Software Assessment Sharing Pilot - Lessons Learned Report

Introduction

The Problem

GC departments and agencies are individually assessing and certifying the same software assets and this duplication of effort is leading to long wait times for employees requesting software (aka digital tools), and to the risk of software problems only being found by some departments.

The IT Strategy team at Employment and Social Development Canada (ESDC) proposed a pilot initiative to address this problem that would allow GC employees doing software assessments to share their software assessments with one another.

Pilot Description

The goal of the pilot was to reduce the time it takes for GC employees to get a response to their software requests. The objectives of the pilot were to increase efficiency across the software assessment process as well as improve the quality of software assessments across the Government of Canada (GC); and increase collaboration between departments by bringing the GC software assessment community together.

It is also aligned with the GC Digital Standards “Work in the open by default” and “Collaborate widely” as well as the Directive on Open Government “Maximizing the release of GC information”. Finally, Canada’s Digital Ambition 2022 said “Digitally, the GC must operate as one to benefit the people of Canada”.

Pilot planning

At ESDC, four teams (architecture, accessibility, security and privacy) are involved in the software assessment process. Each of these teams participated in the pilot planning process and shared valuable information about the current software assessment process at ESDC.

Legal Questions and Information Categorization

During discussions with the teams involved in software assessment (both inside and outside ESDC), many questioned whether ESDC could legally share the software assessment documents. As such the IT Strategy team sought advice from ESDC Legal counsel and the Regional Security Office (RSO). The RSO reviewed some sample assessment documents from architecture, accessibility and security and found that the documents reviewed were Unclassified. Following the review, Legal confirmed that the GC is one legal entity, and so sharing software assessments between departments was allowed with the caveat that a disclaimer be put on the GCxchange page where the documents were to be hosted.

Sharing platform

When considering what platform would be best for sharing software assessment documents across departments, there were initially 3 main criteria: it had to be a platform that employees across the GC had access to, and it had to meet both GC language requirements and accessibility requirements.

The ability to restrict access to the software assessment documents was later added as a criteria when owners of the software assessment documents at ESDC advised that they would only want the documents to be available to Government of Canada employees¹.

GCxchange was chosen as the platform that would best meet the above requirements² and the Software Assessments group page was developed on this platform for the pilot. Within this group a document library was created that would only be accessible by group members. And membership of the group would only be allowed to GC employees.

Through this Group page, members could see what documents have been shared by others, participate in discussions about software assessments and share their software assessment documents. Only unclassified content can be posted on GCxchange due to security limitations of the platform that existed at the time of the pilot.

Pilot Implementation and Results

The pilot was launched in August 2022 and ran for one year.

ESDC shared 127 software assessment documents for 100 different software tools in the Software Assessments group document library over the course of the pilot as follows:

ESDC helped to demonstrate the value of the pilot by being the first to add their documents and this resulted in 140 members from 42 departments joining the group over the course of the pilot.

However, it should be noted that despite this interest, only one document was shared by another department (Canadian Food Inspection Agency (CFIA)) on the GCxchange group page during the pilot.

Kudos received about the pilot

Positive feedback was received from Treasury Board of Canada Secretariat (TBS), Department of National Defence (DND) and many smaller departments or agencies.

TBS featured this pilot in the 2023–2026 Data Strategy for the Federal Public Service under “Mission 4.Empowering the public service – 4.3 Ensure that public servants are equipped with the appropriate tools to support their work” as a way that organizations are piloting new approaches to improve efficiencies.

TBS invited the IT Strategy team to present the pilot at the February 2023 Open Government Working Group session.

The ESDC Chief Information Officer (CIO) mentioned the pilot in his opening remarks of the November 2022 Senior Leadership Forum (SFL), saying: “This is a great initiative that is getting noticed across the GC and I encourage all the teams involved to continue sharing with other departments for this pilot and also to get involved in the wider GC software assessment community”.

Ripple Effects and Collaboration

This pilot sparked collaboration and experimentation that went beyond just software assessments.

Early in the pilot we spoke with the SSC Accessibility, Accommodation and Adaptive Computer Technology (AAACT) team who were working on their own pilot for sharing accessibility assessments in ACR format across the GC. We agreed that our pilot (the accessibility assessment part) could also lead into theirs since our timelines did not overlap and they would be launching after the end of our pilot. They are now planning to use GCxchange for sharing and collaboration instead of building their own solution.

Another team that we connected with at the end of our pilot was the GC Data Community team from CSPS. They were tasked with implementing Mission 4.3 from the 2023–2026 Data Strategy for the Federal Public Service, where our pilot was featured. Specifically they are to coordinate a common location for departments to share information about the toolkits they have developed. Toolkits may encompass a wide range of resources, including guidance, documentation, assessments, manuals, tutorials, as well as codes and scripts. These resources may pertain to various domains such as procurement, security, privacy, accessibility, and architecture within the realm of data, and may be submitted as hyperlinks or files. Through this work they are hoping to achieve similar objectives to the software assessment sharing pilot (i.e., collaboration, efficiency, and knowledge sharing).

The connections made during the pilot have been used to help ESDC’s ITAM and SAM teams to benchmark themselves with other departments.

Lessons Learned

A survey was sent to all members of the GCxchange group and members of the teams doing software assessments at ESDC. Responses were received from 10 members (7 departments) and from three of the ESDC software assessment teams. The lessons learned are informed by their answers to the survey as well as observations from the IT Strategy team over the course of the pilot.

Lack of standardized assessment criteria across departments and agencies hinders collaboration

There is no standardized process across the GC, including expected information of what a software assessment is to contain. As departments and agencies are left to come up with their own, ESDC’s software intake and assessment process has been commented on as one that is mature and a reference to be leveraged. The team from Fisheries and Oceans Canada (DFO), asked that we share our software intake process with them to inform their process renewal. Most departments don’t have the same capacity and structure as ESDC’s to perform software assessments. Participating departments and agencies mentioned that assessments are typically done by small cross-functional teams, that would really benefit from being able to reference assessments done by larger departments with dedicated or specialized teams.

The Pilot confirmed appetite for collaboration on software assessments and related questions within the GC

As soon as the GCxchange group went live and promoted on GCconnex, GCcollab and LinkedIn, the IT Strategy team received positive feedback about the pilot from across the GC.

As mentioned above, during the pilot, 140 people joined the group from 42 departments further indicating the interest in having a place for those doing this work across the GC to collaborate.

Throughout the pilot, there were 13 requests through the GCxchange Teams channel to see if specific software had already been assessed by another department or to ask what is being used in other departments. Similar questions were also received by email to the IT Strategy team who then tried to make the connections to the right teams to get the answers.

One example of this is a question received from NRCan asking about ESDC’s use of the recording feature in Teams and if ESDC had done a security/privacy assessment on that feature in order to align with ATIP requirements. The IT Strategy team connected NRCan with ESDC’s Security and Privacy teams who were able to respond. Privacy also shared a privacy checklist that others can reuse.

Teams from DFO got in touch after seeing the GCxchange group page to learn more about ESDC’s software intake process. Documentation about this process was shared back with them.

The GC has legal cover to share software assessments between departments

Initially there was a lot of concern from software assessment teams and management about the legalities of sharing these sorts of assessments. Some questions raised included whether sharing these documents would break procurement rules, or whether IITB had authority to perform this pilot.

As mentioned above in the planning section, Legal concluded that as the GC is one legal entity, departments and agencies can share documents and collaborate widely to help each other. ESDC Legal counsel recommended that a notice be put on the site that the information was being provided for INFORMATIONAL PURPOSES ONLY, they reviewed the text that we used for this notice and provided the following checklist for ESDC teams to use before sharing:

• the documents shared are for information purposes only – that is, to provide additional information for departments to consider when applying their own assessments for software usage;
• the results of the shared audits do not disqualify a vendor from solicitation with other departments – that is, applying a blanket negative recommendation for all of GC would have important legal risks; and
• if a particular vendor has a highly restrictive non-disclosure agreement (NDA), confirm whether the NDA provides a definition for permitted representatives of the involved parties and undergo additional evaluation on the type of proprietary information included within these assessments.

Assessment documents are often not categorized or over-categorized

Software assessment documents were sometimes not categorized or over-categorized, affecting the ability to collaborate widely.

Fear of making a mistake is believed to contribute to over-categorization. Through the pilot, some teams expressed concerns around causing embarrassment to themselves or the department, whether it be due to questioning the quality of documents, their information categorization, unintentionally placing ESDC at odds with vendors or the uncertain legal implications of sharing outside the department.

The pilot demonstrated that other departments, especially smaller ones that do not have the same capacity as ESDC, were pleased to have ESDC’s assessments shared in their current formats and that ESDC’s Legal Services gave cover for Unclassified information to be shared with other departments and agencies. Even with a limited number of assessments shared, the feedback has been overwhelmingly positive from the wider GC community.

For the pilot only Unclassified information could be shared as GCxchange does not support higher level of information categorization. Unclassified information is information that could be made available to the public (for example, through the Open Government Portal) without expectation of causing an injury to people or businesses.

From the start, the onus of assigning the right information categorization is on the information’s owner, author or originator. ESDC Information Security teams have made tools available on the Intranet to Determine if your information is Classified(ESDC only) (Confidential, Secret or Top Secret) and Determine if your information is Protected(ESDC only) (Unclassified, Protected A, Protected B or Protected C). The tools guide employees on how to handle different categories of information. All data and information must be marked with the correct categorization level.

As well as the tools available on the Intranet, the Corporate Communications team from the Public Affairs and Stakeholder Relations Branch wrote a 4-part blog series on Information Categorization, that highlights the importance of the process and doing it consistently.

• Are you correctly categorizing sensitive information? (ESDC only)
• Assessing the injury and impact of information (ESDC only)
• Downgrading information (ESDC only)
• Declassification of information (ESDC only)

When documents are not categorized it leads to confusion about if it can be shared or not. If information is categorized as Unclassified it should mean that it could be shared publicly. For the pilot we found that some information that could have been shared was not categorized and that teams were not always comfortable sharing Unclassified information, even if only internally to the GC on GCxchange.

On the other hand, over-categorizing information imposes higher restrictions and levels of protection that hinder information from getting where it needs to be. For the pilot we found that some teams create all their documents categorized as Protected by default. When looking at some of the documents it was not obvious what made them Protected compared to other information that was shared during the pilot.

It could be argued that information contained in some software assessments could be considered internal policies and plans not published to the public (as per the ESDC tool to Determine if your information is Protected) or advice on operations of Government (exclusion criteria under the Access to Information Act and the Privacy Act), that could make the document Protected. Information categorization about software assessment is unclear due to not having information that is personal to an individual (e.g.no SIN numbers or names), or of national consideration (e.g., Classified). More guidance is needed for software assessments with re-assurance to not over categorize information. There is a need to have better standards in place for categorization to better enable information sharing within ESDC, the wider GC and through the Open Government Portal.

Recommendations

Establish GC Standard Software Assessment Evaluation Criteria

Because there are no standards on software assessments for the GC, departments develop their own assessment criteria. When we spoke with departments to request their participation in the pilot and share their software assessments, many doubted that information from their assessments would be relevant to other departments due to them all being done differently.

When surveyed at the end of the pilot, some GCxchange group members indicated that they did not find the documents useful because they found the content irrelevant or unclear. If assessments were done in a standardized way across the GC this issue would likely have been minimized.

The need for standard assessment criteria has already been identified by the accessibility community. They identified that the quality of assessments would improve if all departments were using the same criteria. The SSC ACT team is leading an interdepartmental project to develop standardized criteria and a template for everyone in the GC doing accessibility assessments to use. It would be useful to follow the outcome of this project as other areas involved in software assessment could do the same.

Pilot participants noted that standardization would be useful and gave specific examples of possible standards to adopt such as the Open Security Controls Assessment Language (OSCAL) being developed by the National Institute of Standards and Technology (NIST) for security assessments. It was also noted that software assessments should include criteria from the Appendix J: Standard on Systems that Manage Information and Data part of the Directive on Service and Digital.

During the pilot, participants from DFO requested that ESDC share information about their software assessment process (this had not previously been shared as it was only the assessment documents that were shared on the GCxchange page). ESDC could contribute to the GC’s standard software assessment evaluation criteria by joining whoever is leading this work and sharing ESDC’s standard and related processes whether current or in development. By doing so, IITB could help shape a set of standard criteria and templates for software assessments in the GC.

Additionally, some pilot participants noted that it would be useful to organize the software assessment documents on GCxchange by software rather than by assessment team. In this vein it was also felt that an overall assessment of each software (that includes information from all teams involved) would be useful. ESDC is in the process of developing an overall assessment that will span the software assessment teams.

To enable adherence to any standard criteria, templates or process that are developed, they could be centrally located. This could be done by storing the standard documents on the GCxchange page or ensuring links to those are available on the GCxchange page (e.g.to the accessibility templates which will likely be stored elsewhere).

Until the GC identifies the authority to define such standard, working in the open while reviewing our software approval process to be more service oriented and less compliance focused, would help the whole GC be better at assessing solutions. Continuing to share ESDC software assessments, like was done during the pilot, could have a similar effect of becoming a de facto standard for smaller departments and agencies that reuse the information to make their own decisions.

Provide more guidance on information categorization for software assessment documents

We have obtained reassurance from ESDC’s RSO that a sample of assessments were Unclassified. However, because the responsibility of accurately categorizing information lies with teams that produce the information, we are seeing a need to further provide guidance and assurance to confidently share their work with other teams outside ESDC. This would also help with implementation of ESDC’s Information Strategy 23-26 (ESDC only) that seeks to remove barriers to accessing information by having ESDC employees embrace an “open by design, closed by exception” information culture.

All data and information should be clearly marked as either Unclassified or Protected and access managed accordingly. ESDC does not currently automate the categorization of documents when they are created like some other departments do. When information is not marked, it can create confusion as to whether it can be shared within the GC or on the Open Government portal. When information is incorrectly marked it can inadvertently create barriers to timely access of information in case of Access to Information and Privacy (ATIP) requests.

Use Standard Software Assessment Evaluation Criteria for RFPs

The software intake and assessment process at ESDC involves many teams looking at different aspects of adopting certain software in the department. For example, accessibility is a core requirement for solutions to be approved for both external and internal use. But it is often not a requirement or will be granted an exception when doing a larger procurement (RFP) involving a software component.

ESDC should ensure that RFPs include similar requirements to those of the software intake and assessment process.

Enable More Collaboration

The pilot aimed to promote collaboration between departments and demonstrate through behaviors the intent behind the GC Digital Standards (Work in the open and collaborate widely) by sharing software assessments from ESDC teams and onboarding other departments to start sharing. The initial response from other departments joining as well as the connections made, and the discussions held, show that there is a need for collaboration of software intake and assessment teams across the GC.

ESDC maintains two repositories of software: the Definitive Media Library (DML) and, like all departments, the Application Portfolio Management (APM). For the APM, TBS highlighted the lack of visibility across departments, agencies, or corporations due to the lack of consistent APM solution in the Draft Application Portfolio Management (APM) Modernization CONOPS.

ESDC would benefit from having some of the larger departments (e.g., CRA, DND, SSC) share their software assessments information on the GCxchange group to continue these collaborative efforts post-pilot and promote success to ESDC senior management, which is needed to continue supporting this initiative. Continuing this cross-departmental collaboration will provide much needed expertise and information for smaller departments and agencies to use for their own assessment processes.

Finally, organizations sharing their software catalog with other departments along with contacts for each would enable more collaboration within the GC. That was the idea behind the Open Resource Exchange (ORE), to create a list of open-source software (OSS) published and used in the GC along with contact information of those maintaining it. One of the criteria in the GC Enterprise Architecture Framework is for departments to register OSS being used to the ORE.

ESDC should:

• Continue to share Unclassified documents in the current format that teams are producing them (ex.: Risk Assessment Matrix (ESDC only) documents);
• Make the list of software in its Application Catalogue (ESDC only) and Definitive Media Library (DML) (ESDC only) data available to other departments along with related assessments;
• Add all OSS being used at ESDC to the ORE with a relevant contact to get more information.

Community needs fostering. In the early stages of a community, it is not enough to simply have a sharing space and post documents. The IT strategy did a lot of community building work in terms of connecting people, helping to get questions answered etc. Until the community is established and self-sustaining, this work will be important to continue. Software assessment teams should participate in the community by sharing documents as well as starting and contributing to discussions.

Conclusion

The objectives of the pilot were to increase efficiency across the software assessment process as well as improve the quality of software assessments across the Government of Canada (GC); and increase collaboration between departments by bringing the GC software assessment community together.

The pilot’s objectives were partly met.

The GCxchange group clearly filled a collaboration gap. Prior to the pilot there wasn’t a place for those working on software assessments across the GC to ask questions and share information. The implementation of this sharing space was a clear success in increasing collaboration and even though the pilot is officially over, the GCxchange group will continue.

While the objectives of improving the quality of software assessments and the efficiency of software assessment were not met during the timeframe of the pilot, they could be met in the future if the above recommendations are implemented. Collaboration will also improve with the implementation of the above recommendations as the ability to collaborate is greatly influenced by the standardization of information and the reassurance on personnel to share across departments.

We encourage ESDC to continue leading departments and agencies in sharing and collaborating, even if pilot is over!

Inline references