Doing an IT Organization Diagnostic Study
“The core content of a strategy is a diagnosis of the situation at hand” – Richard P. Rumelt, Good Strategy/Bad Strategy: The difference and why it matters
In 2023 the IT Strategy team revisited the IM/IT Organization diagnostic study we did in 2021. We re-interviewed executives within our IM/IT organization (called IITB) to see what had changed in the two years since, documented our findings and reported back to our senior management, as well as presenting the results to the Senior Leadership Forum (made up of close to 100 executives and their managers). While we cannot share the details of the report, we thought it might be useful to share our process and planning documents so that other teams thinking of doing similar studies of their IT organizations might reuse this work.
tl;dr
- Get support from the level above those you plan to interview to ensure those you’ll be interviewing will be allowed the time to participate in the study.
- Consult and collaborate with your Values and Ethics team (as well as your HR, accessibility and OL teams).
- Look for data that already exists and don’t duplicate work.
- Consider as a team how to ensure anonymity, psychological, and cultural safety for participants.
- Keep the interviews short to ensure participation. Executives are busy.
- Listen and keep your biases in check.
- Document and validate what you heard with participants.
Scope of our study
For the 2023 study we interviewed 54% of IITB executives (directors and executive directors). We ensured that we had participation from executives in each Division in IITB. Each interview took approximately 1 hour. It took 2 months to conduct all the interviews. For our organizational study we focused on culture and took a human-oriented approach emphasizing barriers to collaboration and empowerment. We were inspired by the Open Source “movement” and research done by the DevOps Research and Assessment (DORA) institute.
Planning
We reviewed the post-mortem document from the 2021 study as well as participant feedback to determine if any changes were required to the process or interview questions.
Since we wanted to compare results between the 2021 and 2023 studies, we only made a few changes to the questions (e.g. questions related to the pandemic were replaced by questions on the Common Hybrid Work Model). The total number of interview questions remained constant at 20.
As with the 2021 study, we cross referenced planned questions against the Public Service Employee Survey (PSES) and removed any questions where there was duplication. We also connected with our HR team, ESDC Values and Ethics, and Public Opinion Research to get their feedback and cover for our approach. We learned from these teams and made adjustments where required.
Communications
We communicated our plans for starting the 2023 study to the Branch DG level committee. We did so to get their support to allow their staff time to participate. Having validation on our approach from other teams (such as Values and Ethics) reassured DGs that their staff would be in safe hands. It should be noted that we did not communicate the study to DGs to get them to force their staff to participate, as for ethical research, participation should always be voluntary.
We then sent an email from our Executive Director to all Executives (executive directors and directors) to advise that the second iteration of the diagnostic study was about to begin and to expect an email from the IT Strategy Team with more information.
To maximize participation, a reminder email was sent by the IT Strategy team 6 weeks before the planned end of the interviews to directors and executive directors requesting their participation.
Scheduling the interviews
Our team sent individual emails to each Executive to invite them to participate. Invitees were informed that this was their opportunity to be heard, and that the findings of the report will be used to inform plans and strategies for the Branch.
Executives self-registered to participate and identified the times they were available using a calendar in a SharePoint site we created for this purpose. During self-registration, participants advised if they wanted to be interviewed in English or French.
All directors and executive directors self-registered were sent an invitation that included the interview package. This package included:
- Why we are doing the diagnostic study
- What’s in it for you
- How it will work
- Confidentiality/Privacy
- The interview questions
Preparing for the interviews
Participants were encouraged to review the interview package in advance and take some time for reflection, to engage with their team members to collect their perspective, and to ensure the best use of their time during the interview.
Our team members conducting the interviews prepared for the interviews by reviewing and getting comfortable with our field guide and interview script.
The script was used so that key items like consent were done consistently, that interviewers did not inadvertently steer participants (due to their own biases), and gave interviewers examples of prompts and background information for specific questions should they be needed.
The field guide contains general guidance on conducting user interviews. Some of this guidance includes (and draws heavily on the work of Steve Portigal from his book Interviewing users: How to uncover compelling insights):
- Go with the flow. It’s ok to not stick rigidly to the order of the questions
- It’s ok to ask probing/follow up questions (follow up questions don’t have to be the same for everyone). E.g:
- Ask about emotional cues: “why do you laugh when you mention the x system?”
- Ask why: “I’ve tried to get my boss to adopt this format, but she won’t do it…” “Why do you think she hasn’t?”
- Probe delicately: “You mentioned a difficult situation that changed your usage. Can you tell me what that situation was?”
- Ask about code words/native language: “Why do you call it the bat cave?”
- Ask the shortest question you can without directing answers.
- Silence is ok. Count to 5 before you say something.
- Give your interviewee plenty of ways to succeed
- They might ask you questions back to you. Try to always bring it back to them….it’s not about you.
During the interview
To ensure the interviewer could give their full attention to the director being interviewed, a second team member attended each interview as the note-taker. The anonymity of the participants was paramount in ensuring we received participants open and honest feedback. No interview sessions, names, or easily identifiable information were recorded in the notes.
Following each interview
At the end of each interview the interviewer and the note-taker would meet for 30 mins to debrief on what they had heard. This also provided a chance to review the interview notes and ensure that both agreed on what was captured. The notes were saved in a private document library on SharePoint that only the interviewers and note-takers had access to.
Reporting and Closeout
Once all the interviews had been conducted, the team met to analyze all the interview notes and identify the themes to organize the content of the What We Heard Report. To protect anonymity, any quotes included in the report were without attribution and data was rolled up to the branch level.
The report included the following sections:
- Executive Summary
- The interviews (methodology, reception, participants, differences between the 2023 and 2021 study)
- Major Themes
- IITB’s Organizational Culture
- Strategic Actions and Recommendations
The IT Organization Culture section specifically highlighted analysis of the diagnostic study results against the Westrum model recommended by the DevOps Research and Assessment (DORA) institute.
In the Strategic Actions section of the report we outlined possible new actions as well as also highlighted some actions already underway (that participants might not have known about) which could address certain challenges raised by participants.
The draft of this report was first shared with all participants to ensure we had accurately captured their comments and to provide an opportunity for adjustments. The What We Heard Report was then used to develop a summary presentation which was presented to three executive forums: first to the DG committee, then CIO Executive Committee, then to our Senior Leaders Forum (all IT executives and managers). We also completed a post-mortem for the 2023 study. In our post-mortems we discuss:
- What went well
- What did not go well
- What changes should we make
During the 2023 post-mortem, we realized that some of the interview questions focused too heavily on IT work and did not do justice to IITB’s diverse workforce (e.g. such as AS and EC). As such, we felt we may have missed important and relevant factors that could have been used to inform IT strategies. To address this, changes will be made to future interview questions. In addition, we will be reviewing the relevance and usefulness of some questions (e.g. questions relating to functions and team size).
Senior executives and participants have told us that they found that the results of the 2021 and 2023 IT organization diagnostic studies to be really insightful and we plan to complete the study again in 2025. If your team is thinking of doing something similar, don’t hesitate to reach out to us at EDSC.DGIIT.StrategieTI-ITStrategy.IITB.ESDC@hrsdc-rhdcc.gc.ca.