Project summary: Citizenship test optimization project

The Digital Transformation Office (DTO) and Immigration, Refugees and Citizenship Canada (IRCC) collaborated on the citizenship test optimization project in the summer of 2018. In this project, the team sought to make it easier for people to prepare for the test that most new citizens must take to complete their application for Canadian citizenship.

A team of content designers, design researchers and interaction designers worked with IRCC to define a set of 6 task scenarios related to the citizenship test. Using these scenarios, we completed a total of 214 tests with 36 participants. We tested the existing pages and then re-tested our prototyped improvements. Overall, task success rose by 17% percentage points - to an average of 86% - on the revised prototype.

Establishing a baseline

We selected 19 participants to test the existing pages using the 6 tasks that our team had designed. Our team was trying to measure:

  • findability: how often people successfully found the page on the website with the answer to the task question
  • success rate: how often people correctly identified and understood the correct answer to the task question

We generated real-world scenarios that reflected what people are actually trying to do. For example:

  • When I applied for Canadian citizenship, I was informed about the test. I want to study for it, so I can pass
  • When I think I’m ready for the test, I want to find out when I can do it so I can plan the test
  • When I am less confident of passing the test, I want to know what happens if I fail so I can understand how I can become a citizen

Redesigning to make it easier for people

Using the results from the baseline testing, the team designed a prototype with task flow in mind. We wanted to simplify the information about the steps after applying and make it easier for applicants to use the study guide. We clarified topic labels and reduced the number of options on the page to eliminate distracting clutter. By using verbs and clearer language in the links and descriptions, the team made it easier for users to discover the right path to the information they needed.

On the live site, the steps to becoming a Canadian citizen were not numbered or listed sequentially. This caused confusion for users. In the prototype, we numbered the steps and grouped them together so users would understand the sequence in which they should be done.

Sequential processes and steps

We clarified and numbered the steps involved in applying for Canadian citizenship. This made it easier to understand and follow the process.

Study guide navigation

We experimented with a new long-document pattern. We added a “search within” and a left-hand navigation that shows all other chapters. Users were able to easily search the specific chapter and see at a glance other relevant chapters.

No distractions

We removed inline links to help users stay focused on the task. This way they were not distracted by links to additional information. We answered questions they might have directly in the content instead of linking away from the page.

Here is one example of how we reorganized information:


The study guide showing the new long document pattern.

Detailed description

Before:

All the information for this chapter (chapter not numbered) was on one page. The only way to search was to use Ctrl + F. Users would commonly get lost when they began to scroll.

After:

By adding a “search within” and a left-hand navigation, users were more successful at navigating within the document and finding what they needed.

Measuring success

Success! Our target was to improve both findability and task success by a minimum of 20 percentage points, or to exceed 80% for both measures. We re-tested the same 6 tasks on our prototype with 17 new participants. Successful task completion increased by 17 percentage points, to a total of 86%.

This chart shows the success and findability rates of all tasks in baseline and validation phases:

Task completion success rates – table

Baseline measurement at start of project, validation on prototype redesigned by project team.

Task Baseline Validation
1. Steps to citizenship 68% 63%
2. Study methods 90% 100%
3. Language skills 48% 82%
4. Canadian flag 48% 82%
5. Inuit meaning 74% 88%
6. Take which documents to the test 89% 100%

What we learned

Users succeed better when we:

  • present step-by-step processes in a clear sequence
  • include search and navigation features within long documents
  • eliminate distracting links within content
  • use plain language and verbs to make labels clearer

Request the research

If you’d like to see the research findings from this project, let us know. Email us at dto.btn@tbs-sct.gc.ca.

Let us know what you think

Tweet using the hashtag #Canadadotca.

Explore further