Amen Aiwekhoe

The Project

Grammarly is a cloud-based writing assistant that reviews spelling, grammar, punctuation, clarity, engagement, and delivery mistakes.


Durations: 7 Weeks


Methods: Usability Testing, Semi-Structured Interviews, Surveys, Affinity Mapping


Tools: Miro, Google Forms, Zoom, Figma


Role: UX Researcher


Context

Grammarly's Browser Extension is one of the most widely used products offered by Grammarly. The assistant was recently redesigned to address known issues. The purpose of this project was to investigate the usability of the Grammarly Assistant.


My team specific goals of this research include:


  • 1. Investigate how users interact with the features and suggestions of the Grammarly Browser Extension

  • 2. Understand how different features influence users’ perceptions of the Grammarly Browser Extension


At the end of the project, we collated our findings from the study and provided 16 suggestions as well as recommendations to the Grammarly's design and engineer teams. 



My Role

I was the UX Researcher on this project, my tasks include user recruitment, user interviews, usability testing, and synthesizing results.  I worked with a team of four (including myself),  one UX Researcher, and two other UX/UI Generalists . I was fortunate to work on a team that was collaborative, and reliable.


Research Statement and Goals

To start our research process, I conducted a heuristic evaluation to identify potential usability issues. Based on findings from the heuristic evaluation, stakeholder discussions, and insights from the Grammarly product team, I identified the following research questions:


  1. 1. What features impact user perceptions of the Grammarly Browser Extension as a writing assistant (instead of a Grammar Error Correction tool)?


  1. 2. How discoverable are the Grammarly Assistant features, and do users understand their purpose?
  2. (Features include: Set Goals, Suggestions, Tone Detector & Key Takeaways)


  1. 3. How efficiently are users able to transition between the In-line and Assistant features when editing?


  1. 4. What pain points do users experience when interacting with Suggestions? (In-Line & Bundled Suggestions)


  1. 5. What is the most likely approach users take when accepting Suggestions? (In-Line vs. in the Assistant)


Research Plan

Grammarly provides services to many user groups, including individuals, businesses, and educational institutions.


As identified by Grammarly internal stakeholders, we investigated the following distinct user groups:

After we finalized our target group and research questions we developed a study kit including the screener questions, scripts, moderator guide, post-task survey (SEQ), user interview questions, and usability study tasks.

Research Methods

The majority of our usability testing was moderated through a video conference tool (Zoom) and each participant had a designated facilitator that walked the user through the usability testing process. 

Qualitative


I performed direct observation with Cognitive (think-aloud) testing, in order to collect data on users’ impressions of the system as they complete tasks. This method allowed my team to understand user expectations of the browser extension and to identify what users may find confusing in the Grammarly Assistant tool and features.  After our usability tests, I performed Semi-Structured interviews to help gauge user comprehension of the Grammarly Assistant and provide insights to the initial research questions.

Quantitative 


For this project I focused on gathering metrics to explain our user experience, with the Grammarly Assistant tool, these included success/fail rates (how many times was the user able to complete a task successfully without assistance), the number of assists in completing tasks, and error rates. At the end of each task, we asked participants to complete a short post-task questionnaire via Google Forms. The post-task questions were related to task usability (Single Ease of Use Questionnaire) and the participant’s perception of each feature’s value.

In the post-study interviews, I used open-ended questions to allow users to describe their experience with the tool. It was interesting to learn how experienced premium users did not know all the advantages they had with the tool, and how some new users were now curious to explore and perhaps keep their Grammarly Browser Extension tool


Analysis and Synthesis Process

To communicate the findings and recommendations I synthesized the recording from the usability testing session with card sorting along with affinity mapping, to identify the main themes of our user experiences with the Grammarly Browser Extension. Then I held a team meeting with my stakeholders to walk through the SEQ  and the themes.


Output

The output of the project was a 16 recommendations matrix to improve the Grammarly writing assistant. The recommendations were categorized according to the Dumas and Reddish scale:


Level 1: Prevents Task Completion [Red]

Level 2: Creates significant delay and frustration [Dark Orange]

Level 3: Problems have a minor effect on usability [Yellow]

Level 4: Subtle and possible [Green] enhancements/suggestions


Our usability study identified several opportunities for improvement, primarily driven by low discoverability of key features and interface elements. I presented the findings and recommendations during Grammarly's Design All-Hands, where they received strong support from design stakeholders and informed future product enhancements.



Reflections

After completion of this project, I learned the importance of scoping early on in a project especially when timelines are not flexible. While working with Grammarly there were a lot of ideas the design and marketing team desired to explore and this caused some delays in our planning phases, and we could have done better in being transparent with our stakeholders regarding our timeline.