Overview
Team: John Sykes, Marianne Kroutchkevitch, William Faler, Jordan Kolasinski
Stakeholders: Hulu Users, UX Design, Content, Subscriber Growth & Retention, Product Management
Role: Lead Researcher
Duration: 3 Months
Methods: Remote Open Card Sort, In-Person Comparative Usability Study, Task Analysis, Directed Storytelling
Tools: UserZoom, Sketch, Zoom, inVision, Framer
Challenge
The addition of a Live TV offering and complete redesign of Hulu’s platforms presented the opportunity to review and potentially change our overall information architecture (IA) across platforms. In order to ensure that our users receive the best experience possible, UX Research and Design needed to understand users’ mental model of how they naturally group both content and navigation options. This not only allowed me to test a long-held hypothesis of mine, but would also help guide designers and product managers in creating a more effective navigation experience.
Solution
The card sort studies highlighted user tendencies to think Genre first rather by video length, opposing how Hulu’s navigation was historically structured. As I hypothesized, groupings created by participants prioritized genres such as comedy, drama, sci-fi, etc. while content length was of secondary concern. Kid’s TV and Kid’s Movies were a notable exception, as participants considered these distinct from each other.
Using these findings, designers created two prototypes which I presented in a comparative study to participants. They were then asked to complete a number of tasks and questions intended to gather feedback around the new genre-first navigation experience. Users generally found the new navigation easier, reducing the amount of time it took them to complete common browsing and watching tasks.
Formative Research
Open Card Sort
Hypotheses
Hulu’s previous navigation, prioritizing Movies and TV along with select genres.
When searching for content to watch using the top navigation, users prioritize the choice in genre over content length.
The current navigation is confusing to users due to inconsistency and unclear information architecture
Research Questions
How do users’ mental models of categorization of content impact their navigation of the site?
How do users naturally group different types of information within our ecosystem?
Are there any unexpected or unconventional content groupings?
Card Sort Studies
Methodology
I conducted 2 unmoderated virtual card sorting exercises to better understand how users think about content navigation and how Live TV would integrate into the new information architecture. In these exercises, users were presented with 50+ content types and asked to create their own categories, then sort the types into each category as they saw fit. I chose an open card sort method in order to make sure participants were given as much freedom as possible and prevent my assumptions around content groupings from influencing feedback,
Using the aggregated data from 100 users, I created dendrograms to visualize the data and analyze commonalities among participants in order to provide designers and product managers insight into how users naturally sort content.
Content Grouping Study
Content Card Sort Dendrogram
The intention of this card sort was to understand user’s mental models around how users search for content.
Participants generally thought along two axes - Genre and content length, which is designated by a TV or Movie label.
As I hypothesized, participants grouped content by Genre first. This was antithetical to Hulu’s current IA and led to direct changes in the upcoming prototypes.
Participants also perceived some unrelated content as similar enough to warrant its own genre- examples being different sports and ‘Foreign’ content (Latino, Asian, British). Kids and Foreign content was perceived as distinct enough to
warrant different groups for TV and Movies.
Resulting Groups
News
Foreign (TV)
Foreign (Movies)
Kids (Movies)
Kids (TV)
On-Demand
SciFi
Sports
Favorites
Personalized
Drama
Comedy
Live TV
Popular/New Movies
Recommended Movies
Live TV and Administrative Card Sort
The intention of this study was to understand how users perception of Live content would differ from On-Demand content. Additionally, we wanted to see how users grouped administrative features liked Account Management.
Participants indicated Live News and Sports were distinct from their on-demand counterparts. Rather than bury them under an aggregate section with On-Demand content, the designers used this information to add the Live section to the top navigation in the prototype.
Category Card Sort Dendrogram
Resulting Groups
Watched Content
Search / Browse
Popular Content
Saved Content
Account Management
Kids
Live Sports
Live News
Concept and Testing
User Acceptance Testing, Task Analysis, Semi-Structured Interview
Prototypes
Utilizing the findings from the card sorts, our designers created a treatment and control prototype meant to validate these findings through a user study and test potential changes like adding ‘My Stuff’ to the navigation, the clarity of the Live TV iconography, and the placement of the Search interaction.
Prototype A: Treatment
Prototype B: Control
Methodology
Findings
Participants for this study were meant to be representative of a ‘typical’ hulu user, ranging in ages from mid-20s up to mid-40s with an even split of genders.
These participants were already active streaming media users, requiring a minimum of 15 hours of streaming media per week on any platform. This ensured that participants would be somewhat familiar with the concept we were trying to test.
In order to prevent status queue bias, I split the pool between current and non-users. To prevent order bias, I counterbalanced the order in which each prototype was presented.
7 of 9 Participants preferred the treatment prototype, citing ease of use as the main factor. Not only did participants find the treatment prototype easier to navigate, but they also perceived more available content despite there being no difference in information presented.
Previous studies by our Data Scientists had shown that perception of content breadth is strongly correlated with user retention, so this finding was well-received by the the product management, growth, and content teams.
“I didn’t know Hulu had cooking shows or anime!”
Task Analysis validated these stated preferences with objective findings. Using selected tasks we developed for benchmark tests, we saw significantly higher completion rates among participants browsing and finding content they would be interested in.
“[Prototype A] is more distinctive as to knowing what you’d want to watch.”
Results and Implementation
Live version of Hulu Web Navigation
Based on the card sort and validating usability studies, the design and product teams implemented genre-based navigation upon release of the Live TV product on Web. Reviews of the new UI specifically mention the Browse menu as a useful upgrade from the previous iteration.
Reflection
My long-held hypothesis around content navigation had been validated, leading to its successful implementation. This was one of the earlier studies I’d performed as a UX Researcher, but it gave me a greater sense of accomplishment than much of my previous work in analytics which tended to feel reactionary. This role felt like I was having a proactive impact on the company, allowing me to shape opinions on what we should do in the future.
Looking back on the study, I would say I over-recruited for the initial cardsorts as 50 participants per study; I think half as many participants would have been sufficient. I also feel that I let the designers add too many variables into the prototypes that were tested. The inclusion of multiple different navigational elements into the test made it difficult to discern which specific changes users were responding to. If I were to do this study again, I would not move the Search interaction as that seemed to cause some confusion.