Case Study Sauce Labs

Case Study

Sauce Labs

Sauce Labs provides a platform of tools for the functional testing of web and mobile applications.

The platform’s main web interface is the Test Results Workflow: a series of screens for locating, analyzing and taking steps to resolve errors found by our four main products.

Each product was built by a separate team, with different ways of displaying and interacting with what was a very similar underlying pattern.

multiple variations of the same pattern

The Challenge

A Broken Primary Workflow

Over its twelve-year lifespan, Sauce Labs’ original product had expanded into multiple products and features built by siloed agile teams. As a result, the user experience for Sauce Labs’ most highly used  feature — Test Results — wasn’t optimal for our biggest cohort of users, QA’s (testers) and SDET’s (engineers) working in test.

For instance: large customers like Walmart ran thousands of automated tests each night, but the Test Results page only showed the last 50 tests. To view additional test results, users had to go to the Archives page.

Additionally, the Insights and Builds pages had useful information that wasn't available on the results pages, nor was it easy to navigate between these helpful views. Users had to remember which test they were drilling into.

the pages in the workflow were not connected well

My role

When I joined as the Head of Product Design, Sauce Labs had undergone major changes. The new executives decided to pivot to a product-led growth strategy, which involved increasing adoption of our tools by developers. 

I hypothesized that an overhaul of test results would result in a better product for both our current users as well as developers, and unifying the disparate UX patterns would be critical to the scalability of our platform over time. 

Additionally, there had been layoffs that included most of the design team. While I led the initiative to overhaul test results, I was also rebuilding the design org, and gradually stepped back as I brought on new hires. 

With all the new eyes and energy directed at our 12 year-old product, we craved a more robust design system. By focusing on Test Results, we would be covering the most challenging and important design patterns first.

Test Results Before Redesign

Test Results Before Redesign

Before Redesign

Add a caption to the picture, pick a color 70%.

A Complication

Who Owns Web Testing?

Looking at our org chart, there was a team to support our new focus on the Developer Experience and there were dedicated teams for each of the different kinds of testing we supported at the time except for one glaring omission...

There was no single team that owned the UX of our original offering - Web Testing. This would have been the most logical team to partner with to improve Test Results.

The same was true for the cross-platform experience.

Everyone agreed we needed to improve Test Results, but none of our agile teams were incentivized to work on it.

The Process

Making it Work

USING the sailboat exercise to assess the current state

Luckily, the members of the DevX (Developer Experience) team became our allies. Their goal was to increase the usage of our product by developers and, like the design team, they knew we had to provide a better workflow using Test Results to track down bugs.

Their idea to "just make it like Cypress" was not completely off track: we certainly studied Cypress as well as other tools developers use every day.

But Discovery involves more than looking around, it means looking within, not just to capture the "as is" state of the product, but to understand how it got that way in first place. We used the sailboat exercise which helped turn us away from comparing our product to other products, which can only get you so far.

Cross-Team Collaboration

With the DevX team we were able to build enough momentum for the work to be spread out among the other teams. But the initiative still ranked low in priority. I would continue to provide project leadership to make the work easier for the teams to step up and commit. This work involved:

Visualizing the work so I could effectively communicate the scope and coordination needed

Discussing it regularly with product and engineering leaders

Building enthusiasm: calling it “Platform Love” and presenting widely our goal to elevate areas of the product from minimally viable to maximally loveable.

Coaching my designers to get buy-in from their teams: insisting that they use only the newest patterns in all mockups, regardless of each team’s willingness to implement them.

We eventually got that “Web Platform" team, but until then, Design was the only group in the Product Org that met regularly and discussed cross-platform UX issues.

Cross-Team Collaboration

With the DevX team we were able to build enough momentum for the work to be spread out among the other teams. But the initiative still ranked low in priority. I would continue to provide project leadership to make the work easier for the teams to step up and commit. This work involved:

Visualizing the work so I could effectively communicate the scope and coordination needed

Discussing it regularly with product and engineering leaders

Building enthusiasm: calling it “Platform Love” and presenting widely our goal to elevate areas of the product from minimally viable to maximally loveable.

Coaching my designers to get buy-in from their teams: insisting that they use only the newest patterns in all mockups, regardless of each team’s willingness to implement them.

We eventually got that “Web Platform" team, but until then, Design was the only group in the Product Org that met regularly and discussed cross-platform UX issues.

using miro to visualize the work and connect it to Jira

using miro to visualize the work and connect it to Jira

The Solution

What We Achieved

Defined a “results review” workflow that could be used by all current and future products of our platform

Defined a “results review” workflow that could be used by all current and future products of our platform

Improved the logs and surfaced hidden metadata important to developers

Improved the logs and surfaced hidden metadata important to developers

Made a single Test Results page that was robust enough to sunset the other pages

Made a single Test Results page that was robust enough to sunset the other pages

Made the design system WCAG-compliant and used React Aria to provide context and keyboard navigation.

Made the design system WCAG-compliant and used React Aria to provide context and keyboard navigation.

Uncovered deeper issues with our object model and kicked off an an initiative to change it

Uncovered deeper issues with our object model and kicked off an an initiative to change it

Revised Design

Revised Design

Revised Design

Results & Impact

Initial improvements to the Test Results workflow resulted in a doubling of usage among new users. The components were successfully adopted by 4 more of our products within the next year and Archives was finally sunsetted in Aug 2023.

Due to the sudden, severe contraction of tech investment in early 2023, enterprise QA users became a top target for Sauce Labs again.

I believe the Design Org’s focus on improving the QA workflow set Sauce Labs up for a successful pivot back to our most profitable value proposition while improving the foundation to support future innovation and growth. I laid the ground work for using Jobs to Be Done and other frameworks that has helped Sauce Labs address the needs of an expanded audience of test orchestrators, engineering managers, and DevOps leaders and admins.

Results & Impact

Initial improvements to the Test Results workflow resulted in a doubling of usage among new users. The components were successfully adopted by 4 more of our products within the next year and Archives was finally sunsetted in Aug 2023.

Due to the sudden, severe contraction of tech investment in early 2023, enterprise QA users became a top target for Sauce Labs again.

I believe the Design Org’s focus on improving the QA workflow set Sauce Labs up for a successful pivot back to our most profitable value proposition while improving the foundation to support future innovation and growth. I laid the ground work for using Jobs to Be Done and other frameworks that has helped Sauce Labs address the needs of an expanded audience of test orchestrators, engineering managers, and DevOps leaders and admins.

Increase

In usage among new users

Increase

In usage among new users

Time Spent

By internal teams

based on feedback from eng leaders

Time Spent

By internal teams

based on feedback from eng leaders

Positive Feedback

On our UX

based on win-loss analysis by Clozd

Positive Feedback

On our UX

based on win-loss analysis by Clozd

More Products

Adopting the improved patterns

More Products

Adopting the improved patterns

Conclusion

Inspired by fixing a workflow that mattered to the largest cohort of our users, I successfully brought disparate teams together to implement this and other cross-platform initiatives.

When leadership changed in mid 2023, the new executives turned to the design team to provide a Meta Analysis of our users, their journeys and the future possibilities of our platform. I was gone at the time, but I'm proud of the fact that my designers could convey this in a way no one else in Product could.

In my mind, it was proof that I had succeeded at my goal of developing designers as business-informed, product-thinkers, who can communicate the big picture while sweating the details.

About Brenda

I like to say that I am passionate about making high quality professional tools because bad software can make you really hate your job.

I like to say that I am passionate about making high quality enterprise tools because bad software can make you really hate your job.

I started working in software as a QA tester part time to support myself as a professional dancer. Testing led to coding, prototyping, animating and writing for software products as well.

I started a design agency with fellow UX designers and ran that company for over 15 years. Clients included Walmart, Verizon, Autodesk, Hyatt Hotels, Cox Automotive, Applied Biosystems, Sony, Intuit and PG&E.

These days, I bring my years of broad experience to lead design “in house," where my penchant for improving systems and communications can impact not only the end users, but the teams creating those impactful experiences as well.

My team at Sauce Labs had low turnover and gave me the highest score on employee satisfaction surveys.

Brenda's Leadership Philosophy

Brenda's Leadership Philosophy

Brenda's Leadership Philosophy

  • Everything I do should be in service of someone else

  • Assume positive intent

  • Focus on what we can do versus what we can’t

  • Time and psychological safety are vital to the creative process

  • So is a sense of humor

  • When speculation is put forward as fact, ask “is that true”?

  • Give recognition and thanks often and in public

  • Help my team grow as professional software makers and rising leaders themselves

Thank you for visiting my website. I look forward to hearing from you.

"A key contribution of design is creating a context in which people can think.

Alongside final deliverables, clarity, insight and inspiration should be regularly delivered as well."

— Brenda Stine

Brenda Stine

Product Design Leader

Copyright © 2024 Brenda Stine | All Rights Reserved