Insight/Action/Outcome: Through an Amplitude error notification analysis, the IBM Cloud for VMware design team discovered that users were getting error messages in the provision experience. By reviewing the access permissions, they could reduce errors and increase conversions on the provisioning page - and they now monitor progress in real time.
It’s not easy being the new kid on the block. As an intern, you want to make a good impression on an employer and find ways to contribute. A great team makes it easy. I joined the IBM Cloud for VMware UX team at our Budapest Lab in February 2022. From the beginning, I saw there were no gatekeepers. I was free to explore the topics I wanted, lead projects, and learn through these activities.
We needed more hard data about users
I learned that there was a need to establish alignment behind the changes we wanted to make for users that could help to support my assumptions as a designer. I recall working closely with our research practice to collect information through user interviews, design and stakeholder reviews, and other qualitative studies. But to prioritize actions and get buy-in on the plans for change with the broader product team, we needed quantitative data, and that required us to improve our structured user interaction data. I first recommended hiring a data analyst to look at the numbers, but my supervisors encouraged me to lead the charge. I wanted us to focus on eliminating the main frustrations for our users, and began to work with three pillars: data, study, and review. After I played back my first findings to our product partners, I got a resounding response to continue. There was recent investment in transforming the user experience for IBM Cloud, and the work I was pulling helped us to focus on the right outcomes.
In an effort to drive changes for a better user experience, I started to work with quantitative data alongside our standard qualitative studies.
The first barrier to working with data was that our products needed some instrumentation housekeeping and clean up. Our naming conventions weren’t standardized, making proper segmentation of our user base impossible. We had seven offerings with many identical properties, so we couldn’t track or analyze event data with good accuracy.
We needed to standardize naming, segment properties and introduce instrumentation across the pages. Once we had resolved that, we could start to analyze and understand the data. Through this process, we empowered ourselves to dive deeper into rationalizing key user experience decisions through data.
A flexible and intuitive solution
I’m not a data scientist, but I took it upon myself to learn about Amplitude and set up our core instrumentation and charts with help from colleagues on other teams. I couldn’t believe how easy it was compared to other data visualization and reporting tools I have used in the past. I was relieved that I didn’t have to load all the data myself or build visualizations from scratch, and I was impressed at how flexible it was. I didn’t have to learn complex workflows to create dashboards that yielded actionable data to advance our decision-making. Amplitude was also decentralized. Anyone could use it, not just the data science team.
I set up charts to monitor conversion rate, time on task, and error notification analysis. When building a resource or buying a product, users are directed to our provisioning experience and must click the create button to continue. We use Amplitude to track what happens at that point.
I encourage my peers to build Amplitude dashboards, and I monitor these specific metrics myself and engage the appropriate persons or teams of any issues I encounter. I share my findings with our product team, and together, we use them to prioritize projects and align their activities with user needs. It has improved collaboration, and has been a way for designers to shape the output of the product team.
Better tracking improves conversions
Amplitude Analytics has made it easier for the design team to spot and correct errors sooner and faster. Design reviews are part of our workflow, and that’s when we would discuss remedying the errors we found. Amplitude supports those conversations.
For example, through one error notification analysis, we discovered that users were getting error messages in the provision experience. We changed our user access permissions and disabled a button, and informed our users about the reasons for the change. This allowed users to work directly with the person with right access credentials to make the provision, leading to fewer errors and more conversions on the provisioning page. The ability to track our progress in real time allowed us to fine tune towards the greatest levels of success.
In the first year after adopting Amplitude, we launched four new offerings, and by understanding key user patterns, we can back our decisions with data and build confidence in how we accelerate product-led growth.
Since we started tracking our core metrics in Amplitude, time on task has dropped by double digits, and the overall user task completion rate has increased multifold. Our usability score has jumped up a level, and our internal experience score has been reclassified and is now on the verge of “excellent”. In the first year after adopting Amplitude, we launched four new offerings, and by understanding key user patterns, we can back our decisions with data and build confidence in how we accelerate product-led growth.
Amplitude boosted the impact of design
Amplitude has become the first step for every design team project. We use Amplitude data to pinpoint user issues, quantify them, and validate them through usability studies. I often ask our researcher to run mini-tests to confirm that the data indicates a specific issue, and we combine our findings to set goals and prioritize projects. Amplitude enables us to pinpoint our problems, validate user assumptions, and enhance the customer experience. The combination of quantitative data and qualitative studies helps us understand the ‘why’ behind user decisions.
My next step is to share everything I’ve learned about Amplitude with my peers. I’m writing an analytics guide with my colleague with the purpose of helping designers overcome their fear of analytics. It introduces Amplitude to designers and walks them through setting up dashboards and charts. It teaches them everything I learned so they don’t have to figure it out themselves. The guide will also codify naming conventions and introduce simple data governance concepts to simplify, streamline, and standardize analytics workflows for the design team.
Amplitude has helped set priorities, justify and get buy-in for new features and allows us to jointly review the impact on user flows, navigation, visual and textual UI elements, and contextual help.
Virtualization is a fast growing market with immense potential as users look to shift their workloads to Cloud. IBM Cloud for VMware is one of the major hyperscalers in the market, but competition is fierce. Thanks to Amplitude, and the instrumentation of our product suite, we can better understand user behaviors and cater to different client needs, through learning support and simplification. Amplitude has helped set priorities, justify and get buy-in for new features and allows us to jointly review the impact on user flows, navigation, visual and textual UI elements, and contextual help.
Aha moments indicate whether we’re on the right track
Amplitude helps us get aha moments. For example, we tested an experimental feature on our landing page that generated two different buttons when a user hovered over the tiles. Through analysis in Amplitude, data suggested that users click both buttons frequently, so we kept both. It's a small discovery, but it confirmed that we were on the right track. Another moment was on our provision page, where a selector panel stayed open with a pop-up message that seemed unneeded for users. Thanks to Amplitude, data suggested we replace the pop-up with a text telling them how to complete the selection. Amplitude helps us as a design team to improve the user experience, and we continue to use Amplitude to monitor UX experiments and test user reactions to new features.
Everyone can make a difference with Amplitude
Amplitude helped me make a difference as a new UX designer. When I joined, I was worried about making an impact and being heard, but Amplitude helps me reason with data, which is an effective way to voice opinions and test assumptions. It leads to a more constructive conversation within our product team, and we can weigh our efforts and the anticipated gains. My feeling is that if I can do it, anyone can. I urge other designers to overcome their fear of data and embrace analytics as a tool that guides decisions and drives results.
