As ABS-CBN’s head of data analytics, Robbin Brillantes has been leading the organization’s implementation of Amplitude–and an overhaul of its relationship with data. “Amplitude has completely changed” the way even non-technical employees can understand and use data, and revealed important insights into user behavior, she told Amplitude Product Evangelist Adam Greco. Read more of their conversation to discover how Robbin Brillantes has helped forge a culture of data at ABS-CBN.
Adam Greco: Tell me a little bit about yourself. What’s your role? And tell me a little bit about what your organization does.
Robbin Brillantes: I’m currently the data analytics head for ABS-CBN. ABS-CBN is the largest media broadcasting company in the Philippines. And it’s also the oldest television broadcaster in Southeast Asia. With that comes a tradition of excellence in content creation, broadcasting of news, and launching the most popular and loved entertainment programs for Filipinos in the Philippines and all over the world.
AG: And which department or which team are you on?
RB: I’m on the data team. Over the last year, we’ve onboarded Amplitude onto a slew of our digital products. Coming into 2020 with the unfolding of the pandemic and some unfortunate circumstances for our company, our leaders’ bet on a full-on digital approach turned out to have come at the right time. Getting Amplitude onboarded set us on a path of figuring out how to best invest our time, resources, our energy, and efforts into making sure that our products remain relevant to our consumers’ changing tastes and behaviors. Looking back at where we started, pre-Amplitude, and where we are today? It’s been a great ride.
AG: Before you started using Amplitude, what were you using? What other technologies were you using to understand how people were using your website or your mobile app?
RB: For digital products, it was largely Google Analytics. To some extent, there was also some work using big data (instrumenting all of these websites, gathering the data from the platforms). But in terms of finding actionable insights from the massive data that we were capturing, nothing’s brought us as far as Amplitude has.
And in the same vein, while we had all of this data in Big Data and Google Analytics, there were only ever a handful of people in the organization who knew how to use tools to access and make sense of them. Some people had access, but only a few really knew how to read and interpret the reports. Amplitude has completely changed that as it allows non-technical people to look at the data, digest what it means, understand what the users are going through, and gives them actionable steps to take to actually drive improvement in the metrics that matter to them.
AG: So what do you think was the biggest difference in usability between Google Analytics and Amplitude? What made the difference in people finally being able to self-serve in Amplitude where they weren’t in Google Analytics?
RB: I think with Google Analytics, where it tracks everything out-of-the-box, there’s wasn’t anyone from Google we can approach and say, “Hey, can you tell me if I’m reading this report right? Or is there a way to refine how we’re capturing this so that we understand these new developments we’ve made?”
With Amplitude, because it’s event-based, your instrumentation depends on the metrics that matter to you. And that happens within my team — we instrument all of the products, which, firstly, entails speaking with the stakeholders, and understanding what their North Star is so that we would know what we should prioritize for tracking. Once these events are implemented, it’s then easier for the teams to use the data as they know what each event means, what the movement from event A to event B means for them, or how quick the conversion time is, and so on.
AG: And how many people typically use Amplitude on a regular basis?
RB: There’s around maybe 350 in the org that have access to Amplitude at the moment. Among these users, about half of them use it two to three times a week. Since using Amplitude, we’ve more than 10x’d the number of employees accessing data on their own. This didn’t happen overnight though. We had to train teams regularly. For the better part of Q4 2020 until the first half of 2021, I conducted weekly analytics sessions on Fridays where it’s an open Q&A format. Each week I’d set aside some interesting charts to share with the teams, call out some insights, and weave the stories behind them. We had a mixed audience from product, marketing, engineering, content, sales, ad tech, of around 60 employees coming into this weekly Q&A. So when I talk about what we’re measuring now, or highlight new growth areas, I’m able to call out some people and ask them, “Hey, I saw this from your product. Very interesting! Was this spike expected?” “Can you tell us about this marketing effort?” etc. I think this avenue to discuss and ask questions helped get the ball rolling as teams saw how empowering the data could be across different teams. This collective curiosity – we didn’t have this culture back when we were using Google Analytics.
AG: The cool part is that you were showing them data in context, which helps users understand it, and motivates them to learn how to use it themselves. Does your team act as a centralized hub for analytics where you do the implementation and trainings?
RB: Due to our limited headcount at the moment, it is. But as our products scale and as the Data Team’s initiatives expand, I foresee us moving towards a hub and spoke model where the instrumentation will still be managed by our team, and we’ll have dedicated analysts embedded in other product teams so they have the full context of all the plans, ongoing efforts, challenges, opportunities, etc.
By having dedicated analysts within those groups, we can look after teams and ensure that they are looking at meaningful metrics, and that insighting is practiced. Essentially ensuring that teams use data to inform action and decision-making, shifting away from the typical legacy monthly reporting.
AG: I know there are probably a million things that you’re doing with Amplitude. What are some of the top use cases for your organization?
RB: The main product, the largest product we have, is a streaming platform. So in the past, without Amplitude, we had no data tied to behavior, whether it was what content drives higher minutes watched or what marketing content attracts the most valuable users. All we had were transactional data giving us summary figures giving us a sense of total users, total views, and basic dimensions.
Now we’re able to talk about how our different users from around the world behave, who our most valuable users are and what actions have the greatest impact on their experience. From a QOE standpoint, we can use data to assess video quality, like when videos are buffering or stalling before customers complain, allowing us to identify the size and scale of issues to inform what we should work on now, pursue as a long-term fix, or if its an edge case, document and move on to something else. An interesting and recent use case was for analyzing the impact of subtitles on content viewership, user growth, and engagement. From the data, we were able to observe how subtitling a handful of content in a certain language boosted their performance in, surprisingly, many other territories. This has sparked curiosity around how subtitling, in general, not only strengthens content and growth potential in its native market. Subtitling, when done correctly, can help us open up more markets to bring the best of Filipino content to the world. This is one example of how data allows us to pursue new opportunities we may not have known even existed.
AG: One of the trends that we’re seeing now, is that you have the ability to see how you acquired people, and connect that with what they’re doing in your product. A lot of times in Google Analytics, I found that initially, you may get a lot of people from different channels, but you can’t really measure the downstream value of those people. So you might keep plowing money into campaigns that are getting you people but not actually making you money..
RB: Definitely. It’s one thing to know which channel brings you the most users, but the ability to tie traffic source to behavior data is where the gold is at. To that marketing ROI point, right now, we’re able to capture ad events and enrich them. When a user sees an ad, we’re able to define and capture that as an ad event. We’re enriching these ad events with data from Google Ad Manager. So even in markets where our OTT runs on an AVOD model, we’re able to attribute ad revenues to a specific segment of users, a cohort of users from a specific channel or platform, and so on. Looking at these ad revenues against a marketing budget for a particular campaign, we can see the total revenue generated by users from that campaign, even when it’s not driven by an outright transaction or subscription. This capability is very relevant for any OTT. Being able to measure marketing ROI and ad revenues enables us to try strategies where AVOD might work in some markets while an SVOD model might be what works for the United States and Canada. This provides an additional lens that helps the marketers see the return on their investments, and in turn make better bets as they go.
AG: How did you do that? Are you pulling in data from AdWords as events? It sounds like you’re seeing impressions and clicks from ads. How does that work in Amplitude?
RB: We work with Datazoom’s video collectors which get sent into Amplitude. Essentially, there are video events that they capture from the player as users watch content. For any of these video events, we are able to send custom metadata as well as declare custom events on the client. So when we a user sees an ad, Datazoom enriches the ad data with different attributes available from a fresh data pull from Ad Manager.
AG: Then does that data also have the marketing/advertising cost information?
RB: That’s the piece that’s separately handled at the moment. Information on actual marketing spend for campaigns sits with our marketing guys. When putting together marketing ROI, marketers have to input the spend details accordingly.
AG: That’s actually one we’re working on. We want to be able to help you get that into Amplitude as well. The other thing that it sounded like you were mentioning is that you were able to experiment and learn new things about what’s happening with experience. You mentioned what you learned from studying the impact of subtitles. Do you have a process or a strategy around every week trying to learn new things out of Amplitude? And how do you document that, and how do you make that part of your kind of culture?
RB: Apart from the every Friday Q&A session that I used to do, we also hold twice-a-month sessions with the different teams, ideally grouping them with another team to diversify perspectives and stimulate curiosity in our discussions. In each session, teams share something they’re working on and how they used Amplitude, or they show a chart they’re using to prepare for an ongoing initiative. They’d be given homework for the following session, usually a chart or notebook, and share it with the group. This was a good way to give them helpful feedback on how else data could be put together for a more compelling story or to really drive home the point.
At the same time, we challenge them given the data that they share with us. So if we, for example, see that the trend in marketing is that they aren’t acquiring users that stick around, or users that join only visit the platform but don’t perform any viewing event, we lead the team down the path of discovering what are these users are doing on the platform and compare their behavior to those of our power users from previous campaigns. We try to identify what works, what are their similarities and investigate further.
For the marketers looking at subtitling, I think having the data and these learnings, is incredibly helpful when they ask for their budget for subtitling and promoting the availability of subtitled content. The data is put together, insights are surfaced, and they can confidently share their recommendations because they know that it’s backed by data. And once action is taken, they can measure the impact of their campaign, and see opportunities to tune the campaign.
AG: It sounds like they’re using some of the collaboration features that are in the product such as notebooks. Have the notebooks been helpful to storytelling as they go through this process?
RB: Definitely. It’s also reduced the digital waste of having to take screenshots, put them in PowerPoint decks, and so on. By using notebooks, we’re able to write the story and present the charts that best express the key insights we want to share. In many teams, these notebooks have replaced presentation decks. So for those receiving these notebook links, and it’s mostly upper management, we also do sessions to get them familiarized with this new format in presenting insights and ideas. The key element is that they’re interactive. Using it in a meeting, showing the interactive elements, encourages more curiosity and interest from attendees, and invites more questions. Meetings are made more engaging as we can query the data and get answers on the fly. We’ve never had that in the past. Pre-Amplitude it would take you maybe three days up to two weeks to get answers to follow-up questions. Now, using Amplitude, if you ask me to look further into the U.S. or the California market to see how many users we have there, then I can actually show you in real-time what that looks like. I think that’s been spectacular for everyone to see.
AG: I’m curious, there’s a lot of automated insights in Amplitude, with insights based on your data. Are you using any of the features that automate some of the analysis, or the ones that try to nudge your analysts in a certain direction?
RB: At the moment we’re using automated insights largely for detecting anomalies.
AG: And how is that used?
RB: For the OTT platform, I would have real-time and hourly monitoring charts for the key parts of the user funnel, for example, registration flow. The registration/login hourly charts would have alerts set-up to fire based on a specific deviation from the historical mean. Having those alerts set up and being able to investigate using root cause analysis gives me an array of breakdowns according to the different, most-queried user and event properties. It’s a quick way to detect what’s going on and then discuss with the teams managing how to address them.
AG: Yeah, that’s a pretty powerful feature. That plus cohorts is pretty powerful in terms of seeing the different groups of your users.
What other tools are in your data or MarTech stack? Where does Amplitude data fit in the big picture? Are there other systems? Are you taking cohorts of users and sending them emails with another product? Are you sending Amplitude data to a data lake? Are there any other data points that you’re pulling into Amplitude other than the advertising ones we talked about?
RB: We do create audiences using the cohorts and send them to our S3 bucket. And from there, we prepare them and send them to our DMP so that we can target users based on these audiences, first-party interests and demographics, among others.
AG: So in terms of the Amplitude product itself, which kind of reports or features would you say are your go-to ones, the ones that you use most often?
RB: I think the most often would be event segmentation and funnels. I think event segmentation is the most friendly for the average user. It answers the question: what’s happening? How many users saw this? How many users registered? And so on. The natural next step is using funnels to understand what is the conversion between a user doing this event A and event B or the next thing we want them to do.
Some of our teams are already using lifecycle charts to understand users coming and going to the platform. For someone who is looking after the free segment or paying/subscribed users, they’d want to see that lifecycle chart on a weekly basis or on a monthly basis to find answers to questions like “How many new users are on the platform this month?” “How many of the current users in the month become dormant in the next month?” And so on.
AG: Do you use your trainings to let your Amplitude users know about the many other options the platform offers? We’re trying to figure out how to get people to use the other ones more often.
RB: Absolutely. In our training sessions, if there’s an opportunity for us to show them a better chart to answer the same question or take their question further, we show them an alternative chart type that does just that.
AG: I wanted to talk about our user profile. In Google Analytics, you can’t look up a particular person and see every event they did, or merge any different devices that they used. I wondered, how does the user profile in Amplitude help over what you had in Google Analytics, and what are some of the ways that you’re using the user profile?
RB: The user profile is a large differentiator for Amplitude versus Google Analytics. There’s no sense of a single user’s experience in Google Analytics. While for Amplitude, you can look up the stream for an individual user, see how many devices they’re on, and what events they have triggered throughout their session. For example, we have a retail offering where users pay for content that’s available to watch only for a 48-hour window. Occasionally we get feedback from customers saying that they aren’t able to access the content that they paid for and they’re worried they’d have to pay again. Using the user look-up, our customer service team is then able to look up the specific user’s stream and analyze the sequence of events they generated. When was the payment done, what was the payment gateway, and whether the user clicked the content and succeeding video events fired.
This highly detailed user-level view gives the customer service team the confidence to better handle feedback and manage the kind of response or action to take. In the end, that’s also very good for our customers, because reducing guesswork results in faster issue resolution.