I have been in the digital analytics world for over twenty years. During this time, I have been on the client side twice, the consulting side once, and the vendor side twice. So I have seen all perspectives on how organizations select digital analytics products. While there may be some things unique to digital analytics, many things I have observed are true for selecting any software product. But since my area of expertise is digital analytics, I will confine the content of this post to this genre.
At the outset, it is essential to state that digital analytics products (or “tools,” as some like to call them) are probably the least important aspect of a digital analytics program. If the product you select is the difference between your success or failure in digital analytics, you are either doing something wrong or have set the bar way too low. I do believe that some digital analytics products are better than others, but the product itself is not enough to make or break success. I also think some digital analytics products “fit” the organization’s culture better than others. As such, selecting the digital analytics product that aligns with the culture or skillset of the organization can help improve adoption and success. But with the right people and processes, your organization can be successful with any digital analytics product. Conversely, with the wrong people and processes, you can fail with the best analytics product.
In this post, I’d like to share how I see organizations approach the selection of digital analytics products and why I think many current practices are flawed. I will also attempt to share my advice along the way. Even though I currently work for a digital analytics vendor, this post will be vendor-agnostic.
Inertia
Despite what anyone tells you, the most likely determinant of your organization’s digital analytics product for the next five years is the one you use today. Inertia (or what I sometimes think is laziness) is very powerful. I have spoken to many organizations who spend much time complaining about their current vendor but always stay with it. Frequently organizations stick with what they have because it is what they know, and change is hard. But inertia often comprises a few sub-themes:
Training
It takes time to train users on a digital analytics product. There is a fear that moving to a new product will require re-training all of your users. I have seen that only a few people use digital analytics products regularly. I think the fear of re-training is often exaggerated, and I recommend that you quantify how many users you’d have to re-train before letting this be a roadblock.
Procurement
It never ceases to amaze me how many organizations use digital analytics products they hate simply because of the work needed to get a new vendor through procurement. Procurement avoidance is especially prevalent in financial services organizations. Too many organizations let procurement (or the fear of procurement) dictate what products they use. While I previously stated that any organization could be successful with any product, if your current product isn’t working out for you and isn’t a people or process issue, you shouldn’t use procurement as an excuse to avoid looking at new products.
Re-implementation
Another aspect of inertia that impacts digital analytics product decisions is the work involved in re-implementing a new product. Of course, no one wants to re-implement a digital analytics product. It seldom goes well the first time, so why would you want to do it again? But nowadays, most of the work in digital analytics implementations is in the data layer, CDP, or tag management system. If you have implemented it well, switching analytics vendors should only involve sending the last portion of the process to a different endpoint. And since most digital analytics implementations are tracking way too much (and a lot of garbage), re-implementing with a new product can be an opportunity to start fresh with a clean slate!
Historical Data Preservation
Some organizations cite retaining historical data, mainly year-over-year data, as a barrier to moving to a new product. Your organization should be backing up all of its digital analytics data in a data warehouse, which you can backfill into almost any digital analytics product. Plus, most organizations don’t use historical data as much as they claim to…
Career Preservation
While this could also be associated with “inertia,” another way organizations choose digital analytics products is career preservation. Many people in the digital analytics industry are “tool specialists.” They have built their career upon the foundation of a specific tool. I should know, as I used to be one of them. For almost twenty years, I was known as the “Adobe Analytics” guy (formerly known as “Omniman” in the Omniture SiteCatalyst days). I knew everything there was to know about Adobe Analytics. If I had gone to work for a company, I would have only chosen ones that used Adobe Analytics since that was how I could add the most value. When I was a consultant, all consulting customers approaching me knew I was the Adobe Analytics guy. You didn’t come to me to have me advise you to move from Adobe to Google Analytics; you came to me to help you improve your use of Adobe Analytics.
While I was the extreme example of this, many people in the industry know only one digital analytics product. If they work on the corporate side, there is a disincentive for them to suggest that the organization move away from the product they know. Doing so would make them less valuable and could lead to losing their job. For this reason, many people overlook the flaws of the digital analytics product they know because it is in their self-interest to do so. However, if these people were good implementors or analysts, they should be confident enough to adapt their skills to any digital analytics product. Knowing multiple products makes you even more valuable in the long run.
Consultancy/Agency Bias
The career preservation issues I just described also exist at the institutional level within consultancies and agencies. Many organizations have relationships with and get advice from consultancies or agencies specializing in a specific digital analytics vendor. While some consultancies and agencies know multiple digital analytics products, I have found that the majority specialize in one or two. That means their consultants only have expertise in one or two products. Therefore, when you work with them, it will likely be the case that they recommend the products they know (the old “if you only have a hammer, everything looks like a nail” syndrome).
Consultancies and agencies should have their client’s interests at the center of what they do, but too often, I see them put their interests ahead of their clients. Sometimes, consultancies and agencies get kickbacks from vendors for referring clients. Google famously did this when GA 360 first came out. Agencies sold it for $150,000 and kept $75,000 of the purchase price to provide “limited support,” the majority of which was, in reality, a finder’s fee. If you brought enough clients to GA 360, you could make a lot of money, and if other vendors weren’t offering the same kickbacks, which product would you recommend?
Feature Wars!
When you evaluate digital analytics vendors, it is common to get dragged into feature wars. Each vendor will show you what features they have that their competitors do not. While it is essential to understand the detailed features of each product, make sure you compare apples to apples and focus on the features you will use. Avoid the charts of features where vendor A can do everything and vendor B can do absolutely nothing! As with data, there is always a way to skew things and tell the story you want. Vendors () are good at that. Listen, and take notes, but don’t make that your main evaluation criteria!
Off-the-Shelf vs. Build Your Own
At one point in your digital analytics journey, you or someone on your team will suggest that you build your own analytics solution instead of purchasing an off-the-shelf analytics product. Every organization goes through the phase where they think they can combine enough open-source tools and save money. While I applaud the ambition, I have yet to see it work out. If you build the equivalent of an off-the-shelf digital analytics product, your team is now supporting its own business and an independent digital analytics software business. If something breaks, it is your problem, not the vendors. These efforts start with great intentions but tend to fall apart when people leave the organization over time. Unless data is integral to your business (e.g., Twitter) or you are large enough to support this (e.g., Amazon.com), you should pay someone else instead of building it yourself. If you can’t get the bug out of your system, I would build just a few parts of the data architecture (e.g., data pipeline, event collector, etc.).
Buying the Suite
Sometimes, vendors will pitch the benefits of buying their entire suite of products. Since we are talking about digital analytics products, you know which vendors offer product suites and which do not. There is nothing wrong with buying a suite of products from one vendor. Real synergies can be attained by committing to one ecosystem (ask any Apple customer). But those synergies come with some strings attached. In some cases, the products in the suite are not the “best-in-breed” for every solution. It would be almost impossible for any vendor to have the best product in 5-6 solutions. But if each of the products meets enough of your needs and you want to go “all in” on one vendor, the suite approach can be the best move to simplify your tech stack and ease the burden of integrations.
But another downside of the “suite” approach is the potential loss in pricing power. Like it or not, once a vendor has sold you multiple products in their suite, they know that changing to other products will be difficult. They have built a virtual moat around your organization. This gives the suite vendor a lot more price elasticity than they would have if you were only using one of their products. Many suite vendors will argue that purchasing multiple products will save you money, but I have heard from many companies that feel like they suddenly wake up one day and realize they are paying way more today than they were a few years ago.
Over the last few years, there has been a bit of an “anti-suite” mentality in the market, which has also hit the digital analytics market. When people say “modern data stack,” that is sometimes code for leveraging many different vendors for different portions of the data stack. Some organizations want to work with the best possible product in each area and feel like the technology landscape has evolved to the point where integrating multiple disparate products has become more manageable. I don’t think there is a right or wrong answer here, but it is worth discussing within your organization.
Customer Review Sites
In the B2B space, there are many product review sites. On these sites, customers rate their experiences with products and share anecdotes about the product. I have found that these review sites often have reviews by people who are mad at or want to suck up to a vendor. The reviews tend to go to one extreme or the other since most people who use a product daily have no real incentive to voice their opinion in a public forum. So I would take these review sites with a grain of salt.
Industry Analyst Reviews
Another source of vendor feedback is industry analyst reviews. Organizations like Forrester, Gartner, etc., meet with vendors and their customers and evaluate vendors on a long list of criteria, and publish those findings in reports like a Gartner Magic Quadrant or . These evaluations tend to be more scientific than the customer review sites mentioned above, but they could be more objective. For these reports, vendors submit the customer references that industry analysts talk to, so you tend to get a best-case view of the vendor. But, what’s some don’t know is that these analysts field customer inquiry calls throughout the year and hear a variety of feedback – good and bad – outside of these major evaluations, that give them a more realistic view of the vendor in question. overall, these reports are a good starting point for viewing where the industry is going and which vendors operate in the same space.
Support
One of the product evaluation factors that is too often devalued is customer support. When you select a vendor, you enter into a partnership relationship with them. But many organizations find that the support they receive from digital analytics vendors is abysmal. Some vendors don’t offer direct support but instead, leverage the consultancies/agencies mentioned above to provide support. That isn’t helpful when encountering product bugs or having important feature requests. Other vendors outsource support to offshore resources that don’t know the product well enough to provide adequate support. If you spend any time in the #Measure Slack group, you will inevitably find a lot of vendor-ranting threads. But most of the time, those who rant about poor support continue using the analytics product for the above reasons.
More organizations should prioritize the support they receive. I’d rather work with a vendor that occasionally screws up but owns up to it and works hard to improve it than one that ignores me altogether. But too often, I think customers set the bar too low and assume that the crappy support they receive is the same type of support they would get from any other vendor.
Price
In the real world, price matters. Some products are more expensive than others. At the same time, I don’t think the price should be everything. I see that many organizations place too much emphasis on price.
In the grand scheme of things, you need people, processes, and products to be successful. If you are doing things correctly, the people and process portions of digital analytics will cost you way more than the digital analytics product. Therefore, how much you spend on the digital analytics product shouldn’t be the deciding factor.
However, there are cases where purchasing an overly expensive digital analytics product can directly impact your remaining budget on people and processes. If that is the case, I recommend picking people and processes over products every day of the week. For example, I would purchase a digital analytics product that can achieve 80% of the tasks you need at 50% of the cost versus one that is twice as expensive and can achieve 100% of what is needed. Very few users will use the 20% you would be missing.
Executive Relationships
No one likes to talk about it, but executive relationships, fancy dinners, and golf outings work more often than you’d think! People love attending fancy vendor conferences, nice dinners, skiing, driving fancy cars on racetracks, etc. If you are a decision-maker for an analytics product at your company, and a vendor treats you well, you consciously or unconsciously reward them with a contract renewal. I have seen many people get sucked into vendor relationships that turn more into friendships than vendor and customer.
I have also seen cases where a digital analytics team wants to use a different product, but their boss’ boss has a long-standing relationship with an executive at the vendor (it often involves country clubs!). That relationship can easily trump the teams’ opinions about the actual product.
How You Should Evaluate Products
So the preceding has covered much of what not to do when evaluating digital analytics products. How would I suggest that you evaluate digital analytics products? While there is no silver bullet when it comes to evaluating products, here is my advice:
- Focus on Culture – Be honest with yourself about your organization’s culture. Does it need a centralized or a self-service environment to be successful in digital analytics? Do people at your organization want to get their own data or would they prefer to file a support ticket and have someone create reports for them? Make sure you leverage a product that fits the corresponding approach. Is your organization a build vs. buy at heart? Don’t try to cram a build approach down the throat of a buy organization (or vice versa).
- Buy For Today – I see many organizations with an “aspirational” view of their organization and teams. They want to be a 9 out of 10 when it comes to digital analytics, so they purchase the product they would use if they were a 9 out of 10, when in reality, they are currently more like a 3 out of 10. That is like paying for a Ferrari when you are barely using all of the features of your Ford! Be honest about where your organization is today and buy the product that will help you today, not a few years from now. You can always upgrade later…
- Evaluate Every Three Years – I suggest you re-evaluate digital analytics products every three years. Even if you are happy with your current vendor, you have nothing to lose by comparing them to a few others and seeing what else is out there. Much of the evaluation work is done by the vendors. If you still like your current vendor the best, you can feel confident that you will continue using the right product. If you find another vendor doing interesting things, you can decide if those features are worth a change or push your current vendor to add those features.
- Focus on Use Cases – It is easy to get sucked into product feature wars (see above). But if you focus on real-world use cases your organization needs to accomplish; you can have vendors demonstrate how they would address those use cases. That helps get you out of a feature-by-feature comparison.
- Leverage Your Network – Find out which organizations use each digital analytics vendor you are evaluating. Then leverage your network to find people you trust who work at those companies and use those products. Then ask them directly how they like the products, like the support, etc. Ask them to demo how they use the product.
- Implementation Modularity – Don’t tie yourself too closely to any one vendor. Don’t put their code directly on your site if you can avoid it. When you implement, build your own data collection stream to control what data you collect and just send the last mile to the vendor. Stay agile. Make it so you can switch to a new vendor in under four weeks.
- Hire Generalists – Don’t focus too much on vendor-specific skills when hiring digital analytics analysts and implementors. Your employees can learn new digital analytics products; you can’t teach people to be smart or team players! Hiring generalists will avoid any bias on your team of a specific vendor.
So those are some of my observations on how organizations currently evaluate digital analytics products and some of my advice on how to evaluate products in the future.