Avoiding Data Governance Pitfalls

The first thing you need to uncover is how people are letting you know things need to change or if you have to discover it on your own.

Every organization has a different way of discovering what needs changing.

For us at Appfire, a common example might stem from a conversation with an analyst interacting with the raw data downstream. They might give feedback that something isn’t working as expected. Another example would be if a senior business stakeholder interacts with some data product the team has made elsewhere but not the raw data. They might request a new dimension to aggregate.

Anything like this is an input.

The big thing that we found that has been critical is to try to organize and capture the input in the rawest possible format. The aim isn’t to solve inputs straight away but to clarify what they are and get a good view before reconciling them later.

Find out the priority level of requests early. Some people might have a simple ask, unaware of other requests being made. A unified model can help determine a job’s priority.

Once feedback items are collected, it’s important to allocate time to think about them. That’s not just looking at individual items but reconciling your lists.

Let’s say you’ve pushed instrumentation, feel good about it, and don’t feel any changes will need to be made for a while. Feedback begins to trickle in and, eventually, needs reconciling. Could this be making a change or adding something new?

It’s also important to make sure the feedback received doesn’t conflict with each other. Determine if all things fed back can be reconciled and implemented into a clean pass and that you can communicate what will be changed.

Then you want to socialize your proposed changes. Go back to the stakeholders and tell them what changes need to be made and why. This gives you a second opportunity to get feedback, letting you know if the change could impact other areas.

Lastly, you must test where possible.

Testing before implementation can save time. Much of what is recommended for a change must go through ideation, have a firm specification, and be passed through the development team to roll out.

If you haven’t tested something that could have been easily validated and comes back wrong, it’s a waste of a cycle. Developer resources are precious, so it’s important to mock things up before proposing and then run it through Amplitude. Amplitude makes this really easy, you can test properties, pull it into a chart, make sure it all aggregates the way you think it's going to be, and then have that included in your documentation when you're ready to roll it out.

For us at Appfire, with hundreds of employees, documentation is a crucial part of the process.

Once there is a plan of changes the team knows we need to make, and stakeholders have validated them, it is written down verbosely. We detail exactly what is going to change and how it needs to change, including as much reasoning as possible in the documentation to explain why the changes are happening.

Detailed documentation creates a nice summary of all the work that has been happening over a given period. Anyone who reads it, from a technical stakeholder to non-technical business stakeholder, should grasp what the change is and why it’s happening, even if they don’t understand every component.

Over time, a change management log is created. Team members can refer to this log as a reminder of the reason for changes, removing ambiguity. I keep finding myself going back to and referencing our log - it’s a good way to remove the ambiguity "why did we do this, again?”

If you’re not already going through a process of collecting feedback, evaluating the feedback, and making a decision from that communication regularly, it builds up or gets lost.

With a regular review cycle, you can make sure that doesn’t happen and it’s an opportunity to reliably communicate what you’re up to with the rest of the organization.

At Appfire, we agreed a quarterly review made the most sense for our work. A window of this size enables us to collect changes, figure out exactly what to recommend, and then get it to the development team to implement. Any faster than this period would have induced stress among the team, while slower might have made it difficult to capture feedback as fast as it needed to be implemented. A smaller organization might do monthly or even weekly reviews — it’s all about what’s right for your organization.

Committing to a quarterly cycle can be communicated via a calendar so teams know when to expect changes. They know what is coming and when so they can factor this into their own planning.

If you find yourself in a data governance position where you are representing things to other parts of the company, heavily documenting and organizing information up front means you don’t have to worry about stakeholder expectations. You have an answer ready, as it’s already been planned in the cycle.

Let’s say there is something you want to work on but don’t have the necessary components to deliver at present. It can instead be forecasted within a future cycle when you come to release product changes.

Our approach started with a documentation rework, getting a handle on it before it expanded too much.

We agreed to points with stakeholders for what they wanted to accomplish with the changes. The documentation had to be functional for multiple stakeholder groups, including the most technical teams doing active instrumentation and broad business stakeholders who needed a more general update.

We also wanted to ensure documentation didn’t need to be written in multiple places. We knew that we were gonna be producing a lot of documentation so we didn't want to kill ourselves with having to maintain it in multiple places.

This led to the creation of a ‘core data taxonomy’, essentially a rebranding of its documentation. However, it also offered a structured approach to maintaining the taxonomy information for Amplitude instrumentation.

Within taxonomy, every event, property, user property, and functionality is used and treated like a similar-level object with a clean documentation page. That documentation includes metadata and definitions stakeholders might need. We can syndicate that information and produce views with more or less information depending on the audience, for example, senior stakeholders vs. technical teams. Plus, if we need to make a change, we make it in one place, and it gets updated everywhere.