This article helps you:
Distinguish meaningful fluctuations in your core metrics from those caused by statistical noise.
When core metrics fluctuate, it can be hard to know if those movements are meaningful and worthy of investigation, or just random noise. Amplitude's Anomaly + Forecast feature highlights statistically significant deviations from expected values for your metrics, based on historical data.
This can help you:
Forecast lets you project metrics into the future, so that you can set realistic goals for your team and product.
This feature is available to users on Growth and Enterprise plans only.
Anomaly detection and forecast work with time-series data within the following Amplitude charts: Event Segmentation, Conversion over Time (when counting by unique users for the entire funnel), User Sessions, Retention over Time, and Stickiness over Time.
Within Event Segmentation, it works with rolling windows, rolling averages, growth percentage, and custom formulas that support time series analyses.
Anomaly detection isn't compatible with cumulative time series charts, or charts comparing two different time periods.
If you're using an hourly interval, Anomaly + Forecast supports a maximum of one group-by.
The anomaly detection technique used in Anomaly + Forecast grew out of the extensively tested open source tool Prophet. It's a procedure for forecasting time series data that's robust to missing data points, shifts in trends, as well as large outliers.
You can find the control for this feature on the left hand side, right above the main chart area. If your chart isn't supported by the Anomaly + Forecast feature, the button isn't clickable.
To set up anomaly detection, follow these steps:
Amplitude automatically detects seasonality in each mode. The duration depends on the amount of data used to train the model. Agile mode typically uses daily and weekly durations, while Robust mode looks for monthly and yearly durations. In cases where enough data isn't available, Amplitude may not detect or apply seasonality.
Charts with a single series display a light blue band (confidence interval band) and a dashed line representing the expected value beside the solid blue line that's your actual data. Any anomalies detected appear in orange, outside the confidence band. You can describe an anomaly in this way: "Based on 120 days of training data, we can say with 95% confidence that this data point represents an unexpected change."
If no orange dots are present, all data points are within the confidence interval.
You can run Anomaly + Forecast with multiple series on a chart. Hover over each series to view its confidence band (these appear in different colors as well).
For forecasts, you see the solid lines representing actual data and the confidence intervals until the current date. From there, the forecast displays only dashed lines representing the anticipated future values. Prophet projects metrics by assuming the magnitude and frequency of changes observed in the past are similar in the future, with a certain degree of confidence.
You can describe forecast results in this way: “Based on the trend seen with the last 120 days of data, we're 95% confident that this metric will be between [high value] and [low value] on [a future date].”
Identifying an anomaly is only the first step. You probably want to know what caused it in the first place.
Start by looking at a few related metrics to see if you observe anomalies on those as well. In particular, look at the events that fire before or after the step in the funnel. You could also use group-bys on any properties that might yield more insights into why these anomalies occurred.
A third option is to examine the business context surrounding the anomaly. For example, did a new feature ship that day? Could that have been the cause?
Finally, Amplitude's Root Cause Analysis feature is a powerful tool for tracking down the causes of anomalies.
Amplitude uses different default training durations for different time intervals and modes. In custom mode, you can configure this and add it to the chart date range.
For example, using a daily interval and looking at the last 30 days of data on the chart, the default training data duration for daily charts is 120 days prior to the start of the chart date range. In this case, Amplitude uses a total of 150 days of data to train the model.
In agile mode, Amplitude uses the following default data training durations:
>Time interval used on the chart | Default training duration |
---|---|
Real time | Not Available |
Hourly | 7 days |
Daily | 120 days |
Weekly | 26 weeks |
Monthly | 6 months |
Quarterly | 2 quarters |
In robust mode, Amplitude uses the following default data training durations:
>Time interval used on the chart | Default training duration |
---|---|
Real time | Not Available |
Hourly | 7 days |
Daily | 365 days |
Weekly | 52 weeks |
Monthly | 12 months |
Quarterly | 4 quarters |
The upper limit of training data (prior periods + chart duration) used for each interval is as follows:
>Time interval used on the chart | Training duration limits |
---|---|
Real time | Not Available |
Hourly | 14 days |
Daily | 395 days |
Weekly | 56 weeks |
Monthly | 13 months |
Quarterly | 5 quarters |
If you have a specific training duration in mind that agile or robust modes don't offer, you can set that duration by choosing the custom mode.
If you need alerting for anomalies, Amplitude's Insights package includes automatic and custom monitor alerts. You can find more about the package here.
Thanks for your feedback!
June 6th, 2024
Need help? Contact Support
Visit Amplitude.com
Have a look at the Amplitude Blog
Learn more at Amplitude Academy
© 2024 Amplitude, Inc. All rights reserved. Amplitude is a registered trademark of Amplitude, Inc.