Performance and caching

Amplitude Experiment evaluation supports two modes, local and remote. Each mode has different performance characteristics and tradeoffs.

Performance

Evaluation performance depends on the type of evaluation you use, and on where the request originates.

Amplitude hosts data centers in the US (us-west-2) and EU (eu-central-1). Requests to a data center geographically distant from the data center that hosts your Amplitude data experience higher latency.

Remote evaluation

Remote evaluation uses Fastly to cache evaluation results for a user. Cache hits serve variants from the edge, which improves performance.

The following results come from synthetic remote evaluation test requests to Amplitude's US data center, collected over the last six months. Latency includes Domain Name System (DNS) resolution, transport layer security (TLS) connection, and the remote evaluation request response round trip.

Local evaluation

Local evaluation pre-fetches flag configurations and uses them to evaluate all users. This avoids a network request and speeds up evaluation compared to remote evaluation.

The following results measure a single flag evaluation, collected over 10 executions of 10,000 iterations with randomized user inputs evaluated against one flag configuration. Amplitude selected these results at random from three possible flag configurations.

CDN caching

After Experiment computes and retrieves a response for a request, Experiment caches that request for reuse to make future requests faster. Experiment uses a CDN to cache the experiments and feature flags for a user, which provides low-latency access on subsequent requests.

Cache time-to-live (TTL)

Experiment caches requests to the server on the CDN. The cache uses a time-to-live (TTL) policy and expires after 60 minutes regardless of whether the key is accessed within that window. The 60-minute cache window starts from the first page load.

Cache key

The CDN caches the exact request received, including user information. Any change in user info misses the CDN cache unless Experiment cached that exact same request before.

Cache invalidation

To prevent stale results when underlying flags change, Experiment invalidates (deletes) cached results for an entire deployment when you update any flag or experiment associated with that deployment. The SDKs retrieve results for all experiments and feature flags for a given deployment for a user, so Experiment invalidates all results for a deployment whenever a single flag changes. Experiment also invalidates all cached requests for a deployment when you add or remove the deployment from a flag.

Dynamic targeting cache considerations

Experiment lets you target feature flags and experiments based on dynamic properties, such as user properties and behavioral cohorts synced from Analytics. These properties aren't included in the fetch request, so you may receive cached experiment results for up to one hour (the TTL) until the cache misses and Experiment re-evaluates the user with the most recent dynamic properties.

Amplitude user properties

Experiment's remote evaluation servers can target based on user properties that Amplitude previously identified for the user. The CDN caches responses based only on user properties passed explicitly in the request, so the caller may receive stale results for up to one hour, even after Analytics updates the user properties to values that would assign the user to a different variant.

Behavioral cohorts

You may want to use behavioral cohorts defined in Amplitude Analytics in your flag and experiment targeting. Experiment computes cohorts hourly and the CDN cache TTL is also hourly, so targeting a user to a variant may take up to two hours.