How to backfill historical events with HTTP V2 in Node.js

Import a small historical event file with original timestamps and stable event IDs. Add a dry run, a project-bound checkpoint and bounded retries, then verify a real duplicate replay.

  • Node.js
  • JavaScript
  • HTTP V2
  • 25 min

Chris Tam · Last updated 13 November 2026

Product analytics can include past application activity when an import preserves its event times and identities. I'll build a bounded Node.js importer for a synthetic JSONL file with three report actions from an hour ago. It sends through HTTP V2 using an existing test-project ingestion key; no source service or external account is needed.

What you'll build

A small importer with an explicit dry run, batches of at most ten events, deterministic insert/device IDs, a file/project-bound checkpoint and a single-writer lock. A second invocation resumes without resending; a separate test checkpoint replays the same events to verify deduplication.

Prerequisites

  • Node.js 24.14 or later and Corepack for pnpm.
  • An existing Amplitude test project, its ingestion API key, and its US or EU data center. Open Organization settings > API Keys and select the test project.

No additional service account is needed. The app runs locally.

No Amplitude account yet? A free account is enough for this tutorial; nothing here needs a paid plan. Sign up for free to get your project API key.

1. Create the standalone project

I'll use Node's built-in fetch and filesystem APIs with the HTTP V2 endpoint. This example has no application package dependencies; the package file pins the command runner used for testing.

Terminal
mkdir nodejs-historical-backfill
cd nodejs-historical-backfill
mkdir -p src tests
corepack enable
package.json
{
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "packageManager": "pnpm@10.33.2+sha512.a90faf6feeab71ad6c6e57f94e0fe1a12f5dcc22cd754db40ae9593eb6a3e0b6b12e3540218bb37ae083404b1f2ce6db2a4121e979829b4aff94b99f49da1cf8",
  "name": "amplitude-nodejs-historical-backfill",
  "engines": {
    "node": ">=24.14.0",
    "pnpm": ">=10.21.0"
  },
  "scripts": {
    "test": "node --test tests/*.test.mjs",
    "fixture": "node src/fixture.js",
    "backfill": "node --env-file-if-exists=.env.local src/cli.js"
  },
  "dependencies": {}
}
pnpm-workspace.yaml
minimumReleaseAge: 10080
trustPolicy: no-downgrade
.gitignore
node_modules/
.env.local
data/
.env.example
AMPLITUDE_API_KEY=
AMPLITUDE_SERVER_ZONE=US
BUILD_CHECKPOINT=data/checkpoint.json
Terminal
corepack prepare pnpm@10.33.2 --activate
pnpm install
cp .env.example .env.local

Set AMPLITUDE_API_KEY to the test-project ingestion key and AMPLITUDE_SERVER_ZONE to US or EU. The project key has the minimum scope needed to ingest data. Keep your local environment file out of source control. These examples never need an Amplitude secret key.

In an existing app, merge these additions into the current SDK entry point and business operations. Preserve authentication, consent, routes and other instrumentation; use the authenticated account ID rather than accepting an arbitrary client-supplied user ID.

2. Generate a small historical source file

I'll create three synthetic records with original times an hour ago and a stable record UUID for each. The generator refuses to overwrite an existing fixture. Keep this file unchanged throughout resume and replay checks.

src/fixture.js
import { randomUUID } from 'node:crypto';
import {dirname} from 'node:path';
import { mkdirSync, writeFileSync, existsSync } from 'node:fs';
const path=process.argv[2]||'data/fixture.jsonl';
if(existsSync(path))throw Error('Fixture already exists. Keep it unchanged for resume/replay.');
mkdirSync(dirname(path),{recursive:true});const user='build-backfill-'+randomUUID();
const rows=['viewed','saved','viewed'].map((action,i)=>({record_id:randomUUID(),user_id:user,occurred_at:Date.now()-3600000+i*1000,action,private_note:'PRIVATE_BACKFILL_FIXTURE_20261113'}));
writeFileSync(path,rows.map(x=>JSON.stringify(x)).join('\n')+'\n',{mode:0o600,flag:'wx'});
console.log(JSON.stringify({path,user,records:rows.length}));

The synthetic user is an existing identity within this test dataset, not a fabricated match to a real production user. In a real migration, map historical users to your application's actual stable IDs; changing identity formats can split their history.

The private_note marker exists only in source data to check that the projection drops it. The importer accepts this fixture's bounded report-action schema, not arbitrary production exports.

3. Project events and checkpoint accepted batches

I'll map each source action into an approved event and preserve occurred_at as the event time. A deterministic insert ID combines the stable user and record UUID, so retries don't mint new event identities. The HTTP V2 reference defines the event and response fields.

src/backfill.js
import { readFileSync, writeFileSync, renameSync, mkdirSync, rmdirSync, existsSync } from 'node:fs';
import { dirname } from 'node:path';
import { createHash, randomUUID } from 'node:crypto';
import { setTimeout as delay } from 'node:timers/promises';
const hash=text=>createHash('sha256').update(text).digest('hex');
export function readDataset(path){
 const raw=readFileSync(path,'utf8');if(Buffer.byteLength(raw)>200000)throw Error('Use a file under200KB.');
 const lines=raw.trim().split('\n');if(!raw.trim()||lines.length>100)throw Error('Use1-100 JSONL records.');
 const seen=new Set();
 const events=lines.map(line=>{
  const r=JSON.parse(line);if(!r||typeof r!=='object')throw Error('Invalid source record.');
  if(!/^[0-9a-f-]{36}$/i.test(r.record_id)||seen.has(r.record_id)||!/^build-backfill-[a-z0-9-]{5,80}$/.test(r.user_id)||!Number.isSafeInteger(r.occurred_at)||r.occurred_at<=0||r.occurred_at>Date.now()||!['viewed','saved'].includes(r.action))throw Error('Invalid or repeated source record.');
  seen.add(r.record_id);
  return {event_type:r.action==='viewed'?'Historical Report Viewed':'Historical Report Saved',user_id:r.user_id,device_id:r.user_id,time:r.occurred_at,insert_id:'backfill-v1-'+hash(r.user_id+':'+r.record_id),event_properties:{source_record_id:r.record_id,backfill_source:'local-jsonl-v1'}};
 });return {fingerprint:hash(raw),events};
}
function save(path,state){const temp=path+'.'+randomUUID()+'.tmp';writeFileSync(temp,JSON.stringify(state),{mode:0o600});renameSync(temp,path);}
export async function backfill({dataset,checkpoint,apiKey,zone='US',send=false,fetcher=fetch,sleep=delay,now=()=>Date.now()}){
 if(!send)return {mode:'dry-run',events:dataset.events};
 if(!apiKey?.trim())throw Error('Set the test ingestion key before --send.');
 const endpoint=zone==='EU'?'https://api.eu.amplitude.com/2/httpapi':'https://api2.amplitude.com/2/httpapi';
 const binding=hash(dataset.fingerprint+':'+endpoint+':'+hash(apiKey.trim()));
 mkdirSync(dirname(checkpoint),{recursive:true});const lock=checkpoint+'.lock';
 try{mkdirSync(lock);}catch{throw Error('Checkpoint is locked. Confirm no other sender runs before clearing a stale lock.');}
 try{
  const state=existsSync(checkpoint)?JSON.parse(readFileSync(checkpoint,'utf8')):{binding,next:0,createdAt:now(),retryAfter:0};
  if(state.binding!==binding||!Number.isInteger(state.next)||state.next<0||state.next>dataset.events.length||!Number.isFinite(state.createdAt)||!Number.isFinite(state.retryAfter))throw Error('Checkpoint does not match this file/project or is invalid.');
  if(now()-state.createdAt>=7*86400000)throw Error('Checkpoint exceeds replay window; reconcile before resuming.');
  if(now()<state.retryAfter)throw Error('Rate-limit cooldown is still active. Retry later.');
  if(!existsSync(checkpoint))save(checkpoint,state);
  const results=[];
  while(state.next<dataset.events.length){
   const batch=dataset.events.slice(state.next,state.next+10);const body=JSON.stringify({api_key:apiKey.trim(),events:batch});
   if(Buffer.byteLength(body)>=1000000)throw Error('Batch too large.');
   let accepted=false;
   for(let attempt=0;attempt<3;attempt++){
    let response;
    try{response=await fetcher(endpoint,{method:'POST',headers:{'content-type':'application/json'},body,signal:AbortSignal.timeout(5000)});}
    catch{if(attempt===2)throw Error('Upload unavailable after3 attempts; checkpoint preserved.');await sleep(1000*2**attempt);continue;}
    if(response.status!==200)await response.body?.cancel();
    if(response.status===429){
     const retry=response.headers.get('retry-after');const seconds=Number(retry);const parsed=retry&&Number.isFinite(seconds)?now()+seconds*1000:Date.parse(retry||'');
     state.retryAfter=Math.max(now()+60000,Number.isFinite(parsed)?parsed:0);save(checkpoint,state);throw Error('Rate limited. Cooldown saved; retry after at least60 seconds.');
    }
    if(response.status>=500){if(attempt===2)throw Error('Server failure after3 attempts; checkpoint preserved.');await sleep(1000*2**attempt);continue;}
    if(response.status!==200)throw Error(`Upload rejected with HTTP ${response.status}; checkpoint preserved.`);
    let result;try{result=await response.json();}catch{throw Error('Invalid success body; checkpoint preserved.');}
    if(result.code!==200||!Number.isInteger(result.events_ingested)||result.events_ingested<0||result.events_ingested>batch.length)throw Error('Unexpected success body; checkpoint preserved.');
    state.next+=batch.length;state.retryAfter=0;save(checkpoint,state);results.push({accepted_through:state.next,events_ingested:result.events_ingested});accepted=true;break;
   }
   if(!accepted)throw Error('No accepted upload.');
  }
  return {mode:'sent',next:state.next,results};
 }finally{rmdirSync(lock);}
}

The importer admits at most100 records and200KB of source text, then sends batches of at most10 events with bodies under1MB. These are deliberate small-example bounds; batches also fit the documented Free-plan limit of10 events per request. This isn't a high-volume migration worker.

A checkpoint binds to the exact source-file bytes, endpoint and hashed project key. It is written before the first request and atomically replaced after an accepted batch. The lock rejects a second sender using the same checkpoint. A crash can leave a stale lock; confirm no sender remains before manually removing it.

Why. A timeout can occur after a server accepted the request. Keep the same insert/device IDs on retry. The API deduplication window is seven days within the project; the checkpoint stops after seven days from this run's creation for manual reconciliation. Historical event time and retry-window age are different things.

Network and5xx failures get at most three attempts with one- and two-second backoff. A429 stores a cooldown of at least60 seconds, honoring a longer Retry-After, then stops for a later invocation. It does not keep a shell blocked through that cooldown. Other non200 responses stop for inspection rather than repeatedly uploading invalid data.

A200 success body must have code200 and a plausible events_ingested count before the checkpoint advances. Replayed duplicates can be accepted by ingestion and deduplicated later; response counts are not a substitute for processed-history verification.

4. Make dry run the default

I'll keep projection separate from upload. Without --send the command prints only projected fixture events and doesn't need a key, create a checkpoint or make an HTTP request. With --send it uses the selected test project's region and checkpoint path.

src/cli.js
import { readDataset, backfill } from './backfill.js';
const args=process.argv.slice(2);const positional=args.filter(x=>x!=='--send');
try{
 if(positional.length!==1)throw Error('Usage: pnpm backfill data/fixture.jsonl [--send]');
 const result=await backfill({dataset:readDataset(positional[0]),checkpoint:process.env.BUILD_CHECKPOINT||'data/checkpoint.json',apiKey:process.env.AMPLITUDE_API_KEY,zone:process.env.AMPLITUDE_SERVER_ZONE,send:args.includes('--send')});
 console.log(JSON.stringify(result,null,2));
}catch(e){console.error(e.message);process.exitCode=1;}

Use a separate checkpoint path for a deliberately controlled replay, keeping the same source file and project key. Do not delete a production checkpoint to force a rerun. Changes to source bytes or project binding fail closed so an old position isn't silently reused for another dataset.

MCP. Ask an agent to review current HTTP V2 limits and response handling through Docs MCP before adapting this importer. Public docs can't confirm your historical source identities or whether an ambiguous request was processed.

5. Test and run the example

The tests cover projection/dry run, stable retries, checkpoint resume/project binding,429 cooldown,400 and exhausted5xx, stale replay-window rejection, real loopback HTTP and concurrent-sender locking.

tests/backfill.test.mjs
import test from 'node:test';
import assert from 'node:assert/strict';
import {mkdtemp,writeFile,readFile,rm,access} from 'node:fs/promises';
import {tmpdir} from 'node:os';import {join} from 'node:path';import {randomUUID} from 'node:crypto';import {createServer} from 'node:http';
import {readDataset,backfill} from '../src/backfill.js';
async function fixture(){const dir=await mkdtemp(join(tmpdir(),'backfill-'));const path=join(dir,'source.jsonl');const user='build-backfill-'+randomUUID();await writeFile(path,[0,1,2].map(i=>JSON.stringify({record_id:randomUUID(),user_id:user,occurred_at:Date.now()-3600000+i,action:i===1?'saved':'viewed',private_note:'PRIVATE_BACKFILL_FIXTURE'})).join('\n'));return {dir,path,checkpoint:join(dir,'checkpoint.json'),dataset:readDataset(path)};}
const ok=n=>new Response(JSON.stringify({code:200,events_ingested:n}),{status:200});
test('projection, dry run, stable retries, checkpoint resume and binding guard',async()=>{
 const f=await fixture();try{
  assert.equal(JSON.stringify(f.dataset.events).includes('PRIVATE_BACKFILL_FIXTURE'),false);
  const dry=await backfill({...f});assert.equal(dry.mode,'dry-run');await assert.rejects(access(f.checkpoint));
  const requests=[],delays=[];let n=0;
  const options={...f,apiKey:'fake-test-key',send:true,sleep:async ms=>delays.push(ms),fetcher:async(url,o)=>{requests.push(JSON.parse(o.body));if(n++===0)return new Response('',{status:503});return ok(3);}};
  await backfill(options);assert.deepEqual(requests[0],requests[1]);assert.deepEqual(delays,[1000]);
  assert.equal(JSON.parse(await readFile(f.checkpoint)).next,3);await backfill(options);assert.equal(requests.length,2);
  await assert.rejects(backfill({...options,apiKey:'other-test-key'}),/does not match/);
  assert.ok(!(await readFile(f.checkpoint,'utf8')).includes('fake-test-key'));
 }finally{await rm(f.dir,{recursive:true,force:true});}
});
test('429 saves cooldown;400 and exhausted5xx preserve position; stale checkpoint stops',async()=>{
 const f=await fixture();let time=Date.now(),calls=0;const base={...f,apiKey:'fake-test-key',send:true,now:()=>time,sleep:async()=>{}};
 try{
  await assert.rejects(backfill({...base,fetcher:async()=>{calls++;return new Response('',{status:429,headers:{'retry-after':'120'}})}}),/Rate limited/);
  const state=JSON.parse(await readFile(f.checkpoint));assert.equal(state.next,0);assert.equal(state.retryAfter,time+120000);
  await assert.rejects(backfill({...base,fetcher:async()=>{calls++;return ok(3)}}),/cooldown/);assert.equal(calls,1);
  time+=120001;await assert.rejects(backfill({...base,fetcher:async()=>new Response('',{status:400})}),/HTTP 400/);
  let retries=0;await assert.rejects(backfill({...base,fetcher:async()=>{retries++;return new Response('',{status:500})}}),/after3 attempts/);assert.equal(retries,3);assert.equal(JSON.parse(await readFile(f.checkpoint)).next,0);
  time=state.createdAt+8*86400000;await assert.rejects(backfill({...base,fetcher:async()=>ok(3)}),/replay window/);
 }finally{await rm(f.dir,{recursive:true,force:true});}
});
test('actual HTTP sender waits for accepted response; lock rejects concurrent writer; ambiguous body preserves stable replay',async()=>{
 const f=await fixture();let release,entered;const started=new Promise(r=>entered=r);const gate=new Promise(r=>release=r);const bodies=[];
 const server=createServer(async(req,res)=>{let raw='';for await(const c of req)raw+=c;bodies.push(JSON.parse(raw));entered();await gate;res.writeHead(200,{'content-type':'application/json'});res.end(JSON.stringify({code:200,events_ingested:3}));});await new Promise(r=>server.listen(0,'127.0.0.1',r));
 const base={...f,apiKey:'fake-test-key',send:true,fetcher:(url,options)=>fetch(`http://127.0.0.1:${server.address().port}`,options)};
 try{
  const first=backfill(base);await started;await assert.rejects(backfill(base),/locked/);assert.equal(JSON.parse(await readFile(f.checkpoint)).next,0);release();await first;assert.equal(bodies.length,1);
  const nextCheckpoint=join(f.dir,'ambiguous.json');await assert.rejects(backfill({...base,checkpoint:nextCheckpoint,fetcher:async()=>new Response('not-json',{status:200})}),/Invalid success body/);
  assert.equal(JSON.parse(await readFile(nextCheckpoint)).next,0);
  await backfill({...base,checkpoint:nextCheckpoint});assert.deepEqual(bodies[0].events,bodies[1].events);
 }finally{release();await new Promise(r=>server.close(r));await rm(f.dir,{recursive:true,force:true});}
});
Terminal
pnpm test
pnpm fixture
pnpm backfill data/fixture.jsonl
# After reviewing the projected fixture and setting the test key:
pnpm backfill data/fixture.jsonl --send
pnpm backfill data/fixture.jsonl --send

Verify it worked

Use only the generated synthetic fixture for this first live check. Preserve its bytes and note its user/record IDs.

  1. Dry run: With no ingestion key, run the importer without --send. Confirm three projected report events, historical millisecond times and stable IDs. The private_note marker must not appear; no checkpoint or HTTP request should be created.
  2. First upload and resume: Add the existing test-project ingestion key, then run --send. Confirm code200 and checkpoint next=3. Run the same command again:there should be no new upload because the checkpoint is already complete.
  3. Actual duplicate replay: Within seven days, use a fresh, separately named test checkpoint with BUILD_CHECKPOINT=data/replay-checkpoint.json while preserving the source/project. Send once. The three event bodies must retain their original insert/device IDs and timestamps. Confirm acceptance separately from deduplication.
  4. Processed history: Search the printed synthetic user in User Lookup and inspect Activity after ten seconds to one minute. Match the three source_record_id values and original historical times. After the duplicate replay, confirm only the original three events remain, not six. This is the actual deduplication check.
  5. Failures: Run pnpm test. A429 must persist cooldown without advancing; an immediate retry stops before HTTP. A400 or exhausted5xx must preserve position. An ambiguous200 body must retain the same events for a later replay. A second concurrent writer must fail the checkpoint lock.

Troubleshooting

The checkpoint does not match the source or project

Keep the source file bytes, region and test-project key unchanged for resume. For a deliberately different dataset, reconcile the old run and use a different checkpoint path.

A rate-limit error stops the command

The saved cooldown prevents an immediate rerun. Wait until its time has elapsed; the importer chooses at least60 seconds to cover project-level throttling and honors a longer Retry-After.

A lock remains after a crash

Confirm no sender is still running before manually removing that checkpoint's .lock directory. Never remove an active writer's lock to start a second importer.

The API accepted a replay but events_ingested still says3

Ingestion acceptance and downstream deduplication are separate. Match stable insert/device IDs and inspect processed history; do not infer duplicate history from request acceptance counts.

An import is older than the checkpoint window

Event timestamps can be historical. The seven-day checkpoint window concerns safe repeat delivery of this run; reconcile processed results before restarting an old run or changing its IDs.

Full code reference

The complete runnable example includes the lockfile. These are the complete setup, application and test files.

Full code reference
===== package.json =====
{
  "version": "1.0.0",
  "private": true,
  "type": "module",
  "packageManager": "pnpm@10.33.2+sha512.a90faf6feeab71ad6c6e57f94e0fe1a12f5dcc22cd754db40ae9593eb6a3e0b6b12e3540218bb37ae083404b1f2ce6db2a4121e979829b4aff94b99f49da1cf8",
  "name": "amplitude-nodejs-historical-backfill",
  "engines": {
    "node": ">=24.14.0",
    "pnpm": ">=10.21.0"
  },
  "scripts": {
    "test": "node --test tests/*.test.mjs",
    "fixture": "node src/fixture.js",
    "backfill": "node --env-file-if-exists=.env.local src/cli.js"
  },
  "dependencies": {}
}

===== pnpm-workspace.yaml =====
minimumReleaseAge: 10080
trustPolicy: no-downgrade

===== .gitignore =====
node_modules/
.env.local
data/

===== .env.example =====
AMPLITUDE_API_KEY=
AMPLITUDE_SERVER_ZONE=US
BUILD_CHECKPOINT=data/checkpoint.json

===== src/fixture.js =====
import { randomUUID } from 'node:crypto';
import {dirname} from 'node:path';
import { mkdirSync, writeFileSync, existsSync } from 'node:fs';
const path=process.argv[2]||'data/fixture.jsonl';
if(existsSync(path))throw Error('Fixture already exists. Keep it unchanged for resume/replay.');
mkdirSync(dirname(path),{recursive:true});const user='build-backfill-'+randomUUID();
const rows=['viewed','saved','viewed'].map((action,i)=>({record_id:randomUUID(),user_id:user,occurred_at:Date.now()-3600000+i*1000,action,private_note:'PRIVATE_BACKFILL_FIXTURE_20261113'}));
writeFileSync(path,rows.map(x=>JSON.stringify(x)).join('\n')+'\n',{mode:0o600,flag:'wx'});
console.log(JSON.stringify({path,user,records:rows.length}));

===== src/backfill.js =====
import { readFileSync, writeFileSync, renameSync, mkdirSync, rmdirSync, existsSync } from 'node:fs';
import { dirname } from 'node:path';
import { createHash, randomUUID } from 'node:crypto';
import { setTimeout as delay } from 'node:timers/promises';
const hash=text=>createHash('sha256').update(text).digest('hex');
export function readDataset(path){
 const raw=readFileSync(path,'utf8');if(Buffer.byteLength(raw)>200000)throw Error('Use a file under200KB.');
 const lines=raw.trim().split('\n');if(!raw.trim()||lines.length>100)throw Error('Use1-100 JSONL records.');
 const seen=new Set();
 const events=lines.map(line=>{
  const r=JSON.parse(line);if(!r||typeof r!=='object')throw Error('Invalid source record.');
  if(!/^[0-9a-f-]{36}$/i.test(r.record_id)||seen.has(r.record_id)||!/^build-backfill-[a-z0-9-]{5,80}$/.test(r.user_id)||!Number.isSafeInteger(r.occurred_at)||r.occurred_at<=0||r.occurred_at>Date.now()||!['viewed','saved'].includes(r.action))throw Error('Invalid or repeated source record.');
  seen.add(r.record_id);
  return {event_type:r.action==='viewed'?'Historical Report Viewed':'Historical Report Saved',user_id:r.user_id,device_id:r.user_id,time:r.occurred_at,insert_id:'backfill-v1-'+hash(r.user_id+':'+r.record_id),event_properties:{source_record_id:r.record_id,backfill_source:'local-jsonl-v1'}};
 });return {fingerprint:hash(raw),events};
}
function save(path,state){const temp=path+'.'+randomUUID()+'.tmp';writeFileSync(temp,JSON.stringify(state),{mode:0o600});renameSync(temp,path);}
export async function backfill({dataset,checkpoint,apiKey,zone='US',send=false,fetcher=fetch,sleep=delay,now=()=>Date.now()}){
 if(!send)return {mode:'dry-run',events:dataset.events};
 if(!apiKey?.trim())throw Error('Set the test ingestion key before --send.');
 const endpoint=zone==='EU'?'https://api.eu.amplitude.com/2/httpapi':'https://api2.amplitude.com/2/httpapi';
 const binding=hash(dataset.fingerprint+':'+endpoint+':'+hash(apiKey.trim()));
 mkdirSync(dirname(checkpoint),{recursive:true});const lock=checkpoint+'.lock';
 try{mkdirSync(lock);}catch{throw Error('Checkpoint is locked. Confirm no other sender runs before clearing a stale lock.');}
 try{
  const state=existsSync(checkpoint)?JSON.parse(readFileSync(checkpoint,'utf8')):{binding,next:0,createdAt:now(),retryAfter:0};
  if(state.binding!==binding||!Number.isInteger(state.next)||state.next<0||state.next>dataset.events.length||!Number.isFinite(state.createdAt)||!Number.isFinite(state.retryAfter))throw Error('Checkpoint does not match this file/project or is invalid.');
  if(now()-state.createdAt>=7*86400000)throw Error('Checkpoint exceeds replay window; reconcile before resuming.');
  if(now()<state.retryAfter)throw Error('Rate-limit cooldown is still active. Retry later.');
  if(!existsSync(checkpoint))save(checkpoint,state);
  const results=[];
  while(state.next<dataset.events.length){
   const batch=dataset.events.slice(state.next,state.next+10);const body=JSON.stringify({api_key:apiKey.trim(),events:batch});
   if(Buffer.byteLength(body)>=1000000)throw Error('Batch too large.');
   let accepted=false;
   for(let attempt=0;attempt<3;attempt++){
    let response;
    try{response=await fetcher(endpoint,{method:'POST',headers:{'content-type':'application/json'},body,signal:AbortSignal.timeout(5000)});}
    catch{if(attempt===2)throw Error('Upload unavailable after3 attempts; checkpoint preserved.');await sleep(1000*2**attempt);continue;}
    if(response.status!==200)await response.body?.cancel();
    if(response.status===429){
     const retry=response.headers.get('retry-after');const seconds=Number(retry);const parsed=retry&&Number.isFinite(seconds)?now()+seconds*1000:Date.parse(retry||'');
     state.retryAfter=Math.max(now()+60000,Number.isFinite(parsed)?parsed:0);save(checkpoint,state);throw Error('Rate limited. Cooldown saved; retry after at least60 seconds.');
    }
    if(response.status>=500){if(attempt===2)throw Error('Server failure after3 attempts; checkpoint preserved.');await sleep(1000*2**attempt);continue;}
    if(response.status!==200)throw Error(`Upload rejected with HTTP ${response.status}; checkpoint preserved.`);
    let result;try{result=await response.json();}catch{throw Error('Invalid success body; checkpoint preserved.');}
    if(result.code!==200||!Number.isInteger(result.events_ingested)||result.events_ingested<0||result.events_ingested>batch.length)throw Error('Unexpected success body; checkpoint preserved.');
    state.next+=batch.length;state.retryAfter=0;save(checkpoint,state);results.push({accepted_through:state.next,events_ingested:result.events_ingested});accepted=true;break;
   }
   if(!accepted)throw Error('No accepted upload.');
  }
  return {mode:'sent',next:state.next,results};
 }finally{rmdirSync(lock);}
}

===== src/cli.js =====
import { readDataset, backfill } from './backfill.js';
const args=process.argv.slice(2);const positional=args.filter(x=>x!=='--send');
try{
 if(positional.length!==1)throw Error('Usage: pnpm backfill data/fixture.jsonl [--send]');
 const result=await backfill({dataset:readDataset(positional[0]),checkpoint:process.env.BUILD_CHECKPOINT||'data/checkpoint.json',apiKey:process.env.AMPLITUDE_API_KEY,zone:process.env.AMPLITUDE_SERVER_ZONE,send:args.includes('--send')});
 console.log(JSON.stringify(result,null,2));
}catch(e){console.error(e.message);process.exitCode=1;}

===== tests/backfill.test.mjs =====
import test from 'node:test';
import assert from 'node:assert/strict';
import {mkdtemp,writeFile,readFile,rm,access} from 'node:fs/promises';
import {tmpdir} from 'node:os';import {join} from 'node:path';import {randomUUID} from 'node:crypto';import {createServer} from 'node:http';
import {readDataset,backfill} from '../src/backfill.js';
async function fixture(){const dir=await mkdtemp(join(tmpdir(),'backfill-'));const path=join(dir,'source.jsonl');const user='build-backfill-'+randomUUID();await writeFile(path,[0,1,2].map(i=>JSON.stringify({record_id:randomUUID(),user_id:user,occurred_at:Date.now()-3600000+i,action:i===1?'saved':'viewed',private_note:'PRIVATE_BACKFILL_FIXTURE'})).join('\n'));return {dir,path,checkpoint:join(dir,'checkpoint.json'),dataset:readDataset(path)};}
const ok=n=>new Response(JSON.stringify({code:200,events_ingested:n}),{status:200});
test('projection, dry run, stable retries, checkpoint resume and binding guard',async()=>{
 const f=await fixture();try{
  assert.equal(JSON.stringify(f.dataset.events).includes('PRIVATE_BACKFILL_FIXTURE'),false);
  const dry=await backfill({...f});assert.equal(dry.mode,'dry-run');await assert.rejects(access(f.checkpoint));
  const requests=[],delays=[];let n=0;
  const options={...f,apiKey:'fake-test-key',send:true,sleep:async ms=>delays.push(ms),fetcher:async(url,o)=>{requests.push(JSON.parse(o.body));if(n++===0)return new Response('',{status:503});return ok(3);}};
  await backfill(options);assert.deepEqual(requests[0],requests[1]);assert.deepEqual(delays,[1000]);
  assert.equal(JSON.parse(await readFile(f.checkpoint)).next,3);await backfill(options);assert.equal(requests.length,2);
  await assert.rejects(backfill({...options,apiKey:'other-test-key'}),/does not match/);
  assert.ok(!(await readFile(f.checkpoint,'utf8')).includes('fake-test-key'));
 }finally{await rm(f.dir,{recursive:true,force:true});}
});
test('429 saves cooldown;400 and exhausted5xx preserve position; stale checkpoint stops',async()=>{
 const f=await fixture();let time=Date.now(),calls=0;const base={...f,apiKey:'fake-test-key',send:true,now:()=>time,sleep:async()=>{}};
 try{
  await assert.rejects(backfill({...base,fetcher:async()=>{calls++;return new Response('',{status:429,headers:{'retry-after':'120'}})}}),/Rate limited/);
  const state=JSON.parse(await readFile(f.checkpoint));assert.equal(state.next,0);assert.equal(state.retryAfter,time+120000);
  await assert.rejects(backfill({...base,fetcher:async()=>{calls++;return ok(3)}}),/cooldown/);assert.equal(calls,1);
  time+=120001;await assert.rejects(backfill({...base,fetcher:async()=>new Response('',{status:400})}),/HTTP 400/);
  let retries=0;await assert.rejects(backfill({...base,fetcher:async()=>{retries++;return new Response('',{status:500})}}),/after3 attempts/);assert.equal(retries,3);assert.equal(JSON.parse(await readFile(f.checkpoint)).next,0);
  time=state.createdAt+8*86400000;await assert.rejects(backfill({...base,fetcher:async()=>ok(3)}),/replay window/);
 }finally{await rm(f.dir,{recursive:true,force:true});}
});
test('actual HTTP sender waits for accepted response; lock rejects concurrent writer; ambiguous body preserves stable replay',async()=>{
 const f=await fixture();let release,entered;const started=new Promise(r=>entered=r);const gate=new Promise(r=>release=r);const bodies=[];
 const server=createServer(async(req,res)=>{let raw='';for await(const c of req)raw+=c;bodies.push(JSON.parse(raw));entered();await gate;res.writeHead(200,{'content-type':'application/json'});res.end(JSON.stringify({code:200,events_ingested:3}));});await new Promise(r=>server.listen(0,'127.0.0.1',r));
 const base={...f,apiKey:'fake-test-key',send:true,fetcher:(url,options)=>fetch(`http://127.0.0.1:${server.address().port}`,options)};
 try{
  const first=backfill(base);await started;await assert.rejects(backfill(base),/locked/);assert.equal(JSON.parse(await readFile(f.checkpoint)).next,0);release();await first;assert.equal(bodies.length,1);
  const nextCheckpoint=join(f.dir,'ambiguous.json');await assert.rejects(backfill({...base,checkpoint:nextCheckpoint,fetcher:async()=>new Response('not-json',{status:200})}),/Invalid success body/);
  assert.equal(JSON.parse(await readFile(nextCheckpoint)).next,0);
  await backfill({...base,checkpoint:nextCheckpoint});assert.deepEqual(bodies[0].events,bodies[1].events);
 }finally{release();await new Promise(r=>server.close(r));await rm(f.dir,{recursive:true,force:true});}
});

Hand it to your agent

Implementation prompt
Use https://amplitude.com/build/nodejs-historical-backfill and its linked source.
Inspect the existing application and SDK entry point first. Preserve auth, consent,
and other instrumentation. Read the installed SDK's current docs through Docs MCP.

Use original historical times and real stable source/user identities. Keep insert
IDs deterministic across retry and resume. Project only approved fields; inspect
a key-free dry run first. Bind an atomic checkpoint to file/project/region, save
it before the first request, and reject concurrent writers. Bound network/5xx
attempts; persist429 cooldown instead of immediate retry. Stop invalid responses
without advancing. Treat seven-day dedup as bounded and verify actual processed
record IDs after a controlled replay. Do not regenerate the source on resume.

Run local failure and success checks. Verify real HTTP acceptance and processed
history separately. Keep environment secrets and synthetic fixture data local.
Report changed files and remaining coverage limits.

MCP. The Docs MCP server reads public documentation. It cannot inspect this project's events or run the application for you.

What this unlocks

Product analytics can include historical report actions at their original times.

Session replay requires its own capture; event backfill does not reconstruct an old browser recording.

Feature flags can be measured with separately captured exposure/outcome data; this import does not fabricate historical assignments.

Guides and surveys can support future user interactions; importing events does not replay an old guide.

Frequently asked questions

Does importing events recreate historical session replay?

No. This importer sends analytics events and original timestamps. It does not reconstruct recordings, UI state or experiment assignment.

Why not generate a new insert ID for a retry?

A new ID changes the event identity and can produce duplicate history after an ambiguous upload. The same device/insert ID supports the documented seven-day deduplication window.

Can this import a large production warehouse?

It is deliberately bounded to100 rows and a small fixture schema. A production migration needs source identity mapping, volume/quota planning, partitioned throttling, recovery and a reviewed rollout.

Next steps

Amplitude is the leading all-in-one analytics platform for building better products. Product and engineering teams use Amplitude to connect product analytics and web analytics with session replay, heatmaps, experimentation, feature flags, and guides and surveys in one unified workflow, with AI agents working across all of it. Learn how Amplitude is building self-improving products with Wave, helping teams turn product data into PRs.

Connect the next event

Make the outcome and identity explicit, then verify them in your test project.