Databricks
Export Amplitude events, merged Amplitude IDs, and data configuration to Unity Catalog Delta tables in your Databricks workspace. The connection authenticates with OAuth Workload Identity Federation (WIF), so Amplitude never holds a long-lived client secret for your workspace.
Considerations
- Amplitude delivers exports on a best-effort basis. How soon data lands in your Delta table depends on the run interval you choose. With a 5- or 10-minute run interval, data is typically queryable within 30 to 45 minutes of Amplitude receiving it. Longer intervals add up to one interval, so the default hourly interval can take up to about 1 hour 45 minutes. For data that's typically fresh within minutes, use express forwarding.
- You can create multiple Databricks exports from the same Amplitude project. Each export has its own destination table. Event exports also support separate event filters.
- Event filters apply only to event exports. Merged Amplitude ID and data configuration exports don't use the event filter.
Supported export types
Select one or more export types during setup. You can export merged Amplitude IDs or data configuration without exporting events.
| Export type | Data | Schedule |
|---|---|---|
| Events | Raw events that match your event filter, if you configure one. | Standard: configurable, every hour by default. Express: every few minutes. |
| Merged Amplitude IDs | Records that map merged Amplitude IDs to the IDs they merge into. | Every hour. |
| Event Transformations | Snapshots of your project's event transformation definitions. | Configurable data configuration frequency; every 10 minutes by default. |
| Custom Events | Snapshots of your project's custom event definitions. | Configurable data configuration frequency; every 10 minutes by default. |
Prerequisites
You need admin or manager privileges in Amplitude, and a role in Databricks that can manage service principals, create Unity Catalog grants, and create an account-level Federation Policy.
Set up the following in Databricks before you open the wizard:
- A Databricks workspace with a Unity Catalog catalog and schema. Capture the workspace host (for example,
https://dbc-xxxx.cloud.databricks.com). - A SQL warehouse. Capture the warehouse ID from the Databricks UI under SQL Warehouses > your warehouse > View JSON or the URL.
- A service principal in Settings > Identity and access > Service principals. Don't generate an OAuth secret. Capture both:
- The
application_id(a UUID). The wizard calls this theclientId, and it's the credential Amplitude uses at run time. - The numeric service principal ID. The wizard uses this to fill in the Federation Policy CLI command.
- The
Why Workload Identity Federation?
WIF removes the shared client secret that OAuth machine-to-machine (M2M) authentication requires. It's the more secure default for this integration:
- No shared secret: Amplitude never receives a credential that authenticates as your service principal outside of a live export run.
- Nothing for you to rotate: The credential is a short-lived JWT that Amplitude's infrastructure mints and refreshes automatically. There's no password or key on your side that drifts, expires, or leaks through a backup.
- One-line revocation: Deleting the Federation Policy on your service principal ends Amplitude's ability to authenticate immediately, without disturbing other integrations on the same workspace.
The wizard generates the Federation Policy command and the Unity Catalog grants SQL for you. You run them in Databricks during setup.
Set up the integration
The setup wizard has three steps: Get started, Set up, and Verify. The wizard generates the Databricks CLI command and grant SQL as you type, so you can run them in Databricks without leaving the page.
In Amplitude Data, click Catalog and select the Destinations tab.
In the Warehouse Destinations section, click Databricks.
On the Get started step, select the data to export:
- Forward events ingested today and moving forward: Under Forwarding mode, select Standard and set the run interval (the default is every hour), or select Express, where Amplitude sets the run interval. To forward only events that meet certain criteria, add a filter.
- Export all merged Amplitude IDs: Export the user-merge records.
- Data Configuration: Select Event Transformations, Custom Events, or both. Set the Data configuration frequency. The default is every 10 minutes.
Click Next.
On the Set up step, enter a Connection name and your Credential (OAuth WIF) details:
- Workspace host: the full workspace URL including the scheme, for example
https://dbc-xxxx.cloud.databricks.com. - Service Principal ID (numeric): the numeric ID from the Databricks UI under Settings > Identity and access > Service principals > your service principal. The wizard uses this only to fill in the Federation Policy command. Amplitude doesn't store it.
- Service Principal application_id (clientId): the UUID from the same Databricks page. This is the credential Amplitude uses at run time.
Copy the Federation Policy command on the right and run it with the Databricks CLI authenticated against your Databricks account, not a workspace. The command tells Databricks to accept JWTs from Amplitude's environment as proof of identity for the service principal. The wizard sets the OIDC issuer in the command automatically for the Amplitude environment your project runs in.
- Workspace host: the full workspace URL including the scheme, for example
In the same step, enter your Unity Catalog target:
- Catalog and Schema: The Unity Catalog location for your export tables.
- Table names: Enter a distinct table name for each selected export type. The fields are Events table name, Merged IDs table name, Event transformations table name, and Custom events table name. Table names must be unique within this setup, regardless of capitalization. Amplitude creates each table when it first exports data to that table.
- SQL warehouse ID: the warehouse Amplitude uses to run the
COPY INTOstatement.
Copy the Unity Catalog grants SQL on the right and run it in a Databricks SQL editor. Then in the Databricks UI, open SQL Warehouses > your warehouse > Permissions and grant the service principal CAN USE. The warehouse permission isn't grantable through SQL.
Click Next to open the Verify step. Amplitude verifies access and creates a separate export connection for each selected data type. Wait for Exports created, then click Finish.
Each connection runs on its own schedule and writes to its own table. Exports start on their next scheduled run.
Revoke Amplitude's access
To stop Amplitude from authenticating as the service principal, delete the Federation Policy you created in step 4:
databricks account service-principal-federation-policy list <sp-application-id> \
--profile <your-databricks-account-profile>
databricks account service-principal-federation-policy delete <sp-application-id> <policy-id> \
--profile <your-databricks-account-profile>
Use the same Databricks account profile you used in the Federation Policy command during setup. A workspace-scoped profile returns a 401 for these subcommands.
After you delete the policy, the next scheduled run fails with PERMISSION_DENIED and Databricks stops issuing workspace tokens for the service principal. To also block data access for any token that's still valid, remove the Unity Catalog grants from step 5.
Use express forwarding
Express forwarding is in early access.
Express forwarding loads raw events into your Delta table every few minutes, so data is typically fresh within minutes. It's available to organizations on the Enterprise plan. With express forwarding, events that Amplitude imports from a data warehouse or cloud storage still arrive hourly.
To use express forwarding, select Express under Forwarding mode in step 3 of the setup wizard.
Amplitude runs each load as a COPY INTO statement on the SQL warehouse you configure. Loads run every few minutes, so with typical auto-stop settings the SQL warehouse doesn't stop between loads. Budget for a SQL warehouse that runs continuously.
Use a dedicated, small SQL warehouse for express forwarding. A serverless SQL warehouse is the best fit because it starts in seconds and needs no cluster management.
If you already forward events with standard forwarding, you can keep that destination running while you set up express forwarding. To run both destinations at the same time:
- Give the express destination its own table. Two destinations that write to the same table load every event twice.
- Give the express destination its own SQL warehouse. Standard forwarding runs larger, less frequent loads, and express forwarding runs small, continuous ones.
After you confirm that data arrives in the express destination's table, delete the standard destination if you no longer need it.
Databricks export format
Data location
Amplitude writes each export type to the Unity Catalog table you name during setup. The full identifier is {catalog}.{schema}.{table_name}, using the values you enter.
Route different event slices to different tables
Amplitude's Databricks destination lets you create multiple exports from the same project. Each destination can have its own event filter and its own destination table, so you can fan a project's event stream out into purpose-built tables. For example, set up one destination that filters to checkout events and lands them in analytics.finance.checkout_events, and a second that filters to engagement events and lands them in analytics.product.engagement_events.
Event table schema
The event table uses the following Delta columns:
| Column | Type | Description |
|---|---|---|
amplitude_attribution_ids | VARIANT | Hashed attribution IDs on the event. |
amplitude_id | BIGINT | Amplitude ID on the event. Use the Merged User table to resolve user merges. Example: 2234540891. |
app | BIGINT | Project ID found in your project's Settings page. Example: 123456. |
city | STRING | City. Example: "San Francisco". |
client_event_time | TIMESTAMP | Local timestamp (UTC) of when the device logged the event. Example: 2015-08-10T12:00:00.000000. |
client_upload_time | TIMESTAMP | The local timestamp (UTC) of when the device uploaded the event. Example: 2015-08-10T12:00:00.000000. |
country | STRING | Country. Example: "United States". |
data | VARIANT | Dictionary where Amplitude stores certain fields such as first_event and merged_amplitude_id. |
device_carrier | STRING | Device carrier. Example: Verizon. |
device_family | STRING | Device family. Example: Apple iPhone. |
device_id | STRING | The device-specific identifier. Example: C8F9E604-F01A-4BD9-95C6-8E5357DF265D. |
device_type | STRING | Device type. Example: Apple iPhone 5s. |
dma | STRING | Designated marketing area (DMA). Example: San Francisco-Oakland-San Jose, CA. |
event_id | BIGINT | A counter that distinguishes events. Example: 1. |
event_properties | VARIANT | Event properties as semi-structured JSON. |
event_time | TIMESTAMP | Amplitude timestamp (UTC), which is client_event_time adjusted by the difference between server_received_time and client_upload_time: event_time = client_event_time + (server_received_time - client_upload_time). Amplitude uses this timestamp to organize events on Amplitude charts. If the difference between server_received_time and client_upload_time is less than 60 seconds, Amplitude doesn't adjust event_time and it equals client_event_time. Example: 2015-08-10T12:00:00.000000. |
event_type | STRING | Event type. |
group_properties | VARIANT | Group properties as semi-structured JSON. |
groups | VARIANT | Group types. For more information, refer to Account-level reporting. |
ip_address | STRING | IP address. Example: "123.11.111.11". |
language | STRING | Language reported by the device. |
library | STRING | The Amplitude SDK that sent the event. |
location_lat | DOUBLE | Latitude. Example: 12.3456789. |
location_lng | DOUBLE | Longitude. Example: -123.4567890. |
os_name | STRING | OS name. Example: ios. |
os_version | STRING | OS version. |
paying | STRING | True if the user has ever logged any revenue, otherwise none. You can modify the property value with the Identify API. Example: true. |
platform | STRING | Platform that sent the event. |
processed_time | TIMESTAMP | Time Amplitude processed the event. |
region | STRING | Region. Example: California. |
sample_rate | BIGINT | Event sampling rate. |
server_received_time | TIMESTAMP | Server time when Amplitude received the event. |
server_upload_time | TIMESTAMP | Amplitude timestamp (UTC) of when Amplitude servers received the event. Example: 2015-08-10T12:00:00.000000. |
session_id | BIGINT | The session start time in milliseconds since epoch. Example: 1396381378123. |
start_version | STRING | App version when Amplitude first tracked the user. Example: 1.0.0. |
user_creation_time | TIMESTAMP | Timestamp (UTC) of when Amplitude first observed the user on this project. |
user_id | STRING | A readable ID you specify. Use something that doesn't change. Avoid using the user's email address. |
user_properties | VARIANT | User properties as semi-structured JSON. |
uuid | STRING | Per-event unique identifier Amplitude assigns at ingestion. Use this as the join key to deduplicate events across refreshes or backfills. |
version_name | STRING | App version the SDK reports. Example: 1.0.0. |
Merged User table schema
The Merged User table records how Amplitude resolves user merges. Use the mapping to associate events with a user's merged identity.
| Column | Type | Description |
|---|---|---|
amplitude_id | STRING | The Amplitude ID that merges into another Amplitude ID. |
merged_amplitude_id | STRING | The Amplitude ID that amplitude_id merges into. |
merge_event_time | TIMESTAMP | Event timestamp associated with the user merge. |
merge_server_time | TIMESTAMP | Server timestamp associated with the user merge. |
Data configuration snapshots
Event transformation and custom event exports contain definitions, such as transformation rules and custom event filters. These exports don't apply the definitions to the raw event table.
Each scheduled run appends a full snapshot of the project's current definitions to the corresponding table. Earlier snapshots remain in the table. Use snapshot_at and export_job_id to identify rows from the same snapshot.
Event transformations table schema
Amplitude writes event transformation definitions to the table you enter in Event transformations table name.
| Column | Type | Description |
|---|---|---|
project_id | STRING | Amplitude project ID. |
snapshot_at | TIMESTAMP | Time Amplitude produced the snapshot. |
export_job_id | STRING | Identifier of the export run that produced the snapshot. |
transformation_id | STRING | Stable transformation identifier. |
name | STRING | Transformation display name. |
description | STRING | Transformation description. |
transform_type | STRING | Transformation type, such as merged event or merged-event derived property. |
merged_props | VARIANT | Mapping from target properties to source properties for merge transformations. |
merge_priority | VARIANT | Priority rules for merge resolution. |
merged_events | VARIANT | Events that merge to create the transformation. |
derived_props | VARIANT | Properties the transformation creates. This field doesn't represent Amplitude's standalone Derived Properties feature. |
derived_props_for_merge_events | VARIANT | Derived-property mapping for merged events. |
is_enabled | BOOLEAN | Whether Amplitude applies the transformation. |
created_at | TIMESTAMP | Time someone created the transformation. |
created_by | STRING | User who created the transformation. |
last_updated_at | TIMESTAMP | Time someone last modified the transformation. |
last_updated_by | STRING | User who last modified the transformation. |
Custom events table schema
Amplitude writes custom event definitions to the table you enter in Custom events table name.
| Column | Type | Description |
|---|---|---|
project_id | STRING | Amplitude project ID. |
snapshot_at | TIMESTAMP | Time Amplitude produced the snapshot. |
export_job_id | STRING | Identifier of the export run that produced the snapshot. |
custom_event_id | STRING | Stable custom event identifier. |
name | STRING | Canonical custom event name. |
display_name | STRING | Display name in Amplitude. |
description | STRING | Custom event description. |
definition | VARIANT | Definition expression, including filters and groups. |
category | STRING | Optional category or group. |
classifications | VARIANT | Tags or classifications as semi-structured JSON. |
suggestion_status | STRING | Suggestion state, such as suggested, accepted, or rejected. |
has_properties | BOOLEAN | Whether the custom event has custom properties. |
is_official | BOOLEAN | Whether the custom event is an org-official custom event. |
is_restricted | BOOLEAN | Whether the custom event limits access to certain users or roles. |
is_autotrack | BOOLEAN | Whether autotagging generated the custom event. |
is_hidden | BOOLEAN | Whether Amplitude hides the custom event from standard lists. |
created_at | TIMESTAMP | Time someone created the custom event. |
created_by | STRING | User who created the custom event. |
last_updated_at | TIMESTAMP | Time someone last modified the custom event. |
last_updated_by | STRING | User who last modified the custom event. |
Backfills
For event and merged Amplitude ID exports, the Backfills tab on your connection detail page replays a historical date range into the same Delta table as the recurring export. Event backfills use server_upload_time; merged Amplitude ID backfills use merge_server_time. You can also run manual backfills for express destinations.
Use a backfill to load historical data or recover from data loss in Databricks. Data configuration exports contain current definitions and don't support historical backfills.
Was this helpful?
For AI agents: a documentation index is available at /docs/llms.txt. Append .md to any page URL for markdown, or send Accept: text/markdown.