Databricks

Amplitude's Databricks import source enables you to import data from Databricks to your Amplitude account. Databricks import uses the Databricks Change Data Feed feature to securely access and extract live data from your Databricks workspace.

Features

• Import all data types, including events, user properties, and group properties.
• Support for delta sync, to ensure Amplitude imports only new or changed data.
  • For event data, Amplitude imports rows with an insert operation.
  • For user properties and group properties, Amplitude imports rows with insert or update operations.
  • Amplitude ignores rows with delete operations for all data types.

Limitations

• The User Look-Up page doesn't display 100 most recent events ingested.

For guided instructions to setting up this integration, view the Loom video.

Configure Databricks

Before you start to configure the Databricks source in Amplitude, complete the following tasks in Databricks.

Find or create an all-purpose compute cluster

Amplitude creates workflows in this cluster on your behalf to start sync jobs. When complete, copy the Server hostname and HTTP path values to use in a later step. Find both values on the Configuration -> JDBC/ODBC tab. For more information about cluster types, see Compute.

1"workload_type.clients.jobs": {
2    "type": "fixed",
3    "value": false
4}

1TypeError: 'type' object is not subscriptable

Cluster policies and access modes

Amplitude supports all policies and access modes. However, if your clusters have the following policies and access modes, grant the Data Reader permission (USE CATALOG, USE SCHEMA, EXECUTE, READ VOLUME, SELECT) to the your workspace user or service principal, whose personal access token is used to authenticate in Amplitude. Otherwise, you can't access the tables in the unity catalog of your import source.

AWS Databricks:

GCP Databricks:

Authentication

Amplitude's Databricks import supports authentication with personal access tokens for Databricks workspace users, or personal access tokens for Service Principals. Choose Workspace User authentication for faster setup, or Service Principal authentication for finer grained control. For more information, see Databrick's article Authentication for Databricks Automation

Create a workspace user Personal Access Token (PAT)

Amplitude's Databricks import uses Personal Access Tokens to authenticate. For the quickest setup, create a PAT for your workspace user in Databricks. For more information, see Databricks' article Personal Access Tokens for Workspace Users

Create a service principal Personal Access Token (PAT)

Amplitude recommends that you create a service principal in Databricks to allow for more granular control of access.

1. Follow the Databricks instructions to create a service principal. Copy the UUID for use in a later step.

2. Generate a PAT on this Service Principal.

   • If you use AWS or GCP Databricks, follow the instructions in the article Databricks personal access tokens for service principals.
   • If you use Azure-based Databricks, follow the instructions in the article Manage personal access tokens for a service principal.

The service principal you created above requires the following permissions in Databricks:

Enable CDF on your tables

Amplitude uses Databricks' Change Data Feed to continuously import data. To enable CDF on a Databricks table, see Databricks | Enable change data feed

Configure the Amplitude Databricks source

To add Databricks as a source in Amplitude, complete the following steps.

Connect to Databricks

1. In Amplitude Data, click Catalog and navigate to the Sources tab.
2. Search for Databricks.
3. On the Credentials tab of the Connect Databricks screen, enter the credentials you configured during the Databricks configuration:
   • Server hostname
   • HTTP Path
   • Personal Access Token (for the workspace user or Service Principal)
4. Click Next to verify access.

Select data to import

1. Select the data type for data to be imported. The Databricks source supports three data types:

   • Event
   • User properties
   • Group properties

   For the Event data type, optionally select Sync User Properties or Sync Group Properties to sync the corresponding properties within an event.

2. Configure the SQL command that transforms data in Databricks before Amplitude imports it.

   • Amplitude treats each record in the SQL execution output as an event to be import. See the Example body in the Batch Event Upload API documentation to ensure each record you import complies.
   • Amplitude can transform / import from only the tables you specify in step 1 above.
     • For example, if you have access to tables A, B and C but only selected A in step 1, then you can only import data from A.
   • The table names you reference in the SQL command must match exactly the name of the table you select in step 1. For example, if you select catalog.schema.table1, use that exact value in the SQL.

   1select
   2    unix_millis(current_timestamp())                                       as time,
   3    id                                                                     as user_id,
   4    "demo"                                                                 as event_type,
   5    named_struct('name', name, 'home', home, 'age', age, 'income', income) as user_properties,
   6    named_struct('group_type1', ARRAY("group_A", "group_B"))               as groups,
   7    named_struct('group_property', "group_property_value")                 as group_properties
   8from catalog.schema.table1;

3. After you add the SQL, click Test SQL. Amplitude runs a test against your Databricks instance to ensure the SQL is valid. Click Next.

4. Select the table version for initial import. The initial import brings everything the from table as of the selected version. Select First or Latest.

   • First means first version, which is 0.
   • Latest means latest version.

5. Set the sync frequency. This frequency determines the interval at which Amplitude pulls data from Databricks.

6. Enter a descriptive name for this instance of the source.

7. The source appears in your workspace's Sources list.

Verify data import

Events that Amplitude imports assume the name you assign in your SQL statement. In the example above, the events have the name demo

To verify the data coming into Amplitude:

• View the Events page of your Tracking Plan
• Create a Segmentation chart that filters on the event name you specify.
• Go to the Ingestion Jobs tab in your source. You can view the status of the ingestion and debug using ERROR LOG if necessary.