Converter configuration reference

This reference covers examples and operators for the Amazon S3 Import and Google Cloud Storage (GCS) converter configuration. Read the S3 guide or the GCS guide for more information.

skip_user_properties_sync

Because many cloud storage source imports are batch uploads of historical data, syncing the latest user properties for historical events might not make sense. For this reason, Amplitude sets $skip_user_properties_sync to true by default. To include user properties with your events, set it to false in the converter.

For more information about $skip_user_properties_sync, refer to the Data Backfill Guide.

ignoreEventFlag

Use ignoreEventFlag to selectively skip events from ingestion based on a computed boolean condition. Set ignoreEventFlag to a field name (for example, "$ignore"), then define that field in convertToAmplitudeFunc using any DataLang boolean expression. When the field evaluates to true for a given event, Amplitude skips the event without counting it as an error. When it evaluates to false or is absent, Amplitude ingests the event normally.

This option is useful when your source data contains events you want to filter, such as internal test events, bot traffic, or specific event types, without preprocessing the files upstream.

Example: filter out specific event types

The following example skips any event where event_type is page_view or session_start:

{
  "converterConfig": {
    "ignoreEventFlag": "$ignore",
    "convertToAmplitudeFunc": {
      "event_type": "event_type",
      "user_id": "user_id",
      "$ignore": ["or",
        ["equals", ["path", "event_type"], ["value", "page_view"]],
        ["equals", ["path", "event_type"], ["value", "session_start"]]
      ]
    }
  }
}

In this example:

• "ignoreEventFlag": "$ignore" tells Amplitude to look for a field named $ignore in the converted event.
• "$ignore" in convertToAmplitudeFunc defines a boolean expression using the or and equals operators.
• If event_type is page_view or session_start, $ignore evaluates to true and Amplitude skips the event.
• Amplitude ingests all other events normally.

You can use any boolean operator or combination of operators to build the condition.

convertToAmplitudeFunc

Conversion rules in convertToAmplitudeFunc instruct the ingestion service on how to construct events in Amplitude.

Example converter with convertToAmplitudeFunc

{
    "config_name": ["Event sample converter"],
    "converterConfig": {
        "fileType": "parquet",
        "compressionType": "none",
        "convertToAmplitudeFunc": {
          "event_type": "action",
          "user_id": "user",
          "device_id": "device",
          "event_properties": {
              "business_id_encid": "business_id"
          },
          "user_properties": {
              "utm_channel_category": "utm_channel_c",
              "utm_channel_source": "utm_channel_s"
          },
          "time": "epoch",
          "session_id": "session_id",
          "app_version": "app_version"
        }
    },

    "keyValidatorConfig": {
        "filterPattern": "folder1/folder2/ds=202011[1-2][0-9]/.*\\.parquet"
    }
}

Example constructed event

{
  "event_type": "watch tv",
  "user_id": "john",
  "device_id": "host1",
  "event_properties": {
    "business_id_encid": "123"
      },
      "user_properties": {
        "utm_channel_category": "discovery",
        "utm_channel_source": "network"
      },
      "time": "1645066434189",
      "session_id": "1",
      "app_version": "1"
}

Values in the event come from the fields specified by convertToAmplitudeFunc. For example, the value watch tv in field event_type comes from field "action" in ingested data files. Because the event_type value isn't ["value":"$identify"] or ["value":"$groupidentify"], Amplitude ingests events the same way it ingests events with the HTTP V2 API.

Operators

List operators

Use list operators

If the source description is a list, the first item in the list must be a string specifying the function. The rest of the list are the parameters to the function. The "|" character separates non-repeating and repeating arguments. Any arguments after "|" are repeatable, and you can specify them any number of times. The entire list of arguments must be present in any multiple-argument operator (you can't specify just one of three arguments, you must include all three).

Boolean operators

These operators return a JsonPrimitive of type Boolean, so they're valid to use with cond and ifelse.

Integer and float operators

The following Operators return a JsonPrimitive of type Integer, barring the add Operator which returns JsonPrimitive of type Float.

JSON operator

User property operations

The converter supports the same user property operators as the Identify API. Refer to the Identify documentation for details.