/SFDC-change-data-capture

Streaming Event - Change Data Capture (CDC)

Primary LanguageJavaScript

Change Data Capture (CDC)

This GitHub Repo explains specifically for Change Data Capture (CDC). If you are looking for ALL Streaming events then have a look at this detailed GitHub Repo.

Introduction

When to Use Change Data Capture

  • Use Change Data Capture to update data in an external system instead of doing periodic exports or API polling.
  • Capturing changes with Change Data Capture event notifications ensures that your external data can be updated in real time and stays fresh.
  • You can use Change Data Capture as part of the data replication process.
  • Capture all field changes for all records.
  • Get broad access to all data regardless of sharing rules.
  • Get information about the change in the event header, such as the origin of the change, so you can ignore changes that your client generates.
  • Perform data updates using transaction boundaries when more than one operation is part of the same transaction.
  • Use a versioned event schema.
  • Subscribe to mass changes in a scalable way.
  • Get access to retained events for up to 3 days.
  • You can select up to 5 entities, including standard and custom objects.
  • To enable more entities, contact Salesforce to purchase an add-on license.
  • The add-on license removes the limit on the number of entities you can select. Also, it increases the CometD event delivery allocation.

Supported Objects

Sample Change Event JSON

  • {
  "data": {
    "schema": "fl8pv4oUvtbyFAwJQO2NAQ",
    "payload": {
      "LastModifiedDate": "2019-05-02T11:27:36Z",
      "OwnerId": "005B0000005iVeGIAU",
      "CreatedById": "005B0000005iVeGIAU",
      "ChangeEventHeader": {
        "commitNumber": 10426208028536,
        "commitUser": "005B0000005iVeGIAU",
        "sequenceNumber": 1,
        "entityName": "Employee__c",
        "changeType": "CREATE",
        "changeOrigin": "com/salesforce/api/soap/46.0;client=SfdcInternalAPI/",
        "transactionKey": "000267d5-1ab2-e18a-f726-b8849994a99a",
        "commitTimestamp": 1556796456000,
        "recordIds": [
          "a00B000000AVWaZIAX"
        ]
      },
      "CreatedDate": "2019-05-02T11:27:36Z",
      "First_Name__c": "Marc",
      "LastModifiedById": "005B0000005iVeGIAU",
      "Tenure__c": 20,
      "Name": "EN-0003",
      "Last_Name__c": "Benioff"
    },
    "event": {
      "replayId": 32872942
    }
  },
  "channel": "/data/Employee__ChangeEvent"
}
  • changeType: "CREATE", "UPDATE", "DELETE", "UNDELETE", "GAP_CREATE", "GAP_UPDATE", "GAP_DELETE", "GAP_UNDELETE", "GAP_OVERFLOW"
  • changeOrigin: Use this field to detect whether your app initiated the change, so you do not process the change again and potentially avoid a deep cycle of changes. This field contains the Salesforce API and the API client ID that initiated the change, if set by the client.
  • Note: In the example, it is com/salesforce/api/soap/44.0;client=GetCloudy, which means that an app with clientID GetCloudy created the Employee record via SOAP API.
  • The order of the fields in the JSON event message follows the underlying Avro schema that Change Events are based on.
  • Change Event Header Fields
    • For gap events, the change type starts with the GAP_ prefix.
    • GAP_CREATE
    • GAP_UPDATE
    • GAP_DELETE
    • GAP_UNDELETE
    • For overflow events, the change type is GAP_OVERFLOW.

Change Event Schema by Event Name

  • Change Event Message Structure
    • Make a GET request to this resource: 
    /services/data/vXX.X/sobjects/<EventName>/eventSchema?payloadFormat=COMPACT

/services/data/v45.0/sobjects/Employee__ChangeEvent/eventSchema?payloadFormat=EXPANDED

Change Event Schema by Schema ID

  • Change Event Message Structure
    • Make a GET request to this resource: 
    /services/data/vXX.X/event/eventSchema/<Schema_ID>?payloadFormat=COMPACT

/services/data/v45.0/event/eventSchema/ABCiola5arQ6oq4jvUZpqQ?payloadFormat=COMPACT

Subscription Channels

Reference from the Change Data Capture guide.

Subscribe to change events for: Channel Example
All Change Events (all objects) /data/ChangeEvents
Standard Object /data/StandardObjectNameChangeEvent /data/AccountChangeEvent
Custom Object /data/CustomObjectName__ChangeEvent /data/Employee__ChangeEvent
Custom Channel /data/YourChannelName__chn /data/SalesEvents__chn

Field-Level Security

Change Data Capture respects org’s field-level security settings. Delivered events contain only the fields that a subscribed user is allowed to view.

Required User Permissions

  • Reference from the Required Permissions for Subscription.

    Channel Required Permissions
    /data/ChangeEvents View All Data AND View All Users
    /data/UserChangeEvent View All Users
    /data/StandardObjectNameChangeEvent Or

    /data/CustomObjectName__ChangeEvent

    		View All for the object OR View All Data

    Some standard objects don’t have the View All permission, such as Task and Event. In this case, the View All Data permission is required.

    /data/YourChannelName__chn View All for the object OR View All Data View All Users for User change events
  • User permissions are enforced when the user subscribes to a channel. If the user has insufficient permissions, the user can’t subscribe to the channel and an error is returned.

Change Event Objects in Metadata API

Developers can use the PlatformEventChannel metadata type in Metadata API to retrieve and deploy Change Data Capture objects.


Standard Change Event Channels (ChangeEvents.platformEventChannel)
Custom Change Event Channels
Note: As of Summer '19 release, you can create a custom channel with Metadata API. You can’t create or view custom channels in Setup in the Change Data Capture page.
SAPEvents__chn.platformEventChannel
OracleEvents__chn.platformEventChannel
Package.xml

Fields in JSON & Apex Change Event Body

Change Event Body Fields

1 - Subscribe to Change Events with the CometD

CometD (A messaging library that enables listening to events through long polling and simulates push technology).

Interactive Visualforce Page without Replay

Subscribe to Changes with the Streaming API

  • Change Data Capture in Setup
  • Note: Here, the filter is case-sensitive. Make sure that you use the same case as the entity.
  • Subscribe to Channel: /data/Employee__ChangeEvent
  • Create a New Employee__c record 
    {
  "data": {
    "schema": "fl8pv4oUvtbyFAwJQO2NAQ",
    "payload": {
      "LastModifiedDate": "2019-05-02T11:27:36Z",
      "OwnerId": "005B0000005iVeGIAU",
      "CreatedById": "005B0000005iVeGIAU",
      "ChangeEventHeader": {
        "commitNumber": 10426208028536,
        "commitUser": "005B0000005iVeGIAU",
        "sequenceNumber": 1,
        "entityName": "Employee__c",
        "changeType": "CREATE",
        "changeOrigin": "com/salesforce/api/soap/46.0;client=SfdcInternalAPI/",
        "transactionKey": "000267d5-1ab2-e18a-f726-b8849994a99a",
        "commitTimestamp": 1556796456000,
        "recordIds": [
          "a00B000000AVWaZIAX"
        ]
      },
      "CreatedDate": "2019-05-02T11:27:36Z",
      "First_Name__c": "Marc",
      "LastModifiedById": "005B0000005iVeGIAU",
      "Tenure__c": 20,
      "Name": "EN-0003",
      "Last_Name__c": "Benioff"
    },
    "event": {
      "replayId": 32872942
    }
  },
  "channel": "/data/Employee__ChangeEvent"
}
  • Update an Employee__c record 
    {
  "data": {
    "schema": "fl8pv4oUvtbyFAwJQO2NAQ",
    "payload": {
      "LastModifiedDate": "2019-05-02T11:29:00Z",
      "ChangeEventHeader": {
        "commitNumber": 10426208532295,
        "commitUser": "005B0000005iVeGIAU",
        "sequenceNumber": 1,
        "entityName": "Employee__c",
        "changeType": "UPDATE",
        "changeOrigin": "com/salesforce/api/soap/46.0;client=SfdcInternalAPI/",
        "transactionKey": "000267e8-ba20-f483-b6b9-5da9bb5c6c64",
        "commitTimestamp": 1556796540000,
        "recordIds": [
          "a00B000000AVWaZIAX"
        ]
      },
      "First_Name__c": "Marc 2",
      "Tenure__c": 22,
      "Last_Name__c": "Benioff 2"
    },
    "event": {
      "replayId": 32873029
    }
  },
  "channel": "/data/Employee__ChangeEvent"
}
  • Delete an Employee__c record 
    {
  "data": {
    "schema": "fl8pv4oUvtbyFAwJQO2NAQ",
    "payload": {
      "ChangeEventHeader": {
        "commitNumber": 10426208770402,
        "commitUser": "005B0000005iVeGIAU",
        "sequenceNumber": 1,
        "entityName": "Employee__c",
        "changeType": "DELETE",
        "changeOrigin": "com/salesforce/api/soap/46.0;client=SfdcInternalAPI/",
        "transactionKey": "000267f1-31a6-e4e7-24e3-45f54c53a5d0",
        "commitTimestamp": 1556796576000,
        "recordIds": [
          "a00B000000AVWaZIAX"
        ]
      }
    },
    "event": {
      "replayId": 32873116
    }
  },
  "channel": "/data/Employee__ChangeEvent"
}
  • Deleted record in Recycle Bin
  • Undelete an Employee__c record 
    {
  "data": {
    "schema": "fl8pv4oUvtbyFAwJQO2NAQ",
    "payload": {
      "LastModifiedDate": "2019-05-02T11:27:36Z",
      "OwnerId": "005B0000005iVeGIAU",
      "CreatedById": "005B0000005iVeGIAU",
      "ChangeEventHeader": {
        "commitNumber": 10426208028536,
        "commitUser": "005B0000005iVeGIAU",
        "sequenceNumber": 1,
        "entityName": "Employee__c",
        "changeType": "UNDELETE",
        "changeOrigin": "com/salesforce/api/soap/46.0;client=SfdcInternalAPI/",
        "transactionKey": "000267d5-1ab2-e18a-f726-b8849994a99a",
        "commitTimestamp": 1556796456000,
        "recordIds": [
          "a00B000000AVWaZIAX"
        ]
      },
      "CreatedDate": "2019-05-02T11:27:36Z",
      "First_Name__c": "Marc",
      "LastModifiedById": "005B0000005iVeGIAU",
      "Tenure__c": 20,
      "Name": "EN-0003",
      "Last_Name__c": "Benioff"
    },
    "event": {
      "replayId": 32872117
    }
  },
  "channel": "/data/Employee__ChangeEvent"
}

2 - Subscribe to Change Events with the EMP Connector

Enterprise Messaging Platform (EMP) Connector is an open-source sample tool that subscribes to streaming channels using Streaming API and CometD. EMP Connector is a thin wrapper around the CometD library. It hides the complexity of creating a CometD client and subscribing to Streaming API in Java.

3 - Subscribe to Change Events with a Lightning Component

  • Reference from the Streaming API Developer Guide.
  • The lightning:empApi component uses a shared CometD-based Streaming API connection, enabling you to run multiple streaming apps in the browser.
  • To call the component’s methods, add the lightning:empApi component inside your custom component and assign an aura:id attribute to it.
  • Then in the client-side controller, add functions to call the component methods.

4 - Subscribe to Change Events with a Lightning Web Component (LWC)

  • Starting with Summer '19 release, you can subscribe to an event channel using the lightning/empApi module
  • The events that you can receive include platform events, PushTopic events, generic events, and Change Data Capture events.

5 - Subscribe to Change Events with the Apex Triggers

  • Starting with Summer '19 release, you can now Process Change Event Messages in Apex Triggers
  • Subscribe with Apex Triggers
  • Apex triggers for change events are similar to Apex triggers on platform events.
  • Change event triggers run asynchronously after the database transaction is completed.
  • Perform resource-intensive business logic asynchronously in the change event trigger, and implement transaction-based logic in the Apex object trigger.
  • By decoupling the processing of changes, change event triggers can help reduce transaction processing time.

6 - Subscribe to a Custom (Virtual) Channel

  • Starting with Summer '19 release, you can now subscribe to Custom Channels to receive specific types of events (e.g. Account, Contact, Employee__c etc.).
  • You can create a custom channel with Metadata API. You can’t create or view custom channels in Setup in the Change Data Capture page.
  • /data/YourChannelName__chn = /data/SalesEvents__chn

Gap and Overflow Events

Transaction-based Replication Approach

Change Data Capture Allocations

External Change Data Capture

  • Starting with Summer '19 release, External Change Data Capture is now Generally Available (GA) in Salesforce Connect using the OData 4.0 adapter.
  • You can track changes made to the External Object from within Salesforce and changes made outside of Salesforce are tracked.

Useful Resources