/pinpoint-export-plugin

Export PostHog events to Amazon Pinpoint on ingestion

Primary LanguageTypeScriptMIT LicenseMIT

Pinpoint Export Plugin

Export events to Amazon Pinpoint on ingestion. Archive your data, or simply free it up for other kinds of analysis, by integrating export right into your event processing pipeline.

Installation

  1. Access PostHog's Plugins page from the sidebar.
  2. To perform installation either:
    1. go to the Repository tab, find this plugin, and Install it,
    2. or go to the Advanced tab and Fetch and install the following URL in the Install from GitHub, GitLab or npm section:
      http://github.com/PostHog/posthog-plugin-geoip.
  3. Configure the plugin by entering your AWS credentials and S3 bucket details.
  4. Watch events roll into S3!

Obtaining AWS Credentials

  1. Log in to AWS.
  2. Open Pinpoint in the AWS console and create a project to send events.
  3. Open IAM and create a new policy to allow access to this bucket.
    1. Open "Policies" and click "Create policy"
    2. On the "Visual Editor" tab, click "Choose a service" and select "Pinpoint"
    3. Under "Actions" select
      1. "Write" -> "PutEvents"
    4. Specify your bucket name and choose "any" for the object name, so the ARN looks something like this: arn:aws:mobiletargeting:{Region}:{AccountId}:apps/{AppId}/*
    5. On the "Visual Editor" tab, click "Choose a service" and select "Pinpoint"
    6. Under "Actions" select
      1. "Write" -> "UpdateEndpoint"
    7. Specify your bucket name and choose "any" for the object name, so the ARN looks something like this: arn:aws:mobiletargeting:{Region}:{AccountId}:apps/{AppId}/*
    8. On the "Visual Editor" tab, click "Choose a service" and select "Mobile Analytics"
    9. Under "Actions" select
      1. "Write" -> "PutEvents"
    10. Under "Resources" select "Specific" and click "object" -> "All resources"
    11. Click "Next" until you end up on the "Review Policy" page
    12. Give it a name
  4. Open IAM and create a new user who uses this policy
    1. Click "Users" -> "Add User"
    2. Specify a name and choose "Programmatic access"
    3. Click "Next"
    4. Select "Attach existing policies directly"
    5. Select the policy you had just created
    6. Click "Next" until you reach the "Create user" button. Click that as well.
    7. Make sure to copy your "Access key" and "Secret access key". The latter will not be shown again.
  5. Install the plugin in PostHog and fill in the "Access key", "Secret access key", "Pinpoint region" and "Application Id" fields. Adjust other parameters as needed.

Memory Usage

To vastly increase export throughput, this plugin batches events in memory before uploading them to S3. Upload frequency (every minute by default) and maximum upload size (1 MB by default) can be configured when the plugin is installed.

You should make sure to keep these numbers reasonable to avoid running out of memory on your server. Note that the values apply to each concurrent plugin server thread.

Questions?

Thanks

This project was based on "S3 Export Plugin" so thanks to all team.