/hull-sql

SQL Importer Ship

Primary LanguageJavaScript

Hull SQL Ship.

Using :

  • Go to your Hull Dashboard > Ships > Add new
  • Paste the URL for your Heroku deployment, or use ours : https://hull-sql.herokuapp.com/
  • Visit the Setup Page (see below)
  • Add your ship to a Platform.

Developing :

  • Fork
  • Install
npm i -g gulp
npm i
npm run start:dev #starts in dev mode with nodemon
npm run test #runs unit tests, lints and checks dependencies
npm run watch #builds client package
npm run build # build
# Checkout package.json for more tasks

Installing

Dependencies for this connector

  • Redis
  • S3 Bucket
  • IAM profile with read/write access to a S3 bucket

Environment variables

| SECRET | Connector Secret - Generate a random secret string | | AWS_KEY_ID | Access Key for IAM profile | | AWS_SECRET_KEY | Secret Key for IAM profile | | BUCKET_PATH | S3 Bucket name and path for extracted files | | REDIS_URL | Redis URL for the jobs queue |

How to setup the S3 bucket and IAM profile

This article explains how to create an IAM profile and grant access to a S3 bucket : https://aws.amazon.com/blogs/security/writing-iam-policies-how-to-grant-access-to-an-amazon-s3-bucket/

You might also want to configure your S3 bucket's lifecycle to expire files automatically after a few days.

In the S3 section of your AWS console, go to your bucket's Management tab and add a Lifecycle rule to automatically expire Objects after 7 days.

Logs

Logs that are specific for SQL Connector:

  • incoming.job.*

    • jobName: sync
    • stepName: query - showing progress for fetching users from database
    • stepName: upload - showing progress for uploading chunks of data to S3
    • stepName: import - showing progress for triggering imports on the platform

  • incoming.job.query - includes the full query which is run on the external database to import data from

  • connection.error - encountered problems when connecting with database

  • incoming.job.warning - query returned no results

Status

Messages that may come from status endpoint:

  • Connection string is not configured. - when some of database settings are not configured in connector's private settings
  • Error when trying to connect with database. {err} - when trying to establish connection and perform simple query
  • Query string is not configured - when query is not configured in connector's private settings
  • Database does not return any rows for saved query - when database returned no results for saved query.
  • Results have invalid format {messages} - when results from database have invalid format. messages - list that contains specified
  • Interval syncing is enabled but interval time is less or equal zero - when cron syncing is enabled but time provided by user is less than zero.

Notifications

Messages that may be sent to platform from this connector:

  • error:
    • Server Error: {err} - server side error, mostly connection errors
    • Invalid Syntax: {message} - errors about invalid query syntax
    • Invalid structure {messages} - errors about invalid results structure, e.g. missing email/external_id, json column in postgres database ...
  • warning:
    • Warning: Query returned no results - when query that was already saved returned no results