/export-dialogflow-logs-to-bigquery

Export Dialogflow conversation logs to BigQuery with masking PII using DLP API

Primary LanguageJavaScriptApache License 2.0Apache-2.0

Export Dialogflow conversations to BigQuery

Build Status Coverage Status

Export Dialogflow conversations to BigQuery after masking SPII data using DLP API

Solution Approach

Dialogflow Fulfillment mechanism is used to intercept all the conversations using a webhook (deployed as a GAE app). The conversations are logged asynchronously after masking the PII data using Google App Engine and DLP API to a PubSub topic. From there each conversation is then inserted into BigQuery using Dataflow job. (#streaming inserts to BQ)

Architecture

Export Conversations to BQ

Contributions Welcome

Please feel free to contribute to the code base by submitting a pull request.