google/sredocs

sredocs

sredocs is to turn postmortems, charters and other SRE docs into machine readable data for analysis while retaining human readability properties.

This repository contains code for downloading postmortems and/or charters following certain template(s) from Google Docs, parsing the data to CSV and support for uploading the result to Google BigQuery.

Here is a demo dashboard for incident analysis using this, Google Data Studio and BigQuery. It contains an interactive report with information related to five dummy postmortems.

sredocs currently parses postmortems using a single specific template (sample) and charter that will be published separately later.

Download

Here is how to use the download mode:

1. Go to https://console.developers.google.com. I'd recommend creating a new project called "sredocs".
2. Switch to that project. Search for "Google Drive API" then enable it.
3. Go to APIs & Services -> Credentials.
4. Click on "Create credentials" then select Service account key.
5. Give your Service account key a name, e.g "sredocs".
6. Take note of its address (looks like an email address). Do not assign it a role while creating it.
7. Save the private key (it will be automatically downloaded).
8. Switch to Google Drive.
9. Give your service account address, e.g sredocs@...iam.gservice... view access to the folder with SRE docs.
10. Pass the private key to sredocs via -download_credentials_path.
11. Run mkdir <download_dir>
12. Run sredocs -mode=download -cloud_credentials=<creds.json> -download_folder=<docs_folder> -download_destination=<download_dir>

Parse

Here is how to use the parse mode:

1. Run mkdir <parsed_dir>
2. Run sredocs -mode=parse -parse_kind=auto -parse_path=<download_dir> -parse_output_path=<parsed_dir>

Upload

Here is how to use the upload mode:

1. Follow the instructions for Download first.
2. Go to https://console.developers.google.com.
3. Navigate to IAM & Admin.
4. Click on the Add button.
5. Search for your service account, e.g sredocs@... Give it the BigQuery Admin role.
6. Run sredocs -mode=upload -cloud_credentials=<creds.json> -upload_path=<parsed_dir> -upload_project=<org:project> -upload_dataset=<sredocs> -upload_table=<sredocs_20190603> -upload_truncate=true

BYOP

You can also bring your own parser.

type Parser interface {
        CompileRegex(fields []string) ([]*regexp.Regexp, error)
        Parse(fields []string, b []byte) (*bytes.Buffer, error)
        CSVHeader(regexps []*regexp.Regexp) []string
        NamedGroup(field string) string
        Save(buf *bytes.Buffer, filename string) error
}

You will have to modify charter/* and postmortem/* to switch away from the DefaultParser.

You can also extend sredocs by adding a new kind of document by using charter/* as a template.