This is a Singer tap that produces JSON-formatted data following the Singer spec to retrieve Nikabot timesheeting data from the Nikabot API.
Install the package
$ pip install .
Discover the streams
$ tap-nikabot -c config.json --discover > catalog.json
Run the tap (you will need to edit the catalog to add "selected": true
to a stream's metadata)
$ tap-nikabot -c config.json --catalog catalog.json
See the Singer documentation for full usage.
tap-nikabot supports the following streams:
- users
- roles
- groups
- teams
- projects
- records
Where records are the timesheet entries for a user.
A sample config is provided in config_sample.json
.
Property | Default | Description |
---|---|---|
access_token | None | Required. Nikabot API access token |
page_size | 1000 | Allows configuring the number of records returned in each server request. |
start_date | None | The timesheet date to start pulling records from. If not provided will sync from the beginning of time. |
end_date | None | The timesheet date to start pulling records up to. If not provided will sync up to todays date, but see note about cutoff days. |
cutoff_days | 10 | When using incremental replication, will only sync records up to this many days before todays date. |
Start date and end date take preferrence over bookmarks. If you have a bookmark that is earlier than the configured start date, syncing will start from start date. If you have a bookmark that is greater than end date, no records will be returned.
The Nikabot API only allows for filtering timesheet records by timesheet day, and only returns the created date not a modified date, his makes incremental replication a challenge. To support incremental replication, we've introduced a concept of cutoff days.
When replication-method: "INCREMENTAL"
is specified in the catalog for the records stream, the timesheet date is used as a bookmark / replication key and the tap will only sync records up to "cutoff_days" (defaults to 10 days) before today. Users have the cutoff days period to enter their timesheet information after which their entries will not be synced.
Note that because the timesheet date is a date only (not time), you will only be update to sync at most once per day.
Cutoff days cannot be combined with an end date in config. If an end date is provided, cutoff days will be ignored and records will be synced right up to end date.
The Nikabot API returns dates in ISO 8601format without any timezone information. This is not compatible with JSON Schema which requires RFC 3339 formatted dates, which are a subset of the ISO specification but mandates timezones. To allow the Nikabot data to be successfully vaidated against the schema, we post-process the data to add timezone information by assuming all dates are in UTC.
So 2019-09-02T05:13:43.151
becomes 2019-09-02T05:13:43.151000Z
.
This process uses the schema to determine which dates to reformat, if there is no schema information for the datetime field (e.g. blank schema {}
), then the dates will not be adjusted.
A Makefile is provided to manage a virtual environment.
$ make init
Will setup a virtual environment in .venv
and install the package and all (development and production) dependencies.
To run the linter / autoformatter (provided by black)
$ make lint
To run the test suite (which also runs the linter)
$ make test
To run the tap in discovery mode (loads config from config.json
)
$ make discover
Or to run it in sync mode (loads catalog from catalog.json
and config from config.json
)
$ make sync
You can proxy all requests through a proxy tool like OWASP ZAP, BurpSuite, Fiddler or Charles Proxy using environment variables to control the python requests pacakge. And easy way to manage this is with a shell script that sets environment variables. Create the script env.sh
:
#!/bin/sh
##
# Script to execute a command with specific environment variables set.
# Note that environment variables passed as arguments will need to be escaped to avoid shell expansion.
#
# USAGE:
# ./env.sh my-command
# ./env.sh 'echo $MY_VAR'
##
set -e
export HTTP_PROXY="http://localhost:8080"
export HTTPS_PROXY="http://localhost:8080"
export REQUESTS_CA_BUNDLE="C:\path\to\owasp_zap_root_ca.cer"
eval "$@"
Update REQUESTS_CA_BUNDLE to point to the CA certificate for the proxy tool. Now pass your command to the script to run it.
$ ./env.sh make discover > catalog.json
$ ./env.sh tap-nikabot -c config.json --catalog catalog.json
Copyright © 2020 Stitch