Twitter tap class.
Built with the Meltano SDK for Singer Taps and Targets.
catalog
state
discover
about
stream-maps
Setting | Required | Default | Description |
---|---|---|---|
bearer_token | True | None | The bearer token to authenticate against the Twitter API using the OAuth 2.0 flow outlined here - https://developer.twitter.com/en/docs/authentication/oauth-2-0/application-only |
user_ids | True | None | List of user IDs of Twitter accounts for which to fetch data |
url_patterns | False | None | List of URL patterns for which to fetch tweets |
start_date | False | None | The earliest record date to sync |
A full list of supported settings and capabilities is available by running: tap-twitter --about
pipx install git+https://github.com/voxmedia/tap-twitter.git
A full list of supported settings and capabilities for this tap is available by running:
tap-twitter --about
This tap requires a Twitter developer account. Visit Twitter Apps to set one up and get credentials.
You can easily run tap-twitter
by itself or in a pipeline using Meltano.
tap-twitter --version
tap-twitter --help
tap-twitter --config CONFIG --discover > ./catalog.json
pipx install poetry
poetry install
Create tests within the tap_twitter/tests
subfolder and
then run:
poetry run pytest
You can also test the tap-twitter
CLI interface directly using poetry run
:
poetry run tap-twitter --help
Testing with Meltano
Note: This tap will work in any Singer environment and does not require Meltano. Examples here are for convenience and to streamline end-to-end orchestration scenarios.
Your project comes with a custom meltano.yml
project file already created. Open the meltano.yml
and follow any "TODO" items listed in
the file.
Next, install Meltano (if you haven't already) and any needed plugins:
# Install meltano
pipx install meltano
# Initialize meltano within this directory
cd tap-twitter
meltano install
Now you can test and orchestrate using Meltano:
# Test invocation:
meltano invoke tap-twitter --version
# OR run a test `elt` pipeline:
meltano elt tap-twitter target-jsonl
See the dev guide for more instructions on how to use the SDK to develop your own taps and targets.