Extract all tweets for a given query. This uses Twitters private API (via twitter.com) to bypass quotas/limits.
dotnet tool install --global TwitterDump
First, you must setup the authentication.
twitter-dump auth
After you have followed the instructions and succesfully authenticated, you can now dump a search to a JSON file.
twitter-dump search -q "(from:realDonaldTrump)" -o trump.json
To install on Unbuntu 20.04 do:
$ wget https://packages.microsoft.com/config/ubuntu/20.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb
$ sudo dpkg -i packages-microsoft-prod.deb
$ sudo apt-get update
$ sudo apt-get install -y apt-transport-https && sudo apt-get update
$ sudo apt-get install dotnet-sdk-3.1
$ export PATH="$PATH:$HOME/.dotnet/tools"
$ dotnet tool install --global TwitterDump
The software is now installed. You now need to get a token from a logged in session:
- Using the Chromium browser log in to Twitter and then navigate to: https://twitter.com/search
- Open Chromium developer tools (Ctrl-Shift I)
- Open the Network tab on the developer tools
- Filter requests for "adaptive.json"
- Search for anything in the web page (doesn't matter what)
- Scroll down until a network request for "adapative.json" is made
- Right click the request and click "Copy -> Copy as cURL"
- In a terminal run: perl -pe 's/\\n//g' | twitter-dump auth
- Paste the contents of your clipboard
- Press Ctrl-D to close the input to 'perl'
Now you can run:
$ twitter-dump search -q "(from:GnuParallel)" -o gnuparallel.json
MIT