/dril-archive

An archive of all dril tweets until Elron Husk killed the API

Primary LanguagePHPDo What The F*ck You Want To Public LicenseWTFPL

dril-archive

Due to the uncertain future of Twitter dot com many have decided to pack their bags to be ready to move on in case the site ceases to exist. People started to worry for the whereabouts of dril's tweets - a modern prophet - "who could only emerge on an app like Twitter".

I started to write a backup tool to augment the incomplete twitter archive downloads and of course dril's feed was used for testing purposes. Around the same time, Nick Farruggia shared a Google spreadsheet with every dril tweet, that i then used as basis to compile a JSON from the Twitter web API. Eventually, the archive, along with the scripts i used to download and compile ended up in this repository in order to compile the data into other formats and run a static website via GitHub pages.

Downloads

Ok, this is a bit messy, but releases are hardly feasible for this type of repo. Instead, each build creates an artifact with the files that are committed to the gh-pages branch for the static website. You can download these artifacts under "actions/Build" and filter by "scheduled". Click on the latest workflow and scroll down to "artifacts" - there you are!

Requirements

Installation

PHP for Windows

  • download the latest PHP for your system (usually x64, thread safe) from windows.php.net and unzip it to a folder of your choice
  • copy/rename the php.ini-development to php.ini and open the latter in an editor
    • search for extension_dir, uncomment this line (under "on windows")
    • search for extension=curl, uncomment this line (remove the semicolon)
    • search for extension=mbstring, uncomment this line
    • search for extension=openssl, uncomment this line
  • install composer: Windows install
  • optional: add PHP and Composer to your system PATH
  • optional: install git for windows and/or the GitHub desktop client

It might be necessary to provide a CA file for OpenSSL:

  • download the cacert.pem from https://curl.se/ca/cacert.pem (e.g. into the PHP folder)
  • in the php.ini
    • search for curl.cainfo, uncomment this line and add .\cacert.pem or c:\path\to\cacert.pem
    • search for openssl.cafile, uncomment this line and add the same path to the cacert.pem as above

PHP for Linux

  • add the PPA ondrej/php: sudo apt-add-repository ppa:ondrej/php
  • update the package list: sudo apt-get update
  • install php: sudo apt-get install php8.1-cli php8.1-common php8.1-curl php8.1-mbstring php8.1-openssl php8.1-xml
  • install composer: installation guide

A PHP installation guide can be found over here on digitalocean.com.

Library installation

  • download the zip file or clone the repo via git clone https://github.com/codemasher/dril-archive (or use the desktop client)
  • in the library root, next to the composer.json run composer install to install the dependencies
  • in the /config folder, copy the .env_example to .env, open it in an editor and fill in the details for your twitter developer account - currently, only the Bearer token is necessary.
  • download the dril spreadsheet and save it as /.build/dril.csv

Usage

Initial build

The official timeline/search API endpoints only return up to 3200 tweets unless you have academic access. So the initial build uses the undocumented/inoffical adaptive search API that's being used by Twitter's web search, which means, the tokens from the developer account won't work here. Obtaining the credentials is a bit messy and described in the following steps:

  • open /cli/build-clean.php in an editor
  • open https://twitter.com/search in a webbrowser (chrome or firefox recommended), ideally in an incognito tab
  • open the developer console (press F12)
  • type anything in the twitter search box, hit enter
  • go to the "network" tab in the dev console and filter the requests for adaptive.json
  • click that line, a new tab for that request appears
  • there, in the "headers" tab, scroll to "request headers" and look for Authorization: Bearer ...
  • right click that line, select "copy value" and replace the value of $requestToken in build-clean.php
  • scroll a bit further and look for the x-guest-token header, copy the value and replace the $guestToken value in build-clean.php
  • adjust the $query variable to your liking (a valid twitter search) - omit the since and until keywords here to download the user's full timeline

The x-guest-token is valid for about 2 hours, the bearer token at least for a day.

Now that everything's set up, you can run php build-clean.php in the ./cli directory and watch the console for a while... 🍵 The output will be stored under /output and you can open the index.html in a browser, the API responses are cached under /.build/<query-value>.

Incremental update

The incremental update utilizes the v1 API search endpoint, which returns 20 results with standard access. It will also update the user profiles in the timeline. The query should be the same as the one used to run the initial build. Run php incremental-update.php in ./cli and grab ☕ (this script is also used in the daily-run workflow).

Counter update

The counter update uses the v2 tweets endpoint with the tweet.fields=public_metrics expansion to update the stale counter values of an existing timeline. Run php update-counts.php in ./cli 🍰

Disclaimer

The scripts to create the archive are licensed under the WTFPL.
All tweets and media remain under copyright by their respective creators.