/trex

youtube & tiktok analysis + youchoose recommendation custmizer. backend, extensions, and tooling

Primary LanguageHTMLGNU Affero General Public License v3.0AGPL-3.0

Notice of discontinuation

August 2023: Dear reader, as you may have read on the Tracking Exposed website, we're going through a complete restructuring/rebrainding/reframing. This repository, which hosts the TikTok and YouTube analytics toolkit, inherited code from 2016 and was becoming cumbersome to maintain. It has also undergone a major progressive refactoring from javascript to typescript over the last two years, as part of a more robust release pipeline that has served our algorithm accountability efforts.

This code was originally forked from the repositories used to analyse Facebook (extension and backend), and from there the AGPL-3 licence is inherited.

Other sites that have a more accurate deprecation notice are

Tracking Exposed toolkit

This monorepo will eventually include all packages needed and platforms supported by Tracking Exposed:

Requirements

  • node >=16
  • yarn >=3.2.3
  • node-canvas deps depending on your OS
  • docker

Monorepo structure

To start the services in production:

yarn pm2 start platforms/ecosystem.config.js
yarn pm2 status

(also for extension reviewer) how to build the extensions:

  • tiktok: yarn; yarn tk:ext dist; ls platforms/tktrex/extension/dist/*.zip
  • youtube: yarn; yarn yt:ext dist; ls platforms/yttrex/extension/dist/*.zip
  • youchoose: yarn; yarn ycai dist; ls platforms/ycai/studio/build/extension/*.zip

To assist debug

you might run yarn tsc-diagnostics and check out the content of diagnostics/ directory.

Supported Platforms

tktrex

The browser extension of tiktok.tracking.exposed the tiktok algorithm analysis toolkit for researcher, power user, and algorithm analysts.

yttrex

The browser extension of youtube.tracking.exposed the youtube algorithm analysis toolkit for researcher, power user, and algorithm analysts.

Initially sponsored by ALEX from University of Amsterdam DATACTIVE reseaerch group. Maintained by the Technical team of Tracking Exposed, more details on youtube.tracking.exposed.

Guardoni

A complete Pupetteer wrapper to orchestrate reproducible data collection with YTTrEx extension, documented with the name of Guardoni

Maintained by the Technical and Research team of Tracking Exposed, more details on youtube.tracking.exposed.

YCAI

The browser extension for YouChoose.AI and studio dashboard studio.youchoose.ai

Sponsored by the European Commission Ledger project in 2021, Develope by the technical team of YouChoose AI a project by Tracking Exposed. It is listed separately as we consider YouChoose should develop its own governance, reach out to us if you want to know more.

Note on supported platforms

As you can see in Tracking Exposed website a few other platforms are supported, work in progress, or discontinued. For example: Pornhub, Facebook, Amazon. They are not imported in this repository, but making this repository a shared resource and a monorepo is part of the refactor begun in 2021.

Packages

Taboule

A portable data table written in React to display TRex data by pre-configured API.

Tests

Tests are powered by jest and can be run all at once

yarn test

or by specific workspace

yarn yt:ext test

Run spec tests

To execute all the spec (unit testing) test files in the repo run:

yarn test spec --coverage

Run end-to-end tests

yarn pm2 start platforms/ecosystem.dev.config.js --env test
yarn test e2e
yarn pm2 stop all

Coverage output

To produce a coverage report run

yarn test --coverage

and the output will be produced at coverage/lcov-report/index.html

License

Affero-GPL 3, as file attached in this repository display.