Social-Picker-API

This project is used to extract media from various posting platforms like Twitter, Reddit, Pixiv, Youtube, Tiktok, Osnova and many others. It's written for Node.js and it works as local service for other local services, e.g. Anime Ultra Bot or Social Picker Vue.

Usage

Start a server, make HTTP-request to it with a link to the post and receive that post's content back.

Configs

There are some configuration files:

service.json – service port and external service for viewing some content with additional local proxy
tokens.json – tokens for platforms and other filepaths. See Platform specific and types/configs.d.ts
pm2.production.json – config for pm2 daemon
Other configuration files for particular platforms – see Platform specific

Development config files can be created and placed along production ones (e.g. tokens.dev.json or service.dev.json). You can also install all npm modules (including dev) one with npm install. npm run dev will run service in dev-environment, npm run lint will use eslint.

How to run

Install necessary dependencies – npm i --production
Run production server – npm run production

After launching you can access Picker with fetching it like curl http://localhost:8080/?url=__LINK_TO_ANY_POST__ (change 8080 to real port you specified in service.json). If you're planning opening your Picker instance to the world, I'd suggest creating your own middleware service for handling user access, downloading combined videos, etc. (something like that).

List of supported platforms

Twitter – images, videos, GIFs and direct media from *.twimg.com. More details
Pixiv – images, single direct media from *.pximg.net and Ugoira-GIFs. More details
Reddit – images, videos, GIFs and galleries. More details
Youtube – video with response in default type containing all streams (via yt-dlp)
Tiktok – Any video in multiple formats. More details
Instagram – images, videos, galleries and Reels. More details
Osnova – images, videos, GIFs and galleries. More details
Coub – looped videos with linear audio. More details
Joyreactor – images and GIFs from multiple subdomains and direct links to media files
Tumblr – images and galleries. More details
KemonoParty – images
Danbooru – images (unstable)
Gelbooru – images (unstable)
Konachan – images (unstable)
Yandere – images (unstable)
Eshuushuu – images (unstable)
Sankaku – images (unstable)
Zerochan – images (unstable)
AnimePictures – images (unstable)

Platform specific

Twitter

Supports images, videos, GIFs (temporary unstable) and direct media from *.twimg.com.

This project uses Social Picker Twitter Scrapper – wrapper in Go to fetch tweets without API. Thus if you want to get Tweets, you should consider (before running this app) reading that project README and then placing compiled executable and created its config-files into such folders:

Move executable file Social-Picker-Twitter-Scrapper in bin folder
Move generated cookies.json in config, choose a explicit name for it, e.g. twitter-scrapper-cookies.json
- For convenience you can move credentials.json next to cookies.json (this app and executable binary use only cookies file).
Check and edit tokens.json – give a look to properties TWITTER_SCAPPER.binary_file_path and TWITTER_SCAPPER.cookies_file_path: paths are relative to project's root, default ones are in accordance to this example.

Pixiv

Supports images, single direct media from *.pximg.net and Ugoira-GIFs.

Uses external service for end-user viewing of high-res images due to Referer Header issues. Uses ugoira-builder for creating mp4 video from Ugoira ZIP with PNGs (via ffmpeg).

Supports images, videos, GIFs and galleries. Because Reddit's videos are divided from audios, this app utilizes video-audio-merge for merging separated streams (via ffmpeg).

Tiktok

Any video in multiple formats. Does not support image galleries.

Uses yt-dlp for extracting metadata and then video-codec-convert to convert original highest quality HEVC video without watermark to H264 file (via ffmpeg) for compatibility. Still prioritizes HEVC one.

Instagram

Supports images, videos, galleries and Reels.

To pick default regular post (square photos, videos and galleries) set property INSTAGRAM_COOKIE_ONE_LINE_FOR_POSTS in tokens.json. Place there contents of cookies from your web-browser's console (Press F12 —> Console tab —> type document.cookie —> press Enter).

To pick Reels:

Install this browser extension – it's open source and it supports Chrome and Firefox
Go to instagram.com
Copy cookies (click Copy button in extension popup)
Create file config/instagram.cookies.txt and paste there contents from clipboard (it starts with "# Netscape HTTP Cookie File")
Check and edit tokens.json – give a look to property INSTAGRAM_COOKIE_FILE_LOCATION_FOR_REELS: set path relative to project's root, default one is in accordance to this example.

Osnova

Supports images, videos, GIFs and galleries. Also extracts Twitter and Instagram blocks/links from within and handles them with parsers above.

Coub

Creates looped videos with linear audio, limits itself only with audio length and filesize if it ever exceed a reasonable size (20 MB). Uses video-audio-merge for merging separated streams (via ffmpeg).

Tumblr

Supports images and galleries, set OAuth keys and stuff in tokens.json.

hack3rsparadise/Social-Picker-API