twitter-x-scraper

scrape from twitter/x and post to facebook page, control via telegram bot

Installation

Line by line:

git clone https://github.com/RFS-ADRENO/twitter-x-scraper.git
cd twitter-x-scraper

npm install

Configuration

Rename .env.example to .env and fill in the values.

Linux:

mv .env.example .env

Windows:

rename .env.example .env

Telegram Bot

Create new telegram bot via BotFather and get the token.
For more information, see here.

Create new telegram channel and add the bot.
Now fill in the values in .env file

TELEGRAMBOT_TOKEN="your_token_here"
WHITELIST_CHAT_ID="your_channel_id_here"

Start the bot

npm start

Go to your channel and send /uid, the bot will send you your id.
Grab the id and fill in the .env file

ADMIN_IDS="id1,id2,id3"

As you can see, you can add multiple ids, just separate them with comma. The bot will only accept commands if the channel id is equal to WHITELIST_CHAT_ID and the sender id is equal to one of the ADMIN_IDS.

Facebook App

First you need to create a facebook app from here.

Follow the steps below while creating the app:

Choose Other

Chose Business

Then fill in the app name and contact email and click Create App

Now, you need a Privacy Policy URL.
You can create one from here for free.

Then navigate to App settings > Basic and fill in the Privacy Policy URL and click Save Changes

Now toggle the App Mode from Development to Live

Facebook Page

This bot support posting to multiple facebook pages.
First, create a page from here if you haven't already.

Then go to Graph API Explorer and select your app from the dropdown menu.
Then click Get Token > Get Page Access Token

Follow the instructions and select pages that you want to post to.

After you connected pages to your app, click Get Token or User Token, this time you will see your page names.
Select the page that you want to post, add pages_manage_posts permission.
Then click Generate Access Token then copy the token.

Then copy the token and paste it in .env file

Because this bot support posting to multiple pages, you need to follow the naming rule.

FBTOKEN_PAGE_ONE = "your_token_here"
FBTOKEN_PAGE_TWO = "your_token_here"

Start the program

Now all done, you can start the program

npm start

Go to your telegram channel and send /pages to see the list of pages that you connected to your app.

Commands

Command	Description
`/uid`	Get your telegram id
`/pages`	Get the list of pages that you connected to your app
`/startTop`	Start scraping from top section
`/startLatest`	Start scraping from latest section
`/stop`	Stop scraping

Example

startTop and startLatest commands will add a job to the queue.
The job will be executed when the previous job is finished.

/startTop pageNumber | hashtag1 hashtag2 ...
/startLatest pageNumber | hashtag1 hashtag2 ...

for example: /startTop 1 | openai gpt This will scrape from top section of tweets that contains both openai and gpt hashtags.

LICENSE

This project is licensed under the MIT License - see the LICENSE file for details