twai is a tool to scrape tweets and add a score to them using AI.
📢 Connect with us! Join our Telegram group for support and collaboration: t.me/igohub
- Scrape tweets from Twitter using a headless chrome browser
- Add a score from 1 to 10 to each tweet using AI
- Add a score using Elo rating system, comparing tweets to each other
You can use the Golang binary to install twai:
go install github.com/igolaizola/twai/cmd/twai@latest
Or you can download the binary from the releases
You need to capture the cookie from Twitter website.
- Go to https://x.com/home
- Login if you are not already logged in
- Open the developer tools (F12)
- Go to the "Network" tab
- Refresh the page
- Click on the first request to https://x.com/home
- Go to the "Request Headers"
- Copy the "cookie" header and save it in a
cookie.txt
file
You need to have an AI model to score the tweets.
You can use a local model via Ollama.
- Download and install Ollama tool
- Run
ollama run llama3
to download the model - Set the
host
tohttp://localhost:11434/v1
and themodel
tollama3
Or you can use an online model like OpenAI.
- Get an API key from OpenAI
- Set the
model
togpt-3.5-turbo
and thetoken
to your API key
You can launch the twai command without any arguments or simply double-click on the binary to start the web UI. The web UI will be available at the address printed in the console.
Alternatively, you can use the command-line interface to interact with the tool.
Scrape tweets from a user or from the home page:
twai scrape --config scrape.yaml
#scrape.yaml
debug: false #(bool): Debug mode
page: home #(string): Page to fetch (home or username)
n: 50 #(int): Number of tweets to fetch
followers: false #(bool): Fetch followers stats, it will take longer
output: scape.csv #(string): Output file
show-browser: false #(bool): Show browser
cookie-file: cookie.txt #(string): Cookie file
Add a score to the tweets:
twai score --config score.yaml
#score.yaml
debug: false #(bool): Debug mode
concurrency: 1 #(int): Number of concurrent requests to the AI
input: scrape.csv #(string): Input file (generated by scrape command)
output: score.csv #(string): Output file (csv)
prompt: "Rate the following tweet from 1 to 10 based on relevance, clarity, engagement, and impact. Only answer with a number." #(string): Prompt
model: llama3 #(string): AI model (e.g., llama3, gpt-3.5-turbo)
host: "http://localhost:11434/v1" #(string): AI endpoint host (not needed for openai)
token: "" #(string): Authorization token (required for openai)
Add a score to the tweets using the Elo rating system:
twai elo --config elo.yaml
#elo.yaml
debug: false #(bool): Debug mode
concurrency: 1 #(int): Number of concurrent requests to the AI
input: scrape.csv #(string): Input file (generated by scrape command)
output: elo.csv #(string): Output file (csv)
iterations: 10 #(int): Number of iterations
prompt: "Which tweet is best? 1 or 2? Answer only with the number 1 or 2." #(string): Prompt
model: llama3 #(string): AI model (e.g., llama3, gpt-3.5-turbo)
host: "http://localhost:11434/v1" #(string): AI endpoint host (not needed for openai)
token: "" #(string): Authorization token (required for openai)
Launch twai
with the --help
flag to see all available commands and options:
twai --help
You can use the --help
flag with any command to view available options:
twai scrape --help
The automation of Twitter accounts is a violation of their Terms of Service and will result in your account(s) being terminated.
Read about Twitter Terms of Service and Community Guidelines.
twai was written as a proof of concept and the code has been released for educational purposes only. The authors are released of any liabilities which your usage may entail.
If you have found my code helpful, please give the repository a star ⭐
Additionally, if you would like to support my late-night coding efforts and the coffee that keeps me going, I would greatly appreciate a donation.
You can invite me for a coffee at ko-fi (0% fees):
Or at buymeacoffee:
Donate to my PayPal:
Or sponsor me on GitHub:
github.com/sponsors/igolaizola
Thanks for your support!