Graphinator is a network building and visualisation tool, designed to be used by people without extensive IT knowledge.
Easy to learn, hard to master 😎
- Make sure you have
npm
andNode.js
version18.3.0
or greater installed. - Clone this repository into a local directory and
cd
into it. - Rename all configuration files in the
config
directory by removing.template
from their name and changing their contents appropriately (ex: renametwitter.template.json
->twitter.json
, updateusername
andpassword
values inside). - Install dependencies.
npm install
- Start the web server.
npm start
- Open a web browser and navigate to
127.0.0.1:8080
. - Have a look at the user manual, and enjoy scraping!
Please refer to the User manual for a complete usage guide.
- Write a developer manual documenting the module API and a guide for adding new modules.
- Implement more modules.
- Cache the (most) scraped accounts in a database and set up a crontab for updating it on a regular basis.
- Before deploying at scale, add some form of authentication/captcha for scraping tasks to avoid denial-of-service attacks.
The followings retrieval functionality of the Twitter module is rather limited in speed.
This is due to the fact that it performs a live scraping of the followers list on the Twitter website, which loads batches of ~16 followers, ~300ms each.
This can become especially noticeable when scraping the followings of large
accounts (over 1000 followings), and could benefit from a layer of caching
(see Next steps
).
The tool has been written in the form of a browser application because this allows it to remain simple in usage while being able to perform complex scraping tasks on the back-end.
Besides, this allows it to be hosted like any other website and used by investigators all around the world without the need to write a single line of code, SQL query or terminal command.