Plutus is a data crawling project designed to fetch SJC gold price data from a specific website and store it in a database. The collected data can then be visualized using Grafana to generate charts and graphs. Additionally, historical data from the past 1 or 2 years can be obtained from another website.
To use Plutus, follow the steps below:
- Ensure you have Node.js and npm installed on your machine.
- Clone the Plutus repository to your local machine.
- Run
npm install
to install the required dependencies.
To crawl new gold price data from the designated website and store it in the database, you need to set up a cron job to run the script at regular intervals.
-
Define a cron job to run the script every 30 minutes. Use the following command:
*/30 * * * * /path/cronjob.sh >> /path/logs 2>&1 ``` Replace `/path/cronjob.sh` with the actual path to the script.
To crawl historical data from the past 1 or 2 years, follow these steps:
-
Run
npm install
to ensure all dependencies are installed. -
Execute the following command:
bash /path/crawl_data.sh ``` Replace `/path/crawl_data.sh` with the actual path to the script.
-
The script will generate a file named
final.sql
. -
Run the
final.sql
file to insert the crawled historical data into the database.
The structure of the database used by Plutus will be provided at a later stage. Please refer to the documentation for details on the table schema and data organization.
The Database Structure documentation will be updated with the actual structure once it becomes available.
The configuration for the Grafana dashboard will be updated and provided at a later stage. It will include the necessary settings to connect to the Plutus database and visualize the gold price data.
Please note that the Grafana dashboard configuration is currently being worked on and will be updated in the near future.
Contributions to Plutus are welcome! If you encounter any issues or have ideas for improvements, please feel free to submit pull requests or open issues on the project's repository.
Plutus is released under the MIT License.