SpiderX allows you to watch movies by scraping data from the internet. The frontend is built with Vue.js and the scraping process is handled by python scrapy framework.
- The user searches will be first sent to firestore, if the search is not found in the firestore then the scraper (using python scrapy framework) will be called to initiate the scraping process.
- Once the scraper scrapes the data, the data scraped will be sent to firestore.
- Using firestore real-time updates, the vue interface will be updated in real-time.
- You need to setup firestore and initialize the db in src/firebase.js
import firebase from 'firebase/app';
import 'firebase/firestore';
// fill the values you get from firestore
const config = {
apiKey: '<apiKey>',
authDomain: '<authDomain>',
databaseURL: '<databaseURL>',
projectId: '<projectId>',
storageBucket: '<storageBucket>',
messagingSenderId: '<messagingSenderId>',
appId: '<appId>',
measurementId: '<measurementId>',
}
firebase.initializeApp(config);
const db = firebase.firestore();
export default db;
- You need to configure .env at the root of the project
VUE_APP_SPIDERX_HEROKU=<URLHERE> // the python scraper will be up soon
- Then run
npm i && npm run serve
to see the spider in action