This is an api crawler that fetches public apis data from POSTMAN. Support for rate limiting and pagination.
More reference - Public APIs github repo (https://github.com/public-apis/public-apis) is a collective list of free APIs for use in software and web development.
- To fetch all api data to database
python3 main.py
in source file- All data should be fetched by this command to database files
- To run queries on data base simply run
python3 queryDB.py
- Your code should follow concept of OOPS
- Support for Authentication & token expiration
- Support for Pagination to get all data
- Support for Rate limiting
- Crawled all API entries for all categories and stored it in a database
- Database is not scalable for more large number of entries. Use another Scalable Db framework.
- Need to implement checks while writing to db to avoid copies.
- A few unnecessary token calls are bring made and can be avoided.
- Implement better async code to speed up processing.
- Use docker to run the code.