people-lover/request-boost

A better requests and urllib. A simple package for hitting multiple URLs and performing GET/POST requests in parallel.

request-boost

A better requests and urllib package.
A simple package for hitting multiple URLs and performing GET/POST requests in parallel.

Setup

pip install request-boost

Important: Virtual Environment is recommended

Usage

# Sample data
number_of_sample_urls = 1000
urls = [ f'https://postman-echo.com/get?random_data={test_no}' for test_no in range(number_of_sample_urls) ]
headers = [{'sample_header':test_no} for test_no in range(number_of_sample_urls)]

from request_boost import boosted_requests

results = boosted_requests(urls=urls)
print(results)

from request_boost import boosted_requests

results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=headers, verbose=True)
print(results)

from request_boost import boosted_requests

# Config
number_of_sample_urls = 100

# Sample data
urls = [f'https://postman-echo.com/get?random_data={test_no}' for test_no in range(number_of_sample_urls)]
post_urls = [f'https://postman-echo.com/post' for test_no in range(number_of_sample_urls)]
headers = [{'sample_header': test_no} for test_no in range(number_of_sample_urls)]
data = [{'sample_data': test_no} for test_no in range(number_of_sample_urls)] # Required for POST requests, 
#For POST request data can be just list of empty dict but not NONE

simple_results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=None, verbose=False)
header_results = boosted_requests(urls=urls, no_workers=16, max_tries=5, timeout=5, headers=headers)
post_results = boosted_requests(urls=post_urls, no_workers=16, max_tries=5, timeout=5, headers=headers, data=data, verbose=True)

Documentation

boosted_requests(urls, no_workers=8, max_tries=3, timeout=10, headers=None, data=None, verbose=True)

Get data from APIs in parallel by creating workers that process in the background
    :param urls: list of URLS
    :param no_workers: maximum number of parallel processes
    :param max_tries: Maximum number of tries before failing for a specific URL
    :param timeout: Waiting time per request
    :param headers: Headers if any for the URL requests
    :param data: data if any for the URL requests (Wherever not None a POST request is made)
    :param verbose: Show progress [True or False]
    :return: List of response for each API (order is maintained)

Credits

Maintained by

Kuldeep Singh Sidhu

Github: github/singhsidhukuldeep https://github.com/singhsidhukuldeep

Website: Kuldeep Singh Sidhu (Website) http://kuldeepsinghsidhu.com

LinkedIn: Kuldeep Singh Sidhu (LinkedIn) https://www.linkedin.com/in/singhsidhukuldeep/

Contributors

The full list of all the contributors is available here

Say Thanks

If this helped you in any way, it would be great if you could share it with others.

Steps for publishing to pypi [This is just for me, Maybe!]

• pip3 install setuptools twine
• Go to project folder
• python3 setup.py sdist
• twine upload --repository-url https://upload.pypi.org/legacy/ dist/*

OR

Go to your project folder and:

pip3 install setuptools twine

python3 setup.py sdist
twine upload --repository-url https://upload.pypi.org/legacy/ dist/*

Current Status