asyncurl

Asynchronous cURL Requests using python, which is inspired by this benchmark(KR, EN).

Support python versions

python >= 3.6

Dependencies

AsyncURL project consists of the following packages:

Package	Version	Description
asyncio	>=3.4.3	for Asynchronous
requests	>=2.22.0	pycurl substitutes
uvloop	>=0.12.2	for event loop policy

Installation

You can download asyncurl executable and binary distributions from PyPI.

Using pip

pip install asyncurl

Usage

Import AsyncURL.

from asyncurl.fetch import AsyncURLFetch

ac_fetch = AsyncURLFetch()

Default worker's count is 2. you can change it if you want.

ac_fetch.worker = 3

and you can put urls to <AsyncURL.queue>.

for x in range(2):
    ac_fetch.queue.put_nowait('http://localhost')

ac_fetch.parallel()

Then call parallel(). The fucntion fetch urls using <requests>(HTTP library for Python).

and AsyncURL can change <requests>'s method and else properties.

from asyncurl.session import AsyncURLSession
from asyncurl.fetch import AsyncURLFetch

ac_fetch = AsyncURLFetch()

for x in range(2):
    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36'}
    session = AsyncURLSession('GET', 'http://localhost', headers=headers)
    ac_fetch.queue.put_nowait(session)

ac_fetch.parallel()

AsyncURLSession is inheritance of <requests.Session>.

parallel() will return <AsncURLFetch>, and it can show results to you.

Show results:

ac_fetch.parallel().results

The order of result is nonsequential. and it will return list of <requests.Response>.

AsyncURLSession's properties

equals to <requests.Request>

params=None
data=None
headers=None
cookies=None
files=None
auth=None
timeout=None
allow_redirects=True
proxies=None
hooks=None
stream=None
verify=None
cert=None
json=None

Examples

for x in range(3):
    session = AsyncURLSession('GET', 'http://localhost')
    ac_fetch.queue.put_nowait(session)
    
# case.1) with callback
print('[with callback]')
ac_fetch.parallel(callback=lambda x: print('with callback : {0}'.format(x)))

# case.2) return results
print('[return results]')
print(ac_fetch.parallel().results)

>>>
[with callback]
with callback : <Future finished result=<Response [403]>>
with callback : <Future finished result=<Response [403]>>
with callback : <Future finished result=<Response [403]>>

[return results]
[<Response [403]>, <Response [403]>, <Response [403]>]

License

The MIT License (MIT)

hidekuma/asyncurl