aiohttp_chromium

aiohttp-like interface to chromium

based on selenium_driverless to bypass cloudflare

status

working prototype

usage

aiohttp_chromium is a drop-in replacement for aiohttp

import asyncio

#import aiohttp
import aiohttp_chromium as aiohttp

async def main():
    async with aiohttp.ClientSession() as session:
        async with session.get('http://httpbin.org/get') as resp:
            print(resp.status)
            print(await resp.text())

asyncio.run(main())

why

handling file downloads with selenium is too verbose, and too complex to integrate into selenium, so this is a wrapper for selenium

i wanted a "stupid http client", so it has the same interface as aiohttp.client, and handling web pages has lower priority, so the selenium interface is hidden in response._driver

known issues

chromium window is stealing focus

when creating new tabs, or when switching between tabs, the chromium window is grabbing focus

this is an issue with the window manager

workaround for the KDE plasma desktop: move the chromium window to a different desktop, and focus some window

chromium seems to have no command line switch to disable this focus-grabbing

possible solutions

run chromium in a LD_PRELOAD wrapper
binary patching of the chromium executable
configure the window manager

todo

graphical interface where the user can solve challenges: captchas, unexpected responses, ...
integration with captcha solving services
remove unfree dependencies
- selenium_driverless - cc by-nc-sa license
  - selenium_driverless is a high-level wrapper for the Chrome DevTools Protocol (CDP)
grep -r -w FIXME src/
grep -r -w TODO src/

keywords

web scraper
chromium
aiohttp
web scraping
asyncio
bypass cloudflare
headful scraper
headful web scraper
headful chromium
gui scripting
headful webscraper
selenium driverless

similar projects

botasaurus

kaliiiiiiiiii/aiohttp_chromium