TeamHG-Memex

Pinned Repositories

aquarium
Splash + HAProxy + Docker Compose
Language:Python194 16 3043
arachnado
Web Crawling UI and HTTP API, based on Scrapy and Tornado
Language:Python159 27 1865
autologin
A project to attempt to automatically login to a website given a single seed
Language:Python122 15 1843
deep-deep
Adaptive crawler which uses Reinforcement Learning methods
Language:Jupyter Notebook168 25 538
eli5
A library for debugging/inspecting machine learning classifiers and explaining their predictions
Language:Jupyter Notebook2.7k 67 257333
Formasaurus
Formasaurus tells you the type of an HTML form and its fields using machine learning
Language:HTML115 13 2047
html-text
Extract text from HTML
Language:HTML126 15 1624
scrapy-rotating-proxies
use multiple proxies with Scrapy
Language:Python721 21 52156
sklearn-crfsuite
scikit-learn inspired API for CRFsuite
Language:Python425 22 57213
tensorboard_logger
Log TensorBoard events without touching TensorFlow
Language:Python631 29 2454

TeamHG-Memex's Repositories

TeamHG-Memex/eli5
A library for debugging/inspecting machine learning classifiers and explaining their predictions
Language:Jupyter Notebook2.7k 67 257333
TeamHG-Memex/scrapy-rotating-proxies
use multiple proxies with Scrapy
Language:Python721 21 52156
TeamHG-Memex/tensorboard_logger
Log TensorBoard events without touching TensorFlow
Language:Python631 29 2454
TeamHG-Memex/sklearn-crfsuite
scikit-learn inspired API for CRFsuite
Language:Python425 22 57213
TeamHG-Memex/aquarium
Splash + HAProxy + Docker Compose
Language:Python194 16 3043
TeamHG-Memex/deep-deep
Adaptive crawler which uses Reinforcement Learning methods
Language:Jupyter Notebook168 25 538
TeamHG-Memex/arachnado
Web Crawling UI and HTTP API, based on Scrapy and Tornado
Language:Python159 27 1865
TeamHG-Memex/html-text
Extract text from HTML
Language:HTML126 15 1624
TeamHG-Memex/autologin
A project to attempt to automatically login to a website given a single seed
Language:Python122 15 1843
TeamHG-Memex/Formasaurus
Formasaurus tells you the type of an HTML form and its fields using machine learning
Language:HTML115 13 2047
TeamHG-Memex/autopager
Detect and classify pagination links
Language:HTML97 16 825
TeamHG-Memex/page-compare
Simple heuristic for measuring web page similarity (& data set)
Language:HTML90 15 118
TeamHG-Memex/scrapy-crawl-once
Scrapy middleware which allows to crawl only new content
Language:Python77 8 423
TeamHG-Memex/undercrawler
A generic crawler
Language:Python77 16 4026
TeamHG-Memex/soft404
A classifier for detecting soft 404 pages
Language:Jupyter Notebook55 12 1014
TeamHG-Memex/agnostic
Agnostic Database Migrations
Language:Python52 10 1818
TeamHG-Memex/autologin-middleware
Scrapy middleware for the autologin
Language:Python38 8 1015
TeamHG-Memex/json-lines
Read JSON lines (jl) files, including gzipped and broken
Language:Python34 10 59
TeamHG-Memex/scrapy-kafka-export
Scrapy extension which writes crawled items to Kafka
Language:Python29 13 29
TeamHG-Memex/MaybeDont
A component that tries to avoid downloading duplicate content
Language:Python27 6 214
TeamHG-Memex/sitehound-frontend
Site Hound (previously THH) is a Domain Discovery Tool
Language:HTML23 16 513
TeamHG-Memex/domain-discovery-crawler
Broad crawler for domain discovery
Language:Python19 6 310
TeamHG-Memex/url-summary
Show summary of a large number of URLs in a Jupyter Notebook
Language:Python17 14 09
TeamHG-Memex/sitehound
This is the facade for installation and access to the individual components
Language:Shell16 5 08
TeamHG-Memex/docker-tor-rotator
A rotating socks proxy using Tor, Delegate and Haproxy
Language:Dockerfile14 14 15
TeamHG-Memex/hh-page-classifier
Headless Horseman Page Classifier service
Language:Python7 10 06
TeamHG-Memex/scrapy-cdr
Item definition and utils for storing items in CDR format for scrapy
Language:Python7 7 46
TeamHG-Memex/scrash-lua-examples
A collection of example LUA scripts and JS utilities
Language:JavaScript7 9 14
TeamHG-Memex/sitehound-backend
Sitehound's backend
Language:HTML6 12 05
TeamHG-Memex/sshadduser
A simple tool to add a new user with OpenSSH keys.
Language:Python2 4 11