Toimik

Open source projects for an upcoming web search engine

Singapore

Pinned Repositories

CommonCrawl
Common Crawl's processing tools
Language:C#9 2 00
IpAddressEnumeration
IP address enumerators
Language:C#0 1 00
RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language:C#0 2 00
SitemapsProtocol
Parsers for sitemap / sitemap index (aka Sitemaps Protocol)
Language:C#0 2 00
UrlNormalization
URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical
Language:C#4 1 00
WarcProtocol
Parser for WARC (aka WebArchive) files
Language:C#11 1 43
Wikimedia
Wikimedia Downloads' processing tools
Language:C#0 1 00

Toimik's Repositories

toimik/WarcProtocol
Parser for WARC (aka WebArchive) files
Language:C#11 1 43
toimik/CommonCrawl
Common Crawl's processing tools
Language:C#9 2 00
toimik/UrlNormalization
URL normalizer to canonicalize (standardize) the text representation of a URL to determine if differently-formatted URLs are identical
Language:C#4 1 00
toimik/IpAddressEnumeration
IP address enumerators
Language:C#0 1 00
toimik/RobotsProtocol
Parsers for robots.txt (aka Robots Exclusion Standard / Robots Exclusion Protocol), Robots Meta Tag, and X-Robots-Tag
Language:C#0 2 00
toimik/SitemapsProtocol
Parsers for sitemap / sitemap index (aka Sitemaps Protocol)
Language:C#0 2 00
toimik/Wikimedia
Wikimedia Downloads' processing tools
Language:C#0 1 00