CI-Research

Pinned Repositories

cdx-index-client
A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
Language:Python1 3 01
CI-HiBench
Big Data benchmark from Intel called HiBench
Language:Shell0 2 00
common_crawl_index
Index URLs in Common Crawl
Language:Python0 2 01
commoncrawl-examples
A library of examples showing how to use the Common Crawl corpus.
Language:Java0 1 00
CommonCrawlDocumentDownload
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types for mass-testing of frameworks like Apache POI and Apache Tika
Language:Java0 4 01
dkpro-c4corpus
DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate removal, language detection, and near-duplicate removal.
Language:Java0 2 00
HiBench
HiBench is a big data benchmark suite.
Language:Java0 3 00
KeywordAnalysis
Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
57 5 111
spark-Jupyter-AWS
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
Language:Jupyter Notebook1 3 00

CI-Research's Repositories

CI-Research/KeywordAnalysis
Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends
57 5 111
CI-Research/cdx-index-client
A command-line tool for using CommonCrawl Index API at http://index.commoncrawl.org/
Language:Python1 3 01
CI-Research/spark-Jupyter-AWS
A guide on how to set up Jupyter with Pyspark painlessly on AWS EC2 clusters, with S3 I/O support
Language:Jupyter Notebook1 3 00
CI-Research/CI-HiBench
Big Data benchmark from Intel called HiBench
Language:Shell0 2 00
CI-Research/common_crawl_index
Index URLs in Common Crawl
Language:Python0 2 01
CI-Research/commoncrawl-examples
A library of examples showing how to use the Common Crawl corpus.
Language:Java0 1 00
CI-Research/CommonCrawlDocumentDownload
A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types for mass-testing of frameworks like Apache POI and Apache Tika
Language:Java0 4 01
CI-Research/dkpro-c4corpus
DKPro C4CorpusTools is a collection of tools for processing CommonCrawl corpus, including Creative Commons license detection, boilerplate removal, language detection, and near-duplicate removal.
Language:Java0 2 00
CI-Research/HiBench
HiBench is a big data benchmark suite.
Language:Java0 3 00