commoncrawl/cc-pyspark

Issues

• Drop support for Python2

  #7 opened 2 years ago by sebastian-nagel
  1

• Example using Resiliparse's HTML parser or text extractor

  #43 opened 2 years ago by sebastian-nagel
  0

• Provide classes to use FastWARC to read WARC/WAT/WET files

  #37 opened 2 years ago by sebastian-nagel
  0

• Drop support for Python 2.7

  #40 opened 2 years ago by sebastian-nagel
  1

• Use simdjson to read WAT payloads

  #41 opened 2 years ago by sebastian-nagel
  0

• boto3 credentials error when running CCSparkJob with ~100 S3 warc paths as input, but works with <10 S3 warc paths as input

  #32 opened 2 years ago by praveenr019
  5

• Looks like ccspark tried to access everything from local file. What's wrong with the settings?

  #39 opened 2 years ago by GenuineReader
  1

• download only specific language data from wet files like Warc

  #29 opened 2 years ago by aliebrahiiimi
  1

• Incompatible Architecture

  #34 opened 2 years ago by swetepete
  4

• Host link extraction does not represent every IDN as IDNA

  #35 opened 2 years ago by sebastian-nagel
  0

• Bad Substitution

  #33 opened 2 years ago by swetepete
  3

• Use SparkSession instead of SQLContext

  #24 opened 3 years ago by sebastian-nagel
  5

• Class org.apache.hadoop.fs.s3a.S3AFileSystem not found

  #30 opened 3 years ago by BrownXing
  2

• How To: process CC NEWS warc files, most recent first

  #27 opened 3 years ago by lgov
  3

• Can not run server_count example on Windows locally

  #28 opened 4 years ago by brand17
  6

• Broken links in README

  #23 opened 4 years ago by gamtiq
  1

• Common Crawl Index Table - Need for Schema Merging to be documented

  #18 opened 4 years ago by chk2817
  2

• Test and update examples to work with ARC files of the 2008 - 2012 crawls

  #20 opened 4 years ago by sebastian-nagel
  0

• Processing English only archives

  #17 opened 5 years ago by jaehunro
  2

• Webgraph construction: include nodes with zero outgoing links

  #15 opened 5 years ago by sebastian-nagel
  0

• Document dependency of CCIndexSparkJob to Java S3 file system libs

  #13 opened 5 years ago by sebastian-nagel
  0

• Commands to execute python files?

  #12 opened 5 years ago by calee88
  7

• CCIndexWarcSparkJob requires one of --query or --csv

  #14 opened 5 years ago by sebastian-nagel
  0

• Avoid to set Spark configuration properties in code

  #11 opened 5 years ago by sebastian-nagel
  0

• Allow to access WARC record filename and offset

  #6 opened 5 years ago by sebastian-nagel
  2