sul-dlss/wasapi-downloader

Issues

• Automate web archive crawl accessioning pipeline

  #124 opened 6 years ago by aaron-collier
  0

• devops docs: add wasapi-downloader docs

  #49 opened 7 years ago by ndushay
  0

• devops docs: update was-downloader VM docs

  #50 opened 7 years ago by ndushay
  0

• [was-registrar-app/stage] RuntimeError: Fetching WARCs failed: Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "store-time" (class edu.stanford.dlss.was.WasapiFile), not marked as ignorable (10 known properties: "checksums", "size", "crawl-time", "filetype", "filename", "crawl-start", "account", "locations", "crawl", "collection"]) at [Source: org.apache.http.conn.EofSensorInputStream@4b0b0854; line: 1, column: 565] (through reference chain...

  #126 opened 5 years ago by honeybadger
  1

• Enable incremental retries

  #123 opened 6 years ago by aaron-collier
  0

• Reduce RAM needs for large jobs

  #120 opened 7 years ago by ndushay
  2

• allow execution from arbitrary directory paths

  #119 opened 7 years ago by nullhandle
  0

• report download progress to console

  #118 opened 7 years ago by nullhandle
  0

• do a github release and deploy to VMs

  #117 opened 7 years ago by ndushay
  0

• integer settings validation improvements

  #83 opened 7 years ago by ndushay
  0

• actually deploy finished code to prod

  #80 opened 7 years ago by ndushay
  2

• setting? to choose which checksum to validate with

  #92 opened 7 years ago by ndushay
  3

• setting to set the number of retries per file download

  #93 opened 7 years ago by ndushay
  1

• review and approve README

  #99 opened 7 years ago by ndushay
  3

• tests: ensure "expected" is first value, "actual" is second value

  #101 opened 7 years ago by ndushay
  0

• rename "jobId" in command line to "crawlId" ?

  #98 opened 7 years ago by ndushay
  0

• tests: intermittent test failure

  #103 opened 7 years ago by ndushay
  1

• output shows download attempt number

  #109 opened 7 years ago by ndushay
  0

• tests: refactor TestWasapiDownloader_DownloadAndValidateFile for readability

  #105 opened 7 years ago by ndushay
  0

• download action retries until limit

  #64 opened 7 years ago by ndushay
  0

• download WARCs into correct location

  #74 opened 7 years ago by ndushay
  1

• hook up checksum validation of downloaded files

  #75 opened 7 years ago by ndushay
  0

• report network glitch during download to system.err

  #77 opened 7 years ago by ndushay
  0

• refactor: PrintStream instance vars that default to System. out and err

  #106 opened 7 years ago by ndushay
  0

• refactor WasapiValidator.validateResponse()

  #94 opened 7 years ago by ndushay
  0

• command line allows single file download requests

  #81 opened 7 years ago by ndushay
  0

• document how to use wasapi-downloader in the README

  #78 opened 7 years ago by ndushay
  0

• VMs get memory they need

  #76 opened 7 years ago by ndushay
  1

• build/deploy configures "ready-to-run" app

  #79 opened 7 years ago by ndushay
  0

• baseurl can end with or without slash

  #90 opened 7 years ago by ndushay
  0

• output basedir can end with or without slash

  #89 opened 7 years ago by ndushay
  0

• hook up crawl selection

  #73 opened 7 years ago by ndushay
  0

• remove crawlStartBefore setting - not implemented

  #85 opened 7 years ago by jmartin-sul
  4

• command line allows file list output (no download)

  #88 opened 7 years ago by ndushay
  0

• what settings go together, what settings conflict?

  #82 opened 7 years ago by jmartin-sul
  0

• add command line arg for last-known-crawl (and crawl-start-date-after if not there)

  #45 opened 7 years ago by ndushay
  1

• add account ID to settings

  #70 opened 7 years ago by jmartin-sul
  0

• WasapiFile: output json for each warc at crawl id level

  #62 opened 7 years ago by ndushay
  0

• be able to benchmark download timings with on/off setting

  #67 opened 7 years ago by ndushay
  0

• Add logging

  #41 opened 7 years ago by tingulfsen
  0

• WasapiDownloader: actually get filelist response based on args

  #38 opened 7 years ago by ndushay
  0

• validate command line arguments (and settings)

  #42 opened 7 years ago by ndushay
  3

• code doesn't work with https://partner.archive-it.org/wasapi/v1/login

  #68 opened 7 years ago by ndushay
  1

• Add tests for WasapiClient

  #60 opened 7 years ago by tingulfsen
  0

• Test WasapiConnection

  #59 opened 7 years ago by tingulfsen
  0

• move validate checksum methods to WasapiValidator

  #63 opened 7 years ago by ndushay
  0

• all settings should be available from either .properties file or command line args

  #54 opened 7 years ago by jmartin-sul
  0

• add lint checker (checkstyle?)

  #52 opened 7 years ago by ndushay
  1

• if wasapi adds login endpoint as part of spec, then remove it from settings file

  #57 opened 7 years ago by ndushay
  0

• wasapi change to crawl-start - fix WasapiFile class

  #43 opened 7 years ago by ndushay
  0