edgi-govdata-archiving/web-monitoring-processing

Issues

• Figure out how to handle bad gzipped content from Wayback

  #309 opened 5 years ago by Mr0grog
  6

• Rewrite Dockerfile to run imports and healthcheck and run it on ECS

  #757 opened 2 years ago by Mr0grog
  2

• Don't require credentials in DB client

  #844 opened 2 years ago by Mr0grog
  0

• Use a more robust PDF library

  #655 opened 2 years ago by Mr0grog
  4

• Keep track of canonical URLs from page markup in `source_metadata`

  #728 opened 3 years ago by Mr0grog
  1

• Lint with flake8

  #705 opened 3 years ago by Mr0grog
  1

• Handle too many connections to Wayback automatically

  #525 opened 5 years ago by Mr0grog
  3

• Move docs-building to GH Actions

  #671 opened 3 years ago by danielballan
  1

• Import script should upload bodies directly to S3

  #663 opened 4 years ago by Mr0grog
  1

• Most DB API methods should support arbitrary keyword args

  #661 opened 4 years ago by Mr0grog
  2

• Import script should include client-side redirects

  #670 opened 4 years ago by Mr0grog
  1

• Remove differs, which are now in web-monitoring-diff

  #687 opened 4 years ago by danielballan
  0

• Optionally pre-check known versions in `wm import ia-known-pages`

  #664 opened 4 years ago by Mr0grog
  1

• Import script should follow more of a pipeline style

  #669 opened 4 years ago by Mr0grog
  1

• DB API needs methods for iterating over paginated results

  #660 opened 4 years ago by Mr0grog
  0

• DB API should have built-in retry functionality

  #659 opened 4 years ago by Mr0grog
  0

• Import records should not have an empty string for `mime_type`

  #620 opened 4 years ago by Mr0grog
  2

• Provide clearer messaging when zero records imported from IA

  #582 opened 4 years ago by lightandluck
  1

• Update import script to use Wayback v0.3

  #641 opened 4 years ago by Mr0grog
  1

• Extract diffs and diff server into separate repo & Python package

  #638 opened 4 years ago by Mr0grog
  7

• `db.Client` should include timeouts for all HTTP requests

  #486 opened 4 years ago by Mr0grog
  6

• Parameter Names and Types are Displayed Wrong in Docs

  #468 opened 5 years ago by Mr0grog
  3

• Integrate scripts/annotations_import and scripts/ia_healthcheck into scripts/wm

  #529 opened 5 years ago by Mr0grog
  2

• Use dotenv for environment configuration

  #499 opened 5 years ago by Mr0grog
  1

• Diffing server does not release ports on SIGTERM

  #576 opened 5 years ago by Mr0grog
  0

• Imports from Wayback list the final redirect URL twice

  #563 opened 5 years ago by Mr0grog
  0

• `wm import ia` should validate URL before trying to import

  #543 opened 5 years ago by Mr0grog
  0

• ☂️ Cron jobs should run as Kubernetes Cron Jobs

  #376 opened 5 years ago by ibuys
  4

• Put wayback API in a separate package.

  #477 opened 5 years ago by danielballan
  9

• DB client should set the `Accept` header on requests

  #495 opened 5 years ago by Mr0grog
  1

• Use `dateutil` for Date Parsing Everywhere

  #360 opened 5 years ago by Mr0grog
  1

• If fetched content does not match hash in diffing server, don’t return 500 error

  #491 opened 5 years ago by Mr0grog
  1

• Use Python native async syntax in diff server

  #432 opened 5 years ago by Mr0grog
  0

• Switch MAINTAINER to LABEL directive in Docker

  #467 opened 5 years ago by Mr0grog
  0

• Remove Pagefreezer support

  #379 opened 5 years ago by Mr0grog
  2

• Add a built-in, pluggable cache to the diffing server

  #265 opened 5 years ago by danielballan
  6

• Compare URLs with any HTTP status

  #404 opened 5 years ago by vbanos
  2

• Use Sentry’s Tornado integration for diff server

  #342 opened 5 years ago by Mr0grog
  1

• Handle Upcoming Tornado HTTPClient Errors

  #282 opened 6 years ago by Mr0grog
  2

• Upgrade to Python 3.7

  #341 opened 6 years ago by Mr0grog
  1

• Support ETag headers

  #264 opened 6 years ago by danielballan
  1

• Page with an unbelievable number of dropdown options fails to parse in the middle

  #288 opened 6 years ago by Mr0grog
  4

• IA healthcheck script should only check active pages

  #328 opened 6 years ago by Mr0grog
  0

• DB API should support new `active` property in `list_pages`

  #327 opened 6 years ago by Mr0grog
  0

• Investigate why diffing server does not seem to be handling concurrent jobs well

  #303 opened 6 years ago by Mr0grog
  3

• Support invalid encoding `iso-8559-1`

  #310 opened 6 years ago by Mr0grog
  2

• _decode_body fails if body is empty

  #311 opened 6 years ago by Mr0grog
  0

• DiffMatchPatch can’t handle null terminators

  #312 opened 6 years ago by Mr0grog
  0

• Set up Sentry.io for the diffing server

  #306 opened 6 years ago by Mr0grog
  0

• Protect `html_text_dmp` and `html_source_dmp` with content-type checking and sniffing

  #287 opened 6 years ago by Mr0grog
  0