cldellow/datasette-scraper

Issues

• 500 Error with dss_crawl and a few other tables after installing

  #53 opened 7 months ago by unbracketed
  0

• javascript interaction?

  #52 opened a year ago by ar-jan
  0

• table overload: consider hiding tables and using views or other affordances

  #51 opened 2 years ago by cldellow
  0

• usability: if discover-urls returns a str, raise an exception

  #50 opened 2 years ago by cldellow
  0

• dss fails when its not the primary database

  #49 opened 2 years ago by cldellow
  0

• permit installing into a database with pre-existing tables

  #48 opened 2 years ago by cldellow
  0

• plugin: extract-ecommerce-shopify

  #25 opened 2 years ago by cldellow
  0

• support scheduled crawls

  #10 opened 2 years ago by cldellow
  1

• plugin: extract-ecommerce-json-ld

  #24 opened 2 years ago by cldellow
  0

• make plugins pluggable

  #9 opened 2 years ago by cldellow
  0

• link dss_extract_stats table column to actual table

  #47 opened 2 years ago by cldellow
  0

• link dss_job_stats hostname column to dss_crawl_queue_history

  #46 opened 2 years ago by cldellow
  0

• make ops db configurable

  #45 opened 2 years ago by cldellow
  1

• be able to rewrite existing fetch_cache entries

  #43 opened 2 years ago by cldellow
  0

• be able to train dictionaries automatically

  #44 opened 2 years ago by cldellow
  1

• support zstd dictionaries

  #37 opened 2 years ago by cldellow
  0

• consider a knob to hide _dss_ tables

  #5 opened 2 years ago by cldellow
  1

• be able to edit host rates in datasette

  #38 opened 2 years ago by cldellow
  2

• take over row page for _dss_crawl

  #41 opened 2 years ago by cldellow
  0

• take over table page for _dss_crawl table

  #40 opened 2 years ago by cldellow
  0

• crawl_queue_history should have fkey to fetch_cache

  #42 opened 2 years ago by cldellow
  0

• consider setting default sort orders

  #39 opened 2 years ago by cldellow
  0

• plugin: extract-links

  #23 opened 2 years ago by cldellow
  1

• hook: extract_from_response

  #32 opened 2 years ago by cldellow
  1

• Have a way to make an HTTP request that goes through the plugin system

  #34 opened 2 years ago by cldellow
  1

• plugin: seed-sitemaps

  #12 opened 2 years ago by cldellow
  0

• be more aggressive in asserting Wal mode

  #35 opened 2 years ago by cldellow
  0

• plugins: max-pages, max-pages-per-domain

  #20 opened 2 years ago by cldellow
  0

• plugin: canonicalize-shopify-urls

  #18 opened 2 years ago by cldellow
  0

• plugin: discover-allow

  #17 opened 2 years ago by cldellow
  1

• plugin: discover-deny

  #16 opened 2 years ago by cldellow
  0

• consider: re-insert (or update) a crawl item if has a lower depth

  #36 opened 2 years ago by cldellow
  1

• plugin: discover-redirect-urls

  #14 opened 2 years ago by cldellow
  0

• plugin: fetch-cache

  #21 opened 2 years ago by cldellow
  1

• hook: config_schema, config_default_values

  #33 opened 2 years ago by cldellow
  1

• plugin: discover-only-same-origin

  #15 opened 2 years ago by cldellow
  0

• hook: canonicalize_urls

  #31 opened 2 years ago by cldellow
  0

• hook: discover_urls

  #30 opened 2 years ago by cldellow
  0

• plugin: discover-html-urls

  #13 opened 2 years ago by cldellow
  0

• hook: fetch_url

  #29 opened 2 years ago by cldellow
  0

• add background workers that Do The Thing

  #7 opened 2 years ago by cldellow
  5

• plugin: max-depth

  #19 opened 2 years ago by cldellow
  0

• hook: before_fetch_url

  #28 opened 2 years ago by cldellow
  0

• add plugin system

  #8 opened 2 years ago by cldellow
  0

• plugin: seed-urls

  #11 opened 2 years ago by cldellow
  0

• hook: get_seed_urls

  #27 opened 2 years ago by cldellow
  0

• plugin: extract-object

  #26 opened 2 years ago by cldellow
  0

• plugin: extract-seo

  #22 opened 2 years ago by cldellow
  0

• consider dashboards

  #6 opened 2 years ago by cldellow
  0

• throw if the default database is _memory, or isn't mutable

  #4 opened 2 years ago by cldellow
  0