webrecorder/browsertrix-crawler

Issues

• some links on page not crawled

  #723 opened 2 months ago by robert-1043
  0

• WARC Record Write Failed

  #722 opened 2 months ago by benoit74
  1

• Support loading behaviors from a Git repo with branch + path

  #712 opened 2 months ago by ikreymer
  1

• Browser disconnected (crashed?)

  #706 opened 2 months ago by rgaudin
  10

• Youtube video is not crawled when `loading="lazy"`

  #699 opened 2 months ago by benoit74
  2

• Crawler is not failing when seed page returns an HTTP error code

  #719 opened 2 months ago by benoit74
  1

• Stop crawler when we have been hit by a WAF protection

  #711 opened 2 months ago by benoit74
  0

• Issue creating profile for intranet

  #705 opened 3 months ago by MRLeflei
  1

• Archiving particular site kills networking on host machine

  #703 opened 3 months ago by vitorio
  0

• Automatically crawl `<form>` URLs when `method` is `get`

  #702 opened 3 months ago by benoit74
  0

• BUG: Create Browser Profile --headless can't render at viewport

  #696 opened 3 months ago by gitreich
  3

• Document crawl collection layout

  #675 opened 3 months ago by tw4l
  0

• [Bug]: Crawl Configuration Inconsistency: Max Depth and Include Any Linked Page

  #693 opened 3 months ago by mona-ul
  4

• Include depth in pages jsonl files

  #690 opened 3 months ago by tw4l
  0

• Brotli decompression error

  #687 opened 4 months ago by rgaudin
  1

• Browser Crash & Docker Exit Code

  #683 opened 4 months ago by gitreich
  1

• Issue crawling a web property with big PDFs

  #676 opened 4 months ago by benoit74
  3

• A suggestion for making WACZ and WARC-requests

  #663 opened 4 months ago by hamoudak
  4

• SSH Socks5 Tunnel Proxy Support

  #670 opened 4 months ago by ikreymer
  1

• Execution context was destroyed

  #655 opened 5 months ago by rgaudin
  5

• ETA computation

  #660 opened 4 months ago by wsdookadr
  2

• [question] Missing or timed out dynamic request to resource

  #664 opened 5 months ago by wsdookadr
  1

• Use in-place streaming to generate WACZ files

  #674 opened 4 months ago by tw4l
  0

• Don't log behavior-related messages if no behaviors are set

  #651 opened 4 months ago by tw4l
  1

• Crawl button with javascript navigation

  #665 opened 5 months ago by hamzamac
  5

• [Bug]: Pages with no text to extract should be shown as a 100% match

  #666 opened 5 months ago by Shrinks99
  1

• [BUG] invalid gzipped WARC

  #662 opened 5 months ago by wsdookadr
  3

• Youtube Video Quality

  #648 opened 6 months ago by fservida
  1

• Remove invalid crc32 calculation

  #653 opened 5 months ago by ikreymer
  0

• Should invalid URL halt the scraping process?

  #654 opened 5 months ago by rgaudin
  2

• Behavior run partially failed - Protocol error

  #652 opened 5 months ago by zlodejpapiru
  0

• der-postillon.com: crawler considers that scrolling is not necessary while it seems mandatory

  #647 opened 6 months ago by benoit74
  2

• Fix skipping of 206 responses

  #645 opened 6 months ago by ikreymer
  0

• Can an AWS alternative to Access Keys be added?

  #644 opened 6 months ago by jblukach
  2

• Vimeo Playback: Retrieve full stream

  #632 opened 6 months ago by kila58
  1

• Skipping URL from unknown frame

  #643 opened 6 months ago by zlodejpapiru
  3

• Crawler keeps signing out(?)

  #642 opened 6 months ago by Azmodeszer
  2

• "Login form could not be found"

  #637 opened 6 months ago by Azmodeszer
  10

• Add support for WARC-Protocol and WARC-Cipher-Suite headers

  #641 opened 6 months ago by ikreymer
  0

• Better handling of redirect chains to same page

  #634 opened 6 months ago by ikreymer
  0

• Automatically add exclusion rules based on `robots.txt`

  #631 opened 6 months ago by benoit74
  5

• Revisit WARC-Resource-Type or add a new header

  #630 opened 6 months ago by benoit74
  4

• Crawler is not returning full seed page URL in WARC `WARC-Target-URI`

  #627 opened 6 months ago by benoit74
  1

• Could there be a way to create warcs with certain size after one RUN (combinewarc / rolloversize...)

  #617 opened 7 months ago by ssairanen
  5

• Misleading error message

  #598 opened 7 months ago by rgaudin
  2

• Crawl stopping with TypeError: Headers.append:...is an invalid header value.

  #607 opened 7 months ago by ldko
  1

• Crawl stopping with Node throwing ERR_INVALID_STATE.TypeError

  #606 opened 7 months ago by ldko
  2

• Fix seed redirect with multiple crawl instances (eg. scale > 1)

  #604 opened 7 months ago by ikreymer
  1

• Ensure responseReceived callback also checks shouldSkip()

  #601 opened 7 months ago by ikreymer
  1

• Make timeout logging messages warns, not errors

  #599 opened 7 months ago by tw4l
  0