medialab/minet

Issues

• Upgrade ural

  #986 opened 3 months ago by Yomguithereal
  0

• CrawlJob data type should not be wrapped in an optional by default

  #981 opened 4 months ago by Yomguithereal
  0

• Retire minet.buzzsumo, minet.crowdtangle

  #984 opened 4 months ago by Yomguithereal
  0

• Upgrade trafilatura

  #985 opened 4 months ago by Yomguithereal
  0

• ThreadsafeBrowser enhancements

  #963 opened 8 months ago by Yomguithereal
  0

• Add LoadingBar.track

  #964 opened 4 months ago by Yomguithereal
  0

• Crawler should lazily open data files to be written until there is actually something to write

  #951 opened 4 months ago by Yomguithereal
  0

• Adjust twitter scraper retryer and rate limit (again)

  #980 opened 4 months ago by Yomguithereal
  0

• potential changes in rate limit of twitter public API

  #976 opened 4 months ago by taniki
  3

• Add FORWARD_SPIDER option

  #970 opened 4 months ago by Yomguithereal
  0

• Spider process exceptions should at least be raised with some context around them

  #971 opened 4 months ago by Yomguithereal
  0

• Add more automatic context when Spider.process raises

  #979 opened 4 months ago by Yomguithereal
  0

• Issues (core dump or cannot unpack non-iterable FocusCrawlInfo object)

  #977 opened 4 months ago by TeaS0710
  1

• -I should default to "downloaded" in scrape and extract

  #974 opened 4 months ago by Yomguithereal
  1

• Forward SelectionError to minet.scrape

  #983 opened 4 months ago by Yomguithereal
  0

• Command to add jobs to a crawler's queue

  #982 opened 5 months ago by Yomguithereal
  0

• KeyError: 'expanded_url' with minet twitter scrape tweets

  #978 opened 6 months ago by lakonis
  5

• tiktok search-videos error

  #972 opened 7 months ago by csamson-sf
  3

• When -c is not specified, we should default to test all available browsers instead of only firefox

  #975 opened 7 months ago by Yomguithereal
  1

• "Invalid Twitter cookie!" error (possibly due to migration from twitter.com to x.com ?)

  #973 opened 7 months ago by leomignot
  3

• Retrieve videos from instagram hashtag function

  #967 opened 8 months ago by Tyrannas
  9

• Scrapping 1000's of comments on Instagram

  #968 opened 7 months ago by Geminy3
  3

• Error on wikipedia pageviews

  #969 opened 7 months ago by bmaz
  1

• Improve ThreadsafeBrowser.request stability by retrying content acquisition if needed

  #965 opened 8 months ago by Yomguithereal
  0

• instagram post-infos should have line parity in the output and increase a stat rather than log

  #962 opened 8 months ago by Yomguithereal
  0

• /usr/local/bin does not exist on recent mac installs

  #936 opened 8 months ago by Yomguithereal
  0

• Add a playwright version of the crawler

  #957 opened 8 months ago by Yomguithereal
  0

• Refactor Crawler request_args as inheritance

  #961 opened 8 months ago by Yomguithereal
  0

• Spider process error should lead to errorred crawl?

  #958 opened 8 months ago by Yomguithereal
  1

• User friendly error message when spider returns a non-2-tuple

  #941 opened 8 months ago by Yomguithereal
  0

• Multithreaded API clients connection pools should allow more connections

  #946 opened 8 months ago by Yomguithereal
  0

• Upgrade trafilatura and deal with lxml_html_clean

  #959 opened 8 months ago by Yomguithereal
  0

• Move away from lxml as default soup engine?

  #952 opened 8 months ago by Yomguithereal
  0

• Method of CrawlJob to get an identical CrawlTarget to retry

  #955 opened 8 months ago by Yomguithereal
  0

• Upgrade rich and other deps

  #960 opened 8 months ago by Yomguithereal
  0

• Upgrade to min version py3.8

  #956 opened 8 months ago by Yomguithereal
  0

• os.makedirs is already threadsafe

  #953 opened 8 months ago by Yomguithereal
  0

• There should be a Crawler side global callback for each job

  #954 opened 8 months ago by Yomguithereal
  0

• Add some crawler level job filter

  #950 opened 9 months ago by Yomguithereal
  0

• Make minet installable without playwright

  #949 opened 9 months ago by Yomguithereal
  0

• Youtube improvements

  #948 opened 9 months ago by Yomguithereal
  0

• A challenges challenge

  #947 opened 9 months ago by Yomguithereal
  0

• Add a "minet hal" module?

  #945 opened 9 months ago by boogheta
  0

• ErroredCrawlReponse should de facto None response attributes

  #942 opened 9 months ago by Yomguithereal
  0

• Command tw tweets should not require API key

  #943 opened 9 months ago by Yomguithereal
  0

• x.com urls are not usually recognized

  #944 opened 9 months ago by Yomguithereal
  0

• path column is not very useful when using --glob file on scrape/extract

  #939 opened 10 months ago by Yomguithereal
  0

• Stop condition of twitter scraper is not working anymore

  #934 opened 10 months ago by Yomguithereal
  0

• lzma issues

  #937 opened 10 months ago by Yomguithereal
  0

• Telegram channel-messages crashes on string parsing

  #935 opened 10 months ago by Yomguithereal
  0