medialab/sandcrawler

Issues

• Can you do child requests?

  #184 opened 4 years ago by kevinrademan
  5

• body.match is undefined

  #185 opened 4 years ago by kevinrademan
  2

• Can i scrape up to 50,000 pages in reasonable time ?

  #193 opened 8 years ago by scroobius-pip
  1

• Status?

  #192 opened 8 years ago by brandondrew
  14

• not a valid language tag

  #191 opened 9 years ago by ToruHyuga
  2

• Future plans

  #190 opened 9 years ago by moshewe
  0

• Promises in addition to callbacks?

  #189 opened 9 years ago by moshewe
  6

• PhantomJS library integration - phridge/node-phantomjs

  #187 opened 9 years ago by moshewe
  10

• CasperJS support?

  #188 opened 9 years ago by moshewe
  2

• What if no scraper?

  #186 opened 9 years ago by Yomguithereal
  0

• Handle nasty charset polymorphism

  #178 opened 10 years ago by Yomguithereal
  4

• A missing lib on a server caused sandcrawler to crash

  #183 opened 10 years ago by legaultpierre
  0

• Noop flag for scraper methods (navigational edge cases)

  #182 opened 10 years ago by Yomguithereal
  0

• Investigate some limit issues

  #181 opened 10 years ago by Yomguithereal
  0

• Isomorphism

  #180 opened 10 years ago by Yomguithereal
  0

• Display technologies used more conspicuously

  #179 opened 10 years ago by Yomguithereal
  0

• Passing context to scraper function as last argument

  #169 opened 10 years ago by Yomguithereal
  1

• phantomEngine.run won't autoClose

  #148 opened 10 years ago by Yomguithereal
  0

• Possibility to pass config object to the phantom engine run method.

  #153 opened 10 years ago by Yomguithereal
  0

• Phantomjs memory leak

  #160 opened 10 years ago by Yomguithereal
  1

• Add encoding support

  #168 opened 10 years ago by Yomguithereal
  0

• Add a function to addUrlNow

  #171 opened 10 years ago by boogheta
  1

• Handle encoding heuristics

  #176 opened 10 years ago by Yomguithereal
  0

• Handle discards in stats

  #172 opened 10 years ago by Yomguithereal
  0

• Proxy setting polymorphism through nodeUrl

  #175 opened 10 years ago by Yomguithereal
  0

• Document encoding

  #177 opened 10 years ago by Yomguithereal
  0

• Dual callback style result

  #150 opened 10 years ago by Yomguithereal
  0

• Add internal flag for droid and jawa

  #174 opened 10 years ago by Yomguithereal
  0

• Adjust the name string

  #162 opened 10 years ago by Yomguithereal
  0

• Adjust spider default name

  #173 opened 10 years ago by Yomguithereal
  0

• Possibility not to pass a scraper

  #149 opened 10 years ago by Yomguithereal
  0

• Example of json usage

  #170 opened 10 years ago by Yomguithereal
  0

• Check retryNow

  #163 opened 10 years ago by Yomguithereal
  0

• Check throttle

  #164 opened 10 years ago by Yomguithereal
  0

• Iterate index definition is incorrect

  #167 opened 10 years ago by Yomguithereal
  0

• Cheerio options

  #159 opened 10 years ago by Yomguithereal
  0

• Documentation site pull requests

  #165 opened 10 years ago by nucleardreamer
  1

• Experiment with onInitialized and shims/sniffers

  #166 opened 10 years ago by Yomguithereal
  0

• Publish newer version to NPM

  #161 opened 10 years ago by prokilogrammer
  5

• job:end should expose status

  #156 opened 10 years ago by Yomguithereal
  0

• Harmonize res.error

  #152 opened 10 years ago by Yomguithereal
  0

• Document stats

  #158 opened 10 years ago by Yomguithereal
  0

• Bind job to static scraper scope

  #146 opened 10 years ago by boogheta
  2

• Autoretry later/now polymorphism

  #155 opened 10 years ago by Yomguithereal
  0

• Should not throttle the first job

  #151 opened 10 years ago by Yomguithereal
  0

• Navigation edge cases

  #147 opened 10 years ago by Yomguithereal
  0

• Fault tolerance

  #157 opened 10 years ago by Yomguithereal
  0

• Add page-level events

  #154 opened 10 years ago by Yomguithereal
  0

• Example of user agent plugin

  #145 opened 10 years ago by Yomguithereal
  0

• jQuery/cheerio pitfalls

  #144 opened 10 years ago by Yomguithereal
  0