reactor is not waiting till the crawling is completed

[shankar939014]

Hi ,

I am trying to use your code to scrape data in real time. I got the urls from the pickle file and passed to the crawler. Data got scraped and returned the list.

My use case says that whenever user submits the request the from UI, app has to load the urls from the pickle file, pass them to crawler and return the data to UI. Below is the code snippet to wait until the crawling is completed.

import json

from flask import Flask
from scrapy.crawler import CrawlerRunner
from scrapy import log, signals

from quote_scraper import QuoteSpider
from googleapiclient.discovery import build
import pickle
import time
from twisted.python import log
from twisted.internet import endpoints, reactor,task,defer


app = Flask('Scrape With Flask')

crawl_runner = CrawlerRunner()      # requires the Twisted reactor to run
data_list=[]                 # store quotes
urls=[]


@app.route('/crawl')
def crawl_for_quotes():
  
  global urls
  global data_list
  
  with open('urls.pkl', 'rb') as handle:
      urls = pickle.load(handle)
             
  eventual=crawl_runner.crawl(QuoteSpider,url_list=urls,data_list=data_list)
  eventual.addBoth(lambda _: reactor.stop())
  reactor.run()
  return json.dumps(data_list)

if __name__=='__main__':
  from sys import stdout

  from twisted.logger import globalLogBeginner, textFileLogObserver
  from twisted.web import server, wsgi
  from twisted.internet import endpoints, reactor

  # start the logger
  globalLogBeginner.beginLoggingTo([textFileLogObserver(stdout)])

  # start the WSGI server
  root_resource = wsgi.WSGIResource(reactor, reactor.getThreadPool(), app)
  factory = server.Site(root_resource)
  http_server = endpoints.TCP4ServerEndpoint(reactor, 9000)
  http_server.listen(factory)

  # start event loop
  reactor.run()
 

Issue is that the script is not getting blocked until the crawling is finished. App is stopped before crawling. Also every time I curl, data is getting appended to the list. Is there any way to instantiate list every time we curl. Because once the app is started , data to the list will be appended every time user submits the request which is happening with the existing code.

Hoping that you would help me.