maxdepth can not large than 2

[anthony9981]

Hi,
When I try to feed an url with this:
curl localhost:5343/feed -H "Content-Type: application/json" -d '{"url": "https://domain.com", "appid": "gd2", "crawlid": "gdcrawl2", "maxdepth": 5, "allowed_domains": ["domain.com"]}'

I always got:
{"message": "Did not find schema to validate request", "parsed": true, "valid": false, "logger": "kafka-monitor", "timestamp": "2020-12-18T03:06:51.564078Z", "data": {"url":...

But when decrease maxdepth to 2, the crawler worked.

So what exactly maxdepth for?
As I understand is
maxdepth=0: only get the current page. I guess this is the default value
maxdepth=1
maxdepth=2
What exactly maxdepth value for?

[mrasoolmirzaei]

By default, scrapy_cluster won't crawl websites with maxdepth larger than 3. You should change the schema first. To do this, login to your kafka_monitor container:

1- docker exec -it container_id bash
2- cd plugins
3- edit scraper_schema.json (change max value for maxdepth from 3 to anything you want)

from this point, you can crawl websites for maxdepth more than min value and less than the max value you just set.

[anthony9981]

Hi @NeoArio,
Thanks for your reply.
Your answer helps me a alot.
I have a question if you don't mind:

I have some knowable website, I need to crawl title and content in exactly selector for each of them.
Q1: How I can predefine CSS selector for each of them then feed the monitor only the domain?
Q2: And where I can take the scraped items then store to data base like elasticsearch?
I tried with pipelines (scrapy-elasticsearch) but it will ton of additional request to es server.

Sorry I'm new on scrapy. This is awesome!
Best regards,

[mrasoolmirzaei]

Hi! I hope you enjoy scraping :D

Q1: I have another database that stores websites CSS and XPath patterns. I don't know if what you want to try is really applicable.
Q2: scraped items will be pushed to the demo.crawled_firehose topic: https://scrapy-cluster.readthedocs.io/en/latest/topics/kafka-monitor/api.html#kafka-topics
Write a code to consume from this topic then do what you want with that data. Finally, you can send it to elasticsearch by another kafka pipline. I think it is better to insert in elasticsearch with bulk request, I mean each insertion contains 100 crawled link. Add a timeout beside this and you are perfect. 100 crawled link or 10 minutes are good conditions to insert in elasticsearch.

[anthony9981]

Hi @NeoArio ,
Idea about database that stores the selector is so nice, why I don't think about it before 👍
Could you please show me your?
I came from PHP to python then I'm here so Kafka is new for me:)
Thanks to point me up:) Let me learn it deeper.
Best regards,

[madisonb]

I'm happy to chat through custom implementations on Gitter, but per the guidelines I am going to close this issue as a "custom implementation" question which is beyond the scope of a true bug ticket/problem.

More generally - crawling at a depth beyond 2 gets your spider way into the weeds of the internet and is 99% of the time not useful for your actual request. If you wish to crawl at a greater depth you should also implement an allowed_domains filter or regex in the crawl api request to limit your crawler to a specific domain.

If you need to change anything else in the api spec for the request, you can do so at this file https://github.com/istresearch/scrapy-cluster/blob/master/kafka-monitor/plugins/scraper_schema.json