Field coverage always 100%
oyhel opened this issue · 5 comments
I am using spidermon 1.19.0 with Scrapy 2.9.0.
When setting 'SPIDERMON_ADD_FIELD_COVERAGE': True I get output stats for all the fields for the Item after the spider has completed. However, it always report 100% coverage for all fields regardless of whether the item field is empty or not.
The only exception is if I try to set values for all the fields to None
loader.add_value('myvar', None)
In this case the spidermon_item_scraped_count/MyItem/myvar log entry does not list myvar at all.
This renders it impossible to set field coverage limits since it will always be 1.0 or not present...
I assume this is not expected behaviour?
Hello, spidermon calculates coverage based on the fields existing on the item. If you are using a item class that always defines all their fields with some default value like None, "". Alternatively, items like dict and scrapy.Items should work fine.
regardless of whether the item field is empty or not.
Yes, there is a difference between non-empty and non-existent, and coverage is calculated based on the latter.
Thank you for the clarification. If I do not load the Item with the variable at all I still get no spidermon_item_scraped_count for the variable so I am a little unsure what is "triggering" these counts (since neither non-existent nor empty values seem to be counted). Could I bother you with briefly explaining what would be defined as 'non-existent' in this case? It would be very useful to have item coverage working...
Thanks!
If I do not load the Item with the variable at all I still get no spidermon_item_scraped_count for the variable
Which is good?
I am a little unsure what is "triggering" these counts (since neither non-existent nor empty values seem to be counted).
Per your original report empty values were counted while non-existent ones were not, which is the expected behavior.
Could I bother you with briefly explaining what would be defined as 'non-existent' in this case?
Non-existent means the item doesn't have the field at all. Note that setting it to None is the same as not setting it at all, as None values are ignored by ItemLoader, not sure if it's related to your questions though.
Thank you for the clarification, I got it working!
The thing was that my pipeline errored out with a key error if I did not specify a value. For this reason I was passing None values where the value were not found. Once I understood that instead of doing (in the pipeline)
thing.variable = item['variable']
I could do
thing.variable = item.get('variable')
The pipeline allowed non-existent values and the item coverage works as you describe.