Bangbros not scraping
Closed this issue · 3 comments
cloudstrife11 commented
PS Y:\Stash\Webharvy\Scrapy-main\tpdb\spiders> scrapy crawl BangBros -a limit_pages=4 -o bangbrosneeeE342d.csv
2023-08-20 19:20:30 [scrapy.utils.log] INFO: Scrapy 2.10.0 started (bot: tpdb)
2023-08-20 19:20:30 [scrapy.utils.log] INFO: Versions: lxml 4.9.2.0, libxml2 2.9.12, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.3 (tags/v3.11.3:f3909b8, Apr 4 2023, 23:49:59) [MSC v.1934 64 bit (AMD64)],
pyOpenSSL 23.2.0 (OpenSSL 3.1.2 1 Aug 2023), cryptography 41.0.3, Platform Windows-10-10.0.22621-SP0
2023-08-20 19:20:30 [scrapy.addons] INFO: Enabled addons:
[]
2023-08-20 19:20:30 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_DEBUG': True,
'AUTOTHROTTLE_ENABLED': True,
'BOT_NAME': 'tpdb',
'CONCURRENT_REQUESTS': 1,
'CONCURRENT_REQUESTS_PER_DOMAIN': 1,
'CONCURRENT_REQUESTS_PER_IP': 1,
'HTTPCACHE_EXPIRATION_SECS': 7200,
'LOG_LEVEL': 'INFO',
'NEWSPIDER_MODULE': 'tpdb.spiders',
'RETRY_HTTP_CODES': [500, 502, 503, 504, 522, 524, 408, 429, 403, 302],
'SPIDER_MODULES': ['tpdb.spiders'],
'USER_AGENT': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 '
'(KHTML, like Gecko) Chrome/107.0.0.0 Safari/537.36 '
'Edg/107.0.1418.62'}
2023-08-20 19:20:30 [py.warnings] WARNING: C:\Users\Administrator\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\request.py:248: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINT
ER_IMPLEMENTATION' setting.
It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in
a future version of Scrapy.
See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
return cls(crawler)
2023-08-20 19:20:30 [scrapy.extensions.telnet] INFO: Telnet Password: 34da2
2023-08-20 19:20:30 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats',
'scrapy.extensions.throttle.AutoThrottle']
2023-08-20 19:20:30 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-08-20 19:20:30 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-08-20 19:20:31 [scrapy.middleware] INFO: Enabled item pipelines:
['tpdb.pipelines.TpdbApiScenePipeline']
2023-08-20 19:20:31 [scrapy.core.engine] INFO: Spider opened
2023-08-20 19:20:31 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-08-20 19:20:31 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-08-20 19:20:31 [scrapy.extensions.throttle] INFO: slot: bangbros.com | conc: 1 | delay: 5000 ms (+0) | latency: 703 ms | size: 124 bytes
2023-08-20 19:20:39 [scrapy.extensions.throttle] INFO: slot: bangbros.com | conc: 1 | delay: 2702 ms (-2297) | latency: 405 ms | size:152575 bytes
2023-08-20 19:20:39 [scrapy.core.engine] INFO: Closing spider (finished)
2023-08-20 19:20:39 [scrapy.extensions.feedexport] INFO: Stored csv feed (0 items) in: bangbrosneeeE342d.csv
2023-08-20 19:20:39 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/request_bytes': 635,
'downloader/request_count': 2,
'downloader/request_method_count/GET': 2,
'downloader/response_bytes': 154932,
'downloader/response_count': 2,
'downloader/response_status_count/200': 1,
'downloader/response_status_count/301': 1,
'elapsed_time_seconds': 8.142135,
'feedexport/success_count/FileFeedStorage': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2023, 8, 20, 17, 20, 39, 162867),
'httpcompression/response_bytes': 1264364,
'httpcompression/response_count': 1,
'log_count/INFO': 13,
'log_count/WARNING': 1,
'response_received_count': 1,
'scheduler/dequeued': 2,
'scheduler/dequeued/memory': 2,
'scheduler/enqueued': 2,
'scheduler/enqueued/memory': 2,
'start_time': datetime.datetime(2023, 8, 20, 17, 20, 31, 20732)}
2023-08-20 19:20:39 [scrapy.core.engine] INFO: Spider closed (finished)
PS Y:\Stash\Webharvy\Scrapy-main\tpdb\spiders>
cloudstrife11 commented
It not scraping anything, do i do something wrong? Some pages works, but bangbros is not working for me.
Darklyter commented
They changed to the Algolia platform a while back. There's finally a new version up
Kev1n121 commented
If I try ProjectOneService It not scraping anything.
PS Y:\Stash\Webharvy\scrapy-main3\scrapy-main> scrapy crawl ProjectOneService -a limit_pages=1 -o pos3.csv
C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spiderloader.py:49: UserWarning: There are several spiders with the same name:
networkDogfartSpider named 'Dogfart' (in tpdb.spiders.dogfart)
networkDogfartSpider named 'Dogfart' (in tpdb.spiders.networkDogfart)
This can cause unexpected behavior.
warnings.warn(
2023-09-21 15:48:31 [scrapy.utils.log] INFO: Scrapy 2.11.0 started (bot: tpdb)
2023-09-21 15:48:31 [scrapy.utils.log] INFO: Versions: lxml 4.9.3.0, libxml2 2.10.3, cssselect 1.2.0, parsel 1.8.1, w3lib 2.1.2, Twisted 22.10.0, Python 3.11.5 (tags/v3.11.5:cce6ba9, Aug 24 2023, 14:38:34) [MSC v.1936 64 bit (AMD64)],
pyOpenSSL 23.2.0 (OpenSSL 3.1.3 19 Sep 2023), cryptography 41.0.4, Platform Windows-10-10.0.22621-SP0
2023-09-21 15:48:31 [scrapy.addons] INFO: Enabled addons:
[]
2023-09-21 15:48:31 [py.warnings] WARNING: C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\request.py:254: ScrapyDeprecationWarning: '2.6' is a deprecated value for the 'REQUEST_FINGERPRINTER_IMPLE
MENTATION' setting.
It is also the default value. In other words, it is normal to get this warning if you have not defined a value for the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting. This is so for backward compatibility reasons, but it will change in
a future version of Scrapy.
See the documentation of the 'REQUEST_FINGERPRINTER_IMPLEMENTATION' setting for information on how to handle this deprecation.
return cls(crawler)
2023-09-21 15:48:31 [scrapy.extensions.telnet] INFO: Telnet Password: 0f270799d9337d7a
2023-09-21 15:48:31 [scrapy.middleware] INFO: Enabled extensions:
['scrapy.extensions.corestats.CoreStats',
'scrapy.extensions.telnet.TelnetConsole',
'scrapy.extensions.feedexport.FeedExporter',
'scrapy.extensions.logstats.LogStats']
2023-09-21 15:48:31 [scrapy.crawler] INFO: Overridden settings:
{'AUTOTHROTTLE_DEBUG': True,
'BOT_NAME': 'tpdb',
'HTTPCACHE_EXPIRATION_SECS': 7200,
'LOG_LEVEL': 'INFO',
'NEWSPIDER_MODULE': 'tpdb.spiders',
'SPIDER_MODULES': ['tpdb.spiders'],
'USER_AGENT': 'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:86.0) '
'Gecko/20100101 Firefox/86.0'}
2023-09-21 15:48:31 [scrapy.middleware] INFO: Enabled downloader middlewares:
['scrapy.downloadermiddlewares.httpauth.HttpAuthMiddleware',
'scrapy.downloadermiddlewares.downloadtimeout.DownloadTimeoutMiddleware',
'scrapy.downloadermiddlewares.defaultheaders.DefaultHeadersMiddleware',
'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware',
'tpdb.middlewares.TpdbSceneDownloaderMiddleware',
'scrapy.downloadermiddlewares.retry.RetryMiddleware',
'scrapy.downloadermiddlewares.redirect.MetaRefreshMiddleware',
'scrapy.downloadermiddlewares.httpcompression.HttpCompressionMiddleware',
'scrapy.downloadermiddlewares.redirect.RedirectMiddleware',
'scrapy.downloadermiddlewares.cookies.CookiesMiddleware',
'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware',
'scrapy.downloadermiddlewares.stats.DownloaderStats']
2023-09-21 15:48:31 [scrapy.middleware] INFO: Enabled spider middlewares:
['scrapy.spidermiddlewares.httperror.HttpErrorMiddleware',
'scrapy.spidermiddlewares.offsite.OffsiteMiddleware',
'scrapy.spidermiddlewares.referer.RefererMiddleware',
'scrapy.spidermiddlewares.urllength.UrlLengthMiddleware',
'scrapy.spidermiddlewares.depth.DepthMiddleware']
2023-09-21 15:48:31 [scrapy.middleware] INFO: Enabled item pipelines:
['tpdb.pipelines.TpdbApiScenePipeline']
2023-09-21 15:48:31 [scrapy.core.engine] INFO: Spider opened
2023-09-21 15:48:31 [scrapy.extensions.logstats] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-09-21 15:48:31 [ProjectOneService] INFO: Spider opened: ProjectOneService
2023-09-21 15:48:31 [scrapy.extensions.telnet] INFO: Telnet console listening on 127.0.0.1:6023
2023-09-21 15:48:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.stepmomlessons.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:48:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.biempire.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:48:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.sweetsinner.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:48:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.fakehub.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:48:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.sweetheartvideo.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:48:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.milehighmedia.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:48:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.stepmomlessons.com>: HTTP status code is not handled or not allowed
2023-09-21 15:48:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.biempire.com>: HTTP status code is not handled or not allowed
2023-09-21 15:48:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.sweetsinner.com>: HTTP status code is not handled or not allowed
2023-09-21 15:48:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.fakehub.com>: HTTP status code is not handled or not allowed
2023-09-21 15:48:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.sweetheartvideo.com>: HTTP status code is not handled or not allowed
2023-09-21 15:48:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.milehighmedia.com>: HTTP status code is not handled or not allowed
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
2023-09-21 15:49:31 [scrapy.extensions.logstats] INFO: Crawled 13 pages (at 13 pages/min), scraped 0 items (at 0 items/min)
2023-09-21 15:50:31 [scrapy.extensions.logstats] INFO: Crawled 13 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-09-21 15:51:31 [scrapy.extensions.logstats] INFO: Crawled 13 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-09-21 15:51:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.babes.com/scenes?site=218?utm_source=babesunleashed.com&utm_medium=redirect&utm_campaign=redirects> (failed 3 times): 429 Unknown Status
2023-09-21 15:51:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.doghousedigital.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:51:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.sexyhub.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:51:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.babes.com/scenes?site=218?utm_source=babesunleashed.com&utm_medium=redirect&utm_campaign=redirects>: HTTP status code is not handled or n
ot allowed
2023-09-21 15:51:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.doghousedigital.com>: HTTP status code is not handled or not allowed
2023-09-21 15:51:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.transsensual.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:51:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.sexyhub.com>: HTTP status code is not handled or not allowed
2023-09-21 15:51:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.transsensual.com>: HTTP status code is not handled or not allowed
2023-09-21 15:51:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.trueamateurs.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:51:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.tube8vip.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:51:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.trueamateurs.com>: HTTP status code is not handled or not allowed
2023-09-21 15:51:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.tube8vip.com>: HTTP status code is not handled or not allowed
2023-09-21 15:51:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.twistys.com/scenes?site=234&utm_source=momknowsbest.com&utm_medium=redirect&utm_campaign=p1migration> (failed 3 times): 429 Unknown Statu
s
2023-09-21 15:51:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.twistys.com/scenes?site=234&utm_source=momknowsbest.com&utm_medium=redirect&utm_campaign=p1migration>: HTTP status code is not handled or
not allowed
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
NEXT PAGE: 0
2023-09-21 15:51:34 [scrapy.core.scraper] ERROR: Spider error processing <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%2Fwww.br
azzers.com> (referer: https://www.brazzers.com)
Traceback (most recent call last):
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\defer.py", line 279, in iter_errback
yield next(it)
^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\python.py", line 350, in __next__
return next(self.data)
^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\python.py", line 350, in __next__
return next(self.data)
^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 28, in <genexpr>
return (r for r in result or () if self._filter(r, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\referer.py", line 352, in <genexpr>
return (self._set_referer(r, response) for r in result or ())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 27, in <genexpr>
return (r for r in result or () if self._filter(r, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\depth.py", line 31, in <genexpr>
return (r for r in result or () if self._filter(r, response, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "Y:\Stash\Webharvy\scrapy-main3\scrapy-main\tpdb\spiders\networkProjectOneService.py", line 293, in get_scenes
path = '/scene/' + str(item['id']) + '/' + slugify(item['title'])
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\slugify.py", line 24, in slugify
unicode(
^^^^^^^
NameError: name 'unicode' is not defined
2023-09-21 15:52:31 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 14 pages/min), scraped 0 items (at 0 items/min)
2023-09-21 15:53:31 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-09-21 15:54:31 [scrapy.extensions.logstats] INFO: Crawled 27 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2023-09-21 15:54:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.milfed.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.milfed.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.transangels.com> (failed 3 times): User timeout caused connection failure: Getting https://www.transangels.com took longer than 180.0 sec
onds..
2023-09-21 15:54:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://www.transharder.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:32 [scrapy.core.scraper] ERROR: Error downloading <GET https://www.transangels.com>
Traceback (most recent call last):
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\twisted\internet\defer.py", line 1693, in _inlineCallbacks
result = context.run(
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\twisted\python\failure.py", line 518, in throwExceptionIntoGenerator
return g.throw(self.type, self.value, self.tb)
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\downloader\middleware.py", line 54, in process_request
return (yield download_func(request=request, spider=spider))
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\twisted\internet\defer.py", line 892, in _runCallbacks
current.result = callback( # type: ignore[misc]
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\downloader\handlers\http11.py", line 397, in _cb_timeout
raise TimeoutError(f"Getting {url} took longer than {timeout} seconds.")
twisted.internet.error.TimeoutError: User timeout caused connection failure: Getting https://www.transangels.com took longer than 180.0 seconds..
2023-09-21 15:54:32 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://www.transharder.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:32 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.digitalplayground.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.squirted.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.mofos.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.realitykings.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.digitalplayground.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.babes.com%2Fscenes%3Fsite%3D214%26utm_source%3Dofficeobsession.com%26utm_medium%3Dredirect%26utm_campaign%3Dp1migration> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.squirted.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.mofos.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.realitykings.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.babes.com%2Fscenes%3Fsite%3D214%26utm_source%3Dofficeobsession.com%26utm_medium%3Dredirect%26utm_campaign%3Dp1migration>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.familysinners.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.realityjunkies.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.familysinners.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.deviante.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.realityjunkies.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.deviante.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:33 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.transangelsnetwork.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:33 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.transangelsnetwork.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:34 [scrapy.core.scraper] ERROR: Spider error processing <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%2Fwww.ba
bes.com%2Fscenes%3Fsite%3D216%26utm_source%3Delegantanal.com%26utm_medium%3Dredirect%26utm_campaign%3Dp1migration> (referer: https://www.babes.com/scenes?site=216&utm_source=elegantanal.com&utm_medium=redirect&utm_campaign=p1migration)
Traceback (most recent call last):
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\defer.py", line 279, in iter_errback
yield next(it)
^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\python.py", line 350, in __next__
return next(self.data)
^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\utils\python.py", line 350, in __next__
return next(self.data)
^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\offsite.py", line 28, in <genexpr>
return (r for r in result or () if self._filter(r, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\referer.py", line 352, in <genexpr>
return (self._set_referer(r, response) for r in result or ())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\urllength.py", line 27, in <genexpr>
return (r for r in result or () if self._filter(r, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\spidermiddlewares\depth.py", line 31, in <genexpr>
return (r for r in result or () if self._filter(r, response, spider))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\scrapy\core\spidermw.py", line 106, in process_sync
for r in iterable:
File "Y:\Stash\Webharvy\scrapy-main3\scrapy-main\tpdb\spiders\networkProjectOneService.py", line 293, in get_scenes
path = '/scene/' + str(item['id']) + '/' + slugify(item['title'])
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Kev\AppData\Local\Programs\Python\Python311\Lib\site-packages\slugify.py", line 24, in slugify
unicode(
^^^^^^^
NameError: name 'unicode' is not defined
2023-09-21 15:54:34 [scrapy.downloadermiddlewares.retry] ERROR: Gave up retrying <GET https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.babes.com> (failed 3 times): 429 Unknown Status
2023-09-21 15:54:34 [scrapy.spidermiddlewares.httperror] INFO: Ignoring response <429 https://site-api.project1service.com/v2/releases?dateReleased=%3C2023-09-22&limit=100&type=scene&orderBy=-dateReleased&offset=0&referrer=https%3A%2F%
2Fwww.babes.com>: HTTP status code is not handled or not allowed
2023-09-21 15:54:34 [scrapy.core.engine] INFO: Closing spider (finished)
2023-09-21 15:54:34 [scrapy.extensions.feedexport] INFO: Stored csv feed (0 items) in: pos3.csv
2023-09-21 15:54:34 [scrapy.statscollectors] INFO: Dumping Scrapy stats:
{'downloader/exception_count': 27,
'downloader/exception_type_count/twisted.internet.error.TimeoutError': 27,
'downloader/request_bytes': 47095,
'downloader/request_count': 105,
'downloader/request_method_count/GET': 105,
'downloader/response_bytes': 2440993,
'downloader/response_count': 78,
'downloader/response_status_count/200': 15,
'downloader/response_status_count/301': 4,
'downloader/response_status_count/429': 59,
'elapsed_time_seconds': 363.014599,
'feedexport/success_count/FileFeedStorage': 1,
'finish_reason': 'finished',
'finish_time': datetime.datetime(2023, 9, 21, 13, 54, 34, 939425, tzinfo=datetime.timezone.utc),
'httpcompression/response_bytes': 20840951,
'httpcompression/response_count': 15,
'httperror/response_ignored_count': 25,
'httperror/response_ignored_status_count/429': 25,
'log_count/ERROR': 29,
'log_count/INFO': 43,
'log_count/WARNING': 1,
'request_depth_max': 1,
'response_received_count': 40,
'retry/count': 60,
'retry/max_reached': 26,
'retry/reason_count/429 Unknown Status': 34,
'retry/reason_count/twisted.internet.error.TimeoutError': 26,
'scheduler/dequeued': 105,
'scheduler/dequeued/memory': 105,
'scheduler/enqueued': 105,
'scheduler/enqueued/memory': 105,
'spider_exceptions/NameError': 2,
'start_time': datetime.datetime(2023, 9, 21, 13, 48, 31, 924826, tzinfo=datetime.timezone.utc)}
2023-09-21 15:54:34 [scrapy.core.engine] INFO: Spider closed (finished)
PS Y:\Stash\Webharvy\scrapy-main3\scrapy-main>