Does Onionscan still retrieve a snapshot?
Closed this issue · 2 comments
I followed along with the tutorial at http://www.automatingosint.com/blog/2016/07/dark-web-osint-with-python-and-onionscan-part-one/ after looking through the tutorial's sample data and seeing snapshots of onion sites.
However now I've implemented the code myself the JSON responses I'm getting from onionscan don't seem to have a snapshot key at all. I realize the example I've given below appears to be a blank response, however I'm still wondering about the lack of a snapshot key as the tutorial data has blank ones. Has the structure of onionscan's JSON response changed since July 2016 and dropped snapshots?
Example of my JSON response:
{"hiddenService":"marketdcyc56ewpq.onion","dateScanned":"2017-04-26T23:40:35.825510905Z","online":false,"performedScans":["web","tls","ssh","irc","ricochet","ftp","smtp","mongodb","vnc","xmpp","bitcoin","bitcoin_test","litecoin","dogecoin"],"webDetected":false,"tlsDetected":false,"sshDetected":false,"ricochetDetected":false,"ircDetected":false,"ftpDetected":false,"smtpDetected":false,"bitcoinDetected":false,"mongodbDetected":false,"vncDetected":false,"xmppDetected":false,"skynetDetected":false,"crawls":{},"pgpKeys":null,"certificates":null,"bitcoinServices":{"bitcoin":{"detected":false,"userAgent":"","prototocolVersion":0,"onionPeers":null},"bitcoin_test":{"detected":false,"userAgent":"","prototocolVersion":0,"onionPeers":null},"dogecoin":{"detected":false,"userAgent":"","prototocolVersion":0,"onionPeers":null},"litecoin":{"detected":false,"userAgent":"","prototocolVersion":0,"onionPeers":null}},"sshKey":"","sshBanner":"","ftpFingerprint":"","ftpBanner":"","smtpFingerprint":"","smtpBanner":"","lastAction":"dogecoin","timedOut":false,"error":null,"identifierReport":{"privateKeyDetected":false,"foundApacheModStatus":false,"serverVersion":"","relatedOnionServices":null,"relatedOnionDomains":null,"ipAddresses":null,"emailAddresses":null,"analyticsIDs":null,"bitcoinAddresses":null,"linkedOnions":null,"openDirectories":null,"exifImages":null},"simpleReport":{"hiddenService":"marketdcyc56ewpq.onion","risks":null}}
Example of tutorial JSON response with blank snapshot key:
{"hiddenService":"2arj33oil5dbrpyu.onion","webDetected":true,"sshDetected":false,"ricochetDetected":false,"ircDetected":false,"ftpDetected":false,"smtpDetected":false,"bitcoinDetected":false,"mongodbDetected":false,"vncDetected":false,"xmppDetected":false,"serverPoweredBy":"","serverVersion":"nginx/1.8.1","foundApacheModStatus":false,"relatedOnionServices":null,"relatedOnionDomains":null,"linkedSites":null,"internalPages":null,"ipAddresses":null,"openDirectories":null,"exifImages":null,"interestingFiles":null,"pageReferencedDirectories":null,"pgpKeys":null,"hashes":null,"snapshot":"","pageTitle":"","responseHeaders":{"CONNECTION":"keep-alive","CONTENT-TYPE":"text/html","DATE":"Mon, 25 Jul 2016 21:42:31 GMT","SERVER":"nginx/1.8.1"},"bitcoinAddresses":null,"sshKey":"","ftpFingerprint":"","ftpBanner":"","smtpFingerprint":"","smtpBanner":""}
Example of tutorial JSON response with snapshot data:
{"hiddenService":"2agobs57djngatwc.onion","webDetected":true,"sshDetected":true,"ricochetDetected":false,"ircDetected":false,"ftpDetected":false,"smtpDetected":false,"bitcoinDetected":false,"mongodbDetected":false,"vncDetected":false,"xmppDetected":false,"serverPoweredBy":"","serverVersion":"Apache","foundApacheModStatus":false,"relatedOnionServices":null,"relatedOnionDomains":null,"linkedSites":["pyrogoggles.kd.io","ajax.googleapis.com"],"internalPages":[""],"ipAddresses":null,"openDirectories":null,"exifImages":null,"interestingFiles":null,"pageReferencedDirectories":["./css","."],"pgpKeys":null,"hashes":["b34384df1ea88a2138a82b34a3812fe8a9401b3e"],"snapshot":"\u003c!DOCTYPE html\u003e\r\n\u003c!-- saved from url=(0025)http://pyrogoggles.kd.io/ --\u003e\r\n\u003chtml lang=\"sv\"\u003e\r\n\u003chead\u003e\u003cmeta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\"\u003e\r\n\u003cmeta charset=\"utf-8\"\u003e\r\n\u003ctitle\u003eZombeSweden\u003c/title\u003e\r\n\u003cmeta name=\"viewport\" content=\"width=device-width,initial-scale=1\"\u003e\r\n\u003clink rel=\"stylesheet\" href=\"./css/style.css\"\u003e\r\n\u003c!--[if IE]\u003e\r\n \u003cscript src=\"//html5shiv.googlecode.com/svn/trunk/html5.js\"\u003e\u003c/script\u003e\r\n\u003c![endif]--\u003e\r\n\u003clink href=\"./css\" rel=\"stylesheet\" type=\"text/css\"\u003e\r\n\u003clink rel='stylesheet' type='text/css' href='./css/menu_styles.css' /\u003e\r\n\u003cscript src='http://ajax.googleapis.com/ajax/libs/jquery/1.10.2/jquery.min.js'\u003e\u003c/script\u003e\r\n\u003cscript type='text/javascript' src='./js/menu_jquery.js'\u003e\u003c/script\u003e\r\n\u003c!---- My own Font TTF file ---\u003e\r\n\u003cstyle\u003e \r\n@font-face\r\n{\r\nfont-family: bloodlust;\r\nsrc: url(./font/bloodlust.ttf);\r\n}\r\n\r\nblod\r\n{\r\nfont-family:bloodlust;\r\n}\r\n\u003c/style\u003e\r\n\u003cstyle\u003e \r\n@font-face\r\n{\r\nfont-family: ardestine;\r\nsrc: url(./font/ardestine.ttf);\r\n}\r\n\r\nard\r\n{\r\nfont-family:ardestine;\r\n}\r\n\u003c/style\u003e\r\n\u003c/head\u003e\r\n\u003cbody class=\"index\"\u003e\u003cscript\u003e\r\n (function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){\r\n (i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),\r\n m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)\r\n })(window,document,'script','//www.google-analytics.com/analytics.js','ga');\r\n\r\n ga('create', 'UA-45530347-3', 'auto');\r\n ga('send', 'pageview');\r\n\r\n\u003c/script\u003e\u003cdiv id=\"infoBox\" style=\"z-index: 360000; font-family: PT Sans, Trebushet MS, sans-serif; background: -webkit-gradient(linear, 0% 0%, 0% 100%, from(rgba(122, 189, 25,0.9)), to(rgba(102, 167, 16,0.9))); margin-top:-3em; position:fixed; top:3em; left: 0; float: left; width: 100%; color: #fff; padding: 1em; font-size: 11pt; height: 20px; display: none; text-align: left;\"\u003eSelect the blocks that you do not want to see on this page\u003cdiv id=\"saveBlocksButton\" style=\"color: #fff; text-shadow: none; cursor: pointer;display: inline;margin-left: 1.6em;padding: 0.5em 0.8em 0.5em 0.8em;background: rgb(102, 167, 16);border-radius: 5px;\"\u003eSave\u003c/div\u003e\u003cdiv id=\"cancelAddBlocksButton\" style=\"color: #fff; text-shadow: none; cursor: pointer;display: inline;margin-left: 1.6em;padding: 0.5em 0.8em 0.5em 0.8em;background: rgb(102, 167, 16);border-radius: 5px;\"\u003eCancel\u003c/div\u003e\u003c/div\u003e\r\n\r\n\u003cdiv id=\"container\"\u003e\r\n \u003cdiv id=\"main\" role=\"main\" class=\"hellobox\"\u003e\r\n \u003ch1\u003e\u003card\u003e\u003cfont color=\"#FF1111\"\u003eVälkommen\u003c/font\u003e\u003c/ard\u003e\u003c/h1\u003e\r\n \u003ch2\u003e\u003cblod\u003e\u003cfont color=\"#FF1111\"\u003e\u003cspan style=\"font-size:3em;\"\u003eZombeSweden\u003c/span\u003e\u003c/font\u003e\u003c/blod\u003e\u003c/h2\u003e\r\n \u003c/div\u003e\r\n \u003cnav\u003e\r\n \u003cdiv id='cssmenu' align=\"center\"\u003e\r\n\u003cul\u003e\r\n \u003cli class='active'\u003e\u003ca href='#'\u003e\u003cspan\u003eHome\u003c/span\u003e\u003c/a\u003e\r\n \u003cul\u003e\r\n \u003cli\u003e\u003ca href='./index.php'\u003e\u003cspan\u003eHome\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n \u003cli class='last'\u003e\u003ca href='home.html'\u003e\u003cspan\u003e - Old Home\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n\t\t \u003c!-- \u003cli class='last'\u003e\u003ca href='index2.php'\u003e\u003cspan\u003e - Php Home\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e --\u003e\r\n \u003c/ul\u003e\r\n \u003c/li\u003e\r\n \u003cli class='has-sub'\u003e\u003ca href='#'\u003e\u003cspan\u003eMenu\u003c/span\u003e\u003c/a\u003e\r\n \u003cul\u003e\r\n \u003cli\u003e\u003ca href='games_index.php'\u003e\u003cspan\u003eGames\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n \u003cli\u003e\u003ca href='download_index.php'\u003e\u003cspan\u003e- Download\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n \u003cli class='last'\u003e\u003ca href='tipe_index.php'\u003e\u003cspan\u003e- Tips\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n \u003c/ul\u003e\r\n \u003c/li\u003e\r\n \u003cli class='has-sub'\u003e\u003ca href='#'\u003e\u003cspan\u003eOther stuffs\u003c/span\u003e\u003c/a\u003e\r\n \u003cul\u003e\r\n \u003cli\u003e\u003ca href='video.html'\u003e\u003cspan\u003eVideos\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n\t\t \u003cli\u003e\u003ca href='gamesvideo.html'\u003e\u003cspan\u003e- Games Videos\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n\t\t \u003cli\u003e\u003ca href='photoHD.html'\u003e\u003cspan\u003e- HD Photos\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n\t\t \u003cli\u003e\u003ca href='#'\u003e\u003cspan\u003e- About\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n \u003cli class='last'\u003e\u003ca href='#'\u003e\u003cspan id='forum1'\u003e- My Forum\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n \u003c/ul\u003e\r\n\t \u003cli class='has-sub'\u003e\u003c/li\u003e\r\n \u003c/li \u003e\r\n \u003cli class='has-sub'\u003e\u003ca href='#'\u003e\u003cspan id='forum1'\u003eMy Forum\u003c/span\u003e\u003c/a\u003e\r\n \u003c/li\u003e\r\n \u003cli class='last'\u003e\u003ca href='#'\u003e\u003cspan id='forum1'\u003eContact\u003c/span\u003e\u003c/a\u003e\u003c/li\u003e\r\n\u003c/ul\u003e\u003c/div\u003e\r\n \u003c/nav\u003e\r\n \u003cfooter\u003e\r\n \u003ch4\u003eVälkommen!\u003c/h4\u003e\r\n \u003cp\u003eDet här är min lilla hemsida lekstuga. Den här sidan kommer uppdateras då och då med spel info,videos m.m \r\n\t\u003cbr /\u003eKolla gärna runt på sidan. :D\u003c/p\u003e\r\n \u003cpre\u003e\u003cfont color=\"red\"\u003eFör övrigt är inte Fourm sidan inte alltid uppe, den kopplas ner ibland för att den ligger på en gratis server.\u003c/font\u003e\r\n\t\u003cbr \\\u003eDetta är min sida som jag kallar för ZombeSweden. Jag har kvar min gamla designer här också men jag har tänk försöka snygga upp den så att den bli\r\n\t\u003cbr /\u003emer lik designen på denna sita. Den stor bilden ovan ska ändra när jag får tid. Har den bara där temporalt.\r\n\t\u003c/pre\u003e\r\n \u003c/footer\u003e\r\n\u003c/div\u003e\r\n\r\n\r\n\r\n\r\n\r\n\r\n\u003c/body\u003e\u003c/html\u003e","pageTitle":"ZombeSweden","responseHeaders":{"ACCEPT-RANGES":"bytes","CONTENT-TYPE":"text/html","DATE":"Tue, 26 Jul 2016 19:08:44 GMT","ETAG":"\"1209-5090a4b2b38c0-gzip\"","LAST-MODIFIED":"Sun, 30 Nov 2014 02:31:39 GMT","SERVER":"Apache","VARY":"Accept-Encoding","X-CONTENT-TYPE-OPTIONS":"nosniff","X-FRAME-OPTIONS":"sameorigin","X-XSS-PROTECTION":"1; mode=block"},"bitcoinAddresses":null,"sshKey":"81:cb:d2:d4:f7:e4:8c:6b:1a:92:07:42:cd:e2:42:23","ftpFingerprint":"","ftpBanner":"","smtpFingerprint":"","smtpBanner":""}
`
Has the structure of onionscan's JSON response changed since July 2016 and dropped snapshots?
Yes. OnionScan no longer outputs the snapshot to JSON. The snapshots are stored in the localdatabase onionscan creates (by default in ./onionscandb/) - To extract them you can either use the golang package onionscan/crawldb or some other library - snapshots are keyed by "crawls" which is output in the JSON.
This was done to allow OnionScan to scale to snapshotting multiple resources from a given site.
Thank you for your very quick response! Great, I'll take a look at the onionscandb and see how I get on.