Blockchain-downloader need help
kaufmannVincent opened this issue · 6 comments
Hey! Im running a node, have the blockchain saved locally. How do i use the Blockchain-downloader?
Bitcoind has downloaded the entire chain and is running with: bitcoind -daemon.
bitcoin.conf file, located in /home/ubuntu/.bitcoin:
rpcuser=your-test
rpcpassword=your-12345
Add txindex=1
enable rpcport=8332
server=1
rpclogin.txt, located in /home/ubuntu/Blockchain-downloader:
test
12345
when running:
python blockchain_downloader.py
54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713 bitcoin.pdf
Im getting:
RPC connection not available
Traceback (most recent call last):
File "blockchain_downloader.py", line 43, in
class main():
File "blockchain_downloader.py", line 95, in main
dlfn.save_data(BLOCKCHAINADDRESS, LOCAL, INDIVIDUALFILE)
File "/home/ubuntu/Blockchain-downloader/wlffbd/dlfn.py", line 38, in save_data
_, _, data = satoshi.length_checksum_data_from_rawdata(satoshi.unhexutf8(hexdata))
File "/home/ubuntu/Blockchain-downloader/wlffbd/satoshi.py", line 25, in unhexutf8
return unhexlify(s.encode('utf8'))
TypeError: Non-hexadecimal digit found
Your username and password need to match to get the RPC Connection working. In your example you have rpcuser=your-test and rpcpassword=your-12345 but you state that in rpclogin.txt your username is test and password is 12345 it should be your-test and your-12345. The current non-RPC functions of the code do not currently work and that's why you get the TypeError: Non-hexadecimal digit #found.
Edit: I have fixed the functionality for when the RPC connection does not work. It should work in both situations now solving this issue in 493fcc0.
Hello, it looks better now i guess. But im not sure what im doing wrong, when trying to download the bitcoin whitepaper pdf i got this:
python blockchain_downloader.py 54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713 bitcoin.pdf
54e48e5f5c656b26c3bca14a8c95aa583d07ebe84dde3b7dd4a78f4e4186e713 PDF Header Found PDF Footer Found output Satoshi Checksum found
and it created those 3 files instead of a pdf: bitcoin.pdfdata.txt, bitcoin.pdfindata.txt, bitcoin.pdforigdata.txt
EDIT: when changung the filename.pdforigdata.txt to bitcoin.pdf it opens perfectly. Amazing.
Remaining question is, is it possible to download a file that is split in multiple transactions from a text file just as the Cablegate example?
Yes! That was part of the design of this program and the list of cablegate transactions are included in the format required by the .txt file. Put each transaction on a new line and instead of passing the single transaction point it to where the transaction list is like python blockchain_downloader.py cabelgate.txt and it will download the data. You will have to rename it just like you did with the pdf since there is currently no filename reassignment if it finds the header information.
Could you please check if you get the same error?
Looks like it generates a corrupted cablegate.txt:
It starts with:
" ^"^@^@sSEXWikileaks Cablegate Backup "
... and it ends with
" ^@^@^@b�^G�\�'�BS�����P븏^Xȼ�z/^_gF^L���1y�^D�^Y�^H "
when running:
python blockchain_downloader.py cablegate.txt
Im getting: Traceback (most recent call last):
File "blockchain_downloader.py", line 43, in
class main():
File "blockchain_downloader.py", line 77, in main
dlfn.get_tx_list(BLOCKCHAINADDRESS, LOCAL)
File "/home/ubuntu/Blockchain-downloader/wlffbd/dlfn.py", line 66, in get_tx_list
self.save_data(blockhash, LOCAL)
File "/home/ubuntu/Blockchain-downloader/wlffbd/dlfn.py", line 31, in save_data
hexdata = rpc.get_data_local(transaction, self.SERVER)
File "/home/ubuntu/Blockchain-downloader/wlffbd/blockchainrpc.py", line 51, in get_data_local
rawTx = SERVER.getrawtransaction(transaction)
File "/home/ubuntu/.local/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 650, in call
return self.__send(self.__name, args)
File "/home/ubuntu/.local/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 531, in _request
response = self._run_request(request)
File "/home/ubuntu/.local/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 564, in _run_request
verbose=self.__verbose
File "/usr/lib/python2.7/xmlrpclib.py", line 1283, in request
return self.single_request(host, handler, request_body, verbose)
File "/home/ubuntu/.local/lib/python2.7/site-packages/jsonrpclib/jsonrpc.py", line 387, in single_request
response.msg)
jsonrpclib.jsonrpc.ProtocolError: ('your-test:your-12345@localhost:8332/', 500, 'Internal Server Error', <httplib.HTTPMessage instance at 0x7ff95fb3ac20>)
I am getting it to work without issue. I am using this list of transactions in the repo under the data and transactions folders. I have it working in both python 2 and python 3. Your example suggests you need to remove the true first transaction in the list. The first part of cablegate was a transaction containing information regarding cablegate such as all the transactions required. If you use the list I linked you can download it and then change the filedata.txt to cablegate.7z and then extract the file normally. Make sure that after each run you delete the old file*data.txt files otherwise it will write over the old one which is necessary for the list of transactions since it just adds each transactions data onto the end of the file.
I'm having issue as well. setup a venv with python -m venv venv, activated it, installed requirements.txt,
set the rpc options, and I get the following when attempting to get the bitcoin.pdf
RPC connection not available
Traceback (most recent call last):
File "Blockchain-downloader/blockchain_downloader.py", line 96, in <module>
dlfn.save_data(BLOCKCHAINADDRESS, LOCAL, INDIVIDUALFILE)
File "Blockchain-downloader/wlffbd/dlfn.py", line 32, in save_data
page = online.get_blockchain_transaction_json(transaction)
File "Blockchain-downloader/wlffbd/blockchaininfo.py", line 56, in get_blockchain_transaction_json
return json.loads(get_blockchain_request("tx/{}".format(transaction)).read().decode("utf-8"))
File "Blockchain-downloader/wlffbd/blockchaininfo.py", line 42, in get_blockchain_request
return urllib.request.urlopen(req)
File "/usr/lib/python3.9/urllib/request.py", line 214, in urlopen
return opener.open(url, data, timeout)
File "/usr/lib/python3.9/urllib/request.py", line 523, in open
response = meth(req, response)
File "/usr/lib/python3.9/urllib/request.py", line 632, in http_response
response = self.parent.error(
File "/usr/lib/python3.9/urllib/request.py", line 561, in error
return self._call_chain(*args)
File "/usr/lib/python3.9/urllib/request.py", line 494, in _call_chain
result = func(*args)
File "/usr/lib/python3.9/urllib/request.py", line 641, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 404: Not Found