Internal error after a few hours
blockcloud opened this issue · 9 comments
I use the api to store the bitcoin data to the relation database, but when it works for few hours(almost three), there will be an error. I don't know why this happens? Is there anyone get the save problem?
@blockcloud what is the exact error? Can you provide a test with reproduction steps?
@mariosk I receive the block data through the "BlockExplorer" like this
BlockExplorer blockExplorer = new BlockExplorer();
List<Block> block = blockExplorer.getBlocksAtHeight(height);
Last few times I get the "Internal error".
This time I get the error like below.
java.net.SocketException: Connection reset
at java.net.SocketInputStream.read(SocketInputStream.java:209)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
at sun.security.ssl.InputRecord.read(InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:930)
at sun.security.ssl.AppInputStream.read(AppInputStream.java:105)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:286)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.net.www.http.ChunkedInputStream.readAheadBlocking(ChunkedInputStream.java:552)
at sun.net.www.http.ChunkedInputStream.readAhead(ChunkedInputStream.java:609)
at sun.net.www.http.ChunkedInputStream.read(ChunkedInputStream.java:696)
at java.io.FilterInputStream.read(FilterInputStream.java:133)
at sun.net.www.protocol.http.HttpURLConnection$HttpInputStream.read(HttpURLConnection.java:3336)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.BufferedReader.fill(BufferedReader.java:161)
at java.io.BufferedReader.readLine(BufferedReader.java:324)
at java.io.BufferedReader.readLine(BufferedReader.java:389)
at info.blockchain.api.HttpClient.inputStreamToString(HttpClient.java:162)
at info.blockchain.api.HttpClient.openURL(HttpClient.java:108)
at info.blockchain.api.HttpClient.get(HttpClient.java:47)
at info.blockchain.api.blockexplorer.BlockExplorer.getBlocksAtHeight(BlockExplorer.java:134)
at com.cp.services.GetBlocks.getBlockByHeight(GetBlocks.java:89)
at com.cp.services.GetBlocks.getBlocksByHeight(GetBlocks.java:111)
at com.cp.tasks.GetBlockMultiThread.run(GetBlocksInfo.java:37)
at java.lang.Thread.run(Thread.java:745)
@mariosk There is also an error like below:
info.blockchain.api.APIException: Internal Server Error
at info.blockchain.api.HttpClient.openURL(HttpClient.java:106)
at info.blockchain.api.HttpClient.get(HttpClient.java:47)
at info.blockchain.api.blockexplorer.BlockExplorer.getBlocksAtHeight(BlockExplorer.java:134)
at com.cp.services.GetBlocks.getBlockByHeight(GetBlocks.java:88)
at com.cp.services.GetBlocks.getBlocksByHeight(GetBlocks.java:110)
at com.cp.tasks.GetBlocksInfo.main(GetBlocksInfo.java:140)
@blockcloud it seems to me that both of the problems you see are network (blockchain.info) issues that maybe shown during high load or something. @RayXY any other ideas?
@blockcloud you could keep track of the blockheight you have downloaded so far and if such an error occurs, sleep for 3 seconds and retry for the last blockheight. Does it make sense?
@mariosk I have tried your suggestion, finally it works. Thank you.
Are there better ways to parse the bitcoin datas to store in a relationship database?
@blockcloud I have no idea what else could be done to retrieve all the data. I suspect that this is done because the blockchain.info server has a rate limiting enabled and may drop the connections. Can you try to sleep 200ms between the requests?
@mariosk Thank you for your sugguestion. I will try it to see if it could be faster.