Network error while fetching - 0.11.0
Closed this issue · 18 comments
Hi,
in dependency to issue #12 the problem was not full solved with 0.11.0. I have installed 0.11.0 from gems and have furthermore problems.
The client says:
C:\Data\mytest\scm_server>bundle install
Fetching gem metadata from http://173.17.41.6:2000/.Retrying dependency api due
to error (2/4): Bundler::HTTPError Network error while fetching http://172.16.40
.5:2000/api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread
_safe
Retrying dependency api due to error (3/4): Bundler::HTTPError Network error whi
le fetching http://173.17.41.6:2000/api/v1/dependencies?gems=rake,asciidoctor,as
ciidoctor-pdf,coderay,thread_safe
Retrying dependency api due to error (4/4): Bundler::HTTPError Network error whi
le fetching http://173.17.41.6:2000/api/v1/dependencies?gems=rake,asciidoctor,as
ciidoctor-pdf,coderay,thread_safe
Network error while fetching
http://173.17.41.6:2000/quick/Marshal.4.8/thread_safe-0.1.1-java.gemspec.rz
C:\Data\mytest\scm_server>
config.rb
# -*- coding: utf-8 -*-
# This is the main configuration file for your RubyGems mirror. Here you can
# change settings such as the location to store Gem files in and what source
# and Gems you'd like to mirror at start.
Gemirro.configuration.configure do
# Define sinatra environment
environment :production
# The directory to store indexing information as well as the Gem files in.
destination File.expand_path('../public', __FILE__)
# If you're in development mode your probably want to switch to a debug
# logging level.
logger.level = Logger::INFO
# If you want to run your server on a specific host and port, you must
# change the following parameters (server_host and server_port).
#
server.host '173.17.41.6'
server.port '2000'
server.access_log File.expand_path('../logs/access.log', __FILE__)
server.error_log File.expand_path('../logs/error.log', __FILE__)
# You must define a source wich where gems will be downloaded.
# All gem in the block will be downloaded with the update command.
# Other gems will be downloaded with the server.
define_source 'rubygems', 'http://rubygems.org' do
gem 'rack', '>= 1.0.0'
gem 'asciidoctor', '>= 1.5.0'
gem 'asciidoctor-pdf', '>= 1.5.0.alpha.8'
gem 'coderay', '>= 1.1.0'
gem 'thread_safe', '>= 0.1.1'
end
end
access.log
...
I, [2015-08-03T13:21:29.038695 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:29.038976 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:29.039078 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:29.039180 #25391] INFO -- : Try to download thread_safe with version 0.3.3-java
I, [2015-08-03T13:21:29.039467 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:29.039700 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:29.039744 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:29.040279 #25391] INFO -- : Try to download thread_safe with version 0.3.4-java
I, [2015-08-03T13:21:29.040649 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:29.040935 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:29.040980 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:29.041729 #25391] INFO -- : Try to download thread_safe with version 0.3.5-java
I, [2015-08-03T13:21:29.042011 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:29.042245 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:29.042281 #25391] INFO -- : Exiting RubyGems with exit_code 0
172.16.40.111 - - [03/Aug/2015:13:21:29 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 122.1783
I, [2015-08-03T13:21:32.523403 #25391] INFO -- : Try to download thread_safe with version 0.1.1-java
I, [2015-08-03T13:21:32.523981 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.524295 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.524339 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.524886 #25391] INFO -- : Try to download thread_safe with version 0.1.2-java
I, [2015-08-03T13:21:32.525153 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.525378 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.525415 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.525932 #25391] INFO -- : Try to download thread_safe with version 0.1.3-java
I, [2015-08-03T13:21:32.526195 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.526417 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.526453 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.527020 #25391] INFO -- : Try to download thread_safe with version 0.2.0-java
I, [2015-08-03T13:21:32.527287 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.527514 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.527550 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.527634 #25391] INFO -- : Try to download thread_safe with version 0.3.1-java
I, [2015-08-03T13:21:32.527892 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.528112 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.528148 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.528894 #25391] INFO -- : Try to download thread_safe with version 0.3.2-java
I, [2015-08-03T13:21:32.529155 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.529380 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.529415 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.529500 #25391] INFO -- : Try to download thread_safe with version 0.3.3-java
I, [2015-08-03T13:21:32.529755 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.529971 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.530006 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.530501 #25391] INFO -- : Try to download thread_safe with version 0.3.4-java
I, [2015-08-03T13:21:32.530763 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.531028 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.531064 #25391] INFO -- : Exiting RubyGems with exit_code 0
I, [2015-08-03T13:21:32.531612 #25391] INFO -- : Try to download thread_safe with version 0.3.5-java
I, [2015-08-03T13:21:32.531872 #25391] INFO -- : Generating indexes
I, [2015-08-03T13:21:32.532101 #25391] INFO -- : No new gems
I, [2015-08-03T13:21:32.532136 #25391] INFO -- : Exiting RubyGems with exit_code 0
173.17.41.111 - - [03/Aug/2015:13:21:32 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 105.5402
error.log
173.17.43.111 - - [03/Aug/2015:13:18:36 +0200] "GET /api/v1/dependencies HTTP/1.1" 200 - 0.0179
173.17.43.111 - - [03/Aug/2015:13:18:43 +0200] "GET / HTTP/1.1" 200 4916 1.3906
173.17.43.111 - - [03/Aug/2015:13:18:43 +0200] "GET /favicon.ico HTTP/1.1" 404 1162 0.3011
173.17.43.111 - - [03/Aug/2015:13:18:43 +0200] "GET /favicon.ico HTTP/1.1" 404 1162 0.0007
173.17.43.111 - - [03/Aug/2015:13:19:42 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 65.6794
173.17.43.111 - - [03/Aug/2015:13:20:35 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 108.8744
173.17.43.111 - - [03/Aug/2015:13:21:07 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 131.0335
173.17.43.111 - - [03/Aug/2015:13:21:20 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 133.4092
173.17.43.111 - - [03/Aug/2015:13:21:24 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 126.9657
173.17.43.111 - - [03/Aug/2015:13:21:26 +0200] "GET /quick/Marshal.4.8/thread_safe-0.1.1-java.gemspec.rz HTTP/1.1" 404 1162 44.7086
173.17.43.111 - - [03/Aug/2015:13:21:26 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 108.9923
173.17.43.111 - - [03/Aug/2015:13:21:28 +0200] "GET /quick/Marshal.4.8/thread_safe-0.1.1-java.gemspec.rz HTTP/1.1" 404 1162 37.0528
173.17.43.111 - - [03/Aug/2015:13:21:29 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 122.1785
173.17.43.111 - - [03/Aug/2015:13:21:32 +0200] "GET /api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread_safe HTTP/1.1" 200 8783 105.5403
Best regards,
Hi and sorry for the delay.
As it said in the post #12 #12 (comment) I don't really know how to improve download speed when requesting from bundle (Thread? Mutex?).
Seems there are other people using this method, I will try to improve this as soon as possible :)
Hi, no problem.
The problem is not the speed, I have try to add specific versiosn to config.rb like you said in your comment. Then I have run bundle more then one trail and I have wait after every trail until to gemirro has finished the 100% accessing of the CPU. In the first trials I get different gems that can not fetch. This shows gemirro was not fast enough to download and serve the gems, but after a trail the gems was available. That shows gemirro has this successfull downloaded. But in the last trails I get every the same gem which was not servable by gemirro. This looks like gemirro don't load the gem. The problem gem is thread_safe.
Best regards,
woot, seems to be hard to optimize this. Even if I use sinatra in threaded mode I'm getting the error when requesting http://localhost:2000/api/v1/dependencies?gems=rake,rack-test,rspec,simplecov,rubocop,fakefs,slop,httpclient,confstruct,builder,sinatra,thin
Updating all the gems with gemirro update, should fix this now.
sorry for the big delay.
I have made an update:
im@vm-projects:/var/lib/gemirro$ sudo gemirro update
I, [2015-08-24T10:46:43.502305 #27163] INFO -- : Updating rubygems (http://rubygems.org)
im@vm-projects:/var/lib/gemirro$
But I get the same result:
C:\Data\mytest\scm_server>bundle install
Fetching gem metadata from http://173.17.41.6:2000/.Retrying dependency api due
to error (2/4): Bundler::HTTPError Network error while fetching http://172.16.40
.5:2000/api/v1/dependencies?gems=rake,asciidoctor,asciidoctor-pdf,coderay,thread
_safe
Retrying dependency api due to error (3/4): Bundler::HTTPError Network error whi
le fetching http://173.17.41.6:2000/api/v1/dependencies?gems=rake,asciidoctor,as
ciidoctor-pdf,coderay,thread_safe
Retrying dependency api due to error (4/4): Bundler::HTTPError Network error whi
le fetching http://173.17.41.6:2000/api/v1/dependencies?gems=rake,asciidoctor,as
ciidoctor-pdf,coderay,thread_safe
Network error while fetching
http://173.17.41.6:2000/quick/Marshal.4.8/thread_safe-0.1.1-java.gemspec.rz
C:\Data\mytest\scm_server>
Hi, no problem, I'm currently in vacation :)
I will have a look as soon as possible.
Hi,
I add some cache for gem dependencies, it should be faster now, but I must improve it more.
Regards
I think the problem is not speed, maybe for the first call. The main problem is no matter how often if I try the result is the same. In my tests I have waitet until to gemirro fallback to normal CPU usage. This means the processing was finished and all requested gems should now saved on local disk. I think the next call should run without error, then gemirro must not load new gems. But I get always the same errror for thread_safe.
Look it there are CPU and RAM problems :/ As I said, I will investigate more and more on this issue.
Why think you is an CPU and RAM problem? In my first test with clean gemirro I start then gemirro uses a long time the CPU full and I get an error for my first needed gem. After that I repeat the test on the client, but then I get the error for thread_safe gem. All following tests gives the error for thread_safe. This shows now following. In the first run gemirro is slow, but thats ok for my use case. Then the seconed runs shows all needed gems (except thread_safe) was downloaded and now available. For me not the perfomance is the problem, my problem is that gemirro never serve thread_safe in all trials.
Sorry, I meant the CPU and RAM resources as a management problem or sinatra configuration :/
can I help with more informations or what is the next step?
The next step is to try with sinatra configuration and try to refactoring all File, FileUtils, Marshal and other classes that open files. If you know how to improve one or some things, I will be very glade to add them or review PRs :)
sorry I'm not a ruby developer :(. I can only help with testing...
I'm really sorry, I'm currently very busy and don't have the time to have a look for now. I will be back as soon as possible.
have you made changes in 0.12 which refers to this issue?
No, I don't find anything at this moment to optimize the gemspecs and gems :/
Why is bundler trying to fetch a -java- version of a gem? Try hiredis, for ex.
"GET /mirrors/fused/rubygems/public/specs.4.8.gz HTTP/1.1" 200 2706876 "-" "Ruby, RubyGems/2.2.5 x86_64-linux Ruby/2.1.7 (2015-08-18 patchlevel 400)"
"GET /mirrors/fused/rubygems/public/prerelease_specs.4.8.gz HTTP/1.1" 200 170877 "-" "Ruby, RubyGems/2.2.5 x86_64-linux Ruby/2.1.7 (2015-08-18 patchlevel 400)"
"GET /mirrors/fused/rubygems/public/quick/Marshal.4.8/hiredis-0.5.2-java.gemspec.rz HTTP/1.1" 404 345 "-" "bundler/1.11.2 rubygems/2.2.5 ruby/2.1.7 (x86_64-pc-linux-gnu) command/install options/ fed4aa8dae38b6c6"
I'm not running jruby.