stucco/dev-setup

Installation Fails: "TASK [jgoodall.maven3 : get_url]"

packet-rat opened this issue · 17 comments

Latest versions of everything, freshly installed. Same advice: try feature/bulkload branch???

TASK [jgoodall.maven3 : get_url] ***********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: timeout: timed out
fatal: [stucco]: FAILED! => {"changed": false, "failed": true, "msg": "failed to create temporary content file: timed out"}
	to retry, use: --limit @/Users/pmaroney/git/stucco/provisioning/site.retry

PLAY RECAP *********************************************************************
stucco                     : ok=13   changed=9    unreachable=0    failed=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

No, master branch should be up to date now.

I just made a fresh clone of the repo and was able to vagrant up without any problems.

I have run into similar ansible errors before, generally it's either an outdated and unavailable version of something, or else a network or availability problem upstream. In this case everything seems to be fine for me - Maven 3.2.5 is a few versions old, but still available, along with a few older versions.

I think the files below are the only ones it downloads. Can you download these from the host machine? If you reprovision (or destroy) the VM, do you still get the same error?

http://www.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.zip
http://www.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.zip.md5

I've destroyed/re-provisioned. I've also completely wiped out the git repo and re-installed from scratch...

"Oh, somewhere in this favoured land the sun is shining bright,
The band is playing somewhere, and somewhere hearts are light;
And somewhere men are laughing, and somewhere children shout,
But there is no joy in Mudville—mighty Casey has struck out."


PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [stucco]

TASK [jgoodall.common : install common packages (build-essential, git, mercurial, supervisor, zip, unzip, rsync, curl, wget)] ***
changed: [stucco] => (item=[u'build-essential', u'git', u'mercurial', u'supervisor', u'zip', u'unzip', u'rsync', u'curl', u'wget'])

TASK [stucco.common : create group] ********************************************
changed: [stucco]

TASK [stucco.common : create user] *********************************************
changed: [stucco]

TASK [stucco.common : create Stucco directory] *********************************
changed: [stucco]

TASK [stucco.common : create Stucco bin directory] *****************************
changed: [stucco]

TASK [stucco.common : download endogenous-data-uc1 repo] ***********************
changed: [stucco]

TASK [smola.java : include_vars] ***********************************************
ok: [stucco]

TASK [smola.java : Install python-apt] *****************************************
ok: [stucco]

TASK [smola.java : Install WebUpd8 apt key] ************************************
skipping: [stucco]

TASK [smola.java : Install WebUpd8 Team Java PPA (for Oracle Java)] ************
skipping: [stucco]

TASK [smola.java : Remove WebUpd8 Team Java PPA (for Oracle Java)] *************
skipping: [stucco]

TASK [smola.java : Install WebUpd8 Team Java PPA (for Oracle Java)] ************
changed: [stucco]

TASK [smola.java : Remove WebUpd8 Team Java PPA (for Oracle Java)] *************
skipping: [stucco]

TASK [smola.java : Accept Oracle License] **************************************
changed: [stucco] => (item=oracle-java6-installer)
changed: [stucco] => (item=oracle-java7-installer)
changed: [stucco] => (item=oracle-java8-installer)

TASK [smola.java : Install Java packages] **************************************
changed: [stucco] => (item=[u'oracle-java8-installer', u'oracle-java8-set-default'])

TASK [smola.java : Remove unwanted Java packages] ******************************
ok: [stucco] => (item=[u'openjdk-6-jdk', u'openjdk-6-jdk:i386', u'openjdk-7-jdk', u'openjdk-7-jdk:i386'])

TASK [jgoodall.maven3 : get_url] ***********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: timeout: timed out
fatal: [stucco]: FAILED! => {"changed": false, "failed": true, "msg": "failed to create temporary content file: timed out"}
	to retry, use: --limit @/Users/pmaroney/git/stucco/provisioning/site.retry

PLAY RECAP *********************************************************************
stucco                     : ok=13   changed=9    unreachable=0    failed=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.
Benz:stucco pmaroney$ 

I can download both files from the guest:

vagrant@stucco:~$ wget http://www.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.zip
--2017-10-24 14:23:33--  http://www.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.zip
Resolving www.apache.org (www.apache.org)... 140.211.11.105
Connecting to www.apache.org (www.apache.org)|140.211.11.105|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 8071373 (7.7M) [application/zip]
Saving to: ‘apache-maven-3.2.5-bin.zip’

99% [=======================================================================================================================================================================================================> ] 8,070,810   --.-K/s  eta 0s      
99% [=======================================================================================================================================================================================================> ] 8,070,810   --.-K/s   in 2m 48s  

2017-10-24 14:26:24 (46.9 KB/s) - Read error at byte 8070810/8071373 (Connection reset by peer). Retrying.

--2017-10-24 14:26:25--  (try: 2)  http://www.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.zip
Connecting to www.apache.org (www.apache.org)|140.211.11.105|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 8071373 (7.7M), 563 remaining [application/zip]
Saving to: ‘apache-maven-3.2.5-bin.zip’

100%[++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++>] 8,071,373   --.-K/s   in 0s      

2017-10-24 14:26:25 (142 MB/s) - ‘apache-maven-3.2.5-bin.zip’ saved [8071373/8071373]

pretty much a novice with ansible et al, any suggestions for troubleshooting?

Sorry just looked closer at the wget for maven...hmmm. I have a 500Mbs Internet connection.

I wanted to check back on this -- any luck? Was it an issue with the download?

Ok, that looks like a minor connection problem, which wget handles and ansible apparently doesn't.

I just pushed a hotfix branch which adds retries to that download, let me know if that resolves things.

Closer...

..Quoth the Maven, “Nevermore.”

[Sorry it's Halloween and I just couldn't resist]

PLAY [all] *********************************************************************

TASK [Gathering Facts] *********************************************************
ok: [stucco]

TASK [jgoodall.common : install common packages (build-essential, git, mercurial, supervisor, zip, unzip, rsync, curl, wget)] ***
changed: [stucco] => (item=[u'build-essential', u'git', u'mercurial', u'supervisor', u'zip', u'unzip', u'rsync', u'curl', u'wget'])

TASK [stucco.common : create group] ********************************************
changed: [stucco]

TASK [stucco.common : create user] *********************************************
changed: [stucco]

TASK [stucco.common : create Stucco directory] *********************************
changed: [stucco]

TASK [stucco.common : create Stucco bin directory] *****************************
changed: [stucco]

TASK [stucco.common : download endogenous-data-uc1 repo] ***********************
changed: [stucco]

TASK [smola.java : include_vars] ***********************************************
ok: [stucco]

TASK [smola.java : Install python-apt] *****************************************
ok: [stucco]

TASK [smola.java : Install WebUpd8 apt key] ************************************
skipping: [stucco]

TASK [smola.java : Install WebUpd8 Team Java PPA (for Oracle Java)] ************
skipping: [stucco]

TASK [smola.java : Remove WebUpd8 Team Java PPA (for Oracle Java)] *************
skipping: [stucco]

TASK [smola.java : Install WebUpd8 Team Java PPA (for Oracle Java)] ************
changed: [stucco]

TASK [smola.java : Remove WebUpd8 Team Java PPA (for Oracle Java)] *************
skipping: [stucco]

TASK [smola.java : Accept Oracle License] **************************************
changed: [stucco] => (item=oracle-java6-installer)
changed: [stucco] => (item=oracle-java7-installer)
changed: [stucco] => (item=oracle-java8-installer)

TASK [smola.java : Install Java packages] **************************************
changed: [stucco] => (item=[u'oracle-java8-installer', u'oracle-java8-set-default'])

TASK [smola.java : Remove unwanted Java packages] ******************************
ok: [stucco] => (item=[u'openjdk-6-jdk', u'openjdk-6-jdk:i386', u'openjdk-7-jdk', u'openjdk-7-jdk:i386'])

TASK [jgoodall.maven3 : get_url] ***********************************************
FAILED - RETRYING: jgoodall.maven3 : get_url (5 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (4 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (3 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (2 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (1 retries left).
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: timeout: timed out
fatal: [stucco]: FAILED! => {"attempts": 5, "changed": false, "failed": true, "msg": "failed to create temporary content file: timed out"}
	to retry, use: --limit @/Users/pmaroney/git/dev-setup/provisioning/site.retry

PLAY RECAP *********************************************************************
stucco                     : ok=13   changed=9    unreachable=0    failed=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Something strange is definitely happening with their server. With wget, I'm seeing it reach 99.x% at a normal speed, and then drop to 1B/s for the last ~30 bytes, or possibly more. It doesn't cause a problem for get_url in my environment, but I do think I have a fix. I added a fallback to use wget if ansible's get_url fails, since apparently wget is more tolerant of this kind of thing.

Try the latest on the hotfix branch, and let me know if that resolves it.

TASK [jgoodall.maven3 : get_url] ***********************************************
FAILED - RETRYING: jgoodall.maven3 : get_url (5 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (4 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (3 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (2 retries left).
FAILED - RETRYING: jgoodall.maven3 : get_url (1 retries left).
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: timeout: timed out
fatal: [stucco]: FAILED! => {"attempts": 5, "changed": false, "failed": true, "msg": "failed to create temporary content file: timed out"}
	to retry, use: --limit @/Users/pmaroney/git/dev-setup/provisioning/site.retry

PLAY RECAP *********************************************************************
stucco                     : ok=13   changed=0    unreachable=0    failed=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

Latest Hotfix as of today (I think?)

git checkout hotfix
Already on 'hotfix'
Your branch is up-to-date with 'origin/hotfix'.

ok, that was my mistake: I added a retry, but the initial error was still fatal. Should be fixed on hotfix now.

So...we're making progress (I think) . At least failing further down the process:

<<<SNIP>>>
TASK [ansible-role.nodejs : copying node to /usr/local] ************************
changed: [stucco -> 10.10.10.100] => (item=bin)
changed: [stucco -> 10.10.10.100] => (item=lib)
changed: [stucco -> 10.10.10.100] => (item=share)

TASK [jgoodall.maven3 : Download Maven] ****************************************
FAILED - RETRYING: Download Maven (3 retries left).
FAILED - RETRYING: Download Maven (2 retries left).
FAILED - RETRYING: Download Maven (1 retries left).
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: timeout: timed out
fatal: [stucco]: FAILED! => {"attempts": 3, "changed": false, "failed": true, "msg": "failed to create temporary content file: timed out"}
...ignoring

TASK [jgoodall.maven3 : stat] **************************************************
ok: [stucco]

TASK [jgoodall.maven3 : Download Maven (retry)] ********************************
changed: [stucco]

TASK [jgoodall.maven3 : Download Maven checksum] *******************************
changed: [stucco]

TASK [jgoodall.maven3 : stat] **************************************************
ok: [stucco]

TASK [jgoodall.maven3 : Download Maven checksum (retry)] ***********************
skipping: [stucco]

TASK [jgoodall.maven3 : command] ***********************************************
changed: [stucco]

TASK [jgoodall.maven3 : file] **************************************************
ok: [stucco]

TASK [jgoodall.maven3 : unarchive] *********************************************
changed: [stucco]

TASK [jgoodall.maven3 : file] **************************************************
changed: [stucco]

TASK [jgoodall.maven3 : file] **************************************************
changed: [stucco]

TASK [jgoodall.maven3 : template] **********************************************
changed: [stucco]

TASK [jgoodall.maven3 : file] **************************************************
changed: [stucco]

TASK [zenoamaro.postgresql : Adding APT repository key] ************************
changed: [stucco]

TASK [zenoamaro.postgresql : Add PostgreSQL official APT repository] ***********
changed: [stucco]

TASK [zenoamaro.postgresql : Install PostgreSQL] *******************************
changed: [stucco]

TASK [zenoamaro.postgresql : Install dependencies for the Ansible module] ******
changed: [stucco] => (item=[u'python-psycopg2'])

TASK [zenoamaro.postgresql : Install PostgreSQL] *******************************
skipping: [stucco]

TASK [zenoamaro.postgresql : Install dependencies for the Ansible module] ******
skipping: [stucco] => (item=[]) 

TASK [zenoamaro.postgresql : Install development headers] **********************
skipping: [stucco]

TASK [zenoamaro.postgresql : Install PostgreSQL contribs] **********************
skipping: [stucco]

TASK [zenoamaro.postgresql : Add postgis extensions] ***************************
skipping: [stucco] => (item=[]) 

TASK [zenoamaro.postgresql : Create the necessary directories] *****************
ok: [stucco] => (item=/etc/postgresql/9.6/main)
ok: [stucco] => (item=/var/lib/postgresql/9.6/main)
ok: [stucco] => (item=/var/run/postgresql)
changed: [stucco] => (item=/var/log/postgresql)

TASK [zenoamaro.postgresql : Configure PostgreSQL] *****************************
changed: [stucco] => (item=postgresql.conf)
changed: [stucco] => (item=pg_ctl.conf)
changed: [stucco] => (item=environment)

TASK [zenoamaro.postgresql : Configure PostgreSQL (authentication)] ************
changed: [stucco]

TASK [zenoamaro.postgresql : Configure PostgreSQL (ident)] *********************
changed: [stucco]

TASK [stucco.graph : copy postgresql db monitor] *******************************
changed: [stucco]

TASK [stucco.graph : copy supervisord conf for db monitoring script] ***********
changed: [stucco]

TASK [stucco.graph : reload supervisord] ***************************************
changed: [stucco]

TASK [stucco.graph : wait for postgresql] **************************************
ok: [stucco]

TASK [stucco.graph : install setfacl support] **********************************
changed: [stucco]

TASK [stucco.graph : create stucco db user] ************************************
changed: [stucco]

TASK [stucco.graph : create the database as stucco user] ***********************
changed: [stucco] => (item={u'name': u'testdb'})

TASK [stucco.graph : initialize stucco user db] ********************************
changed: [stucco]

TASK [stucco.graph : Add elasticsearch key] ************************************
changed: [stucco]

TASK [stucco.graph : Install apt-transport-https for elasticsearch] ************
ok: [stucco]

TASK [stucco.graph : Add elasticsearch to apt sources] *************************
changed: [stucco]

TASK [stucco.graph : Install elasticsearch] ************************************
changed: [stucco]

TASK [stucco.graph : Set up elasticsearch init] ********************************
changed: [stucco]

TASK [stucco.graph : start elasticsearch] **************************************
changed: [stucco]

TASK [stucco.graph : wait for elasticsearch] ***********************************
ok: [stucco]

TASK [stucco.graph : install elasticdump] **************************************
changed: [stucco]

TASK [stucco.graph : add mapping] **********************************************
changed: [stucco]

TASK [stucco.graph : add data] *************************************************
changed: [stucco]

TASK [jgoodall.golang : get_url] ***********************************************
changed: [stucco]

TASK [jgoodall.golang : file] **************************************************
ok: [stucco]

TASK [jgoodall.golang : unarchive] *********************************************
changed: [stucco]

TASK [jgoodall.golang : file] **************************************************
changed: [stucco]

TASK [jgoodall.golang : template] **********************************************
changed: [stucco]

TASK [jgoodall.golang : file] **************************************************
changed: [stucco]

TASK [stucco.doc : create stucco GOPATH directory for document-service] ********
changed: [stucco]

TASK [stucco.doc : download document-service repo] *****************************
changed: [stucco]

TASK [stucco.doc : test document-service] **************************************
changed: [stucco]

TASK [stucco.doc : build document-service] *************************************
changed: [stucco]

TASK [stucco.doc : install document-service] ***********************************
changed: [stucco]

TASK [stucco.doc : copy document-service supervisord config] *******************
changed: [stucco]

TASK [stucco.doc : reload supervisord] *****************************************
changed: [stucco]

TASK [stucco.rt : download rt repo] ********************************************
changed: [stucco]

TASK [stucco.rt : copy rt config template] *************************************
changed: [stucco]

TASK [stucco.rt : copy rt build scripts] ***************************************
changed: [stucco]

TASK [stucco.rt : make sure postgresqldb is running] ***************************
ok: [stucco]

TASK [stucco.rt : build rt] ****************************************************
fatal: [stucco]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Shared connection to 10.10.10.100 closed.\r\n", "unreachable": true}

RUNNING HANDLER [mikeiannacone.rabbitmq : restart rabbitmq] ********************
fatal: [stucco]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.10.10.100 port 22: Operation timed out\r\n", "unreachable": true}

RUNNING HANDLER [zenoamaro.postgresql : restart postgresql] ********************
fatal: [stucco]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 10.10.10.100 port 22: Host is down\r\n", "unreachable": true}
	to retry, use: --limit @/Users/pmaroney/git/dev-setup/dev-setup/provisioning/site.retry

PLAY RECAP *********************************************************************
stucco                     : ok=100  changed=68   unreachable=3    failed=0   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

I think this looks like an intermittent Vagrant issue - can you try it again (from scratch)?

I thought I saw a note flash by in the GitHub-0-sphere a week or so back that installation issues should be resolved(?). There's no joy in Mudville here:

TASK [smola.java : Accept Oracle License] **************************************
changed: [stucco] => (item=oracle-java6-installer)
changed: [stucco] => (item=oracle-java7-installer)
changed: [stucco] => (item=oracle-java8-installer)

TASK [smola.java : Install Java packages] **************************************
changed: [stucco] => (item=[u'oracle-java8-installer', u'oracle-java8-set-default'])

TASK [smola.java : Remove unwanted Java packages] ******************************
ok: [stucco] => (item=[u'openjdk-6-jdk', u'openjdk-6-jdk:i386', u'openjdk-7-jdk', u'openjdk-7-jdk:i386'])

TASK [jgoodall.maven3 : get_url] ***********************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: timeout: timed out
fatal: [stucco]: FAILED! => {"changed": false, "failed": true, "msg": "failed to create temporary content file: timed out"}
	to retry, use: --limit @/Users/pmaroney/git/dev-setup/provisioning/site.retry

PLAY RECAP *********************************************************************
stucco                     : ok=13   changed=9    unreachable=0    failed=1   

Ansible failed to complete successfully. Any error output should be
visible above. Please fix these errors and try again.

That message looks like a temporary timeout issue, please retry.

Also - be sure you can download the URL, in this case it is trying to get maven:

curl -O http://www.apache.org/dist/maven/maven-3/3.2.5/binaries/apache-maven-3.2.5-bin.zip

FWIW - ~50th time was the charm (wish that was an exaggeration ;-)

Successfully built dev-setup from hotfix branch today!