Build fail at 99% with no reason
orensbruli opened this issue · 8 comments
Can someone give me some clue about what is happening here: http://158.49.112.158/job/237
This is happening many times. Build fails at 99% with no error message in the logs.
Thank you in advance.
Could you use docker events on hosts to see what happen to build container?
maybe you need to set build timeout to higher value, not sure.
Could you use
docker eventson hosts to see what happen to build container?
I only get events for the last day and it hasn't happened so recently so I will try to get docker events as soon as I see this happening again.
maybe you need to set build timeout to higher value, not sure.
We previously had some really long execution jobs (http://robocomp-ci.unex.es/job/199), also ended at 99% with no error, so I guess it's not the problem, but can you let me know where can I look for the build timeout value?
Thank you for your quick answers.
@orensbruli you can set both idleTimeout and jobTimeout in config.json, make sure you restart abstruse after updating this values. btw, can you share console output when build stalles at 99%?
@orensbruli you can set both
idleTimeoutandjobTimeoutinconfig.json, make sure you restart abstruse after updating this values. btw, can you share console output when build stalles at 99%?
I had
"idleTimeout": 600,
"jobTimeout": 3200,
I have increased both and restarted.
@jkuri I thought you could see the console output at the link I shared. Maybe you can see in this one?
http://158.49.112.158/api/jobs/237/log or do I need to paste it here? It's a long output.
Could you use
docker eventson hosts to see what happen to build container?
@ad-m It has just happened but I only can see lines like these ones in the docker events output:
2020-07-30T08:45:00.158992601Z container exec_create: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 e9c00ac43569f4f35d972d7bf7c433cfdf8199b5964c8227b443f1652088e92f (execID=de25208fb25a6c4bcab09783eb5d072935faaca3397bd0aa995b0e67e6fad264, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_mahavira, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:00.159027696Z container exec_start: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 5f5c502ca090eb96d686c64e34213ec16426022cb713b6088d2e28a01c859601 (execID=1099fa31ddc390d2dfa799211b696d64943fac9445e1eb69b15d19ffb454718d, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=friendly_dirac, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:00.159029659Z container exec_start: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 e9c00ac43569f4f35d972d7bf7c433cfdf8199b5964c8227b443f1652088e92f (execID=de25208fb25a6c4bcab09783eb5d072935faaca3397bd0aa995b0e67e6fad264, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_mahavira, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:00.158953891Z container exec_create: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 fbf2df18a891b03f51786da096f9520969bcdb24b92e48e5bd07afda4fd7023a (execID=1f3d102a6dacc8e34dbc3957246c837fddf6c0c5479daa2f366ae76b06a3fba2, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_brattain, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:00.159166309Z container exec_start: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 fbf2df18a891b03f51786da096f9520969bcdb24b92e48e5bd07afda4fd7023a (execID=1f3d102a6dacc8e34dbc3957246c837fddf6c0c5479daa2f366ae76b06a3fba2, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_brattain, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:00.303135339Z container exec_die e9c00ac43569f4f35d972d7bf7c433cfdf8199b5964c8227b443f1652088e92f (execID=de25208fb25a6c4bcab09783eb5d072935faaca3397bd0aa995b0e67e6fad264, exitCode=0, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_mahavira, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:00.304573052Z container exec_die fbf2df18a891b03f51786da096f9520969bcdb24b92e48e5bd07afda4fd7023a (execID=1f3d102a6dacc8e34dbc3957246c837fddf6c0c5479daa2f366ae76b06a3fba2, exitCode=0, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_brattain, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:00.319770810Z container exec_die 5f5c502ca090eb96d686c64e34213ec16426022cb713b6088d2e28a01c859601 (execID=1099fa31ddc390d2dfa799211b696d64943fac9445e1eb69b15d19ffb454718d, exitCode=0, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=friendly_dirac, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.314086619Z container exec_create: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 e9c00ac43569f4f35d972d7bf7c433cfdf8199b5964c8227b443f1652088e92f (execID=1dc9164c4951b2b4ede9108777bfe908e006980d401d920a591a8c505eaef150, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_mahavira, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.314145487Z container exec_start: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 e9c00ac43569f4f35d972d7bf7c433cfdf8199b5964c8227b443f1652088e92f (execID=1dc9164c4951b2b4ede9108777bfe908e006980d401d920a591a8c505eaef150, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_mahavira, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.314921871Z container exec_create: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 fbf2df18a891b03f51786da096f9520969bcdb24b92e48e5bd07afda4fd7023a (execID=2a796a664fead7beeb49c1582c0b611bec0f72291e4d55a77ccd08e66f1e82fa, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_brattain, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.314934733Z container exec_start: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 fbf2df18a891b03f51786da096f9520969bcdb24b92e48e5bd07afda4fd7023a (execID=2a796a664fead7beeb49c1582c0b611bec0f72291e4d55a77ccd08e66f1e82fa, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_brattain, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.327700362Z container exec_create: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 5f5c502ca090eb96d686c64e34213ec16426022cb713b6088d2e28a01c859601 (execID=968c0bbea27ec8e225d0e71726419d85b4078b5b74a455eb4b6fbff7a4728f33, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=friendly_dirac, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.327743376Z container exec_start: /bin/sh -c wget -q -O- http://localhost:6500/status || exit 1 5f5c502ca090eb96d686c64e34213ec16426022cb713b6088d2e28a01c859601 (execID=968c0bbea27ec8e225d0e71726419d85b4078b5b74a455eb4b6fbff7a4728f33, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=friendly_dirac, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.433728594Z container exec_die e9c00ac43569f4f35d972d7bf7c433cfdf8199b5964c8227b443f1652088e92f (execID=1dc9164c4951b2b4ede9108777bfe908e006980d401d920a591a8c505eaef150, exitCode=0, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_mahavira, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.448657874Z container exec_die fbf2df18a891b03f51786da096f9520969bcdb24b92e48e5bd07afda4fd7023a (execID=2a796a664fead7beeb49c1582c0b611bec0f72291e4d55a77ccd08e66f1e82fa, exitCode=0, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=hopeful_brattain, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6) 2020-07-30T08:45:10.455970254Z container exec_die 5f5c502ca090eb96d686c64e34213ec16426022cb713b6088d2e28a01c859601 (execID=968c0bbea27ec8e225d0e71726419d85b4078b5b74a455eb4b6fbff7a4728f33, exitCode=0, image=bleenco/abstruse, maintainer=Jan Kuri <jan@bleenco.com>, name=friendly_dirac, org.label-schema.build-date=2018-10-27T22:24:49Z, org.label-schema.description=Continuous integration platform, simple, scalable and fast, org.label-schema.name=abstruse, org.label-schema.schema-version=1.0, org.label-schema.url=https://abstruse.bleenco.io/, org.label-schema.vcs-ref=1491898cc4ecb3c2c2f45b45396ed4e4269d7074, org.label-schema.vcs-url=https://github.com/bleenco/abstruse, org.label-schema.vendor=Bleenco, org.label-schema.version=v1.6.6)
Is there any kind of filtering that you think I can do to get relevant output?
@orensbruli I meant logs from docker container if you are running abstruse inside docker or plain log output, not job log output.
closed due to inactivity.