Exzuul no longer works --> doesn't run Jenkin's jobs on gerrit events to merge

Question

Exzuul no longer works --> doesn't run Jenkin's jobs on gerrit events to merge

alexbaltman opened this issue 9 years ago · 1 comments

-----Exzuul no longer works --> doesn't run Jenkin's jobs on gerrit events-----
either the independent pipline or the dependent pipeline

All Directions followed in the README.md. No errors currently in the logs when tail -f /var/log/zuul/*

I did see this in the debug.log:
2016-06-07 19:33:16,096 ERROR gerrit.GerritWatcher: Exception on ssh event stream:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/zuul/connection/gerrit.py", line 173, in _run
key_filename=self.keyfile)
File "/usr/lib/python2.7/site-packages/paramiko/client.py", line 311, in connect
raise NoValidConnectionsError(errors)
NoValidConnectionsError: [Errno None] Unable to connect to port 29418 on or 172.17.0.2
2016-06-07 19:38:11,011 DEBUG zuul.Gearman: Looking for lost builds
2016-06-07 19:43:11,012 DEBUG zuul.Gearman: Looking for lost builds
2016-06-07 19:48:11,012 DEBUG zuul.Gearman: Looking for lost builds
2016-06-07 19:49:40,121 DEBUG gerrit.GerritWatcher: Received data from Gerrit event stream:
{u'change': {u'branch': u'master',
u'commitMessage': u'first commit\n\nChange-Id: I51af923ae0966beb198104e0b4863dba4e094add\n',
u'id': u'I51af923ae0966beb198104e0b4863dba4e094add',
u'number': u'1',
u'owner': {u'email': u'admin@ci.localdomain',
u'name': u'admin',
....

So it came up w/ an error, but that error basically disappears right away hence the incoming patch set.

I do see this NO_JOB, but that's not the case. I see the job in jenkins and successfully applied the jenkins job builder:
[root@ci zuul]# grep NO_J *
gearman-server.log:2016-06-07 19:43:02,336 DEBUG gear.ServerConnection: Queuing packet to <gear.ServerConnection 0x2408cd0 name: ci.localdomain_exec-1 host: 172.17.0.2 port: 48826>: <gear.Packet 0x2408f50 type: NO_JOB>
merger-debug.log:2016-06-07 19:33:05,906 INFO gear.Worker.Zuul Merger: Received packet from <gear.Connection 0x22352d0 host: ci.localdomain port: 4730>: <gear.Packet 0x2235690 type: NO_JOB>
merger.log:2016-06-07 19:33:05,906 INFO gear.Worker.Zuul Merger: Received packet from <gear.Connection 0x22352d0 host: ci.localdomain port: 4730>: <gear.Packet 0x2235690 type: NO_JOB>

Standard layout.yaml:
[root@ci jenkins_jobs]# cat jobs/jjb.yaml

defaults:
name: global
project-type: freestyle
concurrent: true
job:
name: testproject-unit-tests
defaults: global
builders:
- shell: |
  zuul-cloner http://ci.localdomain:8080 $ZUUL_PROJECT
  cd $ZUUL_PROJECT
  ./run_tests.sh
project:
name: testproject
node: master
jobs:
- testproject-unit-tests

Gerrit repo:
testproject

Jenkins job:
is there. testproject-unit-tests

[root@ci zuul]# cat /etc/zuul/zuul.conf
[gearman]
server=ci.localdomain

[gearman_server]
start=true
log_config=/etc/zuul/gearman-logging.conf

[gerrit]
server=ci.localdomain
baseurl=http://ci.localdomain:8080
user=zuul
sshkey=/var/lib/keys/id_rsa

[zuul]
layout_config=/etc/zuul/layout.yaml
log_config=/etc/zuul/logging.conf
pidfile=/var/run/zuul/zuul.pid
state_dir=/var/lib/zuul
status_url=http://ci.localdomain

[merger]
git_dir=/var/lib/zuul/git
zuul_url=http://ci.localdomain/p
log_config=/etc/zuul/merger-logging.conf

[smtp]
server=localhost
port=25

[root@ci zuul]# cat /etc/jenkins_jobs/jenkins_jobs.ini
[jenkins]
user=jenkins
password=password
url=http://ci.localdomain:8081/jenkins

[root@ci zuul]# cat /etc/jenkins_jobs/jenkins_jobs.ini
[jenkins]
user=jenkins
password=password
url=http://ci.localdomain:8081/jenkins
[root@ci zuul]# cat /etc/jenkins_jobs/jobs/jjb.yaml

defaults:
name: global
project-type: freestyle
concurrent: true
job:
name: testproject-unit-tests
defaults: global
builders:
- shell: |
  zuul-cloner http://ci.localdomain:8080 $ZUUL_PROJECT
  cd $ZUUL_PROJECT
  ./run_tests.sh
project:
name: testproject
node: master
jobs:
- testproject-unit-tests

[root@ci zuul]# cat /etc/hosts
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
172.17.0.2 ci.localdomain ci

[root@ci zuul]# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 19:32 ? 00:00:00 /bin/bash /start.sh
root 105 0 0 19:32 ? 00:00:00 bash
root 221 1 0 19:33 ? 00:00:01 /usr/bin/python /usr/bin/supervisord -n
root 226 221 0 19:33 ? 00:00:21 java -Xmx256m -jar /opt/gerrit/site_path/gerrit.war daemon -d /opt/gerrit/site_p
root 227 221 0 19:33 ? 00:00:08 /usr/bin/python /usr/bin/zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/layout.
root 228 221 0 19:33 ? 00:00:00 /usr/bin/python /usr/bin/zuul-merger -c /etc/zuul/zuul.conf -d
root 229 221 0 19:33 ? 00:00:00 /usr/sbin/rsyslogd -n -c3
root 230 221 0 19:33 ? 00:00:31 java -Xmx256m -jar /usr/lib/jenkins/jenkins.war --webroot=/var/cache/jenkins/war
root 262 227 0 19:33 ? 00:00:00 /usr/bin/python /usr/bin/zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/layout.
root 394 0 0 19:40 ? 00:00:00 bash
root 410 0 0 19:42 ? 00:00:00 bash
root 442 410 0 19:50 ? 00:00:00 /usr/bin/python /usr/bin/yum install -y telnet
root 447 0 0 19:53 ? 00:00:00 bash
root 580 447 0 20:45 ? 00:00:00 ps -ef

Answer 1 · 2016-06-07T21:06:11.000Z

Still broken, but wanted to update.

I noticed my vhost settings I had played with and httpd was failing to come up so I reverted that and did httpd -k start again.

here's the ps -ef:
[root@ci zuul]# ps -ef
UID PID PPID C STIME TTY TIME CMD
root 1 0 0 19:32 ? 00:00:00 /bin/bash /start.sh
root 105 0 0 19:32 ? 00:00:00 bash
root 221 1 0 19:33 ? 00:00:01 /usr/bin/python /usr/bin/supervisord -n
root 226 221 0 19:33 ? 00:00:24 java -Xmx256m -jar /opt/gerrit/site_path/gerrit.war daemon -d /opt/gerrit/site_p
root 227 221 0 19:33 ? 00:00:11 /usr/bin/python /usr/bin/zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/layout.
root 228 221 0 19:33 ? 00:00:00 /usr/bin/python /usr/bin/zuul-merger -c /etc/zuul/zuul.conf -d
root 229 221 0 19:33 ? 00:00:00 /usr/sbin/rsyslogd -n -c3
root 230 221 0 19:33 ? 00:00:41 java -Xmx256m -jar /usr/lib/jenkins/jenkins.war --webroot=/var/cache/jenkins/war
root 262 227 0 19:33 ? 00:00:00 /usr/bin/python /usr/bin/zuul-server -c /etc/zuul/zuul.conf -l /etc/zuul/layout.
root 394 0 0 19:40 ? 00:00:00 bash
root 410 0 0 19:42 ? 00:00:00 bash
root 447 0 0 19:53 ? 00:00:00 bash
root 636 0 0 20:53 ? 00:00:00 bash
root 732 1 0 21:04 ? 00:00:00 httpd -k start
apache 733 732 0 21:04 ? 00:00:00 httpd -k start
apache 734 732 0 21:04 ? 00:00:00 httpd -k start
apache 735 732 0 21:04 ? 00:00:00 httpd -k start
apache 736 732 0 21:04 ? 00:00:00 httpd -k start
apache 737 732 0 21:04 ? 00:00:00 httpd -k start
root 739 636 0 21:04 ? 00:00:00 ps -ef
[root@ci zuul]#

here's the vhost config:

DocumentRoot /var/www/zuul

LogLevel warn

RewriteEngine on
RewriteRule ^/status.json$ http://127.0.0.1:8001/status.json [P]

SetEnv GIT_PROJECT_ROOT /var/lib/zuul/git/
SetEnv GIT_HTTP_EXPORT_ALL

<Directory /usr/libexec/git-core/>
require all granted

AliasMatch ^/p/(./objects/[0-9a-f]{2}/[0-9a-f]{38})$ /var/lib/zuul/git/$1
AliasMatch ^/p/(./objects/pack/pack-[0-9a-f]{40}.(pack|idx))$ /var/lib/zuul/git/$1
ScriptAlias /p/ /usr/libexec/git-core/git-http-backend/