Is Celery Running?
Rizbe opened this issue · 18 comments
I am using docker compose to run the lint review bot but I am getting this error below
INFO 2017-06-04 17:07:02,973 github3 models:167 JSON was not returned
DEBUG 2017-06-04 17:07:02,973 lintreview.web web:60 lintrc file contents ''
INFO 2017-06-04 17:07:02,973 lintreview.web web:67 Scheduling pull request for name/repo-link
ERROR 2017-06-04 17:07:02,974 lintreview.web web:70 Could not publish job to celery. Make sure its running.
"172.22.0.1 - - [04/Jun/2017:17:07:02] "POST /review/start HTTP/1.1" 500 - "-" "GitHub-Hookshot/a24207a"
INFO 2017-06-04 17:07:02,976 gunicorn.access glogging:309 "172.22.0.1 - - [04/Jun/2017:17:07:02] "POST /review/start HTTP/1.1" 500 - "-" "GitHub-Hookshot/a24206a"
Was wondering if the current docker setup is working or if I am missing something.
How did you start the application with docker-compose?
After defining the vars below in the compose file :
LINTREVIEW_SERVER_NAME
GITHUB_USERNAME
GITHUB_OAUTH_TOKEN
Then I did:
docker-compose up -d broker worker web
Ok. That should work. I'll try to reproduce on my side.
Decided to test out the dependency from above and seem like the same issue still occurs.
Ok I didn't think that change would solve it. But it does help reduce startup errors.
Guessing your able to reproduce?
I've not had a chance to attempt to reproduce. I don't normally run lint-review via docker-compose and instead have it hosted in EC2 on a single node.
Decided to run it outside of docker to see the full workflow and then debug the docker setup, this is what I did.
pip install .
export all needed env var
gunicorn -c settings.py lintreview.web:app
celery -A lintreview.tasks worker
Not seeing any debug from gunicorn
From celery:
[2017-06-06 11:51:51,134: INFO/MainProcess] Received task: lintreview.tasks.cleanup_pull_request[1ceb6563-58ac-4309-84a1-fa2573a2c04f]
[2017-06-06 11:51:51,135: INFO/Worker-4] Cleaning up pull request test/test_link/1
[2017-06-06 11:51:51,136: WARNING/Worker-4] Cannot cleanup '/tmp/testi/test_link/1' path does not exist.
[2017-06-06 11:51:51,136: INFO/MainProcess] Task lintreview.tasks.cleanup_pull_request[1ceb6563-58ac-4309-84a1-fa2573a2c04f] succeeded in 0.00117430200044s: None
That looks like it is working. Gunicorn is likely logging to a file.
I tried tonight using docker-compose and from what I can tell, lintreview works inside the docker-compose file. I did have to:
- Add the GITHUB_USER and GITHUB_OAUTH_TOKEN environment variables on line 26.
- Make sure I posted to the correct URL.
I captured a webhook using mockbin, and then replayed it locally using:
body=$(pbpaste)
curl -XPOST 192.168.99.100:5000/review/start -H 'Content-type: application/json' -d "$body"
While following docker logs with docker-compose logs -f I got the following output:
web_1 | DEBUG 2017-06-19 04:16:51,946 gunicorn.error glogging:257 POST /reviews/start
web_1 | 2017-06-19 04:16:51 [9] [DEBUG] POST /reviews/start
web_1 | INFO 2017-06-19 04:16:51,950 gunicorn.access glogging:309 "192.168.99.1 - - [19/Jun/2017:04:16:51] "POST /reviews/start HTTP/1.1" 404 233 "-" "curl/7.51.0"
web_1 | "192.168.99.1 - - [19/Jun/2017:04:16:51] "POST /reviews/start HTTP/1.1" 404 233 "-" "curl/7.51.0"
web_1 | DEBUG 2017-06-19 04:17:16,126 gunicorn.error glogging:257 POST /review/start
web_1 | 2017-06-19 04:17:16 [9] [DEBUG] POST /review/start
web_1 | INFO 2017-06-19 04:17:16,128 lintreview.web web:48 Received GitHub pull request notification for git://github.com/markstory/lint-test.git 33, (closed) from: git://github.com/markstory/lint-test.git
web_1 | INFO 2017-06-19 04:17:16,131 lintreview.web web:77 Scheduling cleanup for markstory/lint-test
web_1 | DEBUG 2017-06-19 04:17:16,201 amqp connection:755 Start from server, version: 0.9, properties: {u'information': u'Licensed under the MPL. See http://www.rabbitmq.com/', u'product': u'RabbitMQ', u'copyright': u'Copyright (C) 2007-2017 Pivotal Software, Inc.', u'capabilities': {u'exchange_exchange_bindings': True, u'connection.blocked': True, u'authentication_failure_close': True, u'direct_reply_to': True, u'basic.nack': True, u'per_consumer_qos': True, u'consumer_priorities': True, u'consumer_cancel_notify': True, u'publisher_confirms': True}, u'cluster_name': u'rabbit@1e87e2db6a0c', u'platform': u'Erlang/OTP', u'version': u'3.6.10'}, mechanisms: [u'PLAIN', u'AMQPLAIN'], locales: [u'en_US']
broker_1 |
broker_1 | =INFO REPORT==== 19-Jun-2017::04:17:16 ===
broker_1 | accepting AMQP connection <0.596.0> (172.20.0.4:50878 -> 172.20.0.2:5672)
broker_1 |
broker_1 | =INFO REPORT==== 19-Jun-2017::04:17:16 ===
broker_1 | connection <0.596.0> (172.20.0.4:50878 -> 172.20.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
web_1 | DEBUG 2017-06-19 04:17:16,205 amqp connection:641 Open OK!
web_1 | DEBUG 2017-06-19 04:17:16,206 amqp channel:82 using channel_id: 1
web_1 | DEBUG 2017-06-19 04:17:16,210 amqp channel:440 Channel open
worker_1 | [2017-06-19 04:17:16,218: INFO/MainProcess] Received task: lintreview.tasks.cleanup_pull_request[46409420-a44e-405f-87de-59bce487edfb]
worker_1 | [2017-06-19 04:17:16,220: INFO/Worker-1] Cleaning up pull request markstory/lint-test/33
worker_1 | [2017-06-19 04:17:16,221: WARNING/Worker-1] Cannot cleanup '/tmp/workspace/markstory/lint-test/33' path does not exist.
worker_1 | [2017-06-19 04:17:16,225: INFO/MainProcess] Task lintreview.tasks.cleanup_pull_request[46409420-a44e-405f-87de-59bce487edfb] succeeded in 0.00558245599996s: None
web_1 | INFO 2017-06-19 04:17:16,227 gunicorn.access glogging:309 "192.168.99.1 - - [19/Jun/2017:04:17:16] "POST /review/start HTTP/1.1" 204 - "-" "curl/7.51.0"
web_1 | "192.168.99.1 - - [19/Jun/2017:04:17:16] "POST /review/start HTTP/1.1" 204 - "-" "curl/7.51.0"
This indicates that celery and gunicorn are running and that they can talk to each other through rabbitmq.
Sorry for not responding back to this thread, followed exactly what you did but still getting the error below when I run in docker.
ERROR 2017-06-04 17:07:02,974 lintreview.web web:70 Could not publish job to celery. Make sure its running.
I also decided to run it outside of Docker and keep seeing this error in celery
[2017-06-21 10:35:51,593: ERROR/MainProcess] Task lintreview.tasks.process_pull_request[ff41e44d-95ec-4f7c-a981-0057a9397dcf] raised unexpected: File contains no section headers.
file: <???>, line: 1
'{}'
Traceback (most recent call last):
File "/home/user/.virtualenvs/python2/lib/python2.7/site-packages/celery/app/trace.py", line 240, in trace_task
R = retval = fun(*args, **kwargs)
File "/home/user/.virtualenvs/python2/lib/python2.7/site-packages/celery/app/trace.py", line 438, in __protected_call__
return self.run(*args, **kwargs)
File "/home/user/rtmp/lint-review/lintreview/tasks.py", line 25, in process_pull_request
review_config = build_review_config(lintrc, config)
File "/home/user/rtmp/lint-review/lintreview/config.py", line 55, in build_review_config
config.load_ini(ini_config)
File "/home/user/rtmp/lint-review/lintreview/config.py", line 145, in load_ini
parser.readfp(StringIO(ini_config))
File "/usr/lib64/python2.7/ConfigParser.py", line 324, in readfp
self._read(fp, filename)
File "/usr/lib64/python2.7/ConfigParser.py", line 512, in _read
raise MissingSectionHeaderError(fpname, lineno, line)
MissingSectionHeaderError: File contains no section headers.
file: <???>, line: 1
'{}'
Sounds like you might be missing a configuration file in your repository.
Will reclone it fresh and see if everything is good
Cloned it fresh, updated the settings.py with the proper github user, token, and server name, same error as above.
Any idea which config file I might be missing?
Does your repository have a .lintrc file?
It does
➜ cat .lintrc
[tools]
linters = flake8
Odd. That stack trace implies that the repositories config file contained no sections. If you enable debug logging, you should see the lintrc file before the process job is enqueued.