KeyError: 'scheduler'
karpitsky opened this issue · 3 comments
karpitsky commented
Hello,
I've got the following error message:
# mrq-worker --scheduler
File "/usr/local/bin/mrq-worker", line 11, in <module>
load_entry_point('mrq', 'console_scripts', 'mrq-worker')()
File "/usr/local/src/mrq/mrq/bin/mrq_worker.py", line 53, in main
w.work()
File "/usr/local/src/mrq/mrq/worker.py", line 456, in work
self.work_loop(max_jobs=self.max_jobs, max_time=self.max_time)
File "/usr/local/src/mrq/mrq/worker.py", line 510, in work_loop
if self.config["scheduler"] and not self.greenlets["scheduler"]:
KeyError: 'scheduler'
Sentry is attempting to send 1 pending error messages
Waiting up to 10 seconds
Press Ctrl-C to quit
Traceback (most recent call last):
File "/usr/local/lib/python2.7/site-packages/gevent/greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "/usr/local/src/mrq/mrq/processes.py", line 128, in watch
self.watch_processes()
File "/usr/local/src/mrq/mrq/processes.py", line 135, in watch_processes
self.watch_process(process)
File "/usr/local/src/mrq/mrq/processes.py", line 146, in watch_process
status = process["psutil"].status()
File "/usr/local/lib/python2.7/site-packages/psutil/__init__.py", line 669, in status
return self._proc.status()
File "/usr/local/lib/python2.7/site-packages/psutil/_pslinux.py", line 1470, in wrapper
raise NoSuchProcess(self.pid, self._name)
NoSuchProcess: psutil.NoSuchProcess process no longer exists (pid=20)
Do you have an idea how to fix it?
karpitsky commented
I think I've found a solution for my case. My config has QUEUES = ("default",), so when I run a scheduler mrq-worker --scheduler it also runs a worker for the default queue. If this worker gets a job it will fail with the error KeyError 'scheduler', see the value of self.config below. I've removed the "default" queue from my config and it works.
self.config on the line 510 for the worker
{
'scheduler_interval': 60,
'add_network_latency': '0',
'no_import_patch': False,
'worker_class': 'mrq.worker.Worker',
'greenlets': 100,
'redis_max_connections': 1000,
'tasks': {},
'trace_greenlets': False,
'default_job_cancel_ttl': 86400,
'report_interval': 10,
'redis_timeout': 30,
'log_handler': 'mrq.logger.MongoHandler',
'log_level': 'DEBUG',
'mongodb_jobs': 'mongodb://127.0.0.1:30000/mrq?ssl=true&ssl_ca_certs=/opt/ssl/cacert.pem',
'print_mongodb': False,
'redis': 'redis://127.0.0.1:6379/2',
'max_memory': 0,
'version': False,
'redis_prefix': 'mrq',
'config': None,
'profile': False,
'scheduled_tasks': {},
'scheduler_tasks': [***],
'log_handlers': {},
'default_job_ttl': 15552000,
'max_jobs': 0,
'use_large_job_ids': True,
'default_job_timeout': 86400,
'project_dir': '/usr/src/app',
'report_file': '',
'processes': 1,
'trace_memory_type': '',
'name': None,
'trace_io': True,
'default_job_retry_delay': 3,
'default_job_abort_ttl': 86400,
'default_job_max_retries': 3,
'trace_memory_output_dir': 'memory_traces',
'quiet': False,
'mongodb_logs': '1',
'trace_memory': False,
'dequeue_strategy': 'sequential',
'paused_queues_refresh_interval': 10,
'log_format': '%(asctime)s [%(levelname)s] %(message)s',
'queues': ('default',),
'default_job_result_ttl': 604800
}
self.config on the line 510 for the scheduler
{
'ensure_indexes': False,
'add_network_latency': '0',
'no_import_patch': False,
'worker_class': 'mrq.worker.Worker',
'scheduler_interval': 60,
'greenlets': 100,
'redis_max_connections': 1000,
'worker_id': '',
'trace_greenlets': False,
'default_job_cancel_ttl': 86400,
'dequeue_strategy': 'sequential',
'admin_ip': '127.0.0.1',
'scheduled_tasks': {},
'redis_timeout': 30,
'log_handler': 'mrq.logger.MongoHandler',
'log_level': 'DEBUG',
'tasks': {},
'mongodb_jobs': 'mongodb://127.0.0.1:30000/mrq?ssl=true&ssl_ca_certs=/opt/ssl/cacert.pem',
'worker_profile': '',
'print_mongodb': False,
'redis': 'redis://127.0.0.1:6379/2',
'max_memory': 0,
'subqueues_refresh_interval': 60,
'version': False,
'redis_prefix': 'mrq',
'scheduler_tasks': [***],
'config': None,
'profile': False,
'report_interval': 10,
'max_jobs': 0,
'task_blacklist': '',
'admin_port': 0,
'max_latency': 1.0,
'log_handlers': {},
'default_job_ttl': 15552000,
'agent_id': '',
'scheduler': True,
'use_large_job_ids': True,
'default_job_timeout': 86400,
'project_dir': '/usr/src/app',
'worker_group': '',
'report_file': '',
'processes': 1,
'trace_memory_type': '',
'name': None,
'trace_io': True,
'default_job_retry_delay': 3,
'default_job_abort_ttl': 86400,
'default_job_max_retries': 3,
'trace_memory_output_dir': 'memory_traces',
'local_ip': '***',
'quiet': False,
'mongodb_logs': '1',
'external_ip': None,
'trace_memory': False,
'max_time': 0.0,
'paused_queues_refresh_interval': 10,
'log_format': '%(asctime)s [%(levelname)s] %(message)s',
'queues': ('default',),
'default_job_result_ttl': 604800,
'task_whitelist': ''
}
sylvinus commented
Hi @karpitsky !
Thanks for the report. I can't reproduce the behaviour with the latest master.
I don't see how "scheduler" wouldn't be in self.config. It should at least be False because it's in the config in all cases for mrq-worker:
https://github.com/pricingassistant/mrq/blob/master/mrq/config.py#L365
Are you modifying the worker config somewhere maybe?
thanks!