time.sleep in long_task

[Deleted user]

Quick question: in the long_task celery function, i notice that you have used time.sleep(1)

Not sure I fully understand the implications of the sleep statement here ? I tried removing it and cant seem to see any differences in the output atleast cosmetically. Would appreciate any thoughts ?

for i in range(total):
    if not message or random.random() < 0.25:
        message = '{0} {1} {2}...'.format(random.choice(verb),
                                          random.choice(adjective),
                                          random.choice(noun))
    self.update_state(state='PROGRESS',
                      meta={'current': i, 'total': total,
                            'status': message})
    time.sleep(1)

Thanks

[miguelgrinberg]

Typically, when you have a potentially long loop you want to make sure other threads get access to the CPU instead of starving until the long computation ends. Depending on what kind of platform your application runs on, the Python scheduler may or may not distribute the CPU resource among all the tasks on its own (i.e. what's called preemptive multi-tasking). In some cases the tasks need to be nice and release the CPU on their own, and the best way to do that is to issue a sleep. This is called cooperative multi-tasking. For example, if you are using gevent or eventlet, this is required. If you are using standard CPython with threads, it is not required.

I think I must have added that sleep out of habit. Even when pre-emptive multitasking is used, doing a sleep is a good way to give hints to the scheduler on when it is best to pass the CPU to another thread.

Hope this helps. Let me know if this isn't clear.

[Deleted user]

Perfect - that makes sense. Thanks Miguel.

One followup question if i may: im trying to load excel files into pandas using celery as a task queue where user uploads get queued up. Pls correct me if im wrong : since file I/O is not asynchronous, multiple files cannot be loaded in parallel ?
I posted the details here: http://stackoverflow.com/questions/31727688/gevent-socketio-file-i-o-blocking-or-non-blocking-speed-up-file-read

Thanks again for your time

[miguelgrinberg]

This issue will be automatically closed due to being inactive for more than six months. Seeing that I haven't responded to your last comment, it is quite possible that I have dropped the ball on this issue and I apologize about that. If that is the case, do not take the closing of the issue personally as it is an automated process doing it, just reopen it and I'll get back to you.