Nextdoor/ndscheduler

[Question] Scalability

Closed this issue · 2 comments

abnerjacobsen commented

I am interested in using ndscheduler in a solution that I am developing but I have the following requirement:

  1. My app will use the ndscheduler API to generate about 30,000 schedules per day within 2 hours time frame.

  2. These schedules will run only once and their purpose will be to send an attendance confirmation email to a customer.

  3. Schedules will have their execution start date and time distributed over a time period of 6 hours during the day.

  4. After the schedule is successfully executed it can be deleted from the ndscheduler.

Does the ndscheduler support this workload?

Thank you.

kartick commented

Why don't you use a distributed task queue such as Huey, RQ, or Celery for this? Or you can use a combo of a task queue and NDS (though to me, it seems like an overkill)

sabw8217 commented

So I think you could do this, but if you have jobs that will only run once, you probably want a distributed task queue for that. There's not really any reason generating 30K schedules a day wouldn't work, but we certainly haven't tested it.

If you want things to run at certain times, I'd recommend instead having a job that starts up periodically during the day and checks some other distributed queue (e.g. SQS, RabbitMQ) to see what work needs to be done.