feature: Add cron-style periodic worker support
acaloiaro opened this issue · 5 comments
Description
Cron-style periodic workers are required by many applications. Less robust solutions can be concocted with PerformIn, which leaves room for a more explicit way to express job run periods.
Additional Information
I opened this Issue to reference #2308
First, the PR #2308 is a breaking change invalidates all existing (official/unofficial/custom) implementations. It should be considered as a separate interface or extended when we redesign the Worker support in a future major version.
Second, the scope of the cron job is not clear.
The concept of the Workers is basically a kind of long-running task handler to help request handlers, by separating the time-consuming jobs which can be proceeded as a post process and do not directly affect the response. Good examples could be such as N-screen transcoding after video uploading or creating downloadable backup copy of the user data in background when a user requested a backup.
I think this periodic job (or periodic worker) is in the service scope (a kind of batch tasker for a whole service/application), which is not directly related to any HTTP request, and the implementation should be safe even though there are N instances of the same application in a single service if the code is in the application or we need a caution for it.
Providing a web based batch management interface, or Buffalo powered batch worker could be nice, but I am not fully sure if that feature could be a core of Buffalo and I feel we need more consideration if we going forward with it.
Still, this feature request is valuable and I think this is a great input for us as voice of the field. I am not going to merge the PR for now but if you provide ideas about batch job or your real situation, context, or use cases, I will be more than happy!
Hi @sio4. As you may already know, I've not meaningfully contributed to buffalo in the past, so I'm not familiar with the typical procedure for proposing breaking changes. My usual proposal process for projects that don't have a contributing.md outlining the breaking-change process is to do what I've done with the PR #2308 and let project maintainers push back on PRs and propose how to move forward, if the PR is something they want to consider.
If we were to move forward with this concept, I think, like you suggest, that a new PeridoicWorker interface would be in order, and likely a new SimplePeriodic concrete implementation.
The concept of the Workers is basically a kind of long-running task handler to help request handlers, by separating the time-consuming jobs which can be proceeded as a post process and do not directly affect the response.
I think if this is the concept for a worker, then periodic workers are probably a poor fit for this project and I'd be happy to close the PR. However, if you're interested in expanding the concept of Workers to include use cases that are not so closely tied to "requests", then perhaps periodic works are a good fit for this project.
Personally I find periodic workers useful for operations such as
- Running system tasks such as: batch emails, image processing, vacuuming databases, statistics/metrics reporting
- Application-specific features: applications that fire webhooks on an interval; applications that "watch" remote resources for changes; etc.
If thee are use cases that you think Buffalo could serve, I'm happy to work on this further with you. Otherwise, feel free to close this issue and corresponding PR.
p.s. The CONTRIBUTING.md linked to in README.md is a broken link.
Cheers
Oh, Thanks for letting me know about it! It seems like I missed updating README while I centralized CONTRIBUTING.md in the organization-wide .github repo. I will update it soon.
Yes, this kind of batch job is slightly different from the original worker concept, but as you said expanding the concept of worker could be nice and I believe there are users who want a batch job feature. So I would like to reconsider this expansion in the future once we redefine the worker concept in v2.
One of the important considerations of the worker would be reliability, which is not solid for now. If we consider a batch email of the announcement, for example, we should send a message to all users but just only one copy for a single user. So in a production environment with multiple application instances, we should have(or provide) a method to prevent duplicated running of a single job for the same target on multiple instances. There can be various methods for each use case so we can just document it, or we can consider implementing a solid systematic solution for it (could be limited to a specific range of use cases).
It could be nice if you could help us with this, including the design of the future worker, in future. Previously I filed a tracking issue for the new design of the worker for v2 as #2242, but I would like to use this issue as a tracking issue for the concept of batch/periodic worker. Please feel free to share your idea, feedback, or any thought here. It would be very helpful to us!
Many thanks!
I'm closing old issues. Since this has been open for a year without any activity, I'm closing it now.
If anyone wants to continue this issue further, feel free to reopen it.
Cheers