Load data in this project itself
Opened this issue · 2 comments
Is your feature request related to a problem? Please describe.
Now, this project loads data with GitHub action.
A better way is to develop a corn job in this project. and it will deploy to vercel too.
Using GitHub action is convenient for developers because there are many ready-to-use actions. But this is somewhat inconsistent with the whole project in user's view.
Describe the solution you'd like
A clear and concise description of what you want to happen.
Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.
Additional context
Add any other context or screenshots about the feature request here.
The data pipelines was developed in Ruby by @hooopo before this repo. It's quite hard to migrate ruby codebase to Vercel Serverless Function environment.
In addition to the language issues of the data pipeline, even if the data pipeline is written in TypeScript, it is difficult to host it on Vercel for free due to the following reasons:
- Vercel's cron functionality is limited, with the maximum running time being 10 seconds.
- In the future, Vercel's cron will become a paid feature.
Are cron jobs free?
While in beta, cron jobs are free on all plans. However, it'll be a paid feature for general availability.
If users don't host it on Vercel or GitHub Actions, they would need to deploy it using their own servers, which could prove to be too costly for them.