Crons : Simple scheduled builds for your repositories
joshk opened this issue · 149 comments
Earlier this year we dark shipped a new feature for running scheduled builds for repositories, which we called Crons.
This feature was developed with the help from some amazing students at HPI University in Potsdam, Germany.
You can find current, and up to date docs for Crons here: https://docs.travis-ci.com/user/cron-jobs/
This feature is going into General Availability on the 6th of December.
Please leave all feedback related to this feature here as comments.
UPDATE: 2017-01-20
I wanted to give a quick update that I'll being going through all the feedback shortly and provide some information on when this will come out of Beta and what improvements we have lined up.
Thank you everyone for your feedback!
UPDATE: 2017-02-02
Thanks a lot for your feedback everyone!
I am adding each of the suggested changes below for votes.
Once we have gathered votes over a few weeks, we will decide which of these can be implemented earliest to be able to get Crons out of Beta.
UPDATE: 2017-03-02
We have started working on fixing and improving some things to move Crons out of Beta, details here.
I've been using it for a few months now to schedule tests. I wish I could update the time that the job runs.
I re-generate the metadata with Travis by triggering it from AWS Lambda. Was really excited to switch to Travis' native cron, but "daily" is way too rarely.
Any chance to get "every 5 minutes"?
Great feature!
I just tried this on a couple of my repositories, and it fails to add with the following errors on the js console:
raven.js:1062 GET https://api.travis-ci.org/v3/repo/3969047/crons 403 (Forbidden)
Hi @bmcfee, thanks for the error report, could you try doing a reload/refresh of the page and see if that helps. If it doesn't, could you please email support@travis-ci.com and we can look into this.
@alvy54 Thanks for the feedback. We are looking at making this information editable in a future release, possibly in Q1 or Q2 in 2017.
@bsideup Thanks for the feedback. We are considering other interval choices, but it is unlikely we will add 'every 5 mins' for the time being. I will provide more context around this in Q1 once we do some further planning around the next iteration of Crons.
For the record, I'm having the same problem as bmcfee, and I've sent an email to support@travis-ci.com with details. Please don't hesitate to ask if you need additional info, thanks.
This looks like a great feature, thanks. Just another +1 to the request to have other interval option, a week is too often, a month is too rare.
Also would be great to have more option (that scale with the interval) for skipping, e.g. running fortnightly only when there was no build in the last week?
It looks like that I'm not able to add the cron job to a project I maintain (astropy/astroquery), I suppose the jobs should be listed as in the blogpost's screenshot?
Edit: looking at the console, this seems to be the same problem reported above.
👍 to this feature. I am also having trouble adding a cron job to public repos with a similar error as reported by @bmcfee
Thank you for the comments and feedback about the issues you are experiencing in setting up your first Cron schedule.
We are looking into the issue now and will provide an update shortly.
For the time being, please add an emoji reaction of 😕 to this comment if you are experiencing issues adding a Cron schedule to a repository.
We have deployed API changes that should properly enable crons for all. Can anyone who was having a problem before try it again and let us know if it doesn’t work?
It works now, thanks.
@backspace - Adding the job works for me, and it also started the build. Thanks.
Excellent, thank you. We expect it should work for everyone now. You can leave a 👍 on my post if it’s working for you now when it didn’t before. If it’s still broken, please leave a comment!
As a feature request, I wonder whether you consider to add an option to have a separate config file for the cron jobs, or to be able to add extra builds to the matrix that would only run when the build is started from a cron job? It would be a nice workaround to travis-ci/travis-ci#6774 where I seek a way to have builds that only run on master, but not on PRs.
EDIT: anyone using conda environments for their setup may be interested in https://github.com/astropy/ci-helpers. I've added the EVENT_TYPE env variable to control whether to run a build for the given event or not (events are the same as for TRAVIS_EVENT_TYPE: cron, pull_request, push or api)
As a feature request, I wonder whether you consider to add an option to have a separate config file for the cron jobs, or to be able to add extra builds to the matrix that would only run when the build is started from a cron job?
To offer up an alternative use case for this functionality WordPress has just added an official GitHub mirror this morning https://github.com/WordPress/wordpress-develop, I'd like to add a daily cron job that tests the codebase against all the major releases of MySQL and MariadDB. There is no need to test against all these databases for every commit so a daily cron job is ideal.
Great to have this feature! I would really appreciate if the starting time could be configurable - usually I'd prefer to have the automated builds happen at night when nobody's working on the code. A simple dropdown for picking the base hour for all cron jobs (instead of using whatever time it was at initial setup time) would alreay do the trick.
Regarding the request by @bsipocz I can imagine much of this could be solved by simply adding a special env variable that flags automated builds, something along the lines of TRAVIS_CRON_SCHEDULED=yip
I’ll leave the other feedback for Josh to respond to, but regarding the environment variable idea, @colszowka, there already is one! You can check TRAVIS_EVENT_TYPE for the value cron to know a job was triggered this way. 🎉
@backspace - Thanks, this is very useful to know, I think this case I can easily have a workaround script by using this or the other newly recognized variable TRAVIS_PULL_REQUEST from here https://docs.travis-ci.com/user/customizing-the-build#Implementing-Complex-Build-Steps.
I've got a use-case for having a separate config for Cron jobs, I think — I don't want to queue up OS X builds for every PR; I'd much rather do one per day. However, if I list the OS X build in my env matrix, it gets queued up no matter what — I can do an early exit from it, but it's got to start first.
I could use either a separate config or some sort of matrix-job flag for "not on cron-triggered builds", I guess.
(My env matrix has a trusty container, a precise sudo-enabled machine, and an OS X machine, which run JS builds, Android builds, and iOS builds respectively, but I only want the trusty container to run on pushes and PRs. Ideally, I would only trigger the other two from cron.)
We are using Crons for our Coverity builds for FFmpeg.
So far everything runs great, with Cron set to trigger daily.
Coverity only allows us 4 builds per week, so right now we have some code that checks the day-of-week and exits out from the build early.
Some way to control the triggers more fine grained than just Daily/Weekly/Monthly would be nice.
Maybe even full crontab style, but limited to one build per day?
We could potentially workaround this by adding 4 weekly triggers, but that would mean someone would have to set up the 4 trigger exactly at the day and time we want them to run.
So some option to set the exact trigger time would help here.
Would definitely like to be able to control the hour that it runs (want late night east-coast US time, after most developers have reached a stable state for the day.
Would like to be able to build conditionally on there having been changes since the last build.
Would be satisfied with just nightly builds, and not after every change (many documentation changes don't need an immediate build).
Apart from the predefined hourly/daily/weekly options, would be good to have a custom option where we can enter a custom cron expression
Love this feature, works great for month!
Just started using it to ensure I am testing daily builds, so far its been great!
Great idea, thanks!
Great feature, it would be even more useful if you could set the time when the build is executed and if these builds could take precedence over all the others. This would allow proper "nightly" builds starting at the chosen hour, even if there is a long queue of other builds.
Also it would be useful to have separated tabs for builds triggered by commits and builds triggered by cron jobs.
By the way, I noticed something weird, I added the build for 2 different branches, and they took the same build number, see this and this. Is this expected?
My team is using cron jobs to schedule nightly clean builds (no caching dependencies, etc.) and we expect them to take much longer than our standard builds. The ability to decide when jobs run–even if it's a wide range–would be greatly appreciated.
A lot of people want to specify a precise build time. I'd also like the opposite: check a box saying that I'm flexible concerning the build time.
I'm running weekly or monthly builds to check that updates to third-party dependencies did not break anything in my code. I don't care at what time the build runs — for monthly builds I don't even care about the day, it could be ± 3 days.
I would therefore be happy if I could tell travis-ci.org to run my weekly or monthly builds when the traffic is the lowest (around 4 AM CET), or when the machines are the cheapest (if you're using one of these flexible plans).
It works fine but i would be happy to have a more fine grained trigger like "every hours"
@abioteau why is your project worth running an autobuild every hour? Do you expect your dependencies to change very quickly? Please explain why a project would need so much build resources.
As well as the weekly/monthly/daily, can we put in our own cron strings?
I seem to be unable to add a daily build for a branch. A validation error shows saying I need to select a branch but I have done so.
Hey, @mseminatore, can you let us know what repository and branch this is happening for? Is it on travis-ci.org or travis-ci.com? Alternately, can you write in to support so we can have someone look at it?
It looks like there is a bug with the cron scheduling. I activated a weekly build for astropy/astroquery with the option to not run it if there was a build in the last 24hours. Nevertheless it started a cron build right after a build has just finished: https://travis-ci.org/astropy/astroquery/builds
hourly/daily/weekly options +1
👍
feature: cron job events
i wish to setup a cron job for a go package set to the Go-Release-Cycle (two times a year) described at https://github.com/golang/go/wiki/Go-Release-Cycle.
It would be great if the release event is maintained by travis-ci. So go packages could be tested automatically with a new golang release (the next comes soon, in february 2017).
an other scenario is to setup this feature to the go: tip parameter of the travis configuration.
This event could be also created for other language releases supported by travis-ci
Hey, @bsipocz, I tried what you described on a recently-built branch of my own and I found this:
So it says it’s scheduled in a minute, but then a request never showed up:
I think this is a jargon problem on our end. It’s technically true that it’s “scheduled” because that’s what it’s called internally, but it’s confusing because that implies that it will actually run, when really it’s just being checked for whether it should run.
While this is what happened in my attempt and my coworker’s attempt to reproduce your issue, perhaps there was something different in your case and the build actually did run. If so, can you let us know which build it was so we can look into the logs and hopefully understand the difference?
Thanks for the feedback!
ETA: I see that it’s also confusing that after it skips a build because one has already happened in the past 24 hours, it says it ran recently:
@backspace - I saw that the cron run timestamp means that the trigger was made and based on the check it either run or skipped the build. However here the build actually run when it shouldn't have:
The normal build on master was this: https://travis-ci.org/astropy/astroquery/builds/185557677
The cron build right after this was: https://travis-ci.org/astropy/astroquery/builds/185560778
My theory is that the normal build was still running when the cron scheduler run, and it didn't recognize it as a build in the past 24 hours. The other theory is that the builds are not all green, but have some test failures (which sadly is totally normal in this repo) and the cron may check that status, too.
ah yes, thanks for the extra information, @bsipocz. I think it’s true that a currently-running build will not count as a build in the last 24 hours, because it’s looking for completed builds. I don’t think that failed jobs count against the 24-hour limit though. I’ve opened an internal issue to discuss clarifying the terminology around this and added a note about this.
Thanks, than this must be it. I think my workaround will be to wait until Sunday dawn, and then remove and readd the cron job. At that time slot it will be less likely to have builds running for our organization, or to have a system wide osx backlog.
Great feature! We had a build that suddenly failed when we openend a pull request because we didn't have nightly builds. I think it would be a great addition if I could add schedule the job inside my .travis.yml.
In our case we have a seed repository which contains a default project that we copy from time to time. It would be great if it would have a nightly build by default 👍
Just did a write-up on using this feature here. If there's a way to simplify some of this, that'd be good, but I'm not sure that there is a straightforward way to do so. Thanks for the cron feature!
Thanks for the link, @binoculars. That’s quite the setup you’ve got there! I don’t have any suggestions for improving it, though I’m more focused on the front end. One thing though is that you no longer have to write in to support to get access to crons, they are now generally available; this thread is part of the public beta.
Are there plans to add cron configuration to your api?
It’s already part of the “V3” API, @softprops, as that’s what travis-web uses to manage crons. You can see /repo/:id/crons for listing crons and POST /repo/:id/branch/:name/cron for creating one, for instance.
I have so many <3's for travis. You just earned one more!
@backspace I can't actually access those links. How do you get into the developer program?
Wow, my apologies, @softprops, I was unaware that that was not generally available. This seems unfortunate to me, but I gather this version of the API is still in flux and not ready for being public.
I’ve added you to the developer program and I can do so for anyone else who wants. I’ll see about putting the cron API documentation here, with a caveat that it may change.
Sweet. I'll have a closer look tomorrow. Many thanks
I'm loving the crons already! My main feedback so far is that it would be really convenient if we could define environment variables for each cron (just as we can do for each repository).
This would allow a project to "fail fast" by default in response to active development events like PRs and merges, but set the appropriate environment variables for important branches' crons to schedule slower and more comprehensive tests .
This feature is going into General Availability on the 6th of December.
@joshk was the GA date delayed?
@mit-mit Crons have been out since the 6th of December :)
In GA (General Availability)? If so, why does the UI still say 'beta feature'?
Your feature like your product is so great, simple and smooth.
This really helps me to trigger weak dependencies as soon as possible.
A daily period is quite enough for me.
Having an hourly option (or at least something like 4 times per day) would be great for running some cross-product integration tests. Thaaanks!
Two additional features that would be helpful:
-
Some indication in the history (e.g., on the Build History or Branches pages) that a given build was run by the scheduler (rather than manually to triggered by a commit). In fact it would be great in general (where possible) to indicate the "cause" of each build.
-
Notifications of automatic builds, distinct from manual ones. For example, I don't need an e-mail when a manual build success, but I'd like one when a scheduled on does.
@orome - If you look at the icons next to the build number (e.g. in Build History), they are different, based on the TRAVIS_EVENT_TYPE variable.
@bsipocz - Indeed they are! I missed that. Is there a key to what the mean somewhere (a few are clear enough).
I would love to see dependent builds!
Also, can you update your Slack integration to differentiate between a cron build and other builds?
It works now, thanks.But how to set cron expression and timezone?
I think there's something broken with the "Do not run if there has been a build in the last 24h" option. (Or I'm really confused about what it's supposed to do.)
I have it set to run weekly on my master branch, but not if there was a build in the past 24h:
("Ran 7 days ago" is titled "2017-01-12T22:06:54Z".)
But in my project's build history, there hasn't been a build on the master branch in the past 20 days -- and there weren't any builds within 24 hours of 2017-01-12. The last master build was on 2016-12-30:
Is there some reason the cron on 2017-01-12 didn't think it needed to trigger a build? (Same question about a week earlier, 2017-01-05, presumably.) Or am I not understanding how this is meant to to work?
(BTW, awesome feature! My project wraps third-party APIs, so periodically testing they haven't changed out from under me is really valuable.)
I wanted to give a quick update that I'll being going through all the feedback shortly and provide some information on when this will come out of Beta and what improvements we have lined up.
Thank you everyone for your feedback!
I'd be interested in the ability to set up multiple jobs on the same branch, utilizing an environment variable for the ID or name of each build. For instance, I want one job to run daily and execute some steps. I want another job (on the same branch) to run weekly and execute different steps. I'd use the environment variable in my travis.yml to decide which job it was, and subsequently, which steps to execute.
👍 for @jcory
I'm looking for exactly the same feature - either an option to specify some custom env var(s) for each cron build or to assign names/IDs for those builds (TRAVIS_BUILD_NAME?).
Another nice feature (not necessarily related to cron builds only) would be an option to provide different YAML file for each cron build and/or for each event type (push/pull_request/api/cron) - i.e. ".travis_push.yml", ".travis_cron.yml", ..., ".travis_my_cron_build1.yml",
Hi @medmunds
Crons were expected to run thrice in the last 3 weeks for your master branch. All three of them were correctly scheduled, on commit ac8147b but it has [ci skip] in the commit message which is why they were skipped.
The next job, scheduled to run on 26th January, should run fine, considering the last commit on master does not skip ci.
More on Skipping Cron Jobs here.
You can see the list of requests and why they were skipped here.
Happy to hear that crons are a good use case for you!
Hi @aakritigupta -- thanks for the explanation. (And the hidden "/requests" build triggers list is really useful.)
I had been using [ci skip] to try to conserve Travis-CI resources, by marking commits that only change docs or otherwise couldn't affect the tests. But I'd still want the cron builds to run, even following one of those commits, to test against potentially-updated dependencies.
I can see a different use of [ci skip] would be to mark commits that intentionally break the builds, so there'd be no point in running again (even on cron) until a later commit fixed them.
(And really, I should probably be testing the docs in my Travis builds, too. Bottom line is, I'll stop using [ci skip].)
I just tried to set up a new cron build on a non-master branch - I had to refresh the browser page before it picked up my new branch in the cron drop-down. I did navigate away to the resync account page in case I needed to resync, when I came back still no branch in the drop down. F5 page refresh and the branch was there. Minor detail but had me looking around for something I messed up. Thanks for this nice feature.
Just noticed that in Travis CI settings and will try it for sure. Thanks, awasome idea 🥇
Thanks for letting us know @brabster. I have opened an internal ticket to look into this issue.
Thanks a lot for your feedback everyone!
I am adding each of the suggested changes below for votes.
Once we have gathered votes over a few weeks, we will decide which of these can be implemented earliest to be able to get Crons out of Beta.
Suggestion 1
Make options for intervals between cron jobs more frequent, e.g. every n hours or every n minutes
Suggestion 2
Make time/day of running crons configurable, with more granularity, ideally with custom cron expressions.
Suggestion 3
Allow separate config file for cron jobs
Suggestion 4
Improve UX:
- "Ran at .." should show the time of the last build run and not last scheduled run, since a cron could be skipped in some cases.
Suggestion 5
Add an option to run a cron job even if it is for a commit with [ci skip] (or variations) or a separate way to specify when a cron should be skipped, e.g. [cron skip]
Suggestion 6
Add an option to name cron jobs
Suggestion 7
Separate options for enabling notifications for cron and other types of builds
Suggestion 8
Add an option to prioritize cron builds over push and pull_request builds
Suggestion 9
Add an option to define how long after a build should the next cron build. (right now, there is only an option of skipping a build if one has been run in the last 24 hours, or not skipping a cron at all)
Suggestion 10
Add an option to run crons within an interval but with flexibility, i.e., "Run within x days, but precise time or day does not matter"
Suggestion 11
Add an option to include cron job settings within .travis.yml
(see @simondel's comment above)
Suggestion 12
Update Travis' Slack integration to differentiate between cron and other builds
Suggestion 13
Add an option to set up multiple cron jobs on a branch (see @jcory 's comment)
Can I add a suggestion for a a button that allows a scheduled build to be run on demand, like "run now" or something? If I setup for a scheduled build, it'll be run immediately but if there's something wrong with the build I need to make my fix and then delete/recreate the scheduled job to check that it works. No big deal but might be a nicer ux
@brabster agree it's a nice idea, but you can achieve this currently by rebuilding the branch you're testing can't you?
@riggerthegeek good point, I should check the behaviour... so, on the same branch:
- pushed a breaking change into the scheduled build
- normal build ran fine
- scheduled had not rescheduled after waiting 5 minutes
- deleted/recreated scheduled build - ran and broke
- pushed a fix to same branch
- normal build ran fine
- scheduled had not rescheduled after waiting 7 minutes
- deleted/recreated scheduled build - ran fine.
this was builds 343-346 on https://travis-ci.org/brabster/crucible/builds
Hi Travis,
thank you very much for this great feature. What I am personally missing is showing "cron" builds in "Build History". Currently, it looks like that the only place to check if cron build was executed is in "Settings". Also, there is no way how to see build output as it is possible with normal builds.
Cron builds are listed in Build History. They have a different icon before #xyz created/started/passed/failed.
@michael-k , thank you very much for your response, but I still can't find "cron" builds in https://travis-ci.com/linz/bde-processor-tests/builds
@imincik I think you might have found a bug. Could you contact us at support@travis-ci.com?
Hi there.
Would it be possibile to add a cron-job to the latest version-tag?
That would be nice! Because we have our deployment bind to the latest tag.
I might be wrong, but I am pretty sure that there's no such thing as a latest tag. You can sort tags lexicographically, leading to 1.1 < 1.10 < 1.9.