Should we use a Azure Task?
epage opened this issue · 28 comments
spontoreau's Task might be a good basis for us. See discussion below
Original
Including
- Repo name(s)
- One or more repos
- One or more branches
- Folder hierarchy
Inputs
- Allow evolution
- Don't break CIs
- Simple, naiive case should not lead user to unstable resources
- If using branches for different versions, someone might naiively grab the URL for a master version of a script and we could end up breaking them
- Testability
- Azure Pipelines: each template repo has to be explicitly added in the
resources - Easy changelog management
@andoriyu brought up on gitter the idea of making and publishing tasks. For rustup and gh-install (#3), I think this solves a lot of problems brought up in here.
One set of rust tasks already exists though it might have some extra policy compared to our rustup steps. We should reach out and see about collaborating.
I think managing shell scripts inside yaml is too much work and messy. Having ci folder with all the scripts is sub-optimal as well in my opinion - it's not part of the code base. That's why I wanted to write my own tasks but ran into an already existing project.
Which scripts are you concerned about? The ones inside rustup template are trivial and seem to work well. Wanting to better understand the concern here.
If talking about things more like gh-install, see my note on #3
My concern is for consumers of this scripts. If I understand this project correct the idea is to place those templates into a repository and use them in pipeline definition.
Which leads to two way of handling it:
- Git Submodule (doubt anyone will try it)
- Copy scripts over (most people will go this way
Then if script gets better or changes people won't get an updated version. However, if it's treated as black box and hosted somewhere else — everyone uses the same version.
If I understand this project correct the idea is to place those templates into a repository and use them in pipeline definition.
Ah, this is where the confusion is. This is meant to be a central resource rather than something people copy/paste. I want to get tooling in place so we no longer need trust and its approach to CIs (copy everything but install.sh).
(For azure pipelines) What are the tangible benefits of tasks over something like a template? I know that they are more easily versioned. Anything else?
(For azure pipelines) What are the tangible benefits of tasks over something like a template? I know that they are more easily versioned. Anything else?
Discoverable via market place. I guess feels more natural since there're tasks for many other things already. Easier to share with people IMO.
If I understand this project correct the idea is to place those templates into a repository and use them in pipeline definition.
Ah, this is where the confusion is. This is meant to be a central resource rather than something people copy/paste. I want to get tooling in place so we no longer need
trustand its approach to CIs (copy everything butinstall.sh).
Like a submodule or like an example for DIY?
Like a submodule or like an example for DIY?
By adding the org as an endpoint in the GUI, declaring the repo as a resource, and then referencing the templates in that repo
(For azure pipelines) What are the tangible benefits of tasks over something like a template? I know that they are more easily versioned. Anything else?
End-user overhead for adding a remote repo as a resource is one of the reasons I'm preferring using a single repo and a single branch for all our needs but a lot of processes don't work as well in that case.
- We'd have to have separate test folders for every major version
- Branching and integration is weirder
- Browsing commits to see what has changed is weirder
- etc
I'm assuming it is much lighter weight for users to add a task from the market place than to use a remote repo's templates.
The downside is it looks like tasks are more complex to write. If I could package up a template as a task, that'd be slick.
You're right, tasks are not really easy to write. Unit testing capabilities are really low, that's why the extension is moving slowly, because it needs a lot of manual testing before releasing it!
Also, tasks can be used in YAML and with the visual designer (vNext build definition). I'm currently working on a better integration of the extension in YAML.
Anyway, nice repository it can really help :)
- We'd have to have separate test folders for every major version
Would that need to be the case? It seems like testing every individual template major version would be a huge boon as users on an older version wouldn't have any functionality removed out from under them.
- Branching and integration is weirder
Not sure I understand. If we're only dealing with a single repo and one branch, if a template is changed in a PR -- then everything that the PR touched is tested in the PR build
- Browsing commits to see what has changed is weirder
Would a per template or global changelog fix that issue?
I'm assuming it is much lighter weight for users to add a task from the market place than to use a remote repo's templates.
I think I disagree. In the last several days, I have not come across any notion of a marketplace. A marketplace (/ custom tasks) are not described in the pipeline documentation nor listed in the pipeline ui. Templates are. I want pipelines to be copy and paste-able between projects with the minimal amount of out-of-band steps to help me avoid mistakes. So far templates seem to be the best way. Not sure what my past self was thinking -- custom tasks are most definitely documented
You're right, tasks are not really easy to write. Unit testing capabilities are really low, that's why the extension is moving slowly, because it needs a lot of manual testing before releasing it!
Let me check for understanding from the market place page for your tasks
You provide three tasks
- rustup install
- generic cargo
- generic rustup
What is the value of generic cargo and rustup tasks? When looking at writing step templates, I only wrote them for the "high boiler plate" stuff like rustup install and gh-install. I feel like it'd be just as much to type out the yaml for a cargo step template as it would be for just writing the cargo command.
The main benefit I can think of is
- Flags that are CI best practices (always passing
-verbose, replacing test with suity) - Passing the configured
--target
Also, from the screenshot, it looked like your rustup install had a nightly checkbox. Does this mean it only supports nightly and stable? I have found that being able to take in any toolchain name is important, including pinning exact versions (consistent warnings), including nightlies.
I'm curious what you think of my rustup step's "API" compared to the current task's "API"
https://github.com/crate-ci/resources/blob/master/az-pipeline/unstable/rustup.yml
(See also #8 for more work planned)
We'd have to have separate test folders for every major version
Would that need to be the case? It seems like testing every individual template major version would be a huge boon as users on an older version wouldn't have any functionality removed out from under them.
I feel like we're saying the same thing, so I'm unsure where the disconnect is. We'd want every major version tested. If we have everything in the same repo, we'd then need separate test pipelines for every major version. This is in contrast to if we used separate branches, we'd have the same amount of files but we'd be able to leverage the merging, diffing features of git because they'd be treated as different iterations of the same file.
Branching and integration is weirder
Not sure I understand. If we're only dealing with a single repo and one branch, if a template is changed in a PR -- then everything that the PR touched is tested in the PR build
If we decide to make a breaking change in this model, we then copy the v1 folder into the v2 folder (the current unstable is a master with a clear warning). We'd be copying instead of doing git checkout -b v2. We'd have to open up diff programs and compare rather than doing a git merge. etc.
Browsing commits to see what has changed is weirder
Would a per template or global changelog fix that issue?
Ok, this is me being bad at managing a changelog as I go shows through. I normally write them after-the-fact. Granted, that doesn't really work with a "continually deployed" project like templates.
The question is global vs per section (gh-install, az-pipelines) vs per version. I'd lean towards per-version changelogs since people might not want to upgrade past a breaking change for reasons but they want to track what is being changed within that version.
Not sure what my past self was thinking -- custom tasks are most definitely documented
I think we are all finding how terrible the documentation is and we all have our subset of Pipelines knowledge. I knew about native tasks (bash, powershell) and someone at work recently mentioned creating tasks was possible but not the market place. @andoriyu filled my gaps here. It looks like @andoriyu might not have known about getting templates from remote repos.
You're right, tasks are not really easy to write. Unit testing capabilities are really low, that's why the extension is moving slowly, because it needs a lot of manual testing before releasing it!
Let me check for understanding from the market place page for your tasks
You provide three tasks
* rustup install * generic cargo * generic rustupWhat is the value of generic cargo and rustup tasks? When looking at writing step templates, I only wrote them for the "high boiler plate" stuff like rustup install and gh-install. I feel like it'd be just as much to type out the yaml for a cargo step template as it would be for just writing the cargo command.
The main benefit I can think of is
* Flags that are CI best practices (always passing `-verbose`, replacing test with suity) * Passing the configured `--target`Also, from the screenshot, it looked like your rustup install had a
nightlycheckbox. Does this mean it only supports nightly and stable? I have found that being able to take in any toolchain name is important, including pinning exact versions (consistent warnings), including nightlies.I'm curious what you think of my rustup step's "API" compared to the current task's "API"
https://github.com/crate-ci/resources/blob/master/az-pipeline/unstable/rustup.yml
(See also #8 for more work planned)
Ok, let's give you some context ;)
First, it's important you understand that the extension was initially built before the YAML support was released. So it was designed for vNext when Azure DevOps was named Visual Studio Team Service. Before YAML, all the commands has to be included in PowerShell/Bash steps, that was not really userfriendly. That's why I decided to build the extension, then YAML was released.
Now the extension have 4 steps (rustc is missing in your list) and they were designed (as you said) in a generic way. Now there are a lot of Rust developers who are intresting in using Azure Pipeline and templates seems to be the defaut way to handle CI. That's fine for me, it's more powerfull and CI as code is a better option for everybody ;)
My opinion is that Cargo, Rustup and Rustc tasks from the extension are now useless because YAML is the default way to add CI in a project. I just keep them as legacy steps for people who are using it in some build definitions with the visual editor.
On the opposite, the install task can be really usefull in YAML because it's a crossplatform one (it works on Linux/macOS/Windows without doing anything, it's handled inside the extension) and it can reduce the number of line in scripts inside the YAML. I'm currently working to improve YAML support around the extension. It will looks like this:
pool:
vmImage: ubuntu-16.04
steps:
- task: RustTool@1
displayName: 'Rust Tool Installer'
inputs:
nightly: trueSo we have a good option now if you define & maintain templates. Maybe I can just focus the extension as a "Rust Tool Installer" provider like Microsoft do with NodeTool task. It means that the current extension have to be depreceated and replaced by one that just contains a Rust toolchains installer task. As you notice this task doesn't handle all the scenarios... but if i'm just focus on it, maybe I can deliver missing features in few days !
What's your opinion on that?
cc @rylev, @johnterickson & @martinwoodward what do you think about that?
Thanks for the clarification / background @spontoreau.
I was wondering if this was from the pre-yaml days where a lot more task types would make sense.
As you notice this task doesn't handle all the scenarios... but if i'm just focus on it, maybe I can deliver missing features in few days !
I don't know if all scenarios need to be handled but I think the basics plus most common for CI setup. For example, I went ahead and put rustup component add in my step. People could create a script for that but I wanted a "initialize my CI" step. With that said, I don't handle everything but people can make their own rustup calls for very custom logic.
Personally preference but
- I think a combination of our names is ideal, like "RustupInstall". It makes clear the policy being invoked while a "RustTools" could be mistaken (and I originally did) for being equivalent of my "gh-install" (install a pre-built binary from github releases,
trust-style). - Maybe I'm biased, but I think we should look at mirroring my
rustuptemplate's parameters to our task'sinputs. From iterating on my CI a lot over the last two or so years plus looking at others, it seems to be about the right level of flexibility.
Overall though, it sounds like we are aligned. If you want, you are welcome to move it to the crate-ci org to help raise visibility (seems like a lot of us never heard about your work before), help with bus factor, etc.
Like the reasons you gave for RustTools being a task, I think we should probably make gh-install a task as well. That just leaves my rustfmt, clippy, and warnings templates. I mostly made those because I was playing with separate jobs for them (though a single job would be faster, see #4) and the per-job boiler plate was significant. I'm unsure if we should keep them.
I'd offer to help with the tasks but without knowing typescript yet or the APIs, I'm assuming my ramp up would be too much to be of benefit. I eventually need to get up to speed on these things, even if its just for $DAYJOB (migrating from perforce/jenkins to AzDO). If you want me to help, let me know. Otherwise, I'll probably continue my immediate task, rounding out cargo-releases features.
I agree that the "get rustup installed and in the PATH" is the most important one. It should expose as parameters the two rustup parameters:
OPTIONS:
--default-host <default-host> Choose a default host triple
--default-toolchain <default-toolchain> Choose a default toolchain to install
As I mentioned in Gitter, I think there might also be use for a "cargo test that also publishes test results", but I've not thought through that one as much.
Hmm given the size of rustup, maybe we can even get it in the default image (just rustup, no toolchains).
If we decide to make a breaking change in this model, we then copy the
v1folder into thev2folder (the currentunstableis a master with a clear warning). We'd be copying instead of doinggit checkout -b v2. We'd have to open up diff programs and compare rather than doing agit merge. etc.
Hmm yeah I envision the best way would be to lay out this repo in a similar fashion to how the azure pipeline tasks are structured. So we'd have
azure-pipelines/
├── rustup-1.yml
└── rustup-2.yml
rustup-2.yml would be the most current. A breaking change would then necessitate a bump and rustup-3.yml would be current
azure-pipelines/
├── rustup-1.yml
├── rustup-2.yml
└── rustup-3.yml
So no need for branches or copying folders / files or breaking existing pipelines.
I agree that the "get rustup installed and in the PATH" is the most important one. It should expose as parameters the two rustup parameters:
OPTIONS: --default-host <default-host> Choose a default host triple --default-toolchain <default-toolchain> Choose a default toolchain to installAs I mentioned in Gitter, I think there might also be use for a "cargo test that also publishes test results", but I've not thought through that one as much.
Hmm given the size of rustup, maybe we can even get it in the default image (just rustup, no toolchains).
@johnterickson it seems easy to implement inside the extension :)
@epage don't worry about the development, TypeScript is my main langage (Rust isn't yet :p) and if i'm focusing the install step it will be ok ASAP. I just need to finalize the version 1.2 first because it contains YAML support (it will be ok for the end of next week).
A name like "Rust Tool Installer" is more compliant with the current naming convention in Azure DevOps, that's why I choose it in my YAML exemple:
Before moving the repository, we can try something. Let's see if a feature request can be accepted by the Azure DevOps team to integrate an official Rust installer task in Azure Pipelines tasks. If they don't agree, I will move the repository to the crate-ci organization in order to group our efforts around this topic ;)
Hmm yeah I envision the best way would be to lay out this repo in a similar fashion to how the azure pipeline tasks are structured. So we'd have
Thanks, I didn't know this was a standard.
A name like "Rust Tool Installer" is more compliant with the current naming convention in Azure DevOps, that's why I choose it in my YAML exemple:
Good to know. I dislike that standard but sometimes its better to adopt a less than ideal standard than to do your own thing.
Thanks for your help on this!
Hmm given the size of rustup, maybe we can even get it in the default image (just rustup, no toolchains).
What are the trade offs for being in the base image? For example, I don't know how often rustup changes but keeping up to date?
I do know of at least one potential change we'll want ASAP, rust-lang/rustup#998
During the last 2 weekends I rework the CI of one of my rust project, because the travis build on windows was too long (timeout 50min on some run).
I tried cirrus-ci and azure pipeline. To compare and have same behavior for task/step, I decided to move script from ci's yaml into Makefile.toml and to use cargo-make (for test, zipping, upload,...).
So except the rustup toolchain setup, the scripts are ci independent.
The RustTool@1 of @spontoreau is only what I need to remove my only platform dependent steps (I don't use azure pipeline template):
# Linux
- bash: |
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain $RUSTUP_TOOLCHAIN
echo "##vso[task.setvariable variable=PATH;]$PATH:$HOME/.cargo/bin"
condition: eq( variables['Agent.OS'], 'Linux' )
displayName: install rustup
# macOS
- bash: |
curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain $RUSTUP_TOOLCHAIN
echo "##vso[task.setvariable variable=PATH;]$PATH:$HOME/.cargo/bin"
condition: eq( variables['Agent.OS'], 'Darwin' )
displayName: install rustup
# Windows
- script: |
curl -sSf -o rustup-init.exe https://win.rustup.rs
rustup-init.exe -y --default-toolchain %RUSTUP_TOOLCHAIN%
set PATH=%PATH%;%USERPROFILE%\.cargo\bin
echo "##vso[task.setvariable variable=PATH;]%PATH%;%USERPROFILE%\.cargo\bin"
condition: eq( variables['Agent.OS'], 'Windows_NT' )
displayName: install rustupMy 2c
PS: Funny thing, my crate ffizer is a cli to bootstrap or update project/folder from online templates (maybe some collaboration is possible, I planned to add templates for rust ci).
If we can get rust-lang/rustup#559 fixed, then one interesting option we have is to have a "Rustup tool installer" that installs just rustup and not any toolchains. (@epage I think I was getting ahead of myself about putting it in the default image 😊)
steps:
- task: RustupTool@1
- bash: rustup install $RUSTUP_TOOLCHAINor once rust-lang/rustup#998 is resolved (I'm guessing at the syntax)
steps:
- task: RustupTool@1
- bash: rustup install --profile minimal $RUSTUP_TOOLCHAIN The upside is we won't have to iterate the task in reaction to rustup changes like rust-lang/rustup#998. The downside is that it's two steps instead of one.
It looks like rust-lang/rustup#1257 is already fixed rust-lang/rustup#559. We should be able to install without a toolchain unless I'm missing something.
So I guess the question is the trade offs between a minimal RustupTool like you proposed or something more akin to my rustup step in this repo
Let's compare a clippy job with a minimal RustupTool
steps:
- task: RustupTool@1
- bash: rustup install $RUSTUP_TOOLCHAIN
- bash: rustup component add clippy
- bash: cargo clippy Now with a theoretical RustupTool based on my step template:
steps:
- task: RustupTool@1
inputs:
toolchain: ${{ rustup_toolchain }}
component: clippy
- bash: cargo clippyNote that for a build/test job vs a clippy job, instead of specifying component: clippy, we'd have target: ${{ rustup_target }} for those users desiring cross-compilation (generally used by CLIs)
Benefits for minimal RustupTool
- Briefer
- Less policy, so less to change over time
Benefits for the step template approach
- By default implements CI best practices skipped over in the minimal example
Dumping the cargo / rustc version to help with debuggingnm this is only relevant for CIs that support caching or implicit toolchain installs. This information is already visible in the output of the rustup installation and is not needed for Azure Pipelines.- Nicer
displayNames
So it me it looks like a bit of a toss up.
Hello guys,
Some news on my side!
I can't release the YAML version of the current extension because it creates breaking changes on legacy steps (Cargo, Rustup, Rustc). For people who are using the extension it means that they need to resetup all there pipelines. There aren't a lot of developers who install the extension, but I don't want to break something on there side. Also because I'm gonna release a new one which be only focus on tool installer, I will release the last version of the current extension as a legacy one this weekend.
After that's I'm gonna create the new extension based on you're feedback. My goal is to have a first implementation in order to create the feature request inside the azure-pipelines-task repository with an exemple (maybe they will be open for a PR).
I keep you in touch asap!
PS: Also I need your organization name to share the first beta version of the new extension with you!
@spontoreau thank you for your work on this! That sounds fully reasonable.
@epage version 1.2 of the legacy is released. Time to go further now :)
so ran into this after poking around on any options for a more robust rust solution on Azure Pipelines. Found this via the now archived azure rust devops task project, as i started wondering if it'd be worth building a custom task. we're definitely repeating ourselves a bunch in our CI config but also wanting some better integration.
One thing I notice is when rust code is built via normal script tasks, there's no bridge of warnings/errors into the Azure Pipelines UI which we notice happens when built in a VS project in a separate repo. So could be nice to process the cargo build/clippy output so it's discovered and surfaced to the Azure Pipelines UI.
My other hope is probably more work, but is having the option to take the warnings/errors etc from above and surfacing them into GitHub's Checks API. There's already a GitHub Actions plugin for rust clippy but haven't seen anything that'd help do that when using Azure.
So wondering if that'd warrant exploring doing a task as well, beyond the above discussion of the rust tool installer, but also an featureful build/clippy/test task extension? I may poke around in my free time if I can to see if I can at least figure out clippy to github checks.