Allow extensions to publish beta releases and users to opt-in to them

Question

Allow extensions to publish beta releases and users to opt-in to them

DanTup opened this issue 8 years ago · 153 comments

Apologies if this has come up before; I'm unable to find any related issues.

It would be cool to be able to publish extensions in a beta form (similar to Code Insiders) so that they can be run "in production" with real users that have opted-in. This would reduce the disk in publishing an extension because you can push it to a smaller number of users for testing (I probably don't need to sell this to you, I guess you have Insiders for very similar reasons).

I know we can package and distribute extensions ourselves for now, but it's not as easy (eg. updating is not automatic).

I originally thought the "Preview" flag might've provided this functionality but unless I've overlooked something, it does not seem to.

DanTup commented 7 years ago

😔

Answer 1 · 2016-11-20T20:15:46.000Z

I would very much like this myself.

Answer 2 · 2016-11-21T09:24:08.000Z

Why not putting those experimental features behind a configuration setting and have the opt-in mechanism be toggling that setting to true? Once the feature is stable enough just remove the setting.

Answer 3 · 2016-11-21T18:29:32.000Z

Why not putting those experimental features behind a configuration setting and have the opt-in mechanism be toggling that setting to true?

This is an option, but I think it has some drawbacks

If you want to do a beta release every day, your "stable" users will get the update notifications too
It doesn't eliminate all the risk (if you're shipping bugs in the new features, maybe your feature detection is buggy, or how you're dealing with two code paths)
For things that aren't just new features, it's more work to maintain/switch between the branches
Users have to keep opting-in to each feature, rather than choosing to be a beta tester once; this could really limit the testing you get
Having the feature toggles show up in the config auto-complete isn't ideal (this might not be an issue, but I didn't get a response to #1396 (comment) so it's unclear if "hidden options" are supposed to be used)

None of these issues are huge, but since the Insiders builds work so well and it seems like a similar scenario it seemed like a good request to raise for discussion :)

Answer 4 · 2016-11-23T06:34:07.000Z

I agree, this would be a really useful feature. Right now I'm distributing our beta releases via VSIX downloads, but that is cumbersome and I don't reach many folks that way.

Hiding things behind a feature flag wouldn't really help. For example, we are about to release a new version that changes a lot of code, the whole architecture of the extension. There is no way that we could have had that behind a feature flag.

Answer 5 · 2017-03-27T18:03:58.000Z

I would love to see this as well -- while feature flags work in certain cases, it doesn't cover them all. It also provides a sanity check and limits the "damage" of a bad release. And installing via vsix is just too user unfriendly to really be viable imo -- as it is completely manual.

Answer 6 · 2017-09-29T16:59:12.000Z

Just to kick this one again, because the issues surrounding the manual install and REQUIRED uninstall of vsix causes lots of pain for users and seemingly creates cases of corrupt installations -- I've definitely seen an up tick in GitLens issues caused by vsix/marketplace versions on top of one another.

Answer 7 · 2017-09-29T17:30:55.000Z

Yeah, I meant to post back here - when I did a beta release a few months ago I had reports of issues installing/uninstalling the vsix:

Answer 8 · 2017-12-10T10:12:45.000Z

I gave up on vsix testing because of the complications mentioned above. Yesterday I shipped a huge bug that only occurred if dart on your PATH was a symlink (I had to make a lot of changes for multi-root support, and putting it all behind flags was not feasible) :(

I'm starting to think about publishing a second version of my extension with Beta in the name to allow people to choose to test new releases earlier (seems like at least one other has done this).

Answer 9 · 2017-12-11T08:38:27.000Z

@DanTup @eamodio We did some improvements in this milestone (November) to support beta releases in VS Code when installed through VSIX. Refer to this issue #39024 for more details. Can you please check if that helps you in using VSIX?

Answer 10 · 2017-12-11T18:07:46.000Z

@sandy081 Sounds good! Am I right in thinking november milestone is 1.19 and not released to stable yet? I'll try and do some testing latest this/next week.

Answer 11 · 2017-12-12T08:09:02.000Z

Yes 1.19 will be released to stable this week. Let me know if you still want this feature to be opened after that.

Answer 12 · 2017-12-14T19:20:10.000Z

@sandy081 Haven't had chance to test it yet; but I think there's still value in this feature even if vsix works better than it did, for example:

I think end users would have more confidence installing a beta vsix through the marketplace then from some random download url
Allowing people to opt-in to a beta channel would allow them to get auto-updates of betas instead of having to do manual installations all the time
Listing betas on the marketplace is much more discoverable - only a fraction of my users will follow the twitter account (etc.), so there are a lot of potential testers that simply won't know about beta vsix releases

Answer 13 · 2017-12-15T08:11:55.000Z

@DanTup All your requirements are valid. But I think this has to be supported by the Market place first.

@modigrv May I know if there are any plans to have this support in Market place?

Answer 14 · 2017-12-22T09:05:27.000Z

This issue has been closed automatically because it needs more information and has not had recent activity. See also our issue reporting guidelines.

Happy Coding!

Answer 15 · 2017-12-26T05:30:06.000Z

Re-opening this issue as we haven't reached any conclusion yet.
ping @seanmcbreen and @modigrv on the question on marketplace

Answer 16 · 2018-01-06T03:02:09.000Z

Our C/C++ extension wants to be able to publish insider builds to the marketplace so users who opt-in can automatically get either the "insider" version or the release version after that's published -- mostly to increase our real-world test coverage before pushing changes to everyone. We can publish the insider vsix to our GitHub site and email users about it who sign up for it, but it requires user to manually install it, so that isn't likely to get enough usage. We're fine with all VS Code Insiders users also getting the Insider extension instead of having a separate insider setting just for our extension.

Answer 17 · 2018-01-08T11:57:52.000Z

I do not think we can do anything from VS Code side without such support from Market place.

Answer 18 · 2018-01-08T17:22:07.000Z

Where can folks give feedback that this is important? I thought leaving this issue open so that people can still like it might provide that option?

Answer 19 · 2018-01-08T17:59:22.000Z

We're fine with all VS Code Insiders users also getting the Insider extension instead of having a separate insider setting just for our extension.

I don't think it'd be great to tie these together since it'd mean using the insiders version of your extension means that the user needs to be happy to switch also to Insiders Code and with it any other extensions that are publishing Insiders versions. I wouldn't want to go "full unstable" just to do it for one extension (for one, the stability of Insiders extensions could vary wildly across projects).

It also changes the environment somewhat, so you're not getting an accurate test of what happens if you publish the extension (unless ofc you wait for the next stable Code release).

I do not think we can do anything from VS Code side without such support from Market place.

Where can we show support for this? If this case stays closed, I suspect this will be forgotten about but it seems to be a good request desired by several extensions.

Answer 20 · 2018-01-08T20:37:08.000Z

Yeah, having extension "insider-ness" be independent of VS Code itself is fine with me too. We just need the marketplace to allow us to submit an insider version and then VS Code would need to add an "Install Insider" button next to extensions that also have an insider version. Does the marketplace have some feedback site where this is being tracked?

Answer 21 · 2018-01-08T20:41:32.000Z

They could even just re-use the preview flag -- I would bet users don't know what that is even supposed to mean (I don't -- other than some signal with regard to quality/stability/features -- but very nebulous and I bet doesn't stop bad reviews).

Answer 22 · 2018-01-08T20:46:52.000Z

I guess the Marketplace would still need explicit support for this since people who don't want the preview/insiders/pre-release versions would expect to browser the marketplace and see the release notes (etc.) for the latest stable release, not the preview release.

Answer 23 · 2018-01-08T20:53:30.000Z

Yeah, agreed -- there would definitely be a bunch of plumbing that they'd need to do on the marketplace side.

Answer 24 · 2018-01-09T14:04:05.000Z

@viradhamMS @pkuma-msft We have a popular request to support beta/insiders versions of extensions which in turn needs Market place support.

May I know if there are any plans to have this support in Market place and what is the right Market place forum to raise these requests?

Reopening on users demand :)

Answer 25 · 2018-01-11T06:42:32.000Z

Thanks folks. We do have a backlog feature to enable similar requirement.
When we get to it, we will reach out to you folks to understand the requirements to finer detail.

For such feedback/questions on Marketplace, Please send a mail to vsmarketplace @ microsoft.com

Answer 26 · 2018-02-09T03:33:35.000Z

It seems like extensions themselves can implement their own insiders channel without marketplace support for users who have some "yourExtension.insiders" setting set:

Downloading some json file that has the current insiders version (and URL) of the extension on activation (we already do this for on the fly a/b testing).
If there's a new insider version, the extension can automatically download/install the vsix.
And if you want more extension insider usage, you could set the default setting to true for VS Code insiders builds (requiring a package.json rewrite), and users could change the setting to false if they want the release version. This would hopefully solve our current problem in that our manually installed vsix has way too small 0.03% installs compared to the released version. It's on our backlog: microsoft/vscode-cpptools#1526 .

Theoretically, this mechanism could be implemented in a way that is sharable between extensions via something like:

Adding a setting extensions.insiders: [ "ext1": true, "ext2": false].
Adding a property to package.json that sets the URL to download that points to a json file with the current insiders version and URL. We actually have 4 additional "offline" installer URLs for different platforms that wouldn't fit into this design, but we could ship just the online installer via this mechanism.

But I don't know how many other extensions would want to take advantage of this.

Answer 27 · 2018-02-15T21:51:17.000Z

@sean-mcmanus that sounds very interesting. Personally I would like for vscode to somewhat "approve" a mechanism like this before I'd employ it in my extensions.

Answer 28 · 2018-02-22T20:06:23.000Z

While I was wanting this feature too, I have to say, the improved support for installing from .vsix files (without fear of conflicts) in #39024 is working nicely for me. Most people who have an issue with my extension are only interested in that one thing anyway. So I can attempt a fix, upload a beta .vsix file though github releases, and point them there. If everything's ok, I tell them to stick with that and then with the next marketplace release they're updated automatically and back on stable.

Now of course... it work for all cases; large extensions that want a permanent beta channel. Just wanted to share for anyone that might have missed that change.

Answer 29 · 2018-02-22T20:11:00.000Z

The vsix fix is good; but I've had several beta releases in the last week or so. Telling users to keep updating vsix's kinda sucks. Having auto-updates of betas would make is so much easier - I wouldn't have to convince people to go download another vsix and install it :)

Answer 30 · 2018-06-25T23:04:13.000Z

Hey everyone! I work on the PowerShell extension and would LOVE to have a preview/beta channel. In the meantime, I'm probably going to go down the route of having a separate extension similar to what tslint does.

Shoutout to @mattbierner for pointing me here 👍

Answer 31 · 2018-09-28T01:01:52.000Z

FYI, the 0.19.0 C/C++ extension has implemented a C_Cpp.updateChannel setting with an "Insiders" option that will automatically check our GitHub releases page for a newer Insiders .vsix and download/install that: see https://github.com/Microsoft/vscode-cpptools/tree/release/Extension/src . Users of the Insiders VS Code have that setting defaulted to "Insiders".

Answer 32 · 2018-09-28T01:34:59.000Z

@sean-mcmanus That's very cool -- though I would still really like this to be built-in/provided functionality.

Answer 33 · 2018-11-20T19:12:10.000Z

Couldn't the preview tag in a published extension manifest be used for this? If a release has the preview property as true, don't install it, unless a user opts into the "preview" channel for that extension. It seems to me that this wouldn't require any changes on the marketplace but could be implemented in VS Code itself?

Answer 34 · 2018-11-21T01:57:21.000Z

To add to that, Ideally we'd want to be able to grab the latest non-preview if we want the stable channel, or the latest preview if we want the preview channel.

Answer 35 · 2018-11-21T20:50:52.000Z

@davidanthoff "Preview" is currently being used by some extensions for their "release" versions that are not considered "feature complete" enough for 1.0, so you'd be changing the meaning of the "preview" field. It would be better to check the version to see if it contains an "-insiders" which is what our extension uses to implement our "C_Cpp.updateChannel": "Insiders".

Answer 36 · 2018-11-21T23:08:00.000Z

I see. In any case, I don’t understand why this can’t just be implemented in VS Code itself, why is there a need for a marketplace update?

Answer 37 · 2018-11-21T23:50:34.000Z

VS Code's extension API doesn't have a generic concept of "<extension-setting-prefix>.updateChannel", so they would have to add some package.json setting for extensions to provide what setting should be queried for that (or some other equivalent mechanism). If that is done, then they could use an implementation similar to what our C/C++ extension does to bypass the marketplace and query GitHub for new Insider vsix's.

Answer 38 · 2018-11-22T01:26:20.000Z

Couldn't VS Code simply store somewhere in its own settings which extension is on a prerelease channel, and then check the marketplace whether there is an extension that has a version number that qualifies it as a prerelease?

Answer 39 · 2018-11-22T01:36:25.000Z

@davidanthoff How would VS Code determine if an extension is on a prerelease channel? There needs to be some API or package.json for setting that (e.g. an "updateChannel" or "updateChannelSettingName" field in package.json). The marketplace only allows submission/publishing of one version at a time -- we can't submit a 0.20.1 and 0.21.0-insiders without changes to the marketplace implementation (only the last submitted would be used).

Answer 40 · 2018-11-24T01:41:54.000Z

But aren't all versions that have ever been published stored on the marketplace? I just tried this with our julia package, and I can get any old version with a link like

https://marketplace.visualstudio.com/_apis/public/gallery/publishers/julialang/vsextensions/language-julia/0.8.0/vspackage

I guess the problem is that the extension can't get a list of all the uploaded extension versions?

Answer 41 · 2018-11-26T22:19:36.000Z

@davidanthoff Yeah, I think you're right about all the older versions being saved (we used to tell users to use that when there was some breaking regression), but VS Code would need a way to query the marketplace for the list (or the latest Insiders versus non-Insiders). It's possible such an web API exists, but I haven't checked. Our extension queries https://api.github.com/repos/Microsoft/vscode-cpptools/releases for the version list.

Answer 42 · 2018-11-27T01:47:01.000Z

That API you linked has whether or not a version is a prerelease or not. Could we go down the list for the last non-prerelease?

Also couldn't help but notice cpptools is following semver so that could also be leveraged, potentially.

Answer 43 · 2018-11-27T01:58:01.000Z

@TylerLeonhardt Do you mean the GitHub API? That only works for our extension because we post our Insider .vsix's on GitHub and we have custom code that queries it and updates our extension with that. VS Code itself could do something similar to give other extensions the ability to have an Insider channel -- as mentioned previously, there just needs to be some mechanism for exposing the per-extension setting to turn the Insider mode on or off. FYI, our extension has gotten thousands of users on our Insiders, so we consider it a success (compared to tens of users previously, with the "manually discover/download/install the vsix" approach).

Answer 44 · 2018-11-28T11:22:38.000Z

Oof my bad. Got confused on who is who in this thread and totally thought the API you linked was from the marketplace 😄

if the marketplace has an API similar to GitHub's... that could be leveraged in vscode to achieve the NuGet Package Manager-like experience of installing extensions - checkbox to allow prerelease/insiders/previews... which filters the list.

Answer 45 · 2018-12-17T04:24:53.000Z

This seems an API that can pull all versions that are on the marketplace. I think with that all the pieces should be in place that this issue here could be implemented without any changes to the marketplace:

VS Code stores per extension whether it should get pre-release versions for that version or not and exposes a UI for that setting
If an extension is configured for pre-release, it pulls the info about all available versions from the marketplace via the API I linked to, and then gets the newest release that has say beta or something like that in the semver.
If an extension is not configured for prereleases, it just installs the latest "normal" version, based on the version property.

Essentially I think one could implement this in the same spirit that #12764 was implemented, and distinguish between release and prerelease versions on the version property. CCing @sandy081, given that he did the arbitrary version installation stuff.

Answer 46 · 2018-12-17T19:50:22.000Z

@davidanthoff I'm not aware of a per-extension setting in VS Code -- maybe the "setting" could just be exposed via the right-click menu in the Extensions pane, like the "Install Another Version..." option is. Also, in order for us to switch to this VS Code implementation, the per-extension setting would need to default to "Insiders" for users of VSCode Insiders, because that's how we get most of our current Insider usage (and enable people who don't want to be on Insiders to change the default).

Answer 47 · 2019-01-23T23:42:19.000Z

Implemented a workaround for this by shipping two extensions...

Preview
https://marketplace.visualstudio.com/items?itemName=ms-vscode.PowerShell-Preview

Stable
https://marketplace.visualstudio.com/items?itemName=ms-vscode.PowerShell

I ended up running into a strange issue trying to release a 2.0.0-preview.1 version of the Preview package:

No support for SemVer 2! :(

So I tried to use 2.0.0.0 instead and npm doesn't support 4 numbers... only SemVer 2 format!

It's been a wild ride... I really hope the code team can do something about this.

Right now extensions can only be X.X.X versioned.

cc @auchenberg who I was chatting a bit about this on Twitter

Answer 48 · 2019-01-29T12:16:00.000Z

In some ways I like that I can't accidentally publish a dev release. I'm really nervous when I type vsce package that I might push a dev build to all users!

I wonder if we could still have a prefix that was vblocked (eg. -dev) even it became possible to upload other pre-release versions (in order to support this). There would need to be some work in VS Code and on the site to let users see/pick between stable/preview releases, but being able to see versions in VS Code now seems like a step towards this.

Answer 49 · 2019-02-15T23:55:19.000Z

Out of curiosity, where does one file issues against the Marketplace? It sounds like we need to raise awareness to the folks who maintain the Marketplace as well as the vscode team (who are obviously already on this thread.)

Answer 50 · 2019-02-18T08:31:06.000Z

@beriberikix - You can file issues here itself, we work very closely with VSCode group and it will reach us pretty quickly (or) feel free to directly post to vsmarketplace at microsoft dot com

Answer 51 · 2019-04-11T18:13:58.000Z

A feature like this would help out immensely for pushing new features. I'm hoping the marketplace/vscode team will get on it soon!

Answer 52 · 2019-06-12T08:26:56.000Z

Currently most of the extensions are following a separate extension for extension insiders/nightly. This way, users can opt in to install either stable/insiders version of extension and disable the other. It is also NOT limited to insiders/stable.

Any feedback on this solution?

Answer 53 · 2019-06-12T08:52:49.000Z

I don't think it's the best solution - it means you're listed twice in the marketplace, have two sets of reviews/stats, and have to fiddle around when publishing to change the extension ID.

It also means extensions that reference each other by ID (for example Dart & Flutter are dependent and call exported APIs) need some faffing around too (to know to check which version is installed).

And of course, we'd need to do our own checks that the user didn't install multiple versions.

I think being able to publish pre-release versions to the marketplace (even if it's a white-listed set, like -beta) and let users have a flag per-extension to opt in the using pre-release versions would be far better (and I don't think it should be tied to VS Code Insiders).

Answer 54 · 2019-06-12T17:34:47.000Z

Your concerns are valid.

Another approaches we are discussing which is based on what is suggested here

Approach 2: Defining Insiders Flag (VS Code hosted solution)

Introduce an insiders flag for extensions which can be provided in the package.json and follow the same versioning system major.minor.patch.
Provides a way (some UI) to the user to install the stable or opt in for insiders in VS Code. User can always switch between stable and insiders flavour of extension even after installing.

Drawbacks

Following the same versioning system will be confusing to use between insiders and stable. How does an insiders version is numbered without pre-release version tags?
Since insiders version will be always latest, Markeplace shows insiders version of an extension like read me, version etc.,

Approach 3: Insiders version tag (Marketplace hosted solution)

Allow pre-release version tags x.x.x-insiders for extensions.
While browsing (VS Code or Marketplace) always show stable version by default unless requested to see insiders version.
Provides a way (some UI) to the user to install the stable or opt in for insiders in VS Code. User can always switch between stable and insiders flavour of extension even after installing.

I would prefer Approach 3 which is correct IMO and has no drawbacks compared to other solutions. But this has a dependency on Marketplace to support this feature.

Answer 55 · 2019-06-12T17:44:45.000Z

Yes, I'm also much in favor of approach 3!

I think it would be nice if the pre-release tags would support things like v1.4.3-alpha.2, v1.4.3-beta.3, v1.4.3-rc.1 etc. and then would automatically do the right thing. If that is too complicated, maybe just v1.4.3-insiders.3 would do the trick as well. But it would be good if one could publish consecutive pre-release versions that are uniquely identified.

Answer 56 · 2019-06-12T17:57:37.000Z

I also prefer 3, since it seems weird if we have to keep revving non-preview version numbers for preview/insiders builds (however if it's likely to take significantly longer because it needs marketplace changes, I'd revise my preference ;-))

I'd also ask that -insiders builds can have additional build numbers (eg. -insiders.1, -insiders.2, -insiders.3 etc.).

Answer 57 · 2019-06-12T18:55:16.000Z

I also strongly prefer 3

Answer 58 · 2019-06-12T19:52:14.000Z

As mentioned previously, our C/C++ extension implements this ourselves with our C_Cpp.updateChannel settings, so if the marketplace implements something similar it is important that our extension is able to:

Default to Insiders for VS Code Insiders.
Be able to show a prompt such as "Insiders version is available. Would you like to switch to the Insiders channel?: [Yes][Ask Me Later][Don't Show Again]" that can change the version.

Without those 2 features our Insider usage would be too low to be useful. Our code that does this is open source, but it's not packaged to be re-usable by other extensions.

As mentioned by the other users, some sort of 0.23.0-insiders, 0.23.0-insiders2 mechanism is desired: see https://github.com/microsoft/vscode-cpptools/releases .

The main disadvantage with our method is that our Insider versions are not available via the "Install Another Version..." UI in VS Code...so if an API was available to make that fixable I'm not sure why we'd benefit from a marketplace change.

Answer 59 · 2019-06-13T06:58:26.000Z

Without knowing the true architecture of the VSCode marketplace, I can only guess... But since vsix's are very similar to nuget packages, my guess is that the marketplace (VSCode, VisualStudio, etc) is some fork of NuGet.org (or NuGet.org is some fork of the marketplace). Also, the PowerShell Gallery is a fork of NuGet.org.

Both NuGet.org and PowerShell took approach 2. My guess is because of the risk of doing approach 3 and making sure that we are still backcompat. Imagine trying to replace a bunch of SemVer v1 code with SemVer v2... There could be a ton of side effects.

With all that said, I would prefer approach 3 because it's the technically correct approach... But my understanding is that that's a different team and this would be a large ask so it's tough on that front.

Im voting for Approach 2 only because I think a lot of Approach 2 is needed for Approach 3 (like the toggle in vscode to show/hide insiders) and it will take the least amount of time and coordination.

All the while, the marketplace team could be working on Approach 3... And then vscode uses that version instead of the preview flag when their work is done.

Answer 60 · 2019-06-13T10:56:57.000Z

I think with Approach 2 extension authors will have troubles in version their stable and insiders versions. I really do not know what the model would be for versioning stable and insiders with major.minor.patch. For example if the current stable is 1.1.0 what is the next insiders version?

Answer 61 · 2019-06-13T11:28:56.000Z

If you went with 2, I'd probably just use even minor versions for stable releases and odds for insiders to keep things in order:

1.0.0 <-- stable
1.1.1 <-- insiders version of 1.2
1.1.2 <-- insiders version of 1.2
1.2.0 <-- stable
1.3.1 <-- insiders
1.4.0 <-- stable

It's not perfect since you can't have insiders versions of patch releases, but that doesn't seem like a huge issue IMO.

(I'm not saying I'd like 2, but if the difference between delivering that and 3 is a long time - it's been over 2 years since I opened this already - then I'd take it!)

Answer 62 · 2019-06-13T18:29:38.000Z

I agree with @DanTup. That seems reasonable if it means we get the feature a lot sooner.

With the PowerShell extension, we recently followed the Python extension and switched to date-based versioning like 2019.8.0 where it's year.month.buildThatMonth. Leveraging that versioning scheme, with the preview property in the package.json determing prerelease (aka Approach 2) would work nicely for us.

P.S. We switched to that versioning scheme because we accidently shipped the preview extension (2.0.0) to the stable extension... It was a very painful experience to mitigate. We couldn't risk using Semver anymore having 2 extensions to manage.

Answer 63 · 2019-06-13T18:38:50.000Z

If we went with 3), one could have a concept of a "channel': one could have a nightly channel, a beta channel (that is more stable and has fewer releases) etc.

The versioning scheme could be v1.20.3-nightly.4 and v1.20.3-beta.2 etc.

At the end of the day, I think that is what I'd really like to have and it seems to be really useful in lots of other contexts (Windows, Edge etc.)

Answer 64 · 2019-07-24T09:30:18.000Z

This would a great feature addition! Any timeframe on when this might land?

Answer 65 · 2019-11-30T17:24:25.000Z

I seem to be getting different versions depending on whether I am on VSCode Insiders

Name: C/C++
Id: ms-vscode.cpptools
Description: C/C++ IntelliSense, debugging, and code browsing.
Version: 0.26.2-insiders2
Publisher: Microsoft
VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools

Name: C/C++
Id: ms-vscode.cpptools
Description: C/C++ IntelliSense, debugging, and code browsing.
Version: 0.26.1
Publisher: Microsoft
VS Marketplace Link: https://marketplace.visualstudio.com/items?itemName=ms-vscode.cpptools

But it looks like cpptools manually implemented that feature...
microsoft/vscode-cpptools#3221

Answer 66 · 2019-12-02T21:22:21.000Z

@vchuravy Yes, and that's the default behavior, but it can be overridden via our C_Cpp.updateChannel setting. There's a bug preventing it from working for remote machines though (microsoft/vscode-cpptools#3833).

Answer 67 · 2019-12-02T21:29:37.000Z

It would be fantastic if extensions wouldn't have to implement this feature, but VSCode could offer it by default.

Answer 68 · 2019-12-02T22:17:39.000Z

@vchuravy Yeah, I agree. Our implementation is open source if any other extension wants to do something similar, but we haven't packaged it into an easily re-usable npm module. Other than the remote bug, our insider system has worked really good for our extension, allowing us to find/fix important bugs before they get released to a wider audience (and allowing users to get updates earlier than our slower 1.5 month non-Insiders release cadence).

Answer 69 · 2019-12-17T21:45:31.000Z

Is there any hope that this will make it to the top of the list of the core dev team? I think given that this needs integration with the marketplace, it really can only be done by the core team.

Every time we release a new version of our extension, I wish we had this. In my mind this is single feature that has the most amazing potential to increase the quality of the entire extension ecosystem.

Answer 70 · 2020-02-26T16:16:31.000Z

I would love this

Answer 71 · 2020-02-28T04:23:21.000Z

We now caved and are publishing a second version of the Julia extension that is an insider build. The whole situation is really not great because the normal and the insider extension now conflict with each other, but it is better than nothing.

If anyone is interested in the Azure pipeline we use that creates the insider build, take a look here. It makes the necessary modifications to the package.json and swaps out the README.

I still really hope that at some point we'll get a proper solution for this...

Answer 72 · 2020-02-29T12:29:05.000Z

From reddit: https://www.reddit.com/r/vscode/comments/fakre5/hey_extension_authors/fj0r2az/

I can speak to beta extensions. This is one of the most requested features for the marketplace and we're actively working on this one.

Answer 73 · 2020-07-06T15:06:56.000Z

Having 2 extensions - the regular and the insider one raise a problem so the user can't have both of them. I've not digging into it but probably the issues are with duplicate ids of commands, views etc.
In principal, we can run a task to replace the ids with insider ids but it's very uncomfortable.

Answer 74 · 2020-07-06T15:13:09.000Z

@moshfeu I'm eagerly awaiting beta releases as a first-class citizen, but as a workaround for your issue, you should be able to have the regular version look for the insider version (with this api) and disable itself if the latter is installed.

Answer 75 · 2020-07-08T08:57:33.000Z

Thanks @J-Fields . The conflicts coming from the "contributors" - for example, buttons and commands. I'm not sure how using the API can solves this 🤔

Answer 76 · 2020-10-16T14:24:25.000Z

This would prevent a lot of headaches to extension users :)
I'd prefer the possibility for the user to easily rollback to the latest stable version if he accidentally opted-in ;)

Answer 77 · 2021-07-21T21:39:33.000Z

It would be cool like Google Play Store So the entire journey from alpha to production would be covered here.

Answer 78 · 2021-08-26T10:41:42.000Z

Upstream microsoft/vsmarketplace#165

Answer 79 · 2021-09-10T08:27:21.000Z

@sandy081 and me have been thinking about this topic and we understand that there are a lot of problems with the current approach of publishing two extensions: confusing for users, inconsistent across extensions, painful to manage both. We believe that the current separation of insiders and regular into two extensions does not make sense. Here's our proposed solution:

Continue to use current major.minor.patch version semantics but provide a way to extension authors to mark a version as insiders by introducing a flag say --insiders to vsce during the publish step.
Given that MP supports publishing smaller/older versions (given that the version is not published already), Extension authors can still release insiders and stable (including patches) in parallel. One recommendation is to use patch for insiders versions but extension authors are free to use any numbering model given that they use only major.minor.patch semantics. Example:
- Assume that you have a stable released version 1.0.0 and want to release next insiders version.
- You can release next minor insiders versions as follows: 1.1.0, 1.1.1, ....... , 1.1.25 (marking them using `--insiders flag through vsce)
- You can promote above insiders version to stable version by releasing 1.1.26 and start working on next insiders versions 1.2.0... just like above.
- If you want to fix a bug in previous stable ie., 1.1.*, you can fix the bug and release 1.1.27.
When users uses either VS Code Stable or VS Code Insiders to install an extension, the user would get the latest stable version of the extension. This is an open question: should VS Code insiders users get the insiders extension automatically?
Users could manually switch from a Stable extension version to an Insiders (context menu action). Extensions will still auto-update respecting their "quality" - insiders extensions would update to latest insiders, stable only to stable extension versions
Insiders extensions will be decorated specially in the extensions view
It is up to extension authors to deprecate their old separate "Insiders" version of the extension and all settings which might control this (C++ and Python). Either by using notifications or something else. We are willing to help with this transition.
Long term 18 months (upstream dependency): MarketPlace will add support for pre-release versions (e.g. 1.60.2-insiders). Once that is done we might deprecate the --insiders.

Drawbacks of this approach

Version ambiguity: Since one cannot use prerelease part of semver and have to rely on major.minor.patch parts for insiders, this creates strange version history and cannot differentiate insiders vs stable by looking at versions.
Marketplace will always display the latest version of the extension on its website (even if it is marked as insiders version).
Current users of Insiders extensions would have to uninstall the Insider extension, and install the regular extension. And potentially opt in to the insiders version. Might be hassle. Probably we can mitigate this by VS Code handling this.

Above drawbacks will be lifted once MP supports prerelease version tags.

Please let us know what you think about this proposal. We are looking forward to the feedback. Thanks 👏

Authors of extensions that have nightly versions @eamodio @connor4312 @alexr00 @aeschli @brettcannon @davidanthoff @DanTup @WardenGnaw @polinasok @hediet @greazer @TylerLeonhardt

Answer 80 · 2021-09-10T08:43:00.000Z

@isidern I like the proposed solution!

Would vscode-stable users who installed an insider-extension also be updated to stable-builds of that extension if the stable build is newer than the insider build? (I think this makes sense if the extension does not always have an insiders build)

In this case, insider-users who got updated to stable should still get the newest insider build of the extension once it gets published. Thus I think the decision what to update to should not be based on the stability of the installed extension, but rather be a separate (user-configurable) flag.

Answer 81 · 2021-09-10T08:48:43.000Z

@hediet

Thats a good point to note and Yes, we will be doing that - Irrespective of being on insiders/stable version of an extension users we will always have the best latest ie.,

if stable is latest than insiders then always stable.
if insiders is latest than stable then insiders (only when in VS Code insiders or users opted to insiders version of extension)

Answer 82 · 2021-09-10T09:03:32.000Z

@rchiodo Mentioned using the "engines" property to limit a release to VS Code Insiders, not sure everyone is aware of that option:

    "engines": {
        "vscode": "1.59.0-insider"
    },

Answer 83 · 2021-09-10T09:07:37.000Z

@rchiodo Mentioned using the "engines" property to limit a release to VS Code Insiders, not sure everyone is aware of that option:
    "engines": {
        "vscode": "1.59.0-insider"
    },

But 1.60.0 is newer than 1.59.0-insider, right? At least by the semver spec.

Answer 84 · 2021-09-10T09:20:55.000Z

Updated the proposal based on feedback (added drawbacks) and simplified that Stable Extensions would get downloaded for any VS Code flavour and users would have to opt into getting insiders extensions. We can later change this that by default VS Code insiders gets Insiders extensions. This is out of scope for this proposal.

Answer 85 · 2021-09-10T17:07:51.000Z

What is the plan for user discoverability when it comes to this? With a separate nightly extension you get the SEO for your insiders version by showing up as a separate search result. Bundling under the same extension ID means you lose that. So would users then be expected to check every extension to see if there's an insiders version they can switch to? Or would extensions come up with their own way to "advertise" their insiders version to users and ask them to click the right thing to switch? Or would there be an execCommand we could use as part of a notification prompt to help users flip over to insiders?

Answer 86 · 2021-09-10T17:18:00.000Z

The proposed plan would work well for Pylance; I designed our versioning to follow semver (technically, calver) with this eventuality in mind. Our stable releases are like 2021.9.1, and our insiders are like 2021.9.2-pre.1 (insiders builds are always semantically newer than stable builds).

I was worried about the case where insiders users update to stable builds, when we want them to move up to other insiders builds, but I this already happens now as VS Code can update the extension to stable before we get a chance to manually install the VSIX. When a stable build of Pylance is release, we release a corresponding insiders build, so it's probably rare that anyone checks for an update during that small period and would work better than our manual daily checks.

Answer 87 · 2021-09-10T17:20:24.000Z

I was hoping the default for VS code insiders would be to install the latest insiders (or stable if newer build)

Then it would happen automatically. The special ask the user if they want to opt in means nobody will end up using our insiders bits.

Right now we're using the engines property to accomplish just that. All VS code insiders users get our insiders version automatically (or stable version if it's newer).

Answer 88 · 2021-09-10T17:31:16.000Z

@isidorn VS Code Insiders defaulting to the latest Insiders extension (or stable if it's newer) is also a "requirement" for our extension to switch to this mechanism (instead of our current custom one) -- otherwise, Insider usage would be "too low" (we could probably provide telemetry data on this if you want).

Answer 89 · 2021-09-10T17:40:20.000Z

Similar for Pylance; our custom insiders system also respects the Python extension's insiders setting, which gets us a more reasonable number of users. It's hard to make this discoverable.

Answer 90 · 2021-09-10T18:48:47.000Z

I think we should not talk about "newer" or "latest" builds, but higher versioned builds.

Irrespective of being on insiders/stable version of an extension users we will always have the best latest ie., if stable is latest than insiders then always stable.

This is not a good behavior in my opinion. For example, often in recovery releases I'll go back and publish, for example, a 1.60.2 release even when the 1.61.0-insiders.1234 is available. In this case users on 1.60.1 should be updated, but those on the 1.61.0 nightly should not be. That is:

For users on the "insiders" track, always install the highest-versioned build available compatible with engines.vscode
For other users, install the highest-versioned non-prerelease build available compatible with engines.vscode

Aside from that I think the proposal looks good.

Answer 91 · 2021-09-10T18:56:18.000Z

@connor4312 I believe all references to "newer" and "latest" do mean "higher versioned" builds, and the release of 1.60.2 would not affect the 1.61.0-insiders if it's released later, i.e. it's just a word choice difference.

Answer 92 · 2021-09-10T19:15:48.000Z

Yea, just want to clarify that there should not be any chronological implication when we talk about version numbers. I thought there was currently a bug in the marketplace where that was assumed, but it looks like it's actually a separate issue (extension auto update only checks for the highest versioned release of an extension, but if its engines.vscode is not compatible with the current vs code version the auto update stops working; I misremembered and thought the problem was that it only returned the most recently published version)

Answer 93 · 2021-09-11T00:45:18.000Z

if insider avaible, IMO, I want this:
before installed:

after installed:

Answer 94 · 2021-09-11T00:48:26.000Z

I think we should not talk about "newer" or "latest" builds, but higher versioned builds.

I'd like to add, the current behavior works very well for the Jupyter extension. We have releases for both stable and insider builds, and we ship recovery builds for stable every now and then, and insider builds on a daily basis

For Jupyter extension, if used wants insider builds, we just ask then to use the vscode insider builds (ie all vscode insider users automatically get the insider build of Jupyter extension). We haven't had any request from users to have insider builds of the extension in the stable version of vscode.

Answer 95 · 2021-09-13T09:21:48.000Z

If a version has been published as insiders, it can not be promoted to stable, a new version has to be pushed. Can create strange version history for extensions

To clarify - does this mean version numbers? Eg. If I publish a 1.25.0 with --insiders, I cannot then also publish 1.25.0 without --insiders? Or does it just mean that we have to push the file again and can't just flip a flag? I'm assuming it's likely the former if the Marketplace isn't handling this and VS Code is instead tracking which versions were marked as insiders? (out of curiosity - how is this working is vsce just adding some additional metadata to the vsix?).

I think having the marketplace show the Insiders version number might be confusing, but it sounds like that's unavoidable without the Marketplace being updated. I'd certainly live with it to have Insiders versions :-)

I agree with other users though - having Insiders users automatically get Insiders by default would be nice to improve usage (or at least make the install view have the Insiders option more prominent).

Answer 96 · 2021-09-13T09:42:07.000Z

@brettcannon Your discoverability concerns are valid and we will definitely address this while coming up with UX mock ups. Our current thinking is to let VS Code handle this and not by extensions, so that users will get unified consistent experience.

@connor4312 Just to be clear about current proposal - We are not changing current version semantics ie., extension still supports only major.minor.patch. We defer supporting prerelease version tag until Marketplace supports it. Your scenario of publishing 1.60.1 when there is 1.61.0-insiders is supported only then. With the current proposal you still have to follow existing major.minor.patch version semantics but you can tell VSCE / VSCode that a specific version is a an insiders/prerelease one. Unfortunately you have to release a newer/higher version for your insiders, if you have a stable release. So in your case, you have to publish 1.60.1 as stable and 1.60.2 as insiders. This is mentioned as a drawback to the proposal and I agree that it does not sound good and adds some strange version gaps but this temporary and will be lifted once MP supports prerelease tags.

To clarify - does this mean version numbers? Eg. If I publish a 1.25.0 with --insiders, I cannot then also publish 1.25.0 without --insiders? Or does it just mean that we have to push the file again and can't just flip a flag? I'm assuming it's likely the former if the Marketplace isn't handling this and VS Code is instead tracking which versions were marked as insiders? (out of curiosity - how is this working is vsce just adding some additional metadata to the vsix?).

@DanTup your assumption is correct. Yes, with current approach VSCE/VSCode are just tracking the versions marked as insiders. Once MP supports prerelease versions we will move to it. Yes, we will be adding this as metadata to vsix for now.

Answer 97 · 2021-09-13T15:51:40.000Z

What would the MP link to for downloading? The Python extension gets sideloaded a lot due to aggressive firewalls blocking VS Code hitting the MP for updates, so having the download link in the MP be the stable release would be good. Making the link just the highest versioned thing, insiders or not, can cause issues for extensions like Jupyter where users don't realize that the thing they are downloading from the MP is actually the insiders build and not the stable build.

Answer 98 · 2021-09-13T16:32:01.000Z

@brettcannon my understanding is that there will be no changes to the marketplace in the short-term for the proposal above, so it will just show the highest version number regardless of stable/insiders (because it won't know which are stable and which are insiders - only vsce and VS Code will understand that).

It's definitely not perfect, but if it's really likely to be 18 months for the Marketplace to get support, I'd rather have this. It's been almost 5 years since I opened this issue 🙃

Answer 99 · 2021-09-13T17:54:28.000Z

Before I created a separate Insiders extension for GitLens (and the same for GitHub Repositories) I think I would have (for the most part) loved this proposal, but now I feel there are significant issues with using the same extension id for both a stable and unstable build, at least in how VS Code handles data for them.

Having shared data storage for stable and unstable extensions, can be hugely problematic, not only for backward compatibility, but now you will have to deal with users switching back and forth between versions -- and with synced data this can get even more problematic. Granted extensions internally could try to deal with this, by storing their own data in separate stable vs unstable buckets, but that adds even more complexity (and must be understood before you hit the problem).

VS Code could, of course, store stable vs unstable data in separate buckets automatically and that might address most of my concerns.

Outside of the data concerns, I also feel like versioning will be challenging -- not only just getting the right thing published, but also for error reporting (since the main parts of the version are going to be either the same of very close). I definitely like how today both GitLens & GitHub Repositories use standard versioning 1.5.0 while the insiders version are YYYY.MM.DDHHMM. Which make it easy to tell the difference, but also how recent your insiders build it (since they are basically nightly builds). Maybe the --insiders or maybe better yet a --prerelease-channel <name> flag could not only specify the channel, but also affect the versioning structure -- so maybe all prereleases are versioned by the date.

Also if we think of this concept all as pre-release "channels" where the user can choose the channel they want to be in, it will be something easier to manage for authors, users, and the marketplace. On the marketplace, you could then show version histories by channel, download/upgrade by channel, etc. And maybe the number of channels and names are publisher controlled -- or maybe they are something we try to standardize on. And maybe vscode insiders get, by default, opt-ed into a pre-release channel, which they can opt-out of individually or globally. But in this way we aren't just thinking about version numbers -- version numbers are a secondary consideration after the channel.

Anyway that my 2c.