nvaccess/nvda

Ability to automatically update add-ons

nvaccessAuto opened this issue · 64 comments

nvaccessAuto commented

Reported by nvdakor on 2013-05-08 07:08
Hi,
With the launch of the official NvDA add-ons page at addons.nvda-project.org, it became easier to search add-ons from one central location. Although this helps with downloading add-ons for the first time, a user who is running an add-on may find that he or she is running an older version of an add-on. So just like automatic NVDA update feature, there could be a feature where newer versions of add-ons would be downloaded automatically and prompt the user to install it without having to visit the addons site. Thanks.
Blocking #4762

nvaccessAuto commented

Comment 1 by blindbhavya on 2014-09-07 08:27
This proposal would be very useful if implemented.
Could you (the devs James Sir) share your thoughts on this?

nvaccessAuto commented

Comment 3 by bhavyashah on 2015-01-01 07:44
Hi,
Please add #4762 to 'blocking'.

nvaccessAuto commented

Comment 5 by fastfinge on 2015-02-19 16:17
As far as I know, we haven't yet had security issues with an addon. But I'm wondering what the security implications of people running out-of-date addons are? Especially as NVDA needs administrator access. This is another reason why, personally, I think having the ability to check for addon updates from within NVDA is kind of important. Even if the addon itself could specify a URL to check for updates, rather than forcing everything through the centralized NVDA addon website, that might be enough to get people updating old addons when, not if, a serious security hole in an addon is discovered. In it's simplest form, it could be a file with a version number, download URL, and file hash. I don't think addons are currently signed via any kind of private key system at the moment, and I'm not sure they urgently need to be. Maybe this isn't a big deal, though, if addons are sandboxed in some way I don't know about?

nvaccessAuto commented

Comment 6 by jteh on 2015-02-20 01:17
First and most important, NVDA does not require administrator access. You should absolutely not be running NVDA with administrator privileges. You also should not disable UAC.

For what it's worth, no one is disputing that add-on updates would be nice. It's just a matter of someone having the time to do it, including integration with the add-ons repository.

nvaccessAuto commented

Comment 7 by nvdakor on 2015-04-26 05:54
Hi,
After some investigation, it appears it might be doable. The biggest challenge is the mechanism to integrate with the add-ons repository. Recording the URL and then downloading and instaling the add-on is easy (we can borrow ideas from update check's UpdateDownloader class); the issue is how to differentiate between current and the newer add-on (hashing may work).

The big picture could be the following:

  1. Add an update button in the add-ons manager if the add-on manifest specifies an update URL. This could be either the repurposed URL field from the current manifest format, or it could be a new field just for this purpose.
  2. When the update button is clicked, a routine similar to update check routine from NVDA core would run, attempting to compare hashes and version and so on, which will result in the update UI similar to that of NVDA core.
  3. Once the update is downloaded, the add-on installation UI should come up, asking users if they wish to install the new add-on.

Issues to be resolved:

  • How can we calculate the first hash for our add-ons?
  • How should the web server which hosts add-ons website communicate with NvDA core as to comparing hashes, add-on version and so on?

I think anyone can implement the client side, and Mesar knows more about the server side unless if Jamie has access to that server. We also need to hear from add-on authors so we can prepare our add-ons to take advantage of this idea in the future.
Thanks.

nvaccessAuto commented

Comment 8 by jteh (in reply to comment 7) on 2015-04-27 03:50
Replying to nvdakor:

Recording the URL and then downloading and instaling the add-on is easy (we can borrow ideas from update check's UpdateDownloader class);

Ideally, that code should be refactored so that it can be reused rather than duplicating it. I started work on this a while ago (#3504), but never got around to finishing it.

the issue is how to differentiate between current and the newer add-on (hashing may work).

Keep it simple. Just compare the old version with the new version. If it's different, consider it an update. This is all we do for NVDA updates. That doesn't account for someone running a versio that is newer than is available on the site, but if that happens, the user clearly is a developer and/or is working with one.

  1. Add an update button in the add-ons manager

We should also consider automatic updates.

if the add-on manifest specifies an update URL.

I'm leaning towards restricting this functionality to the official add-ons repository, which means we don't need a full update URL, just some key indicating that this can do updates.

  • How can we calculate the first hash for our add-ons?

See above. We don't need hashes for version identification. We do need hashes for verifying file integrity, but those can be generated automatically.

Also, I don't think this needs to magically start working for existing add-ons. I think it's acceptable to require that users download a new version of the add-on initially to have it start updating.

  • How should the web server which hosts add-ons website communicate with NvDA core as to comparing hashes, add-on version and so on?

We could do something similar to the current NVDA update check server script. The bigger issue is actually having a database on the server of add-ons and versions. Currently, as I understand it, version information is only listed in the Markdown for the website, which is not something we can reliably use.

I think anyone can implement the client side, and Mesar knows more about the server side unless if Jamie has access to that server.

The entire translation and add-ons system is now all running on the NV Access server. Mesar is pretty busy with other things these days and I haven't seen much from him lately, so I'm now handling everything related to this (unless Mesar wants to step in). I have a pretty good understanding of some parts of the system, though am still learning my way around others.

nvaccessAuto commented

Comment 9 by nvdakor (in reply to comment 8) on 2015-04-27 03:59
Replying to jteh:

Replying to nvdakor:

Recording the URL and then downloading and instaling the add-on is easy (we can borrow ideas from update check's UpdateDownloader class);

Ideally, that code should be refactored so that it can be reused rather than duplicating it. I started work on this a while ago (#3504), but never got around to finishing it.

If you've started on this, then I think we should wait until #3504 is in place before returning to this ticket. Besides, update downloader and installer GUI has issues - for instance, the GUI for update downloader doesn't close automatically once an update has been downloaded and multiple update progress GUI may appear after a failed update is attempted.

  1. Add an update button in the add-ons manager

We should also consider automatic updates.

if the add-on manifest specifies an update URL.

I'm leaning towards restricting this functionality to the official add-ons repository, which means we don't need a full update URL, just some key indicating that this can do updates.

Understood.

Also, I don't think this needs to magically start working for existing add-ons. I think it's acceptable to require that users download a new version of the add-on initially to have it start updating.

Sure. I'd say we the authors need at least three months time to adjust to the new routines once this ticket has been implemented; whatever major version of the add-on we were working on, that version or the version after that can implement needed changes, similar to what happened last fall when we started implementing add-on help functionality.
Thanks.

nvaccessAuto commented

Comment 10 by nvdakor on 2015-10-02 01:40
Hi,
Coming back to this ticket...
Looking at urllib and NvDA Core's update routines ghave me an idea as to how to implement client side, but for version 1, it'll require user intervention.
The proposal is as follows:

  1. Add-ons should specify the download URL (perhaps a new key). This can be done easily, as many add-ons come with readme files written in Markdown that specifies URL's for stable and development versions.
  2. When update button is clicked, NVDA will gather the add-on manifest for the selected add-on, which will return add-on version, URL and so on. Perhaps a call to locate the add-on installer size should be specified for dev versions.
  3. NVDA will open the URL via urllib.urlopen, parse the URL and look at content length header. Thankfully, because of the add-on file naming convention in use, it is easy to parse the address looking for the version information (a couple calls to strip function).
  4. If the version on the server is newer than the installed add-on, assume that a newer version is available and present the update dialog (borrow NVDA Core's update presentation dialog, rewording parts of it to refer to add-ons).
  5. If a development version is being updated, check the content length against the size of the add-on installer for the currently installed add-on (initially the client side is 0), and if they are different, assume that a new version is available and act accordingly.
  6. For steps 4 or 5, if an update is downloaded, save the content length for future update lookup.
  7. Until #3504 is implemented, NVDA will call os.startfile.
    I'd be happy to write a proof of concept add-on that demonstrates this algorithm.
    Thanks.
nvaccessAuto commented

Comment 11 by jteh (in reply to comment 10) on 2015-10-02 08:29
Replying to nvdakor:

  1. Add-ons should specify the download URL (perhaps a new key).

I'm still thinking we should restrict this to the official add-on repository. Also, there are actually two URLs required in your proposed system: the URL for the manifest and the URL for the add-on itself.

  1. When update button is clicked, NVDA will gather the add-on manifest for the selected add-on, which will return add-on version, URL and so on.

As noted in comment:8, I think this is better done with a server script similar to the current update check server. This centralises things and allows for tracking of statistics (though that's of course not required for the first round).

  1. If a development version is being updated, check the content length against the size of the add-on installer for the currently installed add-on (initially the client side is 0), and if they are different, assume that a new version is available and act accordingly.

This is going to be fragile. Honestly, I think dev versions should just increment the version number (like NVDA snapshots do) if users really are supposed to update.

Thanks.

josephsl commented

Hi all,

@derekriemer and I had a talk about how to implement the server side (client side can be done easily). @nvdaes said on the add-ons mailing list that a file that lists add-on keys and versions would be sufficient, but Derek believes a database would be much better (I agree). The questions are what (database), when (if someone has time), and how (procedure).

CC @jcsteh

derekriemer commented

Hi,
I can open another ticket for this, I'd like an opinion from @jcsteh on this though.
I think when we do this, signing all community add-ons, and only allowing installation of community add-ons until a box is unchecked authorizing that unreviewed add-ons may be installed.
This has these advantages.

  1. Add-ons can check whether the new bundle is properly signed.
  2. When NVDA Boots, it checks the signature of all add-ons if the allow untrusted add-ons option is unchecked. It refuses to load any untrusted add-ons.
  3. When a user tries to install an add-on, if the add-on is not signed, it tells them this is an untrusted andd-on, and prompts them to change their settings if they really want untrusted add-ons.
  4. When updates occur, the add-on bundle must be signed before it will install.
derekriemer commented

Also, a post request for this would be desired, with all the add-ons being specified. I.E.
{
"rf": {
"version" : "2.2.2",
"beta" : false,
"hash" : "987987987987987987987987"
},
"ocr" : {
"version" : "2.1",
"beta" : false,
"hash" : "d53929098d550098409834409824..."
},
...
}
with json being returned of the form
{
"rf" : {
"version" : 2.5.0",
"beta" : false,
...
}
}

rkingett commented

Could all the add on's be hosted on Git Hub and then someone uses the Git Hub server to keep the costs down? & would that be an easier way to have NVDA check for updates?

LeonarddeR commented

This seemed to be quite an interesting ticket to read. Here are my thoughts.

  1. Although I can see why @jcsteh proposes add-ons can only be updated from a central server, this might cause problems for paid add-ons (such as speech synths) which are not in the community repository since they are not or not entirely open source. Even for non-open source add-ons, I think there should be a pathway for automatic updates. Two alternatives:
    A. Only allow third party unregistered add-ons to be updated from a https source with verified SSL certificate
    B. Cumulatively, Allow add-ons from other sources and their corresponding url's to be registered on the centralised add-on system.
jcsteh commented

I agree we want paid add-ons to get automatic updates. I imagined we would have them registered with our central repository, even though you probably wouldn't be able to actually purchase them from there.

feerrenrut commented

P3. This seems like it would take a large amount of time to implement, and is not currently stopping primary use cases. It would however be very convenient.

@jcsteh Perhaps the priority should be higher if this is considered of strategic importance to NVDA?

jcsteh commented
rkingett commented

A lot of people have no idea where to get add on updates, though, or
even that an add on has updated, so this would seriously be more than
just a convenience

On 11/13/2016 11:29 PM, Reef Turner wrote:

P3. This seems like it would take a large amount of time to implement,
and is not currently stopping primary use cases. It would however be
very convenient.

@jcsteh https://github.com/jcsteh Perhaps the priority should be
higher if this is considered of strategic importance to NVDA?

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#3208 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMj7zJ0QpOnkrL1fZnuKML9ZGP0oEp5mks5q9_HAgaJpZM4JXkpI.

josephsl commented

Hi,

After doing some more research and talking with @derekriemer, it appears the best possible server side implementation would be that of a table that includes info such as add-on name, description, version, download link and hash and so on, with another table reserved strictly for add-on updates. When the client looks up add-on updates, the client should specify add-on update channel, then the server should return update info such as version, download link and hash.

For example, suppose we use the URL of the form: https://addons.nvaccess.org/updateCheck?name=addon_channel

For instance, Windows 10 App Essentials dev version could say:
https://addons.nvaccess.org/updateCheck?name=wintenApps-dev

The server should return nothing if the version is up to date or a response similar to how NVDA Core checks for new versions (version, download link, hash).

For add-ons, a new manifest key named "addon_updateChannel" or something similar could be added, with the default value being "None" (no info). However, before the add-on updates database says that the specified add-on does not exist, the server would assume that one wants to update to the latest stable branch. Ideally, a new add-on that does include update name should be used.

Additional comments are appreciated. Thanks.

derekriemer commented
jcsteh commented
derekriemer commented
LeonarddeR commented
jcsteh commented
derekriemer commented
jcsteh commented
derekriemer commented
rkingett commented
derekriemer commented
josephsl commented

Hi,

A progress report: some progress has been made in abstracting away the update downloader, a clearer picture on client, server and UI.

  1. Abstracting the downloader: the latest development snapshots of Windows 10 App Essentials and StationPlaylist Studio add-ons include a subclass of updateCheck.UpdateDownloader tailored for these add-ons. No need to override GUI exec routines (the only thing to be modified are methods that deal with tempfile destination, GUI strings and download URL). As for ShellExecute, just run the .nvda-addon file (no arguments needed). The subclass does allow installed copies of NVDA to download add-on bundles (I've intentionally limited this to installed copies) and it works on both HTTP and HTTPS.
  2. Client GUI: I'm working on this at the moment (branch i3208-updateAddonsClient) in my fork. This branch will add a new button in add-ons manager called "Check for Add-on Update" (pneumonic: U). Once clicked, it'll use similar routine as in NVDA Core update (a background thread to communicate with the server via urllib and parse the results) to look up add-on update info from a website (add-ons website). To guarantee support for different fetch and result parsing mechanisms, a function should be used to let the add-ons manager know if a new add-on version is available (similar to how NVDA Core uses a dictionary to let users know about this). Once the user chooses to update, the add-on will be downloaded and installed if one is using an installed copy (which naturally raises the question about portable versions). In case of installed copy, once the new add-on bundle is downloaded, go through install phase (portable version will be different).
  3. Server: Database is the best solution with at least three tables: add-on versions, download links and hashes, and list of add-ons that might be questionable (illegal synthesizers, for instance). The basic syntax (in SQL/PostGreSQL) is listed below. In terms of server mechanics, questionable add-ons table is consulted, then consult versions if the add-on is safe. Once the version does not match, the server should send update file/hash info as a JSON or a similar format back to clients. As @derekriemer noted, a post method would be best in case multiple add-ons will be looked up.

As for server tables:

Add-on version table:
CREATE TABLE table1(
name TEXT PRIMARY Key NOT NULL,
channel TEXT PRIMARY Key,
version TEXT NOT NULL
)

Add-on info table:
CREATE TABLE table2(
name TEXT PRIMARY KEY NOT NULL,
channel TEXT PRIMARY KEY,
version TEXT PRIMARY KEY NOT NULL,
description TEXT NOT NULL,
author TEXT NOT NULL,
filename TEXT NOT NULL,
filehash TEXT NOT NULL
)

Questionable add-ons:
CREATE TABLE table3(
name TEXT PRIMARY KEY NOT NULL,
version TEXT
)

Notes:

  • Table names are arbitrary (it's up to server-side implementers to choose the table names, but to the outside world, what matters the most is the URL encoding and version info that'll be returned).
  • If "channel" is not present, assume stable channel.
  • Filename refers to the URL from where the update can be downloaded from.
  • Filehash refers to hash for the add-on bundle.
  • Version is a string.
  • The absolute minimum info that should be encoded as URL's are add-on name and version (channel is optional unless a specific channel is requested).

As for my SQL syntax, this is a mixture of real life and idealized syntax, and I used PostGreSQL syntax.

Comments on this updated design is appreciated. Thanks.

derekriemer commented
jcsteh commented
derekriemer commented
jcsteh commented
josephsl commented

Hi,

Blueprint is done (with refactored downloader (#3504) and JSON transport missing).

In regards to downloading bundles: possible routines are:

  • Commit to doing one thing at a time: download all add-on bundles at once, then install all at once.
  • Bundle after bundle: for each bundle, download and install right away.

Some musings on each option:

  • Both should allow the UI to present which add-ons are being downloaded and installed.
  • With the first option, the progress bar will indicate overall download progress of all add-ons combined, whereas the progress bar will reset for each add-on in the second option.
  • The second option has an added benefit of continuing where one left off or if an error occurs, as well as ability to use threads to install a large add-on while downloading the next, although UI issues should be considered.

I'm leaning towards the second option.

Thanks.

nvdaes commented
josephsl commented

Hi all,

Updates as of March 2017:

Operational:

  • Update check emulation is working.
  • Update download is working - using a modified add-on handler and using custom URL's.
  • Manual add-on update check is working, including the case where no add-on updates are available.

What's not working:

  • After updating multiple add-ons at the same time, add-ons manager window does not appear (might be due to limited experience with GUI programming).
  • When updating multiple add-ons, if at least one has an installation UI, progress dialogs will stay open until the installation UI is dealt with.

Ideally:

  1. Get available updates for all add-ons currently installed.
  2. Download and instlal one add-on at a time, taking care of installation UI.
  3. If this is a manual check (from Add-ons Manager), return to Add-ons Manager, and if not, just prompt to restart NVDA.

See #6930 pull request for a more technical discussion regarding this ticket. Thanks.

josephsl commented

Hi,

A major issue found: the reason why add-ons cannot be updated one after the other with current commits is because of thread scheduling and synchronization:

  • In the update loop, the ideal execution sequence is for add-on installer to be downloaded and updated one after the other.
  • In reality, add-ons are downloaded in one go, then all are updated at once.

I'm exploring various options, including use of events. Comments are appreciated. Thanks.

Brian1Gaff commented
LeonarddeR commented

@josephsl: SOme random thoughts:

  • Have you considered putting a multi instance check on updateCheck.UpdateDownloader? This may be a safe way to avoid multiple updates, as well as updating both core or an add-on at the same time
  • Have you tried creating an event in the constructor of AddonUpdateDownloader, than set the event when the downloader has finished its work? You can wait for the event in the for loop that executes the downloaders.

Also, some questions.

  • In the current implementation, does one add-on download after the other, or do they all just download at once?
  • Same for installation, do they nicely install in a row, or also install all together?
  • Does any installation take place while downloading?
josephsl commented

Hi,

Event will not work (I tried), as it'll freeze NVDA (busily waiting for an event to be cleared).

As for update sequence: once the background thread begins to download updates, the next update will be downloaded as threads will take off. This isn't ideal, as add-ons should be downloaded and updated one after the other, and this is a crucial requirement in implementing automatic add-on update checks and downloads.

In general, whenever you do something with progress bars (such as update downloads), doing it in the main thread would result in NVDA freezing, hence the background worker thread is employed. Ideally, the main GUI thread should be the one updating the progress bar, but...

@jcsteh knows more about this, as he dealt with this more often than we did.

Thanks for your input.

josephsl commented

Hi,

Regarding freezing, a clarification: NVDA will appear to freeze when in fact something is happening.

Thanks.

jcsteh commented

If you want to download and install one at a time, the key is to only kick off one downloader at a time. In the success/failure callback for that download/install, you then kick off the next one. You can do this either by maintaining a list and popping off the next item each time or by using a generator which you call next() on in the callback.

jscholes commented

In general, whenever you do something with progress bars (such as update downloads), doing it in the main thread would result in NVDA freezing, hence the background worker thread is employed. Ideally, the main GUI thread should be the one updating the progress bar, but...

It's important to distinguish between the actual updating of the progress bar and the work going on for which progress needs to be communicated. The work should happen on a background thread, but the GUI should be updated in the main one.

Brian1Gaff commented
LeonarddeR commented
derekriemer commented

@jcsteh commented on Jan 27, 2017, 12:01 AM MST:

Thanks. Some feedback:

  1. Rather than using ShellExecute, just call addonHandler directly. This
    also means this will work just fine for portable copies; they'll be exactly
    the same as installed. Things are very different for NVDA itself because it
    needs to install system wide. Add-ons always install to the user config.

is this still true after the work @LeonarddeR did?

derekriemer commented

I do have to wonder, should we not make the key of the table (name, channel, version)?

Ancientlizard commented

Hello!
Thank you for developing nvda! An indispensable program for screen access.
I have a few suggestions regarding nvda.
The relevance and updating of add-ons.
there are many useful add-ons without which it will be very difficult to work in some programs where nvda does not announce everything that happens on the screen. .
And time goes by, and additions are better and better. and they update quickly.
Please tell me, can you implement in nvda checking the relevance of certain additions?
Through the developer's website?
And add a button to the add-on Manager
"check the new version of the add-on." and thus upload a new version when available without going to the website.
interact with Microsoft Office 2016 and other office applications with nvda.
Please tell me What is being done to improve the availability of the Office 2016 package?
And are new versions available starting in 2013?
I need to work with applications like
Microsoft Office InfoPath, Microsoft Office Groove, Microsoft Office OneNote, Microsoft Office Publisher and Microsoft Office SharePoint.
I understand that good accessibility requires microsoft to help you, too,but something is being done to improve accessibility in these and other office applications?
Thanks in advance for your response!!!
Your answer is very important to me.
I already wrote, but accidentally not there wrote.
Asking questions again.
Yes, I am interested specifically in the above written applications and accessibility of office applications in particular.Thanks for the reply.

josephsl commented

Hi,

Okay, now we have the correct content.

Regarding Microsoft Office things: as this is not the ticket for this, I advise creating a new issue (not a pull request) just for Microsoft Office problems. Thanks.

josephsl commented

Hi,

For benefit of @feerrenrut and others who have recently boarded this train, a short summary of what's been happening for the past five years:

In earliest versions of NVDA (say, 0.6, 2009.1 and others), one can "install" Python modules to serve as plugins in folders inside user config directory. These were looked up when NVDA starts, and depending on where the module was found, it could add support for an app or two, or add global features. This was changed in 2012 when add-ons were introduced, at which point everything having to do with extenidng NVDA via external modules became part of add-ons.

Back in the early days of add-ons community (which I'm serving as some sort of a press secretary at the moment), users needing to update add-ons would visit community add-ons website and download updated releases. In some cases, authors such as myself provided direct links to new add-on releases. At the time of creation of this issue (2013), it became apparent to the add-ons people that people were not updating their add-ons unless told to do so, and a way to let NVDA check for, download, and apply add-on updates became desirable.

Currently the server that hosts community add-ons website uses a PHP file to keep a record of ad-dons and their versionss, controlled from a Bitbucket repository. This exposed us to several flaws, including someone accidentally deleting the repo, not being flexible in adding new add-on releases from the website itself and what not. After several rounds of discussions, people involved in this conversation agreed that a database would be much better to not only store add-on information, but also to open up more possibilities including updating add-ons.

As it stands, add-on update client is ready to go (by the way, restrictions on portable copies has been lifted thanks to unified NVDA update facility introduced in 2018.1 as part of postponing updates) provided that the exchange format is standardized. The biggest hurdles as of July 2018 are:

  1. Designing the exchange format: JSON will be used to encapsulate all sorts of data about add-ons.
  2. Server infrastructure: the add-ons server should be ready to host a database for storing add-on information (an SQL database is a most likely scenario).
  3. Resources: not only finding a server to test this on, but also asking folks to help out.
  4. Communication: not only Core people must be aware of what's up, add-ons community, and in extension, the entire NVDA community should be made aware of what's up. The biggest hurdle in communication being talking to authors of add-ons not hosted on community add-ons website at the moment.

As for priority level: P3 at the moment not only because of higher priority work that's being done, but also because this feature will require some outstanding pull requests to be integrated into NVDA, namely checkable list box.

Hope this helps. Thanks.

  • 👍1
Adriani90 commented

@yplassiard, could you please give us a Brief update on your Server infrastructure? In my view the NVDA store created by you along with the update client from @josephsl is the best and most confortable Option.

josephsl commented

Hi,

The NVDA Store add-on doesn't work for me - even if I "fool" the add-on into believing that I have an older version of Windows 10 App Essentials, update functionality fails. This is most likely caused by bad or expired certificate on the NVDA Store server, which makes the add-on inoperative.

Also, based on communication log between the Store client and server, the server implementation is quite inefficient: it returns ALL available add-on files, not the latest one, although it may have been due to locating all available entries stored on a supposed add-ons database (it uses CGI and cookie jar libraries, along with JSON and Requests+URLLib3 module). Note that this assumption might be wrong as I don't know the internals of the server database.

Third, the "changelog" for each add-on update is displayed in French, and for some important add-ons, way out of date and too generic. Surely the changelog for Windows 10 App Essentials is more than "fixed bugs".

Lastly, I'm afraid the code for NVDA Store add-on client and GUI are not compatible with NVDA Core code guidelines. There is a mixture of tabs and spaces for indentation, idiom issues and what not.

In the meantime, I'll release an add-on that'll simulate add-on updating procedure as envisioned by this issue. This new add-on will add a new item in Tools menu and will check for updates from official add-on repo only. Due to technical limitations, it'll support NVDA 2018.2 and later.

Thanks.

josephsl commented

Hi,

A proof of concept add-on has been created and now shows up on add-ons website.

Also, @nvdaes suggests that NVDA should be told to not update certain add-ons via a "do not update" list, as well as favoring certain update channels. As for "do not update" list, it can be done, but the issue would be how users can toggle this state. As for the latter, ultimately it comes down to capabilities of the new update server.

Lastly, when #6275 becomes a reality, it will change how add-on update feature will work, as it must consult add-ons that are not compatible with the latest NvDA release.

Thanks.

nvdaes commented

Should be a mechanism provided to consult the last documentation for each add-on before updating.
This could be shown in a readonly box, to decide if it should be updated. Or at least the last changes, but this may not be enought, but a button to open the add-on webpage (not the website), seems not quite graceful.

jscholes commented

A few thoughts on this going forward, some of which will have already been mentioned elsewhere.

  1. Addressing the current server-side limitations should be a priority, in order to speed up add-on update checks. If nobody is available to carry out this work, perhaps somebody new should be found and appointed.
  2. Going forward, NVDA should allow the user to configure a proxy server for use with outgoing HTTP requests, whether initiated by NVDA Core or add-ons. It should also consume these settings from Windows itself if a proxy is set system-wide.
  3. To avoid verbose focus changes while downloading multiple add-on updates, a single progress dialog should be used instead of the current one-dialog-per-add-on approach.
  4. Add-on packages are generally quite small, so multiple updates should probably be fetched concurrently.
larry801 commented
yplassiard commented

Hi all,
Following the discussion here are my thoughts regarding the update mechanism itself, based on what I implemented within the NVDAStore add-on and feedbacks received after about a year of usage:

User requirements

here are the requirements I think we should address in this implementation:

  • As a user I want to be able to list installed add-ons with their associated name, author, cinstalled version, and latest available version.
  • As a user I want to be able to select an add-on in this list and update it to the latest version.
  • As a user I want to be able to update all installed add-ons to their latest version.

Here are some more requirements that may be really interesting to be implemented, although it may require a slightly more advanced server implementation:

  • As a user I want to be able to access the add-on's changelog priop updating it.
  • As a user I want to be able to update add-ons on NVDA start-up without requiring any interaction: when add-ons are updated through this process, NVDA will notify me that it should be restarted to take advantage of the newest version installed.

Add-on Server

To me, and although it is always encouraged to use the latest NVDA version installed, there is an important "compatibility" check that has to be done for each add-on prior installation. I see here several use cases that should be addressed:

  • NVDA version: An add-on which does not support wxPython 4.x may not be updatable and/or installable if the NVDA version does not match the one flagged as appropriate. wxPython support is an example here, but may be extended to any core NVDA feature the add-on relies on.
  • Windows version: Some add-ons such as Windows 10 App Essentials depends on Windows 10 (obviously), and therefore it is pointless to show such an add-on if the Windows 10 release is not installed.
  • Application presence and version: An add-on that enables or improves accessibility for a particular application should not show up if this application is not installed, or if the application's version is incompatible with the add-on.
  • System features: Some add-on require some hardware and/or software features to work: enhanced Touch Gestures is the most obvious case, being useless if no touch screen is installed on the target system.
  • other add-on dependencies: Especially useful for other TTS add-ons, voices may not be installable if the core TTS add-on is not installed.

To solve this issues, (and that's why the NVDAStore uses this approach), add-on coompatibility checks are done at client level. For generic checks such as Windows release, NVDA version, and identified hardware such as touch screen, checks may be generic and implemented statically within the add-on management system.
For other more specific checks such as the presence of a certain application, it may be useful for add-on authors to provide a routine that would ensure that the given application is installed before trying to download the desired add-on version.

API notes

Based on the previous, here are some notes regarding the API implementation:

  • It, of course, has to be state-less.
  • The server should return the most extensive results: this means that when getting available add-ons, all available add-ons may be returned, with all available versions for each add-on. This would increase the response's size, but considering 100 addons, with 512 bytes for the description, 128 for the summary, name, and author, and something like 256 bytes reserved for each add-on version, including version and the list of associated install requirements, would lead to a size of 3456 bytes for a single add-on with 10 addon version, which would do a kind of 256kb payload for 100 addons.
  • The server should return the addon changelog only when the user asks for addon information: this will save some bandwidth, returning this large amount of text only from time to time.
  • Addon package files have to be stored on a dedicated storage, maybe on the same machine that provides the update server. This is done especially because addon signing support would be stronger if the server also controls the package file, as well as because pointing to an external location is not easily trackable (hot file replacement, file deletion, etc.)

Server implementation

As stated in other comments, I also think a relational database is probably a good choice for this kind of implementation. Each entity would have its own table, with relationships to appropriates entities. Here are the entities I think we should take into account, leading to the following SQL tables:

  • addons: Would contain addon name, summary, author, description, and homepage.
  • addon_versions: Would contain the addon version the associated package filename. Each entry would be associated with an add-on using foreign key mechanism.
  • addon_capabilities: Would contain a capability name, and an optional filename to download to perform the actual capability check.
  • addon_version_requirements: would map a version to a list of zero or more capabilities.

Final design thoughts

This design as, in my opinion, the following benefits:

  • It can be implemented using iterations: No need to address the full user requirements listed above for the first release.
  • It is extendable, we may create other tables, or entities, to represent other addon related concepts when we want to improve this feature. Such concepts may be addon user reviews, ratings, bug reports, and so much more.
  • The API would be REST-compliant, which is much easier to deal with, and which supports several concurrent API versions if needed: /api/v1/addon, and /api/v2/addon for example.
  • The addon management client does not have to know about what addon it is installing: All capability checks code can be downloaded from the server. And because this code is stored on the server itself, it can be signed to ensure it is trustable, and of course be cached locally for future use.

Appendix: translations

Because add-ons contain translations for summary, description, as well as changelogs, this also has to be managed by the server implementation, either:

  • returning all translations for each translated text (very very innefficient).
  • Store translations on the addon server, extending the tables appropriately, so that any returned text associated to a SQL query is translated to the user's language, which may be English by default or if the user language translation is not available for the given text.

Where to go from here

I think we should agree first on the API and user requirements. When this is approved, I would be delighted to start, with others if they are willing to, the addon update server implementation. It would consist in the following:

  • the server itself, providing the API endpoints.
  • The frontend application used to manage registered addons, reference new ones, and, maybe, provide the "public" website if we want to.

Let me know what you think.

jscholes commented

Application presence and version: An add-on that enables or improves accessibility for a particular application should not show up if this application is not installed, or if the application's version is incompatible with the add-on.

This would be unmanageable. A user may have multiple versions of an application installed, use portable copies, run an application from source, or have specific development versions installed that an add-on's checks couldn't handle. There are also many applications which don't offer up the needed information via the VERSIONINFO API. It should be up to add-on authors, in cases where this is appropriate, to detect the version number of an application once the app module receives focus or is instantiated and inform the user of any compatibility concerns.

feerrenrut commented

The server should return the most extensive results: this means that when getting available add-ons, all available add-ons may be returned, with all available versions for each add-on

Returning all available versions of each addon seems unnecessary to me, what is the use case here?

Only displaying addons that match the users Windows release, identified hardware, or installed applications breaks the portability of NVDA. Users must be able to install addons to a portable copy that they can then use elsewhere, they may need to do the installation on a system other than the one that they expect to use the addon.

yplassiard commented

Returning all available versions of each addon seems unnecessary to me, what is the use case here?

The use case is mainly to let the ability for the NVDA addon management client to check by itself that an addon version may be installable before proposing an update to the end-user. The other way to do this ould be to make the client sends its NVDA and Windows version when requesting the addon's list so that the server can filter it, but although it may be nice for statistic purposes, I'm not sure everybody will be happy to send such infos to the server.

Regarding capability checks, you're right when you say that on a portable copy, the hardware checks may be different from a machine to another, but IMHO, letting the user download and install an addon withoud the associated hardware should, at least, present a warning indicating that such capability is currently missing. For an installed copy however, I think that the addon should not be installable.

feerrenrut commented

present a warning indicating that such capability is currently missing.

Perhaps, but I still think this is debatable. Either way, I think this is probably outside of the core goals of this issue. Happy that we ensure that the auto update mechanism does not preclude things like this, but mandating it at this stage is a mistake in my opinion.

josephsl commented
josephsl commented

Hi,

Update for November 2018: I'm slowly incorporating lessons learned from Add-on Updater add-on into this pull request. at the moment almost all features from that add-on are present except for choosing update channel. most notably, two outstanding issues/requests were done:

  1. Better user experience when add-on update results shows up on startup: when add-on updates were found at startup, NvDA didn't focus on update results dialog; now it will thanks to an internal callback.
  2. Users can now turn off update checking for one or more add-ons. As @jcsteh suggested, add-on state will record a set of add-ons where update checking is disabled. Eventually a new property will be added that'll let NVDA know if the add-on supports update checking. Also, enabling/disabling add-on update checks for various add-ons can be configurable via a new dialog in Add-ons Manager.

Thanks.