openzim/zimfarm

Migrate to MWoffliner 1.14

Closed this issue · 6 comments

kelson42 commented

We plan to release MWoffliner 3.14 in the next weeks. This new release bring mainly:

  • Support of the new mobile-html API
  • Deprecation of the old mobile-section API
  • Rearchitecture/Modularisation of the whole scraper based on API end-points
  • Few changes in the arguments

Therefore we need to be in general careful with this move and to adapt the ZIMFarm MWoffliner Recipe template.

Here is the diff regarding the path related paramters:

-  mwApiPath: 'Mediawiki API path (per default "/w/api.php")',
-  mwRestApiPath: 'Mediawiki Rest API path (per default "/api/rest_v1")',
+  mwActionApiPath: 'MediaWiki API path (by default "/w/api.php")',
+  mwRestApiPath: 'MediaWiki REST API path (by default "/w/rest.php")',

In addition we have a new parameter:

  • forceRender:
  • 'Force the usage of a specific API end-point/render, automatically chosen otherwise. Accepted values: [ VisualEditor, WikimediaDesktop. WikimediaMobile ]. More details at https://github.com/openzim/mwoffliner/wiki/API-end-points',
  • insecure: 'Skip HTTPS server authenticity verification step',
kelson42 commented

Rollout of this new template should be made in coordination with @audiodude and @kelson42

benoit74 commented

Do I get it correctly that we need to transfer the two old arguments mwApiPath and mwRestApiPath to the two new ones mwActionApiPath and mwRestApiPath (easy with a bit of SQL) + add two new arguments forceRender (string) and insecure?

I can prepare this now with a PR + SQL script to execute on the DB, and roll this out at your convenience, probably at the same moment we will update all recipes to use 1.14

kelson42 commented

@benoit74 Yes, default value for mwRestApiPath has changed so if not set, should probably be set now. Thx!

benoit74 commented

Oh, I did not saw it is the same parameter name but meaning is different. This is very weird, especially in a minor release, this is a very breaking change I imagine.

But then what do you wanna do? Nothing has to be done for this parameter? Or should I drop this parameter so that it returns to default? Reset this parameter to something else?

benoit74 commented

@kelson42 up? what do you wanna do with this parameter which changed meaning?

kelson42 commented

@benoit74 Not sure, but I would do nothing for the moment