Plugin Site API
Hacking
In order to run the plugin-site-api tests, a service needs to be online which
can serve up a DATA_FILE_URL
. Assuming docker-compose
is installed on the
system, the following commands will run the tests properly:
-
in another terminal:
make run
-
make check
Inspect the Makefile
for more targets, such as all
which will run tests and
build a working container.
Architecture
The REST API calls are powered by Jersey 2.x running inside of Tomcat 8. Plugins are stored and searched via embedded Elasticsearch.
Plugin information sources include:
-
Jenkins Update Center
-
Main source of plugin data
-
-
Jenkins Stats
-
Installation statistics
-
-
Jenkins Wiki
-
Scrape wiki content
-
Elasticsearch mappings are initialized at runtime using a temporary directory that is cleaned up at shutdown. The data is generated via a separate Maven task and stored on an external URL outside of the application. The data file is downloaded from said URL and extracted at boot time. Every 12 hours the application will download the file again and check if it contains new data. If so the application will the reindex the Elasticsearch data using the new data file.
Run Local Plugin Site API
GITHUB_TOKEN=token from https://github.com/settings/tokens DATA_FILE_URL="https://ci.jenkins.io/job/Infra/job/plugin-site-api/job/generate-data/lastSuccessfulBuild/artifact/plugins.json.gzip" mvn jetty:run
This will launch an embedded Jetty container accessible at http://localhost:8080
.
DATA_FILE_URL
can point to http
/https
URLS or to local files using the file://${path}
syntax.
If GITHUB_APP_ID
and GITHUB_APP_PRIVATE_KEY
are not set it will try to use a personal access token provided by GITHUB_TOKEN
and failing that, your site will be limited to fetching 60 plugin documentation pages per hour.
Run in production mode
You can still use the GITHUB_TOKEN method, but preference is to use github apps (https://github.com/settings/apps) Make sure your pem file is convered from the format github apps gives you to the PKCS#8 format by running
openssl pkcs8 -topk8 -inform PEM -outform PEM -in devpluginsjenkinsio.2020-07-28.private-key.pem -out githubapp.pem -nocrypt
Then configure it by
GITHUB_APP_ID=1234 GITHUB_APP_PRIVATE_KEY="/path/to/pem/file" --- == Run Docker Plugin Site API
docker build -t jenkinsciinfra/plugin-site-api . docker run -p 8080:8080 -it -e DATA_FILE_URL="http://url.to/plugins.json.gzip" jenkinsciinfra/plugin-site-api
== Rebuild Elasticsearch data
mvn -P generatePluginData
This will generate a new file in `target/plugins.json.gzip` consisting of plugin information and installation statistics. This file can be now passed as `DATA_FILE_URL`: ``` DATA_FILE_URL="file://$(pwd)/target/plugins.json.gzip" mvn jetty:run ``` == REST API Reference WARNING: This API is still in flux as the frontend and backend are still being developed. As such this API is likely to change in the near future. Unless otherwise indicated, all API calls: * Produce JSON * Do not require authentication * Have no parameters or request body Possible error codes for all calls: * 500 === GET /categories Retrieve categories with their labels. It will be necessary to call /labels to get the titles Sample Response
{ "categories": [ { "id": "languagesPlatforms", "title": "Platforms", "description": "Jenkins plugins that are designed to give added support for building, testing or deploying to specific languages or platforms.", "labels": [ "ios", "dotnet", "android", "ruby", "scala" ] }, … ], "total": 6 }
=== GET /maintainers Retrieve unique maintainers in the plugin data. Sample Response
{ "maintainers": [ "Kohsuke Kawaguchi", "Jesse Farinacci", "Stephen Connolly", "Gregory Boissinot", "Oliver Gondža", "Oleg Nenashev", "Seiji Sogabe", "Lucie Votypkova", "Vojtech Juranek", "Nicolas De Loof", "Bap", "Praqma Josra", "Robert Sandell", "Stefan Brausch", ….. ], "limit": 909 }
=== GET /labels Retrieve available plugin labels. "title" is an optional field so it may be missing from some labels. Sample Response
{ "labels" : [ { "id": "ios", "title": "iOS development" }, { "id": "dotnet", "title": "Azure and .NET" }, { "id": "android", "title": "Android development" }, { "id": "ruby", "title": "Ruby development" }, { "id": "scala", "title": "Scala plugins" }, …. ], "limit": 30 }
=== GET /plugin/:name Retrieves information about a plugin Could return 404 if plugin is not found Sample Response
{ "buildDate": "Jul 04, 2016", "categories": [ "scm" ], "dependencies": [ { "name": "matrix-project", "optional": false, "version": "1.6" }, … ], "maintainers": [ { "id": "kohsuke", "name": "Kohsuke Kawaguchi", "email": null }, … ], "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.", "gav": "org.jenkins-ci.plugins:git:2.5.2", "labels": [ "scm" ], "name": "git", "previousTimestamp": "2016-07-02T20:46:28.00Z", "previousVersion": "2.5.1", "releaseTimetamp": null, "requiredCore": "1.609.3", "scm": "github.com", "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=", "stats": { "installations": [ { "timestamp": 1322697600000, "total": 8906 }, … ], "installationsPercentage": [ { "timestamp": 1459468800000, "percentage": 61.16896694248365 }, … ], "installationsPerVersion": [ { "version": "2.0.3", "total": 141 }, … ], "installationsPercentagePerVersion": [ { "version": "2.2.7", "percentage": 0 }, … ], "currentInstalls": 89232, "trend": 2990 }, "title": "Jenkins Git plugin", "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi", "version": "2.5.2", "wiki": { "content": <HTML content>, "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin" }
=== GET /plugins Search for plugins .Parameters [options="header,footer"] |======================= |Name|Required|Description|Default Value|Possible Values|Example |q|false|Search plugin name, title, excerpt if given|||workflow |sort|false|Define how results are sorted|name|name,updated|name |categories|false|Filter by categories|||scm |labels|false|Filter by labels|||scm,ios |maintainers|false|Filter by maintainers|||kohsuke |core|false|Filter by required core version|||2.13 |limit|false|Specify page limit for results|50||25 |page|false|Specify page number to return. This is not zero based|1||3 |======================= Sample Request
GET /plugins?q=git&sort=name&limit=3&page=1
Sample Response
{ "page": 1, "pages": 3, "plugins": [ { "buildDate": "Jul 04, 2016", "categories": [ "scm" ], "dependencies": [ { "name": "matrix-project", "optional": false, "version": "1.6" }, … ], "maintainers": [ { "id": "kohsuke", "name": "Kohsuke Kawaguchi", "email": null }, … ], "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.", "gav": "org.jenkins-ci.plugins:git:2.5.2", "labels": [ "scm" ], "name": "git", "previousTimestamp": "2016-07-02T20:46:28.00Z", "previousVersion": "2.5.1", "releaseTimetamp": null, "requiredCore": "1.609.3", "scm": "github.com", "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=", "stats": { "installations": [ { "timestamp": 1322697600000, "total": 8906 }, … ], "installationsPercentage": [ { "timestamp": 1459468800000, "percentage": 61.16896694248365 }, … ], "installationsPerVersion": [ { "version": "2.0.3", "total": 141 }, … ], "installationsPercentagePerVersion": [ { "version": "2.2.7", "percentage": 0 }, … ], "currentInstalls": 89232, "trend": 2990 }, "title": "Jenkins Git plugin", "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi", "version": "2.5.2", "wiki": { "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin" } }, … ], "limit": 3, "total": 7 }
=== GET /plugins/installed Get top "limit" install plugins .Parameters [options="header,footer"] |======================= |Name|Required|Description|Default Value|Possible Values|Example |limit|false|Specify limit for results|10||5 |======================= Sample Request
GET /plugins/installed
Sample Response
{ "page": 1, "pages": 3, "plugins": [ { "buildDate": "Jul 04, 2016", "categories": [ "scm" ], "dependencies": [ { "name": "matrix-project", "optional": false, "version": "1.6" }, … ], "maintainers": [ { "id": "kohsuke", "name": "Kohsuke Kawaguchi", "email": null }, … ], "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.", "gav": "org.jenkins-ci.plugins:git:2.5.2", "labels": [ "scm" ], "name": "git", "previousTimestamp": "2016-07-02T20:46:28.00Z", "previousVersion": "2.5.1", "releaseTimetamp": null, "requiredCore": "1.609.3", "scm": "github.com", "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=", "stats": { "installations": [ { "timestamp": 1322697600000, "total": 8906 }, … ], "installationsPercentage": [ { "timestamp": 1459468800000, "percentage": 61.16896694248365 }, … ], "installationsPerVersion": [ { "version": "2.0.3", "total": 141 }, … ], "installationsPercentagePerVersion": [ { "version": "2.2.7", "percentage": 0 }, … ], "currentInstalls": 89232, "trend": 2990 }, "title": "Jenkins Git plugin", "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi", "version": "2.5.2", "wiki": { "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin" } }, … ], "limit": 3, "total": 7 }
=== GET /plugins/trend Get top "limit" trending plugins .Parameters [options="header,footer"] |======================= |Name|Required|Description|Default Value|Possible Values|Example |limit|false|Specify limit for results|10||5 |======================= Sample Request
GET /plugins/trend
Sample Response
{ "page": 1, "pages": 3, "plugins": [ { "buildDate": "Jul 04, 2016", "categories": [ "scm" ], "dependencies": [ { "name": "matrix-project", "optional": false, "version": "1.6" }, … ], "maintainers": [ { "id": "kohsuke", "name": "Kohsuke Kawaguchi", "email": null }, … ], "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.", "gav": "org.jenkins-ci.plugins:git:2.5.2", "labels": [ "scm" ], "name": "git", "previousTimestamp": "2016-07-02T20:46:28.00Z", "previousVersion": "2.5.1", "releaseTimetamp": null, "requiredCore": "1.609.3", "scm": "github.com", "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=", "stats": { "installations": [ { "timestamp": 1322697600000, "total": 8906 }, … ], "installationsPercentage": [ { "timestamp": 1459468800000, "percentage": 61.16896694248365 }, … ], "installationsPerVersion": [ { "version": "2.0.3", "total": 141 }, … ], "installationsPercentagePerVersion": [ { "version": "2.2.7", "percentage": 0 }, … ], "currentInstalls": 89232, "trend": 2990 }, "title": "Jenkins Git plugin", "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi", "version": "2.5.2", "wiki": { "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin" } }, … ], "limit": 3, "total": 7 }
=== GET /plugins/updated Get top "limit" recently updated plugins .Parameters [options="header,footer"] |======================= |Name|Required|Description|Default Value|Possible Values|Example |limit|false|Specify limit for results|10||5 |======================= Sample Request
GET /plugins/updated
Sample Response
{ "page": 1, "pages": 3, "plugins": [ { "buildDate": "Jul 04, 2016", "categories": [ "scm" ], "dependencies": [ { "name": "matrix-project", "optional": false, "version": "1.6" }, … ], "maintainers": [ { "id": "kohsuke", "name": "Kohsuke Kawaguchi", "email": null }, … ], "excerpt": "This plugin allows use of <a href='http://git-scm.com/'>Git</a> as a build SCM, including repository browsers for several providers. A recent Git runtime is required (1.7.9 minimum, 1.8.x recommended). Interaction with the Git runtime is performed by the use of the [JENKINS:Git Client Plugin], which is only tested on official <a href='http://git-scm.com/'>git client</a>. Use exotic installations at your own risk.", "gav": "org.jenkins-ci.plugins:git:2.5.2", "labels": [ "scm" ], "name": "git", "previousTimestamp": "2016-07-02T20:46:28.00Z", "previousVersion": "2.5.1", "releaseTimetamp": null, "requiredCore": "1.609.3", "scm": "github.com", "sha1": "0LNQKJ+Tcn9vTwqMbtxSi1SM+s0=", "stats": { "installations": [ { "timestamp": 1322697600000, "total": 8906 }, … ], "installationsPercentage": [ { "timestamp": 1459468800000, "percentage": 61.16896694248365 }, … ], "installationsPerVersion": [ { "version": "2.0.3", "total": 141 }, … ], "installationsPercentagePerVersion": [ { "version": "2.2.7", "percentage": 0 }, … ], "currentInstalls": 89232, "trend": 2990 }, "title": "Jenkins Git plugin", "url": "http://updates.jenkins-ci.org/download/plugins/git/2.5.2/git.hpi", "version": "2.5.2", "wiki": { "url": "https://wiki.jenkins-ci.org/display/JENKINS/Git+Plugin" } }, … ], "limit": 3, "total": 7 }
=== GET /versions Retrieve unique required Jenkins versions in the plugin data. Sample Response
{ "limit": 226, "versions": [ "1.580.1", "1.625.3", "1.424", "1.609.3", "1.609.1", "1.398", "1.466", "1.480", "1.596.1", "1.532.3", "1.580", "1.509.4", "1.642.3", "1.580.3", "1.447", "1.532", "1.609", "1.509", "1.554.1", "1.480.3", "1.509.3", "1.392", ….. ] }
== Deployment to production This project is containerized via the `Dockerfile` that is located in the `deploy/` directory. The `Jenkinsfile` uses this `Dockerfile` create a container fit for deployment. Unfortunately, the build of this container must occur on a Jenkins cluster which is publicly inaccessible for security reasons. The private job polls SCM every 10 minutes. Deploying: . Have code merged to the master branch of this repository . Wait pateiently 10-15 minutes (coffee time!) . Verify that a new container tag has been published link:https://hub.docker.com/r/jenkinsciinfra/plugin-site-api/tags/[here]. . Submit a pull request to the link:https://github.com/jenkins-infra/jenkins-infra[jenkins-infra/charts] repository updating the `backend.image.tag` value to the latest container's tag (link:https://github.com/jenkins-infra/charts/blob/5e02db1ad84ac0634d256da155717b7664be1849/charts/plugin-site/values.yaml#L10[here]) . Once that is merged the changes will be live