Explore API Data for Core Integration

Question

Explore API Data for Core Integration

striebwj opened this issue 3 years ago · 9 comments

striebwj commented 3 years ago

Explore what data CP site send to WP.org for plugins, and how WP.org responds.

Then compare to the CP Directory API endpoint to verify we have the same data available.

CP Directory Plugin Endpoints:

Answer 1 · 2021-09-01T12:49:23.000Z

This issue has been mentioned on ClassicPress Forums. There might be relevant details there:

https://forums.classicpress.net/t/make-cp-repo-ship-plugin-updates/3501/41

Answer 2 · 2021-09-02T08:52:01.000Z

Seems that the easiest way is to sniff wp_remote_post in wp-includes/http.php.

Changed this way:

function wp_remote_post($url, $args = array()) {
	$xsx_option = get_option( 'xsx-sniff', [] );
	$xsx_option[ date( 'Ymd Y-m-d H:i:s' ) . '  URL' ] = $url;
	$xsx_option[ date( 'Ymd Y-m-d H:i:s' ) . ' ARGS' ] = $args;
	$http = _wp_http_get_object();
	$retval = $http->post( $url, $args );
	$xsx_option[ date( 'Ymd Y-m-d H:i:s' ) . ' RESP' ] = $retval;
	update_option( 'xsx-sniff', $xsx_option );
	return $retval;
}

Then you can look at the logs dumping xsx-sniff in the place you prefer.
Using the Transient Inspector of Update Manager makes it easier to trigger the right request.

Answer 3 · 2021-10-31T19:02:54.000Z

In lieu of posting in the community forum (I'm still deliberating whether to join the forum or not), I'll just engage here, for the time being.

Explore what data CP site send to WP.org for plugins, and how WP.org responds.

As indicated by @xxsimoxx, the request method is post⁵ and the data³ sent, dependent on the situation, is comprised of action & args. In a bit more detail, the data that CP sends is dependent on the trigger/action for a request, of which, will also determine which URL is used for such requests. There are basically 4 different actions (query_plugins, plugin_information, hot_tags, hot_categories) which utilize the same URL and a plugins update request which utilizes a different URL. IOW, calls to:

plugin_api function:¹ hxxps://api.wordpress.org/plugins/info/1.0/ (query_plugins, plugin_information, hot_tags, hot_categories)
wp_update_plugins function:⁸ hxxps://api.wordpress.org/plugins/update-check/1.1/

As an example (default [popular tab] indicated for the sake of simplicity), the 'Add New' submenu for plugins, when clicked:

action: (string) query_plugins
args: (array) page, per_page, fields, locale, installed_plugins, browse

args in more detail:

page: (int) 1
per_page: (int) 30
locale: (string) return value of the function get_user_locale()
fields: (array) last_updated, icons, active_install - note: all values equate to true
installed_plugins: (array) array of keys of the return value from the class method get_installed_plugins()
browse: (string) popular

The response from the server, after some magic has occurred (unserializing, if needed), is an object consisting of:

info: (array) page, pages, results
plugins: array of plugin objects consisting of following top level items/keys:

name, slug, version, author, author_profile, requires, tested, requires_php, rating, ratings, num_ratings, support_threads, support_threads_resolved, active_installs, downloaded, last_updated, added, homepage, sections, short_description, download_link, screenshots, tags, versions, donate_link, icons

N.B.: Keys/items in bold and italicized are missing/unclear in official documentation of what should be returned as described by fields.¹ Fortunately, through an unofficial documentation effort, these keys/items are described as to the values returned but, not whether these key/items can be sent as field arguments.⁴

Additionally, some of the top level items/keys contain an array of items/keys (associative array), reserved for discussion at a later time.

Then compare to the CP Directory API endpoint to verify we have the same data available.

Some of the data is the same, albeit, some may have different labeling (nomenclature). Additionally, some data is similar, for example:

WP - info is similar to CP - meta and WP - plugins is similar to CP - data

In short, the CP Directory API endpoint does not have the same data available, only some of the data. Which can be illustrated as one level of comparison as it pertains to similarity:

WP - plugins:⁶
name, slug, version, author, author_profile, requires, tested, requires_php, rating, ratings, num_ratings, support_threads, support_threads_resolved, active_installs, downloaded, last_updated, added, homepage, sections, short_description, download_link, screenshots, tags, versions, donate_link, icons

CP - data:²
name, description, developer, slug, web_url, minimum_wp_version, minimum_cp_version, current_version, latest_cp_compatible_version, git_provider, repo_url, download_link, comment, type, published_at

Notwithstanding what may appear to be differences, we can match up items such as description² (CP) to be the same as short_description¹ (WP), as well as, some others. Furthermore, there are a few direct matches such as name and slug. However, the CP implementation of slug, may present itself to be a bit troublesome. What may help is to have a few examples of WP responses, as well as what may be troublesome with the CP response, of which, I'm contemplating whether or not to submit a PR to accomplish this (data files). Nevertheless, what I find peculiar, is that information that would match up directly with WP is specified in Plugin readme.txt Requirements.⁷ yet, apparently, this information is not currently included as return information from the ClassicPress Directory API.

CP Directory Plugin Endpoints:

https://directory.classicpress.net/api/plugins

https://directory.classicpress.net/api/plugins/classic-commerce

References:

ClassicPress Functions, plugins_api( string $action, array|object $args = array() ). Retrieved 14 Sep 2021.
ClassicPress Directory API, Plugins API Endpoint. Retrieved 14 Sep 2021.
ENVATO TUTS+, Communicating With the WordPress.org Plugin API (retrieved 14 Sep 2021); WordPress.org API, Plugins (retrieved 14 Sep 2021).
blog://dd32.id.au/, WordPress.org Plugin Information API Docs. Retrieved 14 Sep 2021.
ClassicPress-release/wp-admin/includes/ plugin-install.php. Retrieved 14 Sep 2021.
blog://dd32.id.au/, WordPress.org Plugin Information API Docs (retrieved 14 Sep 2021);
ClassicPress Functions, plugins_api( string $action, array|object $args = array() ) (retrieved 14 Sep 2021).
ClassicPress Plugin Guidelines, Plugin readme.txt Requirements. Retrieved 16 Sep 2021.
ClassicPress Functions, wp_update_plugins( array $extra_stats = array() ). Retrieved 14 Sep 2021.

Answer 4 · 2021-11-21T15:20:18.000Z

This issue has been mentioned on ClassicPress Forums. There might be relevant details there:

https://forums.classicpress.net/t/plugin-directory-api-integration/3715/11

Answer 5 · 2021-11-21T15:58:44.000Z

Nevertheless, what I find peculiar, is that information that would match up directly with WP is specified in Plugin readme.txt Requirements.7 yet, apparently, this information is not currently included as return information from the ClassicPress Directory API.

Be careful about readme.txt. For WP, it is only used for the WP repo. Core WP does not use readme.txt. It gets the same information from the plugin/theme header instead.
It should be decided if this should continue the same way for CP. It could be the difference between repository and directory.

Additionally, the way the versions are compared (which side is doing the comparison) needs to be better for CP than the way WP does it (users are not informed of an update if WP or PHP versions don't match).

Answer 6 · 2021-11-22T00:23:51.000Z

Be careful about readme.txt. For WP, it is only used for the WP repo. Core WP does not use readme.txt. It gets the same information from the plugin/theme header instead. It should be decided if this should continue the same way for CP. It could be the difference between repository and directory.

When you reference Core WP, you must mean WP 5.8, whereby, the functions validate_plugin_requirements and validate_theme_requirements were changed from parsing the readme.txt to parsing the plugin header and stylesheet header, respectively, for Requires at least and Requires PHP. BTW, neither of these indicated functions currently exists in CP nor WP 4.9.

Additionally, the way the versions are compared (which side is doing the comparison) needs to be better for CP than the way WP does it (users are not informed of an update if WP or PHP versions don't match).

There are a number of shortcomings (version comparisons) for both WP and CP, in this regard. As to CP, these shortcomings can be flushed out and enhanced. To note, there are situations whereby CP (inherited from a flaw with WP 4.9) indicates a WP plugin to be incompatible when it is actually compatible due to a version comparison shortcoming (as viewed from admin menu trail plugins -> Add New or as a relative path wp-admin/plugin-install.php). In and of itself, this should be identified as a core issue and tracked as part of the, for lack of better terminology, "Plugin Directory Project", as well as, all other surfaced issues. I'm not sure of the best method to provide visibility, tracking/linking/breadcrumbs to related info (indicated issue TBD, when appropriate).

Neither here nor there and in context to requirements, issue #6 suggests that there is a form with a number of fields that populate the plugin directory database and that some of the fields are not requirements. Since I haven't joined the community forum, I don't know what this form offers as user input fields, so to speak. However, I have a general sense of the form fields from the Plugins API Endpoint info. Given this and other sources (plugin header and readme.txt), a simple table can be constructed that looks like the following (I've flattened the 2 associative arrays [developer and type] to provide a 2 dimensional view of the Plugins API Endpoint):

	plugin_hdr	readme_txt
name	x	x	Plugin Name (plugin_hdr), Plugin Name (readme_txt)
description	x	x	Description (plugin_hdr), Short Description (readme_txt)
developer=>name	x		Author (plugin_hdr)
developer=>slug
developer=>web_url
developer=>username
developer=>website	x		Author URI (plugin_hdr)
developer=>published_at
slug
web_url
minimum_wp_version		x	Requires (readme_txt)
minimum_cp_version
current_version	x		Version (plugin_hdr)
latest_cp_compatible_version
git_provider
repo_url	x		Plugin URI (plugin_hdr)
download_link
comment		@	Other Notes (readme_txt)
type=>key
type=>value
type=>description
published_at

To make a long story shorter, I'll skip legend for the symbols used and just posit a notion that ideally, information that can be obtained from sources other than user input, might present itself as the better path/method. The objective of the table is to provide a talking point in which to satisfy what issue #6 is trying to establish, show sources for info and how they map to the Plugins API Endpoint and reveal what is missing as it pertains to the Endpoint, e.g., Requires PHP amd Tested are missing from the Endpoint.

If I may, I'm going to skip over a lot of detail I would normally cover and get to what the community may want to achieve. Mind you, this is just from my perspective (an alternative to the proposed use of Customizer) and it looks like the 3 following screenshots (note, this is currently in the form of a plugin, but ideally the modifications should be integrated into core; meaning the plugin is for experimental/POC purposes):

Answer 7 · 2022-11-24T20:25:13.000Z

The new directory integration will be built separately from WordPress.org integration, so they will work alongside each other for a while. There's no need for this issue anymore at this time.

Answer 8 · 2022-11-24T20:54:48.000Z

Closing the issue as complete seems a bit optimistic. You didn't explain why this is not needed, either.

Answer 9 · 2022-11-24T22:05:57.000Z

There's nothing new to "explore" here. New directory API returns different data and core integration will utilize this API. As discussed, we don't plan to remove WP.org integration but instead add CP directory integration separately. So both will be available. This will most likely start with Beda's directory plugin utilizing the new API. Directory API will not be changed to match WP.org structure, so the following point in this issue is no longer relevant:

Then compare to the CP Directory API endpoint to verify we have the same data available.

Core integration should be discussed as an issue in the core repository.

This issue was created to help figure out what type of data WP.org returns. We did. It's available in this issue. So the issue served its purpose.

Any changes to the data returned by the directory API should be separate issues now.