CLI tool to check the code samples used in XML docs against the sniff

[jrfnl]

Is your feature request related to a problem?

The XML docs for sniffs should contain "Valid" and "Invalid" code samples.

Currently, one needs to manually check if the code samples as written in the XML docs would correctly (not) trigger the expected errors.

This is time consuming and error prone:

• A reviewer will often only do a visual check of the code samples and they could overlook something
• The docs might become out of date when the sniff gets updated.
• To do the check manually requires to copy & paste and manually adjust the code samples (remove <em> tags) and then do a manual run against the correct sniff etc.

Describe the solution you'd like

Ideally, there would be a CLI script which can be used both by PR reviewers in their CLI, as well as in CI to automatically check the docs against the sniff the docs are for.

Additional context (optional)

This could be done via a PHP script which:

• Reads the XML
• Creates a temporary file with the code samples.
• Runs the sniff over it (gathering the name of the sniff from the name of the docs)

This will no doubt run into some issues in practice, but those can be worked out once someone starts working on creating this script.
An example of an issue I expect might run into: to make this script usable by PHPCompatibility, it would need a way to pass the testVersion to the PHPCS run.

For now, this is just the rough idea for the feature, which I think would be a great new addition to DevTools as XML docs can now become outdated without anyone noticing, which is detrimental for end-users using the documentation.

• I intend to create a pull request to implement this feature.