The total number of test cases in conformance_log.xml is inconsistent with the number in the openxr_conformance.xml

In v1.0.27.0, we found that the total number of test cases is inconsistent with the number in the openxr_conformance.xml.
For example, we run non-interactive test cases on the Android platform.

Test options:
   graphicsPlugin: OpenGLES
   formFactor: Hmd
   hands: both
   environmentBlendMode: Opaque
   viewConfiguration: Stereo
   enabledAPILayers:
   enabledInstanceExtensions:
   invalidHandleValidation: no
   fileLineLoggingEnabled: yes
   debugMode: no

The number of test cases is 61 in conformance_log.xml, but the number of test cases listed in openxr_conformance.xml is 57.
The below 4 test cases are executed in conformance_log.xml, but they are not listed in openxr_conformance.xml.

XR_KHR_convert_timespec_time //It is enabled(or disabled) by macro XR_USE_TIMESPEC
XR_KHR_headless //It is enabled(or disabled) by macro XR_KHR_headless
XR_EXT_performance_settings //It is an empty test case.
XR_EXT_local_floor

We checked the source code and found that this is caused by statistical rules. The current test report openxr_conformance.xml is prone to confusion. From our perspective, it will be better if we use the defined macro SKIP to do some optimization.

Here is the detailed analysis:

The rule that the test case will be written in openxr_conformance.xml:

Has assertions or
stdOut is not empty or
stdErr is not empty.

OpenXR-CTS/src/conformance/framework/catch_reporter_cts.cpp

Line 217 in 3d02d70

if (sectionNode.hasAnyAssertions() || !sectionNode.stdOut.empty() || !sectionNode.stdErr.empty()) {

For test case XR_KHR_convert_timespec_time, the macro XR_USE_TIMESPEC is not defined, it is the equivalent of an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

Proposal:

TEST_CASE("XR_KHR_convert_timespec_time", "")
{
    bool executed = false
    #ifdef XR_USE_TIMESPEC
            ......
            ......
    executed = true
            ......
            ......
    #endif  // XR_USE_TIMESPEC
    If (!executed) {
        SKIP("XR_KHR_convert_timespec_time not supported");
    }
}

For test case XR_KHR_headless, the macro is not defined, it is the equivalent of an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

OpenXR-CTS/src/conformance/conformance_test/test_XR_KHR_headless.cpp

Line 32 in 3d02d70

#ifdef XR_KHR_headless

Proposal:

TEST_CASE("XR_KHR_headless", "")
{
    bool executed = false
    #ifdef XR_KHR_headless
            ......
            ......
    executed = true
            ......
            ......
    #endif  // XR_KHR_headless
    If (!executed) {
    SKIP("XR_KHR_headless not supported");
    }
}

For test case XR_EXT_performance_settings, it is an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

OpenXR-CTS/src/conformance/conformance_test/test_XR_EXT_performance_settings.cpp

Line 22 in 3d02d70

TEST_CASE("XR_EXT_performance_settings", "")

Proposal:

TEST_CASE("XR_EXT_performance_settings", "")
{
    // XrResult xrPerfSettingsSetPerformanceLevelEXT(XrSession session, XrPerfSettingsDomainEXT domain,
    //              XrPerfSettingsLevelEXT level);
    SKIP("XR_EXT_performance_settings not supported");
 }

For test case XR_EXT_local_floor, it is the equivalent of an empty test case if not supported, it doesn’t meet the rule that writes in openxr_conformance.xml.

OpenXR-CTS/src/conformance/conformance_test/test_XR_EXT_local_floor.cpp

Line 33 in 3d02d70

if (!globalData.IsInstanceExtensionSupported(XR_EXT_LOCAL_FLOOR_EXTENSION_NAME)) {

Proposal:

TEST_CASE("XR_EXT_local_floor", "")
{
    GlobalData& globalData = GetGlobalData();
    if (!globalData.IsInstanceExtensionSupported(XR_EXT_LOCAL_FLOOR_EXTENSION_NAME)) {
        SKIP(XR_EXT_LOCAL_FLOOR_EXTENSION_NAME  " not supported");
        return;
    }
        ......
        ......
        ......
}

An issue (number 2072) has been filed to correspond to this issue in the internal Khronos GitLab (Khronos members only: KHR:openxr/openxr#2072 ), to facilitate working group processes.

This GitHub issue will continue to be the main site of discussion.

Thanks for this report. Some of the issues you identified (always empty tests) are resolved in the 1.0.29.0 release just published. The next release will have a few more small fixes.

This has been fixed in the 1.0.30.0 release. A newer release is now the latest, I accidentally missed closing this issue when releasing 1.0.30.0.