KhronosGroup/OpenXR-CTS

The total number of test cases in conformance_log.xml is inconsistent with the number in the openxr_conformance.xml

Closed this issue · 3 comments

In v1.0.27.0, we found that the total number of test cases is inconsistent with the number in the openxr_conformance.xml.
For example, we run non-interactive test cases on the Android platform.

Test options:
   graphicsPlugin: OpenGLES
   formFactor: Hmd
   hands: both
   environmentBlendMode: Opaque
   viewConfiguration: Stereo
   enabledAPILayers:
   enabledInstanceExtensions:
   invalidHandleValidation: no
   fileLineLoggingEnabled: yes
   debugMode: no

The number of test cases is 61 in conformance_log.xml, but the number of test cases listed in openxr_conformance.xml is 57.
The below 4 test cases are executed in conformance_log.xml, but they are not listed in openxr_conformance.xml.

  1. XR_KHR_convert_timespec_time //It is enabled(or disabled) by macro XR_USE_TIMESPEC
  2. XR_KHR_headless //It is enabled(or disabled) by macro XR_KHR_headless
  3. XR_EXT_performance_settings //It is an empty test case.
  4. XR_EXT_local_floor

We checked the source code and found that this is caused by statistical rules. The current test report openxr_conformance.xml is prone to confusion. From our perspective, it will be better if we use the defined macro SKIP to do some optimization.

Here is the detailed analysis:

  • The rule that the test case will be written in openxr_conformance.xml:
  1. Has assertions or
  2. stdOut is not empty or
  3. stdErr is not empty.
    if (sectionNode.hasAnyAssertions() || !sectionNode.stdOut.empty() || !sectionNode.stdErr.empty()) {
  • For test case XR_KHR_convert_timespec_time, the macro XR_USE_TIMESPEC is not defined, it is the equivalent of an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

Proposal:

TEST_CASE("XR_KHR_convert_timespec_time", "")
{
    bool executed = false
    #ifdef XR_USE_TIMESPEC
            ......
            ......
    executed = true
            ......
            ......
    #endif  // XR_USE_TIMESPEC
    If (!executed) {
        SKIP("XR_KHR_convert_timespec_time not supported");
    }
}
  • For test case XR_KHR_headless, the macro is not defined, it is the equivalent of an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

Proposal:

TEST_CASE("XR_KHR_headless", "")
{
    bool executed = false
    #ifdef XR_KHR_headless
            ......
            ......
    executed = true
            ......
            ......
    #endif  // XR_KHR_headless
    If (!executed) {
    SKIP("XR_KHR_headless not supported");
    }
}
  • For test case XR_EXT_performance_settings, it is an empty test case, so it doesn’t meet the rule that writes in openxr_conformance.xml.

TEST_CASE("XR_EXT_performance_settings", "")

Proposal:

TEST_CASE("XR_EXT_performance_settings", "")
{
    // XrResult xrPerfSettingsSetPerformanceLevelEXT(XrSession session, XrPerfSettingsDomainEXT domain,
    //              XrPerfSettingsLevelEXT level);
    SKIP("XR_EXT_performance_settings not supported");
 }
TEST_CASE("XR_EXT_local_floor", "")
{
    GlobalData& globalData = GetGlobalData();
    if (!globalData.IsInstanceExtensionSupported(XR_EXT_LOCAL_FLOOR_EXTENSION_NAME)) {
        SKIP(XR_EXT_LOCAL_FLOOR_EXTENSION_NAME  " not supported");
        return;
    }
        ......
        ......
        ......
}



An issue (number 2072) has been filed to correspond to this issue in the internal Khronos GitLab (Khronos members only: KHR:openxr/openxr#2072 ), to facilitate working group processes.

This GitHub issue will continue to be the main site of discussion.

Thanks for this report. Some of the issues you identified (always empty tests) are resolved in the 1.0.29.0 release just published. The next release will have a few more small fixes.

This has been fixed in the 1.0.30.0 release. A newer release is now the latest, I accidentally missed closing this issue when releasing 1.0.30.0.