Build reports of duration for test execution
Opened this issue · 0 comments
jakubdyszkiewicz commented
Description
so far I used this script
import json
import sys
import os
# Get the directory path from the command-line arguments
directory_path = sys.argv[1]
# Initialize an empty list to collect all test results
all_results = []
# Loop through all files in the specified directory
for filename in os.listdir(directory_path):
if filename.endswith(".json"): # Process only JSON files
file_path = os.path.join(directory_path, filename)
# Load each JSON report
with open(file_path) as f:
report = json.load(f)
# Check if the report is a list
if isinstance(report, list):
# Loop through each item in the list
for spec in report:
if 'SpecReports' in spec: # Adjust this based on the actual structure
for spec_report in spec['SpecReports']:
# Get the container hierarchy texts and join them with a separator
container_texts = spec_report.get('ContainerHierarchyTexts', [])
prefix = " > ".join(container_texts) if container_texts else ''
# Get the test name, type, and runtime
test_name = spec_report.get('LeafNodeText', 'Unnamed Test')
test_type = spec_report.get('LeafNodeType', 'Unknown Type')
full_test_name = f"{prefix} > {test_name}" if prefix else test_name
# Convert runtime from nanoseconds to seconds
duration_ns = spec_report.get('RunTime', 0)
duration_sec = duration_ns / 1_000_000_000 # Convert to seconds
# Store the result in the all_results list
all_results.append({
"test_name": full_test_name,
"test_type": test_type,
"duration_sec": duration_sec
})
# Sort all results by duration (longest first)
all_results.sort(key=lambda x: x['duration_sec'], reverse=True)
# Print the sorted results
for result in all_results:
print(f"Test: [{result['test_type']}] > {result['test_name']}, Duration: {result['duration_sec']:.3f} seconds")
and example result
❯❯❯ python3 test-time-dir.py /Users/jakub/Downloads/e2e-reports-universal-amd64-kind-flannel--0
Test: [It] > MeshRetry > should retry on GRPC connection failure, Duration: 155.078 seconds
Test: [It] > Retry > should retry on HTTP connection failure, Duration: 153.915 seconds
Test: [It] > External Services through Zone Egress > should not access external service when zone egress is down, Duration: 47.096 seconds
Test: [It] > Resilience > should mark data plane proxy as offline after it is killed forcefully when control-plane is down, Duration: 36.348 seconds
Test: [It] > Compatibility > connection between old and new DPP > from version: 2.9.0, Duration: 34.483 seconds
Test: [It] > Leader Election > should elect only one leader and drop the leader on DB disconnect, Duration: 26.758 seconds
Test: [It] > Gateway - Cross-mesh > when mTLS is enabled > should proxy HTTP requests from a different mesh, Duration: 24.444 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshTimeout > should target real MeshExternalService resource, Duration: 19.758 seconds
Test: [It] > MeshHealthCheck > HTTP to real MeshService > should mark host as unhealthy if it doesn't reply on health checks, Duration: 16.943 seconds
Test: [It] > HealthCheck panic threshold > should switch to panic mode and dismiss all requests, Duration: 15.837 seconds
Test: [It] > MeshHealthCheck panic threshold > should switch to panic mode and dismiss all requests, Duration: 15.743 seconds
Test: [It] > Traffic Logging > TrafficLog Logging to TCP > should send a traffic log to TCP port, Duration: 15.467 seconds
Test: [It] > Timeout > should reset the connection by timeout, Duration: 15.446 seconds
Test: [It] > Mesh External Services > without default policies > should route to mesh-external-service over tls, Duration: 15.084 seconds
Test: [It] > Reachable Services > should be able to connect to reachable services, Duration: 15.018 seconds
Test: [It] > HealthCheck > TCP with permissive mTLS > should mark host as unhealthy if it doesn't reply on health checks, Duration: 14.310 seconds
Test: [It] > Gateway > when a rate limit is configured > should be rate limited, Duration: 14.304 seconds
Test: [It] > Compatibility > connection between old and new DPP > from version: 2.7.8, Duration: 14.239 seconds
Test: [It] > MeshLoadBalancingStrategy > should use ring hash load balancing strategy, Duration: 14.164 seconds
Test: [It] > MeshHealthCheck > gRPC > should mark host as unhealthy if it doesn't reply to health checks, Duration: 13.823 seconds
Test: [It] > MeshHealthCheck > HTTP > should mark host as unhealthy if it doesn't reply on health checks, Duration: 13.579 seconds
Test: [It] > HealthCheck > TCP > should mark host as unhealthy if it doesn't reply on health checks, Duration: 13.536 seconds
Test: [It] > Compatibility > connection between old and new DPP > from version: 2.8.4, Duration: 13.486 seconds
Test: [It] > Tracing > should emit traces to jaeger, Duration: 13.392 seconds
Test: [It] > MeshTrace > should emit traces to jaeger, Duration: 13.300 seconds
Test: [It] > MeshHealthCheck > HTTP with MeshHTTPRoute > should mark host as unhealthy if it doesn't reply on health checks, Duration: 13.167 seconds
Test: [It] > Traffic Route > HTTP routing > should split by header and split by default, Duration: 13.026 seconds
Test: [It] > RateLimit > should limit echo server as external service, Duration: 12.978 seconds
Test: [It] > Mtls > enabling PERMISSIVE with no failed requests, Duration: 12.933 seconds
Test: [It] > Mtls > should support mode > STRICT server with TLS, Duration: 12.887 seconds
Test: [It] > MeshTimeout > should reset the connection by timeout > outbound timeout, Duration: 12.767 seconds
Test: [It] > MeshHealthCheck > TCP > should mark host as unhealthy if it doesn't reply on health checks, Duration: 12.762 seconds
Test: [It] > Mtls > should support mode > PERMISSIVE server with TLS, Duration: 12.744 seconds
Test: [It] > Mtls > should support mode > PERMISSIVE server without TLS, Duration: 12.743 seconds
Test: [It] > Membership > should take into account membership when dp is connecting to the CP, Duration: 12.720 seconds
Test: [It] > Mtls > should support mode > STRICT server without TLS, Duration: 12.709 seconds
Test: [It] > MeshHealthCheck > TCP with permissive mTLS > should mark host as unhealthy if it doesn't reply to health checks when Permissive mTLS enabled, Duration: 12.695 seconds
Test: [It] > Service Probes > should mark DP as unhealthy when listeners are draining, Duration: 12.483 seconds
Test: [It] > Traffic Permission > should allow the traffic with default traffic permission, Duration: 12.172 seconds
Test: [It] > Traffic Route > HTTP routing > should split traffic between internal and external destinations, Duration: 11.942 seconds
Test: [It] > MeshTimeout > should reset the connection by timeout > inbound timeout, Duration: 11.874 seconds
Test: [It] > HealthCheck > HTTP > should mark host as unhealthy if it doesn't reply on health checks, Duration: 11.729 seconds
Test: [It] > MeshAccessLog > should log outgoing traffic, Duration: 11.350 seconds
Test: [It] > MeshService > should switch to permissive mTLS without drop of the traffic, Duration: 11.152 seconds
Test: [It] > Mesh Traffic Permission > should be able to allow the traffic with permissive mTLS (tcp), Duration: 11.110 seconds
Test: [It] > Mesh Traffic Permission > should allow the traffic with meshtrafficpermission based on MeshService (http), Duration: 10.370 seconds
Test: [It] > External Services > without default policies > should route to external-service, Duration: 10.187 seconds
Test: [It] > MeshTrace > should emit MeshGateway traces to jaeger, Duration: 10.111 seconds
Test: [It] > Projected Service Account Token > should connect to restarted control plane with new token without dp restart, Duration: 10.105 seconds
Test: [It] > Virtual Outbound > should add hostnames for individual instances, Duration: 10.098 seconds
Test: [It] > Gateway - Cross-mesh > when mTLS is enabled > doesn't allow HTTP requests from outside the mesh, Duration: 10.001 seconds
Test: [It] > Traffic Route > should access all instances of the service, Duration: 9.709 seconds
Test: [It] > GRPC > should emit stats from the server, Duration: 9.488 seconds
Test: [It] > External Services > with default policies > should route to external-service over tls, Duration: 9.381 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshHTTPRoute > should route to other endpoint, Duration: 9.155 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshRetry > should retry on error, Duration: 8.743 seconds
Test: [It] > External Services > without default policies > should route to external-service over tls, Duration: 8.588 seconds
Test: [It] > Traffic Route > HTTP routing > should route matching by path, Duration: 8.573 seconds
Test: [It] > GRPC > MeshHTTPRoute does split the traffic between two gRPC services, Duration: 8.572 seconds
Test: [It] > MeshAccessLog > should log outgoing traffic to real MeshService, Duration: 8.556 seconds
Test: [It] > MeshService > should be able to create and use MeshService with HostnameGenerator, Duration: 8.519 seconds
Test: [It] > MeshAccessLog > should log incoming traffic, Duration: 8.495 seconds
Test: [It] > MeshAccessLog > should log outgoing traffic with JSON formatting, Duration: 8.463 seconds
Test: [It] > MeshAccessLog > should log outgoing passthrough traffic, Duration: 8.434 seconds
Test: [It] > Mesh Traffic Permission > should allow the traffic with traffic permission based on non standard tag, Duration: 8.270 seconds
Test: [It] > MeshTLS > should change single dataplane to Permissive, Duration: 7.990 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshTCPRoute > should route to other backend, Duration: 7.913 seconds
Test: [It] > Matching > should both fault injections with the same destination proxy, Duration: 7.867 seconds
Test: [It] > Traffic Route > should split traffic between internal and external services, Duration: 7.688 seconds
Test: [It] > MeshRateLimit > should limit all sources, Duration: 7.538 seconds
Test: [It] > Traffic Route > HTTP routing > should route matching by header, Duration: 7.531 seconds
Test: [It] > MeshRetry > should retry on HTTP connection failure, Duration: 7.285 seconds
Test: [It] > RateLimit > should limit per source, Duration: 6.969 seconds
Test: [It] > MeshFaultInjection > should be affected by fault and return > 402 when requests from the demo-client-blocked, Duration: 6.626 seconds
Test: [SynchronizedBeforeSuite] > , Duration: 6.619 seconds
Test: [SynchronizedBeforeSuite] > , Duration: 6.403 seconds
Test: [It] > MeshTLS > should set cypher and version, Duration: 6.333 seconds
Test: [It] > Mesh Traffic Permission > should allow the traffic with meshtrafficpermission based on MeshService (tcp), Duration: 5.998 seconds
Test: [It] > Mesh External Services > without default policies > should route to mesh-external-service with same hostname but different ports, Duration: 5.964 seconds
Test: [It] > ProxyTemplate > should add a header using Lua filter, Duration: 5.794 seconds
Test: [It] > Reachable Services > should not be able to non reachable services, Duration: 5.752 seconds
Test: [It] > MeshProxyPatch > should add a header using Lua filter, Duration: 5.586 seconds
Test: [It] > DP Auth > should revoke token and kick out dataplane proxy out of the mesh, Duration: 5.538 seconds
Test: [It] > MeshTLS > should tls version for 1.3, Duration: 5.311 seconds
Test: [It] > Traffic Permission > should use most specific traffic permission, Duration: 5.072 seconds
Test: [It] > Applications Metrics > should scrape metrics defined in mesh and not fail when defined service doesn't exist, Duration: 5.015 seconds
Test: [It] > Zone Egress > Fault Injection > should inject faults for external service, Duration: 4.915 seconds
Test: [It] > Mesh External Services > without default policies > should route to mesh-external-service, Duration: 4.900 seconds
Test: [It] > Offline Auth > should use user-token generated offline, Duration: 4.591 seconds
Test: [It] > Gateway > when mTLS is disabled > should proxy simple HTTP requests, Duration: 4.524 seconds
Test: [It] > MeshFaultInjection > should delay responses for demo-client-timeout, Duration: 4.451 seconds
Test: [It] > MeshAccessLog > supports logging traffic to an ExternalService using MeshService (without ZoneIngress), Duration: 4.445 seconds
Test: [It] > Zone Egress > Proxy > should access external service through zoneegress, Duration: 4.370 seconds
Test: [It] > MeshAccessLog > should log traffic from MeshGateway, Duration: 4.354 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshAccessLog > should log outgoing traffic to MeshExternalService, Duration: 4.179 seconds
Test: [It] > Offline Auth > should use a dp-token generated offline, validated with a non-mesh scoped key, Duration: 3.991 seconds
Test: [It] > Prometheus Metrics > should expose metrics through https, Duration: 3.560 seconds
Test: [It] > MeshRetry > should retry on HTTP connection failure with real MeshService, Duration: 3.398 seconds
Test: [It] > MeshTLS > should change single dataplane to Strict, Duration: 3.278 seconds
Test: [It] > Inspect > should return envoy config_dump, Duration: 3.271 seconds
Test: [It] > External Services > with default policies > should route to external-service, Duration: 2.951 seconds
Test: [It] > Offline Auth > should use dp-token generated offline, Duration: 2.897 seconds
Test: [It] > MeshRetry > should retry on HTTP connection failure applied on MeshHTTPRoute, Duration: 2.875 seconds
Test: [It] > Gateway > when a MeshFaultInjection is configured > should return custom error code for all requests from both gateways, Duration: 2.787 seconds
Test: [It] > Gateway > when targeting an external service > should proxy simple HTTP requests, Duration: 2.678 seconds
Test: [It] > Gateway > when a MeshRateLimit is configured > should be rate limited, Duration: 2.632 seconds
Test: [It] > Zone Egress > MeshFaultInjection > should inject faults for external service, Duration: 2.627 seconds
Test: [It] > Traffic Permission > should allow the traffic with traffic permission based on non standard tag, Duration: 2.542 seconds
Test: [It] > Traffic Route > HTTP routing > should modify host, Duration: 2.469 seconds
Test: [It] > Zone Egress > Rate Limit > should rate limit requests to external service, Duration: 2.270 seconds
Test: [It] > Service Probes > should update dataplane.inbound.health of unhealthy test-server, Duration: 2.264 seconds
Test: [SynchronizedAfterSuite] > , Duration: 2.105 seconds
Test: [It] > Mesh Traffic Permission > should be able to allow the traffic with permissive mTLS (http), Duration: 2.018 seconds
Test: [It] > External Services > without default policies > should route to external-service with same hostname but different ports, Duration: 2.012 seconds
Test: [It] > MeshRateLimit > should limit tcp connections, Duration: 1.846 seconds
Test: [It] > Applications Metrics > should allow to define and expose localhost bound server, Duration: 1.843 seconds
Test: [It] > Gateway - Cross-mesh > when mTLS is enabled > should be reachable without transparent proxy, Duration: 1.836 seconds
Test: [It] > Traffic Route > HTTP routing > should modify path, Duration: 1.818 seconds
Test: [It] > Traffic Route > HTTP routing > should modify headers, Duration: 1.783 seconds
Test: [It] > Traffic Permission > should allow the traffic with traffic permission based on kuma.io/service tag, Duration: 1.744 seconds
Test: [It] > Traffic Permission > should allow the traffic with traffic permission based on many tags, Duration: 1.739 seconds
Test: [It] > Gateway > when targeting a HTTPS gateway listener > should proxy simple HTTPS requests with Host header, Duration: 1.711 seconds
Test: [It] > Traffic Route > should route 100 percent of the traffic to the different service, Duration: 1.640 seconds
Test: [It] > Prometheus Metrics > should expose metrics through http when the certificate wasn't provided, Duration: 1.530 seconds
Test: [It] > Gateway > when a MeshFaultInjection is configured > should return custom error code for all requests, Duration: 1.475 seconds
Test: [It] > Gateway > when targeting a HTTPS gateway listener > should proxy simple HTTPS requests without hostname, Duration: 1.429 seconds
Test: [It] > External Services > with default policies > should route to external-service with same hostname but different ports, Duration: 1.367 seconds
Test: [It] > DP Auth > should not be able to override someone else Dataplane, Duration: 1.233 seconds
Test: [It] > DP Auth > should be able to override old Dataplane of same service, Duration: 1.216 seconds
Test: [It] > Zone Egress > MeshRateLimit > should rate limit requests to external service, Duration: 1.196 seconds
Test: [It] > Gateway > really uses mTLS, Duration: 1.078 seconds
Test: [It] > MeshFaultInjection > should be affected by fault and return > 421 when requests from the demo-client-blocked, Duration: 0.893 seconds
Test: [It] > Applications Metrics > should use only configuration from dataplane, Duration: 0.787 seconds
Test: [It] > Gateway > when mTLS is enabled > should proxy simple HTTP requests, Duration: 0.497 seconds
Test: [It] > Inspect > should return clusters, Duration: 0.321 seconds
Test: [It] > User Auth > should generate user for group member and log in, Duration: 0.209 seconds
Test: [It] > InterCP Server > should run inter cp server, Duration: 0.181 seconds
Test: [It] > Gateway - Cross-mesh > when mTLS is enabled > HTTP requests to a non-crossMesh gateway should still be proxied, Duration: 0.173 seconds
Test: [It] > Gateway - Cross-mesh > when mTLS is enabled > should proxy HTTP requests from the same mesh, Duration: 0.169 seconds
Test: [It] > MeshFaultInjection > should not be affected by any fault, Duration: 0.166 seconds
Test: [It] > MeshFaultInjection > should be affected by fault and return > 421 when requests from the demo-client-blocked, Duration: 0.166 seconds
Test: [It] > User Auth > should generate user for group admin and log in, Duration: 0.157 seconds
Test: [It] > GRPC > should emit stats from the client, Duration: 0.156 seconds
Test: [It] > Applications Metrics > should override mesh configuration with dataplane configuration, Duration: 0.153 seconds
Test: [It] > MeshFaultInjection > should be affected by fault and return > 421 when requests from any client are blocked, Duration: 0.148 seconds
Test: [It] > Gateway > applying ProxyTemplate > shouldn't error with gateway-proxy import, Duration: 0.086 seconds
Test: [It] > Inspect > should return stats, Duration: 0.079 seconds
Test: [It] > Apis > gets secret > global secret, Duration: 0.075 seconds
Test: [It] > Apis > gets secret > mesh secret, Duration: 0.044 seconds
Test: [ReportAfterSuite] > universal after suite, Duration: 0.028 seconds
Test: [It] > Apis > works with /policies, Duration: 0.005 seconds
Test: [It] > User Auth > should ignore auth data on unauthorized endpoints > index, Duration: 0.001 seconds
Test: [It] > User Auth > should ignore auth data on unauthorized endpoints > gui, Duration: 0.001 seconds
Test: [It] > Apis > gets k8s version of default mesh, Duration: 0.001 seconds
Test: [It] > Apis > gets universal version of default mesh, Duration: 0.000 seconds
Test: [It] > Apis > works with /, Duration: 0.000 seconds
Test: [DeferCleanup (Suite)] > , Duration: 0.000 seconds
Test: [SynchronizedAfterSuite] > , Duration: 0.000 seconds
Test: [It] > Mtls > should enforce traffic permissions > builtin CA, Duration: 0.000 seconds
Test: [It] > Mtls > should enforce traffic permissions > Provided CA with Root CA, Duration: 0.000 seconds
Test: [It] > Mtls > should enforce traffic permissions > Provided CA with Intermediate CA, Duration: 0.000 seconds
Test: [It] > External Services > should check allow negotiation, Duration: 0.000 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshHealthCheck > should target real MeshExternalService resource, Duration: 0.000 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshCircuitBreaker > should target real MeshExternalService resource, Duration: 0.000 seconds
Test: [It] > Mesh External Services > MeshExternalService with MeshLoadBalancingStrategy > should target real MeshExternalService resource, Duration: 0.000 seconds
Test: [It] > Transparent Proxy > should be able to re-install transparent proxy, Duration: 0.000 seconds
It would be nice to build as an action, similar to project-report.