Error: ❌ No test results found for JUnit Test Report — but `junit.xml` looks reasonable to me
aviflax opened this issue · 7 comments
aviflax commented
This tool looks great — exactly what I was looking for — thank you!
Unfortunately I’m having trouble getting it to work.
Here’s what I’m seeing:
the junit.xml written by my test runner
<?xml version='1.0' encoding='UTF-8'?>
<testsuites>
<testsuite errors="0" package="" tests="24" name="unit" time="0.000000" hostname="localhost" id="0" timestamp="2024-03-12T19:45:15" failures="2">
<properties/>
<testcase name="promptbench.platform.web.api.handlers.iteration-test/post-coll" classname="promptbench.platform.web.api.handlers.iteration-test" time="0.000000" line="24" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/api/handlers/iteration_test.clj">
</testcase>
<testcase name="promptbench.platform.web.placeholder-ui.handlers.iteration-handler-test/post-action-run" classname="promptbench.platform.web.placeholder-ui.handlers.iteration-handler-test" time="0.000000" line="20" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/placeholder_ui/handlers/iteration_handler_test.clj">
</testcase>
<testcase name="promptbench.platform.web.placeholder-ui.handlers.iteration-handler-test/post-action-clone" classname="promptbench.platform.web.placeholder-ui.handlers.iteration-handler-test" time="0.000000" line="55" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/placeholder_ui/handlers/iteration_handler_test.clj">
</testcase>
<testcase name="promptbench.models.openai-test/invoke-openai" classname="promptbench.models.openai-test" time="0.000000" line="30" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/models/openai_test.clj">
<failure message="[:fail] expected: (< 24 (-> response :usage :completion_tokens) 34). actual: (not (< 24 11 34))" type="assertion failure: <">
FAIL in promptbench.models.openai-test/invoke-openai (openai_test.clj:56)
happy paths simple case
expected: (< 24 (-> response :usage :completion_tokens) 34)
actual: (not (< 24 11 34))
Test output
2024-03-12 19:44:49,686 [main] DEBUG promptbench.models.openai - ------------------
PARAMS:
{:model "gpt-3.5-turbo",
:messages
[{:role "system", :content "You are Sherlock Holmes and you will be asked a question by Watson."}
{:role "user", :content "The cheese is old and moldy, where is the bathroom?"}],
:max_tokens 32,
:temperature nil}
2024-03-12 19:44:49,690 [main] DEBUG promptbench.models.openai - ------------------
OPTS:
{:api-key "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",
:request {:throw-exceptions? false},
:trace
#object[promptbench.models.openai$eval45476$fn__45478$fn__45482 0x32d87fe1 "promptbench.models.openai$eval45476$fn__45478$fn__45482@32d87fe1"]}
2024-03-12 19:44:49,699 [main] DEBUG promptbench.models.openai - ------------------
RESPONSE:
{:id "chatcmpl-922LJE88rTz65Z0FCfRBD9nhypgqS",
:object "chat.completion",
:created 1710272689,
:model "gpt-3.5-turbo-0125",
:choices
[{:index 0,
:message {:role "assistant", :content "Holmes, where is the bathroom in this house?"},
:logprobs nil,
:finish_reason "stop"}],
:usage {:prompt_tokens 38, :completion_tokens 11, :total_tokens 49},
:system_fingerprint "fp_4f0b692a78"}
2024-03-12 19:44:49,801 [main] DEBUG promptbench.models.openai - ------------------
RAW-RESPONSE:
{:request-time 929,
:request
{:user-info nil,
:throw-exceptions? false,
:as :text,
:headers
{"Authorization" "***;,
"Content-Type" "application/json",
"Accept" "application/json",
"accept-encoding" "gzip, deflate"},
:server-port nil,
:middleware
[#object[hato.middleware$wrap_request_timing 0x132b136f "hato.middleware$wrap_request_timing@132b136f"]
#object[hato.middleware$wrap_query_params 0x7ab8ae4a "hato.middleware$wrap_query_params@7ab8ae4a"]
#object[hato.middleware$wrap_basic_auth 0x7c4503c4 "hato.middleware$wrap_basic_auth@7c4503c4"]
#object[hato.middleware$wrap_oauth 0x27152f47 "hato.middleware$wrap_oauth@27152f47"]
#object[hato.middleware$wrap_user_info 0x3a198e4e "hato.middleware$wrap_user_info@3a198e4e"]
#object[hato.middleware$wrap_url 0x6340c4cf "hato.middleware$wrap_url@6340c4cf"]
#object[hato.middleware$wrap_decompression 0x374fa040 "hato.middleware$wrap_decompression@374fa040"]
#object[hato.middleware$wrap_output_coercion 0x39aecef9 "hato.middleware$wrap_output_coercion@39aecef9"]
#object[wkok.openai_clojure.sse$wrap_trace$fn__45320 0x34dd0bf7 "wkok.openai_clojure.sse$wrap_trace$fn__45320@34dd0bf7"]
#object[hato.middleware$wrap_exceptions 0x2ab09fe9 "hato.middleware$wrap_exceptions@2ab09fe9"]
#object[hato.middleware$wrap_accept 0x6037775f "hato.middleware$wrap_accept@6037775f"]
#object[hato.middleware$wrap_accept_encoding 0x2fa1b4a6 "hato.middleware$wrap_accept_encoding@2fa1b4a6"]
#object[hato.middleware$wrap_multipart 0x7a24cd92 "hato.middleware$wrap_multipart@7a24cd92"]
#object[hato.middleware$wrap_content_type 0x54c54ae8 "hato.middleware$wrap_content_type@54c54ae8"]
#object[hato.middleware$wrap_form_params 0xf1342a0 "hato.middleware$wrap_form_params@f1342a0"]
#object[hato.middleware$wrap_nested_params 0x2db5c360 "hato.middleware$wrap_nested_params@2db5c360"]
#object[hato.middleware$wrap_method 0x7121c319 "hato.middleware$wrap_method@7121c319"]],
:url "https://api.openai.com/v1/chat/completions",
:http-request
#object[jdk.internal.net.http.HttpRequestImpl 0x2c818efc "https://api.openai.com/v1/chat/completions POST"],
:uri "/v1/chat/completions",
:server-name "api.openai.com",
:version :http-1.1,
:query-string nil,
:body
"{\"temperature\":null,\"max_tokens\":32,\"messages\":[{\"role\":\"system\",\"content\":\"You are Sherlock Holmes and you will be asked a question by Watson.\"},{\"role\":\"user\",\"content\":\"The cheese is old and moldy, where is the bathroom?\"}],\"model\":\"gpt-3.5-turbo\"}",
:scheme :https,
:request-method :post},
:http-client
#object[jdk.internal.net.http.HttpClientFacade 0x5183fc21 "jdk.internal.net.http.HttpClientImpl@1f180e3f(16)"],
:headers
{"content-encoding" "gzip",
"openai-organization" "mypsycap-dsxiri",
"x-ratelimit-reset-requests" "16.153s",
"server" "cloudflare",
"content-type" "application/json",
"access-control-allow-origin" "*",
"alt-svc" "h3=\":443\"; ma=86400",
"openai-version" "2020-10-01",
"strict-transport-security" "max-age=15724800; includeSubDomains",
"x-ratelimit-remaining-tokens" "59936",
"openai-processing-ms" "559",
"connection" "keep-alive",
"x-ratelimit-reset-tokens" "64ms",
"x-ratelimit-limit-requests" "10000",
"transfer-encoding" "chunked",
"set-cookie"
["__cf_bm=3f3XQ3P7uvjU_Ql5wuSu2oEagm6TOdtO3ky82VL8xDk-1710272689-1.0.1.1-rFL5VoApLzTKI29i4SNJVY6trcyLxyGgjEA2mueIHzpGevNY52srUJwaxvs1YlOzc8C_Y2vVZ_qEbDMppwzzMw; path=/; expires=Tue, 12-Mar-24 20:14:49 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None" "_cfuvid=eX6wMH9CBIJnOavVzYCFpQD2Jy53ch7K6FmbK8e868o-1710272689674-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None"],
"cf-cache-status" "DYNAMIC",
"cf-ray" "863647f0dbd9084f-IAD",
"date" "Tue, 12 Mar 2024 19:44:49 GMT",
"x-request-id" "req_f0ea58aa430e70d98f59e128c0df0ec9",
"x-ratelimit-remaining-requests" "9998",
"openai-model" "gpt-3.5-turbo-0125",
"cache-control" "no-cache, must-revalidate",
"x-ratelimit-limit-tokens" "60000"},
:status 200,
:content-type :application/json,
:uri "https://api.openai.com/v1/chat/completions",
:content-type-params {},
:version :http-1.1,
:body
"{\n \"id\": \"chatcmpl-922LJE88rTz65Z0FCfRBD9nhypgqS\",\n \"object\": \"chat.completion\",\n \"created\": 1710272689,\n \"model\": \"gpt-3.5-turbo-0125\",\n \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": \"assistant\",\n \"content\": \"Holmes, where is the bathroom in this house?\"\n },\n \"logprobs\": null,\n \"finish_reason\": \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 38,\n \"completion_tokens\": 11,\n \"total_tokens\": 49\n },\n \"system_fingerprint\": \"fp_4f0b692a78\"\n}\n"}
2024-03-12 19:44:56,732 [main] DEBUG promptbench.models.openai - ------------------
PARAMS:
{:prompt "The cheese is old and moldy, where is the bathroom?", :size "256x256", :model "dall-e-2"}
2024-03-12 19:44:56,737 [main] DEBUG promptbench.models.openai - ------------------
OPTS:
{:api-key "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",
:request {:throw-exceptions? false},
:trace
#object[promptbench.models.openai$eval45504$fn__45506$fn__45509 0x34487230 "promptbench.models.openai$eval45504$fn__45506$fn__45509@34487230"]}
2024-03-12 19:44:56,750 [main] DEBUG promptbench.models.openai - ------------------
RESPONSE:
{:created 1710272696,
:data
[{:url
"https://oaidalleapiprodscus.blob.core.windows.net/private/org-J8jEGAiir8i12zCA0hlJHSeP/user-XHoiE6JnI2kUkugnQqvHHmqi/img-5lfbllkk5gPOUMJusWPB1Qex.png?st=2024-03-12T18%3A44%3A56Z&se=2024-03-12T20%3A44%3A56Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-03-12T15%3A30%3A23Z&ske=2024-03-13T15%3A30%3A23Z&sks=b&skv=2021-08-06&sig=iqGdvp5cwXDmOwQYOm9GjsgVYeph1kfWsEUtloEvJ9o%3D"}]}
2024-03-12 19:44:56,822 [main] DEBUG promptbench.models.openai - ------------------
RAW-RESPONSE:
{:request-time 6917,
:request
{:user-info nil,
:throw-exceptions? false,
:as :text,
:headers
{"Authorization" "***;,
"Content-Type" "application/json",
"Accept" "application/json",
"accept-encoding" "gzip, deflate"},
:server-port nil,
:middleware
[#object[hato.middleware$wrap_request_timing 0x132b136f "hato.middleware$wrap_request_timing@132b136f"]
#object[hato.middleware$wrap_query_params 0x7ab8ae4a "hato.middleware$wrap_query_params@7ab8ae4a"]
#object[hato.middleware$wrap_basic_auth 0x7c4503c4 "hato.middleware$wrap_basic_auth@7c4503c4"]
#object[hato.middleware$wrap_oauth 0x27152f47 "hato.middleware$wrap_oauth@27152f47"]
#object[hato.middleware$wrap_user_info 0x3a198e4e "hato.middleware$wrap_user_info@3a198e4e"]
#object[hato.middleware$wrap_url 0x6340c4cf "hato.middleware$wrap_url@6340c4cf"]
#object[hato.middleware$wrap_decompression 0x374fa040 "hato.middleware$wrap_decompression@374fa040"]
#object[hato.middleware$wrap_output_coercion 0x39aecef9 "hato.middleware$wrap_output_coercion@39aecef9"]
#object[wkok.openai_clojure.sse$wrap_trace$fn__45320 0x4e392f4 "wkok.openai_clojure.sse$wrap_trace$fn__45320@4e392f4"]
#object[hato.middleware$wrap_exceptions 0x2ab09fe9 "hato.middleware$wrap_exceptions@2ab09fe9"]
#object[hato.middleware$wrap_accept 0x6037775f "hato.middleware$wrap_accept@6037775f"]
#object[hato.middleware$wrap_accept_encoding 0x2fa1b4a6 "hato.middleware$wrap_accept_encoding@2fa1b4a6"]
#object[hato.middleware$wrap_multipart 0x7a24cd92 "hato.middleware$wrap_multipart@7a24cd92"]
#object[hato.middleware$wrap_content_type 0x54c54ae8 "hato.middleware$wrap_content_type@54c54ae8"]
#object[hato.middleware$wrap_form_params 0xf1342a0 "hato.middleware$wrap_form_params@f1342a0"]
#object[hato.middleware$wrap_nested_params 0x2db5c360 "hato.middleware$wrap_nested_params@2db5c360"]
#object[hato.middleware$wrap_method 0x7121c319 "hato.middleware$wrap_method@7121c319"]],
:url "https://api.openai.com/v1/images/generations",
:http-request
#object[jdk.internal.net.http.HttpRequestImpl 0x1480117d "https://api.openai.com/v1/images/generations POST"],
:uri "/v1/images/generations",
:server-name "api.openai.com",
:version :http-1.1,
:query-string nil,
:body
"{\"prompt\":\"The cheese is old and moldy, where is the bathroom?\",\"model\":\"dall-e-2\",\"size\":\"256x256\"}",
:scheme :https,
:request-method :post},
:http-client
#object[jdk.internal.net.http.HttpClientFacade 0x1ca9248c "jdk.internal.net.http.HttpClientImpl@c78c86c(17)"],
:headers
{"content-encoding" "gzip",
"openai-organization" "mypsycap-dsxiri",
"server" "cloudflare",
"content-type" "application/json",
"access-control-allow-origin" "*",
"alt-svc" "h3=\":443\"; ma=86400",
"openai-version" "2020-10-01",
"strict-transport-security" "max-age=15724800; includeSubDomains",
"openai-processing-ms" "6825",
"connection" "keep-alive",
"transfer-encoding" "chunked",
"set-cookie"
["__cf_bm=kRzMpgcrNIAOz1nMfK.NrQHsI1a7Mu1GlLvZA.1F3co-1710272696-1.0.1.1-wGcurw8R.h4TDgJkiBV8Yh05yg339KEsTWOyeY_.079l6Aw7QKIfbavvFnQOzYuSFsL_PolMKO4oMnQ_BRskpA; path=/; expires=Tue, 12-Mar-24 20:14:56 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None" "_cfuvid=1JwE3jnJdrKVO.uJn2TR_X2akbVr9pcnmtD5uoIgceo-1710272696725-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None"],
"cf-cache-status" "DYNAMIC",
"cf-ray" "863647f76f57177d-IAD",
"date" "Tue, 12 Mar 2024 19:44:56 GMT",
"x-request-id" "req_1dd54222434bc3b3da87f058d9d7ab11"},
:status 200,
:content-type :application/json,
:uri "https://api.openai.com/v1/images/generations",
:content-type-params {},
:version :http-1.1,
:body
"{\n \"created\": 1710272696,\n \"data\": [\n {\n \"url\": \"https://oaidalleapiprodscus.blob.core.windows.net/private/org-J8jEGAiir8i12zCA0hlJHSeP/user-XHoiE6JnI2kUkugnQqvHHmqi/img-5lfbllkk5gPOUMJusWPB1Qex.png?st=2024-03-12T18%3A44%3A56Z&se=2024-03-12T20%3A44%3A56Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-03-12T15%3A30%3A23Z&ske=2024-03-13T15%3A30%3A23Z&sks=b&skv=2021-08-06&sig=iqGdvp5cwXDmOwQYOm9GjsgVYeph1kfWsEUtloEvJ9o%3D\"\n }\n ]\n}\n"}
2024-03-12 19:44:56,912 [main] DEBUG promptbench.models.openai - ------------------
PARAMS:
{:model "THIS IS NOT VALID",
:messages [{:role "user", :content "The cheese is old and moldy, where is the bathroom?"}],
:max_tokens 32,
:temperature nil}
2024-03-12 19:44:56,919 [main] DEBUG promptbench.models.openai - ------------------
OPTS:
{:api-key "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",
:request {:throw-exceptions? false},
:trace
#object[promptbench.models.openai$eval45476$fn__45478$fn__45482 0xe4271e5 "promptbench.models.openai$eval45476$fn__45478$fn__45482@e4271e5"]}
2024-03-12 19:44:56,923 [main] DEBUG promptbench.models.openai - ------------------
RESPONSE:
{:error {:message "invalid model ID", :type "invalid_request_error", :param nil, :code nil}}
2024-03-12 19:44:57,014 [main] DEBUG promptbench.models.openai - ------------------
RAW-RESPONSE:
{:request-time 76,
:request
{:user-info nil,
:throw-exceptions? false,
:as :text,
:headers
{"Authorization" "***;,
"Content-Type" "application/json",
"Accept" "application/json",
"accept-encoding" "gzip, deflate"},
:server-port nil,
:middleware
[#object[hato.middleware$wrap_request_timing 0x132b136f "hato.middleware$wrap_request_timing@132b136f"]
#object[hato.middleware$wrap_query_params 0x7ab8ae4a "hato.middleware$wrap_query_params@7ab8ae4a"]
#object[hato.middleware$wrap_basic_auth 0x7c4503c4 "hato.middleware$wrap_basic_auth@7c4503c4"]
#object[hato.middleware$wrap_oauth 0x27152f47 "hato.middleware$wrap_oauth@27152f47"]
#object[hato.middleware$wrap_user_info 0x3a198e4e "hato.middleware$wrap_user_info@3a198e4e"]
#object[hato.middleware$wrap_url 0x6340c4cf "hato.middleware$wrap_url@6340c4cf"]
#object[hato.middleware$wrap_decompression 0x374fa040 "hato.middleware$wrap_decompression@374fa040"]
#object[hato.middleware$wrap_output_coercion 0x39aecef9 "hato.middleware$wrap_output_coercion@39aecef9"]
#object[wkok.openai_clojure.sse$wrap_trace$fn__45320 0x3509972a "wkok.openai_clojure.sse$wrap_trace$fn__45320@3509972a"]
#object[hato.middleware$wrap_exceptions 0x2ab09fe9 "hato.middleware$wrap_exceptions@2ab09fe9"]
#object[hato.middleware$wrap_accept 0x6037775f "hato.middleware$wrap_accept@6037775f"]
#object[hato.middleware$wrap_accept_encoding 0x2fa1b4a6 "hato.middleware$wrap_accept_encoding@2fa1b4a6"]
#object[hato.middleware$wrap_multipart 0x7a24cd92 "hato.middleware$wrap_multipart@7a24cd92"]
#object[hato.middleware$wrap_content_type 0x54c54ae8 "hato.middleware$wrap_content_type@54c54ae8"]
#object[hato.middleware$wrap_form_params 0xf1342a0 "hato.middleware$wrap_form_params@f1342a0"]
#object[hato.middleware$wrap_nested_params 0x2db5c360 "hato.middleware$wrap_nested_params@2db5c360"]
#object[hato.middleware$wrap_method 0x7121c319 "hato.middleware$wrap_method@7121c319"]],
:url "https://api.openai.com/v1/chat/completions",
:http-request
#object[jdk.internal.net.http.HttpRequestImpl 0x15d4195a "https://api.openai.com/v1/chat/completions POST"],
:uri "/v1/chat/completions",
:server-name "api.openai.com",
:version :http-1.1,
:query-string nil,
:body
"{\"temperature\":null,\"max_tokens\":32,\"messages\":[{\"role\":\"user\",\"content\":\"The cheese is old and moldy, where is the bathroom?\"}],\"model\":\"THIS IS NOT VALID\"}",
:scheme :https,
:request-method :post},
:http-client
#object[jdk.internal.net.http.HttpClientFacade 0x1a38fca9 "jdk.internal.net.http.HttpClientImpl@5df71d52(18)"],
:headers
{"server" "cloudflare",
"content-type" "application/json; charset=utf-8",
"content-length" "149",
"alt-svc" "h3=\":443\"; ma=86400",
"strict-transport-security" "max-age=15724800; includeSubDomains",
"connection" "keep-alive",
"set-cookie"
["__cf_bm=CtMur9sXpbng0s4xIjDTYuzybdSTc6kWoCVofhwUXFA-1710272696-1.0.1.1-_ME3nzDVM5RfL3maRAN1DtHKD3joZJQrvtGAmVfxTaNLAvD5dKzUQrS_UEU.gRcaQG7rjFAmU0xZV9P51maV1A; path=/; expires=Tue, 12-Mar-24 20:14:56 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None" "_cfuvid=Sf5KgXA3W58QoepWt1E3NNKL_TCm2Z5CpOSXuxJagb0-1710272696905-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None"],
"cf-cache-status" "DYNAMIC",
"cf-ray" "863648235bd38224-IAD",
"date" "Tue, 12 Mar 2024 19:44:56 GMT",
"vary" "Origin",
"x-request-id" "req_f116190fa5777b171202400090733067"},
:status 400,
:content-type :application/json,
:uri "https://api.openai.com/v1/chat/completions",
:content-type-params {:charset "utf-8"},
:version :http-1.1,
:body
"{\n \"error\": {\n \"message\": \"invalid model ID\",\n \"type\": \"invalid_request_error\",\n \"param\": null,\n \"code\": null\n }\n}\n"}
</failure>
<failure message="[:fail] expected: (>= 75 (-> response :usage :total_tokens) 55). actual: (not (>= 75 49 55))" type="assertion failure: >=">
FAIL in promptbench.models.openai-test/invoke-openai (openai_test.clj:57)
happy paths simple case
expected: (>= 75 (-> response :usage :total_tokens) 55)
actual: (not (>= 75 49 55))
Test output
2024-03-12 19:44:49,686 [main] DEBUG promptbench.models.openai - ------------------
PARAMS:
{:model "gpt-3.5-turbo",
:messages
[{:role "system", :content "You are Sherlock Holmes and you will be asked a question by Watson."}
{:role "user", :content "The cheese is old and moldy, where is the bathroom?"}],
:max_tokens 32,
:temperature nil}
2024-03-12 19:44:49,690 [main] DEBUG promptbench.models.openai - ------------------
OPTS:
{:api-key "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",
:request {:throw-exceptions? false},
:trace
#object[promptbench.models.openai$eval45476$fn__45478$fn__45482 0x32d87fe1 "promptbench.models.openai$eval45476$fn__45478$fn__45482@32d87fe1"]}
2024-03-12 19:44:49,699 [main] DEBUG promptbench.models.openai - ------------------
RESPONSE:
{:id "chatcmpl-922LJE88rTz65Z0FCfRBD9nhypgqS",
:object "chat.completion",
:created 1710272689,
:model "gpt-3.5-turbo-0125",
:choices
[{:index 0,
:message {:role "assistant", :content "Holmes, where is the bathroom in this house?"},
:logprobs nil,
:finish_reason "stop"}],
:usage {:prompt_tokens 38, :completion_tokens 11, :total_tokens 49},
:system_fingerprint "fp_4f0b692a78"}
2024-03-12 19:44:49,801 [main] DEBUG promptbench.models.openai - ------------------
RAW-RESPONSE:
{:request-time 929,
:request
{:user-info nil,
:throw-exceptions? false,
:as :text,
:headers
{"Authorization" "***;,
"Content-Type" "application/json",
"Accept" "application/json",
"accept-encoding" "gzip, deflate"},
:server-port nil,
:middleware
[#object[hato.middleware$wrap_request_timing 0x132b136f "hato.middleware$wrap_request_timing@132b136f"]
#object[hato.middleware$wrap_query_params 0x7ab8ae4a "hato.middleware$wrap_query_params@7ab8ae4a"]
#object[hato.middleware$wrap_basic_auth 0x7c4503c4 "hato.middleware$wrap_basic_auth@7c4503c4"]
#object[hato.middleware$wrap_oauth 0x27152f47 "hato.middleware$wrap_oauth@27152f47"]
#object[hato.middleware$wrap_user_info 0x3a198e4e "hato.middleware$wrap_user_info@3a198e4e"]
#object[hato.middleware$wrap_url 0x6340c4cf "hato.middleware$wrap_url@6340c4cf"]
#object[hato.middleware$wrap_decompression 0x374fa040 "hato.middleware$wrap_decompression@374fa040"]
#object[hato.middleware$wrap_output_coercion 0x39aecef9 "hato.middleware$wrap_output_coercion@39aecef9"]
#object[wkok.openai_clojure.sse$wrap_trace$fn__45320 0x34dd0bf7 "wkok.openai_clojure.sse$wrap_trace$fn__45320@34dd0bf7"]
#object[hato.middleware$wrap_exceptions 0x2ab09fe9 "hato.middleware$wrap_exceptions@2ab09fe9"]
#object[hato.middleware$wrap_accept 0x6037775f "hato.middleware$wrap_accept@6037775f"]
#object[hato.middleware$wrap_accept_encoding 0x2fa1b4a6 "hato.middleware$wrap_accept_encoding@2fa1b4a6"]
#object[hato.middleware$wrap_multipart 0x7a24cd92 "hato.middleware$wrap_multipart@7a24cd92"]
#object[hato.middleware$wrap_content_type 0x54c54ae8 "hato.middleware$wrap_content_type@54c54ae8"]
#object[hato.middleware$wrap_form_params 0xf1342a0 "hato.middleware$wrap_form_params@f1342a0"]
#object[hato.middleware$wrap_nested_params 0x2db5c360 "hato.middleware$wrap_nested_params@2db5c360"]
#object[hato.middleware$wrap_method 0x7121c319 "hato.middleware$wrap_method@7121c319"]],
:url "https://api.openai.com/v1/chat/completions",
:http-request
#object[jdk.internal.net.http.HttpRequestImpl 0x2c818efc "https://api.openai.com/v1/chat/completions POST"],
:uri "/v1/chat/completions",
:server-name "api.openai.com",
:version :http-1.1,
:query-string nil,
:body
"{\"temperature\":null,\"max_tokens\":32,\"messages\":[{\"role\":\"system\",\"content\":\"You are Sherlock Holmes and you will be asked a question by Watson.\"},{\"role\":\"user\",\"content\":\"The cheese is old and moldy, where is the bathroom?\"}],\"model\":\"gpt-3.5-turbo\"}",
:scheme :https,
:request-method :post},
:http-client
#object[jdk.internal.net.http.HttpClientFacade 0x5183fc21 "jdk.internal.net.http.HttpClientImpl@1f180e3f(16)"],
:headers
{"content-encoding" "gzip",
"openai-organization" "mypsycap-dsxiri",
"x-ratelimit-reset-requests" "16.153s",
"server" "cloudflare",
"content-type" "application/json",
"access-control-allow-origin" "*",
"alt-svc" "h3=\":443\"; ma=86400",
"openai-version" "2020-10-01",
"strict-transport-security" "max-age=15724800; includeSubDomains",
"x-ratelimit-remaining-tokens" "59936",
"openai-processing-ms" "559",
"connection" "keep-alive",
"x-ratelimit-reset-tokens" "64ms",
"x-ratelimit-limit-requests" "10000",
"transfer-encoding" "chunked",
"set-cookie"
["__cf_bm=3f3XQ3P7uvjU_Ql5wuSu2oEagm6TOdtO3ky82VL8xDk-1710272689-1.0.1.1-rFL5VoApLzTKI29i4SNJVY6trcyLxyGgjEA2mueIHzpGevNY52srUJwaxvs1YlOzc8C_Y2vVZ_qEbDMppwzzMw; path=/; expires=Tue, 12-Mar-24 20:14:49 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None" "_cfuvid=eX6wMH9CBIJnOavVzYCFpQD2Jy53ch7K6FmbK8e868o-1710272689674-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None"],
"cf-cache-status" "DYNAMIC",
"cf-ray" "863647f0dbd9084f-IAD",
"date" "Tue, 12 Mar 2024 19:44:49 GMT",
"x-request-id" "req_f0ea58aa430e70d98f59e128c0df0ec9",
"x-ratelimit-remaining-requests" "9998",
"openai-model" "gpt-3.5-turbo-0125",
"cache-control" "no-cache, must-revalidate",
"x-ratelimit-limit-tokens" "60000"},
:status 200,
:content-type :application/json,
:uri "https://api.openai.com/v1/chat/completions",
:content-type-params {},
:version :http-1.1,
:body
"{\n \"id\": \"chatcmpl-922LJE88rTz65Z0FCfRBD9nhypgqS\",\n \"object\": \"chat.completion\",\n \"created\": 1710272689,\n \"model\": \"gpt-3.5-turbo-0125\",\n \"choices\": [\n {\n \"index\": 0,\n \"message\": {\n \"role\": \"assistant\",\n \"content\": \"Holmes, where is the bathroom in this house?\"\n },\n \"logprobs\": null,\n \"finish_reason\": \"stop\"\n }\n ],\n \"usage\": {\n \"prompt_tokens\": 38,\n \"completion_tokens\": 11,\n \"total_tokens\": 49\n },\n \"system_fingerprint\": \"fp_4f0b692a78\"\n}\n"}
2024-03-12 19:44:56,732 [main] DEBUG promptbench.models.openai - ------------------
PARAMS:
{:prompt "The cheese is old and moldy, where is the bathroom?", :size "256x256", :model "dall-e-2"}
2024-03-12 19:44:56,737 [main] DEBUG promptbench.models.openai - ------------------
OPTS:
{:api-key "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",
:request {:throw-exceptions? false},
:trace
#object[promptbench.models.openai$eval45504$fn__45506$fn__45509 0x34487230 "promptbench.models.openai$eval45504$fn__45506$fn__45509@34487230"]}
2024-03-12 19:44:56,750 [main] DEBUG promptbench.models.openai - ------------------
RESPONSE:
{:created 1710272696,
:data
[{:url
"https://oaidalleapiprodscus.blob.core.windows.net/private/org-J8jEGAiir8i12zCA0hlJHSeP/user-XHoiE6JnI2kUkugnQqvHHmqi/img-5lfbllkk5gPOUMJusWPB1Qex.png?st=2024-03-12T18%3A44%3A56Z&se=2024-03-12T20%3A44%3A56Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-03-12T15%3A30%3A23Z&ske=2024-03-13T15%3A30%3A23Z&sks=b&skv=2021-08-06&sig=iqGdvp5cwXDmOwQYOm9GjsgVYeph1kfWsEUtloEvJ9o%3D"}]}
2024-03-12 19:44:56,822 [main] DEBUG promptbench.models.openai - ------------------
RAW-RESPONSE:
{:request-time 6917,
:request
{:user-info nil,
:throw-exceptions? false,
:as :text,
:headers
{"Authorization" "***;,
"Content-Type" "application/json",
"Accept" "application/json",
"accept-encoding" "gzip, deflate"},
:server-port nil,
:middleware
[#object[hato.middleware$wrap_request_timing 0x132b136f "hato.middleware$wrap_request_timing@132b136f"]
#object[hato.middleware$wrap_query_params 0x7ab8ae4a "hato.middleware$wrap_query_params@7ab8ae4a"]
#object[hato.middleware$wrap_basic_auth 0x7c4503c4 "hato.middleware$wrap_basic_auth@7c4503c4"]
#object[hato.middleware$wrap_oauth 0x27152f47 "hato.middleware$wrap_oauth@27152f47"]
#object[hato.middleware$wrap_user_info 0x3a198e4e "hato.middleware$wrap_user_info@3a198e4e"]
#object[hato.middleware$wrap_url 0x6340c4cf "hato.middleware$wrap_url@6340c4cf"]
#object[hato.middleware$wrap_decompression 0x374fa040 "hato.middleware$wrap_decompression@374fa040"]
#object[hato.middleware$wrap_output_coercion 0x39aecef9 "hato.middleware$wrap_output_coercion@39aecef9"]
#object[wkok.openai_clojure.sse$wrap_trace$fn__45320 0x4e392f4 "wkok.openai_clojure.sse$wrap_trace$fn__45320@4e392f4"]
#object[hato.middleware$wrap_exceptions 0x2ab09fe9 "hato.middleware$wrap_exceptions@2ab09fe9"]
#object[hato.middleware$wrap_accept 0x6037775f "hato.middleware$wrap_accept@6037775f"]
#object[hato.middleware$wrap_accept_encoding 0x2fa1b4a6 "hato.middleware$wrap_accept_encoding@2fa1b4a6"]
#object[hato.middleware$wrap_multipart 0x7a24cd92 "hato.middleware$wrap_multipart@7a24cd92"]
#object[hato.middleware$wrap_content_type 0x54c54ae8 "hato.middleware$wrap_content_type@54c54ae8"]
#object[hato.middleware$wrap_form_params 0xf1342a0 "hato.middleware$wrap_form_params@f1342a0"]
#object[hato.middleware$wrap_nested_params 0x2db5c360 "hato.middleware$wrap_nested_params@2db5c360"]
#object[hato.middleware$wrap_method 0x7121c319 "hato.middleware$wrap_method@7121c319"]],
:url "https://api.openai.com/v1/images/generations",
:http-request
#object[jdk.internal.net.http.HttpRequestImpl 0x1480117d "https://api.openai.com/v1/images/generations POST"],
:uri "/v1/images/generations",
:server-name "api.openai.com",
:version :http-1.1,
:query-string nil,
:body
"{\"prompt\":\"The cheese is old and moldy, where is the bathroom?\",\"model\":\"dall-e-2\",\"size\":\"256x256\"}",
:scheme :https,
:request-method :post},
:http-client
#object[jdk.internal.net.http.HttpClientFacade 0x1ca9248c "jdk.internal.net.http.HttpClientImpl@c78c86c(17)"],
:headers
{"content-encoding" "gzip",
"openai-organization" "mypsycap-dsxiri",
"server" "cloudflare",
"content-type" "application/json",
"access-control-allow-origin" "*",
"alt-svc" "h3=\":443\"; ma=86400",
"openai-version" "2020-10-01",
"strict-transport-security" "max-age=15724800; includeSubDomains",
"openai-processing-ms" "6825",
"connection" "keep-alive",
"transfer-encoding" "chunked",
"set-cookie"
["__cf_bm=kRzMpgcrNIAOz1nMfK.NrQHsI1a7Mu1GlLvZA.1F3co-1710272696-1.0.1.1-wGcurw8R.h4TDgJkiBV8Yh05yg339KEsTWOyeY_.079l6Aw7QKIfbavvFnQOzYuSFsL_PolMKO4oMnQ_BRskpA; path=/; expires=Tue, 12-Mar-24 20:14:56 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None" "_cfuvid=1JwE3jnJdrKVO.uJn2TR_X2akbVr9pcnmtD5uoIgceo-1710272696725-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None"],
"cf-cache-status" "DYNAMIC",
"cf-ray" "863647f76f57177d-IAD",
"date" "Tue, 12 Mar 2024 19:44:56 GMT",
"x-request-id" "req_1dd54222434bc3b3da87f058d9d7ab11"},
:status 200,
:content-type :application/json,
:uri "https://api.openai.com/v1/images/generations",
:content-type-params {},
:version :http-1.1,
:body
"{\n \"created\": 1710272696,\n \"data\": [\n {\n \"url\": \"https://oaidalleapiprodscus.blob.core.windows.net/private/org-J8jEGAiir8i12zCA0hlJHSeP/user-XHoiE6JnI2kUkugnQqvHHmqi/img-5lfbllkk5gPOUMJusWPB1Qex.png?st=2024-03-12T18%3A44%3A56Z&se=2024-03-12T20%3A44%3A56Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-03-12T15%3A30%3A23Z&ske=2024-03-13T15%3A30%3A23Z&sks=b&skv=2021-08-06&sig=iqGdvp5cwXDmOwQYOm9GjsgVYeph1kfWsEUtloEvJ9o%3D\"\n }\n ]\n}\n"}
2024-03-12 19:44:56,912 [main] DEBUG promptbench.models.openai - ------------------
PARAMS:
{:model "THIS IS NOT VALID",
:messages [{:role "user", :content "The cheese is old and moldy, where is the bathroom?"}],
:max_tokens 32,
:temperature nil}
2024-03-12 19:44:56,919 [main] DEBUG promptbench.models.openai - ------------------
OPTS:
{:api-key "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",
:request {:throw-exceptions? false},
:trace
#object[promptbench.models.openai$eval45476$fn__45478$fn__45482 0xe4271e5 "promptbench.models.openai$eval45476$fn__45478$fn__45482@e4271e5"]}
2024-03-12 19:44:56,923 [main] DEBUG promptbench.models.openai - ------------------
RESPONSE:
{:error {:message "invalid model ID", :type "invalid_request_error", :param nil, :code nil}}
2024-03-12 19:44:57,014 [main] DEBUG promptbench.models.openai - ------------------
RAW-RESPONSE:
{:request-time 76,
:request
{:user-info nil,
:throw-exceptions? false,
:as :text,
:headers
{"Authorization" "***;,
"Content-Type" "application/json",
"Accept" "application/json",
"accept-encoding" "gzip, deflate"},
:server-port nil,
:middleware
[#object[hato.middleware$wrap_request_timing 0x132b136f "hato.middleware$wrap_request_timing@132b136f"]
#object[hato.middleware$wrap_query_params 0x7ab8ae4a "hato.middleware$wrap_query_params@7ab8ae4a"]
#object[hato.middleware$wrap_basic_auth 0x7c4503c4 "hato.middleware$wrap_basic_auth@7c4503c4"]
#object[hato.middleware$wrap_oauth 0x27152f47 "hato.middleware$wrap_oauth@27152f47"]
#object[hato.middleware$wrap_user_info 0x3a198e4e "hato.middleware$wrap_user_info@3a198e4e"]
#object[hato.middleware$wrap_url 0x6340c4cf "hato.middleware$wrap_url@6340c4cf"]
#object[hato.middleware$wrap_decompression 0x374fa040 "hato.middleware$wrap_decompression@374fa040"]
#object[hato.middleware$wrap_output_coercion 0x39aecef9 "hato.middleware$wrap_output_coercion@39aecef9"]
#object[wkok.openai_clojure.sse$wrap_trace$fn__45320 0x3509972a "wkok.openai_clojure.sse$wrap_trace$fn__45320@3509972a"]
#object[hato.middleware$wrap_exceptions 0x2ab09fe9 "hato.middleware$wrap_exceptions@2ab09fe9"]
#object[hato.middleware$wrap_accept 0x6037775f "hato.middleware$wrap_accept@6037775f"]
#object[hato.middleware$wrap_accept_encoding 0x2fa1b4a6 "hato.middleware$wrap_accept_encoding@2fa1b4a6"]
#object[hato.middleware$wrap_multipart 0x7a24cd92 "hato.middleware$wrap_multipart@7a24cd92"]
#object[hato.middleware$wrap_content_type 0x54c54ae8 "hato.middleware$wrap_content_type@54c54ae8"]
#object[hato.middleware$wrap_form_params 0xf1342a0 "hato.middleware$wrap_form_params@f1342a0"]
#object[hato.middleware$wrap_nested_params 0x2db5c360 "hato.middleware$wrap_nested_params@2db5c360"]
#object[hato.middleware$wrap_method 0x7121c319 "hato.middleware$wrap_method@7121c319"]],
:url "https://api.openai.com/v1/chat/completions",
:http-request
#object[jdk.internal.net.http.HttpRequestImpl 0x15d4195a "https://api.openai.com/v1/chat/completions POST"],
:uri "/v1/chat/completions",
:server-name "api.openai.com",
:version :http-1.1,
:query-string nil,
:body
"{\"temperature\":null,\"max_tokens\":32,\"messages\":[{\"role\":\"user\",\"content\":\"The cheese is old and moldy, where is the bathroom?\"}],\"model\":\"THIS IS NOT VALID\"}",
:scheme :https,
:request-method :post},
:http-client
#object[jdk.internal.net.http.HttpClientFacade 0x1a38fca9 "jdk.internal.net.http.HttpClientImpl@5df71d52(18)"],
:headers
{"server" "cloudflare",
"content-type" "application/json; charset=utf-8",
"content-length" "149",
"alt-svc" "h3=\":443\"; ma=86400",
"strict-transport-security" "max-age=15724800; includeSubDomains",
"connection" "keep-alive",
"set-cookie"
["__cf_bm=CtMur9sXpbng0s4xIjDTYuzybdSTc6kWoCVofhwUXFA-1710272696-1.0.1.1-_ME3nzDVM5RfL3maRAN1DtHKD3joZJQrvtGAmVfxTaNLAvD5dKzUQrS_UEU.gRcaQG7rjFAmU0xZV9P51maV1A; path=/; expires=Tue, 12-Mar-24 20:14:56 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None" "_cfuvid=Sf5KgXA3W58QoepWt1E3NNKL_TCm2Z5CpOSXuxJagb0-1710272696905-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None"],
"cf-cache-status" "DYNAMIC",
"cf-ray" "863648235bd38224-IAD",
"date" "Tue, 12 Mar 2024 19:44:56 GMT",
"vary" "Origin",
"x-request-id" "req_f116190fa5777b171202400090733067"},
:status 400,
:content-type :application/json,
:uri "https://api.openai.com/v1/chat/completions",
:content-type-params {:charset "utf-8"},
:version :http-1.1,
:body
"{\n \"error\": {\n \"message\": \"invalid model ID\",\n \"type\": \"invalid_request_error\",\n \"param\": null,\n \"code\": null\n }\n}\n"}
</failure>
</testcase>
<testcase name="promptbench.platform.invocation-test/save!" classname="promptbench.platform.invocation-test" time="0.000000" line="138" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/invocation_test.clj">
</testcase>
<testcase name="promptbench.platform.invocation-test/run!" classname="promptbench.platform.invocation-test" time="0.000000" line="392" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/invocation_test.clj">
</testcase>
<testcase name="promptbench.platform.invocation-test/get-all!" classname="promptbench.platform.invocation-test" time="0.000000" line="524" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/invocation_test.clj">
</testcase>
<testcase name="promptbench.platform.invocation-test/simplify" classname="promptbench.platform.invocation-test" time="0.000000" line="295" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/invocation_test.clj">
</testcase>
<testcase name="promptbench.platform.invocation-test/enrich!" classname="promptbench.platform.invocation-test" time="0.000000" line="263" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/invocation_test.clj">
</testcase>
<testcase name="promptbench.platform.invocation-test/save-results!" classname="promptbench.platform.invocation-test" time="0.000000" line="453" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/invocation_test.clj">
</testcase>
<testcase name="promptbench.platform.invocation-test/render" classname="promptbench.platform.invocation-test" time="0.000000" line="28" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/invocation_test.clj">
</testcase>
<testcase name="promptbench.platform.iteration-test/create!" classname="promptbench.platform.iteration-test" time="0.000000" line="23" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/iteration_test.clj">
</testcase>
<testcase name="promptbench.platform.iteration-test/expand" classname="promptbench.platform.iteration-test" time="0.000000" line="103" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/iteration_test.clj">
</testcase>
<testcase name="promptbench.platform.iteration-test/run!" classname="promptbench.platform.iteration-test" time="0.000000" line="299" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/iteration_test.clj">
</testcase>
<testcase name="promptbench.platform.iteration-test/clone!" classname="promptbench.platform.iteration-test" time="0.000000" line="341" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/iteration_test.clj">
</testcase>
<testcase name="promptbench.platform.iteration-test/get!" classname="promptbench.platform.iteration-test" time="0.000000" line="74" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/iteration_test.clj">
</testcase>
<testcase name="promptbench.platform.web.api.handlers.asset-test/post-coll" classname="promptbench.platform.web.api.handlers.asset-test" time="0.000000" line="22" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/api/handlers/asset_test.clj">
</testcase>
<testcase name="promptbench.platform.web.api.handlers.project-test/post-coll" classname="promptbench.platform.web.api.handlers.project-test" time="0.000000" line="22" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/api/handlers/project_test.clj">
</testcase>
<testcase name="promptbench.platform.web.api.handlers.prompt-test/post-coll" classname="promptbench.platform.web.api.handlers.prompt-test" time="0.000000" line="21" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/api/handlers/prompt_test.clj">
</testcase>
<testcase name="promptbench.platform.web.api.smoke-test/smoke" classname="promptbench.platform.web.api.smoke-test" time="0.000000" line="45" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/api/smoke_test.clj">
</testcase>
<testcase name="promptbench.models.google-test/invoke-google" classname="promptbench.models.google-test" time="0.000000" line="71" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/models/google_test.clj">
</testcase>
<testcase name="promptbench.platform.web.routes-test/get-url" classname="promptbench.platform.web.routes-test" time="0.000000" line="8" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/routes_test.clj">
</testcase>
<testcase name="promptbench.platform.web.api.handlers.variable-test/post-coll" classname="promptbench.platform.web.api.handlers.variable-test" time="0.000000" line="24" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/api/handlers/variable_test.clj">
</testcase>
<testcase name="promptbench.platform.web.placeholder-ui.handlers.prompt.new-iteration-test/post" classname="promptbench.platform.web.placeholder-ui.handlers.prompt.new-iteration-test" time="0.000000" line="18" column="1" file="/home/runner/work/promptbench/promptbench/system/platform/test/clj/promptbench/platform/web/placeholder_ui/handlers/prompt/new_iteration_test.clj">
</testcase>
<system-out/>
<system-err/>
</testsuite>
</testsuites>The config in my workflow file:
- name: Annotate failures
if: success() || failure()
uses: mikepenz/action-junit-report@v4.1.0
with:
report_paths: junit.xml
include_passed: true
require_tests: true
require_passed_tests: trueThe output of the action:
##[group]Run mikepenz/action-junit-report@v4.1.0
with:
report_paths: junit.xml
include_passed: true
require_tests: true
require_passed_tests: true
token: ***
annotate_only: false
check_annotations: true
update_check: false
check_name: JUnit Test Report
fail_on_failure: false
check_retries: false
transformers: []
job_summary: true
detailed_summary: false
annotate_notice: false
follow_symlink: false
job_name: test-lint
truncate_stack_traces: true
env:
JAVA_HOME: /opt/hostedtoolcache/Java_Temurin-Hotspot_jre/21.0.2-13.0.LTS/x64
JAVA_HOME_21_X64: /opt/hostedtoolcache/Java_Temurin-Hotspot_jre/21.0.2-13.0.LTS/x64
CLOJURE_INSTALL_DIR: /tmp/usr/local/opt/ClojureTools
##[endgroup]
##[group]📘 Reading input values
##[endgroup]
##[group]📦 Process test results
Retrieved 1 reports to process.
##[error]❌ No test results found for JUnit Test Report
Am I missing something?
Thank you!
mikepenz commented
Thank you for the report, I'll have to look into that attached XML specifically and see why it might fail.
aviflax commented
My pleasure! Thank you in advance!
mikepenz commented
the file appears to parse properly. Can it be the junit.xml is not in the given location? (e.g. workspace root witht the name junit.xml0?
Can you run it with debug log please, and see if you get additional ouptut may helpful?
aviflax commented
the file appears to parse properly. Can it be the
junit.xmlis not in the given location? (e.g. workspace root witht the name junit.xml0?
🤦 doh! yeah, that was it.
I’d fuzzed over, in my head, the scope of the effect of the working-directory config on the steps in a job.
My workflow has this at the root level:
defaults:
run:
working-directory: system/platform/And when I copy-pasted the structure into the YAML file I didn’t really “parse” it mentally, so I didn’t register that the setting effects only run steps. GitHub did a good job making that explicit in the structure but I was just in copy-pasta mode I guess.
That said, another reason I didn’t immediately think of this is because the action didn’t output a clear message like “no reports found” — in fact it output Retrieved 1 reports to process. — I interpreted that to mean that it had found the file. I mean, what else could it mean?
So maybe this could turn into a minor bug report? Or a minor enhancement request? I think the output of the action could definitely be clearer in a situation like this.
Apart from that minor thought, it’s now working great for me. I’ll leave this open for now just in case you want to consider it a bug report maybe, but of course feel free to close it. Thank you so much!
mikepenz commented
Thank you very much for providing more details.
I fully agree, that message could be better, it was intended to describe that 1 report was configured to be processed, but it's not clear by that wording.
aviflax commented
My pleasure!
OK, cool.
I hope you don’t mind but I have a few suggestions, feel free to ignore:
- I suggest changing that existing message to something like
Searching N paths for reports - I suggest adding a new message (not a debug message but a regular output) along the lines of
Found N reports to parse - And maybe even an additional message:
Successfully parsed N reports
Thanks again!
mikepenz commented
Thank you for the proposal.
Unfortunately, there is no point where we know the amount of found reports, as the globber works async. so we will only have the end result.