Error "Invalid arguments provided to runtime assertion." while running trace

Question

Error "Invalid arguments provided to runtime assertion." while running trace

leokondrashov opened this issue 2 years ago · 5 comments

leokondrashov commented 2 years ago

Version & Branch

main branch

Expected Behaviour

Run the trace

Actual Behaviour

Failure with message "Invalid arguments provided to runtime assertion."

Steps to Reproduce the Problem

Run loader with attached trace
Tested on xl170 with 2 and 3 nodes, loader colocated with master
Trace is obtained via sampler
100_inv.csv
100_mem.csv
100_run.csv

Answer 1 · 2023-02-06T08:57:11.000Z

Same error is observed for traces from trace_reference branch of sampler. With 20 workers + master configuration traces with 500 and more functions fail with the same error, preceded by messages Percentage of failed invocations within a minute has reached 50.00.

Answer 2 · 2023-02-08T14:05:11.000Z

@leokondrashov Regarding the last error, I think you might be pushing too much load into the system. try to run a smaller trace

Answer 3 · 2023-02-08T14:22:35.000Z

Yes, smaller traces run fine, so that's might be true, but CPU and memory utilization show that resources are not fully utilized: ~30%CPU and ~3.5%mem is used on all nodes, total pods is ~1000, which is much less than limit of 20 nodes x 200 pods per node . Without CPU limits, CPU utilization is ~80%.

Answer 4 · 2023-02-08T14:41:24.000Z

hmm, 30 vs 80% is an interesting huge discrepancy...

can you plot the CPU utilization timelines for both configs with the same trace? please include the teardown period as well

Answer 5 · 2023-02-08T15:58:06.000Z

Here's the plot: showing average utilization of all worker nodes over all measuring period. Limited one fails before unlimited, hence the shorter period.