hellofresh/kangal

Support or bug: Unable to submit gzip/base64 test data for jmeter

flah00 opened this issue · 2 comments

flah00 commented 
$ helm ls -n kangal
NAME            NAMESPACE       REVISION        UPDATED                                 STATUS          CHART           APP VERSION
grafana         kangal          1               2023-09-10 09:11:05.024578103 -0400 EDT deployed        grafana-6.59.4  10.1.1
influxdb        kangal          1               2023-09-10 09:11:13.857574427 -0400 EDT deployed        influxdb-4.12.5 1.8.10
kangal          kangal          1               2023-09-10 03:57:12.838650413 -0400 EDT deployed        kangal-2.2.6

I would like to upload a large data set for jmeter performance testing. But thus far I've only successfully been able to upload raw CSVs.

With unencoded CSV POST returns 200

$ curl --fail http://localhost:9999/load-test --verbose \
-H 'Content-Type: multipart/form-data' -F distributedPods=5 \
-F testFile=@r30000000-s1000.jmx -F testData=@r30000000-s1000.csv -F type=JMeter -F overwrite=true
returns 200 and JSON data

With gzip CSV POST returns 400

$ gzip -c <r30000000-s1000.csv > /tmp/tmp.j6jQKvEVKz.gz
$ curl --fail http://localhost:9999/load-test --verbose \
-H 'Content-Type: multipart/form-data' -F distributedPods=5 \
-F testFile=@r30000000-s1000.jmx -F testData=@/tmp/tmp.j6jQKvEVKz.gz -F type=JMeter -F overwrite=true
curl: (22) The requested URL returned error: 400

With gzip base64 POST returns 400

$ gzip -c <r30000000-s1000.csv |base64 > /tmp/tmp.j6jQKvEVKz.gz
$ curl --fail http://localhost:9999/load-test --verbose \
-H 'Content-Type: multipart/form-data' -F distributedPods=5 \
-F testFile=@r30000000-s1000.jmx -F testData=@/tmp/tmp.j6jQKvEVKz.gz -F type=JMeter -F overwrite=true
curl: (22) The requested URL returned error: 400

Perhaps I'm misunderstanding how to make use of the csv gzip base64 feature? Proxy and controller logs don't reveal anything. The jmx file opens in jmeter. I'm not sure what I'm missing?

jmx file

<?xml version="1.0" encoding="UTF-8"?>
<jmeterTestPlan version="1.2" properties="5.0" jmeter="5.6.2">
  <hashTree>
    <TestPlan guiclass="TestPlanGui" testclass="TestPlan" testname="r30000000-s15000" enabled="true">
      <stringProp name="TestPlan.comments">This test will call Kangal status endpoint running locally</stringProp>
      <boolProp name="TestPlan.functional_mode">false</boolProp>
      <boolProp name="TestPlan.serialize_threadgroups">false</boolProp>
      <elementProp name="TestPlan.user_defined_variables" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" testname="User Defined Variables" enabled="true">
        <collectionProp name="Arguments.arguments"/>
      </elementProp>
      <boolProp name="TestPlan.tearDown_on_shutdown">false</boolProp>
    </TestPlan>
    <hashTree>
      <ThreadGroup guiclass="ThreadGroupGui" testclass="ThreadGroup" testname="Thread Group" enabled="true">
        <stringProp name="ThreadGroup.on_sample_error">continue</stringProp>
        <elementProp name="ThreadGroup.main_controller" elementType="LoopController" guiclass="LoopControlPanel" testclass="LoopController" testname="Loop Controller" enabled="true">
          <intProp name="LoopController.loops">1</intProp>
          <boolProp name="LoopController.continue_forever">false</boolProp>
        </elementProp>
        <stringProp name="ThreadGroup.num_threads">10000</stringProp>
        <stringProp name="ThreadGroup.ramp_time">200</stringProp>
        <boolProp name="ThreadGroup.scheduler">false</boolProp>
        <stringProp name="ThreadGroup.duration"></stringProp>
        <stringProp name="ThreadGroup.delay"></stringProp>
        <boolProp name="ThreadGroup.delayedStart">false</boolProp>
        <boolProp name="ThreadGroup.same_user_on_next_iteration">true</boolProp>
      </ThreadGroup>
      <hashTree>
        <HTTPSamplerProxy guiclass="HttpTestSampleGui" testclass="HTTPSamplerProxy" testname="HTTP POST" enabled="true">
          <boolProp name="HTTPSampler.postBodyRaw">true</boolProp>
          <elementProp name="HTTPsampler.Arguments" elementType="Arguments">
            <collectionProp name="Arguments.arguments">
              <elementProp name="" elementType="HTTPArgument">
                <boolProp name="HTTPArgument.always_encode">false</boolProp>
                <stringProp name="Argument.value">${body}</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="HTTPSampler.domain">http.perf.svc.cluster.local</stringProp>
          <stringProp name="HTTPSampler.protocol">http</stringProp>
          <stringProp name="HTTPSampler.path">/api/decision</stringProp>
          <stringProp name="HTTPSampler.port">80</stringProp>
          <stringProp name="HTTPSampler.method">POST</stringProp>
          <boolProp name="HTTPSampler.follow_redirects">false</boolProp>
          <boolProp name="HTTPSampler.auto_redirects">false</boolProp>
          <boolProp name="HTTPSampler.use_keepalive">true</boolProp>
          <boolProp name="HTTPSampler.DO_MULTIPART_POST">false</boolProp>
          <boolProp name="HTTPSampler.BROWSER_COMPATIBLE_MULTIPART">false</boolProp>
          <boolProp name="HTTPSampler.image_parser">false</boolProp>
          <boolProp name="HTTPSampler.concurrentDwn">false</boolProp>
          <stringProp name="HTTPSampler.concurrentPool">6</stringProp>
          <boolProp name="HTTPSampler.md5">false</boolProp>
          <intProp name="HTTPSampler.ipSourceType">0</intProp>
        </HTTPSamplerProxy>
        <hashTree>
          <HeaderManager guiclass="HeaderPanel" testclass="HeaderManager" testname="HTTP Header Manager" enabled="true">
            <collectionProp name="HeaderManager.headers">
              <elementProp name="" elementType="Header">
                <stringProp name="Header.name">Content-Type</stringProp>
                <stringProp name="Header.value">application/json</stringProp>
              </elementProp>
            </collectionProp>
          </HeaderManager>
          <hashTree/>
          <CSVDataSet guiclass="TestBeanGUI" testclass="CSVDataSet" testname="CSV Data Set Config" enabled="true">
            <stringProp name="filename">/testdata/testdata.csv</stringProp>
            <stringProp name="fileEncoding"></stringProp>
            <stringProp name="variableNames">body</stringProp>
            <boolProp name="ignoreFirstLine">false</boolProp>
            <stringProp name="delimiter">,</stringProp>
            <boolProp name="quotedData">true</boolProp>
            <boolProp name="recycle">true</boolProp>
            <boolProp name="stopThread">false</boolProp>
            <stringProp name="shareMode">shareMode.all</stringProp>
          </CSVDataSet>
          <hashTree/>
        </hashTree>
        <BackendListener guiclass="BackendListenerGui" testclass="BackendListener" testname="Backend Listener" enabled="true">
          <elementProp name="arguments" elementType="Arguments" guiclass="ArgumentsPanel" testclass="Arguments" enabled="true">
            <collectionProp name="Arguments.arguments">
              <elementProp name="influxdbMetricsSender" elementType="Argument">
                <stringProp name="Argument.name">influxdbMetricsSender</stringProp>
                <stringProp name="Argument.value">org.apache.jmeter.visualizers.backend.influxdb.HttpMetricsSender</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="influxdbUrl" elementType="Argument">
                <stringProp name="Argument.name">influxdbUrl</stringProp>
                <stringProp name="Argument.value">http://influxdb.kangal.svc.cluster.local:8086/write?db=graphite</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="application" elementType="Argument">
                <stringProp name="Argument.name">application</stringProp>
                <stringProp name="Argument.value">http</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="measurement" elementType="Argument">
                <stringProp name="Argument.name">measurement</stringProp>
                <stringProp name="Argument.value">jmeter</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="summaryOnly" elementType="Argument">
                <stringProp name="Argument.name">summaryOnly</stringProp>
                <stringProp name="Argument.value">false</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="samplersRegex" elementType="Argument">
                <stringProp name="Argument.name">samplersRegex</stringProp>
                <stringProp name="Argument.value">.*</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="percentiles" elementType="Argument">
                <stringProp name="Argument.name">percentiles</stringProp>
                <stringProp name="Argument.value">9d0;95;99</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="testTitle" elementType="Argument">
                <stringProp name="Argument.name">testTitle</stringProp>
                <stringProp name="Argument.value">r30000000-s15000</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
              <elementProp name="eventTags" elementType="Argument">
                <stringProp name="Argument.name">eventTags</stringProp>
                <stringProp name="Argument.metadata">=</stringProp>
              </elementProp>
            </collectionProp>
          </elementProp>
          <stringProp name="TestPlan.comments">This listener will send data to Influx DB.</stringProp>
          <stringProp name="classname">org.apache.jmeter.visualizers.backend.influxdb.InfluxdbBackendListenerClient</stringProp>
        </BackendListener>
        <hashTree/>
      </hashTree>
    </hashTree>
  </hashTree>
</jmeterTestPlan>

example csv file

"{""a"":{},""b"":[],""c"":{}}"
"{""a"":{""d"":1},""b"":[""e""]],""c"":{""f"":1}}"
lucasmdrs commented

Hi @flah00

The testdata compression feature in Jmeter backend is done by the Proxy itself.

The Proxy only accepts csv and protoset file formats at the moment.

flah00 commented

D'oh! thx