Tsung consumes 3x more CPU when using HTTPS

[champtar]

Hello tsung team and users

I'm trying to use tsung to do some simple HTTP/HTTPS bench, 10k users each doing ~1req/s, around 30Gbit/s of traffic. When using HTTP, the tsung server uses around 20% of CPU, just switching to HTTPS, it uses 60/65% of CPU. Looking with perf top:
with ssl

without ssl

openssl doesn't seems to be consuming much CPU, most of it is in erlang.
My tsung server is running CentOS8, erlang 22.0.7-1.el8, tsung 1.7.0.
I've tried ssl_cache_null trick (https://gist.github.com/jj1bdx/0f273ba8f9d445a41e4807932c2543f7) but no changes.
As I'm setting up 10k connections then just use them, I was expecting to have small performance impact.
Now looking at erlang releases notes, I see erlang 21.2 & 21.3 bring ssl improvements, but I'm already using 22.0.

My questions:

• is 3x CPU increase expected ?
• did anyone try one of the replacement of erlang ssl ? (https://github.com/silviucpp/tls_bench)

As I'm an Erlang noob, any guidance on how to investigate this performance issue is welcome

Here my config:

<?xml version="1.0"?>
<!DOCTYPE tsung SYSTEM "/usr/share/tsung/tsung-1.0.dtd" [] >
<tsung loglevel="notice">
<clients>
  <!-- you need ssh access to each client -->
  <client host="localhost" maxusers="30000" cpu="80">
    <iprange version="v4" value="100.100.0-254.1-254"/>
  </client>
</clients>

<servers>
  <!-- Target -->
  <!-- <server host="15.0.0.91" port="80" type="tcp"></server> -->
  <server host="15.0.0.91" port="443" type="ssl"></server>
</servers>

<load duration="2" unit="hour">
  <arrivalphase phase="1" duration="30" unit="minute">
    <users maxnumber="10000" arrivalrate="100" unit="second"/>
  </arrivalphase>
</load>

<options>
  <option type="ts_http" name="user_agent">
    <user_agent probability="100">tsung-ua</user_agent>
  </option>
  <option name="ip_transparent" value="true"/>
  <option name="ports_range" min="30000" max="60000"/>
  <option name="ssl_reuse_sessions" value="false"/>
</options>   

<sessions>
  <session name="bench" probability="100" type="ts_http">
    <setdynvars sourcetype="random_number" start="1" end="1000">
      <var name="channelnum" />
    </setdynvars>
    <request subst="true"><http url="/live/benchbackend/%%_channelnum%%/sa/sizemin=2000_sizemax=2001_ttl=60_master.m3u8" method="GET" version="1.1"></http></request>
    <for from="1" to="1000000000" incr="1" var="i">
      <request subst="true"><http url="/live/benchbackend/%%_channelnum%%/sa/sizemin=600_sizemax=900_ttl=2_video.m3u8" method="GET" version="1.1"></http></request>
      <request subst="true"><http url="/live/benchbackend/%%_channelnum%%/sa/sizemin=800_sizemax=999_ttl=2_audio.m3u8" method="GET" version="1.1"></http></request>
      <request subst="true"><http url="/live/benchbackend/%%_channelnum%%/sa/sizemin=1500000_sizemax=1600000_ttl=86400_video-%%_i%%.ts" method="GET" version="1.1"></http></request>
      <request subst="true"><http url="/live/benchbackend/%%_channelnum%%/sa/sizemin=68000_sizemax=72000_ttl=86400_audio-%%_i%%.ts" method="GET" version="1.1"></http></request>
      <thinktime value="3.8"></thinktime>
    </for>
  </session>
</sessions>
</tsung>

Thanks
Etienne