Sockperf Performance over mesh testing

[tricksterisback]

I have a setup of 66 vms where i want to measure latency between each vm by pairing them in n(n-1)/2 pairs. Now i am using currently 8 ports on the server and at a time 65 clients will connect to any of the 8 ports. This has been working well for a 19 VM Sockperf mesh Test setup but when i scale it up higher number i start to see Cpu and Memory bandwidth issues. I have a 8vcore 64 gib. Has anybody done this before or is sockperf capable of this scale? Any help would be much appreciated

Here is the ansible code to deploy the setup

- name: Install sockperf server systemd
  become: true
  shell: |
    server_vm_name=$(hostname)
    server_vm_ip=$(hostname -I | awk '{print $1}')

    # Define the port file path
    port_file="/etc/default/networktestsv1/ansible/hosts/sockperf_server_ports.txt"

    # Remove the port file if it exists
    [ -f "$port_file" ] && rm "$port_file"

    # Generate the port file content
    for port in {12345..12352}; do
      echo "T:${server_vm_ip}:${port}" >> /etc/default/networktestsv1/ansible/hosts/sockperf_server_ports.txt
    done


    # Define the server command
    server_cmd="sockperf sr -f ${port_file} --threads-num 24"

    # Run the configure server service script
    bash /etc/default/networktestsv1/ansible/Scripts/configure_server_service.sh "${server_cmd}" "sockperf"
  register: configure_results
  ignore_errors: false

• name: Configure Sockperf Client on VMs
  hosts: localhost
  tasks:

  Step 1: Check the structure of the hosts

  • name: Debug group 'all' to ensure hosts are populated
    debug:
    var: groups['all']

  Step 2: Generate pairs of VMs and validate the output

  • name: Generate pairs of VMs
    set_fact:
    vm_pairs: "{{ groups['all'] | combinations(2) }}"

  • name: Debug VM pairs
    debug:
    var: vm_pairs

  Step 3: Test pairing logic before applying delegate_to

  • name: Print VM pairings
    loop: "{{ vm_pairs }}"
    debug:
    msg: "Pairing {{ item.0 }} as client with {{ item.1 }} as server"

• name: Configure sockperf Client systemd
  hosts: localhost
  become: true

  tasks:

  • name: Install sockperf Client systemd

    Step 4: Check if both item.0 and item.1 are defined

    delegate_to: "{{ item.0 }}"
    shell: |
    fqdn=$(getent hosts "{{ item.1 }}" | awk '{print $2}')
    hostname=${fqdn%%.*}
    server_ip=$(getent ahosts "$fqdn" | awk '/^[^ ]/{print $1; exit}')

    port=$((12345 + ({{ index }} % 8)))
    echo $port
    
    server_cmd="sockperf ping-pong -i $server_ip -m 14 -t 120 --tcp -p ${port} --full-rtt"
    test_tag=$hostname
    
    echo $server_cmd
    
    
    chmod +x /etc/default/networktestsv1/ansible/Scripts/configure_client_service.sh
    sh /etc/default/networktestsv1/ansible/Scripts/configure_client_service.sh "$server_cmd" "$test_tag"
    

    loop: "{{ vm_pairs }}"
    loop_control:
    index_var: index
    when: item.0 is defined and item.1 is defined
    register: script_output