/Metric_Collector

Primary LanguagePython

An In-depth Latency Measurement Tool for Large-Scale Systems

Intro

  • In order to improve performance in a large-scale system, it is necessary to manage latency
  • Various latency occur at different points
  • But, There is no tool available for real-time measurement and detailed analysis of latency in a large-scale system

Background

  • This is extended program about ELEMENT ELEMENT
  • This use eBPF/XDP for fast network performance monitoring
    • eBPF
      • eBPF (extended Berkely Packet Filter)
      • it hepls user safely execute user-defined code within certain parts of the kernel
      • it can be used in 10Gbps per-core
    • XDP
      • XDP (eXpress Data Path)
      • it is an eBPF-based high-performane data path used to bypass most operation system networking stacks
      • it can process almost 10Mpps

Architecture

  • Internel Architecture
    • Internal Architecture
  • External Architecture
    • External Architecture

Requirements

  • Redis
    • For Communication Tool between other servers
  • MySQL
    • For a storage as the data collected by Observer
  • paramiko
    • Python Package which is used for SSH connection

Code Structure

  • Metric_Collector
    • ebpf_program_vm
      • metric_measure_vm
        • ebpf_code.py
        • ebpf_conf.py
        • ebpf_database.py
        • ebpf_main.py
        • ebpf_python.py
      • time_sync
        • ebpf_code.py
        • ebpf_conf.py
        • ebpf_main.py
        • ebpf_python.py
    • time_sync_manage
      • ebpf_code.py
      • ebpf_python.py
    • ebpf_preprocess.py
    • ebpf_terminal.py
    • ebpf_database.py
    • ebpf_conf.py
    • ebpf_analyzer.py
    • ebpf_main.py
    • conf
      • connect_info.yaml
      • function_info.yaml
      • ping_info.yaml
      • sampling_info.yaml
      • database_info.yaml
      • management_server_info.yaml
      • redis_info.yaml

Code File

  • time_sync_manage
    • It is used by Management server for Time Synchronization between each servers
    • It make UDP packet and send them to other servers
  • time_sync
    • It is used by Observer for Time Synchronization
    • It installed XDP program
  • ebpf_program_vm
    • Observer Code
    • It collects network metric to central database
  • ebpf_analyzer
    • It calculates the network performance
  • ebpf_preprocess
    • It install Observer code in each server
    • It set tables in Relational Database
  • ebpf_mainprocess
    • It executes Observer in each server
  • ebpf_main
    • It is a entry point

Configuration FILE

  • conf

    • connect_info.yaml (server connect info (address, port, virtual machine etc..))

      variable meaning example
      address address (used by ssh) 10.1.1.1
      port port (used by ssh) 5000
      username name (used by ssh) sonic
      hostname server name (used by analyzer) node1
      metadata_key unique key (used by per server) 1
      novm it is a bare-metaal server metadata_key
      isvm who is the vm's host metadata_key
      eth which interface attach interface names
      other_address other address (used by server) other address
      iscontainer who is the container's host metadata_key

    • function_info.yaml (Which function probed? Not Yet Activated)

    • ping_info.yaml (Information for Time-Synchronization)

      variable meaning example
      address address (where to ping) 10.1.1.1
      port port (where to ping) 5000
      eth interface (used by XDP) enp1s0

    • sampling_info.yaml (Information for Sampling Rate, Address, Port)

      variable meaning example
      size sampling rate (payload size) 72400 (bytes)
      interval sampling rate (time interval) 1 (sec)
      ports sampling port (port that interested) 5000

    • database_info.yaml (Information for Database (Address, Port, Passwd etc..))

      variable meaning example
      user db's user xxxx
      passwd db's password xxxx
      host db's address xxxx
      db which db xxxx

    • management_server_info.yaml (Information for Management server (Reporting, Communication etc..))

      variable meaning example
      address manager server's address 10.1.1.1
      port manager server's port 5000
      username manager server's name xxxx
      password manager server's password xxxx
      hostname manager server's hostname xxxx
      eth manager server's interface (used by XDP) xxxx

    • redis_info.yaml (Information for KV Store (Address, Port, Passwd etc..))

      variable meaning example
      address redis's address 10.1.1.1
      port redis's port 5000

Execution

  • First, Install MySQL, Redis
  • Second, Setting Configuration files properly which is mentioned above
  • Third, Exchange SSH keys between Manager and Proved Server

sudo python3 ebpf_main.py

Example

  • Result Example
    • Result Example1 (LAN)
      • Result Example1 (LAN)
    • Result Example2 (LTE)
      • Result Example2 (LTE)
    • Result Example3 (WIFI)
      • Result Example3 (WIFI)