gojue/ecapture

eCapture consuming lot of memory

h0x0er opened this issue · 9 comments

h0x0er commented

Describe the bug
eCapture is consuming huge amount of memory.

To Reproduce

Note: Output of command free -h may vary.

  1. Checkout memory before starting eCatpure
file -h
               total        used        free      shared  buff/cache   available
Mem:            11Gi       3.8Gi       2.8Gi       1.1Gi       4.9Gi       5.7Gi
Swap:           15Gi       672Mi        14Gi
  1. Start eCapture
sudo ecapture tls
  1. In another terminal re-check the memory
free -h
               total        used        free      shared  buff/cache   available
Mem:            11Gi       6.3Gi       366Mi       1.1Gi       4.8Gi       3.2Gi
Swap:           15Gi       679Mi        14Gi
  1. You will notice the huge amount of consumption

Expected behaviour

eCapture shouldn't consume that much memory.

cfc4n commented

This calculation is inaccurate. It's best to only look at the resource usage of eCapture.

For example, top -p $ECAPTURE_PID.

h0x0er commented

@cfc4n , I am investigating the issue. I will keep you updated.

  • 👍1
h0x0er commented

Following are some details

  1. While creating perf buffer, notice the size of perCpuBuffer

perCpuBuffer := os.Getpagesize() * BufferSizeOfEbpfMap

ecapture/user/module/imodule.go

Lines 192 to 194 in db7e37a

func (m *Module) perfEventReader(errChan chan error, em *ebpf.Map) {
rd, err := perf.NewReader(em, os.Getpagesize()*BufferSizeOfEbpfMap)
if err != nil {

  1. BufferSizeofEbpfMap is declared as

    ecapture/user/module/const_linux.go

    Lines 6 to 8 in db7e37a

    // buffer size times of ebpf perf map
    // buffer size = BufferSizeOfEbpfMap * os.pagesize
    const BufferSizeOfEbpfMap = 1024 * 10

  2. Inside perf.NewReader(), buffer of perCPUBuffer size is allocated for each CPU by calling newPerfEventRing()

https://github.com/cilium/ebpf/blob/f0d238d1934f15fe8c5ef8755337be11bbc114e9/perf/reader.go#L225-L245

  1. Per CPU Memory allocation inside newPerfEventRing() from line 45-49

https://github.com/cilium/ebpf/blob/f0d238d1934f15fe8c5ef8755337be11bbc114e9/perf/ring.go#L25-L49

Calculations

For my machine

  1. os.Getpagesize() = 4096 (bytes)
  2. BufferSizeOfEbpfMap = 10240 (bytes)
  3. perCpuBuffer = os.Getpagesize() * BufferSizeOfEbpfMap = 41943040 (bytes) = 40 MB
  4. Total CPUs = 8
  5. Memory Allocated for 1 module = 40 * 8 = 320 MB
  6. In case of ecapture tls 3 modules are initialised ,
    therefore Memory allocated for ecapture tls = 3 * 320 = 960 MB

Almost 1GB of RAM

fyi @cfc4n

cfc4n commented

Indeed, as you said, eCapture occupies a relatively large amount of memory.

  • BufferSizeOfEbpfMap = 40M : this is to prevent tls events from being lost. Many times, when network traffic is particularly high, it is easy to fill up the ebpf map.
  • per CPU : per CPU type maps have better concurrency safety to avoid errors caused by data write order.
  • 3 modules : This is indeed an area that can be optimized.

Currently, eCapture supports three libraries: openssl\nss\nspr; however, openssl has the highest usage and supports the most mature library compared to the other two which are more niche.

I plan to default close those two modules or create a new subcommand for separate support. Do you have any better ideas?

  • 👍1
h0x0er commented

Regarding BufferSizeOfEbpfMap

this is to prevent tls events from being lost

I agree with this, but I think setting it to 40M by default is not a good idea.

I checked the tetragon implementation, I noticed following things;

  • BufferSize is user configurable. checkout and
  • By default it is set to 65535. checkout

So, I think we should do similar things,

  • Reduce the default size.
  • Make it configurable using a flag, so that end-user can adjust it as per need.

How are your thoughts on this ?

Regarding per CPU types map performance

I am having a little doubt about the performance of per-cpu-buffers
Give a read to: https://nakryiko.com/posts/bpf-ringbuf

I plan to default close those two modules or create a new subcommand for separate support

Disabling unnecessary modules by default seems good idea

fyi @cfc4n

  • 🎉1
cfc4n commented

Thank you for your suggestion.

  • 🎉1
cfc4n commented

I will submit another PR for the custom mapSize flag tomorrow . @h0x0er

good night.

  • 👍1
h0x0er commented

Thanks @cfc4n . Good Night 🌃

cfc4n commented

fixed at #435

Terminal 1

sudo free -m
[sudo] password for cfc4n:
               total        used        free      shared  buff/cache   available
Mem:            3876         477         277           1        3121        3106
Swap:           3893           0        3893

#### exec ecapture at other terminal.
sudo free -m
               total        used        free      shared  buff/cache   available
Mem:            3876         513         240           1        3121        3069
Swap:           3893           0        3893

Terminal 2

sudo bin/ecapture tls

and , openssl module create 3 ebpf maps.

{
    Name: "tls_events",
},
{
    Name: "connect_events",
},
{
    Name: "mastersecret_events",
},
  • mapSizePerCPU = 5M
  • 2 CPUS
  • 3 eBPF maps

all eBPF maps used memory = 2 * 5 * 3 = 30MB.
now, eCapture used memory (include ebpf maps) = 513-477 ≈ 277-240 ≈ 36M .

As expected.