Trex crashed on 12 interface
jpatidar01 opened this issue · 4 comments
Trex v3.02
On running the trex with 12 interface and 4 cores per dual interface (i.e. c=4) . It crashed on starting the traffic.
- I tried to many combination of memory mbuf_XX : in trex cfg file but didn't works.
- Also tried option --mbuf-factor 0.2 , 0.8 0.5 .......
Above two fix works for some packet size but not for all ( 64, 128, 256 ....)
Is their any solution that work of all pkt sizes on c>=4 case ??
log :
Error: Failed allocating mbuf for holding 256 bytes from socket 1
Try to enlarge the amount of mbufs in the configuration file '/etc/trex_cfg.yaml'
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_64 : 0 out of 85176 (0.00 %) free <-- need to enlarge
mbuf_128 : 42588 out of 42588 (100.00 %) free
mbuf_256 : 24576 out of 24576 (100.00 %) free
mbuf_512 : 24576 out of 24576 (100.00 %) free
mbuf_64 : 0 out of 85176 (0.00 %) free <-- need to enlarge
mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_1024 : 24576 out of 24576 (100.00 %) free
mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_128 : 42588 out of 42588 (100.00 %) free
mbuf_256 : 24576 out of 24576 (100.00 %) free
mbuf_512 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_64 : 0 out of 85176 (0.00 %) free <-- need to enlarge
mbuf_1024 : 24576 out of 24576 (100.00 %) free
mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
mbuf_128 : 42588 out of 42588 (100.00 %) free
mbuf_256 : 24576 out of 24576 (100.00 %) free
mbuf_512 : 24576 out of 24576 (100.00 %) free
mbuf_1024 : 24576 out of 24576 (100.00 %) free
mbuf_2048 : 37674 out of 234282 (16.08 %) free
mbuf_4096 : 24576 out of 24576 (100.00 %) free
mbuf_9k : 24576 out of 24576 (100.00 %) free
Error: signal 11:
*** traceback follows ***
1 0x55d7da3b6873 ./_t-rex-64(+0x2f8873) [0x55d7da3b6873]
2 0x7fd1eeff0520 /lib/x86_64-linux-gnu/libc.so.6(+0x42520) [0x7fd1eeff0520]
3 0x55d7da730894 rte_mempool_ops_get_count + 4
4 0x55d7da72fd5d rte_mempool_avail_count + 13
5 0x55d7da385efe CRteMemPool::dump(_IO_FILE*) + 1246
6 0x55d7da3451ac CGenNodeStateless::alloc_node_with_vm() + 4060
7 0x55d7da33f2aa CCoreEthIFStateless::send_node(CGenNode*) + 442
8 0x55d7da3ad1f2 CNodeGenerator::handle_stl_node(CGenNode*, CFlowGenListPerThread*) + 978
9 0x55d7da3a0dd2 CNodeGenerator::flush_file(double, double, bool, CFlowGenListPerThread*, double&) + 706
10 0x55d7da58f414 TrexStatelessDpCore::start_scheduler() + 244
11 0x55d7da504ff9 TrexDpCore::start() + 57
12 0x55d7da21bdb0 CGlobalTRex::run_in_core(unsigned char) + 452
13 0x55d7da34076b ./_t-rex-64(+0x28276b) [0x55d7da34076b]
14 0x55d7da2b83a1 ./_t-rex-64(+0x1fa3a1) [0x55d7da2b83a1]
15 0x7fd1ef042b43 /lib/x86_64-linux-gnu/libc.so.6(+0x94b43) [0x7fd1ef042b43]
16 0x7fd1ef0d4a00 /lib/x86_64-linux-gnu/libc.so.6(+0x126a00) [0x7fd1ef0d4a00]
@jpatidar01 it seems you are doing a 64bytes test and mbuf_64 is missing,
The option --mbuf-factor will enlarge all the pools in the same rate and you should have a factor more than 1 (try 3)
The file option is better as you can tune the number per pool
i tried the --mbuf-factor=2 and getting below error
number of ports : 12
max cores for 2 ports : 4
tx queues per port : 6
RX core uses TX queue number 65535 on all ports
core, c-port, c-queue, s-port, s-queue, lat-queue
1 0 0 1 0 0
2 2 0 3 0 0
3 4 0 5 0 0
4 6 0 7 0 0
5 8 0 9 0 0
6 10 0 11 0 0
7 0 1 1 1 255
8 2 1 3 1 255
9 4 1 5 1 255
10 6 1 7 1 255
11 8 1 9 1 255
12 10 1 11 1 255
13 0 2 1 2 255
14 2 2 3 2 255
15 4 2 5 2 255
16 6 2 7 2 255
17 8 2 9 2 255
18 10 2 11 2 255
19 0 3 1 3 255
20 2 3 3 3 255
21 4 3 5 3 255
22 6 3 7 3 255
23 8 3 9 3 255
24 10 3 11 3 255
EAL: memzone_reserve_aligned_thread_unsafe(): Number of requested memzone segments exceeds RTE_MAX_MEMZONE
ERROR there is not enough huge-pages memory in your system
EAL: Error - exiting with code: 1
Cause: Cannot init nodes mbuf pool nodes-7
Killing Scapy server... Scapy server is killed
@jpatidar01 try to setup huge pages using GRUB
https://doc.dpdk.org/dts/gsg/usr_guide/setup_dut.html
see here.
By default trex script allocate 4GB per NUMA. try to allocate more @ startup and then run the script
Yes i'm setting huge pages using GRUB only
here:
GRUB_CMDLINE_LINUX="hugepagesz=1G hugepages=128 default_hugepagesz=1G "
i'm using 12 interfaces and 4 core per dual interface
All the time i'm getting this error
ERROR there is not enough huge-pages memory in your system EAL: Error - exiting with code: 1 Cause: Cannot init nodes mbuf pool nodes-19
How much huge pages need for this configuration ?