Significant degradation in DL/UL performance between May21st and Jun10th versions of the "test" repository
wozzamac opened this issue · 8 comments
Issue Description
Whilst testing the srsRAN system for our customer Antevia I've seen a significant degradation in performance between the May 21st version of the "test" repository and the Jun 10th version.
I suspect a recent change/changes have adversely affected performance.
Setup Details
I'm using a Picocom 802R and running the same config file on both versions of the "test" repository.
I attach a Motorola MG62 and perform a DL/UL speed test and compare results.
The May 21st version works well, the Jun 10th version doesn't.
Expected Behavior
I would expect very little difference in performance.
Actual Behaviour
Very poor data rates and large error rates when using the Jun10th version.
Steps to reproduce the problem
Please find attached the config file, relevant gnb logs, git log details showing commit-ids and gnb t-trace info.
Additional Information
Attached zip file contains git log details for each srsRAN version
srsRAN_test_May21vJun10.zip
hi @wozzamac, you should not use the test branch. Do you see the same in the main branch?
I quite often use the "test" branch to try more recent changes.
I've noticed the "main" and "test" branch appear to be at the same commit-id point of Jun 10th "c33cacba7d940e734ac7bad08935cbc35578fad9"
I've just built the "main" branch and see the same poor DL/UL performance.
Hey @wozzamac , we have identified an issue in the scheduler which could affect the UL performance when there are lots of retranmissions and will be pushing out a fix soon. I will let you know once the fix is out so that you can give it a try.
Hey @wozzamac , we have identified an issue in the scheduler which could affect the UL performance when there are lots of retranmissions and will be pushing out a fix soon. I will let you know once the fix is out so that you can give it a try.
Ok thanks for the update
@wozzamac The fix has been pushed to test branch. Please give it a try and let me know if it fixes the issue. If it doesn't then please send the gNB (mac and rrc in debug) logs and console trace output
@wozzamac The fix has been pushed to test branch. Please give it a try and let me know if it fixes the issue. If it doesn't then please send the gNB (mac and rrc in debug) logs and console trace output
Hi,
Thanks for the quick response/turn-around.
Just grabbed the latest Jun17th "test" repo and built.
Unfortunately the DL/UL rates are still very poor compared to the May21st "test" repo version.
Jun17-test-version-results.zip
Please find attached the gnb.log file with rrc/mac trace level set to "debug" as requested.
Here's the gnb trace "t" output:
antevia@dusupermicr:~$ sudo taskset -c 0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15 ~/srsRAN_Project/build/apps/gnb/gnb -c ~/Downloads/gnb_ru_picocom_scb_tdd_n77_30mhz_May21.yml
--== srsRAN gNB (commit 40b17b429) ==--
The PRACH detector will not meet the performance requirements with the configuration {Format B4, ZCZ 0, SCS 30kHz, Rx ports 2}.
N2: Connection to AMF on 10.4.31.116:38412 completed
Cell pci=1, bw=30 MHz, 4T4R, dl_arfcn=678334 (n77), dl_freq=4175.01 MHz, dl_ssb_arfcn=677664, ul_freq=4175.01 MHz
Initializing the Open Fronthaul Interface for sector#0: ul_compr=[none,16], dl_compr=[none,16], prach_compr=[none,16], prach_cp_enabled=false, downlink_broadcast=false
Warning: Configured PRACH occasion collides with PUCCH RBs ([0..1) intersects [0..12)). Some interference between PUCCH and PRACH is expected.
Warning: Configured PRACH occasion collides with PUCCH RBs ([0..1) intersects [0..12)). Some interference between PUCCH and PRACH is expected.
==== gNB started ===
Type <h> to view help
t
|--------------------DL---------------------|-------------------------UL------------------------------
pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr
1 4604 | 3 1 5 2.6k 2 0 0% 0 | 26.7 -5.2 9 4.2k 1 0 0% 0 -97n -11
1 4604 | 1 1 1 6.0k 13 4 23% 0 | 6.7 -44.0 23 18k 7 30 81% 0 -16n 0
1 4604 | 1 1 0 0 0 0 0% 0 | 32.4 -4.6 28 4.4k 1 0 0% 0 0 5
1 4604 | 1 1 0 112 1 1 50% 0 | 32.9 -4.7 28 8.7k 2 0 0% 0 24n 5
1 4607 | 1 1 2 6.5k 15 5 25% 0 | 6.8 -44.4 23 22k 7 20 74% 0 0 5
1 4607 | 1 1 4 350k 120 211 63% 0 | -2.8 -49.2 5 206k 56 142 71% 0 0 3
1 4607 | 1 1 1 232k 101 161 61% 0 | -5.1 -52.3 9 244k 25 89 78% 0 0 -6
1 4607 | 1 1 0 4.6k 2 23 92% 0 | n/a n/a 0 0 0 0 0% 0 0 -6
1 4607 | 1 1 0 83k 41 80 66% 0 | 14.3 -30.1 23 76k 15 14 48% 0 0 0
1 4607 | 1 1 1 63k 28 74 72% 0 | 33.2 -4.4 28 28k 7 0 0% 0 0 4
1 4607 | 3 1 1 97k 51 91 64% 0 | -7.7 -52.1 8 55k 14 59 80% 384 -260n -6
|--------------------DL---------------------|-------------------------UL------------------------------
pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr
1 4607 | 1 1 0 17k 13 28 68% 0 | 7.6 -41.5 15 36k 9 14 60% 0 0 2
1 4607 | 1 1 0 123k 64 92 58% 2.27k | 34.0 -3.2 28 61k 14 0 0% 0 0 4
1 4607 | 1 1 1 118k 40 110 73% 0 | 33.3 -4.2 28 17k 4 0 0% 0 0 4
1 4607 | 1 1 0 67k 33 67 67% 0 | 34.8 -4.1 28 8.7k 2 0 0% 0 0 3
1 4607 | 1 1 0 12k 6 0 0% 0 | n/a n/a 0 0 0 0 0% 0 0 3
1 4607 | 1 1 0 9.3k 5 22 81% 0 | n/a n/a 0 0 0 0 0% 0 0 3
1 4607 | 1 1 0 0 0 0 0% 0 | n/a n/a 0 0 0 0 0% 0 0 3
1 4607 | 1 1 0 111k 57 119 67% 0 | 34.2 -4.2 28 22k 5 0 0% 0 0 3
1 4607 | 1 1 1 284k 118 223 65% 0 | 34.0 -4.2 28 39k 9 0 0% 0 0 3
1 4607 | 1 1 1 129k 51 73 58% 0 | 32.6 -4.1 28 17k 4 0 0% 0 0 3
1 4607 | 1 1 1 327k 121 119 49% 0 | 33.2 -4.5 28 39k 9 0 0% 0 0 5
|--------------------DL---------------------|-------------------------UL------------------------------
pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr
1 4607 | 3 1 1 418k 155 298 65% 0 | 33.6 -4.2 28 57k 13 0 0% 0 -764n 4
1 4607 | 1 1 0 410k 169 264 60% 0 | 34.1 -3.2 28 61k 14 0 0% 0 0 4
1 4607 | 1 1 1 263k 110 126 53% 0 | 19.3 -25.2 25 31k 8 10 55% 0 0 -8
1 4607 | 1 1 0 250k 107 228 68% 0 | 35.7 -2.3 28 48k 11 0 0% 0 122n 3
1 4607 | 2 1 0 74k 33 169 83% 0 | 36.0 -2.5 28 8.7k 2 0 0% 0 0 4
1 4607 | 1 1 0 61k 32 90 73% 0 | 35.9 -1.2 28 17k 4 0 0% 0 0 2
1 4607 | 1 1 0 33k 17 56 76% 0 | 34.5 -2.7 28 8.7k 2 0 0% 0 0 3
1 4607 | 1 1 0 12k 6 0 0% 0 | 36.1 -1.3 28 4.4k 1 0 0% 0 0 2
1 4607 | 1 1 0 12k 6 0 0% 0 | n/a n/a 0 0 0 0 0% 0 0 2
1 4607 | 1 1 0 0 0 0 0% 0 | n/a n/a 0 0 0 0 0% 0 0 2
1 4607 | 12 4 7 43k 27 11 28% 0 | -2.2 -44.8 7 962k 75 166 68% 150k -244n -6
|--------------------DL---------------------|-------------------------UL------------------------------
pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr
1 4607 | 1 1 12 54k 26 17 39% 0 | -2.9 -43.8 2 604k 198 402 67% 300k -830n -6
1 4607 | 11 4 9 30k 20 16 44% 0 | -2.9 -43.7 2 456k 200 400 66% 300k -651n -7
1 4607 | 13 4 9 34k 21 6 22% 0 | -2.8 -44.5 2 600k 196 404 67% 300k -146n -6
1 4607 | 12 4 11 46k 25 18 41% 0 | -2.3 -46.5 3 737k 189 411 68% 300k 341n -5
1 4607 | 13 4 7 60k 29 13 30% 0 | -1.7 -45.8 3 451k 198 402 67% 300k 797n -5
1 4607 | 12 4 5 316k 61 51 45% 0 | -1.3 -45.6 5 1.1M 195 405 67% 300k 699n -4
1 4607 | 12 4 10 182k 28 10 26% 0 | -1.6 -46.1 4 451k 198 402 67% 300k 976n -1
1 4607 | 12 4 5 223k 40 61 60% 0 | -2.6 -47.2 4 429k 188 412 68% 300k -260n -3
1 4607 | 1 1 9 72k 26 12 31% 0 | -2.3 -47.1 4 438k 192 408 68% 300k 1u -5
1 4607 | 12 4 12 92k 27 7 20% 0 | -2.4 -46.9 3 746k 193 407 67% 300k 781n -6
1 4607 | 12 4 8 36k 28 8 22% 0 | -2.3 -47.1 4 598k 195 405 67% 300k -813n -6
|--------------------DL---------------------|-------------------------UL------------------------------
pci rnti | cqi ri mcs brate ok nok (%) dl_bs | pusch rsrp mcs brate ok nok (%) bsr ta phr
1 4607 | 11 4 12 34k 23 12 34% 0 | -2.2 -46.8 3 753k 196 404 67% 300k 789n -6
1 4607 | 1 1 8 20k 17 7 29% 0 | -2.5 -47.2 2 435k 191 409 68% 300k -56n -4
1 4607 | 13 3 7 37k 24 19 44% 0 | -1.8 -46.1 5 607k 199 401 66% 300k -81n -6
1 4607 | 2 1 11 17k 15 5 25% 0 | -2.3 -46.2 2 442k 194 406 67% 300k 423n -5
q
Here's the "git log" output:
antevia@dusupermicr:~$ cd ~/srsRAN_Project
antevia@dusupermicr:~/srsRAN_Project$ git log
commit 40b17b429dce2627921d099e8a57674e6f2d82d1 (HEAD -> test, origin/test)
Merge: c33cacba7 480e33432
Author: codebot <group_11283275_bot_29fb07365fb84a160b31ccf01e2e2aa2@noreply.gitlab.com>
Date: Mon Jun 17 10:23:05 2024 +0000
Update main
# Conflicts:
# README.md
# apps/examples/du/CMakeLists.txt
# apps/examples/du/du_example.cpp
# apps/examples/du/fapi_factory.cpp
# apps/examples/du/fapi_factory.h
# apps/examples/du/phy_factory.cpp
# apps/examples/du/phy_factory.h
# apps/examples/du/radio_notifier_sample.h
# apps/services/console_helper.cpp
# apps/services/console_helper.h
# include/srsran/pdcp/pdcp_tx_pdu.h
# include/srsran/support/signal_handler.h
# lib/support/signal_handler.cpp
commit 480e33432e20990f4555209ea8f357a3b7cd0148
Author: faluco <borja@srs.io>
Date: Mon Jun 17 10:38:14 2024 +0200
Disable array-bounds checking without assertions to avoid false positives detected when compiling with GCC
commit 0dd6428f5a6a4b4230e076a2778c5fde88b42ff1
Thank you for the logs. In the latest logs I dont see any issue from scheduler perspective as I see the following warnings in the OFH. I will let my colleagues to take a look at what the issue is.
@sauka can you please take a look what below warning means?
2024-06-17T11:37:39.002282 [OFH ] [W] Real-time timing worker woke up late, sleep time has been '571us', or equivalently, '16' symbols
2024-06-17T11:37:39.002303 [SCHED ] [D] [ 564.9] Slot decisions pci=1 t=4us (0 PDSCHs, 0 PUSCHs):
- PUCCH: c-rnti=0x4607 format=1 prb=[77..78) symb=[0..14) cs=0 occ=1 uci: harq_bits=0 sr=1
2024-06-17T11:37:39.002871 [OFH ] [W] Real-time timing worker woke up late, sleep time has been '607us', or equivalently, '17' symbols
This is not a bug. Please use the discussion forum to get advice from the community on how to tune the configuration to achieve high performance.