Weird gaps between symbols in .bss and .rodata
rojer opened this issue · 13 comments
i'm looking closely at my global variable usage using objdump and i see big gaps in between symbols in .bss
3ffeb3f2 g O .bss 00000002 avg_freqoffset_sta_freqcal
3ffeb3f4 g O .bss 0000001c DefFreqCalTimer
3ffeb410 g O .bss 00000002 PktNumInOneChk
3ffebcd0 g O .bss 00000040 wDevCtrl
3ffebe78 g O .bss 00000001 NMIIrqIsOn
3ffebe7c g O .bss 00000004 WdevTimOffSet
3ffef5c8 g O .bss 00000001 tout_dis_txpwr_track
3ffef5c9 g O .bss 00000001 sleep_mode_flag
PktNumInOneChk: supposedly 2 bytes, a gap of 2240.
WdevTimOffSet: size 4 bytes, a gap of 14156 (!) before the next symbol, tout_dis_txpwr_track
what could be using space in these gaps? i don't think they are static variables, as i can see my statics in the output.
i see the same thing in .rodata:
here's entire segment, it's short enough to reproduce in full:
3ffe8c40 g *ABS* 00000000 _rodata_start
3ffe8c40 g O .rodata 00000101 _ctype_
3ffe8c40 l d .rodata 00000000 .rodata
3ffe8d44 g O .rodata 00000004 _global_impure_ptr
3ffe8d48 l O .rodata 00000008 year_lengths
3ffe8d50 l O .rodata 00000060 mon_lengths
3ffe9168 l O .rodata 00000028 tinytens
3ffe92f4 l O .rodata 00000010 zeroes$4031
3ffe9304 l O .rodata 00000010 blanks$4030
3ffe9314 l O .rodata 00000022 basefix$3944
3ffe9338 l O .rodata 000000fc _C_time_locale
3ffe9598 l O .rodata 00000010 zeroes$4046
3ffe95a8 l O .rodata 00000010 blanks$4045
3ffe95b8 l O .rodata 0000000c p05$2546
3ffe95c8 g O .rodata 00000028 __mprec_tinytens
3ffe95f0 g O .rodata 00000028 __mprec_bigtens
3ffe9618 g O .rodata 000000c8 __mprec_tens
3ffe96e0 l O .rodata 00000010 zeroes$3975
3ffe96f0 l O .rodata 00000010 blanks$3974
3ffe9700 l O .rodata 00000022 basefix$3444
3ffe9722 l O .rodata 00000010 zeroes$3989
3ffe9732 l O .rodata 00000010 blanks$3988
3ffea051 g O .rodata 00000006 ethzero
3ffea057 g O .rodata 00000006 ethbroadcast
3ffea060 g O .rodata 00000004 ip_addr_broadcast
3ffea064 g O .rodata 00000004 ip_addr_any
3ffea068 g O .rodata 00000010 ip6_addr_any
3ffea1dc g *ABS* 00000000 _bss_table_start
3ffea1dc g *ABS* 00000000 __init_array_end
3ffea1dc g *ABS* 00000000 __init_array_start
3ffea1dc g *ABS* 00000000 __XT_EXCEPTION_DESCS__
3ffea1dc g *ABS* 00000000 __XT_EXCEPTION_DESCS_END__
3ffea1dc g *ABS* 00000000 __XT_EXCEPTION_TABLE__
3ffea1e4 g *ABS* 00000000 _bss_table_end
3ffea1e4 g *ABS* 00000000 _rodata_end
total size is 0x3ffea1e4 - 0x3ffe8c40 = 5540 bytes, but if i objcopy, i only get 3468 bytes.
and again, blanks$3988: supposedly 16 bytes, but there is a gap of 2335 bytes before ethzero.
Any way for me to reproduce it?
install Docker, then:
[rojer@nba ~/cesanta ]$ git clone https://github.com/cesanta/mongoose-os.git
Cloning into 'mongoose-os'...
...
[rojer@nba ~/cesanta ]$ cd mongoose-os/fw/platforms/esp8266
[rojer@nba ~/cesanta/mongoose-os/fw/platforms/esp8266 master]$ make
...
Built mongoose-os/esp8266 version 2017021306 (20170213-063055/master@a3afc550)
[rojer@nba ~/cesanta/mongoose-os/fw/platforms/esp8266 master]$ xtensa-lx106-elf-objdump -t .build/mongoose-os.elf | sort | grep '\.rodata'
3ffe8c00 g O .rodata 0000000f SDK_VERSION
3ffe8c00 l d .rodata 00000000 .rodata
3ffe8c0f g O .rodata 00000006 ethzero
3ffe8c15 g O .rodata 00000006 ethbroadcast
3ffe8c1c g O .rodata 00000028 memp_sizes
3ffe8c78 g O .rodata 00000010 tcp_pcb_lists
3ffe8c88 g O .rodata 00000007 tcp_persist_backoff
3ffe8c8f g O .rodata 0000000d tcp_backoff
3ffe8c9c g O .rodata 00000004 ip_addr_broadcast
3ffe8ca0 g O .rodata 00000004 ip_addr_any
3ffe8ca4 l O .rodata 00000004 magic_cookie
3ffe8cc8 g O .rodata 00000101 _ctype_
3ffe8dcc g O .rodata 00000004 _global_impure_ptr
3ffe8dd0 l O .rodata 00000008 year_lengths
3ffe8dd8 l O .rodata 00000060 mon_lengths
3ffe8fb0 l O .rodata 00000028 tinytens
3ffe9144 l O .rodata 00000010 zeroes$4039
3ffe9154 l O .rodata 00000010 blanks$4038
3ffe9164 l O .rodata 00000022 basefix$3949
3ffe9188 l O .rodata 000000fc _C_time_locale
3ffe93f0 l O .rodata 00000010 zeroes$4054
3ffe9400 l O .rodata 00000010 blanks$4053
3ffe9410 l O .rodata 0000000c p05$2549
3ffe9420 g O .rodata 00000028 __mprec_tinytens
3ffe9448 g O .rodata 00000028 __mprec_bigtens
3ffe9470 g O .rodata 000000c8 __mprec_tens
3ffe96a4 l O .rodata 00000010 zeroes$4018
3ffe96b4 l O .rodata 00000010 blanks$4017
the docker image contains a build of our fork of esp-open-sdk - specifically, cesanta/esp-open-sdk @ 496f44d which is based on crosstool-NG @ ecfc19a.
PktNumInOneChk: supposedly 2 bytes, a gap of 2240.
WdevTimOffSet: size 4 bytes, a gap of 14156 (!) before the next symbol, tout_dis_txpwr_trackwhat could be using space in these gaps?
Both these symbols come from the Espressif libpp.a, so I've no idea of how they were made.
'supposedly' is the right word here, because the size printed by the objdump is just an arbitrary metainformation associated with the symbol, it does not have to correspond to actual data size.
Here's an illustration:
$ cat size.s
.data
.global var1
var1:
.long 1
.size var1, 1000
var2:
.long 2
$ xtensa-lx106-elf-gcc -c size.s
$ xtensa-lx106-elf-objdump -x size.o | grep var
00000004 l .data 00000000 var2
00000000 g .data 000003e8 var1
i see the same thing in .rodata:
unfortunately I don't see the same thing. Here's what I see:
3ffe8c00 g O .rodata 0000000f SDK_VERSION
3ffe8c00 l d .rodata 00000000 .rodata
3ffe8c0f g O .rodata 00000006 ethzero
3ffe8c15 g O .rodata 00000006 ethbroadcast
3ffe8c1c g O .rodata 00000028 memp_sizes
3ffe8c64 g O .rodata 00000010 tcp_pcb_lists
3ffe8c74 g O .rodata 00000007 tcp_persist_backoff
3ffe8c7b g O .rodata 0000000d tcp_backoff
3ffe8c9c g O .rodata 00000004 ip_addr_broadcast
3ffe8ca0 g O .rodata 00000004 ip_addr_any
3ffe8ca4 l O .rodata 00000004 magic_cookie
3ffe8cc8 g O .rodata 00000101 _ctype_
3ffe8dcc g O .rodata 00000004 _global_impure_ptr
3ffe8dd0 l O .rodata 00000008 year_lengths
3ffe8dd8 l O .rodata 00000060 mon_lengths
3ffe8fb0 l O .rodata 00000028 tinytens
3ffe9144 l O .rodata 00000010 zeroes$4039
3ffe9154 l O .rodata 00000010 blanks$4038
3ffe9164 l O .rodata 00000022 basefix$3949
3ffe9188 l O .rodata 000000fc _C_time_locale
3ffe93f0 l O .rodata 00000010 zeroes$4054
3ffe9400 l O .rodata 00000010 blanks$4053
3ffe9410 l O .rodata 0000000c p05$2549
3ffe9420 g O .rodata 00000028 __mprec_tinytens
3ffe9448 g O .rodata 00000028 __mprec_bigtens
3ffe9470 g O .rodata 000000c8 __mprec_tens
3ffe96a4 l O .rodata 00000010 zeroes$4018
3ffe96b4 l O .rodata 00000010 blanks$4017
3ffe971f l .rodata 00000000 .LC9
I looked at the first size mismatch, that's
3ffe8c1c g O .rodata 00000028 memp_sizes
3ffe8c64 g O .rodata 00000010 tcp_pcb_lists
and I've found jump table right before the tcp_pcb_lists. Here's the code that uses it:
00000000 <tcp_close_shutdown-0x54>:
...
14: R_XTENSA_32 .rodata
...
00000054 <tcp_close_shutdown>:
...
12c: ffba21 l32r a2, 14 <tcp_close_shutdown-0x40>
12c: R_XTENSA_SLOT0_OP .text.tcp_close_shutdown+0x14
12f: a02320 addx4 a2, a3, a2
132: 0228 l32i.n a2, a2, 0
134: 0002a0 jx a2
and the relocations for it, pointing back to tcp_close_shutdown :
Relocation section '.rela.rodata' at offset 0x1c344 contains 12 entries:
Offset Info Type Sym.Value Sym. Name + Addend
00000000 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
00000004 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
00000008 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
0000000c 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
00000010 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
00000014 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
00000018 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
0000001c 00001d01 R_XTENSA_32 00000000 .text.tcp_close_shutdo + 0
00000020 0000e101 R_XTENSA_32 00000004 tcp_listen_pcbs + 0
00000024 0000dc01 R_XTENSA_32 00000004 tcp_bound_pcbs + 0
00000028 0000f801 R_XTENSA_32 00000004 tcp_active_pcbs + 0
0000002c 0000fc01 R_XTENSA_32 00000004 tcp_tw_pcbs + 0
there are small discrepancies, those could be jump tables. but there are also big ones.
how about mon_lengths (from your output):
3ffe8dd8 l O .rodata 00000060 mon_lengths
3ffe8fb0 l O .rodata 00000028 tinytens
96 bytes size -> 472 bytes gap
The corresponding part of the link map looks like this:
.rodata 0x000000003ffe8dd0 0x68 /opt/Espressif/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.5/../../../../xtensa-lx106-elf/lib/libc.a(lib_a-mktm_r.o)
.rodata 0x000000003ffe8e38 0x158 /opt/Espressif/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.5/../../../../xtensa-lx106-elf/lib/libc.a(lib_a-strftime.o)
.rodata 0x000000003ffe8f90 0x48 /opt/Espressif/esp-open-sdk/xtensa-lx106-elf/lib/gcc/xtensa-lx106-elf/4.8.5/../../../../xtensa-lx106-elf/lib/libc.a(lib_a-strtod.o)
mon_lengths is in the lib_a-mktm_r.o, tinytens is in the lib_a-strtod.o. lib_a-strftime.o between them adds 0x158 byte jump table, again:
00000000 <strftime-0x118>:
...
1c: R_XTENSA_32 .rodata
...
1e8: ff8d31 l32r a3, 1c <strftime-0xfc>
1e8: R_XTENSA_SLOT0_OP .text.strftime+0x1c
1eb: a02230 addx4 a2, a2, a3
1ee: 0228 l32i.n a2, a2, 0
1f0: 0002a0 jx a2
Relocation section '.rela.rodata' at offset 0x557c contains 86 entries:
Offset Info Type Sym.Value Sym. Name + Addend
00000000 00000801 R_XTENSA_32 00000000 .text.strftime + 0
00000004 00000801 R_XTENSA_32 00000000 .text.strftime + 0
00000008 00000801 R_XTENSA_32 00000000 .text.strftime + 0
0000000c 00000801 R_XTENSA_32 00000000 .text.strftime + 0
00000010 00000801 R_XTENSA_32 00000000 .text.strftime + 0
00000014 00000801 R_XTENSA_32 00000000 .text.strftime + 0
...
Plus 0x1c byte sized table in lib_a-strftime.o with exactly the same pattern.
ok, so the gaps are filled with various ancillary data inserted by compiler. since it's code, can it perhaps be pushed onto flash?
since it's code, can it perhaps be pushed onto flash?
It's not actually code, it's data, likely with relocations against it.
As is it can be pushed to flash as a part of .rodata with linker script. This may not always be desirable, depending on the nature of other data in the .rodata.
With -fpic option given to the compiler these jump tables become a part of the .text of the function that uses them. The resulting code is slightly less efficient though.
The compiler uses JUMP_TABLES_IN_TEXT_SECTION macro (in the compiler backend code) to decide whether it should put jump tables into .text or .rodata. For xtensa it's defined as flag_pic, i.e. it puts tables to .text iff it's invoked with -fpic.
hm... yeah, -fPIC is not a good idea - code is larger and memory footprint increases as well.
ok, so that;s .rodata gaps, they are filled with jumptables.
any ideas what the .bss gaps are?
any ideas what the .bss gaps are?
Do you see any .bss gaps between/adjacent to symbols from object files compiled with gcc?
examining the link map and poking around convinced me that these are just static from SDK libs.
sorry for raising the noise. but - i have learned about linker maps :)