Exported variables are not seen by CMake when calling west
efra-mx-aqua opened this issue · 6 comments
Exported variables are not seen by west. I have notices that when signing binaries
In our project we have been building the code using CMake, but now we are doing the transition to use West.
how to reproduce
Before building define these variables
export HEADER_SIZE=0x0200
export SLOT_SIZE=0x67000
export IMG_FW_VERSION=1.2.3+456
Add this to the "prj.conf"
CONFIG_BOOTLOADER_MCUBOOT=y
CONFIG_MCUBOOT_EXTRA_IMGTOOL_ARGS=" --header-size ${HEADER_SIZE} --align 8 --erased-val 0xff --version ${IMG_FW_VERSION} --slot-size ${SLOT_SIZE}"
Build using:
west build -b nrf52_adafruit_feather $ZEPHYR_BASE/samples/hello_world -p always -- -DCONFIG_MCUBOOT_SIGNATURE_KEY_FILE=\"/path/to/any-valid-key.pem\"
Result
The image signing fails because the variables HEADER_SIZE, IMG_FW_VERSION and SLOT_SIZE do not exits
[146/146] Linking C executable zephyr/zephyr.elf
FAILED: zephyr/zephyr.elf zephyr/zephyr.map zephyr/zephyr.hex zephyr/zephyr.bin zephyr/zephyr.lst zephyr/zephyr.stat zephyr/zephyr.signed.bin zephyr/zephyr.signed.confirmed.bin zephyr/zephyr.signed.hex zephyr/zephyr.signed.confirmed.hex /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.map /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.hex /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.bin /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.lst /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.stat /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.bin /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.confirmed.bin /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.hex /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.confirmed.hex
: && ccache /opt/toolchains/gcc-arm-none-eabi-10-2020-q4-major/bin/arm-none-eabi-gcc zephyr/CMakeFiles/zephyr_final.dir/misc/empty_file.c.obj zephyr/CMakeFiles/zephyr_final.dir/isr_tables.c.obj zephyr/CMakeFiles/zephyr_final.dir/dev_handles.c.obj -o zephyr/zephyr.elf zephyr/CMakeFiles/offsets.dir/./arch/arm/core/offsets/offsets.c.obj -fuse-ld=bfd -Wl,-T zephyr/linker.cmd -Wl,-Map=/home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr_final.map -Wl,--whole-archive app/libapp.a zephyr/libzephyr.a zephyr/arch/common/libarch__common.a zephyr/arch/arch/arm/core/aarch32/libarch__arm__core__aarch32.a zephyr/arch/arch/arm/core/aarch32/cortex_m/libarch__arm__core__aarch32__cortex_m.a zephyr/arch/arch/arm/core/aarch32/mpu/libarch__arm__core__aarch32__mpu.a zephyr/lib/libc/minimal/liblib__libc__minimal.a zephyr/lib/posix/liblib__posix.a zephyr/soc/arm/common/cortex_m/libsoc__arm__common__cortex_m.a zephyr/soc/arm/nordic_nrf/nrf52/libsoc__arm__nordic_nrf__nrf52.a zephyr/drivers/clock_control/libdrivers__clock_control.a zephyr/drivers/console/libdrivers__console.a zephyr/drivers/gpio/libdrivers__gpio.a zephyr/drivers/serial/libdrivers__serial.a zephyr/drivers/timer/libdrivers__timer.a modules/hal_nordic/nrfx/libmodules__hal_nordic__nrfx.a -Wl,--no-whole-archive zephyr/kernel/libkernel.a -L"/opt/toolchains/gcc-arm-none-eabi-10-2020-q4-major/bin/../lib/gcc/arm-none-eabi/10.2.1/thumb/v7e-m/nofp" -L/home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr -lgcc -Wl,--print-memory-usage zephyr/arch/common/libisr_tables.a -mcpu=cortex-m4 -mthumb -mabi=aapcs -mfp16-format=ieee -Wl,--gc-sections -Wl,--build-id=none -Wl,--sort-common=descending -Wl,--sort-section=alignment -Wl,-u,_OffsetAbsSyms -Wl,-u,_ConfigAbsSyms -nostdlib -static -no-pie -Wl,-X -Wl,-N -Wl,--orphan-handling=warn && cd /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr && /snap/cmake/1186/bin/cmake -E rename zephyr_final.map zephyr.map && /opt/toolchains/gcc-arm-none-eabi-10-2020-q4-major/bin/arm-none-eabi-objcopy --gap-fill 0xff --output-target=ihex --remove-section=.comment --remove-section=COMMON --remove-section=.eh_frame zephyr.elf zephyr.hex && /opt/toolchains/gcc-arm-none-eabi-10-2020-q4-major/bin/arm-none-eabi-objcopy --gap-fill 0xff --output-target=binary --remove-section=.comment --remove-section=COMMON --remove-section=.eh_frame zephyr.elf zephyr.bin && /opt/toolchains/gcc-arm-none-eabi-10-2020-q4-major/bin/arm-none-eabi-objdump -d -S zephyr.elf > zephyr.lst && /opt/toolchains/gcc-arm-none-eabi-10-2020-q4-major/bin/arm-none-eabi-readelf -e zephyr.elf > zephyr.stat && /home/efra-mx/workspaces/aqua/iotbrd2/.venv/bin/python3 -m west sign --quiet --tool imgtool --tool-path /home/efra-mx/workspaces/aqua/iotbrd2/.venv/bin/imgtool --build-dir /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build --bin --sbin /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.bin --hex --shex /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.hex -- --key /home/efra-mx/aquarobur-rsa-2048.pem --header-size --align 8 --erased-val 0xff --version --slot-size && /home/efra-mx/workspaces/aqua/iotbrd2/.venv/bin/python3 -m west sign --quiet --tool imgtool --tool-path /home/efra-mx/workspaces/aqua/iotbrd2/.venv/bin/imgtool --build-dir /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build --bin --sbin /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.confirmed.bin --hex --shex /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.confirmed.hex -- --key /home/efra-mx/aquarobur-rsa-2048.pem --header-size --align 8 --erased-val 0xff --version --slot-size --pad --confirm
Memory region Used Size Region Size %age Used
FLASH: 16156 B 200 KB 7.89%
SRAM: 4032 B 64 KB 6.15%
IDT_LIST: 0 GB 2 KB 0.00%
Usage: imgtool sign [OPTIONS] INFILE OUTFILE
Try 'imgtool sign -h' for help.
Error: Invalid value for '-v' / '--version': Invalid version number, should be maj.min.rev+build with later parts optional
FATAL ERROR: command exited with status 2: /home/efra-mx/workspaces/aqua/iotbrd2/.venv/bin/imgtool sign --version 0.0.0+0 --align 4 --header-size 512 --slot-size 204800 --key /home/efra-mx/aquarobur-rsa-2048.pem --header-size --align 8 --erased-val 0xff --version --slot-size /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.bin /home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build/zephyr/zephyr.signed.bin
ninja: build stopped: subcommand failed.
Expected result
That the variables a visible, therefore the signing will work
workaround
Calling cmake directly
rm -rf build
cmake -DBOARD=nrf52_adafruit_feather -S $ZEPHYR_BASE/samples/hello_world -B build -DCONFIG_MCUBOOT_SIGNATURE_KEY_FILE=\"/path/to/any-valid-key.pem\"
make -C build
log:
[100%] Linking C executable zephyr.elf
Memory region Used Size Region Size %age Used
FLASH: 16156 B 200 KB 7.89%
SRAM: 4032 B 64 KB 6.15%
IDT_LIST: 0 GB 2 KB 0.00%
Generating files from zephyr.elf for board: nrf52_adafruit_feather
make[2]: Leaving directory '/home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build'
[100%] Built target zephyr_final
make[1]: Leaving directory '/home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build'
make: Leaving directory '/home/efra-mx/workspaces/aqua/iotbrd2/ar-firmware/build'
output:
build/zephyr/zephyr.bin build/zephyr/zephyr_prebuilt.elf
build/zephyr/zephyr.dts build/zephyr/zephyr_prebuilt.map
build/zephyr/zephyr.elf build/zephyr/zephyr.signed.bin
build/zephyr/zephyr.hex build/zephyr/zephyr.signed.hex
build/zephyr/zephyr.lst build/zephyr/zephyr.stat
build/zephyr/zephyr.map
environment
OS: linux
zephyr: 2.7.2
west: 0.11, 0.14
workaround: call CMake directly
Can you please elaborate on that? I'd be very surprised to learn that prj.conf files support environment variables (with or without west which does not seem relevant here)
@efra-mx-aqua thanks for the report; could you please provide the cmake commands you use which work as expected, for comparison?
@marc-hb @mbolivar-nordic I have updated the description with the missing information
Thanks @efra-mx-aqua . I can't build . Are you using the Zephyr SDK? If not could you share a reproduction that uses it?nrf52_adafruit_feather, it fails with some with HAS_NORDIC_DRIVERS error for me
I tried to reproduce with qemu_x86. As I expected, samples/hello_world/prj.conf does NOT expand environment variables. In other words, with export CLI_OPT=--neither-an-option, both west and cmake fail exactly the same below:
--- a/samples/hello_world/prj.conf
+++ b/samples/hello_world/prj.conf
+# Fails with error: unrecognized command-line option '--not-an-option'
+CONFIG_COMPILER_OPT="--not-an-option"--- a/samples/hello_world/prj.conf
+++ b/samples/hello_world/prj.conf
+# Fails with error: ${CLI_OPT}: linker input file not found: No such file or directory
+# NO variable expansion
+CONFIG_COMPILER_OPT="${CLI_OPT}"I'm not sure what weird "magic" causes environment variables to be expanded in the following line but it's the expansion that looks like a bug to me:
CONFIG_MCUBOOT_EXTRA_IMGTOOL_ARGS=" --header-size ${HEADER_SIZE} --align 8 --erased-val 0xff --version ${IMG_FW_VERSION} --slot-size ${SLOT_SIZE}"
Environment variables are generally speaking a "build code smell": they hide very well and they cause subtle differences which are very difficult to find and reproduce. Many commands "cleanse" the environment.
Also, CMake does not support defining build-time environment variables:
https://gitlab.kitware.com/cmake/community/-/wikis/FAQ#how-can-i-get-or-set-environment-variables
I can't build nrf52_adafruit_feather
I figured out what was wrong, sorry for the noise. I can reproduce now, thanks for the detailed steps.
This is all down to a difference between Ninja and Make, it has nothing to do with west. The variable expansion relies on a bug/loophole in Make that was fixed in Ninja. If you ask west to use make then you can exploit the same loophole from west too:
west build -b nrf52_adafruit_feather samples/hello_world -p always -- -G'Unix Makefiles'
I highly recommend against this though because: 1. Environment variables are bad, see above 2. This gives the wrong impression that environment variables are supported in .conf files (they're not). 3. Most Windows users don't have Make 4. Make is old and baroque, don't use it as the CMake backend. 5. Relying on ninja vs make differences is super confusing.
Maybe you could generate some .conf files? Or maybe the signing tool needs a configuration file.
Anyway this is about environment variables, kconfig and Make and has absolutely nothing to do with west: closing.
PS: setting CONFIG_MCUBOOT_SIGNATURE_KEY_FILE in prj.conf is more convenient for reproduction
@marc-hb thank you for the information. I am by no mean expert in how the Kconfig works, so I tried that and noticed that defining CONFIG_MCUBOOT_EXTRA_IMGTOOL_ARGS with environment variables was working.
I have had problems to pass the image version to the signing script, but that is another issue.