Can't build project that contains two files with identical name
dawidchyrzynski opened this issue · 19 comments
Hi,
My project uses ESP8266 core and Arduino Crypto library (https://github.com/OperatorFoundation/Crypto). The problem is that both repositories contain "Crypto.h" and "Crypto.cpp" files. Due to flat structure of the build directory Crypto.cpp from ESP8266 overrides Crypto.cpp from Arduino Crypto and it leads to linking problems (missing symbols). Is there any way to solve this issue (assuming that we're not going to modify third party sources)?
This is a looming issue for me as well for a very long time. I had to work around name clashes in third party libs, most often by forking them and renaming individual files. Building on Mac makes things even worse due to case insensitive file names (by default). I don't know how feasable it would be to keep the directory structure for the object files during the build though.
Ok, point taken. I will look into creating sub directories in the build directory
I have tried several solutions to this problem but selected the easiest one.
Please try out the latest commit
Hi @plerup,
Unfortunately it didn't solve the problem. I can see two files being compiled (one with original name, the second one with "2_" prefix) but linker still complains about missing symbols. When I'm adding prefix manually to the original cpp file the build succeeds without any issue.
Is it possible for you to share these files with me?
hi Peter,
I've just seen the same problem with commit f30b6fe: the second file duplicating a name gets compiled (as 2_...) but doesn't get linked. Renaming the file works.
I first thought that the src_list.mk additions in find_src.pl need to also add the renamed source file to a list of objects to link, but the problem seems to be influenced by execution order. Turning off both make parallelism and ccache seems to help.
My build is here: https://gitlab.com/hamishcunningham/unphone/-/tree/master/examples/2022
The issue manifests in relation to WebServer.cpp in the ESPAsyncWebServer library.
Tnx! Hamish
Also, if any source files are symbolic links, they get renamed (even though they're unique).
@hamishcunningham
Please try setting NO_USER_OBJ_LIB=1
@hamishcunningham Please try setting NO_USER_OBJ_LIB=1
This appears to work :)
Observations:
- putting
NO_USER_OBJ_LIB = 1in my Makefile (before including makeEspArduino.mk) doesn't seem to work: I have to put it on the command line - the problem is intermittent, so probably an order of execution issue; this makes me suspect that sometimes the source code of the duplicate is copied to both the original name and the 2_... name, meaning that the duplicate is never compiled? perhaps if the ordering was restricted somehow in the perl generator script it would help?
- using BUILD_THREADS=1 also seems to work, supporting the order of execution theory (though again the intermittent nature means it is hard to be absolutely sure)
Tnx!
@hamishcunningham
Tried building your example at https://gitlab.com/hamishcunningham/unphone/-/tree/master/examples/2022 and that works without any problem but then again there are no duplicates in this case
I had to create a dummy private.h with the definitions for SSID and PSK, should it contain something more?
@plerup thanks! That example should generate two copies of WebServer.cpp, one from the AsyncWebServer library and one from the Arduino WebServer library...? Then my error is that the linker sometimes can't find the AsyncWebServer symbols.
(SSID and PSK defs should be all you need in private.h)
PS I guess that if you don't have this repo
https://gitlab.com/hamishcunningham/the-internet-of-things
also checked out in $HOME and haven't run support/magic.sh setup then the latter copy of WebServer won't be present
Did this but still no duplicates or problems.
This is what I did in a clean virtual Ubuntu 20.04
git clone https://gitlab.com/hamishcunningham/the-internet-of-things.git
git clone https://gitlab.com/hamishcunningham/unphone.git
cd the-internet-of-things/
support/magic.sh setup
cd
cd unphone/examples/2022
./../../the-internet-of-things/support/magic.sh mkesp
Please advice
Sorry, I updated the libraries and also renamed the clashing file. If you move it back:
mv ~/unphone/lib/ESPAsyncWebServer/src/XXXWebServer.cpp ~/unphone/lib/ESPAsyncWebServer/src/WebServer.cpp
Then you will get the 2_WebServer.cpp file appearing in /tmp/mkESP/sketch_featheresp32.
Subsequent builds fail intermittently with duplicate or missing symbol linker errors, e.g.
~/unphone/examples/2022 $ make -f ~/the-internet-of-things/support/tooling/makeEspArduino/makeEspArduino.mk ESP_ROOT=~/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32 BOARD=featheresp32 CHIP=esp32 LIBS=~/unphone/lib SKETCH=sketch/sketch.ino all
Linking /tmp/mkESP/sketch_featheresp32/sketch.bin
Versions: cdd4aa0-dirty, 2.0.2
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal._Z13initWebServerv+0x38): undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal._Z13initWebServerv+0x3c): undefined reference to `AsyncWebServer::onNotFound(std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal._Z13initWebServerv+0x40): undefined reference to `AsyncWebServer::begin()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal.startup._GLOBAL__sub_I__Z7getHtmlR6StringPPKciP13replacement_ti+0x4): undefined reference to `AsyncWebServer::AsyncWebServer(unsigned short)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o):(.literal.exit._GLOBAL__sub_D__Z7getHtmlR6StringPPKciP13replacement_ti+0x0): undefined reference to `AsyncWebServer::~AsyncWebServer()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o): in function `initWebServer()':
/home/hamish/unphone/examples/2022/sketch/httpd.cpp:176: undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:188: undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:199: undefined reference to `AsyncWebServer::on(char const*, unsigned char, std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:201: undefined reference to `AsyncWebServer::onNotFound(std::function<void (AsyncWebServerRequest*)>)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /home/hamish/unphone/examples/2022/sketch/httpd.cpp:203: undefined reference to `AsyncWebServer::begin()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o): in function `_GLOBAL__sub_I__Z7getHtmlR6StringPPKciP13replacement_ti':
/home/hamish/unphone/examples/2022/sketch/httpd.cpp:113: undefined reference to `AsyncWebServer::AsyncWebServer(unsigned short)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(httpd.cpp.o): in function `_GLOBAL__sub_D__Z7getHtmlR6StringPPKciP13replacement_ti':
/home/hamish/unphone/examples/2022/sketch/httpd.cpp:113: undefined reference to `AsyncWebServer::~AsyncWebServer()'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(WebRequest.cpp.o):(.literal._ZN21AsyncWebServerRequest13_onDisconnectEv+0x0): undefined reference to `AsyncWebServer::_handleDisconnect(AsyncWebServerRequest*)'
/home/hamish/the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/tools/xtensa-esp32-elf/bin/../lib/gcc/xtensa-esp32-elf/8.4.0/../../../../xtensa-esp32-elf/bin/ld: /tmp/mkESP/sketch_featheresp32/user_obj.ar(WebRequest.cpp.o): in function `AsyncWebServerRequest::_onDisconnect()':
/home/hamish/unphone/lib/ESPAsyncWebServer/src/WebRequest.cpp:229: undefined reference to `AsyncWebServer::_handleDisconnect(AsyncWebServerRequest*)'
collect2: error: ld returned 1 exit status
make: *** [/home/hamish/the-internet-of-things/support/tooling/makeEspArduino/makeEspArduino.mk:313: /tmp/mkESP/sketch_featheresp32/sketch.bin] Error 1
It isn't consistent :( but I've also seen it in other similar builds.
What appears to have happened is that the file from the-internet-of-things/support/tooling/Arduino/hardware/espressif/esp32/libraries/WebServer/src/WebServer.cpp has been copied into 2_WebServer.cpp, whereas it should have been copied from unphone/lib/ESPAsyncWebServer/src/WebServer.cpp.
Perhaps make is interleaving two tasks during parallel execution, or the script that sets up the copy is backgrounded, or similar?
Or perhaps which copy of WebServer.cpp is reached first during the build gets copied into 2_WebServer.cpp, and sometimes this is from one tree and sometimes the other? (A fix might be for the perl script to order the list of candidates by full path before processing?)
Thanks, this example made me realize that the current solution for handling clashing file names is only working in some cases.
I will have to implement a new one. Stay tuned...
@hamishcunningham
Please test latest commit
@plerup have only done a brief test but seems to work so far :)
(I can see both 1_... and 2_... versions of the duplicate being compiled.)
Thanks! Will try more tomorrow.
Pretty sure this works now :)
Can be closed from my point-of-view, thanks!