Mass storage upload fails on linux64 when no 32-bit loader is installed

Question

Mass storage upload fails on linux64 when no 32-bit loader is installed

matthijskooijman opened this issue 5 years ago · 15 comments

On a 64-bit Ubuntu Disco system, mass-storage uploads fail. In the IDE window, I get:

java.io.IOException: Cannot run program "/home/matthijs/.arduino15/packages/STM32/tools/STM32Tools/1.3.1/tools/linux/massStorageCopy": error=2, No such file or directory

Closer inspection shows that the file referenced actually does exist, but is a 32-bit binary. Executing it fails with a ENOENT, which is expected (according to https://unix.stackexchange.com/a/13409/233460) when no 32-bit loader is installed.

Manually starting the binary also fails, and strace shows it is the actual execve systemcall that fails:

execve("/home/matthijs/.arduino15/packages/STM32/tools/STM32Tools/1.3.1/tools/linux/massStorageCopy", ["/home/matthijs/.arduino15/packages/STM32/tools/STM32Tools/1.3.1/tools/linux/massStorageCopy"], 0x7ffe812fecd8 /* 95 vars */) = -1 ENOENT (No such file or directory)

A linux64 version is included, which does run and work as expected (tested by editing platform.txt).

Looking at the platform.txt, at first glance it looks like there is a typo in the linux64 version: https://github.com/stm32duino/Arduino_Core_STM32/blob/6808f06a3c183f22784f0486a3034644535c5b2b/platform.txt#L160-L161

As you can see, both refer to the 32-bit version. However, on closer inspection, it turns out that the Arduino IDE does not actually distinguish between linux32 and linux64 currently, so to really fix this, I guess the IDE needs to be changed (I'll file a bug report for that as well).

As a workaround, you can enable execution of 32-bit binaries on a 64-bit Linux Debian-derived (e.g. Ubuntu) system by enabling multiarch and installing a 32-bit libc:

dpkg --add-architecture i386
apt-get update
apt-get install libc6:i386

With that, things work as expected again. But ideally, you would use the 64-bit version on a 64-bit system, if Arduino adds support for that.

To Reproduce

Get a 64-bit system without 32-bit compatibility enabled (default on Ubuntu, it seems).
Get any sketch, and select any STM32 board (e.g. Nucleo63 -> F401RE)
Select upload method mass storage
Press upload

Desktop (please complete the following information):

OS: Linux Ubuntu Disco
Arduino IDE version: Git master (a1448876a1115c9d3ee9e88f29a15bb081a27816), bit newer than 1.8.11
STM32 core version: 1.7.0
Upload method: Mass Storage

Board (please complete the following information):

Name: Nucleo F401RE

Answer 1 · 2019-10-09T13:15:58.000Z

Hi @matthijskooijman
Right this is a know issue for Arduino IDE.
I guess adding a note in the FAQ section of the wiki:
https://github.com/stm32duino/wiki/wiki/FAQ
Do not hesitate to add one subsection to it 😉

Answer 2 · 2019-10-09T13:35:22.000Z

While writing an issue for the Arduino IDE, I actually realized that this can (and IMHO should) be solved in the STM core after all.

Looking more closely, it seems that e.g. the AVR tools on my system only ship 64-bit versions. Looking in ~/.arduino15/package_index.json, I see that the avr-gcc tool is defined as:

        {
          "name": "avr-gcc",
          "version": "7.3.0-atmel3.6.1-arduino5",
          "systems": [
(... more entries removed ...)
            {
              "size": "38710087",
              "checksum": "SHA-256:2ff12739d7ed09688d6b3c2c126e8df69b5bda1a07ab558799f0e576571e0e1d",
              "host": "i686-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino5-i686-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino5-i686-pc-linux-gnu.tar.bz2"
            },
            {
              "size": "39114120",
              "checksum": "SHA-256:3effed8ffa1978b6e4a46f1aa2acc629e440b4d77244f71f9b79a916025206fb",
              "host": "x86_64-linux-gnu",
              "archiveFileName": "avr-gcc-7.3.0-atmel3.6.1-arduino5-x86_64-pc-linux-gnu.tar.bz2",
              "url": "http://downloads.arduino.cc/tools/avr-gcc-7.3.0-atmel3.6.1-arduino5-x86_64-pc-linux-gnu.tar.bz2"
            }
          ]
        },

IOW, the linux32 and linux64 systems have different tarballs, so the installer can simply install the right binaries and platform.txt no longer has to distinguish them.

Looking at package_stm_index.json, I see that both linux32 and linux64 use the same tarball:

            {
              "host": "x86_64-pc-linux-gnu",
              "url": "https://github.com/stm32duino/Arduino_Tools/releases/download/1.3.1/STM32Tools-1.3.1-linux.tar.bz2",
              "archiveFileName": "STM32Tools-1.3.1-linux.tar.bz2",
              "checksum": "SHA-256:82088da960965fe982510fca85993b098246932a53d27afd2f87fe912f323ff3",
              "size": "1005934"
            },
            {
              "host": "i686-pc-linux-gnu",
              "url": "https://github.com/stm32duino/Arduino_Tools/releases/download/1.3.1/STM32Tools-1.3.1-linux.tar.bz2",
              "archiveFileName": "STM32Tools-1.3.1-linux.tar.bz2",
              "checksum": "SHA-256:82088da960965fe982510fca85993b098246932a53d27afd2f87fe912f323ff3",
              "size": "1005934"
            }

So I think the proper solution would be to separate that tarball into a 32-bit and 64-bit version and use the correct one for each system, keeping the paths inside the tarball the same so platform.txt does not need to distinguish.

It seems this is already done for the gcc tool, and the CMSIS tool is platform-independent, so can just use the same tarball for all platforms.

Answer 3 · 2019-10-09T13:50:18.000Z

Yes, this probably could be done like that.

Answer 4 · 2019-10-10T08:18:52.000Z

Thinking about that, there will be always an issue for those using the git repo which include all tools.
This will not solve the pb.
I guess the best way would be to move the massStorageCopy binary to a simple bash script.

Answer 5 · 2019-10-10T09:21:48.000Z

But this repo contains no tools at all? So I think you mean the https://github.com/stm32duino/Arduino_Tools repository?

If someone is using that, you already need to modify platform.txt (or use a .local.txt) to make the IDE find the tools, right?

One alternative for using the platform from git with the default tools can be to install the platform through the boards manager as normal (also installing the tools), and then putting the platform git checkout (i.e. this repo) in your Sketchbook/hardware directory under a different name (e.g. Sketchbook/hardware/STM32-git/stm32). The changed name makes the IDE show both platforms separately in the Boards menu (so changing the name in platform.txts to distinguish them is also helpful). This is the trick I've been using for AVR as well, see arduino/Arduino#9237

I haven't tried this for the STM32 platform yet, but if it works, that's probably something to document and recommend, since it is even more flexible than only installing the core from git.

Answer 6 · 2019-10-10T09:26:52.000Z

Yes I talk about https://github.com/stm32duino/Arduino_Tools
I've already documented one way to use git repo in the Wiki (I know there is other possibility but I could not assume all possible way):
https://github.com/stm32duino/wiki/wiki/Using-git-repository

And no you do not need to modify platform.txt to use this method.

Answer 7 · 2019-10-11T07:50:13.000Z

Ah, I see the method you proposed. It is really similar to what I suggested, except that you delete the files from ~/.arduino15 rather than rename the core from git in the sketchbook.

However, with "your" method, having a split tools tarball also still works, since the tools are installed by the IDE as well.

Previously, you said:

Thinking about that, there will be always an issue for those using the git repo which include all tools.
This will not solve the pb.

I think this refers to using a modified Arduino_Tools repository, not a modified Arduino_Core_STM32 repository. And for doing that, I think you need to modify platform.txt anyway, is what I was saying.

Answer 8 · 2019-10-11T07:56:21.000Z

However, with "your" method, having a split tools tarball also still works, since the tools are installed by the IDE as well.

Rigth

I think this refers to using a modified Arduino_Tools repository, not a modified Arduino_Core_STM32 repository. And for doing that, I think you need to modify platform.txt anyway, is what I was saying.

No, I delete the tools directory then clone the Arduino_Tools repo so no need to update platform.txt.
Anyway, as I said there is lot of method for use git repo, this is finally up to end user to know what he does.

Answer 9 · 2019-10-11T08:44:34.000Z

No, I delete the tools directory then clone the Arduino_Tools repo so no need to update platform.txt.

Ah, I guess that could also work indeed. That probably does not work by cloning it into the sketchbook, only into the ~/.arduino15 directory.

What you could then perhaps do:

Modify platform.txt in the officially shipped core to refer to a generic bin directory (instead of the win / linux / macosx distinction as now).
When generating the tools tarball, rename win, linux, linux64 or macosx to bin depending on the platform (similar to how you now, I think, strip the binaries for other platforms from the tarball).
Then, when installing custom tools from git, you replace the tools in ~/.arduino15/... with a git clone and after cloning you simply create a bin symlink to the appropriate folder (though while writing this, I'm not entirely sure how this would work on Windows. I think NTFS can do something like symlinks?)

This would still not require modifying platform.txt, only adding a symlink in the tools repo (where you are already messing with files anyway).

Does that make sense?

Answer 10 · 2019-10-11T12:17:46.000Z

Arduino provide this feature and I do not want have manually create a symlink to use the git repo. This should work as it is.
I'm currently removing the linux64 directory then no more needs about all that stuff.

Answer 11 · 2019-10-11T13:18:05.000Z

I'm currently removing the linux64 directory then no more needs about all that stuff.

So that means always using the 32-bit binaries, also on 64-bit linux? That requires people to install additional libraries, which might not be so easy for everyone (e.g on Debian it is more than just installing an extra package). Also, that would not work if you ever want to support arm rather than x86 (which the Arduino IDE already does).

Answer 12 · 2019-10-11T13:21:03.000Z

No, I didn't tell that.
As I said I will provide a bash script instead of a binary.
And for other binary I've updated the bash script to call the correct binary depending of the host arch (i.e. dfu-utils).

Answer 13 · 2019-10-11T13:23:28.000Z

Ok, then I misunderstood. I guess a script would work, but it doesn't seem too elegant to duplicate the arch detection again in a script. But it does make it easier to run with tools from git, I guess, so if that's what you prefer :-)

Thanks!

Answer 14 · 2019-10-11T13:36:12.000Z

Well, this avoid to duplicate files in the repo and have it functional without have to add a platform.local.txt, or create a symlink.

Only one arch detection have to be performed:

if [ `uname -m` ==  "x86_64" ]; then
  DFU_UTIL=${DIR}/dfu-util_x86_64/dfu-util
else
  DFU_UTIL=${DIR}/dfu-util/dfu-util
fi

I guess this is not very disturbing

Answer 15 · 2019-10-14T08:07:11.000Z

Closed by #43