dart-lang/sdk

Allow AOT executables to be cross-compiled

nex3 opened this issue Β· 47 comments

nex3 commented

Currently application snapshots (as described here) can only be compiled for the architecture of the machine doing the compilation. This makes releasing cross-platform apps using these snapshots much more difficult, requiring multiple machines (including at least one physical machine, since OS X isn't virtualizable) to compile for all platforms.

It seems that the compilation process could do the initial run on the current platform, but then preserve the profiling information and use it to cross-compile the result to other platforms.

App snapshots are tied to the target ABI of the VM, not the host OS and not the host ABI. If you create an app snapshot using an IA32 VM, you can run the same snapshot on Windows, Mac or Linux. If you create an app snapshot using an X64 VM on Mac or Linux, you run it on the other (they share the sysv ABI), but not Windows (which has its own ABI). If you create an app snapshot using a SIMARM VM on your desktop, you can run it with an ARM VM on, say, an Android device (this is how Flutter works).

We will not produce VMs that target multiple ABIs. The assumption of a single target ABI as a build-time decision is quite deep in the VM.

nex3 commented

I want to push back strongly on this. Startup performance is very important for providing my users with a good experience, and the Dart VM's startup performance without using application snapshots compares poorly to just about any other language (including Ruby, whose Sass implementation we're in the process of deprecating because of performance issues).

Other languages are able to do this. Easy cross-compilation is a major reason that Go is so popular for writing command-line tools, and if Dart provided it with similar ease it would make the server-side Dart world very compelling. But without it, Dart users are forced to deal with sub-par performance when they could easily get performance and portability with another language.

nex3 commented

Ping... efficient startup speeds across all platforms are a very important customer feature.

nex3 commented

I haven't received any response, so I'm re-opening this. It's a pressing customer issue.

One workaround is https://github.com/filiph/dartbin which creates a go wrapper. Not Dart2 compatible though and it requires go as dependency.

nex3 commented

Updating to AOT executables, since that's not the best way to distribute CLI applications.

nex3 commented

@mit-mit has told me offline that "getting to full cross-compilation is going to be a long journey", but "the journey has already started; we just don't have a timeline for when it will end".

Looking forward to see this feature! Loving to work with dart ❀

This feature is a show stopper for us in order to adopt Dart in AWS Lambda. Since the AWS CLI runs the compiled executable in a Docker container, offline development is only possible in a Linux environment. Manual deployment without CI/CD is also impossible in Mac and Windows.

Also a cross-compile option for linux ARM architectures please!

Just found this wiki: https://github.com/dart-lang/sdk/wiki/Building-Dart-SDK-for-ARM-processors#building

I wonder if dart2native and the produced executable/snapshot works on RaspberryPi πŸ€”?

Ah, that would be awesome to have a cross compilation !
I just was looking for this functionality, but found this issue.
If my vote for this topic can change the priorities for dart2native tools improvement - I would be really glad.

I'm trying to use Dart for several projects for Raspberry, but that's hard to easily win this battle without ability to get ARM binary using my host OS (MacOS).

What's about plan? Will this feature be included once?

Is there an update to this? Really interested in cross-compilation!

Would something like this be an interesting mid-term solution?

 $ dart2native foo.dart \
    --gen-snapshot out/ProductSIMARM/dart-sdk/bin/utils/gen_snapshot \
    --aot-runtime out/ProductXARM/dart-sdk/bin/dartaotruntime

jpnurmi@9625310

Would something like this be an interesting mid-term solution?

That's pretty close to how it would be implemented in the long term too (if we had resources or if we had somebody willing to contribute external implementation). Few missing pieces to take it all the way:

  • We need to create X-configurations for producing gen_snapshot for all configurations of host-target OSes and architectures.
  • We need to figure out packaging and distribution for these binary artifacts. I don't think including them into SDK is scalable - we need to figure out "out of band" delivery mechanism (which might somewhat complicate Dart SDK release process).
matsp commented

What needs to be done here? I am not an expert at all for this topic but really want to help and make this feature real. If someone is able to give me some lead I am willing to help.

Ah, that would be awesome to have a cross compilation !
I just was looking for this functionality, but found this issue.
If my vote for this topic can change the priorities for dart2native tools improvement - I would be really glad.

I'm trying to use Dart for several projects for Raspberry, but that's hard to easily win this battle without ability to get ARM binary using my host OS (MacOS).

What's about plan? Will this feature be included once?

Have you found any alternative to compile for Rpi devices?
I think the Dart dev team is very busy in these days.

The way I've done before:

@jairoareyes do the development normally on your Ubuntu/mac/win, download the dart SDK to one rPi and setup a lil script that rsync the project files to the pi and runs pub get and compile.

Not "the most" efficient, but it works nicely

Any updates...? Or this will never be done?

There are currently no immediate plans to address this. There is a bunch of improvements one could make to dart2native (e.g. fix how we link runtime with AOT snapshot to make binaries signable, fix packaging for dart2native artifacts, support cross compilation, etc), but we are too occupied with higher priority issues to work on them.

If somebody is willing to contribute, I will be happy to provide design and guidance.

Hi guys, I'm doing some experiments about using multiarch docker images to build Pi/armhf Dart self-contained exe from a x64 host. All is here : https://github.com/abasty/at_dockerfiles/blob/experiment/armv7/README-armv7.md.

I use ugly workarounds to make it possible... I know what I've done so far is bad ! As we use to say in such situation : well it's just a proof of concept !

I need help from people who know how to fix the certificates problem.

I need help from the Dart team to cleanly fix at least two problems :

1/ Build a SDK where the dart CLI can run with qemu armhf transparency (in a container running on x64)
2/ Have some guidance about the File::Delete problem

Details and step by step instructions are in the above mentioned README file.

Thanks!

1/ Build a SDK where the dart CLI can run with qemu armhf transparency (in a container running on x64)

You should already be able to do that. Pass --use-qemu to tools/build.py when building an SDK.

2/ Have some guidance about the File::Delete problem

This is an issue with 32-on-64 emulation and it requires some kernel changes (which did not get into kernel), see https://bugs.launchpad.net/qemu/+bug/1805913. Not much we can do here.

Thanks @mraleph for your time and the very interesting 32-on-64 emulation problem link.

I understand that maintaining host x guest x OS deliveries is almost impossible at this time, it's why I tried to workaround the problem using emulation and official Dart SDK releases. I even tried in qemu-system running a regular Debian armhf distribution.

Finally my main problem is "Why I need to rebuild the Dart SDK to make it run in QEMU ?" i.e. "Why DART_RUN_IN_QEMU_ARMv7 is a compile time constant ?".

This question is very far from the topic of this discussion...

Finally my main problem is "Why I need to rebuild the Dart SDK to make it run in QEMU ?" i.e. "Why DART_RUN_IN_QEMU_ARMv7 is a compile time constant ?".

Because we expect that /proc/cpuinfo gives us important information about host CPU so that we know which CPU features we can rely on and which we can't rely on. QEMU is emulating ARM cpu, but (seemingly) leaves /proc/cpuinfo in-tact, which confuses our /proc/cpuinfo parsing code.

I understand the problem about /proc/cpuinfo when using qemu-arm-static in a chrooted env. It's why I also tested an official Dart SDK in qemu-system-arm, running a regular debian buster armhf kernel. In this case :

# cat /proc/cpuinfo 
processor	: 0
model name	: ARMv7 Processor rev 1 (v7l)
BogoMIPS	: 125.00
Features	: half thumb fastmult vfp edsp thumbee neon vfpv3 tls vfpv4 idiva idivt vfpd32 lpae evtstrm 
CPU implementer	: 0x41
CPU architecture: 7
CPU variant	: 0x2
CPU part	: 0xc0f
CPU revision	: 1

Hardware	: Generic DT based system
Revision	: 0000
Serial		: 0000000000000000
$ uname -a
Linux devarm 4.19.0-16-armmp-lpae #1 SMP Debian 4.19.181-1 (2021-03-19) armv7l GNU/Linux

I was very confident but :

$ dart

===== CRASH =====
si_signo=Illegal instruction(4), si_code=1, si_addr=0xb640e5a8
version=2.13.3 (stable) (Wed Jun 9 12:44:44 2021 +0200) on "linux_arm"
pid=290, thread=292, isolate_group=vm-service(0x344cc00), isolate=vm-service(0x3463200)
isolate_instructions=1c5a500, vm_instructions=1c5a500
  pc 0xb640e5a8 fp 0xb68eea78 Unknown symbol
  pc 0xb640e1f0 fp 0xb68eea98 Unknown symbol
  pc 0xb640dfc8 fp 0xb68eeab8 Unknown symbol
  pc 0xb640dbb8 fp 0xb68eead8 Unknown symbol
  pc 0xb6400a24 fp 0xb68eeafc Unknown symbol
  pc 0xb6901ef8 fp 0xb68eeb74 Unknown symbol
-- End of DumpStackTrace
Aborted

I think it's really another problem here (illegal instruction)...

I will try with some other QEMU emulated machines (I use -M virt for now).

@abasty yeah, you should probably file a different issue. it would help if you also provide some output from running it under gdb. (especially x/20i $pc-16).

maks commented

@mraleph per @jpnurmi method (https://medium.com/flutter-community/cross-compiling-dart-apps-f88e69824639) and patch to dart2native (or I guess just dart compile these days) is it now a matter of having the Dart SDK release process build & publish the "cross-platform" (x86 that produces for arm, arm that produces for x86) versions of gen_snapshot and dartaotruntime that would at least give end users all the building blocks to do cross-compilation without needing to download SDK src and patch/build those tools ourselves, without initially worrying about figuring out what the "out of band" delivery mechanism needs to be?

@maks well, I don't think we want to include gen_snapshot and dart_precompiled_runtime pair for each configuration (which is actually a combination of CPU and OS - something that @jpnurmi did not tackle at all) into an SDK archive.

FWIW there is no complexity in figuring out-of-an-SDK delivery mechanism - we can just use a cloud storage bucket (in the same way Flutter does it). There will be of course some design around release process for such things, how CI testing would look like, etc.

maks commented

@mraleph thanks for quick reply πŸ‘πŸ» and yes sorry that's what I was trying to say, sorry I didn't explain it very well.

What I thought was just that: as part of your the Dart release process, publishing somewhere like a cloud bucket all the combinations of gen_snapshot and dart_precompiled_runtime binaries and not including them in the SDK and then just let me, the user, download the specific ones I need. Of course it would be nice to have that automated by tooling the way Flutter does with downloading its engines, getting the correct matching versions etc, but as a first step I would be happy to just download them vs needing to build them locally out of the sdk src.

And for sure I guess there's a fair bit of work around doing that in the release process, but unfort its not really something that can be done except by the Dart team so I'll keep 🀞🏻 and watch this issue.

@maks well, I don't think we want to include gen_snapshot and dart_precompiled_runtime pair for each configuration (which is actually a combination of CPU and OS - something that @jpnurmi did not tackle at all) into an SDK archive.

If I understand correctly the dart_precompiled_runtime is per target while gen_snapshot is per (host, target) tuple. I'm wondering why this approach was chosen, as new compilers (clang,rust,etc) tend to be retargetable. I believe gcc only supported a single host,target tuple because in 1987 computers had a very limited amount of memory and disk space.

@dvc94ch Note that gen_snapshot does not actually support arbitrary combinations of host and target.

The root of this is in Dart's origins. Dart is originally a JITed language and JITs tend to have host == target. Additionally there is a tight coupling between compilation toolchain and the runtime system (again because it was first and foremost a JITed language). So most of the code was written like that and properly disentangling this is quite an adventure. We managed to disentangle some of it when we were forced to support generating 32-bit ARM code on 64-bit hosts (due to Mac OS X removing support for 32-bit binaries), but there are still more things to refactor.

I used this GitHub action workflow to get cross-platform builds

# This is a basic workflow to help you get started with Actions

name: CI Building native server

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  build:
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
        include:
          - os: ubuntu-latest
            output-name: server-linux
          - os: macOS-latest
            output-name: server-mac
          - os: windows-latest
            output-name: server-windows.exe
    steps:
          - uses: actions/checkout@v2
          - uses: dart-lang/setup-dart@v1.3
          - name: Install dependencies
            run: dart pub get
          - run: mkdir build
          - name: Install Dependencies
            run: dart pub get
          - run: dart compile exe ./bin/server.dart -v -o build/${{ matrix.output-name }}
          - uses: actions/upload-artifact@v1
            with:
                name: native-executables
                path: build

and here is the result

Screen Shot 2022-02-18 at 12 37 14 AM

I think people are more concerned with cross compiling debug builds on a development machine and testing it on a target machine than creating cross compiled releases. There are a few reasons why it's hard even if dart could cross compile or why it's not a great idea.

  • dmg used for macos packaging and squashfs used in appimages or snaps use file systems not commonly found on other platforms, I expect it to be quite a bit of work to get it working.
  • when building something like an appimage you want to link against old libraries to make the artifact usable on as many systems as possible

It is already possible today by using the kernel_blob.bin (cross compiled windows apps work already) and I'm working on a tool that exposes this (among other things like packaging, signing and publishing) to flutter developers.

Is there any way to build the aot assembly for iOS from Linux/windows? kernel_blob.bin won't work on iOS right? Got other platforms working, still figuring out iOS.

So https://github.com/cloudpeers/xbuild would really benefit from being able to create aot snapshots for ios from linux/windows. It's the only reason you still need a mac for developing ios apps on non jailbroken ios devices. Of course you still need an apple developer subscription.

xbuild now supports signing/notarizing/stapling the appbundle, creating a dmg and signing/notarizing/stapling the dmg from linux and windows. This makes it really easy to provide users with a bugfix for testing.

maks commented

Coming back to this because it occurred to me to ask: how does Flutter already manage to do this?

Flutter is able to both:

  1. compile Dart code on my x86 laptop and deploy it (with hot reload) to a Flutter app running on a Arm emulator or device
  2. compile Dart code and package it into a release apk that can run a Arm emulator or device

In both the above, my assumption is that cross compilation is happening for the Flutter apps Dart code and if so is this all Flutter specific code and is there a reason it cannot be used for Dart cli as well as Flutter apps?

My current use case for cross compiling is trying to develop for a RISC-V device where the CPU is relatively slow and also has a modest (512MB) amount of RAM so running Dart on the cli (which of course requires compiling) is very slow and so on device development is very slow and painful.

@maks Flutter does it in the same way which is described in #28617 (comment) and other comments on the thread.

They build a bunch of gen_snapshot binaries for each host -> target combination they support as well as building a bunch of libflutter.so for all target configurations they support. flutter tool then dynamically downloads necessary pieces.

Dart's dart compile exe could do the same. For combinations that we already support the biggest open question is packaging this all together, rather than anything else. For combinations that we don't support (e.g. building Windows exe on Linux) more work needs to be done in VM innards.

maks commented

Thanks @mraleph πŸ‘πŸ» that gives me a good place to go start looking into the Flutter code to learn how they build the separate gen_snapshot binaries for each target.

In regards to the packaging and distribution for doing this in Dart, is there the possibility of reusing the infrastructure that Flutter has setup or do the Dart & Flutter projects prefer to keep these separate?

Thanks @mraleph πŸ‘πŸ» that gives me a good place to go start looking into the Flutter code to learn how they build the separate gen_snapshot binaries for each target.

You will have to read their buildbot recipes for that (https://flutter.googlesource.com/recipes). It's all done on the CI and uploaded to the cloud storage bucket.

Ultimately it is just about configuring the engine build with the right flags.

In regards to the packaging and distribution for doing this in Dart, is there the possibility of reusing the infrastructure

We will probably have some variation of the same infrastructure. We had a brief chat with @mit-mit and we might lay some ground work for this next quarter.

I think "dart can't cross compile" is simply a lie !!!

Because apparently, flutter can compile an arm apk file directly on window, linux amd64 machine.

With Mac moving over to arm. Being able to cross-compile because even more important. Because people on mac will not be able to run a windows vm and do the compiling there. So we will need 2 seperated machines now just to get our executables for all platforms...

But I'll need to ask then: Why does the dart compile command mention the --target-os option to begin with? Why does it not print a warning that --target-os is currently ignored when entered?

With MacOS having two different architectures arm and intell I need to have two Mac machines to compile all the variants. Note that not all CI platforms, as GitHub offer Arm Mac machines. So you would need to search to cover your Mac compilation needs.

I used this GitHub action workflow to get cross-platform builds

# This is a basic workflow to help you get started with Actions

name: CI Building native server

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  build:
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
        include:
          - os: ubuntu-latest
            output-name: server-linux
          - os: macOS-latest
            output-name: server-mac
          - os: windows-latest
            output-name: server-windows.exe
    steps:
          - uses: actions/checkout@v2
          - uses: dart-lang/setup-dart@v1.3
          - name: Install dependencies
            run: dart pub get
          - run: mkdir build
          - name: Install Dependencies
            run: dart pub get
          - run: dart compile exe ./bin/server.dart -v -o build/${{ matrix.output-name }}
          - uses: actions/upload-artifact@v1
            with:
                name: native-executables
                path: build

and here is the result

Screen Shot 2022-02-18 at 12 37 14 AM

Hi, thank you for this solution. I am new to git workflows/actions!?
I wonder about the Install Dependencies step. Why have the install dependencies 2 times?

          - name: Install dependencies   *1
            run: dart pub get
          - run: mkdir build
          - name: Install Dependencies   *2

My root folder seems like this.

root
     - .github
          - workflows
               - github-actions-demo.yml
               
     - dart_server
          - bin
               - dart_server.dart
          - pubspec.yaml

     - dart_share                       //include the object, and function these were used in dart_share
          - pubspec.yaml

So I should change some steps in github-actions-demo.yml, right?

UPDATE
I am using this.

# This is a basic workflow to help you get started with Actions

name: CI Building native server

# Controls when the workflow will run
on:
  # Triggers the workflow on push or pull request events but only for the main branch
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

jobs:
  build:
    runs-on: ${{ matrix.os }}

    strategy:
      matrix:
        os: [ubuntu-latest, windows-latest, macOS-latest]
        include:
          - os: ubuntu-latest
            output-name: server-linux
          - os: macOS-latest
            output-name: server-mac
          - os: windows-latest
            output-name: server-windows.exe
    steps:
      - uses: actions/checkout@v2
      - uses: dart-lang/setup-dart@v1.3
      - name: Install dependencies for share project
        working-directory: ./share_dart
        run: dart pub get
      - name: Install dependencies for server project
        working-directory: ./server
        run: dart pub get
      - run: mkdir build
      - name: Install Dependencies for share project
        working-directory: ./share_dart
        run: dart pub get
      - name: Install Dependencies for server project
        working-directory: ./server
        run: dart pub get
      - run: dart compile exe ./server/bin/server.dart -v -o build/${{ matrix.output-name }}
      - uses: actions/upload-artifact@v1
        with:
          name: native-executables
          path: build

Alternative approach Using node.js to create executables

I have a work around that I would like to share as it does not require any GitHub services, etc. It can all be done from command line.

I am compiling to JS, then using node.js tools to convert to executable. I wrote a bash script to simplify the process. This is certainly not going to be the most efficient and probably has gaps (that I haven't found), but for the command line tools I've been building it has worked well so far.

Until it is supported natively...

Example Steps

Create the js files

dart compile js bin/hello.dart -o js/hello.js

Command line arguments don’t work in generated js

It turns out that for some reason the command line arguments passed to the program are not picked up by the generated node program, unless you do a special little trick. I found out about the trick from this site: https://japhr.blogspot.com/2014/07/command-line-arguments-with-nodejs-and.html

Basically the trick is to add a function to the generated node.js file:

function dartMainRunner(main, args) {
  main(process.argv.slice(2)); 
}

Create the executable using node.js tools

Create the package.json file in js directory

{
    "name": "hello",
    "version": "0.0.1",
    "description": "Say hello.",
    "main": "hello.js",
    "bin": "hello.js", 
    "author": "anybody",
    "license": "MIT"
}

Then build the executable

pkg .

Script to do all of this

In order to automate this I created a script dart2js in ~/local/bin that does all of these steps.

#!/bin/bash

if [[ "$1" == "" ]]; then
  echo -e "Usage:\n\t$0 [basename]"
  exit 1
fi

if [[ ! -f bin/$1.dart ]]; then
  echo "bin/$1.dart does not exist"
  exit 2
fi

dart compile js bin/$1.dart -o js/$1.js
if [ $? -ne 0 ]; then
  echo "Compiling js failed, exiting build script"
  exit 3
fi

wrapper='function dartMainRunner(main, args) { main(process.argv.slice(2)); }';
echo $wrapper >> js/$1.js

cat <<EOT > js/package.json
{
    "name": "$1",
    "version": "0.0.1",
    "description": "Do amazing $1 stuff.",
    "main": "$1.js",
    "bin": "$1.js", 
    "author": "Geoff Ruscoe",
    "license": "MIT"
}
EOT

cd js
pkg .

Alternative approach Using node.js to create executables

I have a work around that I would like to share as it does not require any GitHub services, etc. It can all be done from command line.

I am compiling to JS, then using node.js tools to convert to executable. I wrote a bash script to simplify the process. This is certainly not going to be the most efficient and probably has gaps (that I haven't found), but for the command line tools I've been building it has worked well so far.

Until it is supported natively...

Example Steps

Create the js files

dart compile js bin/hello.dart -o js/hello.js

Command line arguments don’t work in generated js

It turns out that for some reason the command line arguments passed to the program are not picked up by the generated node program, unless you do a special little trick. I found out about the trick from this site: https://japhr.blogspot.com/2014/07/command-line-arguments-with-nodejs-and.html

Basically the trick is to add a function to the generated node.js file:

function dartMainRunner(main, args) {
  main(process.argv.slice(2)); 
}

Create the executable using node.js tools

Create the package.json file in js directory

{
    "name": "hello",
    "version": "0.0.1",
    "description": "Say hello.",
    "main": "hello.js",
    "bin": "hello.js", 
    "author": "anybody",
    "license": "MIT"
}

Then build the executable

pkg .

Script to do all of this

In order to automate this I created a script dart2js in ~/local/bin that does all of these steps.

#!/bin/bash

if [[ "$1" == "" ]]; then
  echo -e "Usage:\n\t$0 [basename]"
  exit 1
fi

if [[ ! -f bin/$1.dart ]]; then
  echo "bin/$1.dart does not exist"
  exit 2
fi

dart compile js bin/$1.dart -o js/$1.js
if [ $? -ne 0 ]; then
  echo "Compiling js failed, exiting build script"
  exit 3
fi

wrapper='function dartMainRunner(main, args) { main(process.argv.slice(2)); }';
echo $wrapper >> js/$1.js

cat <<EOT > js/package.json
{
    "name": "$1",
    "version": "0.0.1",
    "description": "Do amazing $1 stuff.",
    "main": "$1.js",
    "bin": "$1.js", 
    "author": "Geoff Ruscoe",
    "license": "MIT"
}
EOT

cd js
pkg .

this method produces much big sized binary. For simple hello world example, the binary maybe almost 100MB.

I would prefer to use github action using matrix to build the aot binary.