Support remote compilation

Question

Support remote compilation

waruqi opened this issue 5 years ago · 11 comments

waruqi commented 5 years ago

Roadmap

socket io and coroutine scheduler
xmake connect xxx.xxx.xxx.xxx
xmake (remote build)
xmake config (remote configuration)
xmake run (remote run)

Refs

#165 (comment)
#593

Answer 1 · 2022-02-15T17:21:52.000Z

+1 for this

I was thinking about something like this :

xmake build --host host.com --user me --pass security

or using a custom toolchain maybe

xmake build --toolchain=gcc@host

toolchain("gcc@host")
    set_kind("remote")
    set_user("me")
    ...

I'm not sure if it's possible to make it simple to integrate in an IDE if you have to run xmake connect first

Answer 2 · 2022-02-15T23:38:43.000Z

This feature need take a lot of time and work, and maybe I'll start supporting it again in the future, but not now.

Of course, if more people request this feature, I'll also consider it when I have time.

Answer 3 · 2022-04-08T03:48:38.000Z

做一个插件,可不可以用git来同步远程与本地的源文件，之后在远程建一个socket服务器来接受编译的命令，编译完成，在本地用pull同步结果。

Make a plugin,Maybe we can use 'git' to synchronize remote and local source files, and then run a 'socket server' remotely to accept compiled commands, etc., compile finish ，then we can synchronize the results with 'pull' locally.

Answer 4 · 2022-04-08T04:13:16.000Z

做一个插件,可不可以用git来同步远程与本地的源文件，之后在远程建一个socket服务器来接受编译的命令，编译完成，在本地用pull同步结果。

Make a plugin,Maybe we can use 'git' to synchronize remote and local source files, and then run a 'socket server' remotely to accept compiled commands, etc., compile finish ，then we can synchronize the results with 'pull' locally.

现在的问题不是怎么做，是我没时间精力做

Answer 5 · 2022-04-15T15:49:49.000Z

Start service

$ xmake service
<remote_build_server>: listening 0.0.0.0:90091 ..

with verbose logs

$ xmake service -vD
<remote_build_server>: listening 0.0.0.0:90091 ..

Start and stop service with daemon mode

$ xmake service --start
$ xmake service --restart
$ xmake service --stop

Configure service on server side

run xmake service will generate a default service.conf file in ~/.xmake/service.conf

edit it and modify server user name for using git push to sync files.

{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        server = {
            listen = "0.0.0.0:90091"
        }
    }
}

Configure service on client side

~/.xmake/service.conf

{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "192.168.56.101:90091",
        }
    }
}

Import the given configuration file

$ xmake service --config=/tmp/service.conf

Connect remote build service on client side

enter project directory

$ xmake create test
$ cd test
$ xmake service --connect 
<remote_build_client>: connect 192.168.56.110:90091 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 192.168.56.110:90091 ..
Scanning files ..
Comparing 3 files ..
    [+]: src/main.cpp
    [+]: .gitignore
    [+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!

Build project with remote mode

$ xmake
<remote_build_client>: run xmake in 192.168.56.110:90091 ..
checking for platform ... macosx
checking for architecture ... x86_64
checking for Xcode directory ... /Applications/Xcode.app
checking for Codesign Identity of Xcode ... Apple Development: waruqi@gmail.com (T3NA4MRVPU)
checking for SDK version of Xcode for macosx (x86_64) ... 11.3
checking for Minimal target version of Xcode for macosx (x86_64) ... 11.4
[ 25%]: ccache compiling.release src/main.cpp
[ 50%]: linking.release test
[100%]: build ok!
<remote_build_client>: run command ok!

Run target with remote mode

$ xmake run
<remote_build_client>: run xmake run in 192.168.56.110:90091 ..
hello world!
<remote_build_client>: run command ok!

Rebuild target with remote mode

$ xmake -rv
<remote_build_client>: run xmake -rv in 192.168.56.110:90091 ..
[ 25%]: ccache compiling.release src/main.cpp
/usr/local/bin/ccache /usr/bin/xcrun -sdk macosx clang -c -Qunused-arguments -arch x86_64 -mmacosx-version-min=11.4 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG -o build/.objs/test/macosx/x86_64/release/src/main.cpp.o src/main.cpp
[ 50%]: linking.release test
"/usr/bin/xcrun -sdk macosx clang++" -o build/macosx/x86_64/release/test build/.objs/test/macosx/x86_64/release/src/main.cpp.o -arch x86_64 -mmacosx-version-min=11.4 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk -stdlib=libc++ -Wl,-x -lz
[100%]: build ok!
<remote_build_client>: run command ok!

Configure project with remote mode

$ xmake f --xxx --yy

Sync files to remote server

use git to commit code and run sync

$ git add .
$ git commit -a -m "..."
$ xmake service --sync 
<remote_build_client>: sync files in 192.168.56.110:90091 ..
Scanning files ..
Comparing 3 files ..
    [+]: src/main.cpp
    [+]: .gitignore
    [+]: xmake.lua
3 files has been changed!
Archiving files ..
Uploading files with 1372 bytes ..
<remote_build_client>: sync files ok!

Disconnect remote service for the current project

$ xmake service --disconnect
<remote_build_client>: disconnect 192.168.56.110:90091 ..
<remote_build_client>: disconnected!

View service logs

$ xmake service --logs

Clean all built and cache files for the current project on server

$ cd projectdir
$ xmake service --clean

TODO

Pull built target files to local host

Answer 6 · 2022-04-15T15:58:04.000Z

I have removed git/ssh dependences

Some issues related to git/OpenSSH server on windows

https://github.com/PowerShell/Win32-OpenSSH/wiki/Setting-up-a-Git-server-on-Windows-using-Git-for-Windows-and-Win32_OpenSSH

Set powershell as default shell

Fix git push ruki@host:C:/xx/test failed, it will cause that sync files failure.
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShell -Value "C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe" -PropertyType String -Force
New-ItemProperty -Path "HKLM:\SOFTWARE\OpenSSH" -Name DefaultShellCommandOption -Value "/c" -PropertyType String -Force
Set git/mingw64/bin to %PATH%

to solve git-receive-pack not found!
$gitPath = Join-Path -Path $env:ProgramFiles -ChildPath "git\mingw64\bin"
$machinePath = [Environment]::GetEnvironmentVariable('Path', 'MACHINE')
[Environment]::SetEnvironmentVariable('Path', "$gitPath;$machinePath", 'Machine')
FIXME

it still be failure when we run git push to windows server. : (
Enumerating objects: 6, done.
Counting objects: 100% (6/6), done.
Delta compression using up to 8 threads
Compressing objects: 100% (5/5), done.
Writing objects: 100% (6/6), 1.25 KiB | 638.00 KiB/s, done.
Total 6 (delta 0), reused 0 (delta 0), pack-reused 0
send-pack: unexpected disconnect while reading sideband packet
fatal: the remote end hung up unexpectedly

Answer 7 · 2022-04-16T13:46:01.000Z

I have supported it.

Answer 8 · 2022-04-18T12:28:38.000Z

Can I build the project on the remote machine, but run it on my local machine? How can I install the built binaries on localhost?

Answer 9 · 2022-04-18T12:39:34.000Z

Can I build the project on the remote machine, but run it on my local machine? How can I install the built binaries on localhost?

Build project and run target on the remote machine. And we can also pull built target to the local machine (but I have not supported it yet.)

Answer 10 · 2022-04-30T10:02:22.000Z

User authorization

At the moment I don't have time to do perfect encryption and authentication based on ssl, but I have implemented some basic user authorization authentication. On a LAN it should also avoid most of the security problems.

We can support two authentication modes

User password mode
Token mode

Token mode

Generate default configuration in server

We run xmake service to generate a default configuraion. It will generate an initial token.

$ xmake service
generating the config file to /Users/ruki/.xmake/service.conf ..
an token(4b928c7563a0cba10ff4c3f5ca0c8e24) is generated, we can use this token to connect servi
ce.
$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691",
            token = "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        server = {
            listen = "0.0.0.0:9691",
            workdir = "/Users/ruki/.xmake/service/remote_build"
        }
    },
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        known_hosts = { }
    }
}

Add token in client

Then we add this token in the client configuraion.

$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691",
            token = "4b928c7563a0cba10ff4c3f5ca0c8e24"
        }
    }
}

Connect to server

Now we can connect to server to build project.

$ xmake service --connect
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
    [+]: xmake.lua
    [+]: .gitignore
    [+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!

Generate a new token

We can also generate a new token for new client.

$ xmake service --gen-token
New token a7b9fc2d3bfca1472aabc38bb5f5d612 is generated!

Then we can copy and save this token to new client configuraion file.

User password mode

Add user in server

$ xmake service --add-user=ruki
Please input user ruki password:
123456
Add user ruki ok!

$ cat /Users/ruki/.xmake/service.conf
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691"
        },
        server = {
            listen = "0.0.0.0:9691",
            workdir = "/Users/ruki/.xmake/service/remote_build"
        }
    },
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24",
            "7889e25402413e93fd37395a636bf942"
        },
        known_hosts = { }
    }
}

Remove user in server

$xmake service --rm-user=ruki
Please input user ruki password:
123456
Remove user ruki ok!

Configurate user in client

We can add user name to the connect address. eg. user@address:port

{
    remote_build = {
        client = {
            connect = "root@127.0.0.1:9691"
        }
  }
}

Connect to server

We need input user password before connecting server.

$ xmake service --connect
Please input user root password:
000000
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connected!
<remote_build_client>: sync files in 127.0.0.1:9691 ..
Scanning files ..
Comparing 3 files ..
    [+]: xmake.lua
    [+]: .gitignore
    [+]: src/main.cpp
3 files has been changed!
Archiving files ..
Uploading files with 1591 bytes ..
<remote_build_client>: sync files ok!

If password is incorrect, it will raise errors.

 xmake service --connect
Please input user root password:
123
<remote_build_client>: connect 127.0.0.1:9691 ..
<remote_build_client>: connect 127.0.0.1:9691 failed, user and password are incorrect!

Configure known hosts in server

We can also add known hosts in server to guarantee the connection of trusted hosts.

We can add them to the known_hosts = {} in service.conf

$ cat ~/.xmake/service.conf 
{
    logfile = "/Users/ruki/.xmake/service/logs.txt",
    server = {
        tokens = {
            "4b928c7563a0cba10ff4c3f5ca0c8e24"
        },
        known_hosts = { "127.0.0.1", "xx.xx.xx.xx"}
    }
}

If it is configured, then only hosts included in this list will be able to successfully connect to the server.

Answer 11 · 2022-05-15T08:24:38.000Z

To better manage the configuration, I split service.conf into two configuration files, server.conf and client.conf

$ cat ~/.xmake/service/server.conf 
{
    known_hosts = { },
    logfile = "/Users/ruki/.xmake/service/server/logs.txt",
    remote_build = {
        listen = "0.0.0.0:9691",
        workdir = "/Users/ruki/.xmake/service/server/remote_build"
    },
    tokens = {
        "e438d816c95958667747c318f1532c0f"
    }
}

$ cat ~/.xmake/service/client.conf 
{
    remote_build = {
        connect = "127.0.0.1:9691",
        token = "e438d816c95958667747c318f1532c0f"
    }
}