Cause of Remote API not working

Question

Cause of Remote API not working

underdogest opened this issue 2 years ago · 25 comments

The VRage Remote Api uses the HttpListener class from dotnet, which in turn uses http.sys windows kernel driver. The current wine implementation of http.sys is spotty at best, so it wont work as it is.
One possibility would be to decompile VRage.Dedicated.dll and recompile it against a HttpListener implementation that uses native tcp sockets.

Edit:
After changing the HttpListener prefix from
"http://*:" + this.Port.ToString() + "/vrageremote/"
to
"http://+:" + this.Port.ToString() + "/vrageremote/"
and adding a
context.Response.ContentLength64 = bytes.Length;
at two places before
context.Response.OutputStream.Write(bytes, 0, bytes.Length);
the remote api actually works under docker

Answer 1 · 2023-02-22T05:16:15.000Z

Ok, good to know - but i don't think its the right solution for a docker image.

Answer 2 · 2023-05-17T10:37:26.000Z

Have been looking trough the wine commits and there may be some patches in 8.8 that could impact this.
I'm not that into wine and dotnet to be able to confirm. Will try to setup a test environment with the latest dev wine to test when I have the time

http.sys: Add support for adding multiple urls to request queues.
wine-mirror/wine@0f139a0

http.sys: Add support for relative URLs.
wine-mirror/wine@0ba9567

http.sys: Use SO_EXCLUSIVEADDRUSE on listening socket.
wine-mirror/wine@0ba9567

Answer 3 · 2023-05-17T12:26:24.000Z

Ok, i am currently using wine 8.8 devidian/spaceengineers:winestaging but did not try the remote api yet

Answer 4 · 2023-06-02T19:01:01.000Z

I've updated staging to wine 8.9 still no response when i try to connect to Remote API

Answer 5 · 2024-04-20T11:46:38.000Z

Hi @underdogest! Since your changes are currently the only solution, can you briefly describe how you made them?
I'd like to implement them for my docker SE..

was the change made to code "within" VRage.Dedicated.dll ?
can you recommend tools for disassembly and reassembly?
your help is much appreciated!

Answer 6 · 2024-05-23T11:31:37.000Z

I've tested the latest wine, still not working.

Answer 7 · 2024-05-23T15:13:15.000Z

Hi @underdogest! Since your changes are currently the only solution, can you briefly describe how you made them? I'd like to implement them for my docker SE..

was the change made to code "within" VRage.Dedicated.dll ?

can you recommend tools for disassembly and reassembly?
your help is much appreciated!

yes in the binary itself using dnsspy. Open the DLL, find the class and click edit class, then do the code changes and click compile. Afterwards "save module".
Here you change * to +

Here you add "context.Response.ContentLength64 = bytes.Length;"

Use the newest version of dnspy: https://github.com/dnSpyEx/dnSpy
SpaceEngineers calls the winapi wrong, but the winapi still works correctly on windows, but not in wine.

Afterwards it should look like this:

Since its decompiling and recompiling the names might be chosen randomly on your dnspy.

Answer 8 · 2024-06-06T20:44:27.000Z

Hi @underdogest! Since your changes are currently the only solution, can you briefly describe how you made them? I'd like to implement them for my docker SE..

was the change made to code "within" VRage.Dedicated.dll ?

can you recommend tools for disassembly and reassembly?
your help is much appreciated!

yes in the binary itself using dnsspy. Open the DLL, find the class and click edit class, then do the code changes and click compile. Afterwards "save module". Here you change * to + Here you add "context.Response.ContentLength64 = bytes.Length;"

Use the newest version of dnspy: https://github.com/dnSpyEx/dnSpy SpaceEngineers calls the winapi wrong, but the winapi still works correctly on windows, but not in wine.

Afterwards it should look like this:

Since its decompiling and recompiling the names might be chosen randomly on your dnspy.

Tried this, did not work.

Answer 9 · 2024-06-07T00:02:40.000Z

Tried this, did not work.

Just tried it again with newest version and it works. Using docker in debian12 in VirtualBox:

make sure the server is actually starting and you can join!. That took the longest for me, if there are spaces in ANY filepath it will crash and restart.
Check in
/appdata/space-engineers/instances/TestInstance/SpaceEngineers-Dedicated.cfg
for
<LoadWorld>Z:\appdata\space-engineers\instances\TestInstance\Saves\AlienPlanet</LoadWorld>
I removed the spaces in the save name, after that it worked.

Full walkthrough:
My docker-compose.yaml:

services:
  se-server:
    image: devidian/spaceengineers:winestaging
    container_name: se-ds-docker
    restart: unless-stopped
    volumes:
      # left side: your docker-host machine
      # right side: the paths in the image (!!do not change!!)
      - /appdata/space-engineers/plugins:/appdata/space-engineers/plugins
      - /appdata/space-engineers/instances:/appdata/space-engineers/instances
      - /appdata/space-engineers/SpaceEngineersDedicated:/appdata/space-engineers/SpaceEngineersDedicated
      - /appdata/space-engineers/steamcmd:/root/.steam
    ports:
      - target: 27016
        published: 27016
        protocol: udp
        mode: host
      # very important for the remote client
      - target: 8080
        published: 8080
        protocol: tcp
        mode: host
    environment:
      - WINEDEBUG=-all
      - INSTANCE_NAME=TestInstance
      - PUBLIC_IP=192.168.99.111
      # public ip required for healthcheck

Started up the server with
docker-compose up -d
checked the logs with
docker logs 7c46ced700ed
till I saw it crash-looping.
Then stopped the container
docker stop 7c46ced700ed

I created a world on Windows with
X:\SteamLibrary\steamapps\common\SpaceEngineersDedicatedServer\DedicatedServer64\SpaceEngineersDedicated.exe
Called the instance "docker", saved and started it until "Game ready", then stopped the server.
Copied everything in
C:\ProgramData\SpaceEngineersDedicated\docker\*
to
vbox:/appdata/space-engineers/instances/TestInstance/
Renamed the folder
vbox:/appdata/space-engineers/instances/TestInstance/Saves/Alien Planet lots of numbers
to
vbox:/appdata/space-engineers/instances/TestInstance/Saves/AlienPlanet

Opened up /appdata/space-engineers/instances/TestInstance/SpaceEngineers-Dedicated.cfg
Changed the entry
<LoadWorld>C:\ProgramData\SpaceEngineersDedicated\docker\Saves\Alien Planet lots of numbers\Sandbox.sbc</LoadWorld>
to
<LoadWorld>Z:\appdata\space-engineers\instances\TestInstance\Saves\AlienPlanet</LoadWorld>

Copied my patched VRage.Dedicated.dll to (remember to backup the original to VRage.Dedicated.dll.bak just in case)
vbox:/appdata/space-engineers/SpaceEngineersDedicated/DedicatedServer64/

Restarted the docker container
docker start 7c46ced700ed

Waited for the game to be ready and just connected with VRageRemoteClient.exe to 192.168.99.111:8080 using the RemoteSecurityKey in the SpaceEngineers-Dedicated.cfg

Common errors:

You forgot to expose port 8080 in the container
You have something already listening on port 8080 on the docker host
You didnt check if the server was actually ready for joining players

You can try my attached patched VRage.Dedicated.dll, but once the dedicated server gets updated you need to redo my steps in dnspy like described above. Steam will probably overwrite the dll. Also it might become incompatible with the other updated dlls.
VRage.Dedicated.dll.zip

Answer 10 · 2024-06-07T00:38:45.000Z

In the current wine without my patch I get the following error:
wine: Call from 00006FFFFF463E67 to unimplemented function httpapi.dll.**HttpSendResponseEntityBody**, aborting
Which happens because HttpSendResponseEntityBody is only a stub in wine: https://source.winehq.org/WineAPI/httpapi.html
This method is called when a http server sends the response chunked (contentlength is not known when sending).
When the contentlength is known, it sends it via httpapi.dll.HttpSendHttpResponse, which is implemented fine in wine.

So for wine to support the remote client natively someone has to implement httpapi.dll.HttpSendResponseEntityBody in wine here: https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/httpapi/httpapi_main.c (very hard)
and add a case for http://* next to http://+ here: https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/http.sys/http.c#L388 (literally just a line)

Answer 11 · 2024-06-07T07:00:04.000Z

So it could be fixed in wine in the future. Good to know.

Answer 12 · 2024-06-07T16:49:28.000Z

Writing a contribution for wine right now

Answer 13 · 2024-06-07T17:03:23.000Z

In the current wine without my patch I get the following error: wine: Call from 00006FFFFF463E67 to unimplemented function httpapi.dll.**HttpSendResponseEntityBody**, aborting Which happens because HttpSendResponseEntityBody is only a stub in wine: https://source.winehq.org/WineAPI/httpapi.html This method is called when a http server sends the response chunked (contentlength is not known when sending). When the contentlength is known, it sends it via httpapi.dll.HttpSendHttpResponse, which is implemented fine in wine.

So for wine to support the remote client natively someone has to implement httpapi.dll.HttpSendResponseEntityBody in wine here: https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/httpapi/httpapi_main.c (very hard) and add a case for http://* next to http://+ here: https://gitlab.winehq.org/wine/wine/-/blob/master/dlls/http.sys/http.c#L388 (literally just a line)

I use a native windows httpapi.dll and do not get this error. Instead I don't get ANY message, with or without the patch.

Answer 14 · 2024-06-07T17:55:26.000Z

I use a native windows httpapi.dll and do not get this error. Instead I don't get ANY message, with or without the patch.

you would also need the native http.sys driver, but that one cant be loaded into linux or wine.

Answer 15 · 2024-06-07T19:50:23.000Z

I use a native windows httpapi.dll and do not get this error. Instead I don't get ANY message, with or without the patch.

you would also need the native http.sys driver, but that one cant be loaded into linux or wine.

That makes total sense, my bad. I reverted to native. I still can't connect, but at least when I close the remote client I get "02d4:err:http:receive_data Got error 10054; shutting down connection." in my server console.

Answer 16 · 2024-06-07T22:26:24.000Z

Looking good boys, my wine's httpapi.dll and http.sys work with an unmodified spaceengineers dedicated server now!

Now I just need to get my changes into wine main lol

Answer 17 · 2024-06-09T09:58:23.000Z

@underdogest Very nice! Could you link the wine change? Apart from being curious how you implemented it, i might compile wine myself until the new wine version is released.

Answer 18 · 2024-09-28T02:24:12.000Z

@underdogest Very nice! Could you link the wine change? Apart from being curious how you implemented it, i might compile wine myself until the new wine version is released.

https://gitlab.winehq.org/wine/wine/-/merge_requests/6216

Answer 19 · 2024-09-29T02:46:43.000Z

@underdogest Very nice! Could you link the wine change? Apart from being curious how you implemented it, i might compile wine myself until the new wine version is released.

https://gitlab.winehq.org/wine/wine/-/merge_requests/6216

Oh dear, they just copy pasted code from my branch and merged it into master. There was still an issue I was too lazy to fix regarding keep-alive http connections not working after the first connection ended.

Answer 20 · 2024-10-23T14:01:08.000Z

if you have host network mode, then you don't need to specify ports, just network_mode: "host"

Answer 21 · 2024-10-24T02:06:58.000Z

Can anyone confirm that this is working and I can download, install and run Vrage remotely?

Answer 22 · 2024-10-25T12:10:32.000Z

i've just pushed a new staging branch update to wine staging 9.20 (was still on 9.9). Image should be ready in a hour maybe, dont know how fast dockerhub builder is today. Took 20 Minutes on my pc. Maybe someone can test then and report

Answer 23 · 2024-10-25T12:23:48.000Z

i've just pushed a new staging branch update to wine staging 9.20 (was still on 9.9). Image should be ready in a hour maybe, dont know how fast dockerhub builder is today. Took 20 Minutes on my pc. Maybe someone can test then and report

Thanks! I'll keep my eye on this thread.

Answer 24 · 2024-12-18T21:00:32.000Z

~~Just asking, have these changes been merged into a released version of Wine? If so, maybe we should consider bumping the version of Wine used in this docker to add these fixes.~~

Just now seeing Devidian's comment, I will test this now as I'm running a new SE server. I'll see if I can get the remote client working :)

Answer 25 · 2025-01-07T00:50:33.000Z

So port 8080 is taken for me so I changed the port in the config and in the docker container, I added port mappings for port 8081, both for TCP and UDP and I wasn't able to connect at all. At first I thought it might've been because I was running the remote tool under proton but switching to Windows lead to the same result unfortunately.