Vulkan fail to create swapchain on Wayland (Call to a X11 function ?)

Hi, Bgfx with vulkan on wayland with a window created with glfw doesn't seems to work. it work fine with X11 (XWayland) tho.

Here is a minimum example of code that make the bug happens.

#include <wayland-egl.h>
#define GLFW_EXPOSE_NATIVE_WAYLAND
#define GLFW_EXPOSE_NATIVE_EGL
#include <GLFW/glfw3.h>
#include <GLFW/glfw3native.h>
#include <bgfx/bgfx.h>

#define WINDOW_WIDTH 640
#define WINDOW_HEIGHT 480

int main() {
    if (!glfwInit()) {
        return -1;
    }

    glfwWindowHint(GLFW_CLIENT_API, GLFW_NO_API);
    GLFWwindow* window = glfwCreateWindow(WINDOW_WIDTH, WINDOW_HEIGHT, "Test", nullptr, nullptr);

    if (!window) {
        glfwTerminate();
        return -1;
    }

    wl_surface* surface = glfwGetWaylandWindow(window);
    if (!surface) {
        glfwDestroyWindow(window);
        glfwTerminate();
        return -1;
    }

    bgfx::Init init;
    init.platformData.nwh = wl_egl_window_create(surface, WINDOW_WIDTH, WINDOW_HEIGHT);
    init.platformData.ndt = glfwGetWaylandDisplay();
    init.platformData.type = bgfx::NativeWindowHandleType::Wayland;
    init.resolution.width = WINDOW_WIDTH;
    init.resolution.height = WINDOW_HEIGHT;
    init.resolution.reset = 128;
    init.type = bgfx::RendererType::Vulkan;
    init.vendorId = 0;
    bgfx::init(init); //Crash here on XGetXCBConnection
    
    bgfx::shutdown();
    glfwDestroyWindow(window);
    glfwTerminate();
}

It crashes while trying to create the swapchain on bgfx::init, here is the call stack :

The behaviour i was expecting was bgfx initialize successfully.
If it can help : I'm on linux (6.4.12-arch1-1) with a wayland window manager. (with XWayland for compatibility with X11 app)
on CLion, using the bundled cmake that come with CLion.
compiler are gcc/g++.
I use up to date Nvidia proprietary driver.

Hope my english is acceptable.

Edit : I forgot to mention, it work fine with opengl, same setup, same code, but instead of Vulkan type is OpenGL, and it work perfectly with wayland.

cc @belegdol

Did you rebuild glfw with Wayland support enabled? AFAIK for glfw Wayland or X11 is a compile-time option and only one or the other can be built in at any given time.

Also, on XWayland you should be creating an X11 window I believe. Wayland window is for native Wayland window only.

Thanks for the fast answere :D!
Absolutely i have GLFW_BUILD_WAYLAND set on, otherwise it would not even compile.
I'm on Hyprland wich is a wayland window manager. XWayland is just here for compatibility with app that does not use wayland.

Thanks for confirming. I will try to have a look tomorrow. I recall testing this on Nvidia myself and I recall that the examples were working as intended.

Have you defined WL_EGL_PLATFORM when compiling your code? For some reason the X11 variant of the function is getting called.

I didn't, so i tried, but it seems WL_EGL_PLATFORM isn't used ? i tried defining it with cmake but it tells me that it is not used. It doesn't change if i define it directly in my code either. same problem. if you can guide me if i did something wrong i would really apreciate it.

Well if you are building bgfx from source you need to modify the makefile:

diff --git a/makefile b/makefile
index 06e6573e1..84af399e4 100644
--- a/makefile
+++ b/makefile
@@ -113,7 +113,7 @@ wasm-release: .build/projects/gmake-wasm ## Build - Emscripten Release
 wasm: wasm-debug wasm-release ## Build - Emscripten Debug and Release
 
 .build/projects/gmake-linux:
-       $(GENIE) --with-tools --with-combined-examples --with-shared-lib --gcc=linux-gcc gmake
+       $(GENIE) --with-tools --with-combined-examples --with-shared-lib --with-glfw --with-wayland --gcc=linux-gcc gmake
 linux-debug64: .build/projects/gmake-linux ## Build - Linux x64 Debug
        $(MAKE) -R -C .build/projects/gmake-linux config=debug64
 linux-release64: .build/projects/gmake-linux ## Build - Linux x64 Release

This ensures that the wayland-specific Vulkan code gets built:

bgfx/scripts/genie.lua

Lines 207 to 209 in 17f7730

    
           if _OPTIONS["with-wayland"] then 
        
           	defines { "WL_EGL_PLATFORM=1" } 
        
           end

With this and glfw rebuilt with -DGLFW_USE_WAYLAND=ON added I have vulkan on wayland working on Fedora 38 with 535.xx nvidia drivers:

It worked,
I shouldn't have posted this issue here, i use bgfx.cmake wich is just a cmake wrapper around bgfx, and i didn't thought it could come frome that, but it did. sorry for that :/.
Thanks for the help :)!

	if _OPTIONS["with-wayland"] then
	defines { "WL_EGL_PLATFORM=1" }
	end