Get raw frames into render pass from external

[xdave]

I was wondering if it's possible to get raw frames from a video or a capture feed into the render pass?

I'm trying to create an environment that can render my Desktops into 3d space.

[bjornbytes]

You can make a Texture (maybe with the resolution of your desktop) and use Texture:setPixels to upload pixels to it from an Image object on the CPU. There are a couple ways of getting data to the Image object, but the most efficient is to get a pointer to it using Image:getPointer() to get a lightuserdata pointer to the image's pixel data and passing that to some C/FFI code to fill in with the desktop contents.

But after pixels are uploaded to the Texture you can draw it in 3D space like any other texture.

Let me know if anything needs clarification.

[xdave]

@bjornbytes so, here's what I've come up with:

local xcapture = require 'xcapture'

local skybox
local image
local texture
local instance

function lovr.load()
  skybox = lovr.graphics.newTexture('eso0932a.jpg')
  instance = xcapture:init()

  image = lovr.data.newImage(1920, 1080)
  texture = lovr.graphics.newTexture(image, {
    type = '2d',
    usage = {
      "sample",
      "transfer"
    },
    label = "desktop"
  })
end

function lovr.quit()
    xcapture:destroy(instance)
    return false
end

function lovr.draw(pass)
  pass:skybox(skybox)

  xcapture:write_image(instance, image:getPointer())
  texture:setPixels(image)
  pass:draw(texture, 0, 1.7, -3)
end

xcapture:write_image() memcpys a shm buffer from xcb onto the image pointer; however, it's not rendering anything at all. Am I missing something?

[bjornbytes]

Here's a tiny working example that demonstrates the technique using LuaJIT's FFI:

local image
local texture

function lovr.load()
  image = lovr.data.newImage(1, 1)
  texture = lovr.graphics.newTexture(image, {
    type = '2d',
    usage = {
      "sample",
      "transfer"
    },
    label = "desktop"
  })
end

function lovr.draw(pass)
  local a, s, t = math.abs, math.sin, lovr.timer.getTime()
  local r, g, b = a(s(t + 0)), a(s(t + 1)), a(s(t + 2))

  local pointer = image:getPointer()
  local pixel = require('ffi').cast('uint8_t*', pointer)
  pixel[0], pixel[1], pixel[2], pixel[3] = r * 255, g * 255, b * 255, 255

  texture:setPixels(image)
  pass:draw(texture, 0, 1.7, -3)
end

Double check that the xcapture plugin is writing pixels with the right 8-bit format (including alpha channel).

[xdave]

Hmm after some investigating, I'm pretty sure it's coming though as bgr0 by default. Not sure if that's the same thing. I'm allocating 32bits total

width x height x 4

But if I have to change the bit layout, I can try that.

Edit: lol the manpage says

XCB_IMAGE_FORMAT_Z_PIXMAP
                           TODO: NOT YET DOCUMENTED.

This just gets better and better. I'll mess with this some more when I'm at my desk again 🤣