ddvk/remarkable2-framebuffer

Refresh in Oxide after app is close doesn't work correctly

Witos opened this issue · 38 comments

Witos commented

Environment setup:
remarkable2-framebuffer: 706f4cb..
oxide: af869c6... (with adding LD_PRELOAD=/home/root/fb/librm2fb_client.so before application launch)

LD_PRELOAD=/home/root/fb/librm2fb_server.so /usr/bin/remarkable-shutdown &
systemctl start tarnish

Steps:

  1. Start "Process Manager"
  2. Press "Back"

Actual:
"Process Manager" elements are still visible.

Expected:
Oxide should look like after the launch.

I'm attaching video.

The code responsible for the redraw:
https://github.com/Eeems/oxide/blob/af869c69304a9d8debf21dc705783cc9f1c7fdef/applications/system-service/application.h#L162

IMG_0993.mov.zip

raisjn commented

it could be waveform mode auto, i checked and both remux and oxide are using it.

Eeems commented

it could be waveform mode auto, i checked and both remux and oxide are using it.

I did steal code borrow code take inspiration from remux.

raisjn commented

if you change it to WAVEFORM_MODE_GC16, then it should work

Eeems commented

@Witos would you be able to test that?

Witos commented

Tested this patch:
Witos/oxide@8ca3ea4
It didn't change anything.

Witos commented

Log from server:
reMarkable: ~/ cat server.log
OPENED SHARED MEM: /dev/shm/swtfb.01 at 751a9000, errno: 0
BIN FILE: /usr/bin/remarkable-shutdown
ADDR: 21f54
REPLACING THE IMAGE with shared memory
SWTCON initialized \o/
STARTING RM2FB
Reading waveforms from /usr/share/remarkable/320_R299_AFC421_ED103TC2U2_VB3300-KCD_TC.wbf
Running INIT (135 phases)
1404 1872 16
reMarkable: ~

timower commented

Yeah, I've tried to debug this but the fact that tarnish stops the wifi just makes this unnecessarily difficult.
You can get more info from the server by adding:

#define DEBUG
#define DEBUG_DIRTY

to https://github.com/ddvk/remarkable2-framebuffer/blob/master/src/server/main.cpp#L37

It would be nice to see if the ioctl sent by tarnish is actually received.
I suspect the problem is with oxide or tarnish and the way LD_PRELOAD gets propagated.

Witos commented

reMarkable: ~/ cat server.log
OPENED SHARED MEM: /dev/shm/swtfb.01 at 75272000, errno: 0
BIN FILE: /usr/bin/remarkable-shutdown
ADDR: 21f54
REPLACING THE IMAGE with shared memory
SWTCON initialized \o/
STARTING RM2FB
Reading waveforms from /usr/share/remarkable/320_R299_AFC421_ED103TC2U2_VB3300-KCD_TC.wbf
Running INIT (135 phases)
1404 1872 16
WAITING FOR SEND UPDATE ON MSG Q
Dirty Region: 0 0 1404 1872
doUpdate
mxc: waveform_mode 3
mxc: update mode 0
mxc: update marker 1
final: waveform 3 flags 0

Dirty Region: 1218 6 103 43
doUpdate
mxc: waveform_mode 3
mxc: update mode 0
mxc: update marker 2
final: waveform 3 flags 0

Dirty Region: 1218 6 43 43
doUpdate
mxc: waveform_mode 3
mxc: update mode 0
mxc: update marker 3
final: waveform 3 flags 0

Dirty Region: 23 75 187 222
doUpdate
mxc: waveform_mode 3
mxc: update mode 0
mxc: update marker 4
final: waveform 3 flags 0

Dirty Region: 0 0 1404 1872
doUpdate
mxc: waveform_mode 3
mxc: update mode 0
mxc: update marker 1
final: waveform 3 flags 0

Dirty Region: 0 0 1404 1872
doUpdate
mxc: waveform_mode 2
mxc: update mode 1
mxc: update marker 0
final: waveform 2 flags 1

Dirty Region: 23 6 1238 291
doUpdate
mxc: waveform_mode 3
mxc: update mode 0
mxc: update marker 5
final: waveform 3 flags 0

Witos commented

Tried to get a screenshot:
/opt/bin/rot screen call screenshot
and it's all black.

timower commented 
Dirty Region: 0 0 1404 1872
doUpdate

So the update is received by the server, if that's what gets shown after the restore happens.

I think the screenshot does not work as we don't set the bits per channel in the get screen info ioctl correctly. So vinfo.red.length and vinfo.red.mask will be incorrect. Can you dump the raw framebuffer you save and check if it's correct after restoring, before calling the update ioctl?

Eeems commented

I don't have an rM2 so I can't test anything. @Witos ?
@timower Feel free to also spin up a build and try things out yourself.

Witos commented

@timower those are the values:
vinfo ----------->
bits_per_pixel=16, xres=1404, yres=1872
red: offset=56, length=1982819968, msb_right=2128790624
green: offset=16, length=8, msb_right=31750184
blue: offset=2128790336, length=2128790360, msb_right=63
finfo line_length=2808
vinfo <-----------

Witos commented

The screenshot code is not used when the screen is saved for app switching....

Witos commented

Refreshing works like that:

qDebug() << "Saving screen...";
int frameBufferHandle = open("/dev/fb0", O_RDWR);
char* frameBuffer = (char*)mmap(0, DISPLAYSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, frameBufferHandle, 0);
qDebug() << "Compressing data...";
auto compressedData = qCompress(QByteArray(frameBuffer, DISPLAYSIZE));
close(frameBufferHandle);
screenCapture = new QByteArray(compressedData);

and then:

int frameBufferHandle = open("/dev/fb0", O_RDWR);
auto frameBuffer = (char*)mmap(0, DISPLAYSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, frameBufferHandle, 0);
memcpy(frameBuffer, uncompressedData, DISPLAYSIZE);
mxcfb_update_data update_data;
mxcfb_rect update_rect;
update_rect.top = 0;
update_rect.left = 0;
update_rect.width = DISPLAYWIDTH;
update_rect.height = DISPLAYHEIGHT;
update_data.update_marker = 0;
update_data.update_region = update_rect;
update_data.waveform_mode = WAVEFORM_MODE_GC16;
update_data.update_mode = UPDATE_MODE_FULL;
update_data.dither_mode = EPDC_FLAG_USE_DITHERING_MAX;
update_data.temp = TEMP_USE_REMARKABLE_DRAW;
update_data.flags = 0;
ioctl(frameBufferHandle, MXCFB_SEND_UPDATE, &update_data);
Eeems commented

You might want to edit and wrap the code in ```

Witos commented

Dumped the buffer (char* frameBuffer). I attaching. It's not empty, at least, but says nothing to me.
debugfb.txt

Witos commented

Experiment - I commented out:

//auto frameBuffer = (char*)mmap(0, DISPLAYSIZE, PROT_READ | PROT_WRITE, MAP_SHARED, frameBufferHandle, 0);
//memcpy(frameBuffer, uncompressedData, DISPLAYSIZE);

and tested. It doesn't change anything at all. The screen looks exactly the same as when those above were on.

Eeems commented

@Witos I think we need to solve why copying the framebuffer isn't producing a proper image yet first before we attempt to solve why restoring the framebuffer is broken.

Witos commented

@Eeems , how do you know it's not producing a proper image? Did you check the attached debugfb.txt?

raisjn commented

i wonder if it is related to the call to close(fbfd) in fb2png and other places. rm2fb does not intercept the close and so closing the FB file descriptor and then re-opening may not work properly. one way to check would be to remove the close() calls and re-compile oxide. if it is the problem, then we should override close() and do nothing if close is called on the swtfb FD.

raisjn commented

ok, I checked and I don't think the close() is the reason oxide isn't redrawing itself after closing the currently running process. I think oxide just needs to refresh after the current app closes.

Witos commented

@Eeems, I was curious about the numbers returned by ioctl FBIOGET_VCREENINFO. They seemed invalid. I added memset:
struct fb_var_screeninfo vinfo;
memset(&vinfo, 0, sizeof(struct fb_var_screeninfo));
ioctl(fbfd, FBIOGET_VSCREENINFO, &vinfo)

This results in:
vinfo ----------->
bits_per_pixel=16, xres=1404, yres=1872
red: offset=0, lenght=0
green: offset=0, lenght=0
blue: offset=0, lenght=0
vinfo <-----------

ddvk commented
Witos commented 
       else if (request == FBIOGET_VSCREENINFO) {
    1
    2         fb_var_screeninfo *screeninfo = (fb_var_screeninfo *)ptr;
    3         screeninfo->xres = 1404;
    4         screeninfo->yres = 1872;
    5         screeninfo->grayscale = 0;
    6         screeninfo->bits_per_pixel = 16;
    7         return 0;
    8       }
Witos commented

This is still not the reason why the original refresh is not working, because it doesn't use vscreeninfo at all. It just copies the raw memory around. How to perform raw refresh, @raisjn, without this fb copying?

raisjn commented

there should be something in qt where you can redraw the app or main window

ddvk commented

can you try with the latest master? i see that the screenshot of oxide uses the rgb offsets

Witos commented

@ddvk , will try that this evening (UTC+1)
@raisjn , will try to find something, but it seems it's not that obvious to force repaint in Qt

ddvk commented
Eeems commented

ok, I checked and I don't think the close() is the reason oxide isn't redrawing itself after closing the currently running process. I think oxide just needs to refresh after the current app closes.

That defeats the entire point of recalling the stored screen. Not to mention there is no easy way to ensure that other apps redraw themselves on resume.
Redrawing by hiding and showing the main window is what I use to do, but that was replaced with saving and recalling the screen so that when resuming any application it would work properly, instead of having to have every application specifically support being resumed.

Witos commented

Screenshot feature works fine after @ddvk patch. Attaching the screenshot captured as a proof.
fb

  • 👍1
Eeems commented

Does screen recalling when switching apps work now? Or is that still broken?

Witos commented

No, it's still broken. Attaching screenshot when the screen is malformed.
fb

Eeems commented

And this is with WAVEFORM_MODE_GC16?

raisjn commented

oxide from toltec testing now works for me using master (ddvk's fix for screen info + close() impl)

  • 👍1
  • 🎉1
Witos commented

Good job @raisjn , good job @ddvk - I confirm it works, after applying your 2 patches.