Low resolution leads to blurry images
Opened this issue · 13 comments
I am using an IpadPro with a Fedora 34 computer. RPiPlay mostly works fine. When I start it on my computer and then start the screen mirroring on the ipad, then I can see the screen content of the ipad on my computer in a new window. The only problem is that the window is pretty small and fine details are difficult to read. Of course I can increase the size of the window, but this will just scale up the content and not improve the image quality. So I went ahead and took a screenshot on my computer and measured the size of the initial window: 1392x972 pixels in landscape mode, 680x972 pixels in portrait mode. The ipad's own display resolution is 2388x1668 pixels.
How can I increase the resolution of the image sent by my ipad?
things work fine with a non-pro ipad. Maybe no one tested with an ipad pro before you? it might be a gstreamer issue.
things work fine with a non-pro ipad. Maybe no one tested with an ipad pro before you? it might be a gstreamer issue.
Could be. Would you mind testing which resolution is used by default for your non-pro ipad? Does it match the ipad's screen resolution? The answer might be helpful for finding the cause of this issue.
with a standard ipad (7th gen)
initial gstreamer windows (in pixels)
landscape 1300 x 970 (1000 with top gstreamer window title bar )
portrait 730 x 970 (1000)
so it seems that the vertical dimension of the initial gstreamer window is 1000 pixels including title bar, as
as in your measurement, and the horizontal adjusts to that. you will need to look in the code to see how the renderer sets the videoscale to see if this can be modified
mirroring my ipad, I get clean crisp images, pixel measurements grow as window on linux is expanded.
The action seems to happen in lib/raop_rtp_mirror.c line 350
run rpiplay -d (for debug output), start mirroring and stop mirroring just after it starts, to avoid too much output.
the input stream header says what width and height are required
ipad in portrait mode:
conn_request
Handling request SET_PARAMETER with URL rtsp://192.168.10.20/981463989324038314
raop_rtp_mirror accepting client
raop_rtp_mirror width_source = 730.000000 height_source = 972.000000 width = 762.000000 height = 1016.000000
raop_rtp_mirror sps size = 17
raop_rtp_mirror pps size = 4
ipad in landscape mode
conn_request
Handling request SET_PARAMETER with URL rtsp://192.168.10.20/14839122743874767717
raop_rtp_mirror video ntp = 1627366207176196, now = 1627366207104818, latency = -71378
raop_rtp_mirror video ntp = 1627366207192821, now = 1627366207122881, latency = -69940
raop_rtp_mirror width_source = 1296.000000 height_source = 972.000000 width = 1354.000000 height = 1016.000000
raop_rtp_mirror sps size = 17
raop_rtp_mirror pps size = 4
You should find out what width_source and height_source the ipad-pro reports. width and height also seem to come from the
input stream.
The next step would be to find out how this width and height input is transmitted to and used by gstreamer., and if any modification to accommodate the presumably-larger values requested by ipad pro can be made.
The variables width_source etc are read from some input data in unsigned char packet[128], but are apparently only used for the debug output. So I suppose that the contents of "packet" are being passed to the gstreamer renderer at some other point in the code, to give it the pixel height and width data . This seems to happen at line 399, and the pixel data is I guess in h264_data
wild guess: the only place you could modify gstreamer height etc might be at line 71 in renderers/video_renderer_gstreamer.c
where videoconvert is used
// Begin the video pipeline
GString *launch = g_string_new("appsrc name=video_source stream-type=0 format=GST_FORMAT_TIME is-live=true !"
"queue ! decodebin ! videoconvert ! ");
It seems that Airplay mirroring is limited to 1080p resolution, max height = 1080 pixels. This seems to be what rpiplay is essentially producing.
The commercial product Airserver mentions this limitation for it too. (But surely 1080p is good enough?)
More precisely legacy Airplay protocol (used by RPiPlay) is limited to max 1080p resolution. The current Airplay 2 protocol does 4K. As of 2019, commercial Airplay receivers for Windows (e.g. Airserver) were also only using legacy airplay. From a current search, this still appears to be the case. Hopefully Apple will not discontinue legacy Airplay support in future IOS versions too soon.
The audio (but not video) part of Airplay 2 has been reverse engineered
https://github.com/mikebrady/shairport-sync
Edit. latest 2021 AirServer for windows/mac says it does "retina quality" mirroring at 1080x1920 which is AirPlay 2.
There is a 1 month free trial. If this works, it means they have presumably managed to reverse-engineer Airplay 2 video, so some Linux enthusiasts could presumably manage too.
The commercial product Airserver mentions this limitation for it too. (But surely 1080p is good enough?)
Thanks, I think this is the biggest issue. 1080p would be enough, if the scaling was done better. The way it is done now makes text very blurry, whereas it is crystal clear on the ipad. Even larger font looks washed out.
But I guess there is not much to be done here, as this scaling happens on the ipad before transmission.
Nonetheless I will try to play around with RPIplay and try so see if I can get the window to automatically open with 1080p and not this tiny stuff I am observing. Thanks for the suggestions where to look and try.
OK, using PR #191 and setting the resolution to 1080x1920 by hand does improve the image quality a lot, even though it is displayed smaller than that resolution on my computer. Maybe some issues in the pull request.
I can even select higher resolutions and thus have a larger window during startup and everything looks beautiful. However, I was not able to figure out which resolution I am finally getting. I uploaded a couple of test images (one pixel red, the next blue and then extend to a checkerboard of the desired size) to my ipad, but those are not displayed correctly. When I open the large one in the regular photos app, then the navigation symbols and stuff around the image gets replaced by black, even though it is still visible on my ipad. No clue what is happening there. I will open a new bug report for that.
use rpiplay -d to see what the ipad pro is sending for height, width, as in my post above.
presumably it wlll be height_source = 972. post the result (all 4 numbers) here anyway.
rpiplay -d:
Mirroring initialized successfully
raop_rtp_mirror accepting client
raop_rtp_mirror width_source = 680.000000 height_source = 972.000000 width = 710.000000 height = 1016.000000
raop_rtp_mirror sps size = 17
raop_rtp_mirror pps size = 4
rpiplay -y 3000 -d:
Mirroring initialized successfully
raop_rtp_mirror accepting client
raop_rtp_mirror width_source = 1728.000000 height_source = 2474.000000 width = 1804.000000 height = 2584.000000
raop_rtp_mirror sps size = 18
raop_rtp_mirror pps size = 4
The latter needs PR #191 and is a lot more beautiful.
EDIT: There is no difference between -y 3000 and -y 4000, but -y 2000 results in
Mirroring initialized successfully
raop_rtp_mirror accepting client
raop_rtp_mirror width_source = 1258.000000 height_source = 1800.000000 width = 1314.000000 height = 1880.000000
raop_rtp_mirror sps size = 18
raop_rtp_mirror pps size = 4
For non-R Pi users (those using the gstreamer renderer only) there is a updated version of the RPiPlay derivative UxPlay at
https://github.com/FDH2/UxPlay.git that includes setting the display size among other improvements.