Images preview support for terminals witch sixel
Closed this issue · 58 comments
Hello. Is it possible to implement images preview support for terminal which support sixel?
Since wayland is lacking something like ueberzug, I'm currently using sixels on foot in wayland to display images in a terminal. Therefore I'm also interested in this.
I've never done any golang programming, but looked into the preview window code and noticed that currently Device Control ANSI codes (ESC P .... ST), and hereby also sixels, aren't parsed at all.
The relevant parts of code seem to be in src/ansi.go:nextAnsiEscapeSequence where escape codes are filtered.
The ESC starting a device control sequence is just removed and any sixel data is written as plain text into the terminal.
Since sixel data is often an enormous control sequence, painting any pixel multiple times, it may not be desirable in something focused on speed like fzf, I'd like to hear whether this feature would even be merged if somebody implements it.
Despite that I thought about getting it to work.
I'm trying to keep any sixel escape sequence from the command output until it's printed in the preview.
The problem:
The length and format of a string with sixel data is unpredictable (depends on console, font and font size) and introduces one or more newlines since any text printed after a sixel will be put one line below the last pixel.
This messes with the normal handling of strings in go, especially with regard to newlines.
How to go about it:
Any sixel data must therefore be parsed by fzf using (potentially non portable) xterm specific control sequences (ESC [ 14 t) to get the terminal width/height in pixels on stdin (which is another problem when fzf is currently reading stdin), divide that by number of columns/rows to get the width/height of a character and then parse the whole sixel sequence and determine the size of the sixel data in terms of rows and columns.
One can then provide an effective width and height for the data that would need to be passed along through the code until it gets printed in the src/terminal.go:renderPreviewText function.
Is it worth it?
If I'm not missing any easy workaround, this is a lot of work.
On the upside, one should be able to extend this to also allow for kitty's graphics control.
If it has the potential to be merged, I may try my luck, but not in the foreseeable future.
Please correct me if I have misunderstood parts of fzf, or of the sixel graphics escape codes!
introduces one or more newlines since any text printed after a sixel will be put one line below the last pixel.
If you use private mode 8452, the cursor will be left to the right, on the last line of the sixel, instead of on a newline.
Since sixel data is often an enormous control sequence, painting any pixel multiple times,
In my benchmarks of foot, it was pretty clear that the bottleneck is the producer. I.e. the one encoding the sixel. Sending the escape and parsing it in the terminal is nowhere near as demanding. I have seen mpv, and notcurses, peg one or more cores while foot is using ~20-25% of a single core.
Thus, one might want to consider pre-generating the sixel escapes, and/or cache already rendered sixels.
Any sixel data must therefore be parsed by fzf using (potentially non portable) xterm specific control sequences (ESC [ 14 t) to get the terminal width/height in pixels on stdin.
An alternative, that doesn't read stdin, is to use the TIOCGWINSZ ioctl. Not all terminals fill in the pixel fields, but all terminals that support sixel do (AFAIK).
@junegunn any thoughts about this issue?
I'm also interested in ways to do that.
While sixel support would be amazing, just a sidenote for people who don't not know it: ANSI based terminal rendering of images is pretty decent meanwhile - and works in fzf previews w/o problems: https://hpjansson.org/chafa/gallery/
As now ueberzug is not maintained I think it's a great time to rethink this issue. The sixel implementation would be really nice.
Is there any thoughts given on this?
@PJungkamp summarized it well above. I'm not familiar with the spec so I don't even know if it's possible to implement it inside fzf, and even if it is, it's going to be a hell lot of work. So I'm not interested in pursuing the goal, but feel free to send a pull request if you can pull it off.
I wish you would revisit this.
In terms of the spec, it's not terribly complicated. It's an escape sequence, followed by a prefix, followed by pixel data encoded in a particular way. The main thing to properly forward the escape sequences to the terminal, and to not mangle the pixel data. The terminal does the work of rendering. They're drawn in place at the cursor position. A lot of terminal emulators support full-color sixel graphics now. The more people use it, the more other emulators are likely to adopt it as well. The thing is, it's based on an actual hardware graphics terminal from the 1980s, so it's well-established.
I am not familiar with how FZF's preview window works, but some support for this would really expand the reach of what FZF can do.
After adding Kitty graphics support with a minimal code change using pass-through mechanism, @aeghn made me realize a similar approach could be taken to support Sixel as well (#3479 (reply in thread)).
I'm experimenting with the idea in b1a0ab8. Haven't decided if I'm going to keep it. It works fine with a single Sixel image but it's unlikely to work well with anything beyond that.
After adding Kitty graphics support with a minimal code change using pass-through mechanism, @aeghn made me realize a similar approach could be taken to support Sixel as well (#3479 (reply in thread)).
I'm experimenting with the idea in b1a0ab8. Haven't decided if I'm going to keep it. It works fine with a single Sixel image but it's unlikely to work well with anything beyond that.
FINALLY!!
Seeing this is mail gave me so much excitement and relief ......... that now I can ditch my hack for ueberzug to preview images .........
Looking forward with this.
It works fine with a single Sixel image but it's unlikely to work well with anything beyond that.
So do you have plan to add features like displaying text alongside or scrolling along with the image?
(This video is copied from #3479 from my fork)
preview.webm
I think it would be very cool if we could scroll the images in the fzf preview window (like pdf preview?).
If you decide to keep this option, I will be very happy to continue exploring things about sixel to work with fzf.
Only if the amount of the required code is small. After all, this is a "preview" window rather than a full-fledged viewer. I would advise users to launch a proper viewer program, examine the contents, and come back to fzf using execute bindings. But I'll take a look at your branch when I have some time.
FWIW, it is currently possible to do those things with Kitty graphics protocol thanks to its "placeholder" mechanism.
There are several competing protocols for image support on the terminal. Honestly, I have little experience with them, but for now, I prefer Kitty protocol because it's noticeably faster than Sixel, and it works on tmux, unlike Sixel.
Thank you for such a wonderful app and your great effort on the features which would help people living in terminal.
and it works on tmux, unlike Sixel
tmux officially supports sixel when compiled with ./configure --enable-sixel.
and it works on tmux, unlike Sixel
tmux officially supports sixel when compiled with
./configure --enable-sixel.
I think @junegunn is referring to the default setting, as someone pointed out in discussion #3479 (comment).
I am using sixel in fzf from a few days and it has been working awesomely fine in tmux.
So here's some progress:
sixel-scroll.mov
I have tried to implement the image placement with a minimal amount of Sixel-related code. The code is not polished and it has some issues and limitations some of which are not easy to fix. But it works on a basic level.
Hey @junegunn, thank you very much for this! Really great. I've faced a few issues however (running on Arch, I3, xterm 388, fzf-git):
- Regarding convert(1),
-scaleis a bit nicer with CPU resources than-resize(same values). Being this just a preview, it might be a nice alternative for low-end machines. - When the image is larger than the preview window's height, we have wrong cursor placement and the items list is messed up. I've arbitrarily reduced the height to something that works for me:
$((FZF_PREVIEW_HEIGHT - 50)). This works in most cases, but probably depends on the terminal size (mine: 150x40) - Everything works fine until you set
--preview-windowto a fixed width (say 120): some images, depending on their size I guess, are not displayed.
EDIT: reducing the width (say $((FZF_PREVIEW_WIDTH - 270))) fixes point 3, but still this is an arbitrary value (works for my setup at least).
- When a big image is being loaded (and while we see the "loading..." label) the previous image is reprinted and then replaced by the current one.
Just to be sure, are you testing the latest revision? The previous commit renamed $FZF_PREVIEW_WIDTH to $FZF_PREVIEW_PIXEL_WIDTH.
Using the latest rev now. Removed clear from --preview-window (not needed anymore, that's cool), and replaced var names with the new names. Still experimenting the same issues (and using the same workaround values to "fix it").
I also noticed that sometimes (specially when the previous file was a text file, tough also happens with images) the preview of the previous file is not properly cleared.
Btw, the whole thing feels snappier and less resource intensive now. Switched to -resize again.
The code is not polished and it has some issues and limitations some of which are not easy to fix.
Are you saying that sometimes the input box gets misaligned or the border gets broken, or sometimes there is an extra line printed by sixel (which may have different behaviors in different terminal emulators)? I encountered this issue before and couldn't find a good solution, so I directly removed the unnecessary borders and kept only the border separating the input box.
Additionally, from my personal standpoint, I would recommend using chafa.
@aeghn No, no issues with borders. The scrolling beyond an image can't be perfected in a simple way because fzf doesn't know in advance there is an image in the preview contents and how many lines it takes.
chafa looks great. Since it can resize the image by the number of terminal columns and lines instead of pixels, I can remove FZF_PREVIEW_PIXEL_*. That's great!
chafa can even print an image in kitty format, but unfortunately it doesn't seem to work inside tmux. So we're going to keep kitty icat for now.
I also noticed that sometimes (specially when the previous file was a text file, tough also happens with images) the preview of the previous file is not properly cleared.
I can reproduce. Let me see what I can do.
- Removed
$FZF_PREVIEW_PIXEL_* - Updated bin/fzf-preview.sh script to use chafa
Lines 33 to 34 in ec208af
- Clearing of the previous preview text
Quick testing (latest version, using chafa):
- Previous previewed text properly cleared.
- Previous previewed image not properly cleared (mostly if it was wider than the current one; even more than two images might overlap).
- Again: everything works fine until you set
--preview-windowto a fixed width (say 120): some images (couldn't find any common pattern among these files) are not displayed, and others (no pattern either) cause the items list to get messed up (wrong cursor position, list scrolled upward, and overwritten items). Subtracting 1 from FZF_PREVIEW_LINES fixes the latter (it seems that for some reason the image is printed past the bottom limit of the preview window, causing the described interface quirks). - Chafa feels slower than convert, but I'm not sure about this one.
I'll keep testing though.
EDIT: Some images are not displayed even if --preview-window isn't set at all.
some images (couldn't find any common pattern among these files) are not displayed
I think that has to do with this part of the code, which I'm not sure of its correctness. It's the code that determines how many lines are used to display the image. And the problem will likely go away if we change ceil to floor. But I'm not sure yet if it's the right thing to do.
Lines 2031 to 2034 in ec208af
diff --git a/src/terminal.go b/src/terminal.go
index 918c6e0..7cfc8d2 100644
--- a/src/terminal.go
+++ b/src/terminal.go
@@ -2029,8 +2029,8 @@ Loop:
if isSixel {
t.previewed.wipe = true
if t.termSize.PxHeight > 0 {
- rows := util.Max(0, strings.Count(passThrough, "-")-1)
- requiredLines = int(math.Ceil(float64(rows*6*t.termSize.Lines) / float64(t.termSize.PxHeight)))
+ rows := strings.Count(passThrough, "-")
+ requiredLines = int(math.Floor(float64(rows*6*t.termSize.Lines) / float64(t.termSize.PxHeight)))
}
}
I've found that --height is also involved in this issue, but not sure exactly how.
It's the code that determines how many lines are used to display the image. And the problem will likely go away if we change ceil to floor
That's most likely the issue considering what I've seen so far. I'm willing to test any changes you consider appropriate. Just let me know.
Sadly, it makes no difference at all.
EDIT: I can reproduce the issue by setting --height (and leaving --preview-window completely unset) to different values: at some point near the total amount of terminal lines, the image won't be displayed anymore. I guess this confirms your suspicion: requiredLines and/or rows are not getting the right values.
the image won't be displayed anymore
Is the wireframe (border) displayed in that case? Or it's just an empty screen?
Yes, border completely drawn, but no image (just the black background).
EDIT: another thing I've noticed is that, sometimes, after running fzf (in this case with fzf-preview.sh) an image gets like buffered in the terminal, and then, immediately before executing another terminal application, say vi or nano, even fzf itself (but after entering the corresponding command), this image is flushed (printed into the screen) before switching to the application.
Also, at least since the latest commit, the last lines of the image (maybe 1 or 2) are not printed.
I'm starting to think this issue is not grounded on fzf itself. After playing around with convert(1), chafa(1), and img2sixel(1), I've found that all of them fail to produce a sixel image of some images at particular resolutions (no matter if they're expressed as pixels or as characters). Worst, this resolution is not fixed, but depends on the original image (resolution?, compression? I'm not really sure).
The good thing is that this seems to happen only when the fzf window takes the whole (or almost) screen height. The bad thing is that this is fzf default behavior. In this case, reducing the resulting image to a half of the original works most of the time, but the image (all images actually, since we cannot discriminate beforehand between those who work and those who doesn't) is too small (considering the available space) and is only needed for a little fraction of images (maybe 1-3%), so that the price is maybe too high.
Maybe this is the reason (or at least one of the reasons) why Ranger (the file manager) started to experiment with sixel previews an then dropped the idea altogether.
Btw, consider reverting the changes made in 96e31e4: for what I can see, it only reduces the image height more than it should.
I've found that all of them fail to produce a sixel image of some images at particular resolutions
@leo-arch May I ask if you could upload those images?
Here's one of the images I'm using for testing:
Chafa, for example, can display this at 150x19, but fails with 150x40 (my actual term size), 150x30, 130x40, 130x30, 110x20, and more. Same for convert and img2sixel (though they may fail at different resolutions).
Tried with mltem, and it works great! The problem seems to be related to xterm (I'm using version 388).
EDIT: I still need to subtract 1 from FZF_PREVIEW_LINES to prevent the interface from becoming crazy. Here's my chafa command:
chafa -f sixel -s ${FZF_PREVIEW_COLUMNS}x$((FZF_PREVIEW_LINES - 1)) {}This is required when you add a border to the preview window, say --preview-window=border-left (it seems to happen only with border-left, border-right, and border-vertical). I guess it would be easy to make fzf inform 1 line less in FZF_PREVIEW_LINES when one of these preview window borders is set.
It looks like we were not on the same page. Reverted the height calculation. Use math.Ceil instead of math.Floor; i.e. if the calculated number of required lines is something like 25.88, fzf will regard it as 26 lines.
I still need to subtract 1 from FZF_PREVIEW_LINES to prevent the interface from becoming crazy. Here's my chafa command:
I can't reproduce the problem.
Hey, not a big deal anyway @junegunn, but it's definitely there (at least for me).
Here's an image that triggers this behavior (my term is xterm (same thing on mlterm), size 150x40):
And here's the command used:
ls | ~/build/fzf/target/fzf-linux_amd64 --preview='~/build/fzf/bin/fzf-preview.sh {}' --preview-window=border-left@leo-arch I can't reproduce the problem on my iTerm2 with any type of border.
# Press space to change the border style
FZF_DEFAULT_OPTS= fzf --preview 'fzf-preview.sh {}' --bind 'space:change-preview-window(border-left|border-top|border-bottom|border-horizontal|border-vertical|border-double|border-block|noborder)+clear-screen+refresh-preview'Please post a screenshot (or preferably a screencast) of the situation.
@leo-arch Oh wait, I'm noticing a rendering problem when I move between the selections.
I see the pattern. The issue arises when there is no border at the bottom.
border-leftborder-rightborder-topborder-verticalborder-none
@leo-arch Oh wait, I'm noticing a rendering problem when I move between the selections.
This is the main problem I have run into in my fork previously, I just remove the bottom booder and decrease the image size. Hope you could find the cause and solve it.
chafa "$1" -f sixel -s $((${FZF_PREVIEW_COLUMNS}-2))x$(($FZF_PREVIEW_LINES/9*10)) --animate false
Hmm, it looks like there's no simple solution to the problem. The whole screen shifts up by a row as soon as fzf passes the Sixel image data to the underlying terminal. The situation is also reproducible without using fzf.
# The image is shifted up. The cursor is on the last line which is empty.
chafa -f sixel -s ${COLUMNS}x${LINES} long-image.jpg; sleep 5I guess it would be easy to make fzf inform 1 line less in FZF_PREVIEW_LINES when one of these preview window borders is set.
This might be the only solution at hand, but it's not ideal because fzf can use the last line for other text data just fine. So we're reporting an incorrect value just for Sixel.
# Make sure that the preview window covers the last line
fzf --no-margin --no-padding --no-border --preview-window border-none --preview 'echo -n "$FZF_PREVIEW_LINES / ";seq $FZF_PREVIEW_LINES'This might be the only solution at hand, but it's not ideal because fzf can use the last line for other text data just fine. So we're reporting an incorrect value just for Sixel.
Agreed. So, what about exporting an environment variable and then testing against it to construct the chafa command in fzf_preview.sh? Something like this:
if [ -n "$FZF_PREVIEW_NO_BOTTOM_BORDER" ]; then
chafa -f sixel -s "${FZF_PREVIEW_COLUMNS}x$((FZF_PREVIEW_LINES - 1))" "$file"
else
chafa -f sixel -s "${FZF_PREVIEW_COLUMNS}x${FZF_PREVIEW_LINES}" "$file"
fiI know, it's far from elegant and hacky, but should work.
Another alternative (cleaner but less exact) would be to always perform the subtraction: it's not even a noticeable loss after all.
278dce9 fixes the issue, at least on iTerm2 where I tested it. As noted in the commit message, the momentary movement of the screen is sometimes noticeable, but I can't think of a better solution.
Using latest revision. Screen movement is noticeable, true, but not annoying at all. However, there are still a few glitches:
Do you have any idea under what conditions this happens?
What can be seen in the picture is a directory containing cached images of various sizes (from 400x80 to 1920x1360 aprox). The terminal is xterm (version 388, size 150x40). The misbehavior is triggered by a few images (three of them attached) for which no preview is generated (just a black background). The command used is ls | ~/build/fzf/target/fzf-linux_amd64 --preview='~/build/fzf/bin/fzf-preview.sh {}' --preview-window=border-left
This issue is avoided by just reducing FZF_PREVIEW_LINES by 1 (though previews are still not generated).
It's important to note that I've seen images (cannot find the guilty ones right now) overlapping the left border (which is why I normally reduce FZF_PREVIEW_COLUMNS by 2).
When a Sixel image touches the bottom of the screen, the whole
screen scrolls up one line to make room for the cursor.
Some terminals like Wezterm leave the cursor on the last row of a sixel to avoid this behavior. Because of that, the last commit 278dce9 breaks the UI on Wezterm when using --preview-window border-left.
I think the easiest solution to this is to just use the shorter preview window and leave room for the cursor. Another solution is to use \e[?8452h sequence which leaves the cursor on the right side of the the sixel, but not all terminals support it.
Some terminals like Wezterm leave the cursor on the last row of a sixel to avoid this behavior. Because of that, the last commit 278dce9 breaks the UI on Wezterm when using
--preview-window border-left.
Huh, that was what I was worried about, the inconsistencies between the terminal emulators. I'm going to have to revert the patch then.
I was not completely happy with the result anyway; while scrolling up fixes the whole viewport, the top line is lost.
21ab64e is another take on the issue. We export $FZF_PREVIEW_TOP to the preview command so that it can determine whether it should subtract 1 from $FZF_PREVIEW_LINES or not. Removed the scrolling trick.
\e[?8452h would make things much simpler, but it doesn't seem to work on iTerm2 which I currently use for testing.
Thank you for your help and feedback. I'm going to close this thread. Please open a new one if you encounter any problems or have any suggestions.
I'm very happy to see that sixel support has been added to fzf. However I haven't managed to use it, despite my terminal being compatible.
From the discussion above, I have tried the following:
fd . /tmp --extension png | fzf --preview '/tmp/fzf-preview.sh {}' --bind 'space:change-preview-window(border-left|border-top|border-bottom|border-horizontal|border-vertical|border-double|border-block|noborder)+clear-
screen+refresh-preview'But I just got a preview window with garbled characters. chafa /tmp/pic.png works fine.
An additional question: mpv also supports sixel when compiled with libsixel installed and used with --vo=sixel, which can be very convenient for covers of audio files for instance. However it's probably not using chafa under the hood. Is there a chance it can work in fzf previews too? My attempts have failed, but since I can't seem to make chafa work either, it's hard to conclude for mpv.
[Edit] Oh well, it seems the fzf package for my distribution is very outdated, with 0.41.
Yeah, with the latest release of fzf, sixel works just fine. Sorry for the noise.
mpv's sixel implementation yields issues though, with the image not properly showing (but sometimes it does, briefly) and a noticeable CPU hog:
fd . /tmp --extension mp3 | fzf --no-margin --no-padding --no-border --preview-window border-none --preview 'mpv --no-config --vo=sixel --really-quiet --vo-sixel-cols=${FZF_PREVIEW_COLUMNS} --vo-sixel-rows=${FZF_PREVIEW_LINES} {}Sorry for the spam, but some progress:
The sixel cover does seem to resize when I give it a width and a height in pixels, as opposed to giving it a number of columns and rows (above message) which just resulted in moving the large sixel image horizontally or vertically without resizing it.
- Are there any equivalents of
$FZF_PREVIEW_COLUMNSand$FZF_PREVIEW_LINES, but for width and height in pixels? - I believe
fzfis trying to update the image at each frame of thempvplayback, and somehow this is significantly more CPU intensive than just runningmpvwith sixel and withoutfzf. How can I tellfzfpreview to stop updating the preview until I select another file?