vimpager and vimcat freeze, no output (OS X, homebrew)
braham-snyder opened this issue · 13 comments
On OS X El Capitan, with homebrew vim, using vimpager also installed through homebrew, with and without --HEAD, from bash, in both Terminal.app and iTerm2:
bsmbp:~ bs$ brew info vimpager
vimpager: stable 2.06 (bottled), HEAD
Use ViM as PAGER
https://github.com/rkitover/vimpager
/usr/local/Cellar/vimpager/HEAD-4f1a077 (24 files, 399.9KB) *
Built from source on 2017-10-03 at 20:34:12
From: https://github.com/Homebrew/homebrew-core/blob/master/Formula/vimpager.rb
==> Dependencies
Build: pandoc ✘
Optional: pandoc ✘
==> Options
--with-pandoc
Use pandoc to build and install man pages
--HEAD
Install HEAD version
==> Caveats
To use vimpager as your default pager, add `export PAGER=vimpager` to your
shell configuration.
bsmbp:~ bs$ cat ~/.vimpagerrc
set clipboard=unnamed
bsmbp:~ bs$ vim --version
VIM - Vi IMproved 8.0 (2016 Sep 12, compiled Sep 1 2017 18:55:49)
MacOS X (unix) version
Included patches: 1-997
Compiled by Homebrew
Huge version without GUI. Features included (+) or not (-):
+acl +file_in_path +mouse_sgr +tag_old_static
+arabic +find_in_path -mouse_sysmouse -tag_any_white
+autocmd +float +mouse_urxvt -tcl
-balloon_eval +folding +mouse_xterm +termguicolors
-browse -footer +multi_byte +terminal
++builtin_terms +fork() +multi_lang +terminfo
+byte_offset -gettext -mzscheme +termresponse
+channel -hangul_input +netbeans_intg +textobjects
+cindent +iconv +num64 +timers
+clientserver +insert_expand +packages +title
+clipboard +job +path_extra -toolbar
+cmdline_compl +jumplist +perl +user_commands
+cmdline_hist +keymap +persistent_undo +vertsplit
+cmdline_info +lambda +postscript +virtualedit
+comments +langmap +printer +visual
+conceal +libcall +profile +visualextra
+cryptv +linebreak +python +viminfo
+cscope +lispindent -python3 +vreplace
+cursorbind +listcmds +quickfix +wildignore
+cursorshape +localmap +reltime +wildmenu
+dialog_con -lua +rightleft +windows
+diff +menu +ruby +writebackup
+digraphs +mksession +scrollbind +X11
-dnd +modify_fname +signs +xfontset
-ebcdic +mouse +smartindent -xim
+emacs_tags -mouseshape +startuptime -xpm
+eval +mouse_dec +statusline +xsmp_interact
+ex_extra -mouse_gpm -sun_workshop +xterm_clipboard
+extra_search -mouse_jsbterm +syntax -xterm_save
+farsi +mouse_netterm +tag_binary
system vimrc file: "$VIM/vimrc"
user vimrc file: "$HOME/.vimrc"
2nd user vimrc file: "~/.vim/vimrc"
user exrc file: "$HOME/.exrc"
defaults file: "$VIMRUNTIME/defaults.vim"
fall-back for $VIM: "/usr/local/share/vim"
Compilation: clang -c -I. -Iproto -DHAVE_CONFIG_H -DMACOS_X_UNIX -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1
Linking: clang -L. -fstack-protector -L/usr/local/lib -L/usr/local/opt/libyaml/lib -L/usr/local/opt/openssl/lib -L/usr/local/opt/readline/lib -L/usr/local/lib -o vim -lXt -lX11 -lSM -lICE -lncurses -liconv -framework Cocoa -mmacosx-version-min=10.11 -fstack-protector-strong -L/usr/local/lib -L/usr/local/Cellar/perl/5.26.0/lib/perl5/5.26.0/darwin-thread-multi-2level/CORE -lperl -lm -lutil -lc -F/usr/local/opt/python/Frameworks -framework Python -lruby.2.4.1 -lobjc
bsmbp:~ bs$ vimpager -x ~/.vimpagerrc
+ [ 1 -gt 0 ]
+ break
+ find_tmp_directory
+ mkdir_options='-m 700'
+ [ -n '' ]
+ tmp=/tmp
+ tmp=/tmp/vimpager_81182
+ mkdir -m 700 /tmp/vimpager_81182
+ trap 'quit 1' PIPE HUP INT QUIT ILL TRAP KILL BUS TERM
+ detect_term_size
+ command -v tput
+ 1> /dev/null
+ tput cols
+ 1> /tmp/vimpager_81182/cols
+ tput lines
+ 1> /tmp/vimpager_81182/lines
+ cat /tmp/vimpager_81182/cols
+ cols=117
+ cat /tmp/vimpager_81182/lines
+ lines=32
+ rm -f -- /tmp/vimpager_81182/cols /tmp/vimpager_81182/lines
+ [ -z 117 ]
+ [ -z 117 ]
+ expand_config_vars
+ eval runtime='"/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager"'
+ runtime=/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager
+ eval vimcat='"/usr/local/Cellar/vimpager/HEAD-4f1a077/bin/vimcat"'
+ vimcat=/usr/local/Cellar/vimpager/HEAD-4f1a077/bin/vimcat
+ eval system_vimpagerrc='"/usr/local/Cellar/vimpager/HEAD-4f1a077/etc/vimpagerrc"'
+ system_vimpagerrc=/usr/local/Cellar/vimpager/HEAD-4f1a077/etc/vimpagerrc
+ find_vim_executable
+ [ -n '' ]
+ tvim=vim
+ [ -n vim -a -z '' ]
+ VIMPAGER_VIM=vim
+ export VIMPAGER_VIM
+ [ -n vim ]
+ tvim=vim
+ [ -n '' ]
+ vim_cmd=vim
+ [ ! -n '' -a -n '' ]
+ find_vimpagerrc_files
+ i=1
+ OLDIFS=$' \t\n'
+ IFS=$'\n'
+ IFS=$' \t\n'
+ vim -NEnR -i NONE '+call writefile(["", "VAL:" . $VIM, "VAL:" . $MYVIMRC], "/dev/stderr")' +q
+ 0< /dev/tty 2>& 1
At which point output stops, and all keys I press are simply appended to the output. Well, other than ^Z or ^\ -- the latter of which then shows (probably unremarkably):
+ quit 1
+ rm -f gvim.exe.stackdump
+ cd /tmp
+ 2> /dev/null
+ rm -rf /tmp/vimpager_98242
+ 2> /dev/null
+ exit 1
And same for vimcat, with this debugging output:
bsmbp:~ bs$ vimcat -x ~/.vimpagerrc
+ tmp_dir=/tmp
+ mkdir_options='-m 700'
+ uname -s
+ tmp_dir=/tmp/vimcat_90447
+ mkdir -m 700 /tmp/vimcat_90447
+ trap 'quit 1' PIPE HUP INT QUIT ILL TRAP KILL BUS TERM
+ tmp_file_in=/tmp/vimcat_90447/vimcat_in.txt
+ out_fifo=/tmp/vimcat_90447/vimcat_out.fifo
+ uname -s
+ mkfifo /tmp/vimcat_90447/vimcat_out.fifo
+ [ 2 -gt 0 ]
+ shift
+ [ 1 -gt 0 ]
+ break
+ [ -z '' ]
+ [ ! -t 1 ]
+ [ -z '' ]
+ [ -f /Users/bs/.vimcatrc ]
+ vimcatrc=''
+ [ 1 -eq 0 ]
+ [ -n '' -a 1 -gt 1 ]
+ chunks_dir=/tmp/vimcat_90447/chunks
+ mkdir /tmp/vimcat_90447/chunks
+ i=1
+ [ 1 -ge 2 ]
+ pipeline=''
+ pipeline_start=/Users/bs/.vimpagerrc
+ [ 0 -eq 1 ]
+ exit_code=0
+ [ /Users/bs/.vimpagerrc '!=' - ]
+ [ ! -r /Users/bs/.vimpagerrc ]
+ [ ! -s /Users/bs/.vimpagerrc ]
+ [ -z '' -o '' '=' - ]
+ dest_file=/tmp/vimcat_90447/vimcat_out.fifo
+ tail_pid=90453
+ start_highlight_job
+ set -- -NE -i NONE -n --cmd 'set runtimepath^=/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager' --cmd $'call vimcat#Init({ \'rc\': \'\' })' --cmd visual -c 'call vimcat#Run("/tmp/vimcat_90447/vimcat_out.fifo", 0, "/tmp/vimcat_90447/chunks", "/Users/bs/.vimpagerrc")'
+ [ -n '' ]
+ [ 0 -eq 0 ]
+ vim_pid=90454
+ start_pipeline
+ [ -n '' ]
+ pipeline=write_chunks
+ tail -f /tmp/vimcat_90447/vimcat_out.fifo
+ vim -NE -i NONE -n --cmd 'set runtimepath^=/usr/local/Cellar/vimpager/HEAD-4f1a077/share/vimpager' --cmd $'call vimcat#Init({ \'rc\': \'\' })' --cmd visual -c 'call vimcat#Run("/tmp/vimcat_90447/vimcat_out.fifo", 0, "/tmp/vimcat_90447/chunks", "/Users/bs/.vimpagerrc")'
+ 3<& 0
+ cat -- /Users/bs/.vimpagerrc
+ 0< /dev/tty 1> /dev/null 2>& 1
+ echo 90458
+ eval write_chunks
+ 1> /tmp/vimcat_90447/pipeline_pid
+ 0<& 3
+ write_chunks
+ cd /tmp/vimcat_90447/chunks
+ rm -f -- '*'
+ cat /tmp/vimcat_90447/pipeline_pid
+ split -b 4096 -
+ pipeline_pid=90458
+ [ ! -f /tmp/vimcat_90447/vim_done ]
+ do_sleep 50
+ _ms=50
+ vim -NEsnR -i NONE -u NONE '+sleep 50 m' +q
+ 1> /dev/null 2>& 1
+ touch PIPELINE_DONE
Thanks for vimpager!
I was experimenting with colorizing programs and found this ticket, and felt really sorry for you. So I'll help you out.
-
Vim actually already has a built-in "vimpager" script, called
less.sh. It's installed in a "macros" folder somewhere on your system. Find it withsudo find -x / -name less.sh. Although it isn't very good... -
Either way, the best pager is not vim. Not by a long shot. Vim is slow as hell as a pager (takes 2 seconds to start up before anything is shown). The best pager is
less. But not by default. It needs some options to make it usable. Then you'll get full pagination, UTF-8, color highlighting, line numbers, text search via/just like in vim, and you can even edit files with vim directly from less by pressingv(for "visual editor"). And you pressqto leave less.
Here's how to setup less properly to make it your pager and get syntax highlighted pagination of man-pages etc:
# COLORIZE THE OUTPUT OF "LESS" TO MAKE MANPAGES MORE READABLE
export LESS_TERMCAP_mb=$'\E[01;31m';
export LESS_TERMCAP_md=$'\E[01;31m';
export LESS_TERMCAP_me=$'\E[0m';
export LESS_TERMCAP_se=$'\E[0m';
export LESS_TERMCAP_so=$'\E[01;44;33m';
export LESS_TERMCAP_ue=$'\E[0m';
export LESS_TERMCAP_us=$'\E[01;32m';
# TWEAKING OTHER "LESS" PARAMETERS
export LESSCHARSET='utf-8';
export LESS='-i -N -w -z-4 -g -M -R -P%t?f%f :stdin .?pb%pb\%:?lbLine %lb:?bbByte %bb:-...';
# SETTING "LESS" AS DEFAULT PAGER AND REPLACING ALL OTHER PAGERS WITH IT
export PAGER=less;
alias more=$PAGER;
alias zless=$PAGER;
# SET UP "VIM" AS DEFAULT EDITOR
export VISUAL=vim;
export EDITOR=$VISUAL;
You need to put that in your ~/.bash_profile or ~/.bashrc.
Now all your man git and stuff will load with less in beautiful syntax highlighting, colorized mode. And it's super fast. Unlike vim/vimpager which takes 1-2 seconds to start, less is actually instant.
- You wanted
vimcatto colorize code, right? Well, it's truly garbage. I'm sorry... I'm really sorry. I understand the work involved in making the vimscript that powers it. But it's garbage due to how bad vim's scripting language/limitations are.
For a 1600 line file, I ran vimcat <that file> and it took about 2 minutes to print all lines, line by line... as if I was running on a 300 baud modem in the 1980s. If I tried Ctrl-C to abort that tedious process, it hung and stopped accepting keyboard input and had to be killed via ps aux | grep vim and killing the process... (Probably because Ctrl-C just killed the "tail" pipe program but not actually vim? No idea...)
Why was it so slow? Because vimcat uses a temporary file named /tmp/vimcat_####/vimcat.fifo which is read by the shell 1 line at a time until it is empty.
Tedious...
- The best syntax highlighter by far is actually a Python program named Pygments. It supports over 300 languages and supports outputting the syntax highlighting to all kinds of targets such as HTML, Terminal (BINGO!), etc.
To install it, do the following:
First install Homebrew.
Then run these commands to install Python 2 and Python 3:
brew install python3
brew install python
brew unlink python && brew link --overwrite python # make sure python links to python2
pip2 install --upgrade pip setuptools # upgrade to latest pip2
pip3 install --upgrade pip setuptools # upgrade to latest pip3
# never use the generic “python” and “pip”.
# always use “python[2/3]” and “pip[2/3]”.
Then, to install the Python3-version of Pygments, I just ran:
pip3 install pygments
# to upgrade later:
pip3 install pygments --upgrade
Lastly, make an alias for running Pygmentize with output as colorized 256-color terminal, etc:
alias pcat='pygmentize -f terminal256 -O style=monokai -g'
Now just run pcat main.cpp or pcat index.php or whatever other language.
Enjoy the perfection on your screen! It syntax highlights the 1600 line file in ~1 second (pygmentize) compared to ~120 seconds (vimcat). And supports 300 programming languages.
You can change the style (theme). I use monokai, but others are listed here: pygmentize -L styles. (Another good one is native).
If you need to output to other formats (like BBCode for forums, etc), then just run this command to look at all of its available output formats. :-) pygmentize -L formatters.
Have fun.
Here's an example of the pcat output:
I appreciate the thought, but I'm painfully aware of less, and I'm only incidentally interested in vimcat's behavior.
I'm looking for something more powerful. More precisely: I want to be able to search and select text in my pager as well as I can in emacs or vim (so -- ideally -- I'd like access to easymotion, sneak/snipe, swiper, evil-little-word, etc.).
Meanwhile, less can't even wrap its searches (which alone kills me) -- nor is its search incremental.
One use-case I'm particularly interested in is better terminal scrollback: I'd like to test https://github.com/kovidgoyal/kitty with my scrollback_pager set to vimpager (though I'd surely need to limit the number of lines sent to vimpager to keep startup times reasonable).
- Neovim's
+termalmost covers this use-case, but clips the scrollback upon resizes of the terminal window ( neovim/neovim#5045 ) emacs'sshellmisses the underlying shell's completion, so that's a no-go for me.emacs'sansi-term's font-locking is too slow (all printing is slowed down -- it also gets significantly slower as the buffer gets larger).- https://github.com/mbriggs/emacs-pager and similar Emacs programs all seem to suffer from the same issue:
ansi-color-apply-on-region(or equivalent) simply takes too long -- I suspect it would be fairly difficult to speed up the coloring enough for my taste
- https://github.com/mbriggs/emacs-pager and similar Emacs programs all seem to suffer from the same issue:
- The only reason I haven't tried
eshellis that it's very nonstandard (it's possible I'll use it down the road, but first I want a setup for a more standard shell that I'm relatively happy with)
You need to put that in your ~/.bash_profile or ~/.bashrc
You might want those non-bash-specific exports in ~/.profile instead (which you can then source from ~/.bash_profile if the latter also exists): https://unix.stackexchange.com/a/45687/193985
Vim is slow as hell as a pager (takes 2 seconds to start up before anything is shown).
I see sub-one-second startup times with this, which I greatly prefer to less (though I also need to try emacs's woman):
export MANPAGER="/usr/local/bin/nvim -c 'set ft=man' -"
Which is what led me to want to try vimpager as well.
You wanted vimcat to colorize code, right?
No, actually -- I only included the vimcat -x output above to hopefully help the author (or help anyone) diagnose why I cannot get vimpager working.
For a 1600 line file, I ran vimcat and it took about 2 minutes to print all lines, line by line... as if I was running on a 300 baud modem in the 1980s. If I tried Ctrl-C to abort that tedious process, it hung and stopped accepting keyboard input and had to be killed via ps aux | grep vim and killing the process... (Probably because Ctrl-C just killed the "tail" pipe program but not actually vim? No idea...)
Why was it so slow? Because vimcat uses a temporary file named /tmp/vimcat_####/vimcat.fifo which is read by the shell 1 line at a time until it is empty.
While I would not at all be surprised if vimcat were relatively slow, a two-minute runtime for a 1600 line file must surely be a separate bug? Have you tried with a blank ~/.vimrc? (Though -- again -- I'm primarily interested in vimpager.)
edit: there's always tmux for scrollback paging and selection, but its vim-like keybindings are very limited
Haha, alright good luck with everything! :-) By the way, you can search incrementally forwards and backwards in less: /word, n, n, n, shift-N, shift-N.
And with my tweaks you get search result-highlighting etc. Works perfectly for my needs. But you've got other needs and vimpager seems more up your alley. Good luck.
Haha, alright good luck with everything! :-)
Thank you!
By the way, you can search incrementally
I mean a different kind of incremental search (it's unfortunately quite ambiguously-named -- I might start calling it "real-time search" or "as-you-type-matching" or "incrementally-updated search" or something): https://en.wikipedia.org/wiki/Incremental_search
Ohh that kind of search. Sure that's nice. :)
Hmm just as a warning to you:
Remember the vimcat command I described taking 2 minutes, and which I hit Ctrl-C to abort, which froze vim and the terminal and required me to kill that process manually?
Uhm... well, that exact failed process has somehow continued running in the background since yesterday and is taking 60% CPU:
I know it's that process, because under the Open Files and Ports tab of Activity Monitor, it lists the folder of the file I was vimcating... so it belongs to that (long-ago closed) terminal.
I discovered this now because my fans are going nuts, and I wondered why my CPU was so hot... and why my battery drained in like 40 minutes.
Strange stuff...
Time to reboot my computer.
Edit: Rebooted and all is well and my battery estimate is back to 2-3 hours. So yeah turns out vimcat can really badly kill bash on macOS Sierra. I mention it because it may be related to the crashing you mentioned in this ticket.
Did you ever find a solution for your problem? I'm running against the same wall at the moment ;-)
Regarding the vimpager freeze? Nope, but I haven't toyed with it further.
Or $PAGER more generally? I decided to settle -- for now at least -- with tmux's set-window-option -g mode-keys vi + set -g status-keys emacs (the vi status-keys are quite odd) + some minor modifications to make it slightly more vim-like (since, e.g., for some reason tmux only defaults the emacs-style search to incremental, and yanking does not cancel the visual selection by default).
All of the Emacs solutions print even medium-length outputs too slowly, due to the time it takes Emacs to deal with ANSI-coloring (which I suspect would be difficult to speed up).
neovim/neovim#5054 could be quite relevant (with kitty for scrollback paging without re-running commands), but I'm not quite sure of the scope (is the idea to add ANSI-coloring capability to nvim itself?) -- plus it's "unplanned" at the moment.
@braham-snyder that issue is about the "internal pager" of (n)vim that is used for long output from vim commands (see :h more-prompt).
I'm sorry, I haven't worked on this repo in a while because of various things in life, and am currently deeply engrossed in another project.
I'll try to take a break from the other project soon and take a look at stuff here.
@rkitover Relax and have fun with the other project. ;-) You're not under any obligation to do this. Have fun and come back when you feel like it.
@rkitover Take your time! My question was more out of curiosity since the error I'm getting is not exactly the same as described here and I was in doubt whether to create a separate issue or not ... no real life has caught me again and I'll have to see when I can dive into it a bit more ;-)
@braham-snyder thanks for your quick response! I'll see how I can tackle the problem ...