xonsh-developer-toolkit: A Python repository from anki-code

The xonsh developer toolkit contains all spectrum of instrument to develop xonsh shell.

If you like the idea click ⭐ on the repo and tweet.

Become xonsh contributor

The simplified workflow to contribute to xonsh core

mkdir -p ~/git && cd ~/git
# For example your name is `alex` and you forked https://github.com/xonsh/xonsh on Github
git clone git@github.com:alex/xonsh.git
# You can setup IDE (see next section) to extremely speed up the work and test.
cd xonsh
git checkout -b my_awesome_pr

# Install dev packages
# python -m ensurepip --upgrade  # install pip if you have python without pip
pip install -U pip
pip install '.[dev]'

# Make changes: add new environment variable.
vim xonsh/environ.py
git add xonsh/environ.py

# Create test
vim tests/environ.py
pytest tests/environ.py

# Live test
python -m xonsh --no-rc

# Create news file
cd news
cp TEMPLATE.rst my_new_env_var.rst
vim my_new_env_var.rst
git add my_new_env_var.rst
cd ..

# Push
git commit -m "My new environment variable!"
git push

# Create PR: https://github.com/xonsh/xonsh/pulls

Setup IDE

PyCharm

The easiest way to start contribute to xonsh core:

Install IDE e.g. PyCharm.
Fork https://github.com/xonsh/xonsh and open in IDE.
Install dev dependencies: pip install '.[dev]' (you need pip >= 24, update it: python -m pip install -U pip).

Setup IDE e.g. PyCharm:

Create project based on xonsh code directory.
Click "Run" - "Run..." - "Edit Configurations"
Click "+" and choose "Python". Set:
    Name: "xonsh".
    Run: choose "module" and write "xonsh".
    Script parameters: "--no-rc -DPPP=1" (here "PPP" will help to identify process using `ps ax | grep PPP`).
    Working directory: "/tmp"  # to avoid corrupting the source code during experiments
    Environment variables: add ";XONSH_SHOW_TRACEBACK=1"
    Modify options: click "Emulate terminal in output console".
Save settings.

Open `xonsh/procs/specs.py` and `def run_subproc` function.
Put breakpoint to `specs = cmds_to_specs` code. See also: https://www.jetbrains.com/help/pycharm/using-breakpoints.html
Click "Run" - "Debug..." - "xonsh". Now you can see xonsh prompt.
Run `echo 1` and now you're in the debug mode on the breakpoint.
Press F8 to step forward. Good luck!

Create git branch and solve good first issue or popular issue.
Create pull request to xonsh.

Create a xontrib

Create your xontrib step by step from xontrib-template. Best xontribs:

Xonsh code

Pointers

The main loop for interactive prompt: main.py -> shell.shell.cmdloop().
Replacing sys.std* to catching with Tee(): base_shell.py -> tee = Tee(encoding=enc, errors=err).
The main function to run subprocess: procs/specs.py -> run_subproc.
History backends are in xonsh/history/.
Environment variables are in xonsh/environ.py.

Exceptions

`Bad file descriptor` or `I/O operation on closed file.`

If you catch this when using callable alias first of all remember that callable alias runs in asynchronous thread and it's not needed to trace exactly this error. Instead of this you need to trace how executing is working and when descriptor is opened and when it closed. The common case is when main thread closes descriptor but child thread uses it later or vice versa. See also 5482.

xonsh in docker

Test in pure Linux environment

docker run --rm -it xonsh/xonsh:slim bash -c "pip install -U 'xonsh[full]' && xonsh"
# a1b2c3  # docker container id
apt update && apt install -y vim git procps strace  # to run `ps`

# Connect to container from other terminal:
docker exec -it a1b2c3 bash

# Save docker container state to reuse:
docker ps
docker commit c3f279d17e0a local/my_xonsh  # the same for update
docker run --rm -it local/my_xonsh xonsh

Test old versions of xonsh

docker run --rm -it python:3.6-slim-bullseye bash -c "pip install xonsh==0.5.0 && bash"
xonsh

Capturing: stdout, stderr, tty

Test capturing aliases

xonsh --rc ~/git/xonsh-developer-toolkit/callias.xsh
ca-th?
# Callable alias: threadable.

ca-th-in?
# Callable alias: threadable, input.

ca-th-in | head

Test capturing using bash process manager

You can test any case around processes and signals using bash pipes and streams *:

bash
sleep 100 &
# [1] 332211
jobs
# [1]  + running    sleep 100
ps ax | grep sleep  # pid=332211
# 332211 s005  SN     0:00.00 sleep 100
kill -SIGINT 332211

fzf 2>/dev/null &  # run fzf with hidden TUI (fzf is using stderr to show it)
fzf < /dev/null &  # disable stdin
ps fzf | grep fzf
# etc etc etc

Process and subprocess

Docs

TTY
- The TTY demystified
Signals
Python threads
- Python: signals and threads: "Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread"
- In Python, what are the cons of calling os.waitpid in a program with multiple threads?
Misc
- fzf source code: Render UI directly to /dev/tty
Research
- Stackoverflow questions around subprocess Popen and PIPE
- Github code around subprocess Popen and PIPE
Processes
- The Unix process API is unreliable and unsafe

Tools for modeling the process behavior

echo 1 # Simple capturable and thredable process.
fzf  # Complex app that read STDIN, print TUI to STDERR, read terminal input from /dev/tty, print result to STDOUT.
sudo -k ls # Command that show password prompt to `/dev/tty` and runs process that could be capturable or uncapturable.
echo 123 | less  # Pipe capturable process to uncapturable process.
sleep 10  # Tool that are easy to run in background and interrupt.
python -c 'input()'  # Easy way to run unkillable (in some cases) input-blocked app.
python -c 'import os, signal, time; time.sleep(0.2); os.kill(os.getpid(), signal.SIGTTIN)'  # Self-signaling.
python -c '__import__('sys').exit(2)'  # Exiting with exit code.

Monitor process state codes (STAT)

ps ax  # Tool to monitor processes and have the process state in STAT column.

# STAT:
D    uninterruptible sleep (usually IO)
R    running or runnable (on run queue)
S    interruptible sleep (waiting for an event to complete)
T    stopped, either by a job control signal or because it is being traced.
W    paging (not valid since the 2.6.xx kernel)
X    dead (should never be seen)
Z    defunct ("zombie") process, terminated but not reaped by its parent.

Hot to send signals

ps ax | grep fzf   # get pid=123
kill -SIGINT 123
kill -SIGCONT 123

watch -n1 'ps ax | grep fzf'  # Monitor process state after sending signals (STAT).

Decode process signals from `os.waitpid`

# pid, proc_status = os.waitpid(pid_of_child_process, os.WUNTRACED)
# 123, 5759

from xonsh.tools import describe_waitpid_status
describe_waitpid_status(5759)

WIFEXITED - False  - Return True if the process returning status exited via the exit() system call.
WEXITSTATUS - 22 SIGTTOU - Return the process return code from status.
WIFSIGNALED - False  - Return True if the process returning status was terminated by a signal.
WTERMSIG - 127  - Return the signal that terminated the process that provided the status value.
WIFSTOPPED - True  - Return True if the process returning status was stopped.
WSTOPSIG - 22 SIGTTOU - Return the signal that stopped the process that provided the status value.
WCOREDUMP - False  - Return True if the process returning status was dumped to a core file.

Python app to test catching all signals

python singnals-catch.py

Note! We read in the manual that "Python signal handlers are always executed in the main Python thread of the main interpreter, even if the signal was received in another thread." (source). But this is not related to situation when we run subprocess. First of all the signal (e.g. SIGINT) will go to foreground subprocess task. So you need to test this carefully for case main_thread+thread+popen_subprocess.

Sigmask

Process by itself can catch signals. The sigmask is describing what signals process intended to catch. I used sigmask tool to decrypt fzf sigmask:

apt install -y golang-go
go get -v github.com/r4um/sigmask
ps ax | grep fzf  # pid=123

~/go/bin/sigmask 123  # SigCgt - signals that process wants to catch by itself.
# SigPnd
# ShdPnd
# SigBlk SIGUSR1,SIGUSR2,SIGPIPE,SIGALRM,SIGTSTP,SIGTTIN,SIGTTOU,SIGURG,SIGXCPU,SIGXFSZ,SIGVTALRM,SIGIO,SIGPWR,SIGRTMIN
# SigIgn
# SigCgt SIGHUP,SIGINT,SIGQUIT,SIGILL,SIGTRAP,SIGABRT,SIGBUS,SIGFPE,SIGUSR1,SIGSEGV,SIGUSR2,SIGPIPE,SIGALRM,SIGTERM,SIGSTKFLT

Trace signals with `strace`

docker run --rm -it xonsh/xonsh:slim bash -c "pip install -U 'xonsh[full]' && xonsh"
apt update && apt install -y vim git procps strace 
python -c 'input()' & 
# pid 123
strace -p 123
# strace: Process 123 attached
# [ Process PID=123 runs in x32 mode. ]
# --- stopped by SIGTTIN ---
kill -SIGCONT 72  # From another terminal.
# --- SIGCONT {si_signo=SIGCONT, si_code=SI_USER, si_pid=48, si_uid=0} ---
# syscall_0x7ffffff90558(0x5555555c5a00, 0, 0x7fffff634940, 0, 0, 0x3f) = 0x2
# --- SIGTTIN {si_signo=SIGTTIN, si_code=SI_KERNEL} ---
# --- stopped by SIGTTIN ---

Testing

Test xonsh script with Github Actions

https://github.com/anki-code/xonsh-developer-toolkit/blob/main/.github/workflows/python-app.yml

Stress test

while !(python -c 'import sys; sys.exit(1)') != 0:
    print('.', end='')

Trace

Trace xonsh code using xunter

mkdir -p ~/git/ && cd ~/git/
git clone git+https://github.com/xonsh/xonsh
cd xonsh
xunter --no-rc ++cwd ++filter 'Q(filename_has="procs/")' ++output /tmp/procs.xun
# In another terminal:
tail -f /tmp/procs.xun

Interesting case for tracing

Sleep

for i in range(5):
    print(i)
    sleep 10

After Ctrl-c we have continue with the next iteration. It's annoying. Here is the related code:

https://github.com/xonsh/xonsh/blob/26c6e119e259ee6eac990233f8b0710cb67ea6b2/xonsh/jobs.py#L269-L288

When we run sleep 10 the system process go to wait and we stuck on os.waitpid. After pressing Ctrl+C (sending SIGINT) process get the SIGINT signal and we catch that signal appeared using WIFSIGNALED (line 281) but have no actions about stopping the execution entirely and we continue execution. It would be cool to considering the cases around this (i.e. can we really stop the execution of the whole code?) and stopping the execution carefully here.

Input

@aliases.register("mysudo")
def _mysudo():
    input("Password:")
    echo 123

mysudo | grep 1

I expect the behavior like $(sudo -k echo 123 | grep 1) where you can enter the password and then got captured 123. But it's not working. Tracing this in IDE is very interesting.

Xonsh Development Toolkit

Install:

mkdir ~/git && cd ~/git
git clone git@github.com:anki-code/xonsh-developer-toolkit.git

Usage:

xonsh --rc ~/git/xonsh-developer-toolkit/dev.xsh

Tools

duct.py - a Python library for running child processes
https://github.com/goodboy/tractor
https://dramatiq.io/ - is a background task processing library for Python with a focus on simplicity, reliability and performance.

anki-code/xonsh-developer-toolkit

Become xonsh contributor

The simplified workflow to contribute to xonsh core

Setup IDE

PyCharm

Create a xontrib

Xonsh code

Pointers

Exceptions

Bad file descriptor or I/O operation on closed file.

xonsh in docker

Test in pure Linux environment

Test old versions of xonsh

Capturing: stdout, stderr, tty

Test capturing aliases

Test capturing using bash process manager

Process and subprocess

Docs

Docs

Tools for modeling the process behavior

Monitor process state codes (STAT)

Hot to send signals

Decode process signals from os.waitpid

Python app to test catching all signals

Sigmask

Trace signals with strace

Testing

Test xonsh script with Github Actions

Stress test

Trace

Trace xonsh code using xunter

Interesting case for tracing

Sleep

Input

Xonsh Development Toolkit

Tools

See also

`Bad file descriptor` or `I/O operation on closed file.`

Decode process signals from `os.waitpid`

Trace signals with `strace`