Extreme difficulty building from source on Windows

[pete-debiase]

Describe the bug

I cannot seem to get Plover to build from source under Windows 11, Python 3.10.8, tox 3.26, and bash as supplied via WSL.

To Reproduce

(1) git clone https://github.com/openstenoproject/plover.git
(2) cd plover
(3) tox -e setup bdist_win

Behavior

Command (3) above produces the attached error output three_output.txt. It seems as if the python command is not aliased as expected in my (bash?) environment (line 384).

I tried editing windows/dist_build.sh as attached (change python alias line 45, comment out wine stuff lines 83-86). tox -e setup bdist_win now yields attached error output alias_output.txt. Now it seems like there's some kind of pathing issue (line 1052).

I then tried hacking setup.py as attached (tweak pathing lines 93-97). tox -e setup bdist_win now yields the attached error output pathing_output.txt. Now it seems like there's some kind of pip dependency issue (line 1085).

At this point, I'm at a complete loss of what to try next. Fresh out of ideas. Surely it can't be this hard and I must be doing something wrong? Any help getting Plover building from source in this type of Windows environment would be immensely appreciated :) .

Environment

• Windows 11
• Python 3.10.8
• tox 3.26
• bash as supplied via WSL
• tox commands executed via Powershell

[benoit-pierre]

Is there a reason for using WSL?

[benoit-pierre]

Try using the bash supplied by git Windows installer.

[pete-debiase]

No particular reason for using WSL, it just feels much closer to a native Linux environment if you find yourself doing a lot of Linux stuff on Windows (✋).

In any case, I just tried with git bash and was able to reproduce the same errors each step of the way:

bash version

python: command not found

pathing issue

pip dependency issue

Any ideas? I'm totally stumped.

[benoit-pierre]

That patch would help make sure the bash script is using the right Python:

 setup.py              | 4 +++-
 windows/dist_build.sh | 4 ++++
 2 files changed, 7 insertions(+), 1 deletion(-)

diff --git i/setup.py w/setup.py
index beab2d759..be89fa69c 100755
--- i/setup.py
+++ w/setup.py
@@ -83,7 +83,9 @@ class BinaryDistWin(Command):
         pass
 
     def run(self):
-        cmd = [self.bash or 'bash', 'windows/dist_build.sh']
+        cmd = [self.bash or 'bash',
+               'windows/dist_build.sh',
+               '--python', sys.executable]
         if self.installer:
             cmd.append('--installer')
         if self.trim:
diff --git i/windows/dist_build.sh w/windows/dist_build.sh
index 95543a577..8c2e79f89 100755
--- i/windows/dist_build.sh
+++ w/windows/dist_build.sh
@@ -19,6 +19,10 @@ do
     -i|--installer)
       opt_installer=1
       ;;
+    -p|--python)
+      python="$2"
+      shift
+      ;;
     -t|--trim)
       opt_trim=1
       ;;

[benoit-pierre]

Which should not be necessary, are you sure some bash config is not messing with the PATH?

[benoit-pierre]

For that matter, which bash executable do you get in your shell?

Note: the bdist_win command supports --bash to specify the bash executable.

[pete-debiase]

Starting from a fresh clone of plover, applying that patch takes me back to python: command not found land:

I'm not sure whether some bash config is messing with the path and not really sure how to check. I do know I do all my Linux stuff in WSL via zsh, so I would have thought my git bash is about as stock as it can get (never touched it before today).

But it does look like I have a few bash.exe executables. Is the git one probably the best bet, or just try them all?

[benoit-pierre]

Sorry, my bad try:

 setup.py              | 4 +++-
 windows/dist_build.sh | 6 +++++-
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git i/setup.py w/setup.py
index beab2d759..be89fa69c 100755
--- i/setup.py
+++ w/setup.py
@@ -83,7 +83,9 @@ class BinaryDistWin(Command):
         pass
 
     def run(self):
-        cmd = [self.bash or 'bash', 'windows/dist_build.sh']
+        cmd = [self.bash or 'bash',
+               'windows/dist_build.sh',
+               '--python', sys.executable]
         if self.installer:
             cmd.append('--installer')
         if self.trim:
diff --git i/windows/dist_build.sh w/windows/dist_build.sh
index 95543a577..b2aeb81d7 100755
--- i/windows/dist_build.sh
+++ w/windows/dist_build.sh
@@ -8,6 +8,7 @@ opt_incremental=0
 opt_installer=0
 opt_trim=0
 opt_zipdir=0
+python='python'
 args=()
 
 while [ $# -ne 0 ]
@@ -19,6 +20,10 @@ do
     -i|--installer)
       opt_installer=1
       ;;
+    -p|--python)
+      python="$2"
+      shift
+      ;;
     -t|--trim)
       opt_trim=1
       ;;
@@ -42,7 +47,6 @@ builddir="build/windist"
 distdir="dist/$name-$version-win64"
 distzip="$distdir.zip"
 installer="$distdir.setup.exe"
-python='python'
 
 . ./windows/dist_deps.sh
 

[benoit-pierre]

But it does look like I have a few bash.exe executables. Is the git one probably the best bet, or just try them all?

Yes, try the git one, make sure which bash is returning it, or use --bash.

[pete-debiase]

Looks like the Git Bash app/environment natively points bash to whatever's built into MINGW64.

Went back to fresh clone, applied latest suggested patch, ran build with explicitly specified git bash executable. Took me back to the pip dependency issue. Did I specify the bash executable incorrectly?

tox -e setup -- bdist_win --bash "C:\scoop\apps\git\2.38.1.windows.1\usr\bin\bash.exe"

[benoit-pierre]

Went back to fresh clone, applied latest suggested patch, ran build with explicitly specified git bash executable. Took me back to the pip dependency issue. Did I specify the bash executable incorrectly?

The process would have croaked earlier.

As for the pip error… Just pip being pip… What a sorry excuse for a package manager! The two requirements are not incompatible (plover-stroke>=1.1.0 and plover-stroke==1.1.0)…

We normally pin pip to a known working version (see reqs/constraints.txt & plover_build_utils/get_pip.py).

What's the version that end up being used in build\windist\data? (Look for a dist-info entry).

[benoit-pierre]

Also it would help to know which part of the script is being executed, can you provide the full output?

[pete-debiase]

Definitely! Wanted to attach full output before rather than screenshot but Git Bash shell/app thing was not even piping output into a text file for me for some reason.

Got it though, this was the result of tox -e setup -- bdist_win --bash "C:\scoop\apps\git\2.38.1.windows.1\usr\bin\bash.exe" starting from fresh clone with latest suggested patch applied: output.txt.

May be being a bonehead but can't seem to find a dist-info in build\windist\data. Is it maybe failing before that gets generated?

Will try downgrading pip to one of the specified versions (was using 22.3 on all prior attempts).

Edit: Same pip dependency error after manually downgrading pip to 21.3.1.

[benoit-pierre]

May be being a bonehead but can't seem to find a dist-info in build\windist\data. Is it maybe failing before that gets generated?

I should have specified: "Recursively look for…" :P

Try this:

 setup.py              |  4 +++-
 windows/dist_build.sh | 10 +---------
 2 files changed, 4 insertions(+), 10 deletions(-)

diff --git i/setup.py w/setup.py
index f137ec5d4..154c659b9 100755
--- i/setup.py
+++ w/setup.py
@@ -83,7 +83,9 @@ class BinaryDistWin(Command):
         pass
 
     def run(self):
-        cmd = [self.bash or 'bash', 'windows/dist_build.sh']
+        cmd = [self.bash or 'bash',
+               'windows/dist_build.sh',
+               '--python', sys.executable]
         if self.installer:
             cmd.append('--installer')
         if self.trim:
diff --git i/windows/dist_build.sh w/windows/dist_build.sh
index b2aeb81d7..25290dc46 100755
--- i/windows/dist_build.sh
+++ w/windows/dist_build.sh
@@ -57,14 +57,6 @@ build_dist()
 {(
   kernel="$(uname -s)" || die
 
-  # Build/cache the wheels we need first using the default Python, so C
-  # extensions can be successfully built (the embedded distribution does
-  # not include the necessary files, e.g. `Python.h`).
-  if [ "$kernel" != "Linux" ]
-  then
-    bootstrap_dist "$wheel" --no-warn-script-location --no-install
-  fi
-
   # Fetch official embedded distribution.
   py_embed_zip="$(run "$python" -m plover_build_utils.download "https://www.python.org/ftp/python/$py_embed_version/python-$py_embed_version-embed-amd64.zip" "$py_embed_sha1")" || die
 
@@ -92,7 +84,7 @@ build_dist()
   python='dist_python'
 
   # Install Plover and dependencies.
-  bootstrap_dist "$wheel" --no-warn-script-location --no-index
+  bootstrap_dist "$wheel" --no-warn-script-location
 
   # Trim the fat...
   if [ $opt_trim -eq 1 ]

[pete-debiase]

That patch got me building successfully! You freaking rock!!! All of the following now appear to work:

• tox -e setup -- bdist_win --bash "C:\scoop\apps\git\2.38.1.windows.1\usr\bin\bash.exe" (explicitly specify git bash executable)
• tox -e setup -- bdist_win in Git Bash app/shell thing, where bash points to /usr/bin/bash in MINGW64 environment
• tox -e setup -- bdist_win in WSL, where my bash is aliased to bash.exe which points to C:\Windows\System32\bash.exe (whatever bash comes with WSL...I think)

What can I do to help document/resolve this issue? Were you able to reproduce any of this on your machine or was it all just me apparently having a weird environment configuration (please let it not be that).

[pete-debiase]

Also, thank you so much for your help :) . You rock. Now I get to go play around with a plover.qss for a while and be stenoing in style 😎 .

[benoit-pierre]

You did not need to build a version yourself to use plover.qss.

[benoit-pierre]

Why not use the official distribution by the way?

[pete-debiase]

You did not need to build a version yourself to use plover.qss.

I know - that's just one new feature that I saw in recent release notes that I was excited to play with :) .

Why not use the official distribution by the way?

I developed a custom theory + custom hardware that require a few tweaks to Plover to work. I've been using it full time since ~2017 and am working on documenting it to share with the community at some point, but that's a big project I'm doing a little at a time.

In particular, to the best of my knowledge the current built-in support for alternative theories did not exist back when I started, and I've also gotten a little better at writing software in the last 5 years (I hope). Cleanly reimplementing support for the system in modern Plover is easy, mostly it's just writing documentation that I'm slacking on. A truly shameful state of affairs for a steno user indeed :) .

Anyways, thanks again for your help! Looking a little ahead, if I wanted to get more involved with Plover development, where would be the best place to jump in? Trying to squash exisiting bugs/reduce the number of open issues? I've read the contributing page about a dozen times over the years but always get shy/unsure about jumping in myself.

[excalamus]

if I wanted to get more involved with Plover development, where would be the best place to jump in

Come say "hi" on the Discord if you haven't already.