Make sure promnesia works under Windows
Pfedj opened this issue · 23 comments
Windows 10 with Python 3.7.6.
When I run promnesia demo --port 16789 https://github.com/karlicoss/exobrain
I get error:
c:\anaconda\lib\site-packages\promnesia\kython\klogging2.py:27: UserWarning: You might want to install 'logzero' for nice colored logs!
warnings.warn("You might want to install 'logzero' for nice colored logs!")
[INFO 2020-05-18 20:49:57,102 promnesia common.py:329] extracting via promnesia.sources.guess:index ('https://github.com/karlicoss/exobrain',) {} ... ...
[INFO 2020-05-18 20:49:57,102 promnesia common.py:329] extracting via promnesia.sources.guess:index ('https://github.com/karlicoss/exobrain',) {} ... ...
Traceback (most recent call last):
File "c:\anaconda\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File "c:\anaconda\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\anaconda\Scripts\promnesia.exe\__main__.py", line 7, in <module>
File "c:\anaconda\lib\site-packages\promnesia\__main__.py", line 173, in main
do_demo(index_as=getattr(args, 'as'), params=args.params, port=args.port, config_file=args.config)
File "c:\anaconda\lib\site-packages\promnesia\__main__.py", line 110, in do_demo
errors = _do_index()
File "c:\anaconda\lib\site-packages\promnesia\__main__.py", line 49, in _do_index
hist, errors = previsits_to_history(ex, src=ex.src)
File "c:\anaconda\lib\site-packages\promnesia\common.py", line 333, in previsits_to_history
previsits = list(extr()) # TODO DEFENSIVE HERE!!!
File "c:\anaconda\lib\site-packages\promnesia\sources\guess.py", line 33, in index
yield from index_(path, *args, **kwargs)
File "c:\anaconda\lib\site-packages\promnesia\sources\vcs.py", line 15, in index
check_call(['git', 'clone', repo, tp])
File "c:\anaconda\lib\subprocess.py", line 358, in check_call
retcode = call(*popenargs, **kwargs)
File "c:\anaconda\lib\subprocess.py", line 339, in call
with Popen(*popenargs, **kwargs) as p:
File "c:\anaconda\lib\subprocess.py", line 800, in __init__
restore_signals, start_new_session)
File "c:\anaconda\lib\subprocess.py", line 1148, in _execute_child
args = list2cmdline(args)
File "c:\anaconda\lib\subprocess.py", line 555, in list2cmdline
needquote = (" " in arg) or ("\t" in arg) or not arg
TypeError: argument of type 'WindowsPath' is not iterable
So how can I run this program on Windows?
Hey.. Uhoh, it would need some work for that first, sorry! I'm trying to keep things as portable as possible, but had to keep Windows in mind, I haven't used it for years.
I think github actions got windows available for CI, so maybe at some point I'll have time for this.
Some related comments here (apparently can be run via WSL?) #114 (comment)
Right, I played a bit with CircleCI Windows runners and SSH debugging -- not too pleasant experience, but did tackle some issues.
#143
This particular problem is a bug in Python 3.7! https://bugs.python.org/issue33617
I patched this particular place, but it's possible that there are more. If anyone notices more of these, happy to receive a PR (or at least post a comment).
Most tests are passing now, however, hug (the HTTP framework I'm using) itself doesn't seem to start at all, which is of course a major issue.
I wonder if it's related to the firewall (e.g. similar to hugapi/hug#470), but don't know about Windows, so will have to research it.
Not sure when I'll have time for another round of investigation, but hopefully this motivates someone else to carry on (especially if you have a proper Windows system around!).
Alright. I don't have much python knowledge, but here's where I'm at
c:\>promnesia index
[INFO 2020-09-30 07:25:36,953 promnesia common.py:417] extracting via promnesia.sources.auto:index () {} ... ...
[ERROR 2020-09-30 07:25:36,954 promnesia common.py:424] index() missing 1 required positional argument: 'path'
Traceback (most recent call last):
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\common.py", line 422, in previsits_to_history
previsits = list(extr())
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\common.py", line 410, in <lambda>
extr = lambda: ex.ff(*ex.args, **ex.kwargs)
TypeError: index() missing 1 required positional argument: 'path'
[INFO 2020-09-30 07:25:37,015 promnesia dump.py:51] saved database to C:\Users\Uncle Awesome\AppData\Local\promnesia\promnesia\promnesia.sqlite
c:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia>promnesia index
Traceback (most recent call last):
File "c:\users\uncle awesome\appdata\local\programs\python\python38-32\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "c:\users\uncle awesome\appdata\local\programs\python\python38-32\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\Scripts\promnesia.exe\__main__.py", line 7, in <module>
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\__main__.py", line 254, in main
do_index(config_file=args.config)
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\__main__.py", line 59, in do_index
config.load_from(config_file)
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\config.py", line 87, in load_from
instance = import_config(config_file)
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\config.py", line 103, in import_config
spec.loader.exec_module(mod) # type: ignore
File "<frozen importlib._bootstrap_external>", line 783, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "c:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\promnesia\config.py", line 9, in <module>
from .common import PathIsh, get_tmpdir, appdirs, default_output_dir
ImportError: attempted relative import with no known parent package
c:\>promnesia serve
Immediate responds with
[INFO 2020-09-30 07:32:49,837 promnesia server.py:305] Running server: ['python', '-m', 'hug', '-p', '13131', '-f', 'C:\\Users\\Uncle Awesome\\AppData\\Roaming\\Python\\Python38\\site-packages\\promnesia\\server.py']
c:\>
But a second later gives
Traceback (most recent call last):
File "C:\Users\Uncle Awesome\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 194, in _run_module_as_main
return _run_code(code, main_globals, None,
File "C:\Users\Uncle Awesome\AppData\Local\Programs\Python\Python38-32\lib\runpy.py", line 87, in _run_code
exec(code, run_globals)
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\__main__.py", line 3, in <module>
hug.development_runner.hug.interface.cli()
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\interface.py", line 650, in __call__
raise exception
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\interface.py", line 646, in __call__
result = self.output(self.interface(**pass_to_function), context)
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\interface.py", line 129, in __call__
return __hug_internal_self._function(*args, **kwargs)
File "C:\Users\Uncle Awesome\AppData\Roaming\Python\Python38\site-packages\hug\development_runner.py", line 65, in hug
api_module = importlib.machinery.SourceFileLoader(file.split(".")[0], file).load_module()
File "<frozen importlib._bootstrap_external>", line 462, in _check_name_wrapper
File "<frozen importlib._bootstrap_external>", line 962, in load_module
File "<frozen importlib._bootstrap_external>", line 787, in load_module
File "<frozen importlib._bootstrap>", line 265, in _load_module_shim
File "<frozen importlib._bootstrap>", line 702, in _load
File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 779, in exec_module
File "<frozen importlib._bootstrap_external>", line 915, in get_code
File "<frozen importlib._bootstrap_external>", line 972, in get_data
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\Uncle'
Just promnesia index won't do much if you don't have a config, I recommend following the setup/troubleshooting guide https://github.com/karlicoss/promnesia#setup
As for promnesia serve, at the first glance, this looks like a bug in hug with the whitespace handling in paths it splits Uncle Awesome for some reason..
RE: promnesia index ah of course.🤦♂️ I added a proper path and looks good now 👍
RE: promnesia serve I added an issue at hug hugapi/hug/issues/872
I looked a bit and I think I was wrong about hug -- more likely an issue with handling whitespace in os.execvpe call.
- https://bugs.python.org/issue19066
- https://stackoverflow.com/questions/53218984/how-can-i-avoid-the-windows-exec-and-spawn-functions-splitting-arguments-at
I think you can try enclosing __file__ here in extra quotes:
promnesia/src/promnesia/server.py
Line 303 in 15088ba
, e.g. '-f', '"' + __file__ + '"',.
If that works, I guess I'll look up what should be used instead so it's portable.. or perhaps hack it just for Windows.
can confirm '-f', '"' + __file__ + '"', works 👍
User reported similar error when checking promnesia doctor config (fails during compileall check). I guess need to setup some mypy rule to check that all paths are passed as strings
Also sqlitebrowser doesn't (or might not?) register itself in PATH on windows, so maybe need a better way of launching it
Perhaps need to document: paths on Windows need to be prepended with r because of backslashes https://stackoverflow.com/questions/18084554/why-do-i-get-a-syntaxerror-for-a-unicode-escape-in-my-file-path/18084594#18084594
Also find command on Windows is... maybe need to fallback to os.walk to start with
all right, fixed a bunch of things here... #197
- the
argument of type 'WindowsPath' is not iterablething -- worked around via a whapper forsubprocessfunctions 8ec3a24 - traversing the files/grepping should work now 42e2604
- not sure how to configure
libmagicon Windows so just worked around it defensively 313fd25 - 6/75 tests are still failing, most have to do with that
Path.is_dirdoesn't like when path contains an asterisk, which globbing in HPI relies upon.. will need to swap these lines, will do later https://github.com/karlicoss/HPI/blob/6d9bc2964b24cfe6187945f4634940673dfe9c27/my/core/common.py#L160-L165
still haven't added static analysis for these, so without running tests on Windows some of it might go stale.. but hopefully I'll remember this for a while
karlicoss/HPI#124 .. and after this update all Promnesia tests pass (except test_example_config, which has some linux/macos specific paths -- not sure what's the best to do with it)
Have you considered just saying Linux support works only under WSL, @karlicoss ? With the reliance on fd, find, and other *nix utilities, it would probably be easier to just direct users to use WSL instead of writing new implementations. It works especially well with a library like this because at the end of the day, it's a CLI program + filesystem ingest + webserver, all of which are able to traverse the WSL boundary fairly easily.
It would be more overhead on the part of the users to install and configure WSL, but it's something that's documented in a lot of places and IMO was easy the last time I did it (I recently switched to Linux after about 10+ years of mainly Windows dev-ing)
Yeah, I guess I don't know much about it, so figured I'd give it a go to fix (last time I did anything on Windows was ~10 years ago). I guess worth mentioning in the docs that via WSL it would be much more likely to work properly!
So, is the WSL version working properly?
Update: It does! Strangely, there's some path issues so have to use python3 -m promnesia vs promnesia.
Oh that's good to know. Yeah issues like these can always happen, and on linux as well...
Generally running everything as python3.x -m is the safest thing to do, it's just a bit verbose
Recently setup a old thinkpad on windows -- will probably be testing HPI/promnesia on there eventually, will report/PR some docs if I have any issues
Edit: Had no issues setting it up in WSL
I have a (fairly) fresh Windows install and had no issues with Promnesia this go around!
Right in Windows, not via WSL.
- Installed Python 3.10.2 (Windows installer (64-bit)) via python.org
pip3 install --user promnesia- added a folder to PATH
promnesia demo https://github.com/karlicoss/exobrainseemed to run finepromnesia config create, added a local folder with .md files,promnesia indexpromnesia serve
Seems good 👍
Very cool! I've shifted back to Arch and haven't had the time to try it out yet.
I did a bunch of fixes, so pretty sure it should work both under WSL or regular shell (or whatever it's called in Windows world). It also runs on CI now so hopefully won't regress. Feel free to reopen/comment if you still have issues with Windows
- added a folder to PATH
What do you mean in "add a folder to PATH", sir?
I installed Python 3.10.7.
I run "pip3 install --user promnesia" and successfully.
but when I run "promnesia demo https://github.com/karlicoss/exobrain" as your suggestion, I received an error like this:
C:\Python3107>promnesia demo https://github.com/karlicoss/exobrain
'promnesia' is not recognized as an internal or external command,
operable program or batch file.
Do I make mistake in something, sir?
python promnesia since you installed it as a user package and not a global package.