praktomat.wsgi incompatible with virtualenv 20.x
chris21k opened this issue · 11 comments
Ubuntu (20.04 LTS) upgraded virtualenv/python3-virtualenv from 15.1 to 20.x . This is a complete rewrite. Unfortunately, calling praktomat via wsgi and the newly created env does then not work any more. Maybe a new wsgi-file hast to be created which is aware of the new virtualenv stucture?
A workaround is to create an intermediate virtualenv where to install the old virtualenv package via pip and use this to create the virtualenv for praktomat:
virtualenv -p python3 --system-site-packages env-temp/
. env-temp/bin/activate
pip3 install virtualenv==15.1
virtualenv -p python3 --system-site-packages env/
deactivate
. env/bin/activate
pip install -r Praktomat/requirements.txt
With an virtualenv created directly by virtualenv 20.x Apache's error.log lists:
Thread 0x00007f41fc24a700 (most recent call first):
[Fri Jun 19 07:22:00.546651 2020] [wsgi:error] [pid 950:tid 139921025062656] [client 10.10.2.3:59133] Truncated or oversized response headers received from daemon process 'praktomat_prog2_20_SS': /srv/praktomat/prog2_20_SS/Praktomat/wsgi/praktomat.wsgi, referer: https://10.10.6.1/
Python path configuration:
PYTHONHOME = '/srv/praktomat/prog2_20_SS/env'
PYTHONPATH = (not set)
program name = 'python3'
isolated = 0
environment = 1
user site = 1
import site = 1
sys._base_executable = '/usr/bin/python3'
sys.base_prefix = '/srv/praktomat/prog2_20_SS/env'
sys.base_exec_prefix = '/srv/praktomat/prog2_20_SS/env'
sys.executable = '/usr/bin/python3'
sys.prefix = '/srv/praktomat/prog2_20_SS/env'
sys.exec_prefix = '/srv/praktomat/prog2_20_SS/env'
sys.path = [
'/srv/praktomat/prog2_20_SS/env/lib/python38.zip',
'/srv/praktomat/prog2_20_SS/env/lib/python3.8',
'/srv/praktomat/prog2_20_SS/env/lib/python3.8/lib-dynload',
]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: initialized
Thread 0x00007f41fc24a700 (most recent call first):
[Fri Jun 19 07:41:49.918278 2020] [wsgi:error] [pid 949:tid 139921117382400] [client 10.10.2.3:59227] Truncated or oversized response headers received from daemon process 'praktomat_prog2_20_SS': /srv/praktomat/prog2_20_SS/Praktomat/wsgi/praktomat.wsgi, referer: https://10.10.6.1/prog2_20_SS/tasks/
Do you have python 2 related Praktomat versions running on the same apache server?
@ifrh Thanks, for the hint, but I have no python2 (or services based on python2) on this machine:
$ python
Command 'python' not found, did you mean:
command 'python3' from deb python3
command 'python' from deb python-is-python3
$ python2
Command 'python2' not found, but can be installed with:
sudo apt install python2
The described problem continous to exists with meanwhile python 3.8.6 and python3-virtualenv 20.0.29+ds-1 .
which wsgi module for i.e. apache have you installed on your server?
libapache2-mod-wsgi (for using with Python2)
libapache2-mod-wsgi-py3 (for using with Python3)
@chris21k Have you tried to start on the servers console using the virtual-env from 20.x
pip freeze
python --version && python -c "import sqlite3; print(\"... uses pysqlite \" + sqlite3.version +\" with SQLite \" + sqlite3.sqlite_version);"
./src/manage-test.py test accounts attestation checker configuration solutions tasks
What is the output?
With an virtualenv created directly by virtualenv 20.x Apache's error.log lists:
Thread 0x00007f41fc24a700 (most recent call first):
[Fri Jun 19 07:22:00.546651 2020] [wsgi:error] [pid 950:tid 139921025062656] [client 10.10.2.3:59133] Truncated or oversized response headers received from daemon process 'praktomat_prog2_20_SS': /srv/praktomat/prog2_20_SS/Praktomat/wsgi/praktomat.wsgi, referer: https://10.10.6.1/
[...]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
What is the output inside servers console running with activated virtual-env from 20.x
python -c 'import sys; print(sys.getfilesystemencoding())'
just a next guess your server locales are not completely configured:
I need to configure a local testsystem inside a docker with
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y locales
RUN sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
dpkg-reconfigure --frontend=noninteractive locales && \
update-locale LANG=en_US.UTF-8
ENV export LANG en_US.UTF-8
ENV export LANGUAGE en_US:en
ENV export LC_ALL en_US.UTF-8
a small python script to show configuration of encodings and locales
import sys
import locale
print("sys:defaultencoding: " , sys.getdefaultencoding())
print("sys:filesystemencoding: " ,sys.getfilesystemencoding())
print("locale:preferredencoding: " , locale.getpreferredencoding())
print("locale:defaultlocale ",locale.getdefaultlocale())
print("locale:locale ",locale.getlocale())
Thanks for your help and sorry for the late replay.
which wsgi module for i.e. apache have you installed on your server?
$ dpkg -l | grep libapache2-mod-wsgi
ii libapache2-mod-wsgi-py3 4.7.1-3 amd64 Python 3 WSGI adapter module for Apache
Have you tried to start on the servers console using the virtual-env from 20.x
$ virtualenv -p python3 --system-site-packages env/
$ . env/bin/activate
$ pip freeze
appdirs==1.4.4
blinker==1.4
certifi==2020.6.20
chardet==3.0.4
command-not-found==0.3
cryptography==3.2.1
dbus-python==1.2.16
distlib==0.3.1
distro==1.5.0
distro-info===0.24ubuntu2
Django==2.2.17
django-debug-toolbar==3.1.1
django-extensions==3.0.9
django-tinymce==3.1.0
docutils==0.16
filelock==3.0.12
httplib2==0.18.1
idna==2.10
importlib-metadata==1.6.0
jeepney==0.5.0
keyring==21.5.0
keyrings.alt==4.0.1
language-selector==0.1
launchpadlib==1.10.13
lazr.restfulclient==0.14.2
lazr.uri==1.0.5
Markdown==3.3.3
more-itertools==4.2.0
netifaces==0.10.4
oauthlib==3.1.0
psycopg2==2.8.5
psycopg2-binary==2.8.6
pycrypto==2.6.1
pycryptodomex==3.9.7
pycurl==7.43.0.2
Pygments==2.7.2
PyGObject==3.38.0
PyJWT==1.7.1
python-apt==2.1.3+ubuntu3
python-debian==0.1.37
pytz==2020.4
pyxdg==0.26
PyYAML==5.3.1
requests==2.24.0
requests-unixsocket==0.2.0
SecretStorage==3.2.0
simplejson==3.17.0
six==1.15.0
sqlparse==0.4.1
ssh-import-id==5.10
ubuntu-advantage-tools==24.4
ufw==0.36
unattended-upgrades==0.1
urllib3==1.25.9
virtualenv==20.0.29+ds
wadllib==1.3.4
zipp==1.0.0
$ python --version && python -c "import sqlite3; print(\"... uses pysqlite \" + sqlite3.version +\" with SQLite \" + sqlite3.sqlite_version);"
Python 3.8.6
... uses pysqlite 2.6.0 with SQLite 3.33.0
$ /srv/praktomat/prog2_20_SS/Praktomat/src/manage-test.py test accounts attestation checker configuration solutions tasks
Creating test database for alias 'default'...
System check identified no issues (0 silenced).
.............................EF...FsF................
======================================================================
ERROR: test_r_checker (checker.tests.TestChecker)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/praktomat/prog2_20_SS/Praktomat/src/checker/tests.py", line 316, in test_r_checker
self.solution.check_solution()
File "/srv/praktomat/prog2_20_SS/Praktomat/src/solutions/models.py", line 76, in check_solution
check_solution(self, run_secret, debug_keep_tmp)
File "/srv/praktomat/prog2_20_SS/Praktomat/src/checker/basemodels.py", line 299, in check_solution
run_checks(solution, env, run_all)
File "/srv/praktomat/prog2_20_SS/Praktomat/src/checker/basemodels.py", line 360, in run_checks
result = checker.run(env)
File "/srv/praktomat/prog2_20_SS/Praktomat/src/checker/checker/RChecker.py", line 79, in run
(output, error, exitcode, timed_out, oom_ed) = execute_arglist(
File "/srv/praktomat/prog2_20_SS/Praktomat/src/utilities/safeexec.py", line 73, in execute_arglist
process = subprocess.Popen(
File "/usr/lib/python3.8/subprocess.py", line 854, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1702, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'Rscript'
======================================================================
FAIL: test_r_checker_2 (checker.tests.TestChecker)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/praktomat/prog2_20_SS/Praktomat/src/checker/tests.py", line 342, in test_r_checker_2
self.assertIn('2', checkerresult.log, "Test did not calculate 1 + 2 (%s)" % checkerresult.log)
AssertionError: '2' not found in '<p>Could not find expected R script example.R.</p><p>R scripts found: example_Jau5NGg.R</p>' : Test did not calculate 1 + 2 (<p>Could not find expected R script example.R.</p><p>R scripts found: example_Jau5NGg.R</p>)
======================================================================
FAIL: test_script_filesizelimit (checker.tests.TestChecker)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/praktomat/prog2_20_SS/Praktomat/src/checker/tests.py", line 208, in test_script_filesizelimit
self.assertNotIn('End', checkerresult.log, "Test did finish (no timeout?) (%s)" % checkerresult.log)
AssertionError: 'End' unexpectedly found in '<pre>Begin\nEnd\n</pre>' : Test did finish (no timeout?) (<pre>Begin
End
</pre>)
======================================================================
FAIL: test_script_timeout (checker.tests.TestChecker)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/srv/praktomat/prog2_20_SS/Praktomat/src/checker/tests.py", line 189, in test_script_timeout
self.assertNotIn('done', checkerresult.log, "Test did finish (no timeout?)")
AssertionError: 'done' unexpectedly found in '<div class="error">Timeout occured!</div><pre>0\n1\n2\n3\n4\n5\n6\n7\n8\n9\n10\n11\n12\n13\n14\n15\n16\n17\n18\n19\n20\n21\n22\n23\n24\n25\n26\n27\n28\n29\n30\ndone\n</pre>' : Test did finish (no timeout?)
----------------------------------------------------------------------
Ran 53 tests in 14.263s
FAILED (failures=3, errors=1, skipped=1)
Destroying test database for alias 'default'...
What is the output inside servers console running with activated virtual-env from 20.x?
$ python -c 'import sys; print(sys.getfilesystemencoding())'
utf-8
$ dpkg -l | grep locales
ii krb5-locales 1.17-10 all internationalization support for MIT Kerberos
ii locales 2.32-0ubuntu3 all GNU C Library: National Language (locale) data [support]
ii python-apt-common 2.1.3ubuntu3 all Python interface to libapt-pkg (locales)
$ env | grep LANG
LANGUAGE=en_US:en
LANG=en_US.UTF-8
(Output of) a small python script to show configuration of encodings and locales
$ ./show_locales.py
sys:defaultencoding: utf-8
sys:filesystemencoding: utf-8
locale:preferredencoding: UTF-8
locale:defaultlocale ('en_US', 'UTF-8')
locale:locale ('en_US', 'UTF-8')
$ env | grep LANG
LANGUAGE=en_US:en
LANG=en_US.UTF-8
$ cat /etc/locale.gen
# ...
de_DE.UTF-8 UTF-8
# ...
en_US.UTF-8 UTF-8
I need to configure a local testsystem inside a docker with...
$ dpkg-reconfigure locales
This single command helped after a restart of apache.
However, I am quite suprised why this helps. As far as I can see, this command runs on every (automatic) system apt-get upgrade where locales etc. have new versions, and this is quite often the case. And according to the above, I have no misconfigurations, right? And why did it work with the old virtualenv?
@ifrh MANY THANKS!
Chris
/srv/praktomat/prog2_20_SS/Praktomat/src/manage-test.py test accounts attestation checker configuration solutions tasks
gave you some Unittest errors. I think, that R was not installed, or not fully configured.
If that messages are gone now, than (maybe) because dpkg-reconfigure :-)
@ifrh Yes, R is not is not installed by intention, we do only Java on Praktomat. So the error messages about a missing R are "normal". dpkg-reconfigure eliminated the virtualenv problem, however, I cannot fully figure out why.
Just for the record:
in standard configuration on Debian / Ubuntu Apache is running WSGI-Moduls with LANG=C as environment.
if settings in /etc/apache2/envvars changed to UTF-8, than using non-ascii filenames i.e. in solutions are working.
see last point in django bug-report https://code.djangoproject.com/ticket/6009#comment:18