Error creating MongoClient in penguindome/server.py
Closed this issue · 6 comments
Hi,
I'm a bit confused, I had to tweak penguindome/server.py in order to create a MongoClient object to connect to a mongodb test database.
I know a bit of python, and only played with mongodb 10 years ago ... so I may be saying stupid things, sorry for that.
I started a local mongodb 7.0.4 server.
I met following error :
File "/home/xxxxxx/src/penguindome/src/PenguinDome/penguindome/server.py", line 131, in get_db
newdb.authenticate(username, password)
TypeError: 'Collection' object is not callable. If you meant to call the 'authenticate' method on a 'Database' object it is failing because no such method exists.
From what I read it look s like it's related to dprecations in PyMongo lib : https://pymongo.readthedocs.io/en/stable/migrate-to-pymongo4.html#database-authenticate-and-database-logout-are-removed
Database.authenticate and Database.logout are removed
From the 'requirement.txt' I found in the project, PyMongo in use in venv is 4.5 so it makes sense.
I tried to fix locally with the changes below, which are absolutely not backward compatible and probably utterly fragile regarding the configuration file structure, I only considered my test case :
--- a/penguindome/server.py
+++ b/penguindome/server.py
@@ -109,27 +109,37 @@ def get_db(force_db=None):
db = force_db
return db
- database_name = get_setting('database:name')
- host = get_setting('database:host')
+ database_name = get_setting('database:name', 'penguindome')
+ hostport = get_setting('database:host', ['localhost:27017'])
+ if isinstance(hostport, str):
+ host = hostport.split(':')[0]
+ port = hostport.split(':')[1]
+ else:
+ hosts=[]
+ for h in hostport:
+ hosts.append(h.split(':')[0])
+ port = int(h.split(':')[1])
+ # TODO manage multi port values I have no idea how ?!
+ host=', '.join(hosts)
+ replicaset = get_setting('database:replicaset')
+ username = get_setting('database:username')
+ password = get_setting('database:password')
if not host:
connection = MongoClient()
else:
- if not isinstance(host, str):
- host = ','.join(host)
- kwargs = {}
- replicaset = get_setting('database:replicaset')
+ kwargs = {
+ 'host':host,
+ 'port':port,
+ 'username': username,
+ 'password': password
+ }
+
if replicaset:
kwargs['replicaset'] = replicaset
- connection = MongoClient(host, **kwargs)
-
+ connection = MongoClient(**kwargs)
newdb = connection[database_name]
-
- username = get_setting('database:username')
- if username:
- password = get_setting('database:password')
- newdb.authenticate(username, password)
-Which reads better like this :
def get_db(force_db=None):
global db, pid
if force_db is None and db is not None and os.getpid() == pid:
return db
pid = os.getpid()
if force_db is not None:
db = force_db
return db
database_name = get_setting('database:name', 'penguindome')
hostport = get_setting('database:host', ['localhost:27017'])
if isinstance(hostport, str):
host = hostport.split(':')[0]
port = hostport.split(':')[1]
else:
hosts=[]
for h in hostport:
hosts.append(h.split(':')[0])
port = int(h.split(':')[1])
# TODO manage multi port values I have no idea how ?!
host=', '.join(hosts)
replicaset = get_setting('database:replicaset')
username = get_setting('database:username')
password = get_setting('database:password')
if not host:
connection = MongoClient()
else:
kwargs = {
'host':host,
'port':port,
'username': username,
'password': password
}
if replicaset:
kwargs['replicaset'] = replicaset
connection = MongoClient(**kwargs)
newdb = connection[database_name]
db = MongoProxy(newdb)
return dbI wonder why I face the issue to begin with, I'm pretty sure I followed instructions.
Any advice will be welcome !
thank you
The code is compatible with the version of pymongo specified in server/requirements.txt. You appear to be using a newer version of pymongo. Most likely you are invoking the server incorrectly so you aren't running it inside the server virtualenv created by the setup script and as a result you are using the version of pymongo installed in the OS instead of the version installed in the virtualenv, though that's just a guess. You haven't provided any details about how you set up the server or how you're invoking it ("I'm pretty sure I followed instructions" is not a particularly detailed description of what you did), so I can't begin to guess how you're ending up with the server using the wrong version of pymongo.
I can't make the code compatible with the current version of pymongo at this time, because MongoDBProxy isn't compatible with the current version of pymongo and I don't have time to fix MongoDBProxy.
Having said that, I did just push several commits with some non-functional code cleanup as well as several functional changes:
- Upgrade to requests 2.32.3 for a security fix.
- Upgrade to pymongo 4.6.3 (which remains compatible with the PenguinDome code) for a security fix.
- Switch to a patched version of MongoDBProxy which installs successfully.
- Switch to a patched version of stopit which is compatible with Python 3.12.
- Replace use of the deprecated
distutils.versionwithpackaging.version. - Replace deprecated use of
datetime.datetime.utcnow()withdatetime.datetime.now(datetime.timezone.utc).
Hope this helps.
Thank you for the detailed clarification, I'll investigate what's wrong. At least I know it definitely should work.
Being not that experienced in python, allow me to detail things, maybe it will ring a bell ? (or just forget about that, you're not supposed to do python teaching :) )
I run /bin/server
which sources the virtualenv that was built.
cat ./bin/server
#!/bin/bash -e
. "/home/xxxxx/src/PenguinDome/var/server-venv/bin/activate"
exec python "/home/xxxxx/src/PenguinDome/server/server.py" "$@"
the error I get comes from the 'right' virtualenv
File "/home/xxxxx/src/PenguinDome/var/server-venv/lib/python3.10/site-packages/pymongo/collection.py", line 3503, in __call__
raise TypeError(
TypeError: 'Collection' object is not callable. If you meant to call the 'authenticate' method on a 'Database' object it is failing because no such method exists.
and pymongo in this case shows as 4.5.0 (which should not be concerned with the issue I have, and which is the version in requirements.txt)
find src/PenguinDome | grep pymongo
src/PenguinDome/var/server-venv/lib/python3.10/site-packages/pymongo-4.5.0.dist-info
src/PenguinDome/var/server-venv/lib/python3.10/site-packages/pymongo-4.5.0.dist-info/INSTALLER
src/PenguinDome/var/server-venv/lib/python3.10/site-packages/pymongo-4.5.0.dist-info/LICENSE
...
I'd better start from scratch anyway :)
Started from scratch, here's what I did, and got the same result.
And venv contains the "good" pymongo version , now 4.6.3
I'm clueless.
10202 28.9.2024 11:21 git clone https://github.com/quantopian/PenguinDome.git
10203 28.9.2024 11:21 cd PenguinDome
10205 28.9.2024 11:21 ./server/server-setup.sh
10206 28.9.2024 11:23 cat ./bin/server
10207 28.9.2024 11:23 ./bin/server
and here's the output
./server/server-setup.sh
created virtual environment CPython3.10.12.final.0-64 in 128ms
creator CPython3Posix(dest=/home/xxxxx/srcAlt/PenguinDome/var/server-venv, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/xxxxx/.local/share/virtualenv)
added seed packages: pip==24.2, setuptools==74.0.0, wheel==0.44.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
If this takes a long time to run, you may want to install haveged or
some other tool for adding entropy to the kernel.
Do you want to configure things interactively? [y]
What port should the server listen on? [80]
What local-only port should the server use? [5000]
Do you want the server to use SSL? [n]
Database host:port: localhost:27017
Database host:port: [hit Enter when finished]
Replicaset name:
Database name: [penguindome]
Database username: penguin
Database password: password
Server logbook handler (e.g., stderr, syslog) (empty for none): [stderr] [enter "none" to clear]
Server logging level (e.g., debug, info): [debug]
Do you want to enable the audit cron job? [n]
URL base, e.g., http://hostname, for clients to reach server: http://localhost
Google geolocation API key, if any:
How often (minutes) do you want to collect data? [5]
How often (minutes) do you want re-try submits? [1]
Client logbook handler (e.g., stderr, syslog) (empty for none): [stderr] [enter "none" to clear]
Client logging level (e.g., debug, info): [debug]
Saved server settings.
Saved client settings.
Do you want to add the server to systemd? [y] n
Do you want to build a release with the new client settings? [y]
Done!
./bin/server
Traceback (most recent call last):
File "/home/xxxxx/srcAlt/PenguinDome/server/server.py", line 1012, in <module>
main()
File "/home/xxxxx/srcAlt/PenguinDome/server/server.py", line 971, in main
prepare_database()
File "/home/xxxxx/srcAlt/PenguinDome/server/server.py", line 915, in prepare_database
db = get_db()
File "/home/xxxxx/srcAlt/PenguinDome/penguindome/server.py", line 131, in get_db
newdb.authenticate(username, password)
File "/home/xxxxx/srcAlt/PenguinDome/var/server-venv/lib/python3.10/site-packages/pymongo/collection.py", line 3501, in __call__
raise TypeError(
TypeError: 'Collection' object is not callable. If you meant to call the 'authenticate' method on a 'Database' object it is failing because no such method exists.
OK, you've convinced me this is my fault. ;-)
Investigating further.
OK, this is fixed. My apologies for doubting you, I didn't run into this when I tested because it only happens when you use a username and password for the MongoDB connection (which, in retrospect, is obvious!) and I forgot to test that.
While I was in the process of fixing this I upgraded all the Python packages to their current versions.
Don't apologize ! And thank you !!
I was convinced I did something wrong to be honest, and could not see what :)
Works like a charm now !