Better documentation of _v_*
Closed this issue · 10 comments
In https://github.com/zopefoundation/ZODB/blob/master/doc/guide/writing-persistent-objects.rst#special-attributes The _v_
attribute is documented.
I never been sure how it works exactly. Now I had the need to look deeper into it. I always thought (for >15years), in a ZEO setup _v_
is local to the connection and does not get propagated to other clients, which is wrong. No idea where I got that from... but in fact the _v_
is propagated to the other connections/clients after commit. It is just not stored in the Data.fs as far as I understand.
So, lucky me, this made it simple for my case.
But the documentation is bit thin here. Maybe this needs to go in the ZEO documentation, because its ZEO special? How does RelStorage and other backends do here?
v is local to the connection and does not get propagated to other clients, which is wrong. No idea where I got that from... but in fact the v is propagated to the other connections/clients after commit.
I don't think that's right. Volatile attributes are tied to an object instance; an object instance is inherently tied to a process and (via its _p_jar
and the p_jar
's cache) a connection.
From the code in cPersistence.c, we can see that volatile attributes are never saved as part of an object's state. Since storages like RelStorage, FileStorage and ZEO's ClientStorage only ever deal with the serialized form of an object's state (store
, loadSerial
, loadBefore
all accept and return byte strings), there's no way for volatile attribute data to move to a storage and then get distributed to other clients of that storage.
Because ZODB connections can be used (in a serial fashion) from one thread and then another, it's possible to see volatile attributes that were set in one thread in a process appear in another thread in that process.
This was easy to test with two processes acting as ZEO clients and connecting to a ZEO server. If the first process sets _v_thing
on conn.root()
and commits:
# Process 1
>>> conf = """
...: <zodb zeofs>
...: <zeoclient>
...: server localhost:24003
...: </zeoclient>
...: </zodb>
...: """
>>> db = ZODB.config.databaseFromString(conf)
>>> conn = db.open()
>>> root = conn.root()
>>> root._v_thing = 42
>>> root['key'] = 64
>>> root.__dict__
{'data': {'key': 64}, '_v_thing': 42}
>>> transaction.commit()
Only the non-volatile attribute is visible to a second process (the same applies to a second database in the same process, or, as we'll see later, to a second connection to the same database in the same process):
# Process 2
>>> conf = """
...: <zodb zeofs>
...: <zeoclient>
...: server localhost:24003
...: </zeoclient>
...: </zodb>
...: """
>>> db = ZODB.config.databaseFromString(conf)
>>> conn = db.open()
>>> root = conn.root()
>>> root # unghost
{'key': 64}
>>> root.__dict__
{'data': {'key': 64}} # no volatile attribute
The same goes for a separate connection in the first process, even though the volatile attribute remains visible to the first connection.
# Process 1
>>> closed_conn = conn
>>> conn.close()
>>> conn2 = db.open()
>>> conn2 is closed_conn # Connections are LIFO; conn2 is the original conn
True
>>> conn3 = db.open()
>>> conn3 is closed_conn
False
>>> conn2.root().__dict__
{'data': {'key': 64}, '_v_thing': 42} # same object —> same volatile attribute
>>> conn3.root() # Unghost
{'key': 64}
>>> conn3.root().__dict__
{'data': {'key': 64}} # No volatile attribute
If we throw away the object instance state and force it to load from storage again, the volatile attribute will be gone:
>>> conn2.cacheMinimize()
>>> conn2.root()
{'key': 64}
>>> conn2.root().__dict__
{'data': {'key': 64}}
Yup, what @jamadden said. _v_
attributes are local to a specific active object in memory and thus to a specific connection.
I wouldn't object to calling this out in the documentation. :)
Strange, I got different results which led me to my statement above. I took two terminal sessions where I did pip install ZEO
(in a pyenv) which got me ZEO 5.2.1. In first terminal I start the ZEO and let it run:
(p5.2.0-py3.7) ~/ws/sandbox/volatiletest$ python
Python 3.7.5 (default, Dec 16 2019, 18:10:01)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ZEO
>>> address, stop = ZEO.server()
>>> address
('127.0.0.1', 32829)
>>>
In the second:
(p5.2.0-py3.7) ~/ws/sandbox/volatiletest$ python
Python 3.7.5 (default, Dec 16 2019, 18:10:01)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ZEO
>>> connection = ZEO.connection(('127.0.0.1', 32829))
>>> connection.root.foo = "bar"
>>> connection.root
<root: foo>
>>> connection.root._v_baz = 1
>>> connection.root
<root: _v_baz foo>
>>> transaction.commit()
>>> exit()
(p5.2.0-py3.7) ~/ws/sandbox/volatiletest$ python
Python 3.7.5 (default, Dec 16 2019, 18:10:01)
[GCC 9.2.1 20191008] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ZEO
>>> connection = ZEO.connection(('127.0.0.1', 32829))
>>> connection.root
<root: _v_baz foo>
>>> connection.root._v_baz
1
>>>
Given your reply above _v_baz
should not be there after exiting the interpreter and reconnecting to ZEO. What happens here?
connection.root
isn't a persistent object. :) It's actually a wrapper around the root object, which for hysterical reasons, is a PersistentMapping. When you set _v_baz
, you're actually setting a key, not an attribute.
One could be annoyed by this, :), but you're not supposed to do anything interesting or application-dependent with the root object. It's just a place to hang more interesting objects.
It would likely be nice if that wrapper provided some special handling of _v_
attributes. (I don't recall if it does anything special with _p_
attributes. )
Oh, indeed, that's the difference here. Thanks!
A last question for my understanding: I have a volatile attribute on an object on the first connection. Now on a second connection the object is modified and committed.
From my understanding the object on the first connection gets invalidated and thus removed from the cache?
Given I am right with this, the volatile attribute on the first connection gets deleted as well?
the volatile attribute on the first connection gets deleted as well?
Correct. Volatile attributes exist solely in memory on a specific object instance. Any time that instance is ghosted or otherwise invalidated, all volatile attributes vanish (that's why they went away on cacheMinimize()
).
connection.root
isn't a persistent object. :) It's actually a wrapper around the root object, which for hysterical reasons, is a PersistentMapping. When you set_v_baz
, you're actually setting a key, not an attribute.
I just found where I copied from to use connection.root
and not connection.root()
: In the documentation under https://github.com/zopefoundation/ZODB/blob/master/doc/guide/transactions-and-threading.rst there it is. Should this be corrected?
I just found where I copied from to use connection.root and not connection.root(): In the documentation … there it is. Should this be corrected?
IMO, yes. I would consider using .root()["k"] = v
to be the best practice, while .root.k = v
is a convenience reserved for REPL interactions and quick one-off scripts. But reasonable people can disagree.