Ability to purge deleted object nodes after a set time period

Question

Ability to purge deleted object nodes after a set time period

Opened this issue 7 months ago · 22 comments

Overview

If I remove everything from an actual .kdbx file (including recycle bin) and save it (or use save as backup), I notice that the file is not decreasing. There is no compact process so I suspect that everything that was deleted (as per the user request) is still present (and could be recovered?).

Steps to Reproduce

duplicate a kdbx file
open the new one and delete everything
empty and delete the recycle bin
Save or save database backup
check the size of the output

Expected Behavior

an empty database should not be 6Mb

Actual Behavior

file size stay the same

Context

steps followed to investigate another issue on keepass-rs

KeePassXC - 2.7.8
Revision: f6757d3

Operating System: Windows

Answer 1 · 2024-06-19T10:20:52.000Z

6 MB? Did you have a huge attachment on an entry?

Answer 2 · 2024-06-19T12:05:38.000Z

Go to settings, set history size to 0, save, then set it back to whatever it was.

Answer 3 · 2024-06-19T12:19:36.000Z

Technically clearing the recycle bin should wipe out the history since the entry nodes themselves are removed at that point.

Answer 4 · 2024-06-19T12:31:41.000Z

I just added a bunch of large attachments and custom icons to entries and deleted them (from the recycle bin) and the size went from 4 MB -> 60 KB. So you must have not actually deleted the items or you are looking at the wrong file.

Answer 5 · 2024-06-19T14:48:19.000Z

The file is 6,232,643 bytes. For me it is clearly empty (or maybe I am missing something)
I can't send you the file but let me know what I could share with you to solve it.

Answer 6 · 2024-06-19T15:28:03.000Z

Show the file properties sheet (from your file manger) for:

Answer 7 · 2024-06-19T19:06:28.000Z

is it what you need?

Answer 8 · 2024-06-19T19:17:59.000Z

Well that sure is interesting

Answer 9 · 2024-06-19T19:22:35.000Z

Do you have anything listed in previous versions? I'm not 100% sure, but that might be counting against your size.

Answer 10 · 2024-06-20T07:57:05.000Z

nothing in previous version; this file was obtained as a save as Backup but I had the same issue with regular save.

I tried to disable the compression and save as backup and the file is even larger (9Mb).
I don't feel confortable sending you the file but I may run a debug version of keepassxc (on windows or linux)?
I am able to replicate the issue on a Mac.

Answer 11 · 2024-06-20T08:03:27.000Z

Going back to the reason why I opened this issue (Narigo/keepass-diff#76); could it be that my file is "corrupted" with a silent error, and keepassxc is not purging the file due to a silent error or a bug in the recent version of the parser?

Answer 12 · 2024-06-20T10:28:16.000Z

My only thought is entry attachments, there must be one "stuck" somehow to a phantom entry. Attachments are very odd in the kdbx specification and there have been corruption points in the past with other programs, but that was usually attachments getting replaced randomly, not stuck.

Can you try export to xml to see if there are any entries dumped?

Answer 13 · 2024-06-20T11:51:03.000Z

Attachments should be rewritten entirely on save.

Answer 14 · 2024-06-20T13:15:22.000Z

so export in CSV returns nothing
export in XML returns a lot of (3/4 of the file):

			<DeletedObject>
				<UUID>eVV21DDsrEylbYmQ15p+8Q==</UUID>
				<DeletionTime>YZwE3g4AAAA=</DeletionTime>
			</DeletedObject>

the 1st part is about the icons.
I don't see anything related to the attachments.

Answer 15 · 2024-06-20T13:29:59.000Z

Deleted object lines are fully expected.

Answer 16 · 2024-06-20T13:46:19.000Z

The xml exported file of the empty kdbx is 9MB; not compressed the kdbx file is 9MB
The full DB exported as xml is 28MB. The kdbx not compressed is 29MB
The 2 files, compressed are about 6MB (same size).

Since I emptied the recycle bin, I would not have expected DeletedObject?
Is there an option to import the xml into a new keepassxc database? I wanted to compare the size of the database to see if there is anything else in the kdbx?
My understanding is that the keepass entries are removed from the kdbx file (maybe some headers stay), so no space saving is expected. Therefore, everything is fine? I let you decide if my point 1 is a concern or not.

Answer 17 · 2024-06-20T15:34:38.000Z

9MB of what? That is a LOT of XML. How many entries did you delete, over 1 million?

Answer 18 · 2024-06-21T09:37:27.000Z

I have 4700 entries (maybe I am an extreme user). With about 123 characters for each deleted line in XML, it is about 5.7Mb of XML for this section. There is a big part for the icons so it sounds right for me?

Answer 19 · 2024-06-21T10:07:32.000Z

4700*123=570k. Did you miss a zero somewhere?

Answer 20 · 2024-06-21T10:07:59.000Z

That still doesn't add up, but what happens when you go to Database -> Database Settings -> Maintenance tab. Then purge unused icons. I coupd buy you have 8MB of unused favicons.

Answer 21 · 2024-06-22T17:13:44.000Z

correct, I made a mistake with a 0 :(
So here are the new stats:

3190 icons, about 830 characters each (for the 1st one in the list) = 2.6MB, but the full XML section is 8.7MB. I identified some section that are 20K for one icon.
7225 deleted objects * 123 characters (I have some with 130 characters) = 1MB

So in term of size of the file, it all comes from the icons. :)
When I purge the icons from keepass-xc, the file is now 200KB
Would it be possible to purge the DeletedObject?

Answer 22 · 2024-06-22T17:34:30.000Z

No you cannot purge deleted objects, you would never want to do that in practice to all of them. When compressed those xml entries constitute a negligible size.

However, being able to clear deleted objects over a year old (for example) would be a nice maintenance item in the database settings. As long as there is little chance for those entries to be merged back in.