Media upload errors and general guidance
Closed this issue · 16 comments
So, I just created a Grampshub account, imported my Gramps export, and then zipped all of my images (from within the media path, so the ZIP will just have a flat list of images) and uploaded that, but none of the images were showing even though there were no errors and I refreshed the page a dozen times.
All of the image references in the original GEDCOM (from which I created my Gramps DB) are not path-qualified (filenames only) and my Gramps media path points to the image path, which is a big, flat directory of files (similar to the ZIP file I created above). The images load as they should from within Gramps, and I previously wrote and regularly run a tool that ensures that all of the images referenced by the source GEDCOM file match the media path whenever I change anything or bring in updates from elsewhere.
Next I installed Gramps Web Sync in my Gramps and ran a sync. It said there were no tree changes, but that every image existed on local and every image was missing from remote. It then took five minutes for the media-sync and then eventually said that every upload failed:
(There were 186 images in total, all of which were missing.)
There was no further verbosity given in the UI, and no errors were emitted by the plugin in the terminal verbosity.
What do I do? I've read what few sites came up from some Google searches, checked the Gramps documentation, checked Grampshub, checked the Gramps Web documentation, and reviewed the project (Github) site before posting, but there's no mention of upload errors or how to resolve them.
I could use some help. I have a presentation of all of this coming up in a day or two. It'd be meaningfully impactful if the images were to be present. I was hoping for the images to be working in order to sell the architecture as well as the face detection.
Hi,
thanks for the detailed report. I just looked in the server logs and am seeing a lot of 409s on PUT /api/media. This happens when the checksum of the uploaded file does not match the Gramps media object's checksum. Can you please check whether that is the case?
See here for some background: https://www.grampsweb.org/user-guide/sync/
Have you run check & repair before exporting your XML? This is strongly recommended (see https://www.grampsweb.org/user-guide/import/) and should also fix the checksums.
If your checksums are correct, we can investigate further.
It looks like the checksums were originally missing, which seems like a bug.
If you could tell me who to tag or where to report it, I can do that.
Upon running the sync, again, it found a bunch of unexpected changes, included a huge number of people, which makes me uncomfortable. I only just recreated this DB from the GEDCOM, which is version-controlled. I understand how there might be small abnormalities that might be normalized-out by the check-and-repair process, but I would expect the appearance or disappearance of person records.
Since I don't think we can guarantee the consistent assignment of certain IDs to certain personals from my GEDCOM import to the next, I can't really investigate that much further.
I went ahead and applied the changes. It still complained of the same number of errors.
I then reran the process, and there were still changes to apply:
There were also still missing remote images (presumably 186 of them):
Sorry, I'm confused. You cannot sync with a database created from GEDCOM. The sync relies on the database handles being the same, so you can only sync with a remote database sourced from an XML exported by the local Gramps.
Also, why did you go ahead and apply the changes if things were obviously wrong?
I suggest you do the following now:
- Restore your local Gramps to the state it is supposed to be in (if not already done)
- Follow "Prepare your database" here https://www.grampsweb.org/user-guide/import/
- Make sure your Gramps objects have checksums now
- Install the latest version of the Sync Addon from this repo (if not already done)
- In the "Summary" page, choose "reset remote to local" and apply.
Seems like a buggy process. I created a brand new Gramps DB, imported from the GEDCOM, synced to Grampshub using the gramps export, and then used the web-sync and there were zero net changes. This was the first time. I then ran the check-and-repair tool on local (when you recommended it), ran another sync, using the tool, to Grampshub, and now there were a bunch of general changes as well as removed persons. So, we're not talking about GEDCOMs at this point. We're talking about a sync via the add-on.
Well, we're trying to figure things out at the moment. Since the whole point of the error-checking tool is to fix inconsistencies and I have no choice but to assume it's a stable tool, and differences that were then found by the web-sync would have to be reasonable and not the fault of the sync tool itself. I can try to reproduce those discrepancies later today and follow-up on some of them.
I'm less concerned about the integrity of data in my local Gramps and the integrity of the data in my Grampshub account than I am about getting more familiar with this process and its eccentricities. Like I said, my data is version-controlled (as both a GEDCOM, sorted GEDCOM, and a gramps file), and I regularly reconstruct my working Gramps DB. We can also clear and reset my Grampshub tree later if it gets royally screwed-up. It's a new account and probably not the first time it'll get screwed-up while we work through this.
#20 (comment)
<- Will do.
Ok, I see. I agree it's a buggy process. Thanks for all the input - not only about what does not work, but also about what can go wrong in the workflow. The outcome of this hopefully includes a clearer documentation, and perhaps a few warnings in various UI components.
One of the most important is that check and repair is run first, before any importing or syncing.
Do you still have access to the XMLs before and after the check & repair? Perhaps it can shed some light on why it would want to delete 2000 people.
If you can share the raw XML of one of the to-be-deleted people, like I0000, that could be useful.
I'll recreate the DB right now and compare those exports. Can I assume that the nodes in the XML probably aren't sorted?
I sort-of assumed that the verbiage was less than accurate, as there should only be about 650 people.
..and no problem. I love data networks, especially when the fruits of our labor can help fortify each others' family trees. I'm not always the most available person, but I'm happy to help.
So, I created a new DB, imported the GED, set the media-path, exported an uncompressed gramps file, committed, and then ran the check-and-repair tool. Just a couple outright inconsequential changes. This was the case a couple of times when I ran it, yesterday, but then the one time was much more dramatic even though I hadn't made any changes. In as much as this is a "sync" process, I'm guessing that the sync tool was responsible for these.
In addition to the couple of minor changes mentioned above, the media nodes got checksums: The pre-apply breakdown didn't mention these, but probably should.
The delta isn't so bad, so I'm including it here. Note that I've removed most of the media adjustments since it was highly consistent and wouldn't add further value.
diff --git a/kander.gramps.uncompressed b/kander.gramps.uncompressed
index 87fde03..89e003b 100644
--- a/kander.gramps.uncompressed
+++ b/kander.gramps.uncompressed
@@ -125,8 +125,9 @@
<event handle="_f644e8e487659927b6809b9dd7" change="1692348909" id="E0031">
<type>Death</type>
</event>
- <event handle="_f644e8e487f45c46a175b1e3d51" change="1692348909" id="E0032">
+ <event handle="_f644e8e487f45c46a175b1e3d51" change="1692349469" id="E0032">
<type>Birth</type>
+ <noteref hlink="_f644ee39b3d26df0a6755388733"/>
</event>
<event handle="_f644e8e4881380a15de1c5f3d51" change="1692348909" id="E0033">
<type>Death</type>
@@ -1641,8 +1642,9 @@
<type>Death</type>
<dateval val="1981"/>
</event>
- <event handle="_f644e8e550d4894b43aebf1e8c" change="1692348909" id="E0475">
- <type>Birth</type>
+ <event handle="_f644e8e550d4894b43aebf1e8c" change="1692349469" id="E0475">
+ <type>Unknown</type>
+ <noteref hlink="_f644ee39b3d26df0a6755388733"/>
</event>
<event handle="_f644e8e553326380adf38a06acd" change="1692348909" id="E0476">
<type>Birth</type>
@@ -1734,8 +1736,9 @@
<type>Birth</type>
<dateval val="1946-07"/>
</event>
- <event handle="_f644e8e560e5fcc363992619f18" change="1692348909" id="E0500">
- <type>Birth</type>
+ <event handle="_f644e8e560e5fcc363992619f18" change="1692349469" id="E0500">
+ <type>Unknown</type>
+ <noteref hlink="_f644ee39b3d26df0a6755388733"/>
</event>
<event handle="_f644e8e56117ccedf4a414f57d6" change="1692348909" id="E0501">
<type>Death</type>
@@ -1788,8 +1791,9 @@
<type>Birth</type>
<dateval val="1991-12-11"/>
</event>
- <event handle="_f644e8e56a6709c56ba51dc4afa" change="1692348909" id="E0515">
- <type>Birth</type>
+ <event handle="_f644e8e56a6709c56ba51dc4afa" change="1692349469" id="E0515">
+ <type>Unknown</type>
+ <noteref hlink="_f644ee39b3d26df0a6755388733"/>
</event>
<event handle="_f644e8e56a8279444da60fc4271" change="1692348909" id="E0516">
<type>Death</type>
@@ -2332,8 +2336,9 @@
<type>Birth</type>
<dateval val="1982-05-13"/>
</event>
- <event handle="_f644e8e5c612571267817f5d508" change="1692348909" id="E0669">
- <type>Birth</type>
+ <event handle="_f644e8e5c612571267817f5d508" change="1692349469" id="E0669">
+ <type>Unknown</type>
+ <noteref hlink="_f644ee39b3d26df0a6755388733"/>
</event>
<event handle="_f644e8e5c636bd5ca874b22e7bc" change="1692348909" id="E0670">
<type>Death</type>
@@ -2383,8 +2388,9 @@
<type>Birth</type>
<dateval val="1889"/>
</event>
- <event handle="_f644e8e5d3a73632c225bb4b05c" change="1692348909" id="E0684">
- <type>Birth</type>
+ <event handle="_f644e8e5d3a73632c225bb4b05c" change="1692349469" id="E0684">
+ <type>Unknown</type>
+ <noteref hlink="_f644ee39b3d26df0a6755388733"/>
</event>
<event handle="_f644e8e5d3c6557bf4d858caedb" change="1692348909" id="E0685">
<type>Death</type>
@@ -10391,10 +10397,10 @@
<childref hlink="_f644e8e4ff94f81c90fdbcf3da7"/>
<childref hlink="_f644e8e5f9945ad08edce17ec58"/>
</family>
- <family handle="_f644e8e498912cc34e5943e3c13" change="1692348909" id="F0208">
+ <family handle="_f644e8e498912cc34e5943e3c13" change="1692349468" id="F0208">
<rel type="Unknown"/>
- <father hlink="_f644e8e6264c99ae8c2ed56aa3"/>
- <mother hlink="_f644e8e62714862341173f5dbe0"/>
+ <father hlink="_f644e8e62714862341173f5dbe0"/>
+ <mother hlink="_f644e8e6264c99ae8c2ed56aa3"/>
<childref hlink="_f644e8e495c14b4a4327631cde3"/>
</family>
<family handle="_f644e8e499a2acae37ca5feea18" change="1692348909" id="F0018">
@@ -11852,26 +11858,26 @@
</placeobj>
</places>
<objects>
- <object handle="_f644e8e477d2f544553698b36d5" change="1692348909" id="O0000">
- <file src="Muttel Kander.jpg" mime="image/jpeg" description="Muttel Kander.jpg"/>
+ <object handle="_f644e8e477d2f544553698b36d5" change="1692349469" id="O0000">
+ <file src="Muttel Kander.jpg" mime="image/jpeg" checksum="cc4a27bce9425527eadd3bea87ad0ddb" description="Muttel Kander.jpg"/>
</object>
- <object handle="_f644e8e47eb71ede5beacec8ef1" change="1692348909" id="O0001">
- <file src="MoisheKander.jpeg" mime="image/jpeg" description="MoisheKander.jpeg"/>
+ <object handle="_f644e8e47eb71ede5beacec8ef1" change="1692349469" id="O0001">
+ <file src="MoisheKander.jpeg" mime="image/jpeg" checksum="e80c197a4dff282e753ae8a4a0a58d58" description="MoisheKander.jpeg"/>
</object>
- <object handle="_f644e8e482220fc8a8dad0abd24" change="1692348909" id="O0002">
- <file src="VictorHindaIdaKander.jpg" mime="image/jpeg" description="VIctor, Ida and Hattie Kander"/>
+ <object handle="_f644e8e482220fc8a8dad0abd24" change="1692349469" id="O0002">
+ <file src="VictorHindaIdaKander.jpg" mime="image/jpeg" checksum="4f7710ea15579877731ff2abfa312333" description="VIctor, Ida and Hattie Kander"/>
</object>
- <object handle="_f644e8e4827382251680225823b" change="1692348909" id="O0003">
- <file src="AnneLevitin.jpg" mime="image/jpeg" description="Anne Kander Levitin and Hattie Kander"/>
+ <object handle="_f644e8e4827382251680225823b" change="1692349469" id="O0003">
+ <file src="AnneLevitin.jpg" mime="image/jpeg" checksum="954c254f538350768ae309a07f7f6185" description="Anne Kander Levitin and Hattie Kander"/>
</object>
- <object handle="_f644e8e482d4716194d09b2d9e7" change="1692348909" id="O0004">
- <file src="VictorHelen.jpg" mime="image/jpeg" description="Victor and Helen Kander"/>
+ <object handle="_f644e8e482d4716194d09b2d9e7" change="1692349469" id="O0004">
+ <file src="VictorHelen.jpg" mime="image/jpeg" checksum="f7efb268b08da88ecdcc116fc7f969c6" description="Victor and Helen Kander"/>
</object>
- <object handle="_f644e8e48326f2135b75b01c32f" change="1692348909" id="O0005">
- <file src="VictorHinda.jpg" mime="image/jpeg" description="VictorHinda.jpg"/>
+ <object handle="_f644e8e48326f2135b75b01c32f" change="1692349469" id="O0005">
+ <file src="VictorHinda.jpg" mime="image/jpeg" checksum="a7254fa4dd62e21f7d8ef40ed1f4e515" description="VictorHinda.jpg"/>
</object>
- <object handle="_f644e8e493f43d9a90730e1ed1" change="1692348910" id="O0006">
- <file src="Kowalsky2.jpeg" mime="image/jpeg" description="Kowalsky2.jpeg"/>
+ <object handle="_f644e8e493f43d9a90730e1ed1" change="1692349469" id="O0006">
+ <file src="Kowalsky2.jpeg" mime="image/jpeg" checksum="e732c654f7d1cc1096065ae83cba0058" description="Kowalsky2.jpeg"/>
<noteref hlink="_f644e8e493a60482854716d1599"/>
<noteref hlink="_f644e8e497d1412027d205ea19f"/>
<noteref hlink="_f644e8e49943cab66d373baa15c"/>
@@ -11880,608 +11886,608 @@
<noteref hlink="_f644e8e6268698d2674bde0a672"/>
<noteref hlink="_f644e8e62781dadc3bafe2c0aff"/>
</object>
- <object handle="_f644e8e4d8060dee90b2a5d9def" change="1692348909" id="O0020">
- <file src="DavidLawson.png" mime="image/png" description="DavidLawson.png"/>
+ <object handle="_f644e8e4d8060dee90b2a5d9def" change="1692349469" id="O0020">
+ <file src="DavidLawson.png" mime="image/png" checksum="0b20c5758196147aeababd42f0c8b7f8" description="DavidLawson.png"/>
</object>
- <object handle="_f644e8e4d923560e5a606d920e6" change="1692348909" id="O0021">
- <file src="Ellen Levin.png" mime="image/png" description="Ellen Levin.png"/>
+ <object handle="_f644e8e4d923560e5a606d920e6" change="1692349469" id="O0021">
+ <file src="Ellen Levin.png" mime="image/png" checksum="aac8568df23d3cf41400572ad395c14b" description="Ellen Levin.png"/>
</object>
(...)
- <object handle="_f644e8e63d56d44244795e88ae8" change="1692348910" id="O0184">
- <file src="Jonathan Lawson.png" mime="image/png" description="Jonathan Lawson.png"/>
+ <object handle="_f644e8e63d56d44244795e88ae8" change="1692349469" id="O0184">
+ <file src="Jonathan Lawson.png" mime="image/png" checksum="c368701fea119455174ceb66cd57ff8c" description="Jonathan Lawson.png"/>
</object>
- <object handle="_f644e8e63e69971d32da553640" change="1692348910" id="O0185">
- <file src="LisaLawson.png" mime="image/png" description="LisaLawson.png"/>
+ <object handle="_f644e8e63e69971d32da553640" change="1692349469" id="O0185">
+ <file src="LisaLawson.png" mime="image/png" checksum="b73163eb9e74e4cf93017957f1d94f74" description="LisaLawson.png"/>
</object>
</objects>
<notes>
@@ -16129,5 +16135,8 @@ Skipped subordinate line Line 10742:
<range start="0" end="1060"/>
</style>
</note>
+ <note handle="_f644ee39b3d26df0a6755388733" change="1692349469" id="N0351" type="General">
+ <text>Objects referenced by this note were referenced but missing so that is why they have been created when you ran Check and Repair on 08/18/2023 05:04:28 AM.</text>
+ </note>
</notes>
</database>
It would be useful to give us a button for a nuclear reset in the 'Administration' panel. Given that some of us might occasionally attempt actions known to have destabilizing effects, it'd be useful to be able to flush data (just the data, or both the data and the media). Just food for thought. I know resources are limited.
I agree, that's gramps-project/gramps-web-api#396.
Thanks for sharing the XMLs, will look at them later.
I did a direct update of the gramps file to the UI, and the pictures look to be loading now.
Do you have any thoughts on what results I can expect if I were to continue to delete and recreate my Gramps DB and then run a sync every time? The IDs will presumably always be different.
I noticed that there may now be duplicates: I am there twice in Grampshub but only once in my local DB. What underlying process do we use to execute the import? Are we just invoking the Import and Merge Tool programmatically or via the CLI, and accepting all changes?
Do you have any thoughts on what results I can expect if I were to continue to delete and recreate my Gramps DB and then run a sync every time? The IDs will presumably always be different.
Sorry, I don't know what you mean by "delete and recreate" or by "every time". Which IDs? The Gramps IDs, the handles or the checksums?
I noticed that there may now be duplicates: I am there twice in Grampshub but only once in my local DB. What underlying process do we use to execute the import? Are we just invoking the Import and Merge Tool programmatically or via the CLI, and accepting all changes?
It's exactly the same (Python) function used by the CLI import or the Gramps desktop import. But this is (very) differerent from the Import Merge tool!
You can check that, when you import the same XML file twice in Gramps desktop, you will have each object twice. That's intentional. Imports are meant to add stuff.
Executing the Import Merge tool programmatically is not possible as it requires manual interaction with the UI.
I mentioned that I will very frequently blow away the DB and rebuilt it from the latest GEDCOM file. Therefore, as I play with the site I may be importing and reimporting data. I'd assume that, with the tool (since it's called the "Import and Merge" tool, not the "Import and Add" tool), any node IDs (which probably be the Gramps IDs) would be compared with existing nodes, and their constitutions overwritten/reconciled. Maybe this is the case, but the standard import functionality, which is used by Gramps Web (it sounds like due to being unable to automate/orchestrate the IM tool), does not do this. Thoughts? This might not be the correct place to discuss this, but thank you either way.
So, the takeaway is just to always sync and never import if the account is non-empty.