Broken XMLs may prevent proper manifest creation
xKevin04 opened this issue · 19 comments
Unfortunately I'm experiencing a similar behavior to #16 again. I've run gogcli.exe manifest generate twice and each time it failed handling some broken data related to XMLs. I set the log-level to debug the second time around but didn't get any useful info beyond calling GetDownloadFileInfo() for the problematic file. The version used was v0.19.0.
As creation takes a very long time and uses a lot of network resources ever since the XML problems on GOG began, I'd rather not do a couple more test runs with the current version. Here are the two created manifests I ended up with in their entirety:
{
"Errors": [
"GetDownloadFileInfo(downloadPath=/downloads/order_of_battle_us_marines/en1patch1) -> retrieval request error: Get \"https://gog-cdn-lumen.secure2.footprint.net/token=nva=1650461891~dirs=6~token=0bade971fde4080f53a15/secure/offline/1744082009/1192658776/52863558974871196/4509/setup_order_of_battle_u.s._marines_8.3.0_%2832bit%29_%2834403%29.exe.xml\": unexpected EOF"
]
}{
"Errors": [
"GetDownloadFileInfo(downloadPath=/downloads/caves_of_qud/en1patch2) -> Could not parse file xml metadata and the first line was not the expected format: EOF"
]
}Thanks for implementing the workaround method by the way, I'm fully aware of the consequences of using it. Getting an almost complete manifest that only misses those couple of problematic entries would be perfect. I would just retry to retrieve their data via gogcli.exe manifest update later then.
Thanks for reporting it. I've validated a fair number of times at this point that the workaround works with the "manifest update" command, but now that you mention it, I don't think I've done a "manifest generate" command yet since the fix so there could be a bug there.
I'll validate in the upcoming days and see if I find anything.
@xKevin04 I looked at your problem for a few minutes and I could not immediately replicate your issue with version 0.19.0 of gogcli.
I don't have the game that caused you an issue, but I generated a manifest with only "Rings of Saturn" that has one small installer with bad xml data like so (you can use the title filter btw when generating a manifest to select only certain games, I do it all the time when troubleshooting the tool):
gogcli manifest generate --title="Rings of Saturn"
[sdk] Bad metadata for /downloads/dv_rings_of_saturn_anthropogenesis/en1installer0: File metadata was still fetched using much longer workaround method.
As expected, the manifest generation was successful with the work-around having been applied for the file with bad xml metadata (with an accompanying warning showing up in the logs).
Did you observe the above warning patterns (ie, [sdk] Bad metadata...) in your logs prior to experiencing your error?
What flags did you use when you invoked the command?
Full output for command gogcli.exe manifest generate:
[sdk] Bad metadata for /downloads/fallout_new_vegas_ultimate_edition_game/es1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/overclocked_a_history_of_violence/it1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/superhot/en3installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/links_ls_1998_edition/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sacred_2_gold/de1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/overclocked_a_history_of_violence/pl1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dying_light_hellraid/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/no_mans_sky_preorder_dlc/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/phoenix_point_year_one_edition/en1patch3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/spec_ops_the_line/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/kerbal_space_program_making_history/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/stellaris_nemesis/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cyberpunk_2077_game/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/superhot_mind_control_delete/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/turnip_boy_commits_tax_evasion/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/order_of_battle_red_steel/en1patch1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/pl1installer11: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/wing_commander_prophecy/de1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/diplomacy_is_not_an_option_demo/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sudden_strike_2_resource_war/de1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/space_rogue/ru2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/ai_war_2_the_spire_rises/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/besiege/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/asterix_obelix_xxl_3_the_crystal_menhir/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/talisman_the_highland_expansion_legendary_deck/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/tropico_6/en1patch2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/openttd/en3installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/post_mortem/fr1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/between_the_stars_original_soundtrack/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/still_life_2/it1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/forgotten_fields/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/a_light_in_the_dark/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/swine_hd_remaster/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/br1installer15: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/metro_last_light_redux/cz1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/craft_the_world_sisters_in_arms/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/talisman_base_game_legendary_deck/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dead_cells_fatal_falls/en3installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/empires_dawn_of_the_modern_world/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/destroy_all_humans_game/en1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/hedon_extra_thicc_edition_upgrade/en3installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/double_kick_heroes/en1installer2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/rescue_rover_collection/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/house_flipper/en1patch1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/war_for_the_overworld_heart_of_gold/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/a_musical_story/en3installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/ultima_4/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/overclocked_a_history_of_violence/es1installer2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/galactic_assault_prisoner_of_power/ru1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/xcom_2_war_of_the_chosen/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/wing_commander_prophecy/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/children_of_morta/en1patch1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cyberpunk_2077_game/en1patch0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/little_nightmares_expansion_3/ko1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cyberpunk_2077_game/en1patch5: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/shadows_awakening/en1patch3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/aven_colony/en1installer2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/stronghold_crusader/es1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dying_light_hellraid/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/aven_colony/en1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/talisman_the_dungeon_expansion_legendary_deck/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/iron_storm/pl1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad download link for /downloads/imperator_rome/en1patch3: File was not added to manifest.
[sdk] Bad metadata for /downloads/lure_of_the_temptress/es1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/talisman_character_samurai/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sid_meiers_colonization/fr1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/original_war_v20/es1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/curved_space/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/fallout_new_vegas_ultimate_edition_game/pl1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/tropico_6/it1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/steel_division_2/en1patch4: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/ufo_aftershock/fr1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/war_for_the_overworld_underlord_edition_upgrade/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/grimshade/en1installer2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/metro_last_light_redux/pl1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/victor_vran/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/reassembly_fields_expansion/jp2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/pl1installer15: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/still_life/es1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dv_rings_of_saturn/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/walking_dead_a_new_frontier_the/en1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/tropico_6/en1patch1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/bioshock_infinite/en1installer10: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/heroes_of_might_and_magic_v/ru1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/cultist_simulator_the_ghoul/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/iron_harvest_base/en1installer2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sacred_2_gold/en1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/sacred_2_gold/de1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/iron_harvest_operation_eagle/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/galactic_civilizations_iii_lost_treasures_dlc/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/downwell/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/crossroads_inn_pests_puppies/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/galactic_civilizations_iii_game/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dead_cells_rise_of_the_giant/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/slipstream/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/metro_exodus/en1installer5: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/the_witcher_3_wild_hunt_game_of_the_year_edition_game/br1langpack2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/hakuoki_kyoto_winds/jp1installer2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/unpacking/en1patch0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/little_nightmares_expansion_3/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/brothers_in_arms_earned_in_blood/fr1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/still_life/it1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/resonance/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/talisman_character_shaman/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/wing_commander_academy/en1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/star_trek_25th_anniversary/en1installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/dropsy/en2installer5: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/spiritfarer/en2installer0: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/fallout_new_vegas_ultimate_edition_game/de1installer3: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/runaway_a_road_adventure/fr1installer1: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/fallout_new_vegas_ultimate_edition_game/en1installer2: File metadata was still fetched using much longer workaround method.
[sdk] Bad metadata for /downloads/knightshift_game/cz1installer1: File metadata was still fetched using much longer workaround method.
imperator_rome/en1patch3 was handled nicely here, something similar to this would be great for these hard EOF errors.
The second command was gogcli.exe manifest generate --log-level debug 1>debug.out 2>debug.err, which generated a 32MiB debug.out and an empty debug.err file. I won't paste or attach the whole thing here for obvious reasons, but it has 141 workaround method lines and 4 bad download link lines. Here's what I get when searching for the problematic file via awk '/caves_of_qud/ && /en1patch2/' debug.out:
"manualUrl": "\/downloads\/caves_of_qud\/en1patch2",
[sdk] GetDownloadFileInfo(downloadPath=/downloads/caves_of_qud/en1patch2) -> GET https://www.gog.com/downloads/caves_of_qud/en1patch2
The first line containing "manualUrl" is just from the return value of GetGameDetails(gameId=1625207125). The second one is close to the end of output, only two similar lines follow, probably from the same batch of links processed.
I can try to do another run today and see what happens, may fail on yet another title - or maybe GOG's servers are in a better mood this time.
I don't trust GOG with flagging updates correctly so I decided to go with option 2 from the project's README for maintaining my local backups.
Thanks, I took more time to study your error messages.
It appears I was previously a little too conservative in labelling only body errors as bad xml metadata, but I'll add errors earlier in the connection as well as downloadble, but malformed xml data (ie, completely unparsable, even with my hat trick of parsing only the first line if the xml data is otherwise bad) as bad metadata too.
I'll release the modification under v0.19.1
PS: Yes, my under-utilization of stderr is in bad taste ;p
Thank you for the quick update. I tried to generate another manifest. It was looking good, however after a whopping 26 hours the process again finished with exit code 1 and I got only the following manifest.json:
{
"Errors": [
"GetDownloadFileInfo(downloadPath=/downloads/sword_of_the_stars_the_pit_osmium_edition/68543) -> Expected response status code of 200, but got 504"
]
}I have attached the command used and the full output from my cli for you here (no debug this time): output.txt
I'm particularly unsure about the "protocol error" lines, as there's no indication about the files causing them at all. Hopefully, a generic update command would suffice in getting the metadata for these.
The state of the XML situation is really bad it seems, making it practically impossible for users with very large libraries to generate a complete, unfiltered manifest. However, I'd like to make this work even if it takes a long time to test.
I will probably switch to updating my manifest in the future and only return to generating a fresh one once GOG finally fixes these issues, though.
Thanks for the feedback. I'd really like to make the manifest generation dependable. There are least 2-3 further mitigation strategies I can think off from the top of my head. I will analyze this issue further and get back to you.
@xKevin04 I added some improvements in v0.19.2:
- Expended the application of workaround method for metadata errors to everything except 403 and 404 error codes (which historically meant a dangling download link for a file that no longer exists)
- I added retries for most bad metadata cases before doing the workaround just in case it is a transient error
- To maximize the detection of transient errors (and avoiding the longer workaround if not necessary), I increased the number of retries from 3 to 5 and added a 100ms delay between retries
Unfortunately, still no dice. This time it ran for approx. 18 hours and again finished with exit code 1. This is the manifest.json:
{
"Errors": [
"GetDownloadFileInfo(downloadPath=/downloads/submarine_titans/en1installer0) -> retrieval request error: Get \"https://www.gog.com/downloads/submarine_titans/en1installer0\": unexpected EOF",
"GetDownloadFileInfo(downloadPath=/downloads/galador_the_prince_and_the_coward/ru1installer1) -> retrieval request error: Get \"https://www.gog.com/downloads/galador_the_prince_and_the_coward/ru1installer1\": unexpected EOF"
]
}And here is the output: output-v0.19.2.txt
One peculiar aspect of this version is that for some files, there seems to be a racing condition or something similar going on between two files. They loop way more often than they should according to the maximum retry number. Examples:
- steel_division_2_reinforcement_pack_12_kostritza/en1installer0 and tangledeep/en1installer0
- galador_the_prince_and_the_coward/ru1installer1 and submarine_titans/en1installer0
The latter pair also consists of the last files processed according to stdout.
Thanks. I think I'll do a refactoring pass on that part of the code to simplify (I had to do a lot of unexpected workarounds due to gog api shenanigans) and try a manifest generation on my end.
Maybe I'll also do an incremental manifest generation similar to actions for game file updates. We might be at that point now.
We'll get the gog api to play ball, even if we have to drag it though the door, kicking and screaming.
I was busy with stuff this weekend, but I managed to cram in the sdk refactor in the main branch.
I'll validate my changes with a manifest generation early in the week and if everything looks good, I'll do a release.
If it is still unstable after that, I'll tackle incremental and resumable manifest generation, but unless other parts of the GOG api beyond the metadata endpoint become very flaky, I think we'll probably be good with just the sdk refactor for now.
I released v0.19.3. I successfully generated a manifest with it. Let me know if you experience further troubles generating new manifests.
The process finishes successfully now, and it only took two hours this time. Sadly, I have to report that there's still something wrong: All checksums in the generated manifest are empty string. It already felt weird when gogcli didn't need to download a single file due to bad metadata, I just couldn't believe GOG suddenly fixed everything.
One installer from the generated manifest as an example:
{
"Languages": [
"english"
],
"Os": "mac",
"Url": "/downloads/fallout_classic/en2installer1",
"Title": "Fallout Classic",
"Name": "fallout_2.0.0.7.dmg",
"Version": "",
"Date": "",
"EstimatedSize": "566 MB",
"VerifiedSize": 593884431,
"Checksum": ""
}Here is the complete output I got this time around:
>gogcli.exe version
v0.19.3
>gogcli.exe manifest generate
[sdk] Bad download link for /downloads/imperator_rome/en2installer0: File was not added to manifest.
[sdk] Bad download link for /downloads/imperator_rome/en1installer0: File was not added to manifest.
[sdk] Bad download link for /downloads/imperator_rome/en1installer1: File was not added to manifest.
[sdk] Bad download link for /downloads/heart_of_the_woods/en1patch1: File was not added to manifest.
[sdk] Bad download link for /downloads/imperator_rome/en1patch3: File was not added to manifest.
[sdk] Bad download link for /downloads/crossroads_inn_digital_goodies_pack/83927: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88945: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88948: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88951: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88891: File was not added to manifest.
[sdk] Bad download link for /downloads/dying_light_the_following_enhanced_edition/88954: File was not added to manifest.
[sdk] Bad download link for /downloads/metro_exodus/85424: File was not added to manifest.
[sdk] Bad download link for /downloads/the_bards_tale_iv_directors_cut_deluxe_edition_bonus/83005: File was not added to manifest.
[sdk] Bad download link for /downloads/republique/41353: File was not added to manifest.
[sdk] Bad download link for /downloads/wasteland_2_directors_cut_digital_deluxe_edition_upgrade/34743: File was not added to manifest.
I did a second run to verify and both ended up without checksums.
My bad, this is what happens when you tackle too many things at the same time.
The manifest on my end was missing checksums as well for the installers. I should have looked at it more closely.
I just fixed it and will push it to main. It was a bug that caused the installers to follow the same path as the extras (which have no checksum).
I'll generate a manifest again and if all goes well, will release the fix on v0.19.4
EDIT:
./gogcli manifest generate -o windows -o linux -l english -l french -l spanish -l japanese
[sdk] GetDownloadFileInfo(downloadPath=/downloads/sacred_2_gold/en1installer2) -> body retrieval error: stream error: stream ID 129; PROTOCOL_ERROR; received from peer. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/sacred_2_gold/en1installer2) -> body retrieval error: unexpected EOF. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/sacred_2_gold/en1installer2) -> body retrieval error: unexpected EOF. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/sacred_2_gold/en1installer2) -> body retrieval error: unexpected EOF. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/sacred_2_gold/en1installer2) -> body retrieval error: unexpected EOF. Will retry.
[sdk] Bad metadata for /downloads/sacred_2_gold/en1installer2: File metadata was still fetched using much longer workaround method.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/atom_rpg_trudograd/en3installer0) -> body retrieval error: unexpected EOF. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/atom_rpg_trudograd/en3installer0) -> body retrieval error: unexpected EOF. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/atom_rpg_trudograd/en3installer0) -> body retrieval error: unexpected EOF. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/atom_rpg_trudograd/en3installer0) -> body retrieval error: unexpected EOF. Will retry.
[sdk] GetDownloadFileInfo(downloadPath=/downloads/atom_rpg_trudograd/en3installer0) -> body retrieval error: unexpected EOF. Will retry.
Seems to be encountering and dealing with some of the expected errors from GOG's end now with what I got on the main branch. I'll let it finish and make sure it doesn't encounter an unrecoverable error partway through.
Quick update: Manifest generation is still running. A little bit insane that it is now taking over 12 hours to generate a manifest from scratch due to files with bad metadata. I would rather have avoided it, but I think I'll investigate a cloud integration the coming weeks to see if hitting different parts of the cdn makes a significant difference.
Ok, the manifest was successfully generated and the checksums for the installers are there.
I just released v0.19.4 which contains the adjustments.
It failed after about 12 hours with yet another new error. I will try again later, because it reads like something on GOG's side broke hard for an extended period of time. I think your new version should generally work with GOG's XML shenanigans now. In any case, here is the generated manifest.json:
{
"Errors": [
" GetDownloadFileInfoWorkaroundWay(downloadPath=/downloads/abandon_ship/en3installer0) -> retrieval request error: Get \"https://www.gog.com/downloads/abandon_ship/en3installer0\": dial tcp 88.221.61.153:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.",
" GetDownloadFileInfoWorkaroundWay(downloadPath=/downloads/x3_terran_war_pack/ru3installer0) -> retrieval request error: Get \"https://www.gog.com/downloads/x3_terran_war_pack/ru3installer0\": dial tcp 88.221.61.153:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.",
" GetDownloadFileInfoWorkaroundWay(downloadPath=/downloads/speedball_2_hd/en1patch2) -> retrieval request error: Get \"https://www.gog.com/downloads/speedball_2_hd/en1patch2\": dial tcp 88.221.61.153:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.",
" GetDownloadFileInfoWorkaroundWay(downloadPath=/downloads/little_big_adventure_2/en1installer0) -> retrieval request error: Get \"https://www.gog.com/downloads/little_big_adventure_2/en1installer0\": dial tcp 88.221.61.153:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond."
]
}And the output: output-v0.19.4.txt
I will report back after retrying.
Thanks for the feedback. Let me know how your next try goes.
I still have some venues left to mitigate things:
- implement resumable manifest generation
- investigate if a cloud integration could reduce the number of occurrences of the bad metadata problem
The first would ensure that hours of data retrieval are not lost if a problem is encountered and the second has the potential to reduce the delay for manifest generation. I'll see what makes the most sense to implement.
I'm happy to report that the second try worked. It took 34 hours, but at long last I've got a complete, unfiltered fresh manifest and all expected checksums are included!
Resumable generation would be great, as would a progress indicator and final output status (maybe both only when toggled on via switches). But those would fit better as separate feature requests.
Thanks for fixing this, now onto actually updating my backup.
Glad to know it finally worked. Given that the time for manifest generation have gone from ~25 minutes to 12+ hours, it does indeed make sense to review the design of that functionality to provider better feedback and the ability to resume on failure.