Original Arena saved game decoding and import
Opened this issue · 19 comments
This may already be in your longer-term plans, but I think it would be very useful for testing and reverse-engineering the original game data to be able to import native save game files from original Arena, as Daggerfall Unity can do with saved games from original Daggerfall.
There are some saved game editors on UESP for Arena, and these are useful for finding out how data is stored in the save files.
Here are basics on how Arena handles saved game files, based on testing with a newly created character on a fresh Arena installation:
Files modified when saving, where ## is the save number (00, 01, etc.):
AUTOMAP.64 (Identical to AUTOMAP.##?)
AUTOMAP.##
CITYDATA.## (Has city names)
LOG.## (empty on new game)
NAMES.DAT
SAVEENGN.## (contains character stats data)
SAVEGAME.##
SPELLS.## (empty on new game, even with a spellcaster. Maybe records custom-created spells?)
SPELLSG.## (standard spells for sale?)
STATES.## (empty on new game)
WILD001.## (wilderness data probably)
WILD002.##
WILD003.##
WILD004.##
Files modified when program started:
CITYDATA.64
LOG.64
SPELLS.64
SPELLSG.64
STATES.64
Files modified when loading game:
AUTOMAP.64
CITYDATA.64
LOG.64
SAVEENGN.64
SAVEGAME.64
SPELLS.64
SPELLSG.64
STATES.64
WILD001.64
WILD002.64
WILD003.64
WILD004.64
Temporary files that can be created. Deleted on program close:
CHARSHET.$$$
SAVEFACE.$$$
SCREENSV.$$2
TMP2.$$$
**.64 files remain even after closing the program, but are probably active data being used in the current session.
NAMES.DAT
Saved game names that appear on the load screen. No actual link to the rest of the save data.
Each name can have up to 47 characters. This leaves space for at least 1 00
to terminate the string.
Any leftover character spaces are filled out with 00
.
LOG.**
Journal entries.
Format (all numbers stylized as "code" are in hexadecimal):
26
(&) - Skips down 1 line, hides preceding 2 characters if they would render and sets following text to be rendered in red color, but only if there are 2 or fewer characters between the start of the file and 26
, or if 26
was preceded by 2A
(perhaps other conditions as well). Otherwise all that happens is "&" is rendered. Used before date entries. Red color lasts until the next "new line" (0A
or 2A
).
0D
- appears at end of date entry. Doesn't render anything, purpose unknown.
0A
- new line
20
- appears before 2A
at end of text entries. Doesn't render anything, purpose unknown.
2A
(*) - Used at end of text entry. Causes previous byte to not be rendered if it would have rendered something, starts a new line.
Basic usage for a log entry seems to be:
26
Date entry 0D
0A
Text entry (using 0A
for new lines as needed) 20
2A
26
(&) behavior seems a bit weird, but for example:
If the file starts with:
&Text
, a&Text
or aa&Text
then a line is skipped and "Text" is rendered in red, with the preceding "a"s not shown.
If the file starts with aaa&Text
or with more before the &
, it will just be rendered as it is, in the default color and without skipping a line.
If you just want to read in the automatically-generated journal entries, though, I don't think it would be necessary to replicate the behavior this exactly.
Up to 13 rows of text can appear. More than that and you get a "More..." button to turn the page.
SAVEENGN.**
(all numbers stylized as "code" are in hexadecimal)
Offset:
04
, byte, Race ID
Values:
60
Redguard
61
Breton
62
Dark Elf
63
Nord
64
Wood Elf
65
High Elf
66
Argonian
67
Khajiit
Setting to an invalid value will make the race name blank, and the character invisible on the character sheet and the character face in the menu bar, with a black background on the character sheet.
I'll stop here and wait for feedback. I'm trying to keep this organized but I have a feeling it could easily become a mess if we aren't careful.
It would probably be good to separate out each type of data to be imported into its own issue. Anyway, I'll wait for your opinion.
I will describe the save file format in the wiki.
Yeah, I was about to suggest that. Having the save file format in the wiki would be preferable for long-term records. We can keep this issue here as a reminder.
I didn't even notice the wiki until now. Good to see Carmina16 has already made a lot of progress.
It might be further useful to have individual wiki pages on data elements when there is more information. For example, the valid race ID values. Or just a comment may be enough.
The text formatting and journal information (such as the maximum number of lines per journal page) also seem like they could go in the wiki.
@Carmina16, I'm happy to see so much progress on the wiki, including not just saved game structure but things like travel code!
I am as well. I'm not sure what the extent of their knowledge is, but they seem to know a lot about Arena's inner-workings. Although I might not have the chance to implement all of those things in the wiki right away, I plan on looking into each one over time. I've still got a lot of renderer stuff and level data I need to work on first.
I've gathered a bit more info about the spell format (which by the way is also used in the file SPELLS.LST). Referring to the structure in the wiki:
WORD Param0[3];
WORD Param1[3];
WORD Param2[3];
WORD Param3[3];
WORD Param4[3];
WORD Param5[3];
BYTE TargetType;
BYTE unk;
BYTE Element;
WORD Flags;
BYTE Effects[3]; // e.g. 'Fortify'; 0xFF = no effect
BYTE SubEffects[3]; // e.g. 'Attribute:'
BYTE AffectedAttr[3]; // e.g. 'Strength'
WORD Cost;
char Name[33];
The parameters param0-5 contain three entries, each of which is relative to one of the three possible effects a spell can have (specified in Effects[3]).
Valid values for TargetType are:
0x00 self
0x01 touch
0x02 target
Values for Element are:
0x00 Fire
0x01 Ice
0x02 Poison
0x03 Shock
0x04 Magic
0x05 None
Possible values for Effects:
0x00 Cause
0x01 Continuous Damage
0x02 Create
0x03 Cure
0x04 Damage Health
0x05 Designate as non target
0x06 Destroy wall
0x07 Disintagrate
0x08 Dispel
0x09 Drain Attribute
0x0A Elemental resistance
0x0B Fortify attribute
0x0C Heal Attribute
0x0D Transfer Attribute
0x0E Imprison
0x0F Invisibility
0x10 Levitate
0x11 Light
0x12 Lock
0x13 Open
0x14 Regenerate
0x15 Silence
0x16 Spell Absorption
0x17 Spell Reflection
0x18 Spell Resistence
Different spell effects use different parameters:
Cause
param0 chance to inflict {disease/poison/curse/paralysis}, deterioration is
param3 pts of ?? every round + param1% every param4 levels. duration is
param5 rounds per level
Continuous damage {health/fatigue/spell points}
param0 to param1 pts of damage every param4 rnds. duration is param5 rnds.
damage is param2 to param3 pts every param4 levels
Create Shield
param0 hit pts shield created, +param1 pts every param4 levels
Create Wall/Floor
param0 walls/floors created
Cure
param0% chance to cure {disease/poison/para/curse} of equal level to the
caster. + param1% every param4 levels
Damage {health/fatigue/spell points}
param0 to param1 pts damage +param2 to param3 pts per param4 levels
Designate non target
param0% chance caster is ignored + param1% per param4 levels
param5 rounds per param2 levels. (you may cast other spells: check flags)s
Destroy wall/floor
param0 walls permanently negated
Disintagrate
param0% chance to disintegrate +param1% every param4 levels
Dispel
param0% chance + param1% every param4 level. Area is param5 meter radius
Drain Attribute
param0 pts of attribute drained for param5 rounds per level. Target recover
attribute at param1 pts per param4 rounds
Elemental resistance
resistance to element +param0% for param5 rounds per param2 levels of caster
+param1% per param4 levels
Fortify attribute
+param0 pts of attribute for param5 rnds per level. attribute loss of param1
pts per param4 rounds.
Heal attribute
param0 to param1 pts to attribute +param2 to param3 per param4 levels
Transfer attribute
Tranfers param0 pts of attribute from target to caster
Imprison
param0 hit pts cage created, + param1 pts per param4 levels
Invisibility
param0 rounds + param1 rounds per param4 levels
Levitate
param0 rounds + param1 rounds per param4 levels
Light
param5 rounds per level
Lock
param0 chance of locking. duration is param5 rounds per param2 levels
lock strength + param1 % per param4 levels
Open
param0 chance of unlocking, + param1% per param4 levels
Regenerate
param0 health every param1 rounds for param4 rounds per level
Silence
param0 chance of silencing for param5 rounds per param2 levels, + param1 %
per param4 levels
Spell Absorption
param0 chance to absorb + param1 % per param4 levels. duration is param5
rounds per param2 levels
Spell reflection
param0 chance to reflect + param1 % per param4 levels. duration is param5
rounds per param2 levels
Spell resistence
param0 chance to resist + param1 % per param4 levels. duration is param5
rounds per param2 levels
What needs further investigation is the meaning and possible values of Flags/SubEffects/AffectedAttr for the different Spell Effects. Hope it's helpful.
Looks like player coordinates in a level are at offset 0x105C8 in SAVEGAME.xx. First a DWORD that is the X coordinate, then a DWORD that is the Y coordinate. This could be useful for cheating one's way to a particular square in an Arena save.
Originally in #100
F8 - Toggle compass
Toggling compass should be a XOR of 0x2000 or 0x0020, possibly on the GameOptions WORD in the > SAVEENGN file, as described in the Wiki. For a new character I created the changing value was at offset > 0x425 in the SAVEENGN file.
This seemed the appropriate place, so I hope this isn't necro posting. I'd just like to leave a caution note about importing original save games into OpenTESArena. The original game has many ways that you can corrupt the save game files which end up causing glitches later on. Simply opening the "SAVE/LOAD" menu before creating a character will introduce glitches into your game. This example will cause the starting date to be the 1st of Morningstar instead of the 1st of Hearthfire as intended. This can cause many glitches and even the 1.06 patch notes recommends against using this option which is part of the NEW GAME option.
"Can't get an Artifact Quest: This seems to be related to picking the Start New Game option while in the middle of another game. We recommend that you do not choose this option."
There appear to be several ways to mess up the saved games, but the glitches show up later and we usually blame the game rather that the save file. Which is my point. I'd hate to see people loading their corrupted save files and blaming problems on OpenTESArena. Control of input is crucial. It's still a great Idea, but I'd recommend heavy-handed control over the process. (So grateful for this project that has allowed me to peek under the hood - great work!)
Thanks for the info. It's sometimes reasonable to do sanitizing of data read from the original game if it means more quality of life for the modern engine. From your example though, I can't see how that would break a save if the month was different. Seems like some global variable bug in the original executable that presumably gets saved to the save file. I don't know how artifact quests were implemented originally but that bug would probably not carry over to here.
If there are known problems with saves from the original game, those could be perhaps be (optionally?) patched if importing them into OpenTESArena.
I made created this issue after helping work on Daggerfall Unity. Saves from the original game were useful to have when researching and implementing various game mechanics, so I figured they would also be useful here.
Another big reason, though, is because something that happened in both Daggerfall Unity and OpenMW that would be nice to prevent as much as possible: having to worry patch or worry about conflicts with old versions of OpenTESArena-format saves. As in, say, you release OpenTESArena with save/load support (in a unique OpenTESArena format), then later change the way some kind of data is stored in order to more accurately recreate something from the original game, etc. You then have the problem of supporting, or not supporting, the old version's saves. In OpenMW, for example, I've seen "that would change the save format" often come up as an obstacle to why something can't be easily changed.
I thought that perhaps, if importing saves from the original game is implemented well, maybe for some time those could be used, perhaps exclusively, for testing/development before releasing OpenTESArena save/load capability "out into the wild", which might minimize the problem of changing save game formats, since it would be somewhat mature/complete already from staying close to the original data format. @afritz1, it's all your call of course, but I'm just letting you know why I kind of wanted to push the idea of importing original saves and opened this issue.
I don't think it's going to be a problem. Most of the save file is just a dump of internal variables not needed for this engine, so we can easily and safely cherrypick what we actually need.
@afritz1
The month change was just the obvious notice that your game was glitched. It was the other glitches that came along with it that were the problem. I still don't know how these cause such problems from the save data. If you press ESC when asked if you want to be Male or Female, it will default to Female, but create some weird offsets. When you try to change weapons, it will instead change your character's face. (the weapon works, but sprites in the menu are screwy.)
@Allofich
I "do" think it's a good idea and would have many benefits, I just wanted to caution to ensure it didn't carry bugs over which leads me to my next comment.
@Carmina16
That's good. I don't know how this engine works (nor the original for that matter), I just remember how frustrating it was working on a project where I had control over the program, but no control over the input files. They were all over the place. I think they thought the program was just supposed to magically know. "I only moved one thing. It's obvious." But yes, that's what I was hoping. Cherry picking which allows you to test if they are valid variables. Thanks.
Regarding the calendar, this page says that the reference sheet included with the game shows a Tamriel-ified Gregorian calendar (which is pretty much what Morrowind and later use), but in the actual game, every month has 30 days and each month begins on Tirdas (Tuesday) the 1st, despite the each month ending on Middas (Wednesday) the 30th.
The actual implementation details are probably better suited for the Time & Calendar wiki, but I'll add my two cents for game importing. My understanding is that the game stores time as minutes since the start of the game (12:00 noon, Tirdas, 1st of Hearthfire in the year 3E 389). If we have OpenTESArena use a "fixed" calendar that matches the Arena reference sheet and later games (and I think we should instead of using the broken existing calendar) I would be okay with having the internal minutes since game start stay the same and the cosmetic display change. For instance, I would be okay with 90 days after game start being:
- Original Arena: Middas, 30th of Sun's Dusk
- OpenTESArena: Sundas, 29th of Sun's Dusk
So long as our save import documentation explains that this is happening and that this is cosmetic; no quest deadlines or whatever are being modified because internally they are still being calculated as minutes from game start.