AnnoDesigner/anno-designer

Import/export Anno 1800 stamps

Atria1234 opened this issue · 12 comments

Atria1234 commented

Anno developers added "Stamps" in last game update. Stamp is a collections of buildings and roads (maybe other things). It would be super helpful to be able to import them from to AD and also export them to Anno 1800 compatible stamp.

  • So far the format of the stamp is not well known so some decoding has to be done
  • Export might be problematic since there can be buildings which don't exist in the game
NiHoel commented

Decompression

Stamps are zlib compressed RDA files (use the attached interpreter file: stamp.zip).

  1. Decompress them in Python, e.g.
import zlib
with open("path/to/stamp", "rb") as f:
    content = zlib.decompress(f.read())
    with open("stamp.bin", 'wb') as f:
        f.write(content)
  1. Run FileDBReader: FileDBReader.exe decompress -i "FileFormats\stamp.xml" -f stamp.bin

Trivia

  • Stamps are sorted by name
  • Folders are only visible if they contain a stamps
  • The icon of the first stamp determines the folder icon
  • The game needs to be restarted after editing stamps (content, name, path) in the explorer

Dev notes

  • The folder remains empty, if it does not contain a valid stamp
  • Stamps are loaded by the game if they are saved as a (compressed) rda file – not xml file
  • File extensions are ignored
  • ComplexOwnerID can be any number
  • Stamps are invalid if
    • Applying zlib compression on a valid (rda uncompressed) stamp using Python’s zlib library
  • Stamps are still valid
    • With incorrect or no street count, but streets are missing
    • Without dir nodes
    • Without icon
    • Without railway info
    • Fields and Farms do not have a ComplexOwnerID
    • They exceed the limit of buildings selectable for blueprints
    • They contain a harbour warehouse (in creative mode, the player can place two harbour warehouses on an island using such a modified stamp; but when trying to destroy one, the game crashes -> tools that generate stamps should ensure that stamps do not contain harbour warehouses)
    • with stuff not in construction menu (e.g. stone roads or modules in arctic)
    • they contain the guid of mining slots, pits etc. That way one can add these to islands (in both creative and normal mode)

My plans

  • Add a click-and-run tool to SavegameVisualizer that generates images for all stamps
  • Add a save stamp button to export stamps for whole islands

Caveats

Exporting AD files to stamps requires a lot of investigation:

  • Derive GUID based on icon
  • Derive rotation and handle decentered buildings (basically inverting what I do in the SavegameVisualizer)
  • Determine which objects are roads
  • Determine which objects are rails and turn them into a rail network
  • Handle modules (probably discard them because it's often not clear to which main building they belong)
  • ❤️2
Atria1234 commented

Open questions about the stamp format

  • How is coordinate system oriented?
    • After reseting rotation:
    • obrazek
  • How is center of coordinate system calculated?
    • It appears to be +/- 1 tile from center of selection (depending on when was the selection started)
  • What does StampPath contain?
    • Appears to be constant (5000000)
  • How to translate building GUID to AD building
  • What does ComplexOwnerID contain?
  • What does Variation contain?
  • How are multiple types of roads represented?
    • StreetInfo contains pairs of elements. First element (None) in the pair contains GUID of the road of this pair. Second element in the pair contains list of positions where that road type is placed.
  • How are rails represented?
    • RailwayInfos contain list of straight segments of rail. Each segment is defined by two points.
    • obrazek
  • How are farms/modules handled?
  • How are multifactories handled?
    • Multifactories with selected recipe appear to be brokes as they are trying to have ":" in the name but filesystem doesn't allow that
NiHoel commented
  • StampPath is the GUID of the region: that way, the game determines which stamps to display in the current session
  • How to translate building GUID to AD building: You can use the information @StingMcRay added for me: GUID per AD template and replacement.csv
  • What does ComplexOwnerID contain? Object-ID of the main building (all modules and the main building have the same ComplexOwnerID - the exact value doesn't)
  • What does Variation contain? The identifier of the skin (usually a value <10; can be safely ignored for reading/writing)
  • How are farms/modules handled? Each object is a separate entry; what belongs together is saved in ComplexOwnerID. The main building must be first in the list.
AgmasGold commented

Its worth mentioning as well - we already have some logic to map an object in a layout to the Anno building it probably represents in the statistics calculation code. I'm not 100% sure if we can use it to map to a GUID, but would be worth a look.

NiHoel commented

Complete process of turning a stamp into an AD file: https://github.com/NiHoel/Anno1800SavegameVisualizer/blob/main/stamp_converter.py

  • 👍1
taubenangriff commented

I cobbled together a quick data model for serialization of stamps, maybe that can help you guys with the export: https://github.com/taubenangriff/StampDataModel/blob/master/StampDataModel/Stamp.cs

  • 👍1
Atria1234 commented

I haven't have success with recompressing decompressed stamp so far. Would you have some insight how Anno uses zlib to compress files? I decompressed/recompressed with python implementation of zlib (compression done with all 9 compression levels and all valid wbit values) but no luck so far. The results looked the most similar (about 80% binary equal with long streaks of same bytes) with either 8 or 9 level of compression

Atria1234 commented

Also @taubenangriff since your FileDbReader is needed to convert stamps to XML: how would you propose we use your FileDbReader in AD to read stamps?

  • git fork?
  • copy built executables?
  • something else?
taubenangriff commented

I recommend just using the filedb library (FileDBSerializer.dll) for this usecase, create the stamp data model from annodesigner data, and then serialize the data model to the file. You can see https://github.com/taubenangriff/StampDataModel/blob/master/StampSerializingTest/Program.cs for how creating and loading a stamp is done.

  • 👍1
taubenangriff commented

also, the game uses zlib compression level 8, BUT with 12 bytes added at the end of the compressed result:

252536 0 <filesize> all int32s, filesize is the file size of the uncompressed stamp. That might be why decompressing works, but your stamps are invalid after recompressing.

  • ❤️1
Serpens66 commented

You can not simply use "0" for the Pos, some need 0.5. So the combinations you have to try for a single centered building are:
0 0 , 0.5 0 , 0 0.5 and 0.5 0.5 and see which one works for your building ingame.
There is most likely a better calculation for the "Pos" of a building in a stamp and maybe you already know it, but just in case here what I found out for my single-building Stamp mod:
.................
Gibt vermutlich noch eine korrekte immer funktionierende Berechnung für die "Pos" eines Gebäudes im Stamp und vllt kennt ihr die bereits, aber dennoch mal hier das was ich dazu für meinen Mod rausgefunden hab (der nur ein einzelnes Gebäude in eine Stamp Datei packt: https://www.nexusmods.com/anno1800/mods/566

buildingsize ist zb: [6,6] für ein 6 mla 6 Gebäude.

# Ausnahmen zu der Regel (gibt nur sehr wenige, keine ahnung warum es sie überhaupt gibt). zb Ventilatorenfabrik Artistas ist 6x6 und dennoch brauchts 0 0,5 damit stempel geht 
    # Quarzgrube 1010560 braucht <Pos>1,5 0</Pos>, aber ist 6x10 und mit der 6 als erstes ist diese Pos auch unmöglich
    # Dockland 601470 und ihre Module könnten auch Ausnahmen haben, aber die Module packen wir nicht in stamp, weil man sie direkt zu beginn aus der speicherstadt blaupause setzen kann.
    PosAusnahmen = {5862:"0 0,5",1010560:"0,5 0",601470:"0 0,5",6264:"0,5 0,5",
        100519:"0,5 0,5", 101344:"0,5 0,5", 116030:"0,5 0,5", 117871:"0,5 0,5", #  Anlegestelle 100519 braucht <Pos>1,5 -0,5</Pos> laut Spiel, aber hat Maße 7x6 wobei die 7 fest ist und der Hafenbereich nur die 6 vergrößern kann. Doch mit 7 als ersten Wert ist diese Pos nach meinen aktuellen Regeln unmgöglich.
        118729:"0,5 0",114440:"0,5 0,5",112666:"0,5 0,5",112674:"0,5 0,5",
        114544:"0 0",117743:"0 0",117744:"0 0", # flussgebäude
        }
    
    def calc_pos(buildingGUID,buildingsize):
        pos = PosAusnahmen.get(buildingGUID)
        if pos is None:
            beidesgerade = buildingsize[0]%2==0 and buildingsize[1]%2==0
            beidesungerade = buildingsize[0]%2!=0 and buildingsize[1]%2!=0
            erstesungerade = buildingsize[0]%2!=0 and buildingsize[1]%2==0
            zweitesungerade = buildingsize[0]%2==0 and buildingsize[1]%2!=0
            if beidesgerade:
                pos = "0,5 0,5" # der doofe stamp parser von DuxVitae verlangt Kommazahlen mit Komma, bei Punkt funzt das Ergebnis nicht!
            elif beidesungerade:
                pos = "0 0"
            elif erstesungerade: # funzt so, frag mich nicht warum das hier umgedreht wird und eine gerade zahl jetzt zu Pos 0 wird, während das oben umgekehrt war...
                pos = "0,5 0"
            elif zweitesungerade:
                pos = "0 0,5"
        return pos

buildingsize can be calculated like this (code base from Dux Vitae):

def get_buildsize(GUID,buildingnode): # calculation from DuxVitae
    size = None
    ifo_part = buildingnode.find("./Values/Object/Variations/Item/Filename")
    if ifo_part is not None:
        ifo_part = ifo_part.text.replace(".cfg",".ifo") # die ifo datei heißt genauso mit anderer Endnung
        ifo_path = f"{datapath}/data0bis27/{ifo_part}" # ist jetzt alles gesammelt in diesem ordner
        corners = []
        ifo_tree = ET.parse(ifo_path)
        withharbourarea = False
        DummyNames = ifo_tree.findall(".//Dummy/Name")
        for dummyname in DummyNames:
            if dummyname is not None and dummyname.text=="harbourblock01":
                withharbourarea = True
                break
        if withharbourarea: # check in assets.xml if it is extended or not
            HarbourAreaExpand = get_property(buildingnode,GUID,"Blocking/HarbourAreaExpand",text=True,integer=True)

        for corner in ifo_tree.findall(f".//BuildBlocker/Position") :
            corners.append([
                float(corner.find("xf").text),
                float(corner.find("zf").text)
            ])
        corners = np.array(corners).transpose()
        if not withharbourarea and len(corners[0]) >= 8:
            # skip second building blocker of mines - scheint tatsächlich noetig, Mine ist dan 3x3 anstatt die kompletten 5x8 und die Pos eines Eisenminenstempels ist 0,0 ,was heißt size muss ungerade ungerade sein, also ist 3x3 wohl richtig in diesem Kontext.
            diag0 = np.linalg.norm(np.max(corners[:, 0:4], axis=1) - np.min(corners[:, 0:4], axis=1))
            diag1 = np.linalg.norm(np.max(corners[:, 4:8], axis=1) - np.min(corners[:, 4:8], axis=1))
            if diag1 > diag0 + 0.1:
                corners = corners[:, 0:4]
        to_int = lambda arr: np.array([int(round(val)) for val in arr])
        size = list( to_int((np.max(corners, axis=1) - np.min(corners, axis=1))[::-1]) )
        
        if withharbourarea: # das folgende klappt bei vielen Gebäuden, aber es gibt ein paar Ausnahmen, die keinen Sinn ergeben. Diese werden unten in PosAusnahmen gepackt
            if not HarbourAreaExpand:
                size[1] += size[0] # es wird ein quadrat in die size[1] richtung mit kantelänge size[0] angehängt als Hafenbereich
            else:
                size[1] *= 2 # die berechnete Hafenbereich ist wohl einfach nochmal die Gebäudegröße drangehängt

    return size
NiHoel commented

@Serpens66 You can find all your exceptions (and more) here: https://github.com/NiHoel/Anno1800SavegameVisualizer/blob/bcd4b26983c8a8f9946d957a623dcc62afa7453f/tools/params.py#L3536-L3863

Coordinates are corners of the blocked area.