cknave/kevedit

ZXT format support

iamgreaser opened this issue · 3 comments

iamgreaser commented

Obligatory reading:

We've had ZXT 1.0 be a thing kinda sorta for about 6 months now and it's still not supported in any of the two editors I hear about on the Discord of ZZT IRC Bridge. There is an "about as official as it's gonna get" branch of ClassicZoo which supports it as well as a handful of extensions, and there's also ZXTMGR, a standard implementation for anyone to bring into their own forks, which I've used myself for yet another unreleased fork.

I'm currently working on a branch of KevEdit which I hope won't become yet another unreleased fork (yes, I even have those for KevEdit, not just for ZZT). It took about an hour and a half to get basic support for reading a ZXT file with absolutely no support for any actual extensions. However, in order to actually support ZXTs, I do need a way of tracking extension details, and I will need to have integration with the editor.

Well, "I" meaning "whoever ends up using a KevEdit with ZXT support". Which is why I'm putting this issue forward for discussion.

This issue exists to get some discussion out of the way so I don't end up bowling the entirety of KevEdit over to the shock of the next person who needs to touch the code. Adding ZXT support to KevEdit will require me to change a lot of things.

For an initial implementation I'd like the following kinds of extensions, largely to ensure that the implementation remains flexible:

  • Everything in the 000000xx:xxxx space which is easy to implement from the editor side and is not marked as a draft. The only thing there which isn't merely a quirk flag or a bugfix flag is 00000000:0001 (#IFEXT).
  • Something basic.
    • 0000A51E:0004 (#IF RND) really only affects the syntax highlighter from the editor side. However, said syntax highlighter will also need to be rearchitected, as the highlighting will need to be conditional. Due to it being declarative, I suspect this isn't going to be a royal pain, but still, rearchitecting needs to happen.
    • Admittedly, 00000000:0001 also falls in this camp.
  • Something which actually has data in it. Some candidates, of which I'd probably only implement one of for the initial ZXT implementation:
    • 00022038:0003 (Custom fonts) could be quite useful, although this would require KevEdit to be able to support an arbitrary custom font for drawing board data... while possibly keeping around the default font for GUI stuff, I'm not sure how necessary this will be.
      • For an initial implementation I'd be tempted to limit it to 8x14 fonts and to tell people to use MegaZeux as a font editor.
    • 79656D74:0003 (Key counters and limits) has an unsigned 16-bit value as its data, indicating the maximum number of a given key colour one can hold. This is marked as a draft but it's mostly because I haven't removed the draft tag on it yet (and there's still that pasto in the sample code, dammit). Implications:
      • The syntax highlighter needs to be able to conditionally support the *key counters.
      • Keys need to conditionally be presented as booleans or unsigned 8-bit integers when being edited somehow.
    • 0000A51E:0005 (Maximum stat count) has two unsigned 16-bit integers, although it's currently marked as a draft. Implications:
      • The stat array (in KevEdit terminology, the param array) will need to be dynamically-sized. From a quick look at libzzt2/zzt.h, this is already the case. Good job.
      • The maximum number of stats (params) needs to be dynamic. It appears that in libzzt2 this is already the case, and this needs to be taken advantage of in the editor itself. This should be set to the "Minimum stat count" field, as if one cannot go above that number in the implementation, then a board with more stats than that cannot be loaded in such an engine, and thus one would be theoretically violating the extension spec.

So, how do we store this information in memory?

  • ZZTworld needs a field to indicate that it's using one of the following:

    • Vanilla ZZT
    • ZXT as one file
    • ZXT split into a ZAX file and a ZZT file - this would most likely forbid the use of format-breaking extensions.

  • There needs to be a ZXTEXT_* enum type for all supported ZXT extensions, and a ZXTEXT_COUNT field based on the last entry, which has the following purposes:

    • Having a ZXTblock * array of ZXTEXT_COUNT elements in ZZTworld, which acts as a quick indicator for the presence or absence (== NULL) of a block. This will point to the latest block in the list. This is an optimisation - the authorative version follows...

  • There needs to be a ZXTblock struct type in order to track ZXT extension blocks. This should be placed into a dynamically-sized pointer array in ZZTworld as a pointer+length pair (that is, ZXTblock **zxt_blocks; size_t zxt_block_count;).

    • Fields will be as follows:
      • uint16_t flags; uint32_t owner_id; uint16_t selector_id; uint32_t field_length; char *field_data; as per the ZXT 1.0 specification.
      • void *userdata; for extensions to track extra stuff, or track stuff in a format more convenient than the packed field data format.
      • Possibly some kind of vtable for a given extension to follow.
      • ZXTblock *earlier_block; as an optimisation to work out what the next block is in the chain of blocks with the same ID when one searches via the ZXTEXT_*-indexed array.

If there are problems with the above, speak now or forever hold your peace. I don't want to be wasting time on waiting for people to get back to me on how to do this.

I'm going to leave the "adding ZXTs to the KevEdit codebase" side of things as an open question for now:

How do we make it easy to cleanly add support for a given ZXT extension to KevEdit?

Part of this is going to require writing a document on how to extend the codebase for this purpose.

docs/ is potentially the wrong place for this as that's for the in-editor help, but we're going to need some documentation there anyway as we're going to want people to know what the extensions do. As a result, said "how do I add support for a ZXT extension" document will need to cover where one needs to put the information into the in-editor help.

Ideally I'd want there to be one set of files per extension (e.g. under src/zxt/0000A51E/0004/ one would have ifrnd.c and ifrnd.h), and some vtable-based API to handle loading and saving and various other things. There's potential to allow auto-discovery of these during the build process. Not sure if we really want to go down that track, however.

If I haven't managed to make sense of this in about a week's time then please prod me. I do want this support to make it into at least one commonly-used editor, and this is the one I tend to use. Thanks.

asiekierka commented

As the leader of the ZXT specification, my rule of thumb is "implementing something is only worth it when at least one game uses it". I know it creates a bit of a catch-22, but it's the best way to rule out a whole host of problems which may happen on the road to one.

  • 👍1
cknave commented

Appreciate the detailed write-up. I've had a chance to read over the ZXT spec, and the outline of your implementation makes sense to me.

Regarding where to put extension development documentation, EXTENDING.md in the project root, linked from the README?

Regarding the extension sources, I have a strong preference for a human-navigable source tree. If we're starting out with just a handful of extensions, I'd put them in a single file if it's just a few hundred lines. If this gets out of hand, I'd expect some natural "fault lines" to appear, maybe around common code or similar functionality.

Looking forward to seeing your implementation!

asiekierka commented

https://github.com/OpenZoo/ClassicZoo/tree/project/classiczoo-zxt-native-take2

An implementation lives here now, but not so much any games using it at this time - WeaveZZT is not as specified, but that allows it to move in a more agile manner and thus is much more popular.