Essentials Unpack'd

unpackd is a tool for the Pokémon Essentials, to extract data binaries (.rxdata) to readable .rb and .yaml files and to combine them back, Thus making your game to be version-controlled and to be collaborated on.

Usage

$ unpackd.exe --help
Essentials Unpack\'d v3.0.0

`unpackd` is a tool \for Pokémon Essentials, to extract data binaries (.rxdata)
  to readable .rb and .yaml files and to combine them back, thus making
  your game to be version-controlled and to be collaborated on.

Usage:
        unpackd.exe [options] {--extract|--combine|-b|-r}

Commands:
  -e, --extract        Extract given binaries(.rxdata) into individual .yaml/.rb
  -c, --combine        Combine given .yaml/.rb files into binaries(.rxdata)

  -b, --backup         Make Backup for given binary(.rxdata) files
  -r, --revert         Revert given binary(.rxdata) from Backup Folder

Options:
  -d, --project=<s>    Essentials project path. (default: Current Folder)
  -f, --files=<s+>     File Names for .rxdata/.yaml/.rb to operate on. (Default: *)

  -F, --force          Used with `--combine` to Pack Data Forcefully
  -s, --silent         Do not output any information while processing

  -v, --version        Print version and exit
  -h, --help           Show this message

Examples

To Extract Scripts.rxdata and Tilesets.rxdata of a game in "D:\Examples\MyEssentialsGame":

$ unpackd.exe --extract --project "D:\Examples\MyEssentialsGame" --files scripts tilesets
This will create 3 Folders, Backup, Scripts, and YAML in the "D:\Examples\MyEssentialsGame\Data" folder.

Firstly, Backup files will be created in Data\Backup (*.rxdata.backup files)

Scripts.rxdata consists of many ruby scripts and these now will be extracted to individual .rb files placed in grouped folders within the Data\Scripts folder.

Scripts.rxdata will be replaced with a loader file, this file can read the individual .rb files in the Data\Scripts Folder, Therefore making Game.exe still playable! This would not work, if the game is encrypted !

Tilesets.rxdata will be extracted to a readable Tilesets.yaml file within Data\YAML folder.

If at any point, unpackd is unable to perform the extraction, mentioned Backup files will be reinstated.

To Combine { ruby scripts in Data/Scripts to Scripts.rxdata } and { Tilesets.yaml to Tilesets.rxdata }:

$ unpackd.exe --combine --project "D:\Examples\MyEssentialsGame" --files scripts tilesets
This will create 3 Folders, Backup, Scripts, and YAML in the "D:\Examples\MyEssentialsGame\Data" folder.

Firstly, Backup files will be created in Data\Backup (*.rxdata.backup files)

Will check if the Scripts.rxdata is a loader file or an already data-packed file.

If in case, it is already a packed data file, this operation will skip, unless --force flag is passed along.

Else, ruby scripts will be reintegrated back into Scripts.rxdata

Tilesets.yaml will be converted back to Tilesets.rxdata.

If at any point, the script is unable to perform a combination, mentioned Backup files will be reinstated.

To create a general Backup for Scripts.rxdata and Tilesets.rxdata:

$ unpackd.exe --backup --project "D:\Examples\MyEssentialsGame" --files scripts tilesets

To Revert an already created backup of Scripts.rxdata and Tilesets.rxdata:

$ unpackd.exe --revert --project "D:\Examples\MyEssentialsGame" --files scripts tilesets

Tips and Tricks

Put unpackd.exe in the game directory, it is a light file and it removes the need to pass the --project flag
Currently, only Tilesets and Scripts are understandable, I am not sure about the rest of the files.
To add YAML in your version control add !Data/YAML/ to .gitignore.
- ```
$ echo !Data/YAML/ >> .gitignore
```
Extracted Scripts.rxdata cannot be loaded into RPG Maker, even with the loader file, always combine it, if you plan on using the script editor of RPG Maker.
Using ---files flag again and again for the same files, can be a bit effortful, create a batch file or makefile for your workflow, I have added a makefile.template in the repo, as a base to add on.
Suppose your game throws an error, but you couldn't note it down. You try to run the game again, but now Game.exe just shows a blank screen and it exits. This happens due to caching of binary files, Now, I don't know how to delete/ignore those cache but, I know, by --combine --force again (or by --combine then --extract, if you are working with loader), Game.exe assumes it is a new binary. (Warn: This would corrupt backup)
In Case, you don't use a VCS (big mistake), be aware of backups! Suppose you made changes to a file, and combined it to run the game and it did not work (right now the file is bad but the backup is good). but if you make another change and combined forcefully, even if the file is good, the backup becomes bad!

Credits

Essentials Unpack'd is quite different from original files and libs, but Authors must be credited for the grand majority of the work that unpackd do, without them this would have not been possible.

Howard "SiCrane" Jeng for original YAML importer/exporter; serialization, data conversion.
Aaron Patterson for psych 2.0.0 bug fixes.
Andrew Kesterson for converting a simple forum post to a working version-controlled ruby gem!
Rachel Wall for code optimization and maintenance since 2014.
Maruno for all processes regarding Scripts.rxdata, extract, combine, and loader.

Dev-Installation

$ git clone https://github.com/ra101/Essentials-Unpackd.git
$ cd Essentials-Unpackd
$ gem install bundler
$ bundle install

Dev-Usage

$ bundle exec unpackd {--extract|--combine|-b|-r} [options]

Workflow

General

unpackd consists of following parts:

rgss.rb : Stub classes for serialization of RPG Maker game data
serialize.rb : core mechanice behind data processing
psych.rb: Overriding Psych lib to make output more readable
unpackd: The script you call on the frontend.

Avoiding Map Collisions

One thing that unpackd really can't help you with right now (and, ironically, probably one of the reasons you want it) is map collisions. Consider this situation:

The project has 10 maps in it, total.
Developer A makes a new map; it gets saved by the editor as 'Map011'.
Developer B makes a new map, in a different branch; it also gets saved by the editor as 'Map011'.
Developer A and Developer B attempt to merge their changes -- the merge fails because of the collision on the 'Map011' file.

The best way to avoid this is to use blocks of pre-allocated maps. You appoint one person in your project to be principally responsible for the map assets; it then becomes this person's responsibility to allocate maps in "blocks" so that people can work on maps in a distributed way without clobbering one another. The workflow looks like this:

The project has 10 maps in it, total.
Developer A needs to make 4 maps. He sends a request to the "map owner", requesting a block of 4 maps.
The map owner creates 4 default, blank maps, and names them all "Request #12345" for Developer A
Developer A starts working on his maps
Developer B needs to make 6 maps. He sends a request to the "map owner", requesting a block of 6 maps.
The map owner creates 6 default, blank maps, and names them all "Request #12346" for Developer B
Developer B starts working on his maps

Using this workflow, it doesn't matter what order Developers A and B request their map blocks in or what order the map owner creates their map blocks in. By giving the map owner the authority to create the map blocks, individual developers can work freely in their map blocks: they can rename them, reorder them, change all of the map attributes (size, tileset, and so on), without getting in danger of a map collision.

While this may seem like an unnecessary process, it is a reasonable workaround. For a better explanation of why unpackd can't do this for you, read the next section.

Automatic ID generation

You can add new elements to the YAML files manually, and leave their id: field set to null. This will cause the unpackd pack action to automatically assign them a new ID number at the end of the sequence (e.g., if you have 17 items, the new one becomes ID 18). This is mainly handy for adding new scripts to the project without having to open the RPG Maker editor and paste the script in; just make the new script file, add its entry in YAML/Scripts.yaml, and the designer will have your script accessible the next time they repack and open the project.

Also, the unpackd tool sets the ID of script files to an autoincrementing integer. The scripts exist in the database with a magic number that I can't recreate, and nothing in the editor (RPG VX Ace anyway) seems to care if the magic number changes. It doesn't even affect the ordering. So in order to support adding new scripts with null IDs, like everything else, the magic numbers on scripts are disregarded and a new ID number is forced on the scripts when the unpackd pack action occurs.

Note that this does not apply to map files; do not try changing the map ID numbers manually (see the "Avoiding Map Collisions" workflow, above, and "Why unpackd can't help with map collisions", below).

Why `unpackd` can't help with map collisions

If you look at the map collision problem described above, the way out of this situation might seem obvious: "Rename Map011.yaml in one of the branches to Map012.yaml, and problem solved." However, there are several significant problems with this approach:

The ID numbers on the map files correspond to ID number entries in MapInfos.yaml (and the corresponding MapInfos binary files)
The ID numbers are used to specify a parent/child relationship between one or more maps
The ID numbers are used to specify the target of a map transition/warp event in event scripting

This means that changing the ID number assigned to a map (and, thereby, making it possible to merge 2 maps with the same ID number) becomes very nontrivial. The event scripting portion, especially, presents a difficult problem for unpackd to overcome. It is simple enough for unpackd to change the IDs of any new map created, and to change the reference to that ID number from any child maps; however, the events are where it gets sticky. The format of event calls in RPG Maker map files is not terribly well defined, and even if it was, I sincerely doubt that you want unpackd tearing around in the guts of your map events.

Psych 2.0.0 Dependency

From SiCrane:

I used cygwin's ruby 1.9.3 and the Psych 2.0.0 ruby gem, which appears to be the most recent version. However, Psych 2.0.0 has some bugs that impacted the generated YAML (one major and one minor) which I monkey patched, and since I was already rewriting the Psych code, I added some functionality to make the generated YAML prettier. Long story short, this code probably won't work with any version of Psych but 2.0.0.