How to serialize void[]
InfiniteRegression opened this issue · 8 comments
InfiniteRegression commented
I would like to serialize a void[]. I get Error: expression value[i] is void and has no value when trying.
I imagine this is easy to overcome since the void[] is a series of bytes. It is a very large array(audio), so I'm curious if it can be done and be done using binary to avoid bloat? While I might be able to hack it myself, it might be nice if you could add it so it would be done right. Even if it simply serialized it to a binary file and use a link in the serialized output(might be more work). I realize that the binary inside an xml text file might be considered a bad idea but in my cause it wouldn't matter. I need binary. I am using xml so I'm not sure if it will even work so you might simply have to save it to a file then make a reference in the xml to that file and load it on serialization. Maybe use a special attribute to tell it to serialize to a file or something similar.
Thanks.
Stretto commented
Here is something I just through together that seems to work for the serializer:
private void serializeArray (T) (ref T value, string key, Id id)
{
static if (!is(T == const(void[])))
{
auto array = Array(value.ptr, value.length, ElementTypeOfArray!(T).sizeof);
archive.archiveArray(array, arrayToString!(T), key, id, {
for (size_t i = 0; i < value.length; i++)
{
const e = value[i];
serializeInternal(e, toData(i));
}
});
if (value.length > 0)
addSerializedArray(array, id);
} else
{
import std.stdio, std.file;
// Added by me to handle void[]
auto array = Array(value.ptr, value.length, ElementTypeOfArray!(T).sizeof);
archive.archiveArray(array, arrayToString!(T), key, id, {
if (value == null || value.length == 0) return;
// generate random file name to store binary data and use
string filename = genNonce();
while (exists(filename)) filename = genNonce();
for (size_t i = 0; i < filename.length; i++)
{
const e = filename[i];
serializeInternal(e, toData(i));
}
// save binary data to file
std.file.write(filename, value);
});
if (value.length > 0) addSerializedArray(array, id);
}
}
string genNonce()
{
import std.algorithm, std.ascii, std.base64, std.conv, std.random, std.range;
auto rndNums = rndGen().map!(a => cast(ubyte)a)().take(32);
auto result = appender!string();
Base64.encode(rndNums, result);
return result.data.filter!isAlphaNum().to!string();
}
While I might continue to work on it to make it better, I think it has 2 major issues that should be address.
- The file name is stored as an array, it might be better to store it directly as a string as to reduce bloat.
- For small array's(say less than 100) it might be better to inline them directly as a string and use a base64 conversion scheme. To do that, an attribute will need to be added to the xml to distinguish this case.
- Instead of generating random file names, maybe better to generate a hash of some fixed property(type, path of type, tag, etc) so that one doesn't accumulate files.
In fact, maybe the inlining as a base64 string should be or something better (7-bit ascii with 8 bit being an extension so no encoding or decoding is required for normal alphanumeric).
Anyways, I'm not sure how to do that so I'll leave it up to you. The deserialization must also take place.
InfiniteRegression commented
Here is the completed serializer that will handle short arrays, inline representation, external file representation using static path. Uses id and hash of key and id to generate the file name, would better to use the full path if you can get at that(e.g., rather than just the fieldname, the class name, etc) but since id should be unique, it shouldn't matter, although changing the structure being serialized will result in orphaned files.
private void serializeArray (T) (ref T value, string key, Id id)
{
static if (!is(T == const(void[])))
{
auto array = Array(value.ptr, value.length, ElementTypeOfArray!(T).sizeof);
archive.archiveArray(array, arrayToString!(T), key, id, {
for (size_t i = 0; i < value.length; i++)
{
const e = value[i];
serializeInternal(e, toData(i));
}
});
if (value.length > 0)
addSerializedArray(array, id);
} else
{
import std.stdio, std.file, std.base64, std.conv;
auto array = Array(value.ptr, value.length, ElementTypeOfArray!(T).sizeof);
archive.archiveArray(array, arrayToString!(T), key, id, {
if (value == null || value.length == 0) return;
if (value.length < 250)
{
auto result = appender!string();
Base64.encode(cast(byte[])value, result);
auto res = result.data;
serializeInternal(res, toData(id));
} else
{
auto filename = to!string(id)~"_"~to!string(hashOf(key~to!string(id)),16)~".xmldata";
if (exists(filename)) remove(filename);
serializeInternal(filename, toData(id));
std.file.write(filename, value);
}
});
if (value.length > 0) addSerializedArray(array, id);
}
}
InfiniteRegression commented
Here is the deserializer. You might want to have extend this for very long arrays of ints, floats, etc. This way one could use the serializer as a sort of one stop shop.. Could save images, etc. This would reduce the file size significantly since arrays are stored per element it seems(so an array of length N will have something like M*N the size where M is all the extra xml overhead for the element).
Maybe use attributes to specify that the field is to be serialized to an external file.
private T deserializeArray (T) (string key, out Id id)
{
auto slice = deserializeSlice(key);
if (auto tmp = getDeserializedSlice!(T)(slice))
return tmp;
alias ElementTypeOfArray!(T) E;
alias Unqual!(E) UnqualfiedE;
static if (!is(T == void[]))
{
UnqualfiedE[] buffer;
T value;
auto dg = (size_t length) {
buffer.length = length;
foreach (i, ref e ; buffer)
e = deserializeInternal!(typeof(e))(toData(i));
};
if (slice.id != size_t.max) // Deserialize slice
{
id = slice.id;
archive.unarchiveArray(slice.id, dg);
assumeUnique(buffer, value);
addDeserializedSlice(value, slice.id);
return toSlice(value, slice);
}
else // Deserialize array
{
id = archive.unarchiveArray(key, dg);
if (auto arr = id in deserializedSlices)
return cast(T) *arr;
assumeUnique(buffer, value);
addDeserializedSlice(value, id);
return value;
}
} else
{
import std.stdio, std.file, std.base64, std.conv;
void[] value;
auto dg = (size_t length) {
if (length < _MaxSerializedVoidArray)
{
string data = deserializeInternal!(string)("_inlineArray");
value = cast(void[])Base64.decode(data);
} else
{
// Load data from file
auto filename = deserializeInternal!(string)("filename");
auto id = slice.id;
if (!exists(filename)) assert("xml data not found for `"~filename~"`!");
if (getSize(filename) != length) assert("xml data not correct size for `"~filename~"`! Expected "~to!string(value.length)~" got "~to!string(getSize(filename))~".");
value = cast(void[])std.file.read(filename);
}
};
if (slice.id != size_t.max) // Deserialize slice
{
id = slice.id;
archive.unarchiveArray(slice.id, dg);
addDeserializedSlice(value, slice.id);
return toSlice(value, slice);
}
else // Deserialize array
{
id = archive.unarchiveArray(key, dg);
if (auto arr = id in deserializedSlices)
return cast(void[])*arr;
addDeserializedSlice(value, id);
return value;
}
}
}
jacob-carlborg commented
I'm not sure that I'm that comfortable supporting serializing void[]. void[] is basically an unknown type. For dealing with bytes I would recommend byte[] or ubyte[]. Byte arrays could be treated specially, like doing base 64 encoding or other transformations.
InfiniteRegression commented
A void[] is just data, that gets serialized as bytes and back. There can be no harm in doing so because that is all there is. I'm sure you do not serialize byte[] to a file though, do you? Or any array, and that is important to me because I store audio data and image data in arrays. They make the xml file extremely large because of how inefficient arrays are.
You could always add an attribute to get them to serialize. Not supporting something based on a feeling will just screw those that don't have that feeling. The fact of the matter is that now, without being able to serialize a void[], a loss of data situation is created, and that is, IMO, unacceptable(of course, it does err out, but then that leads to one having to jump through hoops to get what they want done).
jacob-carlborg commented
A void[] is just data, that gets serialized as bytes and back. There can be no harm in doing so because that is all there is.
No: "There is a special type of array which acts as a wildcard that can hold arrays of any kind, declared as void[]" [1]. If you want to store just bytes then use ubyte[] or byte[]. If you store pointers or references in a void array it won't be able to properly (de)serialize that.
I'm sure you do not serialize byte[] to a file though, do you?
No. The serialized XML would most likely end up in a file anyway. Why add another file?
Or any array, and that is important to me because I store audio data and image data in arrays. They make the xml file extremely large because of how inefficient arrays are.
I could special case ubyte/byte arrays to serialize in a different more efficient format, but not void arrays.
Not supporting something based on a feeling will just screw those that don't have that feeling.
It's not a feeling. Serializing arbitrary void arrays won't work.
What's wrong with using a type that is intended to store bytes?
InfiniteRegression commented
The problem is that the array can be any type. If I specify bytes then it is assumed it is a byte. The arrays are audio data and the type depends on runtime behavior(what the user selected, what the audio interface uses, etc). Your arguments are not meaningful. bytes can hold pointers, one can have an array of int's that are pointers too. So you are just picking and choosing. byte[]'s and void[] are identical as they are just logical differences, not physical. Everything is a bit in programming.
If you don't like the idea of automatically serializing a void[] then at least allow a way to force it instead of making the assumption that it is bad. It is not bad in all cases so why make me jump through hoops to do what I need to do?
@serializeAs(byte[])
void[] data;
In which it treats it as a byte[].
That kinda thing would be fine for me. But to prevent it all together is not good. Of course, I've already modified it to do what I want, so all this is moot to some degree. If I knew how to use attributes with orange I'd probably add that ability.
It's true, I could just use byte[] instead and cast to whatever type I need to, but void makes it explicit to me that it's not actually meant to be a byte value(which exist for audio).
When you serialize arrays one end sup with an xml tag per element. The tag is very long, about 50 times the size of the element. That ends up exploding the xml array. e.g., suppose I want to store an mp3. 5MB = 250MB. Storing it to an external file will only still be about 5MB.
Same would go with images and other stuff.
I'd rather have to be a little explicit in getting what I want orange to do then it prevent me from being able to do it at all. I'll probably write my own serializer at some point anyways, so I'm not too concerned with what you want to do. I've already customized orange to do what I need and it was pretty easy. Seems to work well, I might continue using it, I'm not sure. I just figured that others might benefit from having the ability to do what I wanted. Since the code is already added, you could simply make it optional or add some attributes to enable it.
InfiniteRegression commented
I guess nothing. You could at least try out the code
/**
* Copyright: Copyright (c) 2010-2011 Jacob Carlborg.
* Authors: Jacob Carlborg
* Version: Initial created: Jan 26, 2010
* License: $(LINK2 http://www.boost.org/LICENSE_1_0.txt, Boost Software License 1.0)
*/
module orange.serialization.Serializer;
import std.conv;
import std.array;
import orange.core.Attribute;
import orange.serialization._;
import orange.serialization.archives.Archive;
import orange.util._;
import orange.util.collection.Array;
enum _MaxSerializedVoidArray = 1000;
enum _filenameKey = "___filename___";
enum _inlineArrayKey = "___inlineArray___";
private
{
alias orange.util.CTFE.contains ctfeContains;
enum Mode
{
serializing,
deserializing
}
alias Mode.serializing serializing;
alias Mode.deserializing deserializing;
private char toUpper () (char c)
{
if (c >= 'a' && c <= 'z')
return cast(char) (c - 32);
return c;
}
}
/**
* This class represents a serializer. It's the main interface to the (de)serialization
* process and it's this class that actually performs most of the (de)serialization.
*
* The serializer is the frontend in the serialization process, it's independent of the
* underlying archive type. It's responsible for collecting and tracking all values that
* should be (de)serialized. It's the serializer that adds keys and ID's to all values,
* keeps track of references to make sure that a given value (of reference type) is only
* (de)serialized once.
*
* The serializer is also responsible for breaking up types that the underlying archive
* cannot handle, into primitive types that archive know how to (de)serialize.
*
* Keys are used by the serializer to associate a name with a value. It's used to
* deserialize values independently of the order of the fields of a class or struct.
* They can also be used by the user to give a name to a value. Keys are unique within
* it's scope.
*
* ID's are an unique identifier associated with each serialized value. The serializer
* uses the ID's to track values when (de)serializing reference types. An ID is unique
* across the whole serialized data.
*
* Examples:
* ---
* import std.stdio;
* import orange.serialization._;
* import orange.serialization.archives._;
*
* class Foo
* {
* int a;
* }
*
* void main ()
* {
* auto archive = new XmlArchive!();
* auto serializer = new Serializer;
*
* auto foo = new Foo;
* foo.a = 3;
*
* serializer.serialize(foo);
* auto foo2 = serializer.deserialize!(Foo)(archive.untypedData);
*
* writeln(foo2.a); // prints "3"
* assert(foo.a == foo2.a);
* }
* ---
*/
class Serializer
{
/// The type of error callback.
alias Archive.ErrorCallback ErrorCallback;
/// The type of the serialized data. This is an untyped format.
alias Archive.UntypedData Data;
/// The type of an ID.
alias Archive.Id Id;
/**
* This callback will be called when an unexpected event occurs, i.e. an expected element
* is missing in the deserialization process.
*
* Examples:
* ---
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
* serializer.errorCallback = (SerializationException exception) {
* writeln(exception);
* throw exception;
* };
* ---
*/
ErrorCallback errorCallback ()
{
return archive.errorCallback;
}
/**
* This callback will be called when an unexpected event occurs, i.e. an expected element
* is missing in the deserialization process.
*
* Examples:
* ---
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
* serializer.errorCallback = (SerializationException exception) {
* writeln(exception);
* throw exception;
* };
* ---
*/
ErrorCallback errorCallback (ErrorCallback errorCallback)
{
return archive.errorCallback = errorCallback;
}
private
{
struct ValueMeta
{
Id id = Id.max;
string key;
bool isValid ()
{
return id != Id.max && key.length > 0;
}
}
static
{
void function (Serializer serializer, in Object, Mode mode) [ClassInfo] registeredTypes;
RegisterBase[string] serializers;
RegisterBase[string] deserializers;
}
Archive archive_;
size_t keyCounter;
Id idCounter;
RegisterBase[string] overriddenSerializers;
RegisterBase[string] overriddenDeserializers;
Id[void*] serializedReferences;
void*[Id] deserializedReferences;
Array[Id] serializedArrays;
void[][Id] deserializedSlices;
void**[Id] deserializedPointers;
ValueMeta[void*] serializedValues;
const(void)*[Id] deserializedValues;
bool hasBegunSerializing;
bool hasBegunDeserializing;
void delegate (SerializationException exception) throwOnErrorCallback;
void delegate (SerializationException exception) doNothingOnErrorCallback;
Mode mode;
}
/**
* Creates a new serializer using the given archive.
*
* The archive is the backend of the (de)serialization process, it performs the low
* level (de)serialization of primitive values and it decides the final format of the
* serialized data.
*
* Params:
* archive = the archive that should be used for this serializer
*
* Examples:
* ---
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
* ---
*/
this (Archive archive)
{
this.archive_ = archive;
throwOnErrorCallback = (SerializationException exception) { throw exception; };
doNothingOnErrorCallback = (SerializationException exception) { /* do nothing */ };
setThrowOnErrorCallback();
}
/**
* Registers the given type for (de)serialization.
*
* This method is used for register classes that will be (de)serialized through base
* class references, no other types need to be registered. If the the user tries to
* (de)serialize an instance through a base class reference which runtime type is not
* registered an exception will be thrown.
*
* Params:
* T = the type to register, must be a class
*
* Examples:
* ---
* class Base {}
* class Sub : Base {}
*
* Serializer.register!(Sub);
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* Base b = new Sub;
* serializer.serialize(b);
* ---
*
* See_Also: registerSerializer
* See_Also: registerDeserializer
*/
static void register (T : Object) ()
{
registeredTypes[T.classinfo] = &downcastSerialize!(T);
}
private static void downcastSerialize (U : Object) (Serializer serializer, in Object value, Mode mode)
{
alias Unqual!(U) T;
static if (!isNonSerialized!(T)())
{
auto casted = cast(T) value;
assert(casted);
assert(casted.classinfo is T.classinfo);
if (mode == serializing)
serializer.objectStructSerializeHelper(casted);
else
serializer.objectStructDeserializeHelper(casted);
}
}
/**
* Registers a serializer for the given type.
*
* The given callback will be called when a value of the given type is about to
* be serialized. This method can be used as an alternative to $(I register). This
* method can also be used as an alternative to Serializable.toData.
*
* This is method should also be used to perform custom serialization of third party
* types or when otherwise chaining an already existing type is not desired.
*
* Params:
* dg = the callback that will be called when value of the given type is about to be serialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* auto dg = (Base value, Serializer serializer, Data key) {
* // perform serialization
* };
*
* Serializer.registerSerializer!(Foo)(dg);
* ---
*
* See_Also: register
* See_Also: registerDeserializer
* See_Also: Serializable.toData
*/
static void registerSerializer (Derived, Base) (void delegate (Base, Serializer, Data) dg)
{
Serializer.serializers[typeid(Derived).toString] = toSerializeRegisterWrapper(dg);
}
/**
* Registers a serializer for the given type.
*
* The given callback will be called when a value of the given type is about to
* be serialized. This method can be used as an alternative to $(I register). This
* method can also be used as an alternative to Serializable.toData.
*
* This is method should also be used to perform custom serialization of third party
* types or when otherwise chaining an already existing type is not desired.
*
* Params:
* dg = the callback that will be called when value of the given type is about to be serialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* void func (Base value, Serializer serializer, Data key) {
* // perform serialization
* }
*
* Serializer.registerSerializer!(Foo)(&func);
* ---
*
* See_Also: register
* See_Also: registerDeserializer
* See_Also: Serializable.toData
*/
static void registerSerializer (Derived, Base) (void function (Base, Serializer, Data) func)
{
Serializer.serializers[typeid(Derived).toString] = toSerializeRegisterWrapper(func);
}
/**
* Registers a deserializer for the given type.
*
* The given callback will be called when a value of the given type is about to
* be deserialized. This method can be used as an alternative to $(I register). This
* method can also be used as an alternative to Serializable.fromData.
*
* This is method should also be used to perform custom deserialization of third party
* types or when otherwise chaining an already existing type is not desired.
*
* Params:
* dg = the callback that will be called when value of the given type is about to be deserialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* auto dg = (ref Base value, Serializer serializer, Data key) {
* // perform deserialization
* };
*
* Serializer.registerDeserializer!(Foo)(dg);
* ---
*
* See_Also: register
* See_Also: registerSerializer
* See_Also: Serializable.fromData
*/
static void registerDeserializer (Derived, Base) (void delegate (ref Base, Serializer, Data) dg)
{
Serializer.deserializers[typeid(Derived).toString] = toDeserializeRegisterWrapper(dg);
}
/**
* Registers a deserializer for the given type.
*
* The given callback will be called when a value of the given type is about to
* be deserialized. This method can be used as an alternative to $(I register). This
* method can also be used as an alternative to Serializable.fromData.
*
* This is method should also be used to perform custom deserialization of third party
* types or when otherwise chaining an already existing type is not desired.
*
* Params:
* dg = the callback that will be called when value of the given type is about to be deserialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* void func (ref Base value, Serializer serializer, Data key) {
* // perform deserialization
* }
*
* Serializer.registerDeserializer!(Foo)(&func);
* ---
*
* See_Also: register
* See_Also: registerSerializer
* See_Also: Serializable.fromData
*/
static void registerDeserializer (Derived, Base) (void function (ref Base, Serializer, Data) func)
{
Serializer.deserializers[typeid(Derived).toString] = toDeserializeRegisterWrapper(func);
}
/**
* Overrides a globally registered serializer for the given type with a serializer
* local to the receiver.
*
* The receiver will first check if a local serializer is registered, otherwise a global
* serializer will be used (if available).
*
* Params:
* dg = the callback that will be called when value of the given type is about to be serialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* auto dg = (Base value, Serializer serializer, Data key) {
* // perform serialization
* };
*
* Serializer.registerSerializer!(Foo)(dg);
*
* auto overrideDg = (Base value, Serializer serializer, Data key) {
* // this will override the above serializer
* }
*
* serializer.overrideSerializer!(Foo)(overrideDg);
* ---
*/
void overrideSerializer (Derived, Base) (void delegate (Base, Serializer, Data) dg)
{
overriddenSerializers[typeid(Derived).toString] = toSerializeRegisterWrapper(dg);
}
/**
* Overrides a globally registered serializer for the given type with a serializer
* local to the receiver.
*
* The receiver will first check if a local serializer is registered, otherwise a global
* serializer will be used (if available).
*
* Params:
* dg = the callback that will be called when value of the given type is about to be serialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* void func (Base value, Serializer serializer, Data key) {
* // perform serialization
* }
*
* Serializer.registerSerializer!(Foo)(&func);
*
* void overrideFunc (Base value, Serializer serializer, Data key) {
* // this will override the above serializer
* }
*
* serializer.overrideSerializer!(Foo)(&overrideFunc);
* ---
*/
void overrideSerializer (Derived, Base) (void function (Base, Serializer, Data) func)
{
overriddenSerializers[typeid(Derived).toString] = toSerializeRegisterWrapper(func);
}
/**
* Overrides a globally registered deserializer for the given type with a deserializer
* local to the receiver.
*
* The receiver will first check if a local deserializer is registered, otherwise a global
* deserializer will be used (if available).
*
* Params:
* dg = the callback that will be called when value of the given type is about to be deserialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* auto dg = (ref Base value, Serializer serializer, Data key) {
* // perform deserialization
* };
*
* Serializer.registerSerializer!(Foo)(dg);
*
* auto overrideDg = (ref Base value, Serializer serializer, Data key) {
* // this will override the above deserializer
* };
*
* serializer.overrideSerializer!(Foo)(overrideDg);
* ---
*/
void overrideDeserializer (Derived, Base) (void delegate (ref Base, Serializer, Data) dg)
{
overriddenDeserializers[typeid(Derived).toString] = toDeserializeRegisterWrapper(dg);
}
/**
* Overrides a globally registered deserializer for the given type with a deserializer
* local to the receiver.
*
* The receiver will first check if a local deserializer is registered, otherwise a global
* deserializer will be used (if available).
*
* Params:
* dg = the callback that will be called when value of the given type is about to be deserialized
*
* Examples:
* ---
* class Base {}
* class Foo : Base {}
*
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* void func (ref Base value, Serializer serializer, Data key) {
* // perform deserialization
* }
*
* Serializer.registerSerializer!(Foo)(&func);
*
* void overrideFunc (ref Base value, Serializer serializer, Data key) {
* // this will override the above deserializer
* }
*
* serializer.overrideSerializer!(Foo)(&overrideFunc);
* ---
*/
void overrideDeserializer (Derived, Base) (void function (ref Base, Serializer, Data) func)
{
overriddenDeserializers[typeid(Derived).toString] = toDeserializeRegisterWrapper(func);
}
/// Returns the receivers archive
Archive archive ()
{
return archive_;
}
/**
* Set the error callback to throw when an error occurs
*
* See_Also: setDoNothingOnErrorCallback
*/
void setThrowOnErrorCallback ()
{
errorCallback = throwOnErrorCallback;
}
/**
* Set the error callback do nothing when an error occurs
*
* See_Also: setThrowOnErrorCallback
*/
void setDoNothingOnErrorCallback ()
{
errorCallback = doNothingOnErrorCallback;
}
/**
* Resets all registered types registered via the "register" method
*
* See_Also: register
*/
static void resetRegisteredTypes ()
{
registeredTypes = null;
}
/**
* Resets all registered (de)serializers registered via the "registerSerializer" method.
* This method will not reset the overridden (de)serializers.
*
* See_Also: registerSerializer
* See_Also: registerDeserializer
*/
static void resetSerializers ()
{
serializers = null;
deserializers = null;
}
/**
* Resets the serializer.
*
* All internal data is reset, including the archive. After calling this method the
* serializer can be used to start a completely new (de)serialization process.
*/
void reset ()
{
resetCounters();
overriddenSerializers = null;
overriddenDeserializers = null;
serializedReferences = null;
deserializedReferences = null;
serializedArrays = null;
deserializedSlices = null;
serializedValues = null;
deserializedValues = null;
deserializedPointers = null;
hasBegunSerializing = false;
hasBegunDeserializing = false;
archive.reset;
mode = Mode.init;
}
/**
* Serializes the given value.
*
* Params:
* value = the value to serialize
* key = associates the value with the given key. This key can later be used to
* deserialize the value
*
* Examples:
* ---
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* serializer.serialize(1);
* serializer.serialize(2, "b");
* ---
*
* Returns: return the serialized data, in an untyped format.
*
* Throws: SerializationException if an error occurs
*/
Data serialize (T) (T value, string key = null)
{
mode = serializing;
if (!hasBegunSerializing)
hasBegunSerializing = true;
serializeInternal(value, key);
postProcess;
return archive.untypedData;
}
/**
* Serializes the base class(es) of an instance.
*
* This method is used when performing custom serialization of a given type. If this
* method is not called when performing custom serialization none of the instance's
* base classes will be serialized.
*
* Params:
* value = the instance which base class(es) should be serialized, usually $(D_CODE this)
*
* Examples:
* ---
* class Base {}
* class Sub : Base
* {
* void toData (Serializer serializer, Serializer.Data key)
* {
* // perform serialization
* serializer.serializeBase(this);
* }
* }
* ---
*
* Throws: SerializationException if an error occurs
*/
void serializeBase (T) (T value)
{
static if (isObject!(T) && !is(Unqual!(T) == Object))
serializeBaseTypes(value);
}
private void serializeInternal (U) (ref U value, string key = null, Id id = Id.max)
{
alias Unqual!(U) T;
if (!key)
key = nextKey;
if (id == Id.max)
id = nextId;
archive.beginArchiving();
static if (isObject!(T))
serializeObject(value, key, id);
else static if (isStruct!(T))
serializeStruct(value, key, id);
else static if (isString!(T))
serializeString(value, key, id);
else static if (isArray!(T))
serializeArray(value, key, id);
else static if (isAssociativeArray!(T))
serializeAssociativeArray(value, key, id);
else static if (isPrimitive!(T))
serializePrimitive(value, key, id);
else static if (isPointer!(T))
{
static if (isFunctionPointer!(T))
error(format!(`The type "`, T, `" cannot be serialized.`));
else
serializePointer(value, key, id);
}
else static if (isEnum!(T))
serializeEnum(value, key, id);
}
private void serializeObject (T) (T value, string key, Id id) if (isObject!(T))
{
auto typeName = typeid(T).toString;
static if (!isNonSerialized!(T)())
{
if (!value)
return archive.archiveNull(typeName, key);
auto reference = getSerializedReference(value);
if (reference != Id.max)
return archive.archiveReference(key, reference);
auto untypedValue = cast(Object) value;
auto runtimeType = untypedValue.classinfo.name;
addSerializedReference(value, id);
triggerEvents(value, {
archive.archiveObject(runtimeType, typeName, key, id, {
if (auto serializer = runtimeType in overriddenSerializers)
callSerializer(serializer, value, key);
else if (auto serializer = runtimeType in Serializer.serializers)
callSerializer(serializer, value, key);
else static if (isSerializable!(T))
{
pragma(msg, T);
pragma(msg, typeof(this));
pragma(msg, typeof(key));
pragma(msg, typeof(value));
value.toData(this, key);
}
else
{
if (isInterface!(T) || isBaseClass(value))
{
if (auto serializer = untypedValue.classinfo in registeredTypes)
(*serializer)(this, untypedValue, serializing);
else
error(`The object of the static type "` ~ typeName ~
`" have a different runtime type (` ~ runtimeType ~
`) and therefore needs to either register its type or register a serializer for its type "`
~ runtimeType ~ `".`);
}
else
objectStructSerializeHelper(value);
}
});
});
}
}
private void serializeStruct (T) (T value, string key, Id id)
{
static if (!isNonSerialized!(T)())
{
string type = typeid(T).toString;
triggerEvents(value, {
archive.archiveStruct(type, key, id, {
if (auto serializer = type in overriddenSerializers)
callSerializer(serializer, value, key);
else if (auto serializer = type in Serializer.serializers)
callSerializer(serializer, value, key);
else
{
static if (isSerializable!(T))
value.toData(this, key);
else
objectStructSerializeHelper(value);
}
});
});
}
}
private void serializeString (T) (T value, string key, Id id)
{
auto array = Array(cast(void*) value.ptr, value.length, ElementTypeOfArray!(T).sizeof);
archive.archive(value, key, id);
if (value.length > 0)
addSerializedArray(array, id);
}
private void serializeArray (T) (ref T value, string key, Id id)
{
alias ElementTypeOfArray!(T) E;
alias Unqual!(E) UnqualfiedE;
auto array = Array(value.ptr, value.length, ElementTypeOfArray!(T).sizeof);
archive.archiveArray(array, arrayToString!(T), key, id,
{
static if (!(is(UnqualfiedE == void) || is(isScalarType!UnqualfiedE)))
{
for (size_t i = 0; i < value.length; i++)
{
const e = value[i];
serializeInternal(e, toData(i));
}
} else
{
import std.stdio, std.file, std.base64, std.conv, std.traits;
static if (!isStaticArray!T)
if (value == null || value.length == 0) return;
if (value.length < _MaxSerializedVoidArray)
{
auto result = appender!string();
Base64.encode(cast(byte[])value, result);
auto res = result.data;
serializeInternal(res, _inlineArrayKey);
} else
{
auto filename = to!string(id)~"_"~to!string(hashOf(key~to!string(id)),16)~".xmldat";
if (exists(filename)) remove(filename);
serializeInternal(filename, _filenameKey);
std.file.write(filename, value);
}
}
});
if (value.length > 0) addSerializedArray(array, id);
}
private void serializeAssociativeArray (T) (T value, string key, Id id)
{
auto reference = getSerializedReference(value);
if (reference != Id.max)
return archive.archiveReference(key, reference);
addSerializedReference(value, id);
string keyType = typeid(KeyTypeOfAssociativeArray!(T)).toString;
string valueType = typeid(ValueTypeOfAssociativeArray!(T)).toString;
archive.archiveAssociativeArray(keyType, valueType, value.length, key, id, {
size_t i;
foreach(k, v ; value)
{
archive.archiveAssociativeArrayKey(toData(i), {
serializeInternal(k, toData(i));
});
archive.archiveAssociativeArrayValue(toData(i), {
serializeInternal(v, toData(i));
});
i++;
}
});
}
private void serializePointer (T) (T value, string key, Id id)
{
if (!value)
return archive.archiveNull(typeid(T).toString, key);
auto reference = getSerializedReference(value);
if (reference != Id.max)
return archive.archiveReference(key, reference);
archive.archivePointer(key, id, {
if (auto serializer = key in overriddenSerializers)
callSerializer(serializer, value, key);
else if (auto serializer = key in Serializer.serializers)
callSerializer(serializer, value, key);
else static if (isSerializable!(T))
value.toData(this, key);
else
{
static if (isVoid!(BaseTypeOfPointer!(T)))
error(`The value with the key "` ~ to!(string)(key) ~ `"` ~
format!(` of the type "`, T, `" cannot be serialized on `,
`its own, either implement orange.serialization.Serializable`,
`.isSerializable or register a serializer.`));
else
{
auto valueMeta = getSerializedValue(value);
if (valueMeta.isValid)
archive.archiveReference(nextKey, valueMeta.id);
else
serializeInternal(*value, nextKey);
}
}
});
addSerializedReference(value, id);
}
private void serializeEnum (T) (T value, string key, Id id)
{
alias BaseTypeOfEnum!(T) EnumBaseType;
auto val = cast(EnumBaseType) value;
string type = typeid(T).toString;
archive.archiveEnum(val, type, key, id);
}
private void serializePrimitive (T) (T value, string key, Id id)
{
archive.archive(value, key, id);
}
/**
* Deserializes the given data to value of the given type.
*
* This is the main method used for deserializing data.
*
* Examples:
* ---
* auto archive = new XmlArchive!();
* auto serializer = new Serializer(archive);
*
* auto data = serializer.serialize(1);
* auto i = serializer.deserialize!(int)(data);
*
* assert(i == 1);
* ---
*
* Params:
* T = the type to deserialize the data into
* data = the serialized untyped data to deserialize
* key = the key associate with the value that was used during serialization.
* Do not specify a key if no key was used during serialization.
*
* Returns: the deserialized value. A different runtime type can be returned
* if the given type is a base class.
*
* Throws: SerializationException if an error occurs
*/
T deserialize (T) (Data data, string key = "")
{
mode = deserializing;
if (hasBegunSerializing && !hasBegunDeserializing)
resetCounters();
if (!hasBegunDeserializing)
hasBegunDeserializing = true;
if (key.empty())
key = nextKey;
archive.beginUnarchiving(data);
auto value = deserializeInternal!(T)(key);
deserializingPostProcess;
return value;
}
/**
* Deserializes the value with the given associated key.
*
* This method should only be called when performing custom an deserializing a value
* that is part of an class or struct. If this method is called before that actual
* deserialization process has begun an SerializationException will be thrown.
* Use this method if a key was specified during the serialization process.
*
* Examples:
* ---
* class Foo
* {
* int a;
*
* void fromData (Serializer serializer, Serializer.Data key)
* {
* a = serializer!(int)("a");
* }
* }
* ---
*
* Params:
* key = the key associate with the value that was used during serialization.
*
* Returns: the deserialized value. A different runtime type can be returned
* if the given type is a base class.
*
* Throws: SerializationException if this method is called before
* the actual deserialization process has begun.
*
* Throws: SerializationException if an error occurs
*/
T deserialize (T) (string key)
{
if (!hasBegunDeserializing)
error("Cannot deserialize without any data, this method should" ~
"only be called after deserialization has begun.");
return deserialize!(T)(archive.untypedData, key);
}
/**
* Deserializes the value with the given associated key.
*
* This method should only be called when performing custom an deserializing a value
* that is part of an class or struct. If this method is called before that actual
* deserialization process has begun an SerializationException will be thrown.
* Use this method if no key was specified during the serialization process.
*
* Examples:
* ---
* class Foo
* {
* int a;
*
* void fromData (Serializer serializer, Serializer.Data key)
* {
* a = serializer!(int)();
* }
* }
* ---
*
* Params:
* key = the key associate with the value that was used during serialization.
*
* Returns: the deserialized value. A different runtime type can be returned
* if the given type is a base class.
*
* Throws: SerializationException if this method is called before
* the actual deserialization process has begun.
*
* Throws: SerializationException if an error occurs
*/
T deserialize (T) ()
{
return deserialize!(T)("");
}
/**
* Deserializes the base class(es) of an instance.
*
* This method is used when performing custom deserialization of a given type. If this
* method is not called when performing custom deserialization none of the instance's
* base classes will be serialized.
*
* Params:
* value = the instance which base class(es) should be deserialized,
* usually $(D_CODE this)
*
* Examples:
* ---
* class Base {}
* class Sub : Base
* {
* void fromData (Serializer serializer, Serializer.Data key)
* {
* // perform deserialization
* serializer.deserializeBase(this);
* }
* }
* ---
*
* Throws: SerializationException if an error occurs
*/
void deserializeBase (T) (T value)
{
static if (isObject!(T) && !is(Unqual!(T) == Object))
deserializeBaseTypes(value);
}
private Unqual!(U) deserializeInternal (U, Key) (Key keyOrId)
{
Id dummy;
return deserializeInternal!(U, Key)(keyOrId, dummy);
}
private Unqual!(U) deserializeInternal (U, Key) (Key keyOrId, out Id id)
{
alias Unqual!(U) T;
static if (isObject!(T))
return deserializeObject!(T)(keyOrId, id);
else static if (isStruct!(T))
return deserializeStruct!(T)(keyOrId, id);
else static if (isString!(T))
return deserializeString!(T)(keyOrId, id);
else static if (isStaticArray!(T))
return deserializeStaticArray!(T)(keyOrId, id);
else static if (isArray!(T))
return deserializeArray!(T)(keyOrId, id);
else static if (isAssociativeArray!(T))
return deserializeAssociativeArray!(T)(keyOrId, id);
else static if (isPrimitive!(T))
return deserializePrimitive!(T)(keyOrId, id);
else static if (isPointer!(T))
{
static if (isFunctionPointer!(T))
goto error;
return deserializePointer!(T)(keyOrId, id).value;
}
else static if (isEnum!(T))
return deserializeEnum!(T)(keyOrId, id);
else
{
error:
error(format!(`The type "`, T, `" cannot be deserialized.`));
}
}
private Unqual!(U) deserializeObject (U, Key) (Key keyOrId, out Id id)
{
alias Unqual!(U) T;
static if (!isNonSerialized!(T)())
{
id = deserializeReference(keyOrId);
if (auto reference = getDeserializedReference!(T)(id))
return *reference;
T value;
Object untypedValue;
nextId;
archive.unarchiveObject(keyOrId, id, untypedValue, {
value = cast(T) untypedValue;
addDeserializedReference(value, id);
assert(untypedValue !is null, format!("Failed to unarchive object of type '", T, "' with key '") ~ to!(string)(keyOrId) ~ "'");
triggerEvents(value, {
auto runtimeType = untypedValue.classinfo.name;
auto runHelper = false;
static if (isString!(Key))
{
if (auto deserializer = runtimeType in overriddenDeserializers)
callSerializer(deserializer, value, keyOrId);
else if (auto deserializer = runtimeType in Serializer.deserializers)
callSerializer(deserializer, value, keyOrId);
else static if (isSerializable!(T))
value.fromData(this, keyOrId);
else
runHelper = true;
}
else
runHelper = true;
if (runHelper)
{
if (isInterface!(T) || isBaseClass(value))
{
if (auto deserializer = untypedValue.classinfo in registeredTypes)
(*deserializer)(this, untypedValue, deserializing);
else
error(`The object of the static type "` ~ typeid(T).toString ~
`" have a different runtime type (` ~ runtimeType ~
`) and therefore needs to either register its type or register a deserializer for its type "`
~ runtimeType ~ `".`);
}
else
objectStructDeserializeHelper(value);
}
});
});
return value;
}
return T.init;
}
private T deserializeStruct (T, U) (U key, out Id id)
{
T value;
static if (!isNonSerialized!(T)())
{
nextId;
id = archive.unarchiveStruct(key, {
triggerEvents(value, {
auto type = toData(typeid(T).toString);
auto runHelper = false;
static if (isString!(U))
{
if (auto deserializer = type in overriddenDeserializers)
callSerializer(deserializer, value, key);
else if (auto deserializer = type in Serializer.deserializers)
callSerializer(deserializer, value, key);
else
runHelper = true;
}
else
runHelper = true;
if (runHelper)
{
static if (isSerializable!(T))
value.fromData(this, key);
else
objectStructDeserializeHelper(value);
}
});
});
}
return value;
}
private T deserializeString (T) (string key, out Id id)
{
auto slice = deserializeSlice(key);
if (auto tmp = getDeserializedSlice!(T)(slice))
return tmp;
T value;
if (slice.id != size_t.max)
{
static if (is(T == string))
value = toSlice(archive.unarchiveString(slice.id), slice);
else static if (is(T == wstring))
value = toSlice(archive.unarchiveWstring(slice.id), slice);
else static if (is(T == dstring))
value = toSlice(archive.unarchiveDstring(slice.id), slice);
}
else
{
static if (is(T == string))
value = archive.unarchiveString(key, slice.id);
else static if (is(T == wstring))
value = archive.unarchiveWstring(key, slice.id);
else static if (is(T == dstring))
value = archive.unarchiveDstring(key, slice.id);
}
addDeserializedSlice(value, slice.id);
id = slice.id;
return value;
}
private T deserializeArray (T) (string key, out Id id)
{
auto slice = deserializeSlice(key);
if (auto tmp = getDeserializedSlice!(T)(slice))
return tmp;
alias ElementTypeOfArray!(T) E;
alias Unqual!(E) UnqualfiedE;
T value;
UnqualfiedE[] buffer;
import std.stdio, std.file, std.base64, std.conv;
// Called to serialize array elements
auto dg = (size_t length)
{
static if (!(is(UnqualfiedE == void) || is(isScalarType!UnqualfiedE)))
{
buffer.length = length;
foreach (i, ref e ; buffer)
e = deserializeInternal!(typeof(e))(toData(i));
}
else
{
if (length < _MaxSerializedVoidArray)
{
try
{
string data = deserializeInternal!(string)(_inlineArrayKey);
buffer = cast(UnqualfiedE[])Base64.decode(data);
} catch
{
buffer.length = 0;
}
} else
{
// Load data from file
auto filename = deserializeInternal!(string)(_filenameKey);
auto id = slice.id;
if (!exists(filename)) assert("xml data not found for `"~filename~"`!");
if (getSize(filename) != length) assert("xml data not correct size for `"~filename~"`! Expected "~to!string(value.length)~" got "~to!string(getSize(filename))~".");
buffer = cast(UnqualfiedE[])std.file.read(filename);
}
}
};
if (slice.id != size_t.max) // Deserialize slice
{
id = slice.id;
archive.unarchiveArray(slice.id, dg);
assumeUnique(buffer, value);
addDeserializedSlice(value, slice.id);
return toSlice(value, slice);
}
else // Deserialize array
{
id = archive.unarchiveArray(key, dg);
if (auto arr = id in deserializedSlices)
return cast(T) *arr;
assumeUnique(buffer, value);
addDeserializedSlice(value, id);
return value;
}
}
private T deserializeStaticArray (T) (string key, out Id id)
{
T value;
id = archive.unarchiveArray(key, (size_t length) {
foreach (i, ref e ; value)
e = deserializeInternal!(typeof(e))(toData(i));
});
return value;
}
private T deserializeAssociativeArray (T) (string key, out Id id)
{
id = deserializeReference(key);
if (auto reference = getDeserializedReference!(T)(id))
return *reference;
alias KeyTypeOfAssociativeArray!(T) Key;
alias ValueTypeOfAssociativeArray!(T) Value;
alias Unqual!(Key) UKey;
alias Unqual!(Value) UValue;
UValue[UKey] buffer;
id = archive.unarchiveAssociativeArray(key, (size_t length) {
for (size_t i = 0; i < length; i++)
{
UKey aaKey;
UValue aaValue;
auto k = toData(i);
archive.unarchiveAssociativeArrayKey(k, {
aaKey = deserializeInternal!(Key)(k);
});
archive.unarchiveAssociativeArrayValue(k, {
aaValue = deserializeInternal!(Value)(k);
});
buffer[aaKey] = aaValue;
}
});
T value = buffer;
addDeserializedReference(value, id);
return value;
}
private Pointer!(T) deserializePointer (T) (string key)
{
auto pointeeId = deserializeReference(key);
if (auto reference = getDeserializedReference!(T)(pointeeId))
return Pointer!(T)(*reference, Id.max);
alias BaseTypeOfPointer!(T) BaseType;
alias Unqual!(BaseType) UnqualfiedBaseType;
auto pointer = new UnqualfiedBaseType;
auto pointerId = archive.unarchivePointer(key, {
if (auto deserializer = key in overriddenDeserializers)
callSerializer(deserializer, pointer, key);
else if (auto deserializer = key in Serializer.deserializers)
callSerializer(deserializer, pointer, key);
else static if (isSerializable!(T))
pointer.fromData(this, key);
else
{
static if (isVoid!(BaseTypeOfPointer!(T)))
error(`The value with the key "` ~ to!(string)(key) ~ `"` ~
format!(` of the type "`, T, `" cannot be ` ~
"deserialized on its own, either implement " ~
"orange.serialization.Serializable.isSerializable or " ~
"register a deserializer."));
else
{
auto k = nextKey;
pointeeId = deserializeReference(k);
if (pointeeId == Id.max)
*pointer = deserializeInternal!(UnqualfiedBaseType)(k);
}
}
});
if (pointeeId != Id.max)
*pointer = deserializeInternal!(UnqualfiedBaseType)(pointeeId);
addDeserializedReference(pointer, pointerId);
return Pointer!(T)(cast(T) pointer, pointerId, pointeeId);
}
private T deserializeEnum (T, U) (U keyOrId, out Id id)
{
alias BaseTypeOfEnum!(T) Enum;
enum functionName = toUpper(Enum.stringof[0]) ~ Enum.stringof[1 .. $];
static if (is(U == Id))
enum params = "(keyOrId);";
else
enum params = "(keyOrId, id);";
mixin("return cast(T) archive.unarchiveEnum" ~ functionName ~ params);
}
private T deserializePrimitive (T, U) (U keyOrId, out Id id)
{
enum functionName = toUpper(T.stringof[0]) ~ T.stringof[1 .. $];
static if (is(U == Id))
enum params = "(keyOrId);";
else
enum params = "(keyOrId, id);";
mixin("return archive.unarchive" ~ functionName ~ params);
}
private Id deserializeReference (string key)
{
return archive.unarchiveReference(key);
}
private Slice deserializeSlice (string key)
{
return archive.unarchiveSlice(key);
}
private void objectStructSerializeHelper (T) (ref T value)
{
static assert(isStruct!(T) || isObject!(T), format!(`The given value of the type "`, T, `" is not a valid type, the only valid types for this method are objects and structs.`));
enum nonSerializedFields = collectAnnotations!(T);
foreach (i, _ ; typeof(T.tupleof))
{
enum field = nameOfFieldAt!(T, i);
alias attributes = getAttributes!(T.tupleof[i]);
static if (attributes.contains!(nonSerialized) ||
ctfeContains!(string)(internalFields, field) ||
ctfeContains!(string)(nonSerializedFields, field))
{
continue;
}
alias typeof(T.tupleof[i]) Type;
auto id = nextId;
static if (isPointer!(Type))
auto pointer = value.tupleof[i];
else
auto pointer = &value.tupleof[i];
auto reference = getSerializedReference(value.tupleof[i]);
if (reference != Id.max)
archive.archiveReference(field, reference);
else
{
auto valueMeta = getSerializedValue(pointer);
if (valueMeta.isValid)
serializePointer(pointer, toData(field), id);
else
{
serializeInternal(value.tupleof[i], toData(field), id);
addSerializedValue(pointer, id, toData(keyCounter));
}
}
}
static if (isObject!(T) && !is(Unqual!(T) == Object))
serializeBaseTypes(value);
}
private void objectStructDeserializeHelper (T) (ref T value)
{
static assert(isStruct!(T) || isObject!(T), format!(`The given value of the type "`, T, `" is not a valid type, the only valid types for this method are objects and structs.`));
enum nonSerializedFields = collectAnnotations!(T);
static if (isObject!(T))
auto rawObject = cast(void*) value;
else
auto rawObject = cast(void*) &value;
foreach (i, _ ; typeof(T.tupleof))
{
enum field = nameOfFieldAt!(T, i);
alias attributes = getAttributes!(T.tupleof[i]);
static if (attributes.contains!(nonSerialized) ||
ctfeContains!(string)(internalFields, field) ||
ctfeContains!(string)(nonSerializedFields, field))
{
continue;
}
alias TypeOfField!(T, field) QualifiedType;
alias Unqual!(QualifiedType) Type;
auto id = deserializeReference(field);
auto isReference = id != Id.max;
auto offset = value.tupleof[i].offsetof;
auto fieldAddress = cast(Type*) (rawObject + offset);
static if (isPointer!(Type))
{
auto pointer = deserializePointer!(Type)(toData(field));
Type pointerValue;
if (pointer.hasPointee)
pointerValue = getDeserializedValue!(Type)(pointer.pointee);
else
pointerValue = pointer.value;
*fieldAddress = pointerValue;
addDeserializedPointer(value.tupleof[i], pointer.id);
}
else
{
auto pointer = getDeserializedPointer!(Type*)(id);
if (isReference && pointer)
{
*fieldAddress = **pointer;
*pointer = cast(Type*) &value.tupleof[i];
}
else
{
*fieldAddress = deserializeInternal!(Type)(toData(field), id);
addDeserializedValue(value.tupleof[i], id);
static if (isStaticArray!(Type))
addDeserializedSlice(value.tupleof[i], id);
}
}
}
static if (isObject!(T) && !is(Unqual!(T) == Object))
deserializeBaseTypes(value);
}
private void serializeBaseTypes (T) (inout T value) if (isObject!(T))
{
static if (hasNonObjectBaseType!(T))
{
alias Base = BaseTypeTupleOf!(T)[0];
archive.archiveBaseClass(typeid(Base).toString, nextKey, nextId);
inout Base base = value;
objectStructSerializeHelper(base);
}
}
private void deserializeBaseTypes (T) (T value) if (isObject!(T))
{
static if (hasNonObjectBaseType!(T))
{
alias Base = BaseTypeTupleOf!(T)[0];
archive.unarchiveBaseClass(nextKey);
Base base = value;
objectStructDeserializeHelper(base);
}
}
private void addSerializedReference (T) (T value, Id id)
{
alias Unqual!(T) Type;
static assert(isReference!(Type) || isAssociativeArray!(Type), format!(`The given type "`, T, `" is not a reference type, i.e. object, pointer or associative array.`));
serializedReferences[cast(void*) value] = id;
}
private void addDeserializedReference (T) (T value, Id id)
{
static assert(isReference!(T) || isAssociativeArray!(T), format!(`The given type "`, T, `" is not a reference type, i.e. object, pointer or associative array.`));
deserializedReferences[id] = cast(void*) value;
}
private void addDeserializedSlice (T) (ref T value, Id id)
{
static assert(isArray!(T) || isString!(T), format!(`The given type "`, T, `" is not a slice type, i.e. array or string.`));
deserializedSlices[id] = cast(void[]) value;
}
private void addSerializedValue (T) (T* value, Id id, string key)
{
serializedValues[value] = ValueMeta(id, key);
}
private void addDeserializedValue (T) (ref T value, Id id)
{
deserializedValues[id] = &value;
}
private void addDeserializedPointer (T) (ref T value, Id id)
{
deserializedPointers[id] = cast(void**) &value;
}
private Id getSerializedReference (T) (T value)
{
if (auto tmp = *(cast(void**) &value) in serializedReferences)
return *tmp;
return Id.max;
}
private ValueMeta getSerializedValue (T) (T* value)
{
if (auto tmp = value in serializedValues)
return *tmp;
return ValueMeta();
}
private T* getDeserializedReference (T) (Id id)
{
if (auto reference = id in deserializedReferences)
return cast(T*) reference;
return null;
}
private T getDeserializedSlice (T) (Slice slice)
{
if (auto array = slice.id in deserializedSlices)
{
auto typed = cast(T) *array;
return typed[slice.offset .. slice.offset + slice.length];
}
return null;
}
private T getDeserializedValue (T) (Id id)
{
if (auto value = id in deserializedValues)
return cast(T) *value;
return null;
}
private T* getDeserializedPointer (T) (Id id)
{
if (auto pointer = id in deserializedPointers)
return cast(T*) *pointer;
return null;
}
private T[] toSlice (T) (T[] array, Slice slice)
{
return array[slice.offset .. slice.offset + slice.length];
}
void callSerializer (T) (RegisterBase* baseWrapper, ref T value, string key)
{
if (mode == serializing)
{
auto wrapper = cast(SerializeRegisterWrapper!(T)) *baseWrapper;
wrapper(value, this, key);
}
else
{
auto wrapper = cast(DeserializeRegisterWrapper!(T)) *baseWrapper;
wrapper(value, this, key);
}
}
static private SerializeRegisterWrapper!(T) toSerializeRegisterWrapper (T) (void delegate (T, Serializer, Data) dg)
{
return new SerializeRegisterWrapper!(T)(dg);
}
static private SerializeRegisterWrapper!(T) toSerializeRegisterWrapper (T) (void function (T, Serializer, Data) func)
{
return new SerializeRegisterWrapper!(T)(func);
}
static private DeserializeRegisterWrapper!(T) toDeserializeRegisterWrapper (T) (void delegate (ref T, Serializer, Data) dg)
{
return new DeserializeRegisterWrapper!(T)(dg);
}
static private DeserializeRegisterWrapper!(T) toDeserializeRegisterWrapper (T) (void function (ref T, Serializer, Data) func)
{
return new DeserializeRegisterWrapper!(T)(func);
}
private void addSerializedArray (Array array, Id id)
{
serializedArrays[id] = array;
}
private void postProcess ()
{
postProcessArrays();
}
private void postProcessArrays ()
{
bool foundSlice = true;
foreach (sliceKey, slice ; serializedArrays)
{
foreach (arrayKey, array ; serializedArrays)
{
if (slice.isSliceOf(array) && slice != array)
{
auto s = Slice(slice.length, (slice.ptr - array.ptr) / slice.elementSize);
archive.archiveSlice(s, sliceKey, arrayKey);
foundSlice = true;
break;
}
else
foundSlice = false;
}
if (!foundSlice)
archive.postProcessArray(sliceKey);
}
}
private void deserializingPostProcess ()
{
deserializingPostProcessPointers;
}
private void deserializingPostProcessPointers ()
{
// foreach (pointeeId, pointee ; deserializedValues)
// {
// if (auto pointer = pointeeId in deserializedPointers)
// **pointer = pointee;
// }
}
private string arrayToString (T) ()
{
return typeid(ElementTypeOfArray!(T)).toString;
}
private bool isBaseClass (T) (T value)
{
return value.classinfo !is T.classinfo;
}
private Id nextId ()
{
return idCounter++;
}
private string nextKey ()
{
return toData(keyCounter++);
}
private void resetCounters ()
{
keyCounter = 0;
idCounter = 0;
}
private string toData (T) (T value)
{
return to!(string)(value);
}
private void triggerEvent (string name, T) (T value)
{
static assert (isObject!(T) || isStruct!(T), format!(`The given value of the type "`, T, `" is not a valid type, the only valid types for this method are objects and structs.`));
static if (hasAnnotation!(T, name))
{
mixin("auto event = T." ~ name ~ ";");
event(value);
}
}
private void triggertUdaEvent (alias event, T) (T value)
{
static assert (isObject!(T) || isStruct!(T), format!(`The given value of the type "`, T, `" is not a valid type, the only valid types for this method are objects and structs.`));
foreach (m ; __traits(allMembers, T))
{
static if (m != nonSerializedField)
{
mixin(`alias attrs = Attributes!(m, __traits(getAttributes, T.` ~ m ~ `));`);
static if (attrs.contains!(event))
__traits(getMember, value, m)();
}
}
}
private void triggerEvents (T) (T value, void delegate () dg)
{
if (mode == serializing)
{
triggerEvent!(onSerializingField)(value);
triggertUdaEvent!(onSerializing)(value);
}
else
{
triggerEvent!(onDeserializingField)(value);
triggertUdaEvent!(onDeserializing)(value);
}
dg();
if (mode == serializing)
{
triggerEvent!(onSerializedField)(value);
triggertUdaEvent!(onSerialized)(value);
}
else
{
triggerEvent!(onDeserializedField)(value);
triggertUdaEvent!(onDeserialized)(value);
}
}
private static bool isNonSerialized (T) ()
{
enum nonSerializedFields = collectAnnotations!(T);
return ctfeContains(nonSerializedFields, "this") || getAttributes!(T).contains!(nonSerialized);
}
private static template hasAnnotation (T, string annotation)
{
enum hasAnnotation = is(typeof({ mixin("const a = T." ~ annotation ~ ";"); }));
}
private static string[] collectAnnotations (T) ()
{
static assert (isObject!(T) || isStruct!(T), format!(`The given value of the type "`, T, `" is not a valid type, the only valid types for this method are objects and structs.`));
static if (hasAnnotation!(T, nonSerializedField))
return T.__nonSerialized;
else
return [];
}
private void error (string message, size_t line = __LINE__)
{
if (errorCallback)
errorCallback()(new SerializationException(message, __FILE__, line));
}
struct Pointer (T)
{
T value;
Id id = Id.max;
Id pointee = Id.max;
bool hasPointee ()
{
return pointee != Id.max;
}
}
private template hasNonObjectBaseType (T)
{
alias BaseTypes = BaseTypeTupleOf!(T);
enum hasNonObjectBaseType = BaseTypes.length > 0 &&
!is(Unqual!(BaseTypes[0]) == Object);
}
}
/**
* This struct is a type independent representation of an array. This struct is used
* when sending an array for archiving from the serializer to the archive.
*/
struct Array
{
const(void)* ptr;
/// The length of the array
size_t length;
/// The size of an individual element stored in the array, in bytes.
size_t elementSize;
/**
* Returns true if the given array is a slice of the receiver.
*
* Params:
* b = the array to check if it's a slice
*
* Returns: true if the given array is a slice of the receiver.
*/
bool isSliceOf (Array b)
{
return ptr >= b.ptr && ptr + length * elementSize <= b.ptr + b.length * b.elementSize;
}
}
/**
* This struct is a type independent representation of a slice. This struct is used
* when sending a slice for archiving from the serializer to the archive.
*/
struct Slice
{
/// The length of the slice.
size_t length;
/// The offset of the slice, i.e. where the slice begins in the array.
size_t offset;
/// The id of the slice. (Only used during unarchiving).
size_t id = size_t.max;
}
Again, you are forcing one to change their internal code to suit orange so it will serialize.
The code I've added shows how to serialize data to files and works well. The reason not to realize to the same file is that large data sets could be duplicated if one uses them in multiple deserialization processes.
The bigger problem is that serializing them inline makes the size an order of magnitude larger!!
The reason you don't think this is a proper solution is because you have not used orange in the real work on large arrays!
Even if I changed it to ubyte[] it would still increase the size 10x. But my data is void[] which can be any type so forcing ubyte requires me to constantly cast to void. There is no point.
You can support the feature and make it optional by parsing an attribute that will serialize the file to an external binary file. I've already did most of the work for you!
What happens is I cannot use your library as is because it is deficient for my purposes. Any updates you will then cause prevent me from easily updating.
You are trying to justify not doing this when you should be finding ways to justify to do it. It is not a silly request, it is in fact very practical, but just because you haven't used orange in such a way you feel it is not but your feelings is not reality.
class Item
{
byte[] largebinarydata;
//@serializeAsBinary(getFileName, Item, largebinarydata) void[] largebinarydata;
}
large binary data could be 100M. Stuck in the output of the serializer it becomes 1G+ because of all the extra xml text per byte. THIS IS NOT EXCEPTABLE!
Being an optional inclusion does not hurt anyone as they can opt in. This solves the problem with void being pointers or whatever.
<array id="48" type="const(immutable(wchar)[])" length="1" key="largebinarydata">
<string id="49" type="immutable(wchar)" length="16" key="0">test.dat</string>
</array>
rather than
<array id="8" type="inout(int)" length="1000" key="largebinarydata">
<int id="9" key="0">0</int>
<int id="10" key="1">0</int>
<int id="11" key="2">0</int>
<int id="12" key="3">0</int>
<int id="13" key="4">0</int>
<int id="14" key="5">0</int>
<int id="15" key="6">0</int>
<int id="16" key="7">0</int>
<int id="17" key="8">0</int>
<int id="18" key="9">0</int>
<int id="19" key="10">0</int>
<int id="20" key="11">0</int>
<int id="21" key="12">0</int>
<int id="22" key="13">0</int>
<int id="23" key="14">0</int>
<int id="24" key="15">0</int>
<int id="25" key="16">0</int>
<int id="26" key="17">0</int>
<int id="27" key="18">0</int>
<int id="28" key="19">0</int>
<int id="29" key="20">0</int>
<int id="30" key="21">0</int>
<int id="31" key="22">0</int>
<int id="32" key="23">0</int>
<int id="33" key="24">0</int>
<int id="34" key="25">0</int>
<int id="35" key="26">0</int>
<int id="36" key="27">0</int>
<int id="37" key="28">0</int>
<int id="38" key="29">0</int>
<int id="39" key="30">0</int>
<int id="40" key="31">0</int>
<int id="41" key="32">0</int>
<int id="42" key="33">0</int>
<int id="43" key="34">0</int>
<int id="44" key="35">0</int>
<int id="45" key="36">0</int>
<int id="46" key="37">0</int>
<int id="47" key="38">0</int>
<int id="48" key="39">0</int>
<int id="49" key="40">0</int>
<int id="50" key="41">0</int>
<int id="51" key="42">0</int>
<int id="52" key="43">0</int>
<int id="53" key="44">0</int>
<int id="54" key="45">0</int>
<int id="55" key="46">0</int>
<int id="56" key="47">0</int>
<int id="57" key="48">0</int>
<int id="58" key="49">0</int>
<int id="59" key="50">0</int>
<int id="60" key="51">0</int>
<int id="61" key="52">0</int>
<int id="62" key="53">0</int>
<int id="63" key="54">0</int>
<int id="64" key="55">0</int>
<int id="65" key="56">0</int>
<int id="66" key="57">0</int>
<int id="67" key="58">0</int>
<int id="68" key="59">0</int>
<int id="69" key="60">0</int>
<int id="70" key="61">0</int>
<int id="71" key="62">0</int>
<int id="72" key="63">0</int>
<int id="73" key="64">0</int>
<int id="74" key="65">0</int>
<int id="75" key="66">0</int>
<int id="76" key="67">0</int>
<int id="77" key="68">0</int>
<int id="78" key="69">0</int>
<int id="79" key="70">0</int>
<int id="80" key="71">0</int>
<int id="81" key="72">0</int>
<int id="82" key="73">0</int>
<int id="83" key="74">0</int>
<int id="84" key="75">0</int>
<int id="85" key="76">0</int>
<int id="86" key="77">0</int>
<int id="87" key="78">0</int>
<int id="88" key="79">0</int>
<int id="89" key="80">0</int>
<int id="90" key="81">0</int>
<int id="91" key="82">0</int>
<int id="92" key="83">0</int>
<int id="93" key="84">0</int>
<int id="94" key="85">0</int>
<int id="95" key="86">0</int>
<int id="96" key="87">0</int>
<int id="97" key="88">0</int>
<int id="98" key="89">0</int>
<int id="99" key="90">0</int>
<int id="100" key="91">0</int>
<int id="101" key="92">0</int>
<int id="102" key="93">0</int>
<int id="103" key="94">0</int>
<int id="104" key="95">0</int>
<int id="105" key="96">0</int>
<int id="106" key="97">0</int>
<int id="107" key="98">0</int>
<int id="108" key="99">0</int>
<int id="109" key="100">0</int>
<int id="110" key="101">0</int>
<int id="111" key="102">0</int>
<int id="112" key="103">0</int>
<int id="113" key="104">0</int>
<int id="114" key="105">0</int>
<int id="115" key="106">0</int>
<int id="116" key="107">0</int>
<int id="117" key="108">0</int>
<int id="118" key="109">0</int>
<int id="119" key="110">0</int>
<int id="120" key="111">0</int>
<int id="121" key="112">0</int>
<int id="122" key="113">0</int>
<int id="123" key="114">0</int>
<int id="124" key="115">0</int>
<int id="125" key="116">0</int>
<int id="126" key="117">0</int>
<int id="127" key="118">0</int>
<int id="128" key="119">0</int>
<int id="129" key="120">0</int>
<int id="130" key="121">0</int>
<int id="131" key="122">0</int>
<int id="132" key="123">0</int>
<int id="133" key="124">0</int>
<int id="134" key="125">0</int>
<int id="135" key="126">0</int>
<int id="136" key="127">0</int>
<int id="137" key="128">0</int>
<int id="138" key="129">0</int>
<int id="139" key="130">0</int>
<int id="140" key="131">0</int>
<int id="141" key="132">0</int>
<int id="142" key="133">0</int>
<int id="143" key="134">0</int>
<int id="144" key="135">0</int>
<int id="145" key="136">0</int>
<int id="146" key="137">0</int>
<int id="147" key="138">0</int>
<int id="148" key="139">0</int>
<int id="149" key="140">0</int>
<int id="150" key="141">0</int>
<int id="151" key="142">0</int>
<int id="152" key="143">0</int>
<int id="153" key="144">0</int>
<int id="154" key="145">0</int>
<int id="155" key="146">0</int>
<int id="156" key="147">0</int>
<int id="157" key="148">0</int>
<int id="158" key="149">0</int>
<int id="159" key="150">0</int>
<int id="160" key="151">0</int>
<int id="161" key="152">0</int>
<int id="162" key="153">0</int>
<int id="163" key="154">0</int>
<int id="164" key="155">0</int>
<int id="165" key="156">0</int>
<int id="166" key="157">0</int>
<int id="167" key="158">0</int>
<int id="168" key="159">0</int>
<int id="169" key="160">0</int>
<int id="170" key="161">0</int>
<int id="171" key="162">0</int>
<int id="172" key="163">0</int>
<int id="173" key="164">0</int>
<int id="174" key="165">0</int>
<int id="175" key="166">0</int>
<int id="176" key="167">0</int>
<int id="177" key="168">0</int>
<int id="178" key="169">0</int>
<int id="179" key="170">0</int>
<int id="180" key="171">0</int>
<int id="181" key="172">0</int>
<int id="182" key="173">0</int>
<int id="183" key="174">0</int>
<int id="184" key="175">0</int>
<int id="185" key="176">0</int>
<int id="186" key="177">0</int>
<int id="187" key="178">0</int>
<int id="188" key="179">0</int>
<int id="189" key="180">0</int>
<int id="190" key="181">0</int>
<int id="191" key="182">0</int>
<int id="192" key="183">0</int>
<int id="193" key="184">0</int>
<int id="194" key="185">0</int>
<int id="195" key="186">0</int>
<int id="196" key="187">0</int>
<int id="197" key="188">0</int>
<int id="198" key="189">0</int>
<int id="199" key="190">0</int>
<int id="200" key="191">0</int>
<int id="201" key="192">0</int>
<int id="202" key="193">0</int>
<int id="203" key="194">0</int>
<int id="204" key="195">0</int>
<int id="205" key="196">0</int>
<int id="206" key="197">0</int>
<int id="207" key="198">0</int>
<int id="208" key="199">0</int>
<int id="209" key="200">0</int>
<int id="210" key="201">0</int>
<int id="211" key="202">0</int>
<int id="212" key="203">0</int>
<int id="213" key="204">0</int>
<int id="214" key="205">0</int>
<int id="215" key="206">0</int>
<int id="216" key="207">0</int>
<int id="217" key="208">0</int>
<int id="218" key="209">0</int>
<int id="219" key="210">0</int>
<int id="220" key="211">0</int>
<int id="221" key="212">0</int>
<int id="222" key="213">0</int>
<int id="223" key="214">0</int>
<int id="224" key="215">0</int>
<int id="225" key="216">0</int>
<int id="226" key="217">0</int>
<int id="227" key="218">0</int>
<int id="228" key="219">0</int>
<int id="229" key="220">0</int>
<int id="230" key="221">0</int>
<int id="231" key="222">0</int>
<int id="232" key="223">0</int>
<int id="233" key="224">0</int>
<int id="234" key="225">0</int>
which you seem to actually believe is acceptable! IT IS NOT! Get it through your head! Please, it is ridiculous!
For a single byte you are adding over 30 bytes in the xml file! A single meg will then take 30 megs to store!
Do it my way and it takes only an extra few bytes. This is not hard, basic programming! Don't be lazy! I've done 90% of the work!