Serialize Data C++

[Guest]

TL;DR
I need to serialize data to files & want to do that with C++ & don't know how.

 

Long:

I'm working on a Game Engine & currently working on the map editor. I want to be able to save the map data into types of collections. IE "Place box at x coordinate 5 & y coordinate 7."

I might add map editor permissions to add Lua scripts to certain locations if I figure out how to create bindings later.

 

Basically I need to be able to store blocks of data as objects & determine what kind of game object it is before placing it into the world. That does kinda end up with me designing the code appropriately to know how to do that, but I'm looking for an API or something that is easy to use that will let me do these things.

 

Someone told me to use Google's Protocol Buffers, but I can't figure them out for the life of me. The guide makes no sense to me.
I was originally considering to use JSON, but idk.

 

I don't really want to write my own file format & stuff. I did that before in one of my previous threads. I'm looking for something more "professional."

Unless defining one's own file structure & format is viewed as professional.

[Bartholomew]

25 minutes ago, fpo said:

TL;DR
I need to serialize data to files & want to do that with C++ & don't know how.

 

Long:

I'm working on a Game Engine & currently working on the map editor. I want to be able to save the map data into types of collections. IE "Place box at x coordinate 5 & y coordinate 7."

I might add map editor permissions to add Lua scripts to certain locations if I figure out how to create bindings later.

 

Basically I need to be able to store blocks of data as objects & determine what kind of game object it is before placing it into the world. That does kinda end up with me designing the code appropriately to know how to do that, but I'm looking for an API or something that is easy to use that will let me do these things.

 

Someone told me to use Google's Protocol Buffers, but I can't figure them out for the life of me. The guide makes no sense to me.
I was originally considering to use JSON, but idk.

 

I don't really want to write my own file format & stuff. I did that before in one of my previous threads. I'm looking for something more "professional."

Unless defining one's own file structure & format is viewed as professional.

Serialisation can be broken up in a few steps:

 

First, "usualy" theres the separation of actual object instances used by the program and the store/retrieve representation objects, often called "DTO"'s (as Data Transfer Objects). This is optional, and theres more strategies possible (like through annotations on fields of live objects) but generally its seen as a good practice as then the next step, transform, only deals with pure data objects.

 

Second theres the transform of DTO to a format (json, xml, etc.).

 

So for serialize, the object hierarchy is traversed and the dtos/fields  (and/or annotations) are scanned and transformed by whatever transform library (try not to roll your own as its error prone).

 

For deserialize, let the transformer create the dto's, and then process those to make the actual functional object instances. This is also why its a common strategy as then any construct-time issues or other stuff (like co-object dependencies of stored objects cross referencing each other) is lifted completely out of the parse/deserialize phase, making error handling a lot easier and allows for further  processing of the data (like version migrations/transforms) before instantiating the actual related objects (which can change appearance between your software versions over time).

 

Depending on the use case and languages and libraries/platform constructs involved other details or strategies can be used (or be better/worse) but overall the above outlines the overall process.

 

Hope this helps at least to look in a few directions from here

 

[Unimportant]

21 hours ago, fpo said:

<snip>

We've rolled our own,so this is just one possible implementation to give you some inspiration.

 

First we have a interface class that represents a serial stream and promises operator  >> and << overloads for all basic types (and some raw byte I/O):

class SerialStream
{
public:

	virtual SerialStream&
	operator >> (int&) = 0;

	virtual SerialStream&
	operator << (int) = 0;

	//And so on for all basic types...

	//Raw reading and writing
	virtual void
	RawRead(char *data, int len) = 0;

	virtual void
	RawWrite(const char *data, int len) = 0;
};

Then, obviously there's some concrete implementations for this interface.

 

Then we have a Deserialise function:

/* Deserialises the template type T from the SerialStream and returns it.
 * Extra Args are passed on to the specialised deserialise function for default constructible types.
 * Extra Args are passed on the constructor for non default constructible types. */
template <class T, class... Args>
T
Deserialise(SerialStream& ss, Args&&... args)
{
    if constexpr (!std::is_enum_v<T>)
    {
        if constexpr (std::is_default_constructible_v<T>)
        {
            /* Default constructible types. */
            T t;
            SpecialisedDeserialise(ss, t, std::forward<Args>(args)...);
            return t;
        }
        else
        {
            /* Non default constructible types.
             * Should have ctor that constructs from SerialStream. */
            return T(ss, std::forward<Args>(args)...);
        }
    }
    else
    {
        /* Enum types. */
        std::underlying_type_t<T> t;
        ss >> t;
        return Util::UnderlyingToEnum<T>(t); //Utility function that converts underlying type to enum.
    }
}

So your own non-default constructable classes will need a constructor that takes a SerialStream as argument and constructs itself from it.

Default constructable classes are handled by SpecialisedDeserialise, which is a template function for the majority of cases:

/* Deserialise for default constructible types that are supported by SerialStream::operator >>.
 * Extra Args are ignored. */
template <class T, class... Args>
void
SpecialisedDeserialise(SerialStream& ss, T& t, Args&&...)
{
    ss >> t;
}

This allows for further specialization if required, for example for std::vector<T>...

/* Deserialise for std::vector<T>.
 * Extra Args are passed on to the Deserialise function for each element. */
template <class T, class... Args>
void
SpecialisedDeserialise(SerialStream& ss, std::vector<T>& v, Args&&... args)
{
    uint64_t size;		
    ss >> size;
    v.clear();
    v.reserve(size);
    for (uint64_t i = 0; i < size; ++i)
    {
        v.emplace_back(Deserialise<T>(ss, std::forward<Args>(args)...));
    }
}

For serialization we have a template operator <<:

/* Will be called for types where no better matching operator << overload is found.
 * Calls the specialised serialise function for the type. */
template <class T>
SerialStream&
operator << (SerialStream& ss, const T& t)
{
    if constexpr (!std::is_enum_v<T>)
    {
        SpecialisedSerialise(ss, t);
    }
    else
    {
        /* Enum types. */
        ss << Util::EnumToUnderlying(t); //Utility function that converts enum to underlying type.
    }

    return ss;
}

SpecialisedSerialise is also a template function that simply delegates to T::Serialise. This means your own classes need to incorporate a Serialise function that takes a SerialStream as argument and serialises itself to it:

/* Serialise for classes that have a Serialise member function. */
template <class T>
void
SpecialisedSerialise(SerialStream& ss, const T& t)
{
    t.Serialise(ss);
}

But again allows for specialization if required:

/* Serialise for std::vector<T>. */
template <class T>
void
SpecialisedSerialise(SerialStream& ss, const std::vector<T>& v)
{
    const uint64_t size = v.size();
    ss << size;
    for (const auto& element : v)
    {
        ss << element;
    }
}

Example usage:

class Example
{
private:

	//Some members for serialisation...
	int mSomeInt;
	float mSomeFloat;
	std::vector<int> mSomeVector;
	Foo mSomeFoo;

public:

	Example(SerialStream& ss) :
		mSomeInt(Deserialise<int>(ss)),			//Will call SerialStream::operator >> for int trough the basic SpecialisedDeserialise template
		mSomeFloat(Deserialise<float>(ss)),     	//Will call SerialStream::operator >> for float trough the basic SpecialisedDeserialise template
		mSomeVector(Deserialise<std::vector<int>>(ss)), //Will call SpecialisedDeserialise for std::vector<T>
		mSomeFoo(Deserialise<Foo>(ss))			//Will call Foo::Foo(SerialStream&) 	
	{}

	void
	Serialise(SerialStream& ss) const
	{
		ss << mSomeInt;		//Plain call to SerialStream::operator << for int.			
		ss << mSomeFloat;	//Plain call to SerialStream::operator << for float.
		ss << mSomeVector;	//Calls SpecialisedSerialise for std::vector<T> trough the operator << template.
		ss << mSomeFoo;		//Calls Foo::Serialise(SerialStream&) trough the operator << template.
	}
};

The Deserialise function also takes additional parameters to be passed to the deserialisation constructor of non-default constructible types if required.