[Adonthell-devel] Improved saving/loading suggestion

[Kai Sterker]

Hi everybody,

sorry for the long silence, but I have been quite busy lately. I should 
have a little more time soon, though.

Anyway, today I did some more coding on the event stuff, and I had some 
thoughts about loading and saving.

Basically, all our load/save routines write stuff to a binary file 
without much error checking. If the file format changes, loading old 
files with the new app or new files with an old app will not only fail, 
but almost certainly crash.

In order to fix this, I have the idea of a sort of intermediate layer. 
Basically, all 'writes' do not go to the file directly, but into a 
cache-like class. This keeps track of the data size, can optionally 
calculate a checksum and, when all the data has been accumulated, write 
it to disk.

That means, when reading a file, the cache class would read exactly the 
record that has been written to file, regardless of the engine version, 
since the size of the record is in the file. Next, it can do an 
integrity check by comparing the checksum. After all the stuff has been 
loaded, the actual object can request its fields from the cache.

Therefore, if new fields are being added to a file, it won't break 
compatibility with old apps, as they would simply ignore the new 
fields. OTOH, if a new app tries to read data with some fields missing, 
it can communicate exactly what is wrong. In rare cases, it might even 
be able to supply defaults and continue.


The interface of the cache class would have the major methods:

* void put (string name, <data> data)
  add data to the cache under the given name. The cache will append 
name and
  data to its internal buffer in binary form.

* <data> get (string name, [bool exit = false, <data> default = NULL])
  Return the data identified by name. There are two optional parameters:
  if 'exit' is set to true, the program will exit with an error message 
in
  case the data is not in the record. That is useful if the data is 
critical,
  so that the program would crash anyway.
  'default' may contain a default value that is returned when the data 
is
  not found and 'exit' is false.

  The two parameters are there for convenience, to avoid repetetive if-
  statements to check whether the data was returned or not. With the two
  parameters the check can be made by the get() method.

* bool load ()
  Load the next record from file. Closes the current record, if any.

* bool save ()
  Saves the record to file and resets / flushes the cache object.


Our put_state and get_state methods would get a cache object instead of 
i/ogzstream objects, since the stream to use is already a member of the 
cache object.

And best of all, later on such a cache class could well be used to 
prepare our objects for transport across the network.


A note about performance:
Data inside the cache will probably be stored in a single char-array in 
the form
namevaluenamevalue... without any delimiters or such. get() will search 
for the given name and retrieve the according type from the bytes 
following the name. If we keep track of the pointer to the array and 
call get in the same order as we put stuff into the cache, then the 
overhead will be minimal.

And a final note about type-safety:
There's none. If you put(u_int16) and get(s_int32) then there will be 
trouble. That's not different from the current situation. However, if 
we are willing to spend an extra byte, we could even keep track of the 
type and make sure that you can't mess things up by accident.


So as a whole, our data files would look like this

 +---------------+
 | record length |
 + - - - - - - - +            Data Section
 |               |            +----+------+-------+----+------------+
 |     data      |   ----->   | id : type : value | id : type : ... |
 |               |            +----+------+-------+----+------------+
 + - - - - - - - +               |     |      |
 |   checksum    |               |     |      +--- variable length
 +---------------+               |     +---------- 1 byte
 | record length |               +---------------- variable length
 + - - - - - - - +
 |               |
 |     data      |
       ...


Of course, there are still some questions open and more will come up 
when implementing stuff. But if you think the advantages are worth the 
effort, then I would have an even closer look at things. Best to 
implement it now, before we have to change too many load/save methods.

Kai