Need better format for save
-
If you want to encourage others to make 3rd-party software that can handle TPT saves and stamps, you're gonna need to make it in a way that is easy to program for. All multibyte values are little endian. This will be similar to the original save format, but with some notable differences, that will make it easier to write 3rd-party software for.
My proposed format:
Header:
Bytes 0 to 3 = String "TPTS" //this means The Powder Toy Save
Byte 4 = Major Version Number //valid values are 0x00 to 0x7F, as the MSB is used to determine if beta (1) or normal (0)
Byte 5 = Minor Version Number //valid values are 0x00 to 0xFF
Byte 6 = Block Width //in units of pixels, 4 by default
Byte 7 = Block Height //in units of pixels, 4 by default
Byte 8 = Simulation Screen Width //in units of blocks
Byte 9 = Simulation Screen Height //in units of blocks
Byte 10 = Number of Signs
Byte 11 = Game State Flags
Byte 12 = Gravity and Air Flags
Bytes 13 to 16 = SInt32 Uncompressed Data Size //or UInt32, as long as file game data size doesn't exceed 0x7fffffff bytes
Bytes 17 to LastByteInFile = BZip2 Compressed Data
Game State Flags:
Bit 0 = pause state //default is 0
Bit 1 = use Newtonian gravity //default is 0
Bit 2 = use heat simulation //default is 1
Bit 3 = use ambient heat //default is 0
Bit 4 = use equalize liquid levels //default is 0
Bit 5 = game screen has solid edges //default is 0
Bit 6 = show decorations //default is 0
Bit 7 = unused or reserved for future use
Gravity and Air Flags:
Bits 0 to 1 = Gravity Mode //default is 1 (0 is no gravity, 1 is vertical, 2 is radial, 3 is unused/reserved)
Bits 2 to 4 = Air Mode //default is 1 (0 is no air, 1 is air on, 2 is pressure off, 3 is velocity off, 4 is no update, 5 to 8 are unused/reserved)
Bits 5 to 7 = unused or reserved for future use
The game data that is to be compressed or decompressed should have this format:
Wall Array
Particle Array
Sign Array
Wall Array should have 2 dimensions.
The 1st dimension is x position of the wall (measured in blocks)
The 2nd dimension is y position of the wall (measured in blocks)
Each cell in the array should be of the below specified user defined type:
Type Name = WallData
Byte 0 = Wall Type
Bytes 1 to 4 = IEEE-Float Fan X Velocity //should be 0 if wall is not a fan
Bytes 1 to 4 = IEEE-Float Fan Y Velocity //should be 0 if wall is not a fan
Particle Array should have 3 dimensions.
The 1st dimension is x position of the wall (measured in pixels)
The 2nd dimension is y position of the wall (measured in pixels)
The 3nd dimension is stacking layer (should always have a size of 1000, meaning at each pixel, up to 1000 particles can be stacked, and layer 0 should always be the primary layer)
Each cell in the array should be of the below specified user defined type:
Type Name = ParticleData
Byte 0 = Particle Type
Byte 1 = CType
Bytes 2 to 5 = SInt32 Life //see Note 1 at the bottom of the page
Bytes 6 to 9 = SInt32 Tmp //see Note 1 at the bottom of the page
Bytes 10 to 13 = SInt32 Tmp2 //see Note 1 at the bottom of the page
Bytes 14 to 17 = IEEE-Float Temperature
Bytes 18 to 21 = IEEE-Float X Velocity
Bytes 22 to 25 = IEEE-Float Y Velocity
Bytes 26 to 29 = IEEE-Float Exact X Position
Bytes 30 to 33 = IEEE-Float Exact Y Position
Bytes 34 to 37 = SInt32 Decoration Color
Any particle who's type does not support saving a particular value field (such as Tmp or Tmp2) should store a value of 0 in that field.
Sign Array should 1 dimension.
The 1 dimension should be used to which sign number you wish to manipulate.
Each cell in the array should be of the below specified user defined type:
Type Name = SignData
Bytes 0 to 1 = UInt16 Length of String //must be no greater than 0xFFFE, see Note 1
Bytes 2 to EndOfString = String containting the text of the sign and any special characters you may have used in the sign
Note 1:
The data types or other specifications may not seem the most logical, for users of C or C++. However VB6 has many limitations, and the exact specifications I picked are intended to make it easy for 3rd party software creators to make software using VB6, which is to be capable of reading and writing TPT saves and stamps. VB6 is common among hobbyist programmers, because of its simplicity of use. While you are using C++ to make TPT, it is imperative that while the internal data data in TPT as it runs in memory may continue to be incompatible with loading directly in VB6 programs, that the save/stamp files it outputs MUST compatible with VB6's native data types and other specs, if one is to easily write a TPT save file editor in VB6. Furthermore, the specs I've included in this document I believe should in no way be incompatible with C or C++. I can only hope that the developers of TPT choose to adopt this save file format for the next version of TPT.Edited 5 times by Videogamer555. Last: 15th Nov 2013 -
Frankly, designing a save format around POS6^H^H^H^HVB6 is a terrible idea in the first place. BSON is fine, works nice, and is well documented, so I don't see what the big deal is.
EDIT:
Wait, VB6 is common among hobbyist programmers?
EDIT: Sorry for multiple images, my adblocker had blocked two of them so I didn't know I had spammed three. If anyone ever sees me post multiples of the same image feel free to get rid of the extras.
Edited 6 times by MiningMarsh, triclops200. Last: 17th Nov 2013 -
There are two main reasons to continue using the current save format instead of this one: extensibility, and compressed filesize.
BSON is used so that it's easy to extend the save format with new fields. Likewise with particle data - field descriptors specifying which properties were saved make it easy to save new ones. We are not going to switch back to a format based on specific byte positions having specific meanings.
Specific byte positions having specific meanings was basically what PSv was. It eventually becomes horrible to maintain, as new properties are added over time and more things need to be saved (TPT has to be able to read saves from all versions, from the first version that ever had saving, all the way to the current version. That adds up to a lot of messy code if the layout of save data is dependent on version.).
Storing full float values makes the data compress very poorly, and is in general unnecessary. Integers, or fixed-point numbers if decimal places are needed, are usually sufficient.
Sorry, but I'm not particularly bothered about VB6 compatibility, and I doubt the other developers are. If you really need to use VB6, maybe you could write a converter in C or C++ that reads OPS saves and outputs whatever you think is easiest to read in VB6?
BSON libraries exist for a variety of popular languages, so I would imagine that most other people wanting to write a save parser will have a BSON library available.
As for how the save format might actually be improved: I think the main thing is the mapping of bits in the field descriptor to how many bytes of each property are saved, which evolved over time so is a bit convoluted and likely not optimal. Also worth trying might be storing each particle property in a separate array, as this might group identical values together better and provide better compression.
Edited 6 times by jacksonmj. Last: 15th Nov 2013 -
New versions of the program can have newer versions of the protocol for the save. The save is still extensible even in my way. If a new version of the program introduced a new version of the save spec, then if the save file contains a version number >= that version, TPT will interpret the save file under the newer spec. If the save file contains a version number < that version, then the save file is to be interpreted under the old spec. So yes, my save format is still extensible, by different version numbers indicating how to interpret the file.
-
So is there a BSON parser for VB6?
So far I've had to make my own, very crude (definitely not full featured) BSON parser. It only can read the few types of data packets used by TPT (boolean, int32, and byte array). It completely ignores the subdocument type by skipping the correct number of bytes over it (no recursive capability in my parser, so nothing can be embedded within anything here, cause VB6 doesn't do recursive stuff very well).
I've gotten it finally to successfully split the data into different files (one for each BSON packet, minus the subdocument packet called "origin"). So I've got a number of files, including the very important parts and partsPos and wallMap files. Unfortunately the parts partsPos and wallMap arrays are not in a format that has been published into any easy-to-read documentation. I'm waiting for someone to do that.
However I think that the implementation of TPT save should be a lot easier than this.Edited 2 times by Videogamer555. Last: 15th Nov 2013 -
But not as easily as the current format. As I said, we tried that sort of thing with the old PSv save format and it's a lot less pleasant to maintain.
I don't know of a BSON parser for VB6, and there isn't one listed on the BSON website. And as I said in http://tpt.io/.258510, I believe boxmein is writing some documentation for those arrays.
-
@jacksonmj (View Post)
Am indeed writing a documentation.
@Videogamer555 (View Post)
Please start talking about a better format when you move off Visual Basic 6. That language sucks. -
