Skip to content

Unpacked Loadout

This lists the format of the binary data used by an unpacked loadout.

For the location of folder containing unpacked loadout, see the Locations page.

Item Path Description
Header header.bin Header with current loadout pointers. Facilitates 'transactions'.
Events events.bin List of all emitted events in the loadout.
Timestamps timestamps.bin Timestamps for each commit.
Commit Parameters commit-parameter-types.bin
+commit-parameter-lengths-{x}.bin
+ commit-parameters-{x}.bin		 List of commit message parameters for each event.
Configs config.bin
+ config-data.bin		 Package Configurations.
External Configs external-config.bin
+ external-config-data.bin
+ external-config-paths.bin		 Package Configurations.
Package Reference (IDs) package-ids.bin Hashes of package IDs in this loadout.
Package Reference (Versions) package-versions-len.bin
+ package-versions.bin		 String versions of package IDs in this loadout.
Store Manifests stores.bin
+ store-data.bin		 Game store specific info to restore game to last version if possible.
Commandline Parameters commandline-parameter-data.bin Raw data for commandline parameters. Length specified in event.

These files are deliberately set up in such a way that making a change in a loadout means appending to the existing files. No data is overwritten. Rolling back in turn means truncating the files to the desired length.

In some cases, data is grouped to improve compression ratios by bundling similar data together when sharing.

And in other cases, we put cold data that is infrequently accessed, e.g. commit message params in a separate file as that information is rarely accessed.

All values are in little endian unless specified otherwise.

They are shown in lowest to highest bit order.
So an order like u8, and u24 means 0:8 bits, then 8:32 bits.

header.bin

This is a master file which tracks the state of other files in the loadout.

This stores the version of the loadout and structure counts for remainder of the loadout files.

In the event of an unexpected crash, this file is used to determine the last state of the Loadout before performing a cleanup of unused data (by truncating remaining files).

Format:

Data Type Name Description
u16 Version Version of the loadout format.
u16 Reserved
u32 NumEvents Total number of events and timestamps in this loadout.
u32 NumPackageIds Total number of unique package ID(s) files in this loadout.
u32 NumPackageVersions Total number of unique package version(s) files in this loadout.
u32 NumConfigs Total number of package configuration files in this loadout.
u32 NumGameVersions Total number of game versions (store entries).
u32 NumExternalConfigs Total number of external configuration files in this loadout.

Backwards compatibility is supported but not forwards.

If you're loading a Version that is newer than what you support, you should reject the file to avoid errors.

events.bin

This file contains all of the events that occurred in this loadout.

Each event has a 1:1 mapping to a timestamp in timestamps.bin. The number of events stored here is stored in header.bin.

The event format is documented in the Event List Page.

As a summary. Each event is composed of an u8 EventType and 0, 8, 24 or 56 bits of InlineData (depending on EventType). Events are laid out such that they align with 8 byte boundaries.

Any data that doesn't fit in the InlineData field is stored in another file and loaded by index. Details of that can be seen on each individual event entry.

Optimizing Events

Sometimes events can be optimized.

For example, if a package is added and then immediately enabled, we can cancel out the events.

As the nature of the events is such that they are always appended, we don't do this during normal operation. However, when we pack the loadout we will run certain clever optimizations like this to reduce clutter and save space.

Situations where optimizations are applied at pack stage will be noted in the event's description.

timestamps.bin

This contains the timestamp for each event.

Each timestamp here corresponds to an event in events.bin.

This is an array of 32-bit timestamps (R3TimeStamp[]). The number of items is defined in header.bin.

config.bin

This stores all historical package configurations for any point in time.

This is the array of file sizes, each being:

Data Type Name Description
u16 FileSize Size of the configuration file.

Every new config is appended to config-data.bin as it is added.

Each unique config has an index, a.k.a. ConfigIdx, which is an incrementing value from 0 every time a config is added. Emitted events refer to this index.

How do you use this data?

When loading a loadout, calculate the offsets of each config in memory, by iterating through the FileSize field(s).

  • First config is at 0
  • Second is at 0 + FileSize(first)
  • Third is at second + FileSize(second).

etc.

As you do this, hash the configs. For this, the snapshots use XXH3. When a new config is created, hash it and check if it's a duplicate, if it isn't, add it to the config list.

config-data.bin

This is a buffer of raw, unmodified, unique configuration files.

You can get the file size and offsets from the config.bin file.

external-config.bin

This stores all historical external configurations for any point in time.

External configurations are these generated by tools, as opposed to the ones generated by R3.

This is the array of file sizes, each being:

Data Type Name Description
u32 FileSize Size of the configuration file.

Every new config is appended to external-config-data.bin as it is added.

The path of every config is appended to external-config-paths.bin.

Each unique config has an index, a.k.a. ConfigIdx, which is an incrementing value from 0 every time a config is added. Emitted events refer to this index.

How do you use this data?

See: config.bin's How do you use this data? section.

external-config-data.bin

This is a buffer of raw, unmodified, unique configuration files.

You can get the file size and offsets from the config.bin file.

external-config-paths.bin

This is a buffer of paths to configuration files generated by external tools/packages/programs.

This is a list of unique String8s (1 byte prefixed UTF-8 Strings).

Each path is a path relative to the path specified in Config File Paths.

This is best illustrated with an example.

For a config with the path set to

[[ConfigFiles]]
Id = 3
Type = "Folder" # or 'ByExtension
Description = "OS-specific data folder"
[[ConfigFiles.Paths]]
OS = "any"
Path = "{LocalAppData}/ToolName/Data"

Storing a path of config.json in external-config-paths.bin would equal the final path to be:

  • "{LocalAppData}/ToolName/Data/config.json"

If the Type is File. An empty path would be used instead.

Packages

We refer to a 'unique package' by XXH3(PackageID)

The field PackageID being the Id field from Package.toml.

There can only be one version of a package with given ID in a loadout at a given time.

References to Package IDs (XXH3(PackageID)) are referred to by an index known as PackageIdIdx in the events:

  • A PackageIdIdx == 1 in an event means fetch the entry at index 1 of package-ids.bin.

Most events will only require a PackageIdIdx. However, in some cases the version PackageVerIdx is also needed, for example, to upgrade packages.

As for how to use the data, it is similar to config.bin. We deduplicate entries by in-memory hash. So an event can always refer to a PackageIdIdx created in an earlier event to save space.

Launcher MUST ensure each published mod has valid update/download data.

Otherwise this system could fail, as a hash of packageID on its own is not useful.

package-ids.bin

This is a buffer of XXH3(PackageID)

Each entry is 8 bytes long.

Using a 64-bit hash, we need around 5 billion hashes until we reach a 50% chance of collision, that's quite plenty!

System can still always fail, we just pray it won't.

Some Numbers

Nexus Mods alone hosts 815999 mods as of 30th of May 2024 (obtained via GraphQL API).

The probability of a hash collision on whole mod set is roughly the following:

>>> r=815999
>>> N=2**64
>>> ratio = 2*N / r**2
>>> ratio
55407743.67551148
>>> 1-math.exp(-1/ratio)
1.8048018635141716e-08

That ends up being ~0.0000018% I'll be damned if R3 comes anywhere close to that.

Anyway, assuming a more modest '100000' mods will be made in R3's lifetime, we can expect a probability of 0.0000000027%, or more than 1 in 3.7 trillion.

If I'm ever that successful, I'd probably be funded enough that I could extend this to 128-bit hash, and at that point a meteor is more likely to land on your house (no joke).

This ID is used to restore the package.

Package Versions

In some contexts, it may also be useful to know the package version.

Reference to unique Package Version are referred to by an index known as PackageVerIdx in the events:

This information is sometimes used to e.g. upgrade packages.

package-versions-len.bin

Contains the lengths of entries in package-versions.bin.

Data Type Name Description
u8 VersionLength Size of the Version string.

This data compresses extremely well.

Most versions are of form X.Y.Z so there is a lot of repetition of 05.

package-versions.bin

This is a buffer consisting of package versions, whose length is defined in package-reference.bin

These versions are stored as UTF-8 strings. No null terminator.

This data compresses extremely well.

Because the randomness (entropy) of values is low, the version components are super commonly 1s and 0s, and almost always all first two numbers 0-1 and dot .

Restoring Actual Package Files

We follow a multi step process in order to reliably try restore Reloaded3 packages.

First we attempt to obtain full package metadata from Central Server.

But what if Central Server is down?

We will query the Static CDN API. That contains a dump of the latest package update info.

stores.bin

This stores all game store specific info.

Why do we store this info?

This info can be used to identify the game when you share the loadout with a friend, and the game isn't known by the Community Repository.

Or in the event that you cloud sync a game (between your machines) that's not known by the Community Repository.

It can also be used to identify when game updates have taken place when auditing the log.

Data Type Name Description
u8 (StoreType) StoreType The store from which the game came from.
u16 FileSize Size of the configuration file.
u8 Currently Unused

The offsets can be derived from file sizes.

Basically this contains data specific to game stores such as GOG, Steam, Epic etc. that can be used to revert the game to an older version.

Reverting to earlier versions is not possible in all game stores.

store-data.bin

When values, e.g. strings are not available, they are encoded as 0 length strings, i.e. constant 00.

  • String8 is assumed to be a 1 byte length prefixed UTF-8 string.
  • String16 is assumed to be a 2 byte length prefixed UTF-8 string.

CommmonData Struct

This struct is shared between all store entries.

i.e. This game was manually added.

Data Type Name Description
u64 ExeHash The hash of the game executable (using (XXH3))
String16 ExePath The path to the game executable
String8 AppId The application ID of the game

We store this for every game, regardless of store.

Unknown

Data Type Name Description
u8 Version The version of the structure
CommonData CommonData The common data structure

Steam

Data Type Name Description
u8 Version The version of the structure
CommonData CommonData The common data structure
u64 AppId The Steam application ID
u64 DepotId The Steam depot ID
u64 ManifestId The Steam manifest ID
String8 Branch The Steam branch name
String8 BranchPassword The password for the Steam branch (if password-protected)

To perform rollback, will maintain basic minimal change fork of DepotDownloader, no need to reinvent wheel. Manifest contains SHA checksums and all file paths, we might be able to only do partial downloads.

To determine current version, check the App's .acf file in steamapps. The InstalledDepots will give you the current Depot and Manifest ID. Steam does not unfortunately have user friendly version names.

To determine downloadable manifests, we'll probably have to use SteamKit2. Use DepotDownloader code for inspiration.

GOG

Extended details in Stores: GOG.

We can get the info from the registry at HKEY_LOCAL_MACHINE\Software\GOG.com\Games\{GameId}

Data Type Name Description
u8 Version The version of the structure
CommonData CommonData The common data structure
u64 GameId The unique identifier for the game on GOG
u64 BuildId The unique identifier for the build
String8 VersionName The user-friendly version name for display purposes

The VersionName is also copied into the commit message on each update.

To identify the version reliably, it seems we will need to compare the hashes against the ones in the different depots.

This will also allow us to support e.g. Heroic on Linux.

Heroic & Playnite

These are 3rd party launchers that support GOG

They need to be supported, because there's no official Linux launcher.

TODO: To be determined.

Epic

Version downgrade with Epic isn't possible.

We will store the minimal amount of data required to identify the game in the hopes it is one day.

With Epic we can nip this data from C:\Program Data\Epic\EpicGamesLauncher\Data\Manifests. We want the following:

Data Type Name Description
u8 Version The version of the structure
CommonData CommonData The common data structure
u128 CatalogItemId The MD5 hash identifier for the game on Epic
String8 AppVersionString The version string of the game on Epic

These values are directly extracted from the manifest file.

Microsoft

Version downgrade with Microsoft isn't possible.

We will store the minimal amount of data required to identify the game in the hopes it is one day.

We're interested in AppXManifest.xml in this case.

Data Type Name Description
u8 Version The version of the structure
CommonData CommonData The common data structure
String8 PackageFamilyName The unique identifier for the game on the Microsoft Store. {Identity.Name}_{hash(Identity.Publisher)}
String8 PackageVersion The version of the game package on the Microsoft Store, from Identity field.

The PackageVersion is actually a four part version, but is stored as string, so just in case an invalid version exists in some manifest, we will string it.

commandline-parameter-data.bin

This file contains the raw strings for commandline parameters. The lengths of the parameters are specified in the UpdateCommandline event.

Commit Parameters

These files contain the parameters for any event that requires additional info in its commit message.

The Commit Message file lists when messages appear in this file for each message.

When the message is not a contextual-parameter, it is stored in this file.

A timestamp is shown beside each event, it does not need to be embedded into description.

An Example

You emit the PackageStatusChanged event with the message commit-messages-packageadded:

Added '**{Name}**' with ID '**{ID}**' and version '**{Version}**'.

Which could be marked as:

Added '**Super Cool Mod**' with ID '**reloaded3.utility.somexample**' and version '**1.0.0**'

The Version is a 'Contextual Parameter', and thus is derived from context.

It is not stored in the commit parameters.

Encoding

Parameters are encoded in the order in which they appear in the template!!

This would be encoded as:

  1. commit-parameter-types.bin: [0, 0]

    Explanation:

    • 0: UTF-8 Char Array for "Super Cool Mod" (Name)
    • 0: UTF-8 Char Array for "reloaded3.utility.somexample" (ID)

  2. commit-parameters-lengths-8.bin: [14, 28]

    Explanation:

    • 14: Length of "Super Cool Mod"
    • 28: Length of "reloaded3.utility.somexample"

  3. commit-parameters-text.bin:

    • Super Cool Mod
    • reloaded3.utility.somexample

    These strings are written directly to the commit-parameters-text.bin file, without any null terminator or padding.

  4. commit-parameters-versions.bin: [0]

    Explanation:

    • 0: Version of the commit message.

    It's 0 because this is the initial version of the message format.

With Back References

Suppose you wanted to repeat the earlier parameter, we would use back references.

  1. commit-parameter-types.bin: [5, 5]

    Explanation:

    • 5: BackReference8 for "Super Cool Mod"
    • 5: BackReference8 for "reloaded3.utility.somexample"

  2. commit-parameters-backrefs-8.bin: [0, 1]

    Explanation:

    • 0: Index of "Super Cool Mod"
    • 1: Index of "reloaded3.utility.somexample"

  3. commit-parameters-versions.bin: [0]

    Explanation:

    • 0: Version of the commit message.

    It's still 0 because we're using the same message format, just with back references.

With Back References (Optimized)

Suppose you have multiple parameters to backreference, there are optimized variants.

Let's say we want to reference all three parameters from the previous example in a new event:

  1. commit-parameter-types.bin: [11]

    Explanation:

    • 11: BackReference3_8 for all three parameters

  2. commit-parameters-backrefs-8.bin: [0, 1]

    Explanation:

    • 0: Index of "Super Cool Mod"
    • 1: Index of "reloaded3.utility.somexample"

  3. commit-parameters-versions.bin: [0]

    Explanation:

    • 0: Version of the commit message.

This optimized approach uses a single ParameterType (11: BackReference3_8) to reference all three parameters at once, reducing the overall size of the encoded data.

It's particularly efficient when you need to reference multiple consecutive parameters from a previous event.

Decoding

To construct commit messages from the unpacked loadout data, follow these steps.

  1. Read Events Sequentially:
    Process the events in events.bin in the order they appear.

  2. Determine Commit Message Type:
    Based on the event type, identify the corresponding commit message template from Commit-Messages.md.

    These are listed in the Event List page for each event under the Messages section (Example)

  3. Check Message Version:
    Read the version of the commit message from commit-parameters-versions.bin.
    This ensures you're using the correct message format for that event type.

  4. Read and Process Parameters:

    1. Fetch the pre-parsed message template. (and number of parameters)
    2. Read the parameter types from commit-parameter-types.bin.
    3. Based on the parameter types, retrieve the actual parameter data from the appropriate locations:

  5. Construct the Message: Use the template from step 2 and fill in the parameters obtained in steps 4 and 5.

commit-parameters-types.bin

This is an array of:

Data Type Name Description
u8 ParameterType Type of the parameter.

ParameterType

ParameterType is defined as:

Type Data Type Example Description
0 UTF-8 Char Array (u8 length) Hello, World! UTF-8 characters, length stored in commit-parameters-lengths-8.bin
1 UTF-8 Char Array (u16 length) A longer string... UTF-8 characters, length stored in commit-parameters-lengths-16.bin
2 UTF-8 Char Array (u32 length) An even longer string... UTF-8 characters, length stored in commit-parameters-lengths-32.bin
3 u32 (R3TimeStamp) 1st of January 2024 Renders as human readable time.
4 u32 (R3TimeStamp) 5 minutes ago Renders as relative time.
5 u8 (BackReference8) Entry 1 Reference to a single previous item.
6 u16 (BackReference16) Entry 2 Reference to a single previous item.
7 u24 (BackReference24) Entry 3 Reference to a single previous item.
8 u32 (BackReference32) Entry 4 Reference to a single previous item.
9 variable List See Parameter Lists Defines the start of a list.
10 u8, u8 (BackReference2_8) Entries 1, 2 Reference to two previous items, each index stored as u8.
11 u8, u8, u8 (BackReference3_8) Entries 1, 2, 3 Reference to three previous items, each index stored as u8.
12 u16, u16 (BackReference2_16) Entries 1, 2 Reference to two previous items, each index stored as u16.
13 u16, u16, u16 (BackReference3_16) Entries 1, 2, 3 Reference to three previous items, each index stored as u16.
14 u24, u24 (BackReference2_24) Entries 1, 2 Reference to two previous items, each index stored as u24.
15 u24, u24, u24 (BackReference3_24) Entries 1, 2, 3 Reference to three previous items, each index stored as u24.
16 u32, u32 (BackReference2_32) Entries 1, 2 Reference to two previous items, each index stored as u32.
17 u32, u32, u32 (BackReference3_32) Entries 1, 2, 3 Reference to three previous items, each index stored as u32.

The parameter data is split into multiple files to aid compression:

  • Text is expected to be mostly (English) ASCII and thus be mostly limited to a certain character set.
  • Timestamps are expected to mostly be increasing.
  • Other/Misc integers go in a separate file.
  • Other/Misc floats go in a separate file.

Here is a listing of which parameter types go where:

Type Data Type File
0 UTF-8 Char Array (u8 length) commit-parameters-text.bin
1 UTF-8 Char Array (u16 length) commit-parameters-text.bin
2 UTF-8 Char Array (u32 length) commit-parameters-text.bin
3 u32 (R3TimeStamp) commit-parameters-timestamps.bin
4 u32 (R3TimeStamp) commit-parameters-timestamps.bin
5 u8 (BackReference8) commit-parameters-backrefs-8.bin
6 u16 (BackReference16) commit-parameters-backrefs-16.bin
7 u24 (BackReference24) commit-parameters-backrefs-24.bin
8 u32 (BackReference32) commit-parameters-backrefs-32.bin
9 variable List commit-parameters-lists.bin
10 u8, u8 (BackReference2_8) commit-parameters-backrefs-8.bin
11 u8, u8, u8 (BackReference3_8) commit-parameters-backrefs-8.bin
12 u16, u16 (BackReference2_16) commit-parameters-backrefs-16.bin
13 u16, u16, u16 (BackReference3_16) commit-parameters-backrefs-16.bin
14 u24, u24 (BackReference2_24) commit-parameters-backrefs-24.bin
15 u24, u24, u24 (BackReference3_24) commit-parameters-backrefs-24.bin
16 u32, u32 (BackReference2_32) commit-parameters-backrefs-32.bin
17 u32, u32, u32 (BackReference3_32) commit-parameters-backrefs-32.bin

Commit Parameter Lengths

The following files store the parameter lengths.

These files are only used whenever the used ParameterType requires it.

See the description section of each ParameterType for more information.

commit-parameters-lengths-8.bin

This is an array of:

Data Type Name Description
u8 ParameterLength Length of the parameter in bytes.

commit-parameters-lengths-16.bin

This is an array of:

Data Type Name Description
u16 ParameterLength Length of the parameter in bytes.

commit-parameters-lengths-32.bin

This is an array of:

Data Type Name Description
u32 ParameterLength Length of the parameter in bytes.

commit-parameters-versions.bin

This enables versioning, ensuring that different variations of the same commit message can coexist.

There should be 1 entry for each event!! Regardless of whether it has a message or not!!

This is an array of:

Data Type Name Description
u8 Version Version of the commit message.

The version number corresponds to the version suffix in the message key.

For example:

  • If the message key is PACKAGE_ADDED_V0, the version would be 0.
  • If the message key is MOD_CONFIG_UPDATED_V1, the version would be 1.

This array contains u8 values which correspond to the version of the commit message last issued for each event.

For example, if the message for an event like PackageStatusChanged is encoded with the key PACKAGE_ADDED_V0, it would be written as 0:

Added '**{Name}**' with ID '**{ID}**' and version '**{Version}**'.

However, if a new version of the message is introduced with a different meaning, order of parameters, or number of parameters, it would use a new key like PACKAGE_ADDED_V1, and the version number would be 1.

In practice, expect to see mostly 0, as the text for most commit messages is unlikely to change often. When changes are needed, a new version of the message is created with an incremented version number in its key.

Compressing 1M zeroes with zstd yields file size of ~50 bytes.

Back References

Back References are a Special Type of Parameter that references a previous item.

Back References are used to deduplicate parameters.

The writer maintains a hash of all parameters so far and reuses the same parameter index if the parameter ends up being a duplicate.

This improves loadout sizes by reducing existing previous data.

Back References are defined as 1 or more ParameterIndex fields, whose location and data type depends on ParameterType.

A ParameterIndex of 0 means 'the first commit parameter' in file. 1 means 'the second commit parameter' etc.

These are essentially indices into the commit-parameters-types.bin file.

In order to quickly handle back-references, the reader should keep offsets of all parameters.

That is offsets in their perspective files, e.g. offsets into commit-parameters-text.bin etc.

Parameter Lists

This primitive is used when you have an unknown number of items.

Imagine you have a message which says:

Changes were made, here they are:

{ChangeList}

And you want ChangeList to have multiple items, so it could be something like:

Changes were made, here they are:

- Value **ResolutionX** changed to **1920**
- Value **ResolutionY** changed to **1080**

Where each localizable Change item could be:

- Value **{Name}** changed to **{NewValue}**

This is where Parameter Lists come in.

A Parameter List is defined as:

Data Type Name Description
u8 ParameterType Type of the parameter.
u4 Version [Event Specific] version of the list.
u20 NumParameters Number of parameters.

For the example above, we can treat each Change as 2 parameters. In which case, if we had 2 changes, we would set NumParameters to 4.

The individual parameters for Name and NewValue would then follow as regular parameters in Commit Parameters.

Why is there a Version field?

Sometimes it may be desireable to change the structure. Suppose you wanted to change Change item to also have the previous value:

- Value **{Name}** changed from **{OldValue}** to **{NewValue}**

In order to perform this change, you would set the Version field to 1. So when you read loadouts you can interpret both the old and new format side by side.

Message Template List

Find the full list of templates on the Commit Messages Page.