LodeBase repository format, v1
Status: v1, frozen (Nov 13, 2026). Chunk parameters were chosen from a 40 GiB Unreal 5.6 project (see "Why these numbers"). Anything marked frozen below cannot change without a format v2, because it changes object ids and breaks dedup.
Reference implementation: crates/lode-core.
Object kinds and ids
| Kind | Code | BLAKE3 derive-key context (frozen) |
|---|---|---|
| Chunk | 1 | lodebase v1 chunk |
| Manifest | 2 | lodebase v1 manifest |
| Tree | 3 | lodebase v1 tree |
| Changeset | 4 | lodebase v1 changeset |
id = BLAKE3-256 derive_key(context(kind), payload), 32 bytes, shown as lowercase hex. A separate context per kind means a chunk can never collide with a tree that happens to contain the same bytes.
Payloads
- Chunk payload: the raw file bytes of that chunk.
- Manifest / Tree / Changeset payload: one version byte (
0x01) followed by the postcard encoding of the struct below. Postcard has exactly one encoding per value.
rust
struct ChunkRef { id: [u8; 32], size: u32 }
struct FileManifest { size: u64, chunks: Vec<ChunkRef> } // empty file = no chunks
struct FileEntry { manifest: [u8; 32], size: u64, executable: bool }
enum EntryKind { File(FileEntry), Dir([u8; 32]) } // Dir holds a tree id
struct TreeEntry { name: String, kind: EntryKind }
struct Tree { entries: Vec<TreeEntry> } // sorted by name bytes, unique
struct Changeset { parents: Vec<[u8; 32]>, root: [u8; 32], author: String,
message: String, timestamp_ms: i64, metadata: Vec<(String, String)> }Tree rules: names are non-empty UTF-8, not . or .., no /, \ or NUL; entries sorted by byte order; no duplicates. Decoders reject non-canonical trees. Empty directories do not exist (a tree with no entries is only valid as the repo root).
Paths on the wire are workspace-relative with / separators.
Stored form (frozen)
Every object is stored and transferred sealed: one codec byte, then data.
| Byte | Meaning |
|---|---|
0x00 | raw payload |
0x01 | zstd(payload), any level |
Writers use zstd level 3 and keep it only if it saves at least 2%; otherwise they store raw (most Unreal assets are already compressed). Receivers decode (capped at 64 MiB) and check the hash before accepting. Object store key: <kind>/<hex[0..2]>/<hex>.
Chunking (frozen)
FastCDC 2020, normalization level 1, gear seed 0 (fastcdc crate v5), the same for every file size: min 64 KiB, average 256 KiB, max 1 MiB.
Why these numbers
lode-bench on RoveSoSimulator (UE 5.6, 16,599 files, 39.96 GiB; 71% of bytes in 229 files of 64 MiB or more):
| Set (min/avg/max KiB) | Unique chunks | Dedup saved | Manifests |
|---|---|---|---|
| 16/64/256 small + 256/1024/4096 large (draft) | 180,109 | 2.34 GiB | 6.4 MiB |
| 8/32/128 + 128/512/2048 | 344,788 | 2.42 GiB | 12.2 MiB |
| 32/128/512 + 512/2048/8192 | 97,563 | 2.23 GiB | 3.5 MiB |
| 16/64/256 everywhere | 514,920 | 2.48 GiB | 18.4 MiB |
| 64/256/1024 everywhere (v1) | 140,553 | 2.39 GiB | 5.0 MiB |
- Storage barely differs between sets (0.25 GiB on 40 GiB), so it didn't decide.
- v1 has fewer objects than the draft and better dedup, while its 256 KiB average inside large files makes re-saving a big asset upload ~4x less than the draft's 1 MiB.
- One parameter set for all sizes avoids the draft's flaw where a file growing past 64 MiB re-chunks (and re-uploads) entirely.
- zstd saved only ~2% overall, hence the raw codec.
Commit semantics
A submit is a list of per-path compare-and-swaps: path currently holds manifest expected (or nothing), replace it with new (or delete). The server applies the list to the current branch head, not to the client's base, so non-overlapping submits never conflict (server-side rebase). A path is a conflict if its current content matches neither expected nor new. Changeset numbers are allocated per repo, in commit order, starting at 1.