Chunked storage

The placement and connectivity data the BSB produces is naturally chunked: a chunk is a fixed-size axis-aligned region of space, identified by an integer ID. Workers process one chunk at a time, and the engine stores each chunk’s data under its own HDF5 group so workers can append without contending on the same dataset.

This page describes how that chunking lands in the file and how to use the ChunkLoader mixin to read it back.

The chunk size attribute

The root of the file carries a chunk_size attribute, set the first time any resource writes a chunk. Every subsequent write must match it (mismatch raises). This is the source of truth for the spatial size of every chunk in the file.

Chunks at rest

A ChunkLoader lives at self._path. Each chunk it owns gets a group at self._path/<chunk.id>. Inside that group:

• One dataset per ChunkedProperty, named after the property (positions, rotations, morphology, labels for a PlacementSet).

• One group per ChunkedCollection, named after the collection (additional for a PlacementSet). The collection group holds one dataset per arbitrary key stored under it.

Example for /placement/granule_cell:

/placement/granule_cell/
├── 12345/                 (chunk id 12345)
│   ├── positions          (N, 3) float
│   ├── rotations          (N, 3) float
│   ├── morphology         (N,) int
│   ├── labels             (N,) int
│   └── additional/        (group)
│       └── ...
├── 12346/
│   └── ...
└── attrs:
    ├── len                (total cells across all chunks)
    └── chunks             (JSON: per-chunk-id count)

The ChunkLoader mixin

A resource that wants chunked storage inherits from ChunkLoader and declares its properties / collections in the subclass parameters:

class PlacementSet(
    Resource,
    ChunkLoader,
    IPlacementSet,
    properties=(
        lambda loader: ChunkedProperty(loader, "position", shape=(0, 3), dtype=float),
        lambda loader: ChunkedProperty(loader, "rotation", shape=(0, 3), dtype=float),
        ...
    ),
    collections=(
        lambda loader: ChunkedCollection(loader, "additional", shape=None, dtype=float),
    ),
):
    ...

The mixin gives the subclass:

• get_loaded_chunks returns the chunks currently in the load filter, or every chunk in the file if no filter is set. Accepts handle= so a decorated caller can pass its handle on to this non-decorated helper. See Handles.

• get_all_chunks reads the keys of self._path and returns the full chunk list. Decorated with @handles_handles("r") so it opens its own handle if not given one.

• get_chunk_path returns the full HDF5 path of a chunk’s group, or a property/collection dataset inside it.

• require_chunk creates the chunk group + property datasets on first write. Decorated with @handles_handles("a").

• include_chunk, exclude_chunk, set_chunk_filter, clear_chunk_filter manage the in-memory load filter.

• chunk_context is a context manager that temporarily replaces the load filter for the duration of a block.

The chunk-id ordering matters

ChunkedProperty.load concatenates chunks in the order they appear in get_loaded_chunks. The order must be stable between writers and readers; chunk lists are produced via bsb.storage._chunks.chunklist() which sorts on the integer chunk id.

If you bypass the chunk loader and read raw chunk groups directly, you must sort the same way before concatenating, or downstream consumers (load_ids, load_positions, label masks) will desync.

Stats and the global chunks attribute

The file root carries a JSON chunks attribute mapping chunk id to placement and connection counts:

{
  "12345": {"placed": 1024, "connections": {"inc": 0, "out": 0}},
  "12346": {"placed":  512, "connections": {"inc": 0, "out": 0}}
}

This is maintained by PlacementSet._track_add (on append) and the connectivity-write paths. It lets the BSB report counts and decide work-distribution without having to walk every chunk group.