The exnode.
An exnode is the per-file metadata object that makes LStore configurable per dataset. It records the hierarchy of segments that hold a file's bytes — their encoding, placement, integrity options, and lifecycle behavior — as the file's own metadata rather than as a system-wide setting. This page describes what the exnode contains, how segments compose, what the segment drivers do, and how exnodes are inspected and modified in practice.
A small document that describes one file.
Every file in LStore has its own exnode — a small document, separate from the file's contents, that records how the file is laid out across the storage substrate. The contents live in fragments distributed across IBP depots; the exnode is the map that knows how to reassemble those fragments when the file is read, and how to update the layout when the file is written.
The architectural reason the exnode exists is straightforward. A filesystem that wants to give every file independent policy — independent encoding, independent placement, independent integrity options — needs somewhere to put that policy. A system-wide configuration file cannot do it; the file has to carry its own configuration with it. The exnode is that per-file configuration, stored as a structured document associated with the file, separate from but addressable through the file's identity in the filesystem.
When an application opens a file, the system reads the exnode first. The exnode tells it which segments compose the file, in what arrangement, under which encoding, with which placement constraints. The actual data path that follows — the I/O against the depots — is shaped by what the exnode says. When the file is written, the exnode is updated to reflect the new layout. Two files in the same directory can have completely different exnodes, with different segment compositions, different erasure parameters, different placement policies, and the filesystem treats both as ordinary files.
This is what gives LStore its per-dataset configurability claim weight. Encoding, placement, integrity, and lifecycle behavior are not appliance defaults applied uniformly; they are properties of each file's exnode, set when the file is created or modified later. The mechanism that makes per-dataset policy possible is the exnode being a document rather than a row in a global table.
Sections, segments, attributes.
An exnode is structured as a set of sections, each describing one piece of the file's layout. The top section is the exnode header itself; below it sit one or more segment sections, each describing one segment of the file. The relationships between sections — which segment wraps which — produce the file's segment tree.
In practical terms, a minimal exnode for an empty file with default policy contains three segments arranged in a recursive composition. A cache segment sits at the top, providing in-flight handling for arbitrary read and write offsets and lengths. Below it, a jerasure segment provides Reed-Solomon erasure encoding, dividing data into fragments with calculated parity. Below that, a lun segment maps the encoded fragments to actual depot allocations. The exnode's view section declares which segment is the entry point for the file's data — the cache segment, by default, since that is where the application's read or write first arrives.
Alongside the segments, the file carries a set of attributes — the file size, ownership, timestamps, the segment composition just described, and policy-related metadata such as integrity counters (block-level error counts, both soft and hard). Attributes in the exnode are key-value pairs stored as part of the file's metadata, separately from the data. The Resource Service uses certain attributes when interpreting placement queries; the cache layer uses others to decide what to retain; the audit and repair processes use the integrity counters to detect and respond to data degradation over time.
The arrangement is deliberately uniform: every file's metadata follows the same structural pattern, even though the specific segments, drivers, and parameters can differ from file to file. The structural uniformity is what allows the same client code to operate against files configured very differently; the parametric variability is what allows each file's storage to be tuned to its own dataset's needs.
Each segment is a complete I/O contract.
LStore's segment model treats each segment as a self-contained I/O contract: a segment knows how to handle reads and writes against an offset range, and how to delegate the actual work to whatever is below it. Segments compose because they speak the same interface to their callers and to whatever they wrap.
The default exnode demonstrates the model concretely. A read against the file enters at the cache segment, which maps arbitrary offsets and lengths to the page granularity the jerasure segment requires. The cache segment passes the request down to the jerasure segment, which breaks the request into chunks — typically a (6+3)×16 KB jerasure stripe — and adds a small page identifier to each chunk to detect out-of-sync allocations. The jerasure segment then issues the encoded operations against the lun segment beneath it, which maps each chunk to the specific depot allocation that holds it and issues the actual I/O. Reconstruction, when needed, runs back up through the same path: the lun segment retrieves available fragments from the depots, the jerasure segment reconstructs the missing fragments using the parity it holds, and the cache segment hands the rebuilt data to whatever called it.
This composition is recursive in principle. Nothing in the model prevents a jerasure segment from wrapping another jerasure segment, or a cache segment from wrapping a mirror segment, or a more elaborate composition that combines several drivers in a way no appliance product would ship as a single configuration. In practice, the default three-segment composition handles the common case; other compositions appear when specific datasets need them — replicated rather than erasure-coded, or layered for tape integration through a log segment, or addressed through a mirror segment for redundant storage paths.
The architectural property worth surfacing is that the segment tree is metadata, not code. Adding a new segment driver does not require changing the filesystem's I/O code; it requires implementing the segment interface and adding the driver as a plugin. Once it is present, exnodes can compose with it the same way they compose with the standard set. New storage classes, new integrity strategies, new tape lifecycle drivers — each enters the system as a segment driver, and from that point forward, can appear in any exnode that wants it.
The segment tree is metadata, not code. New storage strategies enter LStore as new segment drivers and become available to every file that wants them.
Six drivers, six compositional roles.
The current LStore distribution ships with six segment drivers. Each fills a specific compositional role in the exnode — some are common in nearly every exnode, others apply to specific operational patterns. The set is not closed; new drivers can be added without disturbing the rest of the system.
The cache driver is the topmost segment in nearly every exnode. Its role is to translate arbitrary offset-and-length requests from applications into the page-aligned chunks that the layers beneath require. It also retains recently-used pages in memory, which has the operational consequence that adjacent reads to the same file benefit from already-loaded data. The cache driver has its own configurable plugin set — least-recently-used, adaptive multi-policy, round-robin, direct — chosen based on the workload pattern the file or dataset is expected to face.
The jerasure driver is where Reed-Solomon erasure encoding happens. It divides data into chunks, calculates parity chunks, and stores both as fragments in whatever segment it wraps. The driver supports the full set of encoding methods from the underlying Jerasure library — Reed-Solomon, Cauchy variants — and the choice of method, the chunk size, and the data-and-parity counts are all parameters carried in the exnode rather than fixed system defaults.
The lun driver is the workhorse that maps fragments to actual depot allocations. It speaks IBP directly and is where the abstract bytes from above become concrete operations against specific resources on specific depots. The lun driver carries the per-fragment depot mapping in its segment section — n_devices, chunk size, and placement query parameters that the Resource Service interprets when choosing depots for new allocations. Resource Service & Placement covers the query language and the placement model in depth.
The log driver provides versioning and snapshot capabilities. Where a cache-jerasure-lun composition handles current data, adding a log segment to the composition records changes over time and supports point-in-time access to earlier versions of the file. The log driver is also a foundational piece of the archive and tape integration story, where lifecycle progression is recorded as part of the file's history rather than as a separate index.
The mirror driver provides replicated storage where erasure encoding is not the right choice — small files where the encoding overhead exceeds the durability benefit, or datasets where reconstruction latency matters more than storage cost. A mirror segment wraps multiple child segments and ensures that writes propagate to all of them, with reads served from whichever returns first.
The file driver is the simplest member of the catalog. It maps a segment directly to a local file on the depot's filesystem rather than to an IBP allocation. The file driver is primarily used for development and testing — running LStore against local storage to exercise the segment composition logic without needing a full depot infrastructure — but it is also useful for small-scale deployments that do not need the depot abstraction.
What the catalog is not is a fixed list. The driver model is exactly the plugin model described in Architecture — Layer 4: a driver is a shared library that implements a small interface and registers itself with the segment service at startup. Adding a new driver requires implementing the interface and dropping the library into the plugin directory. The existing exnodes do not change; new exnodes can compose with the new driver immediately.
The exnode is where decisions live.
The phrase per-dataset policy appears throughout this section's coverage of LStore. The exnode is the place where that phrase becomes operational. Decisions about how a file is stored are not made by a global configuration applied uniformly to all files in the system; they are made when the exnode is constructed or modified, and they live in the exnode for the life of the file.
Encoding parameters — which erasure method, how many data chunks, how many parity chunks, what chunk size — are recorded in the jerasure segment's section of the exnode. A file written with Reed-Solomon 6+3 carries those parameters in its exnode and the system uses them on every operation against that file. A file written with Cauchy 8+4 carries different parameters and the system uses those. The two files coexist in the same filesystem, in the same directory if needed, and the encoding decision is a property of each file rather than a property of the system.
Placement constraints — which depots are eligible to hold this file's fragments — are carried as query parameters in the lun segment. The query language is rich enough to describe constraints like “hold this file on depots tagged as part of this institution's pool, restricted to drives of this capacity class, with these performance characteristics.” The Resource Service interprets the query when new allocations are needed and returns depots that satisfy it. Two files in adjacent directories can have completely different placement constraints, and each placement decision is made against the live state of the depot pool at the time the allocation is requested.
Integrity options — block-level checksums, soft and hard error attributes, audit cadence — are carried at the depot allocation level but propagate through the exnode's attributes. Files whose exnodes have integrity counters elevated by ongoing audit activity surface that information without needing to consult external systems. Reliability & Integrity covers the audit and repair machinery in depth.
Lifecycle stage — whether the file is current data, has been versioned to a log segment, or has been transitioned to an archive tier — is also visible in the exnode's segment composition. A file that has been archived to tape has a log segment in its composition that records the transition; a file that is being mirrored across geographies has a mirror segment that records the parallel storage. The lifecycle status of any file in the system is readable from its exnode.
An exnode is not a record about a file. It is the file's contract with the storage system, written down where every operation can read it.
The operational exnode.
Exnodes are not just an architectural concept — they are a working part of every LStore deployment that operators interact with directly. Two command-line tools cover the common operations: lio_getattr retrieves the exnode for a given file and renders it as a readable document; lio_setattr updates exnode attributes, including swapping out the segment composition for a different one.
The lio_getattr command is the standard way to inspect what a file's exnode currently contains. It returns the full segment composition — cache segment with its parameters, jerasure segment with its encoding choice and chunk geometry, lun segment with the actual depot allocations and their IDs — along with the file's attributes. Operators use this routinely when diagnosing performance, verifying that a file was written with the expected encoding, or auditing placement to confirm that a dataset is distributed across the depots it should be on.
A companion tool, lio_signature, reports the encoding signature of a file — the composition and its parameters — in a compact form useful for confirming at a glance how a file is stored. The output names each segment in the composition and lists the jerasure and lun parameters directly:
$ lio_signature --fcreds /etc/lio/lfs-example.psk -c @@example.accre.vu @:/alan/walk.dummy
cache()
jerase(
method=cauchy_good
n_data_devs=6
n_parity_devs=3
chunk_size=16384
magic_cksum=1
w=-1
)
lun(
n_devices=9
n_shift=0
chunk_size=16388
)
The signature makes the per-file encoding visible without rendering the full exnode: the jerase block shows the Cauchy Good method with six data and three parity devices at a 16 KB chunk size, and the lun block shows the nine devices the resulting fragments map to. It is the quickest way to confirm that a file was written with the encoding its dataset intended.
The lio_setattr command is the corresponding tool for modification. Some changes are straightforward attribute updates; others require more careful handling, particularly the case of changing a file's encoding parameters. The standard pattern for changing encoding on existing data is to write the new exnode to an empty directory, copy data into that directory under the new encoding, and replace the original. Once the new exnode is in place for a directory, files created in that directory inherit the new composition. Existing files keep their existing exnode unless explicitly migrated, which preserves the data while letting new writes use the updated policy.
What makes this practical is that exnodes are documents, not opaque structures. An operator inspecting an exnode can read it directly, understand the composition, and reason about what a different composition would do. Changes to encoding, placement, or composition are not abstracted behind an appliance's UI; they are visible decisions an operator can examine, plan, and apply.
The Object Service stores them.
An exnode has to be stored somewhere. In LStore, the Object Service — one of the four services that run on the LServer — is the container that holds it, alongside the file's other metadata attributes. The Object Service does not interpret or process those attributes; it stores them, atomically, and serves them on request. Reading an exnode is a request to the Object Service for that attribute. Writing one is the same.
The Object Service stores each file's exnode as one attribute of the file in the filesystem's namespace. The namespace itself is a tree of directories, files, hardlinks, and symlinks that the Object Service maintains on its own local storage. Each file in the namespace has a set of attributes attached to it — the exnode is one of those attributes, alongside the file's size, ownership, timestamps, and any other metadata the system or the user has associated with it. The exnode is not mandatory: an object can exist in the namespace without one. The reason the exnode receives so much architectural attention is that the LIO file system treats that particular attribute specially — using it as the recipe for how the file's bulk data is laid out across depots. The Object Service itself does not act on the exnode; it stores it like any other attribute and hands it back when LIO asks for it.
The Object Service's implementation has its own significant depth: how the namespace is laid out on disk, how attribute directories are organized, how hardlinks are handled, how the namespace can be sharded across multiple devices when the metadata workload outgrows a single backend, how inode tracking supports NFS persistent file handles, how a Time Cache implementation can reduce load on a remote OS for clients that access many files repeatedly. That depth belongs in the LServer documentation rather than here, since it is properly a property of the metadata service rather than of the exnode itself. It is the subject of the dedicated LServer page, which covers the Object Service and the broader metadata-server architecture.
The relationship worth carrying forward from this page is the architectural separation. The exnode is a document the file owns. The Object Service is the service that stores documents and serves them on request. Changes to where exnodes are stored, how the Object Service is replicated, or how metadata workloads are distributed across multiple servers affect the storage and retrieval of exnodes — not the structure of the exnodes themselves. The exnode is portable across implementations of the metadata service in the same sense that any document is portable across storage backends.
The format is not load-bearing.
An exnode is a document. The architecture does not commit to any particular serialization format for that document — it commits to the document's structure and to the segment composition model. The serialization is an engineering choice that has changed over time and can change again without disturbing the rest of the system.
The original exnode serialization used XML. XML is machine-readable in a precise, schema-validated way, and the early generations of LStore relied on that precision. In practice, XML is also heavyweight to parse, particularly for a metadata structure that is read on every file open. The format choice was reconsidered and the current default is an INI-style serialization — lighter to parse, also human-readable, and adequate for the structural depth an exnode requires. Stubs for the XML pathway remain in the codebase but have atrophied; they are not currently exercised in production.
The architectural point worth carrying out of this discussion is that the exnode's format is incidental to what the exnode does. What is load-bearing in LStore is the segment composition model, the per-file independence of policy, and the document-shaped representation. The serialization format is an engineering detail that has been reconsidered before, can be reconsidered again, and does not affect the architecture above or below it. This is the same plugin philosophy that defines the rest of LStore: a piece of the system is replaceable as long as it speaks the right contract to its neighbors.