The architecture.

Each IBP Server Depot is a physical or virtual storage node running the IBP daemon. ACCRE's production deployment runs more than 80 depots holding several thousand drives across nine simultaneously-deployed capacities — from 2 TB drives a decade old through 24 TB drives installed in the most recent refresh cycle. The hardware heterogeneity is not a limitation worked around; it is a design property, particularly relevant in an era of rapid hardware price escalation where the ability to mix old and new capacity preserves prior investment while integrating current generation drives as they become economical. The exact fleet size is a moving target by design — depots are added as hardware becomes available and retired as servers reach end of life, or failed drives being replaced by new bigger drives. The platform’s redundancy is robust enough that multiple drive failures or IBP server outages for upgrades or hardware break/fixes is an operational non-event rather than an emergency.

The LStore Metadata Server — the LServer — runs the services that make the filesystem coherent: the Object Service (filesystem semantics), the Resource Service (placement queries against the depot pool), the Authentication service (capability and identity), and the message-queue broker that connects clients to all of them. The LServer is the metadata authority. It is not, however, in the data path. A client reads or writes a file by going to the depots directly via IBP; the LServer participates only in the metadata exchange around the operation. This separation is what allows LStore's throughput to scale with the number of depots rather than against any single server's capacity.

The LStore Client is whatever machine attaches to an LServer and runs applications against it — via the FUSE mount, the command-line tools, or programmatic bindings. Clients participate in both the data and metadata paths, but the two paths are architecturally distinct: data flows directly to and from depots, metadata flows to and from the LServer.

Current production deploys a single LServer per cluster. The message-queue transport that carries client-to-LServer communication smooths transient network issues between client and server, but it is not a substitute for LServer availability: LStore is designed to fail hard and fail fast. Every operation carries a timeout, and if the LServer does not respond within it, the operation fails and the calling application is responsible for handling that failure. Redundancy against sustained LServer outage — through replicated metadata servers — is a future Object Service driver enhancement. The metadata server has its own dedicated treatment — the four services it runs, how the Object Service stores the namespace on disk, sharding, inode tracking, and the resilience model — on The LServer sub-page.

This is the same trade IP makes for network packets. IP guarantees nothing about delivery — not arrival, not order, not even survival of the packet from one router to the next. What IP guarantees is that if a packet survives, it will be addressable everywhere. The reliability layer above IP (TCP) takes responsibility for retransmission, ordering, and integrity. The unreliable substrate underneath is what makes the reliable abstraction above it possible. Engineering durability into the lowest layer would have made the network rigid; leaving it out made the network fluid.

IBP applies that same insight to storage. The depot does not promise that any single byte array will exist forever. It promises something more useful: that the byte arrays which currently exist are reachable, addressable, and accessed under explicit capability tokens. Durability is the responsibility of the layers above — erasure coding, replication, lease warming, and the exnode metadata that tracks what should exist where. By treating storage as deliberately fluid at the foundation, LStore gains the ability to redistribute, migrate, and reshape data continuously without fighting against the storage abstraction itself.

Practically, this means a depot can host mixed media and mixed capacities side by side. Older 2 TB rotational drives coexist with current 24 TB drives in the same depot, exposed as separate resource IDs. Storage of different cost, performance, and longevity profiles can be mixed within a single namespace and policy can route data to the appropriate class on a per-dataset basis. None of this requires special handling above the IBP layer; it is the natural consequence of treating storage as a leased, capability-addressed substrate.

Capability tokens are the access model. An IBP allocation produces three tokens — read, write, and manage — that are independently transferable. Possession of the token grants the corresponding right; absence of the token denies it. There is no separate authentication state and no notion of session identity at the IBP layer. This is intentional: it keeps the foundation simple, auditable, and fast, while leaving identity, policy, and access governance to the layers above where those concerns naturally belong.

What sits at this layer in the LStore source tree is a single data service plugin, ds_ibp, which implements the depot client. The plugin nature is deliberate: the data service is replaceable. Future foundations — alternative protocols, different distributed storage substrates — can be substituted at this layer without touching anything above it. Today, IBP is the foundation. The architecture does not require it to be the only foundation forever.

Replication — the obvious approach, simply storing multiple copies — is wasteful. Three copies of a file consume three times the storage to survive the loss of any two. Erasure coding does the same job using mathematics: the file is split into k data fragments, and an additional m parity fragments are computed from them. The file can be reconstructed from any k of the k+m total fragments. The cost is far lower than replication; the protection is more flexible.

LStore implements this through the Jerasure library, the open-source Reed-Solomon implementation maintained by James Plank and his collaborators. The default deployment configuration is 6 data fragments and 3 parity — the figure above shows that arrangement — which delivers protection against any three simultaneous fragment losses at fifty percent storage overhead. Triple replication, by contrast, survives the loss of only two copies while consuming 200 percent overhead; erasure coding provides more protection at a quarter of the storage cost, a difference that compounds across petabyte-scale deployments. Jerasure is the encoding engine in production today; support for Intel’s ISA-L (Intelligent Storage Acceleration Library) is in development as a future option, bringing hardware-accelerated encoding to the same per-file model.

The 6+3 default is just that — a default. The encoding parameters are configurable per dataset rather than fixed at the system level. A working dataset that demands faster reconstruction may use a smaller k; an archive dataset that prioritizes storage efficiency over recovery speed may use a larger k. The encoding method itself is also a parameter: cauchy_good (the Cauchy variant tuned by Plank for performance), cauchy_orig, and replication for cases where erasure does not fit. Each dataset's choice is recorded in its exnode — the layer above — rather than baked into the storage system.

What lives on this page is the architectural layer — what erasure coding is, where it sits, and how it is configured. The depth that this page does not cover — reconstruction performance under different fault patterns, the audit and repair processes that run continuously to catch bit-rot, the integrity verification at the block level — is the subject of Reliability & Integrity, the dedicated sub-page within this section. We will visit it shortly as part of the LStore coverage.

Every file in LStore has its own exnode, separate from the file's contents. The contents live in fragments distributed across IBP depots; the exnode is a small document held in a metadata service that knows how to reassemble those fragments into the file. When an application opens a file, the system reads the exnode first, follows it to retrieve the fragments, and presents the result through the filesystem above. When the file is written, the exnode is updated to reflect the new layout.

What this enables is configuration per dataset rather than configuration per system. A dataset of frequently-read working data may be encoded for low-latency reconstruction with a particular placement policy across high-performance drives. A dataset of long-tail archival data, in the same deployment, may use a more space-efficient encoding with a placement policy that prefers cold capacity. A dataset of regulated records may add integrity verification at every block. These are not separate systems — they are the same system, reading exnodes and following the policy each one carries. Single-file capacity reaches 8 exabytes by the same mechanism, since the exnode can describe segment hierarchies of arbitrary depth.

The exnode layer is where most of LStore's distinctiveness lives, and it deserves treatment beyond what an architecture overview can give it. The Exnode, the dedicated sub-page within this section, covers the segment composition model, the placement query language, the integrity options, the lifecycle of an exnode through the metadata service, and the operational tools (lio_setattr, lio_inspect) that make exnodes inspectable and editable. We will visit it shortly as part of the LStore coverage.

Three interfaces share the LIO surface. A FUSE mount — the LStore File System, or LFS — renders LStore as a directory tree, with familiar POSIX semantics, accessible to any application that can open files. A command-line toolset (lio_cp, lio_ls, lio_setattr, lio_inspect, and others) provides operator access for direct work with the namespace. A programmatic binding lets applications that benefit from skipping the POSIX surface call into LIO directly — useful for high-throughput pipelines that move large amounts of data into or out of the filesystem and want to avoid the per-syscall overhead.

Underneath those interfaces, LIO manages the work of holding capability tokens, caching what should be cached, and routing each operation to the right combination of segments and depots based on the file's exnode. The user does not see tokens. The application does not see segment composition. LIO holds those concerns and presents what looks like a filesystem.

What sets LIO apart at the architectural level is the Generic Operation Pipeline — the GOP. The GOP is a parallelism framework that treats different kinds of work as the same kind of object: a depot I/O operation, a local thread task, a message-queue exchange with the LStore metadata server, all flow through one pipeline that can intermingle them with lightweight coordination. Operations of any of these types can be queued together, executed in parallel, and have their results processed as they return. The architectural distinction is not parallelism alone — conventional filesystems parallelize where they can — but the unified treatment of operations across the pipeline, so the layers above can issue work without having to coordinate between operation types.

One consequence of using message queues as one of the operation types follows from the nature of message queueing itself. Routing LServer communication through the GOP’s message-queue path decouples the submission of an operation from a direct synchronous call, lets metadata exchanges intermingle with data and local work in the same pipeline, and absorbs transient network issues between client and server — a brief network interruption does not, by itself, fail an operation. What it does not do is mask an LServer that is actually down. LStore is deliberately designed to fail hard and fail fast: every operation carries a timeout, and if the server does not respond within it, the operation fails and the calling routine handles it. Fast, explicit failure is the design choice here — more useful to an application than indefinite blocking on a server that may not return. Redundancy against a sustained LServer outage is the separate matter of replicated metadata servers, a future enhancement.

Caching is configurable at this layer through pluggable cache strategies — amp for adaptive multi-policy management, lru for recency-prioritized workloads, round_robin for balanced distribution, direct for workloads that bypass the cache entirely. The cache plugin is selected per deployment based on access patterns and performance requirements; like every other layer in the stack, it is a configurable choice rather than a fixed default.

Operations that span beyond a single read or write — orchestration with external systems like Hammerspace, integration with tape archive subsystems, lifecycle migrations across tiers — are introduced by LIO at this layer and handled in detail in Operations & Lifecycle. We will visit that sub-page shortly as part of the LStore coverage.

Everything underneath this layer exists to make this layer unremarkable. The leased-storage substrate, the erasure encoding, the per-file exnode, the asynchronous operation pipeline — all of that work happens beneath what the user sees. A research scientist running an analysis does not need to know that the file behind the open file descriptor is reconstructed from fragments distributed across multiple depots. A graduate student saving simulation output does not need to understand the encoding parameters their dataset is using. A scientific instrument writing telemetry does not need to manage capability tokens. The architecture exists so that none of that complexity surfaces at the layer where work actually gets done.

This is the same outcome that a conventional storage appliance delivers — VDURA, Pure FlashBlade, NetApp, Isilon, Qumulo, and the established market of POSIX-presenting systems all give their users an ordinary filesystem to work with. The difference is in what stands behind that filesystem. An appliance is a fixed product: hardware and software bound together, refreshed on the vendor's cadence, retired as a unit, replaced through significant capital investment and an operating relationship that follows the appliance's lifecycle. LStore presents the same ordinary filesystem to its users through composable software running on commodity hardware that the institution owns and operates on its own terms. The user sees no difference. The institution sees a fundamentally different operating model.

What this means in practice is that LStore can be evaluated on the merits that matter at scale — storage cost, durability, flexibility, longevity, the absence of forced refresh cycles — without forcing the institution to retrain its users or restructure its applications. The filesystem is the filesystem. The work that happens on top of it is the same work that happened before. The architectural change is invisible to the people doing that work, and the institutional change is everywhere.

Two architectural facts about LStore, neither obvious from any single layer, become significant when read together. First, every layer in the stack has multiple plugin implementations — the data service, the erasure method, the segment drivers, the cache strategy. The working configuration is assembled per deployment. There is no single canonical LStore; there is an architecture and a family of valid configurations.

Second, segments compose recursively. The exnode layer does not merely choose which segment driver implements a file's storage. It chooses how segment drivers wrap each other. A common default exnode describes a cache segment wrapping a jerase segment wrapping LUN segments — recent reads served from cache, cache misses reconstructed from erasure-coded fragments, fragments backed by capability-leased allocations on the IBP foundation. Each level in the wrapping is itself a segment, and the composition is open. New segment drivers can be added; existing ones can be recombined.

Read together, these two facts produce the architectural insurance that has kept LStore viable as storage practice has evolved. The cache strategy can change without touching the encoder. The encoder can change without touching the depot. The data service can be replaced without touching anything above it. New media classes can be introduced as new segment drivers, fitting into composition trees alongside what already exists. The system evolves in pieces rather than as a unit, and the pieces evolve at their own pace without forcing every other piece to keep up.

This is the engineering dividend of doing the harder thing at the start. An appliance is simpler to design and faster to ship; the cost is paid later, when the world changes and the appliance does not. A composable architecture is more demanding to design and slower to deliver; the dividend is paid forever, in the form of a system that does not have to be replaced when its assumptions shift. LStore was built on the second path. The layered structure described on this page is the visible expression of that choice.

Three architectural facts speak to the exabyte horizon without overreaching. The exnode metadata layer supports single-file capacities up to eight exabytes by design — the segment composition can describe hierarchies of arbitrary depth. The plugin model at every layer means new media classes, new encoding methods, new distributed substrates can be added as they become available, without forcing existing deployments to be replaced. And the Spectra Logic TFinity SLC tape library, available in the LStore solution set today, ships with 2.2 exabytes of native capacity per library at LTO-10, with the architecture able to integrate that capacity through the operational tier described in Operations & Lifecycle.

The distinction worth keeping clear is the one between architecture and deployment. LStore is in production today at multi-petabyte scale — ACCRE's deployment is what production demands at this point in the platform's life. The architecture is designed for the scale that research computing is heading into, so that today's deployment remains the same deployment when the data outgrows the petabyte regime. The institution that adopts LStore at petabyte scale does not have to revisit the architectural decision when the scale grows; it has to add capacity, not replace systems.

The remaining pages in this section unpack the architecture from different angles — the exnode in detail, the resource service and placement model, the reliability and integrity machinery, the operational and lifecycle layer where Hammerspace and tape integration live, the engagement models that describe how Unique Checksum works with clients, and the provenance of LStore as a research and engineering effort. Each page picks up where this one ends.

This reference list grows as the platform documentation deepens. Additional foundational technologies — erasure coding, the resource and metadata services, and the orchestration and tape layers — are documented across the neighboring LStore pages and will be cited here as their references are confirmed.