Reliability & integrity.

LStore organizes its reliability machinery as a layered stack. The layers compose, but each addresses one specific category of failure and operates on its own timescale and against its own data structures. The table below names them. The rest of the page walks through each in turn.

Layer	Addresses
Encoding	Loss of one or more depots. Survives drive failure, host failure, network partition, and depot decommission by reconstructing missing fragments from surviving ones. Implemented as the jerasure segment driver, using Cauchy Reed-Solomon coding.
Block checksums	Silent bit errors — bits flipped during write, in storage, or on read without the drive reporting an error. Implemented inside IBP at the allocation level, with verification on every read and write.
Error tracking	Cumulative integrity history at the file level. Soft and hard error counters surface on each exnode, triggering deeper inspection of files whose error history suggests degradation in their underlying allocations.
Allocation lifecycle	Loss of allocations through expiration, depot refusal, or host departure. Allocations are leased rather than permanent; warming extends them, and refused leases trigger data migration to surviving resources.
Operations practice	Hardware-level failures that no software mechanism can handle: failing drives, dying SAS controllers, OS drive death. Operational tooling for SMART monitoring, drive replacement, RID sequestering, and rebuild from erasure coding.

Each layer has a different domain. The encoding layer addresses loss of whole depots and is invisible to the application. The checksum layer addresses corruption of individual blocks and triggers in the I/O path. The error-tracking layer addresses cumulative degradation over time and surfaces through file attributes. The lifecycle layer addresses the inherent transience of IBP allocations and runs as a background process. The operations layer addresses what software cannot — the physical reality of a multi-thousand-drive production fleet that needs hands and procedures.

The architectural property worth surfacing at the outset is that no single layer is doing the work alone. A bit flip caught by a block checksum may trigger a soft error counter on the exnode, which over time may trigger an inspection that finds the underlying allocation is on a degrading drive, which the depot operations layer responds to by sequestering the drive, recovering the data through erasure reconstruction, and replacing the hardware. The mechanisms compose, and the composition is what makes the integrity envelope hold.

Replication is the simpler strategy. A data chunk is written to multiple depots, each holding the same bytes. Surviving a depot loss requires only that another depot with the same data remains reachable. The cost is storage: three full copies of every byte costs three times the storage of the underlying data. The benefit is operational simplicity — reconstruction is a copy operation, with no encoding or decoding involved.

Erasure coding takes a different approach. A data chunk is divided into k data fragments, then encoded to produce m additional parity fragments. All k + m fragments are distributed across distinct depots. The original data can be reconstructed from any k of the k + m fragments, which means up to m depots can be unavailable simultaneously without data loss. The storage cost is (k + m) / k — significantly lower than replication for comparable fault tolerance, at the cost of additional CPU work to encode on write and decode on read.

Strategy	Storage overhead	Fault tolerance
Replication 3×	3.0× data size	Survives loss of any 2 depots
Erasure k=6 m=3	1.5× data size	Survives loss of any 3 depots

For research-computing workloads with multi-petabyte data sets and decade-scale retention, the storage-cost difference compounds significantly. Erasure coding at k=6 m=3 tolerates the loss of any three fragments while 3× replication tolerates two, and it does so at a quarter of the storage cost — fifty percent parity overhead against replication’s 200 percent. The trade is CPU and reconstruction time, which are both manageable at the scale LStore is built for and which Reed-Solomon encoding has been engineered to handle efficiently for decades.

The encoding strategy is recorded in the exnode — specifically in the jerasure segment driver’s parameters — rather than being a system-wide setting. Different files in the same deployment can use different strategies. A dataset of working data being actively reconstructed can use 3× replication for the lowest reconstruction latency; a dataset of long-tail archival data can use erasure coding for the lowest storage cost; both run on the same LStore platform without partitioning.

The reconstruction model is what makes erasure coding viable at petabyte scale. Traditional RAID arrays reconstruct a failed drive onto a single replacement drive — the entire surviving stripe is read, the missing data is computed, and the result is written to one new disk, with rebuild time bounded by the write speed of that single drive. For multi-terabyte capacities this means hours or days during which the array is more exposed to a second failure that could push it past its redundancy. LStore inverts this. Because a file’s nine fragments are scattered across the depot fleet, reconstruction is inherently parallel: surviving fragments are read from many drives simultaneously, the missing fragments are computed, and the reconstructed allocations can be placed on free space anywhere in the fleet rather than waiting on the rebuild capacity of a single replacement drive. Rebuild time scales with network bandwidth and the count of drives participating, not with single-drive write speed, and the window of exposure to additional failures shrinks accordingly. This is the architectural property that lets erasure coding scale at petabyte capacity: more drives in the fleet means faster reconstruction and smaller exposure window per file.

The encoding is performed by the jerasure segment driver, which uses the bundled Jerasure library — the well-established Cauchy Reed-Solomon implementation maintained alongside LStore in the ACCRE GitHub organization. Jerasure is the standard reference implementation for this encoding family in the distributed-storage community; the choice to use it is the choice to use proven algorithms rather than reinvented ones, and it is the same library that other research-computing storage platforms rely on for similar reasons.

The encoding parameters are visible in the exnode for any file. Querying a file with lio_getattr returns its segment composition, including the jerasure segment’s parameters — the encoding method, the data and parity fragment counts, the chunk size, and the LUN segment underneath that maps the fragments to specific depot allocations. An abbreviated example:

[segment]
type=jerasure
method=cauchy_good
n_data_devs=6
n_parity_devs=3
chunk_size=16384

[segment]
type=lun
n_devices=9
n_shift=0

The jerasure segment sits above the LUN segment in the standard exnode composition. The jerasure segment defines the encoding; the LUN segment maps the resulting nine fragments to nine depot allocations, one per fragment, with the placement governed by the Resource Service query attached to the LUN segment. When the file is read, the LUN segment fetches the fragments from their depots, the jerasure segment decodes them back into the original bytes, and the application sees the reconstructed data. If one to three of the depots are unreachable, the decode proceeds from the remaining six or more, and the application is not aware of the failure.

Changing the encoding parameters for new files is done through the exnode tooling described on the exnode page — capture the current exnode with lio_getattr, edit the jerasure segment parameters in the captured text, and apply the modified exnode to a new directory with lio_setattr. Files created in the modified directory inherit the new encoding; existing files retain their original encoding unless explicitly migrated. The standard pattern for migrating existing data to a new encoding is to write the new exnode to an empty directory, copy data into that directory, and replace the original.

The mechanism is block-level checksum interleaving. When an allocation is created on a depot with block checksums enabled, the depot reserves a small amount of storage alongside the data for checksum metadata, computed at a granularity that matches the depot’s I/O block size. Every write operation computes a checksum for each block as the data is written and stores it alongside the block. Every read operation recomputes the checksum from the data being read and compares it against the stored value. A mismatch is detected at the point of the read, before the bad data leaves the depot.

The trade for this defense is processing rather than storage. Calculating block checksums on every read and write requires CPU on the depot. Storage depots have substantial excess CPU capacity, since their workload is primarily I/O rather than computation, so the checksum work fits well within the depot’s envelope without creating contention. The alternative — forcing every write to synchronously flush to disk before returning success — would impose far more severe performance penalties without addressing the silent-bit-error problem at all.

When a checksum mismatch is detected, the depot reports the error to the caller rather than returning corrupt data. From the LUN segment driver’s perspective, an allocation that returns a checksum error looks like an allocation that returned bad data — and the response is the same as the response to a missing or unreachable allocation: reconstruct the affected data from the surviving fragments through the encoding layer, and record the error against the file’s integrity counters so that the cumulative degradation can be tracked. The detection happens at the lowest layer; the recovery happens at a higher one; the file system above sees a successful read.

Block-level checksums are enabled at allocation creation time. In the ACCRE deployment, they are turned on by default for production RIDs — set_rid_option.py <Rid> rid enable_chksum 1 is part of the standard RID provisioning procedure documented in the depot operations toolkit.

A second class of silent error sits not in storage but in transit. Bits can flip on the wire — through transient signal-integrity issues, marginal cabling, or memory errors in intermediate network hardware — and the standard transport-layer checksums are not always sufficient to catch them. TCP’s checksum is a deliberately simple algorithm designed to detect single-bit flips; multi-bit flips can cancel each other out and pass through undetected, and the other network-layer checks are similarly simplistic. The at-rest block checksum doesn’t see these errors either, because the corruption occurred between the client and the depot rather than in the depot’s storage.

IBP defends against the in-transit class with transfer-level checksum verification, enabled through a flag in the IBP server’s configuration (similar to enabling connection encryption). When the flag is on, every block sent or received over an IBP connection carries a checksum computed at the sender and verified at the receiver. On mismatch, the IBP layer surfaces the error to the layers above it — LIO, and the application beyond — rather than committing the bad bytes; what to do with the error (retry, fail, escalate) is the higher layer’s decision. The transfer-level checksum and the at-rest block checksum compose: one guards the wire, the other guards storage, and together they cover the two distinct populations of bits at which silent corruption can occur.

Two exnode attributes carry this signal: system.soft_errors and system.hard_errors. Both are updated by the segment drivers when errors are detected during I/O. A soft error indicates a problem that was corrected at a lower layer — a checksum mismatch resolved by reading a redundant fragment, a transient depot unreachability that resolved on retry, a network glitch that did not produce data loss. A hard error indicates a problem that could not be resolved at the layer where it was detected — an unrecoverable corruption, a permanently missing fragment, an allocation that cannot be reconstructed from the surviving siblings.

The value of these counters is not the individual increment but the cumulative history. A file whose system.soft_errors has incremented occasionally over months is normal for a deployment of any meaningful size — storage hardware produces occasional errors, the recovery layers handle them, and the file remains intact. A file whose system.soft_errors has incremented hundreds of times in a week is signaling that its underlying allocations are on hardware that is degrading. A file whose system.hard_errors is non-zero is signaling that the recovery layers have already failed for it once and the file needs immediate attention.

A third exnode attribute, system.write_errors, complements the soft and hard counters with a different kind of signal. Where soft and hard errors are recorded by the segment drivers during I/O, system.write_errors is set whenever a write operation fails for any reason — an unreachable depot, a refused allocation, a checksum mismatch on the write path. The signal is sticky: once set, it persists until a successful repair on the affected file clears it. The daily allocation warmer (lio_warm, described in §06 below) picks these up during its scan and reports them to the operations team, which means files that experienced a write failure to one or more of their allocations are detected without waiting for the next read attempt against the affected fragments.

The counters are surfaced through the standard exnode attribute mechanism, which means they are visible to the same operational tooling that handles other file attributes. Audit processes that periodically scan the namespace can use the counters to prioritize their work — inspecting files with elevated soft errors before files with clean histories, surfacing files with hard errors to operator attention, triggering reconstruction of files whose error pattern indicates degradation in their underlying allocations. The counters are not the failure; they are the signal that allows the system to respond to failure before it becomes loss.

The architectural point worth surfacing is that integrity tracking in LStore is a property of the file rather than a property of the storage device. A drive that produces errors in one file may not produce errors in another, because the allocations on the drive serve different files. A file that loses a fragment to a dead drive does not lose data — it loses one of the nine fragments needed to reconstruct it. The granularity that matters for integrity decisions is the file, and the counters live where the file lives, in the exnode.

Warming is the process of periodically contacting the depots that hold a file’s allocations and requesting that the leases be extended. The request can be granted or refused; a healthy depot with adequate space typically grants extensions, while a depot that is full, decommissioning, or under operational pressure may refuse them. A refused warming request is information — the depot is signaling that the allocation cannot remain there indefinitely, and the system needs to plan accordingly.

The response to a refused warming request is to migrate the affected data to another location. The encoding layer makes this practical: a single fragment that cannot be warmed can be reconstructed from the surviving fragments and rewritten as a new allocation on a different depot, without disrupting the file’s availability. A single allocation that cannot be warmed becomes a routine maintenance event rather than a data loss event. The mechanism is the same one that handles depot failure: surviving fragments, reconstruction through the encoding layer, new allocation, updated exnode.

The lifecycle of an allocation is therefore not “permanent or lost” but “current, warmed, refused, migrated, or replaced.” The system runs warming continuously in the background, applies refusals as triggers for migration, and maintains the file’s availability through whatever depot lifecycle events happen underneath. Files outlive their allocations, and they do so without operator intervention for the routine cases.

The architectural framing here is the same end-to-end principle that runs through the rest of LStore. The IBP layer is deliberately weak — allocations expire, depots can refuse, storage can be lost — in exchange for being globally scalable and operationally autonomous. The end-to-end layers above IBP take responsibility for making the abstraction durable. Warming is one of those layers. Erasure reconstruction is another. Error tracking is a third. The IBP layer below them is what allows them to operate at the scale they do.

The ACCRE production deployment runs more than 80 depots holding several thousand drives across nine simultaneously-deployed drive capacities ranging from 2 TB to 24 TB. Servers in the fleet are built around 36 drive slots. The hardware generations in service span approximately sixteen years of server design — a deliberate operational choice that demonstrates LStore’s indifference to hardware homogeneity. The platform does not require fleet-wide hardware refresh; it requires that the resource registry know what is present and that the placement engine respect the failure-domain attributes attached to each resource.

Each depot has one OS SSD (used both for the operating system and for caching depot metadata) and 36 data drives. Data drives carry two partitions: a 10 GB partition for metadata and a remaining-capacity partition for data. In normal operation, the metadata partition is imported to the OS SSD for performance; this is the arrangement that makes OS-SSD failure a more involved recovery procedure than data-drive failure, since the metadata cache has to be rebuilt from the data partitions before the depot can return to service.

The fleet’s heterogeneity is a design choice, not an artifact of incremental acquisition. The textbook reliability math for erasure-coded storage assumes drive failures are independent — at the level of the math, that assumption is fine; operationally it isn’t. Drives bought together typically come from the same build batch with sequential serial numbers, sharing manufacturing defects and firmware revisions that produce correlated failure modes, and dense enclosures contribute their own correlations through mechanical vibration and backplane wear. A homogeneous fleet concentrates these correlations; a heterogeneous fleet mixed across vendors, generations, and form factors — Intel and AMD CPUs, multiple HBA chipset families, drives spanning 2 TB through 24 TB across three vendors — distributes them. The fragments of any given file’s erasure code land on drives that did not come from the same factory at the same time on the same firmware, so a defect or a vibration mode that takes out one drive does not take out the others in the same stripe.

Drive failure is the most frequent operational event the depot fleet handles, and the response is categorized by failure mode. The depot operations toolkit — a Python and shell script collection maintained alongside the LStore codebase — provides the verbs that distinguish the cases: smart_failure_scanner.py detects degrading drives through SMART attributes; sequester_rid.py takes a suspect RID out of service without disrupting the running IBP server; ibp_detach_rid removes the RID from the running IBP server; lsslot identifies the physical drive bay; light_slot activates the indicator LED to direct the hands-on replacement; and after the new drive is installed, create_rid.py, mount_rid.py, and ibp_attach_rid bring the replacement back into service. Replacement drives are not required to match the failed unit — any size, make, or model can take the slot, since the heterogeneous fleet described above is the operational norm rather than the exception.

When a drive vanishes — has died and is no longer visible to the operating system — there is nothing to preserve on the dead drive itself. The replacement is formatted and mounted, attached to the running IBP server, and the data the dead drive held is reconstructed from its erasure-coded siblings through the encoding layer onto the new drive. The affected files’ exnodes are updated to reflect the new allocation locations.

When a drive is degrading but still readable, the operations team has a choice. The first option is to clone the failing drive to a similar-sized replacement with ddrescue on a separate machine and then reinsert the replacement into the original depot — faster than reconstruction, since the new drive arrives in service holding the data the old one held. The second option is to replace, format, mount, attach, and reconstruct from the erasure-coded siblings as for a vanished drive — slower, but it does not require a separate machine to perform the clone. The choice is operational, made per failure based on what the hardware is doing and what equipment is at hand.

When a drive degrades ungracefully — flooding its SAS controller with errors that affect adjacent drives on the same backplane — the diagnostic problem is harder. The failing drive can make neighboring drives appear unhealthy, and locating the culprit may require bisecting the bus by removing drives in groups until the offender is isolated. Drives in this state are rarely clonable, so the response is the replace-and-reconstruct path.

For aging hardware whose drives are failing gracefully but whose end of service life is approaching, an additional option preserves access to the data while replacement is planned on the operations team’s schedule. The rid_rw_state command marks affected RIDs read-only: they continue to serve reads against the data they hold, but new writes route to healthier resources elsewhere in the fleet through the placement engine. The read-only RIDs are reconstructed and retired in due course rather than under emergency — the aging hardware coasts through to retirement, and the data it carried is migrated as part of routine operations.

A consequence of the platform’s redundancy that is worth making explicit is how it changes the economics of drive replacement. On a conventional file system, a failed drive is an urgent event — the array is degraded, rebuild windows are risky, and the failure demands immediate hands. LStore inverts this. Because a file’s data survives the loss of multiple depots through erasure coding, the failure of a single drive — or a handful of drives across the fleet — degrades nothing the application can see. This means the operations team does not have to respond to individual drive failures one at a time. Bad drives are allowed to accumulate to the point where replacing several at once is an efficient use of a maintenance window, rather than interrupting operations for each one. The same property governs how depots themselves are managed: the oldest servers, whose hardware is marginal enough that installing fresh drives in them would be poor stewardship, are allowed to coast on whatever drives still function and are gradually depopulated rather than urgently refreshed. Depots are added when suitable hardware becomes available — including hardware repurposed from other parts of the institution — and lost occasionally to events as mundane as a power glitch. The fleet is not a fixed installation; it is a living population whose exact size at any moment reflects this continuous, unhurried turnover. That the platform tolerates all of this without application-visible disruption is not incidental to the reliability story — it is the reliability story, observed from the operations side.

This is what reliability looks like in practice on a production deployment: not just the encoding layer holding, but a depot operations team running scheduled SMART scans, responding to alerts within hours, replacing drives on a continuous cadence, and exercising the reconstruction pathways often enough that they are well-understood when they are needed under pressure. The encoding layer is what makes individual drive failures recoverable; the operations practice is what makes them routine.

The depot operations work described in the previous section is the surface where the mechanisms become practice. Every drive that gets replaced is an exercise of the reconstruction pathway. Every RID that gets sequestered is an exercise of the migration pathway. Every SMART scan that catches a degrading drive before it dies is an exercise of the early-detection pathway. Operations runs these continuously, which means that by the time a genuinely surprising failure occurs — a backplane failure, an HBA failure, a cascade of correlated drive failures on the same chassis — the recovery procedures are not theoretical.

The lifecycle work that operates above the depot fleet adds another layer of continuous practice. Allocation warming is running constantly. Audit processes scanning for elevated soft-error counters are running constantly. The Hammerspace orchestration layer described in Operations & Lifecycle is running the data placement policies that move files between tiers continuously. Each of these is an additional exercise of the underlying reliability mechanisms — another opportunity to detect a problem early, another pathway through which a degraded file gets noticed and addressed.

This is also the framing that connects reliability to backup. Constancy, the Unique Checksum-developed enterprise network backup system in production at ACCRE, operates on the same continuous-practice principle from a different angle: integrity at scale is produced through ongoing operational discipline — synthesis, reconsolidation, file-level checksum verification — rather than through periodic audit. The LStore reliability stack and the Constancy backup stack are not duplicating effort; they are addressing different aspects of the same problem, with the same operational philosophy underneath both.

For institutions sizing a research-computing storage platform on the basis of its reliability properties, the question worth asking is not which mechanisms it implements but which ones it exercises. The mechanisms in this page’s catalog appear in many storage products. The discipline of running them continuously, against a production fleet of more than 80 depots and several thousand drives, with operations procedures that have evolved over more than a decade of actual failure response — that combination is what makes LStore’s reliability claim operational rather than aspirational. Unique Checksum’s work with institutions deploying LStore includes the design of the depot operations practice that sustains the platform once it is in service, because the platform is only as reliable as the practice that operates it.