The LServer.

The separation is worth dwelling on, because it is the property that distinguishes LStore’s metadata architecture from designs where the metadata server is also a data gateway. In those designs, every byte read or written passes through the metadata server, which becomes the throughput ceiling for the entire system — add more storage and the server still bottlenecks the aggregate bandwidth. LStore takes a different path. The LServer answers the question “where does this file’s data live and am I allowed to touch it,” hands back the exnode and the capabilities, and steps out of the way. The client then talks to the depots named in the exnode directly, over IBP, for the actual movement of bytes.

The consequence is that the LServer’s load is proportional to metadata activity — file opens, directory listings, attribute reads and writes — rather than to data volume. A deployment can move petabytes through its depots while the LServer handles a comparatively modest stream of metadata operations. The data path scales horizontally as depots are added; the metadata path is a separate concern with its own scaling story, which the rest of this page describes.

This page is the dedicated treatment promised on the exnode page, which establishes that the exnode is a document the file owns and the Object Service is the service that stores and serves those documents. Here we look at the server that runs the Object Service and the three services alongside it — how it is structured, how it stores the namespace on disk, and how it scales when the metadata workload grows.

The LServer runs these four services as a single server process per cluster in current production deployments. The services share the server but address independent concerns, and the plugin model means a deployment can compose the specific implementations it needs — a local Object Service backend here, a remote forwarding Object Service there, one authentication backend for one namespace and another for a second. The sections that follow concentrate on the Object Service, since its job — storing and serving the namespace — is where the architecturally interesting decisions are.

The mechanism is a convention worth naming precisely, because it is how the namespace actually works on disk. Underneath the Object Service’s base path are two directories: one holding the namespace tree, and one holding hardlinked files (described in the next section). Within the namespace tree, every directory contains a special attribute subdirectory — named _^FA^_ (for file attributes) — that holds both the directory’s own attributes and the attributes of every file in that directory. The attributes are stored as files whose contents are the attribute data itself.

This design has a quiet elegance. The namespace itself is just a directory tree on an ordinary filesystem, which means standard filesystem tools can traverse it, back it up, and reason about it. The metadata is separated from the namespace entry into the _^FA^_ attribute directory, which keeps the namespace tree clean and lets attributes be added, read, and modified as individual files. The exnode — the document that describes where a file’s data lives across depots — is simply one of the attribute files, stored as system.exnode alongside the file’s size, ownership, and any user-defined attributes. The Object Service does not need a database for the common case; it uses the host filesystem’s own directory structure as its primary store.

All prefixes the Object Service is configured with must be absolute paths, and symbolic links are not allowed as configuration anchors — the namespace layout is deterministic and explicit by design. That determinism is what makes the more advanced capabilities, sharding and inode tracking, possible to layer on cleanly.

The file-based Object Service described above began life as a proof-of-concept implementation, deliberately designed to be easy to debug. It has aged well. The combination of solid-state storage and directory-level striping in modern deployments has let it scale further than the original framing anticipated, and in production it is one of the most stable pieces of code in LStore, with uptimes measured in years. It is, however, a single point of failure within an LStore deployment — a tradeoff that has been acceptable in practice but does not generalize indefinitely. A distributed implementation of the Object Service is in active development; it makes the LServer effectively stateless, holding all of the metadata state in a distributed database used by the new implementation as its primary store. The workhorse implementation is not going away; the distributed implementation is the planned evolution for deployments where the single-point-of-failure constraint becomes the binding one.

Files begin life stored directly in the namespace. When a file is hardlinked — given a second name that refers to the same underlying object — the Object Service migrates it. The object moves to a dedicated hardlink directory, which holds hardlinked objects in a set of numbered fan-out directories. Inside that directory the layout follows the same pattern as the main namespace, with one difference: the hardlinked object’s inode number is used as its name and as the basis for its attribute directory. The inode file in the hardlink directory is then hardlinked back to the original location in the namespace, and the original object’s attribute directory is replaced by a symlink to the one in the hardlink directory. The result is that every name for the file resolves to the same object and the same attributes, exactly as POSIX hardlink semantics require.

Symbolic links are handled as you would expect in a normal POSIX environment, supporting both absolute and relative targets. Because a symlinked object has its own inode, it also has its own attribute directory — but the attributes that describe the data, system.exnode and the exnode size, are replaced by symlinks to the target’s corresponding attributes. The symlink describes a path; the data attributes defer to the target. Again, the behavior matches what a POSIX environment expects, implemented through the same attribute-directory mechanism that stores everything else.

The point worth drawing out is that the Object Service does not invent its own semantics for these cases. Hardlinks behave like hardlinks; symlinks behave like symlinks. The implementation achieves this by using the host filesystem’s own hardlink and symlink primitives in a disciplined layout, rather than by reimplementing link semantics in a database. The metadata service is, in a real sense, a carefully-structured use of an ordinary filesystem.

Sharding is configured by declaring one or more shard prefixes. Each shard prefix anchors a fan-out directory; underneath it, directory metadata is stored in directories named by the original directory’s inode number, each containing that directory’s attributes and its objects’ attributes. When a directory’s metadata is sharded, its in-namespace attribute directory — the _^FA^_ directory described earlier — is replaced by a symlink pointing to the shard location. From the perspective of namespace traversal nothing changes; the symlink redirects transparently to wherever the metadata actually lives.

The capability is designed for operational evolution rather than only for initial sizing. Shards can be added after the namespace has already been created, and existing metadata can be rebalanced across the expanded set of shards — the kind of capacity expansion that a growing deployment needs to perform without rebuilding from scratch. Adding shards requires a restart of the server process, but not a migration of the namespace; the existing structure is extended, not replaced.

This is the metadata-tier analogue of the placement flexibility that runs through the rest of LStore. Just as the data tier spreads file fragments across many depots and can absorb new depots as they are added, the metadata tier can spread the namespace across many devices and absorb new ones as the workload grows. The architecture avoids the single-device capacity ceiling at both tiers, through mechanisms appropriate to each. Sharding addresses capacity and I/O throughput; it does not address availability. Each shard remains a single point of failure for the files whose attributes live on it — the same constraint that the workhorse Object Service has as a whole, and the constraint that the distributed Object Service is being developed to remove.

When inode tracking is enabled, the Object Service maintains a mapping from inode number to current path and location, exposed through virtual attributes that resolve an inode to its object. The lookup is backed by two databases rather than one, and the split is a deliberate performance choice. One database tracks all objects — files, symlinks, and directories. The second tracks only directories, and is therefore much smaller and far more likely to remain resident in cache. An inode lookup begins in the full database to locate the object, then switches to the directory database to reconstruct the path by walking up the directory tree. Because the directory database is small enough to stay cached, the path-reconstruction step — which would otherwise be the expensive part of resolving an inode to a path — is fast.

Concretely, the Object Service exposes inode tracking through two extension virtual attributes. The first, system.inode, manages the inode itself: it ensures the inode number is unique within the namespace and updates the internal databases automatically whenever a file is created or destroyed. The inode number is what callers see when they query the attribute on a file.

$ getfattr -n system.inode /lfs/example.accre.vu/data/project/source.c
# file: lfs/example.accre.vu/data/project/source.c
system.inode="<inode-number>"

The second, os.inode, handles lookups. It is a prefix virtual attribute — meaning the Object Service recognizes a family of attribute names that share the os.inode prefix — and the suffix selects the form of answer the caller wants. A bare query for os.inode returns the same value as system.inode. A query for os.inode.lookup returns the structured path-and-inode chain that NFS file-handle resolution needs: the path, the inode of each component along the way, and a hex file-type code marking which components are directories and which is the file. A query for os.inode.lookup.binary returns the same information in a compact binary form, which is what the LFS FUSE client uses internally for NFS lookups so the wire format is efficient.

$ getfattr -n os.inode.lookup /lfs/example.accre.vu/data/project/source.c
os.inode.lookup=ENTRIES: 4
PATH: /data/project/source.c
INODE: 1/<dir>/<dir>/<file>
FTYPE(hex): 2/2/2/1

The inode databases can be sharded the same way directory metadata can, with the shard chosen by a simple modulo of the inode number across the number of shards. A deployment that has sharded its namespace for capacity can shard its inode tracking the same way, keeping the inode-resolution workload distributed alongside the metadata it describes.

This inode-tracking machinery is what underpins the NFS export described on the Operations & Lifecycle page. When LStore presents itself to Hammerspace as an NFS-exported path, the persistent file handles that NFS requires are provided by this inode tracking — the tier orchestration depends on a metadata capability several layers down. The same mechanism supports a Time Cache for clients that access many files repeatedly, reducing the lookup load a remote Object Service would otherwise carry. The point is the layering: a high-level integration with an orchestration platform rests on a low-level metadata feature, and the feature was built to support exactly that kind of integration.

The mechanism is the message-queue model introduced in Section 02 and discussed in more depth, as part of the LIO operation pipeline, on the architecture page. The message-queue transport absorbs transient network issues between a client and the LServer — a brief network interruption does not, by itself, fail an operation. It is not, however, a buffer against the LServer being down: LStore is deliberately designed to fail hard and fail fast. Every operation carries a timeout, and if no response arrives within it, the operation fails and the calling routine is responsible for handling the failure. This is a design choice, not a gap — fast, explicit failure is more useful to an application than indefinite blocking on a server that may not return. Redundancy against a sustained LServer outage is addressed by replicated metadata servers, a future Object Service driver enhancement.

For sustained-outage redundancy — protection against an LServer being down long enough that queued operations are not a sufficient answer — the path forward is replicated metadata servers, a future Object Service driver enhancement. The plugin model is what makes this a tractable addition rather than a redesign: because the Object Service is plugin-driven, a replicating implementation can be introduced as a new driver without disturbing the services above it or the clients that talk to them. The architecture was built to accommodate this kind of evolution; the remote-forwarding Object Service drivers already in the codebase — which let one Object Service forward operations to another — are evidence that distributed metadata topologies are within the model’s reach.

The honest present-state summary is this: the LServer is a single server per cluster today, made resilient to transient outages by its transport, with replicated redundancy a future enhancement. For institutions evaluating LStore, the metadata tier is a well-understood part of the system with a clear engineering direction — and a metadata durability story, described in the previous section, that is already as strong as the data tier’s. The work of deploying and operating the metadata tier for a given institution is part of what Unique Checksum’s engagement covers, alongside the depot operations and lifecycle orchestration described on the neighboring pages.