Resource Service & placement.

In LStore, that place is the Resource Service. Every time a file needs new allocations — on write, on rebalance, on repair — the segment that owns the allocation hands the Resource Service a query and asks for resources that match. The Resource Service consults its current view of available depots and the attributes those depots carry, evaluates the query against that view, and returns a set of resources the segment can write to. The actual I/O happens elsewhere, in the Data Service against the IBP depots; placement happens here.

The mechanism is small — a query language, a resource registry, a matching engine — but the consequences are not. Two files in the same directory can land on completely different hardware classes because their exnodes carry different placement queries. New data and aged data can be steered to different depot pools by writing the policy into the exnode at the appropriate moment. Cross-institutional partitions, project reservations, hardware-class restrictions, regional preferences — all of these are expressed through the Resource Service’s query language and live with the file rather than in an external configuration.

This is what makes LStore’s per-dataset configurability claim operational. The architecture page describes a filesystem in which encoding, integrity, and placement are properties of each file rather than appliance defaults. The exnode is the container that holds those properties. The Resource Service is the layer that interprets the placement portion of them.

The bookkeeping function runs continuously. Resources are defined in a configuration file that the Resource Service loads at startup; the service tracks each resource’s reachability and adjusts its view of available space as allocations come and go. When the configuration file changes — a new depot added, an attribute updated, a host renamed — the service detects the change and reloads. When a host goes down, the service marks its resources unavailable until the host returns. The view of available resources is always current to within a configured check interval.

The mapping function runs on demand. When a segment needs new allocations — on file creation, on rebalance, on repair — it hands the Resource Service a query along with the number of resources required and the size of the allocation. The Resource Service evaluates the query against its current view of available resources, returns a matching set, and the segment proceeds with the allocation. The matching itself is randomized: the Resource Service picks candidate resources at random and tests each against the query, keeping the ones that pass and continuing until it has enough. The approach is fast, requires no exhaustive search, and produces good fleet-wide spread without the cost of an optimization pass. The mapping function is the surface through which placement policy actually applies; the bookkeeping function is what makes the mapping function’s answers reflect the current state of the deployment.

The two functions are coupled in one important way: the bookkeeping function tracks space usage and adjusts resource availability based on it. A resource that is full will not be returned by the mapping function regardless of what the query says. A resource on a host that has gone down will not be returned until the host returns. Space usage is tracked as percent used rather than as absolute bytes, which has a useful consequence for heterogeneous fleets: a 2 TB drive and a 24 TB drive fill at the same rate, so smaller drives do not run out of space first and force a rebalance. The Resource Service holds fill rates across the fleet to within a configurable tolerance — five percent by default — so the fleet stays balanced as data is written. These are not policy decisions — they are reality checks the service applies before it consults the query at all.

The mandatory attributes are rid_key, ds_key, and host. The rid_key is the resource’s unique identifier within the deployment — a stable name that does not change even if the underlying hardware is moved or the host is renamed. It can be any character string up to 128 bytes; simple integers were the original form and remain common in practice. The ds_key is the value the Data Service uses to actually reserve space; it is typically an amalgam that includes the depot hostname, the IBP port, and the resource ID, and is passed directly to the Data Service when an allocation is requested. The host attribute names the depot the resource lives on; it is technically optional but is included on essentially every deployed resource because the most common failure unit is the host, and the placement engine needs to know which resources share a failure unit in order to stripe data across distinct hosts.

Beyond the three mandatory attributes, a resource can carry any number of user-defined attributes. Common ones include lun, used to mark a resource as belonging to a particular institutional or project pool, and site, used to mark a resource as part of a regional or replication unit. These attributes are what the placement query operates against. An institution can add whatever attributes its policy needs — hardware class, age, drive size, performance tier, capacity allocation — and the query language will work against them without any change to the system.

A resource definition looks like this:

[rid]
rid_key=1301
ds_key=my-depot1.reddnet.org:6714/1301
host=my-depot1.reddnet.org
lun=reserved-project-a

The rid_key is 1301 — the unique identifier for this resource within the deployment. The ds_key packages the host (my-depot1.reddnet.org), the IBP port (6714), and the RID (1301) into the form the Data Service uses to reserve space on the depot. The host attribute identifies the failure unit. The lun attribute marks this resource as reserved for a particular project. A placement query that wants to land on this kind of resource asks for it by lun — not by rid_key, and not by host. The job of the query is to select resources by the policy attributes attached to them, not by their identity.

This separation — identity in the mandatory attributes, policy in the optional ones — is what makes the query language usefully expressive. The query never has to name a specific depot; it describes the kind of depot that satisfies the policy, and the Resource Service returns whichever depots currently match.

The operators fall into two groups: boolean combinators and key/value matchers.

Operator	Meaning
AND	Both sub-expressions must match. Used to combine constraints — for example, a particular institutional pool and a particular hardware class.
OR	Either sub-expression may match. Used to permit multiple acceptable pools, sites, or classes.
NOT	The sub-expression must not match. Used to exclude — for example, exclude a depot pool that is being decommissioned.
KV_EXACT_MATCH	A key/value match against a resource attribute. Matches if the attribute equals the supplied value exactly. The most common matcher.
KV_PREFIX_MATCH	A key/value match where the attribute value begins with the supplied prefix. Used for hierarchical attribute schemes — a single prefix can select all values that share it.
KV_ANY	Matches if the resource has the named key at all, regardless of value. Used to require an attribute’s presence without constraining its value.

The matchers can be qualified by two modifiers that change how the matching engine selects results across the set of candidates:

Modifier	Meaning
KV_UNIQUE	The matched values across the returned resources must be unique. Prevents the placement engine from returning multiple resources that share the same value of the matched attribute — useful for striping across distinct sites, hardware pools, or failure units.
KV_PICKONE	Normally paired with KV_PREFIX_MATCH. Picks a single value matching the prefix at the first evaluation and reuses that same value for all subsequent matches in the same query. Produces the effect of “choose one pool and stay in it.”

The choice of postfix form is deliberate. Postfix expressions parse left to right without precedence rules or backtracking. Evaluation requires only a simple stack to hold values: operands are pushed onto the stack as they appear, and each operator pops the values it needs and pushes its result. The form is easy to parse and easy to compute. For a language that is evaluated on every allocation request and stored inside every exnode, those properties matter more than syntactic elegance. A placement query is meant to be read and written by operators occasionally and evaluated by the placement engine constantly; the form optimizes for the second case.

The natural way to express this is a prefix match on the lun attribute, qualified by the KV_PICKONE modifier. The prefix match identifies the candidate pool values; the modifier commits to a single pool and applies it across the entire allocation. Concretely, the query is constructed using KV_PREFIX_MATCH against lun=vu_, with the KV_PICKONE modifier attached.

When the segment driver passes this query to the Resource Service along with a request for some number of allocations, the matching engine first finds all resources whose lun attribute starts with vu_. From the candidate set, it picks one specific value — either vu_accre or vu_physics, depending on what is available and what the picking policy selects — and then returns only resources whose lun equals that specific value for the remainder of the allocation request. The file’s data lands in one pool, the second pool is excluded, and the file does not straddle the institutional partition.

Without the modifier, the prefix match alone would match resources from both pools, and the segment would happily stripe data across them — producing a file that lives partly in each pool, which is the exact arrangement the policy was meant to prevent. The KV_PICKONE modifier is what converts “these candidates are acceptable” into “commit to one of these candidates and use only that one.”

Two further observations about this example. First, the policy is expressed against attribute values, not against host identities — the query does not care which specific depots carry the vu_accre tag, only that the resources it selects all share the same lun value. Hardware can be added, retired, or replaced without changing the query, as long as the new hardware is tagged appropriately. Second, the policy lives in the exnode, not in the Resource Service’s configuration. Two files in the same directory can have different placement queries — one constrained to vu_accre, the other free to land anywhere — and the Resource Service applies each query as the file’s segments request allocations. The policy mechanism is the file’s, not the appliance’s.

The segment that holds placement queries in the standard exnode composition is the LUN segment driver — the layer that maps the file’s logical bytes to specific depot allocations. The LUN driver takes four inputs from its segment section in the exnode: the number of virtual disks to stripe across, the chunk size used for striping, an optional shift parameter that controls how data and parity blocks are laid out across stripes, and the Resource Service query that selects which resources are eligible to hold the stripes. The query parameter is how the file’s placement policy is expressed; the other three parameters describe the geometry of the striping that happens once the resources are chosen. The shift value controls the rotation pattern across stripes: shift=0 keeps parity on a fixed set of allocations, while shift=1 (or higher) intermingles data and parity, so each allocation carries a mix of both as the driver marches through successive stripes. The default chunk size in LStore is 16 KB.

Because the query lives inside the segment definition inside the exnode, and because every file has its own exnode, every file can carry its own placement policy. A directory does not have a placement query; the directory has files, and each file has an exnode with a query inside it. New files created in a directory inherit the default exnode that applies in that context, which means the default placement query for that directory propagates to its files — but it propagates by being written into each file’s own exnode at creation, not by being looked up from the directory at every allocation. Once written, the file’s placement query is the file’s.

This arrangement has practical consequences for migration. Changing the placement policy for a directory affects new files; existing files retain whatever query was written into their exnodes when they were created. Migrating existing files to a new policy is a deliberate operation — rewriting the exnode for each file, then optionally rebalancing the data to reflect the new constraints. LStore provides tooling for this kind of migration as part of the exnode operations covered in the exnode page, and the operational rhythm for it — when to run, how to throttle, how to verify — is part of the broader lifecycle work covered separately.

Dynamic reloading is supported across all three implementations. The pattern is straightforward: touch the configuration file on the host running rs_simple or rs_remote_server, and the service picks up the change on its next check cycle. If dynamic mapping is enabled, resource changes propagate to clients on the next read or write operation without restart. This is what makes adding, retiring, or reconfiguring depots a routine operational action rather than a maintenance window event.

The three-implementation pattern is the same plugin philosophy that runs through the rest of LStore. The Resource Service is a contract — given a query and a count, return matching resources — and the three implementations are different ways of satisfying that contract for different deployment shapes. Single-process tooling uses rs_simple directly. Multi-process deployments share an rs_remote_server with many clients. A site that grows from one shape to the other does not change its policy or its query language; it changes which implementation is loaded.

The expressive power that matters in practice comes from the freedom to define resource attributes. The system imposes the three mandatory ones; everything else is up to the operator. Institutions tag resources with the categories their policy cares about — institutional pool, hardware class, drive generation, project reservation, age, performance tier, regional designation — and the query language operates against those tags. A policy that says “new high-throughput data lands on this hardware class, this institutional pool, striped across distinct hosts” becomes a single query line, embedded in the exnode of every file that policy applies to.

The architectural property worth surfacing is that placement is not a global property of the filesystem. It is a per-file property, decided when the file is created, expressed in the file’s own exnode, evaluated against the live state of the resource registry at the moment new allocations are requested. A site can run multiple placement policies simultaneously without partitioning the filesystem. Two directories can carry different default policies; two files in the same directory can carry different explicit policies; data that has aged into a different lifecycle stage can be migrated to a different policy without disturbing other data. The Resource Service is the mechanism that makes all of this routine rather than exceptional.

For sites that have inherited a more rigid storage architecture — where placement is set at array-creation time, or is implicit in which mount point a file is written to — the discipline of treating placement as per-file policy expressed in a query language takes adjustment. The reward is that institutional decisions about where data should live become operational decisions about how to tag resources and how to write queries, rather than capital decisions about how to partition hardware. Unique Checksum’s work with institutions deploying LStore typically includes designing the attribute scheme and the default query set that encodes the institution’s placement policy into the file-level metadata where LStore can act on it.