This is part three of three. The other parts are the early history and the route to standardization.

ECLIPS: returning to the architectural question

This year I have returned to a successor of SPLICE named ECLIPS. The ideas began well before the later DDS developments described in the preceding instalment. In the late 1990s I was thinking about the limitations of a flat data space, the awkwardness of generic durability, the confinement of mutually distrustful subsystems, the removal of keyed objects and the joining of systems whose states had diverged. These were related doubts and possibilities, not yet a design.

Other work took precedence for much of the following decade. With some time in 2008 and 2009, I finally managed to fit the pieces together. In May 2010 I submitted the long paper, ECLIPS, a distributed data space with induced structure and capability-based protection, to ACM Transactions on Computer Systems. A much shorter version appeared later that year at the IASTED conference on Parallel and Distributed Computing and Systems in Marina del Rey.

The long paper was explicit: “ECLIPS currently is but a thought experiment.” There was no implementation, performance study or formal proof, and the paper was rejected. It argued that formalisation should precede implementation, both to expose fundamental errors early and because testing alone could not establish the security claims. An implementation would still be needed to investigate performance.

The assumed environment was deliberately bounded: a local-area network of modest size, perhaps a few hundred nodes on Gigabit Ethernet, with a known upper bound on network latency. That was unfashionable even then, amid enthusiasm for clouds and wide-area systems, but it remained a natural setting for operational systems whose requirements concerned availability, bounded resources and correct behaviour.

My father patiently listened while the ideas were still nebulous and scrutinised several drafts. ECLIPS was my design, subjected to well-informed criticism by the inventor of SPLICE. Its ancestry is therefore personal as well as technical, but it was not simply a further SPLICE implementation. I wanted structure, retention, protection and deletion to acquire meanings within the data model itself. The resulting interface was smaller than DDS’s; the obligations beneath it were considerably more ambitious.

This year, I realised that AI might allow me to construct a proof-of-concept implementation within the limited time available. More than twenty-five years after the first ideas began to take shape, and sixteen years after I managed to write them down as a thought experiment, it is quite amazing to see ECLIPS come into existence.

This then provides the second occasion for this history. Forty years after those early SPLICE presentations, I am again thinking about the question from which it all began: how should a distributed system be structured so that useful properties follow from its architecture? SPLICE answered that question with a selectively replicated shared data space. DDS retained parts of that answer, but its subsequent development concentrated increasingly on communication mechanisms and representation. ECLIPS asks what the data space itself must represent if it is to carry more of the architectural argument.

The questions become concrete as soon as ordinary operation is interrupted. A producing process disappears: where does its state survive? An object is deleted: what happens to an old reference to it? Two disconnected systems continue operating and later meet: which of their decisions and identities can be reconciled? A process claims authority: who else must agree before it acts? These are related problems of system construction, although middleware commonly presents their supporting mechanisms separately.

ECLIPS attempts to bring these questions into one model. Whether that model is sound and economical remains to be established. The ambition is for middleware to provide a basis on which properties of the resulting system can be argued. Doing so imposes stronger assumptions and harder implementation obligations. Whether those costs are justified is an engineering question; abandoning the architectural question is not an answer to it.

A graph of information flows

The familiar starting point in ECLIPS is typed, keyed data. An instance describes the state of the object selected by its key, and a newer value may replace an older state of that object in a local store. Applications query these stores. Replication is selective and asynchronous, with eventual rather than transactional consistency.

The difference is that the structure controlling replication becomes an explicit part of the application model. A process publishes at a nabla and reads from a delta. Nablas and deltas are vertices in a dynamic directed graph, alongside neutral vertices that neither publish nor store data. Each nabla and delta belongs to one data sort, principally a type combined with a key. A value published at a nabla can contribute only to deltas for that sort that are reachable through the graph.

This graph describes permitted information flows, not cables, switches or hosts. Moving a process to another computer need not mean changing the architectural boundary through which its information passes. Conversely, two processes on the same computer need not share access to the same state. The graph expresses a relationship between publications and stores that is distinct from the machinery carrying updates between them.

DDS domains and partitions divide a largely flat communication space by names. ECLIPS can reproduce such divisions: a neutral vertex can represent a partition, with publications connected towards it and subscriptions away from it. It can also represent gateways, confined subsystems and selective routes. Instead of inferring these relationships from conventions for naming topics and partitions, an application can make them part of the space’s structure.

A process asks a herald to perform operations on its behalf. The herald maintains its local stores and the internal state required by the model, exchanging updates with other heralds. There is no single place containing the shared data space. The abstraction arises from those stores, their update rules and the graph connecting them.

Nablas and deltas resemble publications and subscriptions, but identifying them with messaging endpoints too quickly loses the point—which is why I gave them such abstract names. A delta has stored state, a nabla can update the state of a keyed object, and changing the graph can change what a store is obliged to know. Delivering the next message is only one part of maintaining that model.

Where memory belongs

In the demonstration described in the first instalment, processes recovered without the track identifiers changing. Such continuity requires replacement processes to recover the state they need as well as restart their computations. If that state survives only in the histories of its original producers, recovery depends on those producers still existing and being able to provide it. If other stores retain it independently, the process may recover even though the producers have gone. The distinction between DDS’s transient-local and transient durability, and SPLICE’s context data before DDS, concerns this allocation of responsibility.

ECLIPS has no primitive durability category. Whether earlier data must be supplied to a newly attached delta follows from its position in the graph and from the state already held in other deltas. Retention becomes a property of the structure and the applications operating its stores.

For a delta w, the model defines a context: a particular set of upstream deltas for the same sort. Informally, these are the nearest relevant stores from which w can be reached. The exact construction first contracts the graph’s strongly connected components and then considers paths that pass through no other delta for that sort. Deltas within one such component are equivalent for this purpose, providing a natural unit of redundant storage without selecting one distinguished repository.

The associated invariant matters more than the terminology. Subject to propagation delay and topology changes, the store at w must contain, or must have contained, data at least as recent as the corresponding data held by every delta in its context. The qualification “or must have contained” matters: this is an obligation concerning the state supplied to a store, not a demand that every value remain in that store forever. Adding a context source may create an obligation to copy retained state, even when no producer is publishing anything new.

This makes both the location and policy of memory application-programmable. A process operating deltas can combine information from several sorts, discard a track that has ceased to be meaningful, or retain one whose publisher has disappeared. A generic service cannot decide all those questions merely by knowing the age and key of a sample. In ECLIPS, such a process participates in the retention structure through the same model as the applications using its results.

I had already explored application-controlled retention in SPLICE-lite, in the demonstration with cars on a Möbius strip. The ECLIPS paper’s appendix also shows the converse: SPLICE context data can be constructed on top of this model, using context-daemon processes and strongly connected storage around vertices representing SPLICE partitions. The older facility becomes one possible arrangement of the more general structure.

The cost is substantial. Adding or removing an edge can merge or split a strongly connected component and thereby change contexts far from the edge itself. New obligations can require data movement. Maintaining reachability in a changing directed graph is expensive in the general case. ECLIPS makes the allocation of memory explicit and derivable, but that is a semantic gain whose implementation cost still has to be paid.

What a later state means

Retaining a value is useful only if the system can determine what that value represents and which later states may replace it. DDS provides types, keys and instance lifecycle, but defines no referential semantics between topics. Disposing of an instance need not supply a final valid value. A later write can make the same key alive again.

That behaviour may be appropriate for some data. It is troublesome if another object still refers to the key and silently begins referring to a different object. Reader-local generation counts can distinguish episodes in that reader’s history, but they are not a stable identity carried by references between topics. Applications with consequential deletion must supply the missing meaning themselves.

An ECLIPS sort consequently describes more than a representation and key. It supplies predicates for validity and obsolescence, an application-defined partial order on successive states, a minimum retention time, and optionally immutability or controlled identity. The implementation extends the partial order to a total one, subject to the overriding rule for obsolescence. Publication order at a single nabla supplies a further constraint where neither of those rules decides the order. A later state can thus be defined using the meaning of the data, rather than only its arrival time or a middleware timestamp.

The application supplies the predicates and ordering. Their inclusion in the model makes them part of the contract under which replicas replace values, rather than a separate interpretation that every reader must reconstruct after receiving them.

Obsolescence is especially important. An obsolete value is a final state of the keyed object. It may supersede a current value, but no subsequent current value may supersede it. local-take may remove this final value from one local store; the herald retains enough administrative state for a prescribed interval to reject late or reordered updates. Removing something from local storage and declaring that object finished are therefore different acts.

There is a deliberate distinction between ordinary, regular sorts and controlled sorts. For a regular sort the retained administration amounts to a timed blacklist. Once all such state has expired, attempted resurrection need not produce identical results at every delta. Controlled sorts use explicitly allocated, once-only identifiers, allowing an obsolete controlled object to remain permanently excluded. In the paper’s terminology, regular sorts blacklist obsolete objects, while controlled sorts whitelist current ones.

The stronger finality therefore does not follow merely from attaching an “obsolete” flag to arbitrary keys. It depends on the identity discipline of controlled objects. This is one of the places where the design tries to make an architectural choice visible: a system that needs permanent exclusion must use the model that provides it and accept the associated administration.

Structure induced by contents

The graph is itself represented in the data space. ECLIPS has no separate administrative interface for creating sorts, vertices and endpoints. Objects in predefined sorts describe those entities. Publishing an edge object induces the corresponding graph edge; publishing descriptions of a nabla, delta or neutral vertex induces those entities. Global deletion, or eventual disappearance from all local stores, changes the operational graph.

That is the induced structure in the paper’s title. Configuration participates in the same distribution and protection rules as application data. A partitioning scheme, an access-control manager or a SPLICE-style context service can be constructed from objects and the ordinary operations, rather than introduced as another privileged facility beside them.

It is a useful reversal because the configuration of a distributed system is itself distributed state. Processes may disagree temporarily about a new connection just as they may disagree temporarily about an application object. Giving configuration a separate API does not eliminate the disagreement; it merely leaves its relationship to the data model to be explained elsewhere. ECLIPS deliberately puts that relationship into the design.

The recursion still needs a base. Heralds require a bootstrap environment and private state, and they must maintain sufficiently complete replicas of the predefined sorts. The paper sketches the necessary cooperation and identifies a race in which an application might create and remove structural objects before every herald had received them. The structure through which changes are distributed can itself be changed by those changes.

Different heralds may receive modifications at different times and in different orders. Concurrent updates can alter the contexts of the very sorts describing the graph. Whether every permitted ordering induces compatible structures is therefore a central correctness question, not an incidental matter of optimising propagation. The paper explicitly demanded a formal answer.

Two histories remain two histories

Suppose a network partition divides a system. Each fragment may continue processing sensor input and issuing commands. When connectivity returns, the fragments contain the results of two histories with a common origin, rather than delayed copies of one history. A newly detected object may have acquired different identifiers on the two sides, while two different objects may have acquired the same identifier. Decisions and physical actions made in the meantime remain part of what happened.

Transport repair (what DDS does) cannot determine what these states ought to mean together and a generic last-writer rule cannot decide whether to identify, preserve or retract an object. The same difficulty arises without a common ancestor when independently administered systems join a coalition: equal names may denote different entities, and different names equivalent ones. Merging namespaces does not reconcile their contents.

ECLIPS specifies no automatic reconciliation. Sort identifiers are hashes of complete sort descriptors, so identical descriptions acquire the same identity without negotiation. Controlled objects have unique identifiers, and each process has a mapping between its private and global identifiers. Global identifiers can consequently be changed during a merge without rewriting the identifiers held in the process’s private state.

Most importantly, the graph is intended to allow two systems to enter one namespace while keeping their information flows separate. Selected edges can then expose selected sorts to reconciliation processes. Those processes can examine both sides and publish an application-defined result. Joining the systems need not immediately cause every store to mix every object that happens to carry a matching name.

This does not solve the semantic problem on behalf of the application. It supplies somewhere to put the solution, and a way to control which existing state is exposed while that solution runs. The same graph that describes ordinary information boundaries can describe the boundaries maintained during reunion.

Restoring a cable cannot undo an actuator movement, recover an already used identifier or reverse a decision made while the other fragment was unreachable. A useful account of recovery must acknowledge that fact. ECLIPS attempts to represent the distinct histories, the permitted flows between them and the construction of a new state, leaving the domain-specific decisions with the processes that understand them.

Authority and the cost of agreement

Who may construct those flows or modify the objects they expose? In ECLIPS, having an object of a controlled sort in a local store can confer a capability. Depending on its state and the strength of the replica, a process may be permitted to forward it, use it as a name, update or label it, or operate on the entity it denotes.

A weakening edge can propagate an object without propagating its full authority. A monitoring subsystem may receive enough state to observe part of the data space without acquiring the right to alter or extend it. Confinement follows from the graph and its sort-restricted edges: a subsystem can receive precisely the nablas and deltas constituting its interface, without access to unrelated state.

The newenv operation creates a private bootstrap-like environment whose identifiers are initially available only to its caller. Revocable access can be constructed by retaining a private intermediate edge or vertex. Deleting it severs the route; it does not attempt the impossible recovery of a capability already disclosed. Authority and information flow are related within the model, but distributing authority still has consequences that cannot simply be recalled.

One graph thus determines permitted information flows, obligations to supply retained state and the propagation of capabilities. Its edges are themselves controlled objects, so changes to these relationships are subject to the same rules for authority and deletion. A system designer can examine these consequences together when deciding how to connect or confine a subsystem.

This is not a complete secure system. Authentication was delegated to the execution environment, while integrity and non-repudiation could require application-level cryptography. The intended contribution was an account of authority, propagation and confinement inside the data space.

Authority also raises a harder question than access: when must several processes agree? DDS exclusive ownership selects a winner at each reader according to writer strengths, observed liveliness and deadline status. Different readers may temporarily select different owners. That can be entirely acceptable for redundant sources; it does not establish a unique leader for an action requiring agreement across the system.

ECLIPS assigns this stronger obligation to one conspicuous operation. label atomically changes the label of every copy of a controlled object to a process identifier, to no owner, or to delete. With the appropriate label and name capabilities, ownership can be recovered after the labelled process fails. A controlled object can also sequence a set of nablas: updates written before a label transfer must become visible before the new label, while later updates must not become visible before it. This supplies a basis for token handover and agreement about the visibility of the preceding process’s work.

The paper calls label “generally very costly”. It specifies the semantic contract without the distributed mechanism needed to realise the general operation. That is a substantial omission: an atomic change still requires an account of when it can complete and at what cost.

Continued operation within a network fragment does not imply that every operation can complete there. For label, an implementation must specify which heralds constitute the system over which its promise holds, how failures can change that membership, and under what conditions the required agreement can still be obtained. Some operations may have to wait or fail. The assumed bound on network latency does not by itself settle what happens when communication is lost.

The interface and its obligations

The interface consists of newid, newenv, write, forward, read, local-take, wait and label: eight operations, or seven headings in the paper because read and local-take are treated together. Much of their contract resides in sort definitions, graph reachability, context rules, state ordering and capabilities. A graph modification expressed as an ordinary write can change distant retention obligations; a label transfer can order the visibility of publications across replicas. The small interface places substantial obligations on the implementation.

Applications must still choose appropriate state models and information boundaries. ECLIPS also omits some DDS facilities: transactions (“coherent sets”), for example, would require an extension dealing with atomic groups of updates while the graph itself may change. The useful comparison concerns what can be expressed and reasoned about using the model, and what it costs to implement. Neither the number of operations nor a comparison with DDS’s list of QoS policies answers that question.

Another direction: Zenoh

Zenoh provides a useful contemporary contrast that originates in DDS. It brings live publication and subscription, stored values and queries to application services under a common model of names and matching. Key expressions select the resources of interest, and the routing system connects the relevant publishers, subscribers and queryables. The same naming model can span constrained devices, edge systems and cloud services across heterogeneous network topologies.

This includes access to state, not just the next publication. A Zenoh get can obtain a reply from storage or from a queryable that computes an answer. The operation reaches out to providers of information. An ECLIPS read, by comparison, examines a local store whose contents are maintained according to the rules of the data space. Both can give an application access to existing information, but the obligations behind that access are different. A common name through which information can be requested provides a different contract from an obligation among replicas to supply retained state.

I see the broader contrast as a choice about how much the participants must agree upon. Zenoh makes a common naming, matching and routing model useful across a wide range of participants. ECLIPS specifies more of the relationships among retained states, object lifecycle and authority, but in a much narrower assumed environment. A small common contract can make participation easier; a stronger contract can support more deductions about the resulting system. Neither choice removes the questions outside its contract.

I was involved in Zenoh’s early development: Angelo Corsaro initiated the work in 2016, we refined the early designs together and I implemented an early version (“Zenoh-He” or just “Zhe”). Subsequent development was done by colleagues in our French office led by Angelo, while I concentrated on Cyclone DDS and eventually withdrew from the Zenoh work. I recognise the attraction of its direction as well as that of ECLIPS. The comparison is between architectural choices, not a claim that one design descended from the other or ought to replace it.

From the paper to the present experiment

The 2010 paper could offer only a qualitative performance argument for its construction of SPLICE: routes through a stable topology could be cached, and the cost of graph changes amortised over frequent writes. It also identified common cases in which label might admit cheaper implementations. These were plausible directions, without evidence of their cost in a working system.

This year’s proof-of-concept work gives the heralds and operations concrete behaviour. So far, the model has held up surprisingly well, but implementation has also forced decisions about interactions that the paper left insufficiently explicit.

One of the problems currently being worked through is deciding when an object has really disappeared. Finding no copy in the stores examined so far is insufficient: a publication may still be waiting for delivery, or retained state may still be on its way to another store. The implementation has to account for that work before drawing a system-wide conclusion from local absence. This is the kind of detail that makes a graph induced by its contents much harder to implement than to describe.

Such decisions reach into the model itself. The context invariant must survive changes to the graph while its defining objects are being distributed. Retention and obsolescence must interact with local removal and delayed updates. The resulting operational rules may preserve the design, require a different formulation or reveal an error.

Proof and measurement remain separate tasks. A convincing demonstration cannot cover every ordering of concurrent structural changes, and passing examples cannot establish confinement. Conversely, a formal argument for the model would not show that its implementation is economical.

It is also possible that only parts of ECLIPS deserve to survive: topology-derived retention, explicit finality, private-to-global identity mappings, weakening edges, or a single costly agreement operation among otherwise asynchronous primitives. Exploring the complete design gives those ideas a setting in which their interactions can be examined. It need not end in preserving every decision made in 2010.

That makes this a suitable place to pause the history. ECLIPS did not succeed SPLICE as an operational system or displace DDS. It returned to the architectural question my father had pursued and that I believe to be as relevant as it was then: what structure allows timeliness, consistency, bounded resources, redundancy, recovery and evolution to be considered together?

ECLIPS may prove too costly, incomplete or simply mistaken. That matters less than whether we are still willing to ask such questions. Middleware has become very good at moving data around, but I think we should aim higher.

Sources

A list of sources is available. Much of the early history rests on recollections and a few documents my father had kept.