Interaction-Oriented Software Engineering: Programming abstractions for autonomy and decentralization

2.1. Norms as high-level protocols

A norm is a directed social expectation that one principal (the expector) has of another (the expectee) [29]. Norms yield accountability: the expector has standing to demand an account from the expectee for violations [7]. Commitment, power, authorization, and prohibition are exemplar kinds of norms. Accommodating autonomy requires specifying the norms that govern the interactions between the principals. The norms in fact capture the social meaning of interaction, which is the level at which principals understand interactions. Whereas autonomy means a principal may decide as it pleases, norms yield a correctness standard for decisions.

Norms specify high-level protocols and support flexible interactions. E.g., the seller may violate the commitment to do Delivery or do Delivery before it receives Payment. In case of a violation, norms that capture sanctions may be triggered, e.g., that seller should refund 110% of the Payment as compensation. For repeated violations, the seller’s trading credentials may be revoked by authorities in the STS. Representing a system in software requires representation of the relevant norms and supporting reasoning about them.

The alternative to specifying norms is regimenting the interactions via mechanisms so that violation is not possible [18]. However, given that our setting is decentralized, such regimentation is generally neither possible nor is it desirable. A decision to violate a commitment is as much an expression of autonomy as a decision to discharge it. Violations may, in fact, be judged as having been beneficial and lead to improvements in the system.

2.2. Information protocols as operational protocols

Once we specify the system in terms of norms, the question of where the states of the norms are being computed arises naturally. Since the system is decentralized, there isn’t a distinguished locus of computation that computes the norms. Indeed, the norms must be computed locally by the agents in the system based on shared events they have observed separately. It is not difficult to see in fact that at least the expector and expectee of a norm must compute it, otherwise they will scarcely be interoperable.

To enable the decentralized computing of norms, we need to specify operational (low-level) interaction protocols that produce the base-level events such as Offer, Delivery, and Payment. Focusing on meaning makes clear that the operational constraints on communication may be based only on the information content of messages, specifically: (i) information causality, which captures the information dependencies that must be satisfied to emit a message; and (ii) information integrity, which captures that no two messages may contain inconsistent information. Examples of both come from the domain. E.g., in any transaction, the seller must know the item in order to do Delivery, and Payment and Delivery must refer to the same item.

Accordingly, we adopt the idea of declarative information protocols [25], a novel approach for specifying protocols in terms of the information content of messages and causality and integrity constraints on them. By enacting information protocols, agents communicate shared events, based upon whose observations, they compute the norms. Notably, information protocols depart from traditional notations and languages for specifying interactions via UML sequence diagrams [21] and choreographies [30].

Focusing on information lets us avoid irrelevant operational restrictions, the most egregious of which is message reception order. This ability is truly liberating because it opens up decision making: If Delivery conveys the information needed to send Payment (e.g., the price), then the buyer may send Payment without receiving Offer. Meaning thus obviates ordered-delivery communication services, which are not only expensive but cause delays and hide meaning in low-level control structures.

OPEN IN VIEWER

Fig. 3.

Layers in a decentralized MAS, schematically.

Figure 3 shows the resulting architecture schematically. The computation of norms is layered on top of base events produced by information protocols. Each agent’s decision making (private) takes advantage of a database of norm events. Messages between agents are transported by an ordered, unreliable (lossy) communication service, e.g., UDP.

An agent’s communications represents its public decisions – an expression of its autonomy. It complements internal decision making based on intelligent algorithms. E.g., internally, a seller’s algorithm may determine the price at which it should offer an item for sale. However, the public decision is made when it communicates the Offer. And from the social perspective, it is public decisions that count. The history of an agent’s public decisions represents where it stands in the world. Figure 4 captures the centrality of meaning: it lies in information and connects decision making with communication.

OPEN IN VIEWER

Fig. 4.

Meaning intuition.

High-level meanings are not limited to norms, even if they have been the primary focus of study in multiagent systems. Meaning encompasses any high-level event in the application domain that is relevant to decision making. Importantly, meaning is a matter of specification via protocols.

3.1. Norms

We have contributed a novel and expressive language, called Cupid, for specifying and computing norms over databases of business events [5,8]. The motivating idea is that information in any organizational database would reflect the contractual relationships that the organization is involved in. For example, information in a seller’s database would reflect the states of the commitments to buyers. Today, if the seller wanted to generate reports about commitments by their state (e.g., pending, discharged, and violated), she would have to rely on a manual translation from the relevant natural language contract to highly-complicated database queries.

Cupid instead enables declaratively specifying the norms (commitments, prohibitions, authorizations, and powers) and automatically generating the required queries in SQL. The generated queries are typically an order of magnitude longer and significantly more complex than the specifications they are generated from, thereby highlighting the practical significance and value of the language. In fact, it is not unusual to obtain a thousand lines of SQL (as formatted by MySQL Workbench) for five lines of a Cupid specification.

OPEN IN VIEWER

Listing 1.

Commitment specification in Cupid

Listing 1 gives a Cupid specification. It specifies the base events (the underlined attributes are keys and attribute t represents timestamp). PurchaseCom specifies a commitment from seller S to buyer B: Every quote occurrence creates a PurchaseCom commitment; the occurrence of both accept and pay within five days of quote detaches the commitment provided that the amt (in pay) is at least as high as the quoted price; and the occurrence of deliver within 10 days of detachment discharges the commitment. Compensation specifies a commitment from S to B that if a PurchaseCom commitment is violated, then a refund will be issued to B within two days of the violation.

Cupid enables interpreting operational (base or protocol) events in terms of normative events, as illustrated in Fig. 3. We have built implementations of Cupid over both MySQL (a relational database) and CouchDB (a document-oriented database) [11].

Cupid enables a norm-oriented view over a single database of communication events. The problem is in a decentralized application, each agent has its own local database. The challenge is to keep these databases sufficiently in sync so that agents don’t draw incompatible conclusions about the states of the norms they are involved in. For example, if a seller inferred from its database that a buyer were committed to pay, but the buyer did not infer the commitment from its own database, then the parties are misaligned, which could lead to a breakdown in interoperability. To address this challenge, we formulated the novel property of alignment [6] and gave a distributed algorithm that each agent runs locally to guarantee it.

Although originally we studied message ordering-based protocol specification approaches [1], we quickly abandoned them – as complicated as they were, they were unsuited to decentralization because they required the agents to move in lockstep. In a recent evaluation of protocol languages [3], we have in fact demonstrated that information protocols offer significant advantages over well-known protocol specification approaches based on traces and session types. E.g., there is no instrinsic reason why Payment by buyer and Delivery by seller cannot happen concurrenctly. However, the alternative approaches cannot express such concurrency.

Listing 2 gives the information protocol Purchase. Purchase is enacted between roles B and S and each enactment corresponds to a tuple of information as specified in the parameter line. This tuple is progressively computed as the roles exchange the messages specified, such as quote, accept, and so on. The adornments in and out capture causality by specifying information dependencies. For e.g., to send an accept, the bindings for ID and item must be known from prior interactions but addr can be bound to any value. Notice how information enables concurrency: once both buyer and seller have observed accept, the information dependencies for both pay and deliver are satisfied and they can happen concurrently.

OPEN IN VIEWER

Listing 2.

An information protocol

We have introduced expressiveness-related extensions (e.g., multicast, multiple agents playing a role, and dynamic role binding) [2], and novel properties such as atomicity and refinement of protocols and their verification [9,10].

Notice how the message names correspond to event names in Listing 1. The idea is that the base events correspond to message observations, based on which higher-level meanings such as commitments can be computed. Tosca [19] explores the layering of commitments on top of information protocols from the point of view of alignment. Clouseau [26] aims at a protocol language that unifies the operational and higher-level aspects in a single language.

Smart contracts are motivated as representations of real-world contracts. Their unique selling point is that they are inviolable. Despite the rhetoric, a smart contract is no substitute for a real contract [27]. A contract describes a decentralized system that is embedded in the social fabric via notions such as norms, accountability, and trust. Violability is central to the notion of a contract. A smart contract is a glorified program – a unitary machine – removed from any social relationship. In fact, the cardinal error in the philosophy of smart contracts is the notion that social structures should and can be done away with. Autonomy necessitates social structures; what we must do is represent them computationally.

We have applied Cupid toward representing contracts on blockchain [27]. Figure 5 captures our contribution. We put violable contracts specified in a Cupid-like language (as described above) on the blockchain. As events are recorded on the blockchain, a contract state evaluator enables determining the state of the contract, e.g., whether satisfied or violated. In contrast to the traditional smart contract approach, the contract machinery does not modify the blockchain by recording new events, thus fully supporting the autonomy of the participants and at the same time supporting an immutable history of contract states.

OPEN IN VIEWER

Fig. 5.

Smart contracts and violable contracts based on norms, schematically [27].

In earlier work, we focused on methodologies for specifying interactions and engineering agents. Given that we now understand better the architectural constraints that decentralization imposes and we also have languages that work within those constraints, the ideas behind those methodologies have become crisper and evolved into a programming model for decentralized applications.

Specifically, the programming model enables engineering an agent given a protocol. The heart of the model is a generic protocol adapter that exposes a programming interface for sending messages and handling received messages. The interface is specifically generated for the protocol and enables a programmer to easily plug in decision making (the business logic) in the appropriate places. An agent developed following the programming model is guaranteed to communicate correctly with other agents. In recent work, we have instantiated the programming model in Java [17], Node-RED (a popular IoT programming framework) [15], and on AWS Lambda (a popular cloud-based serverless platform) [12]. We have demonstrated that our programming model saves significant programming effort and avoid errors. Most notably, our programming model relies on nothing more than UDP for communications between agents; that is, our technique for coordinating agents works over lossy, unordered infrastructure.

In recent work, we have turned our attention to systems themes. One such theme is fault tolerance, which we investigated first in the Node-RED instantiation of the programming model. There we consider the fault of missing information, which may be due to message loss or due to the fact that an agent never sent the message. To handle such faults, agents implement application-level retransmission policies in the agents. We contrasted our approach with the traditional IoT approach, which is MQTT over TCP (in contrast to UDP, TCP offers reliable ordered delivery). We found that for different loss rates, our application-level retransmission over UDP strategy performed favorably compared to MQTT over TCP in terms of throughput measured as the numbers of interactions completed. Mandrake [14] goes further in proposing a policy language that takes advantage of information protocols and shows how to implement fault tolerance policies at the application level. Bungie [13] gives succinct annotations for augmenting information protocols with features that support fault tolerance.

In the instantiation of our programming model on AWS Lambda, the agents are highly modular, each module being a function (as in Function as a Service). Again, the modularity naturally follows from the programming model being protocol based. Preliminary experiments on AWS Lambda demonstrated that several AWS functions in fact had several concurrent instances running at the same time, thus establishing the potential scalability of our design.

4.1. Contract-based computing

A (business) contract enables engagements between autonomous principals by declaratively laying out the norms (e.g., commitments, prohibitions, authorizations, powers, and so on) that capture expectations about the behavior of the principals. Now the beauty of a contract is that it supports both autonomy and correctness. That is, a principal may do decide to act as it pleases, however, if it violates a norm, that would amount to incorrect behavior. Further, a violation would in principle be verifiable from the observation of the relevant events. Given the centrality of contracts to engagements between autonomous principals, one would imagine that the fundamental software representation of a system that supports such engagements would be in terms of business contracts. But where currently do we represent contracts when we build software?

My vision is that a declarative contract (based on norms, and therefore violable) would yield everything needed for interoperation between the principals who sign up to it. Naturally, the engineering of a particular principal’s agent would be facilitated by the implementation of programming models based on contracts. Our ongoing work on protocol-based programming models and fault tolerance and efficiency would be subsumed in the contract-based model. For example, the contract would tell an agent which messages are crucial and therefore worth retransmitting in the absence of an expected event. Contract-based computing stands in stark contrast with what passes for interoperation today – based on either APIs or message formats (which are prevalent in finance and health).

We have recently elaborated upon this vision, detailing how the aforementioned contributions already constitute a foundation for realizing it and how it represents an alternative to smart contracts [27]. We have also developed a proof-of-concept implementation of violable contracts for the R3 Corda distributed ledger. Our contributions in this line of work have attracted attention from industry. However, more needs to be done in terms of languages needed to express real business contracts and methodology, including design and verification tools, and programming and deployment tools. In particular, blockchain is just one platform for carrying out contract-based computing. A more appealing alternative is to support contract-based computing in the cloud, e.g., on serverless and container-based platforms.

4.2. Microservices

In the last few years, microservices have emerged as a dominant paradigm for building loosely-coupled distributed applications. What is interesting about microservices is that each microservice is treated as if it were autonomous – developed, deployed, and scaled independently of other microservices in the application. Microservices are here to stay, since interest in them in underpinned by advances in cloud computing, especially containerization.

Building a loosely-coupled system requires explicitly modeling the extent of the coupling between system components. In other words, it requires modeling the interactions between components. Surprisingly though, current microservice approaches are not based upon a model of interactions. This leads to overly complex microservices sitting atop overly complex and specialized communication infrastructures (services meshes, event buses, and so on). The overall result is – contrary to aims – a tightly-coupled system.

We want to apply protocols toward realizing an application as a truly loosely-coupled system of microservices. A protocol would capture explicitly the extent of the coupling between the microservices in an application. Further, since a protocol may be enacted asynchronously – without ordering guarantees from the infrastructure – no specialized communication infrastructure is required.

4.3. Programming models for distributed applications

Focusing on meaning represents a radical departure from conventional systems wisdom, which emphasizes doing ordering and fault tolerance in communication services. E.g., TCP (the de facto service for Internet applications) and message queues (the holy cow of business messaging) both guarantee reliable, FIFO-ordered communication. Underlying such communication services is a mindset that ties programming convenience with synchrony achieved by enforcing a global ordering of the events the communications represent. However, as the end-to-end argument [22] anticipated, in providing ordering guarantees, communication services limit decision making at the application level and that programming based on meaning (“semantics”) was the way out [4]. However, a meaning-based programming model has remained deeply elusive. As Internet pioneer David Clark [16] makes clears (after noting Ken Birman’s remark that TCP wasn’t good for anything): “The alternative (to TCP) is to push to the app the implementation of the desired semantics (over UDP)…, but then the (app) designer is implementing the protocol…and we don’t know how to do that.” Even Shannon appeared to be aware of the importance of communication meaning [23]. Meaning is the forgotten holy grail of systems research.

OPEN IN VIEWER

Fig. 6.

How IOSE simplifies the network stack.

Not focusing on meaning has given rise to today’s complex system stack (Fig. 6a) where every layer in the stack except the application is modeled as a protocol. In modeling applications in terms of meaning-based protocols, IOSE is unique among existing software development methodologies in harboring the potential to obviate layers of complexity in the stack and making it possible to develop Internet-native applications, that is, applications that sit directly on top of the Internet (possibly through a shim such as UDP) (Fig. 6b). The stack of Fig. 6a is threatened by many developments, including data centers and the IoT, but few have dared to imagine a stack as simple as Fig. 6b.

Current programming models, which bake in assumptions of ordering and fault tolerance, are entrenched. Programmers have gotten used to all kinds of complex middleware and the client-server mindset. Changing this mindset will require showing that protocol-based application programming models make it possible to build flexible, fault-tolerant, and loosely-coupled decentralized applications just as conveniently as existing programming models. Further, the implemented applications should offer comparable performance and opportunities for scalability. Although we have started investigating these directions, much more needs to done to realize this vision.

We are currently exploring several ideas for making protocol adapters more efficient. In our AWS Lambda implementation, scalability is limited by the fact that the protocol adapter may have only one instance (because it must process messages serially). Under reasonable assumptions about identifiers though, we can in fact run several instances of the adapter. We are currently exploring how protocols may support redundancy for fault tolerance and how redundancy trades off with efficiency. We are exploring code generation-supported programming models that make it easy for a microservice developer to specify fault-tolerance policies.

More broadly, we are examining how important ideas in distributed computing map to IOP. One of them is the notion of potential causality [20], along with the family of timestamp-based coordination approaches it has inspired. We believe that the information protocols approach, being based on a specification of causality, and therefore true causality, is a superior alternative to potential causality. In fact, work on causality in distributed systems has historically been concerned with detecting causal relationships when really the correct approach – as information protocols make clear – is to specify the relationships. We tend to forget that computing is a science of the artificial.