Available Session Recovery Protocol

Available Session Recovery Protocol CMCC

No. 58 Kunlunshan Road Suzhou 215000 China lluozy@yeah.net

CMCC

No. 58 Kunlunshan Road Suzhou 215000 China yanhaishuang_yewu@cmss.chinamobile.com

Applications Network Working Group Internet-Draft This document describes an experimental protocol named the Available Session Recovery Protocol (ASRP). The protocol is designed to optimize high-availability network cluster architectures, providing a superior high-availability solution for clusters offering stateful network services such as load balancing and Network Address Translation (NAT ). ASRP defines the procedures for session backup and recovery, as well as the message formats used during these interactions, enabling efficient and streamlined session state management. In contrast to traditional high-availability techniques that back up session state within the cluster itself, the core innovation of ASRP lies in its distributed backup of state information to the client or server side. This approach offers multiple advantages: theoretically unlimited elastic scaling capacity; support for rapid recovery from multi-point failures; reduction of resource redundancy through the elimination of centralized backup nodes; and significant simplification of cluster implementation complexity. The ASRP protocol provides a standardized method for constructing elastic service clusters, facilitating broader participation from software and hardware developers in building elastic cloud network service clusters.

Introduction Traditional high-availability network clusters based on a master-backup architecture rely on session state synchronization between the master and backup nodes. While functionally complete, this architecture faces challenges in the cloud era, such as insufficient flexibility for elastic scaling, resource redundancy, and high implementation complexity. To address these challenges, the industry has proposed the Elastic Stateful Cluster. An Elastic Stateful Cluster is a high-availability network service cluster composed of multiple cooperative nodes. The number of nodes within the cluster can be elastically scaled, enabling it to provide stateful network services such as load balancing (SLB) and Network Address Translation (NAT). To achieve elastic scaling, conventional Elastic Stateful Clusters adopt a Fast/Slow Path design philosophy, separating session management from packet forwarding. This allows the fast path node layer to achieve good elastic scaling capabilities.

Conventional Elastic Stateful Cluster

| Fast Path Node | <--------> | Server | | | | | +----------------+ | | | | +----------+ | | ... | | +----------+ | | ... | | | +----------------------+ | +--------------------------+ ]]> The slow path nodes are responsible for session creation and synchronization, while the fast path nodes are responsible for rapid packet forwarding. The drawback of this Elastic Stateful Cluster architecture is the weak elastic scaling capability of the slow path nodes. Implementing session synchronization among slow path nodes is complex. A typical implementation reference is the AWS Hyperplane NFV platform.

ASRP Elastic Stateful Cluster

| ASRP Node | <--------> | Server | | | | +----------------+ | | | +----------+ | ... | +----------+ | ... | +----------------------+ ]]> The Available Session Recovery Protocol (ASRP) proposes an innovative high-availability solution, aiming to provide a standardized method for constructing elastic service clusters. This facilitates broader participation from software and hardware developers in building elastic cloud network service clusters. Its core idea is to innovatively distribute session state information to the client or server. The lifecycle of the backup state is synchronized with the real session, eliminating the need for independent keepalive and timeout mechanisms. This design ensures the timeliness and availability of the backup information. ASRP defines corresponding session backup and recovery mechanisms. The protocol allows protocol messages to be transmitted together with the original service data packets, thereby reducing control overhead for state synchronization. In an elastic stateful cluster built on ASRP, network nodes possess atomic and mutually independent properties. There is no need for communication between nodes, nor is session synchronization required within the cluster. This fundamental design provides theoretically unlimited scaling capability and supports rapid recovery from multi-point failures.

Terminology The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in BCP 14 when, and only when, they appear in all capitals, as shown here.

Protocol Overview

Two Operational Modes For the ASRP protocol to function correctly, two prerequisites must be met. First, all network nodes within the cluster MUST run service software supporting the ASRP protocol. Second, the server or client responsible for backing up sessions MUST deploy a kernel module or an eBPF module that supports ASRP. Depending on whether this module is deployed on the server or the client, the protocol operates in one of two corresponding modes: Passive (PSV) Mode and Active (ACT) Mode.

PSV Mode In PSV mode, the network node is typically located within the same trusted network domain as the server (e.g., inside a data center). Its typical service is load balancing.

ACT Mode In ACT mode, the network node is typically located within the same trusted network domain as the client (e.g., an enterprise intranet). Its typical service is Source Network Address Translation (SNAT). The operational mode (PSV or ACT) is determined by the deployment configuration of the network node and is not carried in ASRP messages.

Two Routing Behaviors

Symmetric Routing

| node X | <----------> | Server | | | | +------------+ | | | +----------+ | ... | +----------+ +------------------+ ]]> Symmetric routing refers to the path mode where bidirectional traffic of the same session between a client and a server is always routed to the same node within the cluster.

Asymmetric Routing

| node X | -----------> | | | | | +------------+ | | | | Client | | ... | | Server | | | | +------------+ | | | | | <----------- | node Y | <----------- | | +----------+ | +------------+ | +----------+ | ... | +------------------+ ]]> Asymmetric routing refers to the scenario where bidirectional traffic of the same session may be routed (e.g., by mechanisms such as ECMP , ) to different nodes within a cluster. In cloud networking environments, asymmetric routing is a common phenomenon, which imposes higher demands on the implementation of elastic stateful clusters.

Protocol Message ASRP achieves distributed backup and recovery of session state information by exchanging specific protocol messages among the client, server, and network nodes (such as load balancers or NAT devices). In a load-balancing scenario, session state is distributed and backed up to individual servers; in an SNAT scenario, session state is distributed and backed up to individual clients. ASRP defines the following protocol messages: New Session message (NS), New Session Acknowledge message (NA), New Session Conflict message (NC), Query Session message (QS), Recover Session message (RS), Recovery no-session message (RX), Hello Session message (HS) and Push Session message (PS).

NS Message Generated by the network node, it is used to send session state information to a designated client (in ACT mode) or server (in PSV mode) for backup when creating a new session.

NA Message Generated by the server as a response to an NS message in PSV mode, acknowledging the receipt of the NS message.

NC Message Generated by the client or server when the session state carried in a received NS message conflicts with an existing local session, indicating the session conflict to the network node.

QS Message Generated by the network node, it is used to query the client or server for backup session state information when a received packet cannot match any local session and a session cannot be directly created.

RS Message Generated by the client or server holding the backup as a response to a QS message, it contains the state information required to recover the session. The network node parses the RS message and reconstructs or marks the local session, thereby achieving failure recovery.

RX Message Generated by the client or server holding the backup as a response to a QS message, indicating that the session queried by the QS message was not found. Except for the MsgType field, the RX message is identical to the corresponding QS message.

HS Message Generated by the client, it is used in ACT mode to announce to the network node its capability to support the ASRP protocol and to trigger the network node to return an NS message to complete session backup.

PS Message Generated by the server, it is used in PSV mode to push session state information to the network node. In the case of asymmetric routing, the network node utilizes the PS message to create/update sessions for fast packet forwarding.

Transmission Modes and Signature ASRP messages can be transmitted in two modes: Inline Message and Standalone Message.

Inline Message The Inline Message is transmitted with the same 5-tuples as the flow, either inserted into the original packet or carried in a separate packet that copies the original packet header. When the total packet length does not exceed the MTU, NS, HS, and PS messages SHOULD be inserted into the original packet; otherwise, they are carried in a separate packet. HS messages are sent after forwarded packets, and PS messages are sent before forwarded packets.

Standalone Message The Standalone Message is transmitted in an independent packet with its own 5-tuples. It is encapsulated using IP/UDP , with IP addresses configured to ensure mutual reachability and a fixed destination port ASRP-PORT (e.g., 51200). QS, RS, RX, NA and NC messages typically use this transmission mode.

ASRP Signature In Inline Message mode, an ASRP Signature is used to indicate that a packet contains an ASRP message. Similar to the Proxy Protocol, a 12-byte ASRP Signature is used: 0x0D, 0x0A, 0x0D, 0x0A, 0x00, 0x0D, 0x0A, 0x41, 0x53, 0x52, 0x50, 0xF1. The ASRP Signature is inserted in front of the ASRP message. In Standalone Message mode, the UDP destination port is ASRP-PORT, and the ASRP Signature is not required.

Session Creation/Recovery Scenarios This section elaborates on, through a series of typical scenarios, how the ASRP protocol achieves session backup and recovery via message interaction in the event of network node failures under different operational modes. Each scenario details the involved protocol message flows and the processing steps of each entity.

PSV-Scenario-1

| | -----2:NS----> | | | | | | <----3:NC----- | | | client | | ASRP Nodes | | server | | | | | | | | | <--5:PKT--- | | <----4:PS----- | | +----------+ +--------------+ +----------+ ]]> This scenario describes that, in PSV mode, a network node receives an explicit first packet (i.e., a packet whose characteristics explicitly indicate the start of a session) and directly creates a session flow. Common examples of explicit first packets include TCP SYN and DNS query, among others. The processing flow is as follows: Session Creation: Upon receiving a packet from the client (e.g., TCP SYN), the network node first creates a new session and then sends an NS message to the selected server. Server Response: Upon receiving the NS message, the server stores the session state information and associates it with its local session. If a conflict occurs, it sends an NC message. In the case of asymmetric routing, when sending its first response packet, the server sends a PS message to the network node. Session Recovery: Upon receiving the PS message, the network node restores the session and forwards packets accordingly. The session state information backed up by the server is released upon local session termination, without requiring any additional teardown message.

PSV-Scenario-2

| server | | | | | | | | | | | <---3:RS/RX--- | | +----------+ +--------------+ +----------+ ]]> This scenario describes the session recovery flow triggered by a server packet. The processing flow is as follows: Session Query: Upon receiving a packet from the server, the network node searches its local session table. If no matching session is found, the node SHOULD first buffer the packet for forwarding, then sends a QS message back to the server. Server Response: After receiving the QS message, the server looks up the backed-up session state information and sends an RS/RX message. Session Recovery: Upon receiving an RS message, the network node creates a new local session and forwards packets accordingly. Upon receiving an RX message, the node SHOULD discard the pending packets.

PSV-Scenario-3

| ... | | | ---1:PKT--> | | <---3:RS/RX--- | +--------+ | | | | | ... | | server | | | | | | ... | +--------+ | | client | | ASRP Nodes | ---4:PKT/NS--> | ... | | | | | | +--------+ | | | | | | | server | | | | <--7:PKT--- | | <--5:NA/NC---- | +--------+ | | | | | <----6:PS----- | ... | +----------+ +------------+ +------------+ ]]> This scenario describes the session creation/recovery flow triggered by a client packet. The processing flow is as follows: Query Local Session: Upon receiving a packet from the client, if no matching session is found, the network node first obtains a list of candidate servers (possibly multiple) for querying. Query Backup Session: The network node sends QS messages to each candidate server. Each server replies with an RS or RX message. Process Query Results: If a session is found, the network node restores the session and forwards the packet. Otherwise, for TCP packets: drop the packet. For UDP packets: create a new local session and send an NS message to the selected server. Server Creates New Session: Upon receiving an NS message, the server stores the session state information and associates it with its local session. If a conflict occurs, it sends an NC message; otherwise, if the F-NA flag is set, it immediately replies with an NA packet. Session Recovery: In an asymmetric routing environment, when sending its first response packet, the server sends a PS message to restore the session at the network node. In this scenario, obtaining the list of candidate servers is a key challenge. Two solutions are proposed: The network node employs a deterministic server selection algorithm-such as consistent hashing or history-aware consistent hashing-to quickly map incoming packets to backend servers. Enhance the client so that client packets carry backend server information, allowing the network node to directly extract the target server from the client packet.

ACT-Scenario-1

| | | | | | <----2:NS----- | | ---3:PKT--> | | | client | | ASRP Nodes | | server | | | <----5:QS----- | | <--4:PKT--- | | | | ---6:RS/RX---> | | | | +----------+ +--------------+ +----------+ ]]> This scenario describes session creation at a network node and server-initiated session restoration. The processing flow is as follows: Session Backup: When sending a packet without receiving an NS message, the client sends an HS message to the network node to request session backup (the client SHOULD rate-limit HS). Upon receiving the HS message, the node replies with an NS message. Session Lookup: For packets from a server with no matching session, the network node identifies the target client and sends it a QS message. Session Recovery: Upon receiving QS, the client sends an RS/RX message to the network node to restore the session. In step 2, identifying the target client is challenging. Two solutions are proposed: Use static mapping (e.g., map destination port to client). For SNAT, client IP addresses can be statically mapped to distinct port ranges. Enhance the server to embed client information in its packets, allowing the network node to extract the client address directly.

ACT-Scenario-2

| | | | | | | | | | | client | <----2:QS----- | ASRP Nodes | ---4:PKT--> | server | | | | | | | | | ---3:RS/RX---> | | | | +----------+ +--------------+ +----------+ ]]> This scenario describes the client-packet-triggered session recovery. The processing flow is as follows: Session Query: Upon receiving a packet from a client with no local session and without an HS message, the network node sends a QS message to the client. Session Recovery: The client responds with an RS/RX message; the RS message enables session restoration.

Protocol Details

Message Format An ASRP message consists of nine fields in total. The message header has a fixed length and is composed of the first six fields. The message body comprises the remaining three fields: the fixed-length Protocol field, and the variable-length Session-Tuple and Session-Data fields. The fields are defined as follows:

The values and semantics of the six header fields (Version through Reserved) are specified in the following table.

ST4(length 12) Format:

ST6 sequence) ST64(length 48) Format: Mixed ST pair(ST6->ST4 sequence) ]]> For messages carrying two Session-Tuples (NS, NA, NC, RS, and PS), the first Session-Tuple (ST1) MUST be the client-side tuple, and the second Session-Tuple (ST2) MUST be the server-side tuple. For messages carrying a single Session-Tuple (QS and RX), the Session-Tuple MUST be the 5-tuple of the triggering packet.

NS Message Format The NS message contains a Node-Local-IP field and two Session-Tuples. The Node-Local-IP field is located at the end of the NS message and carries the local IP address of the network node that sends the NS message. Its length is 16 octets when the F-LIP6 flag is set, and 4 octets otherwise. The client or server uses this address as the destination IP to send the NA message back to the network node. NS Message Format:

The two Session-Tuples represent the network node's connections with the client and the server, respectively.

NA Message Format The NA message acknowledges the receipt of an NS message that carries the F-NA flag. The NA message has the same structure as the NS message, except that the Node-Local-IP field is not present.

NC Message Format The structure of the message is the same as that of the NA message. In the NC message, the client-side Session-Tuple remains unchanged, while the server-side Session-Tuple may be modified to reflect the conflicting session. The F-CC flag indicates whether the conflict is on the client side or the server side.

QS Message Format QS Message Format:

RS Message Format RS Message Format: The structure of the message is the same as that of the NA message.

RX Message Format RX Message Format: The structure of the message is the same as that of the QS message.

HS Message Format HS Message Format:

PS Message Format PS Message Format: The structure of the message is the same as that of the NA message.

ASRP packet Format A packet that carries a single ASRP message is referred to as an ASRP packet. Based on the two transmission modes of ASRP messages, the format of ASRP packets can also be classified into two types.

Inline-ASRP packet In Inline Message mode, the ASRP packet shares the 5-tuples with the flow. The ASRP message is either inserted into the original packet:

or carried in a separate packet that copies the original packet header, without the original packet data:

Standalone-ASRP packet In Standalone Message mode, the ASRP packet uses its own 5-tuples and is encapsulated using IP/UDP. The packet format is as follows:

Message Processing Standalone-ASRP packets are identified by the UDP destination port, while Inline-ASRP packets are identified by the ASRP Signature. Once an ASRP packet is identified, the ASRP messages within it are then parsed and processed.

NS Message Processing When a client or server receives an NS message, it MUST first check for session conflicts on the client side and then on the server side. A client-side conflict occurs when the client-side tuple matches an existing backed-up session on either side. If a conflict is found, it MUST clear the F-NA flag and send an NC message, with the F-CC flag set to indicate a client-side conflict. If no conflict is found, it MUST back up the session state information and hand the embedded packet (if present) over to the system. If the F-NA flag is set, it MUST send an NA message. The F-NA flag SHOULD be set when the session is created from a non-explicit first packet; explicit first packets (e.g., TCP SYN or DNS query) do not require it. If an NS message is lost, in PSV mode a TCP SYN retransmission or subsequent packets regenerate NS messages until an NA message is received; in ACT mode, subsequent packets generate HS messages, prompting the network node to retransmit the NS message. NS messages may be generated in both PSV and ACT modes. The handling procedures are described in , , and .

NA Message Processing The destination IP of the NA packet MUST be set to the Node-Local-IP field carried in the received NS message, and the destination port MUST be set to ASRP-PORT. Upon receiving an NA message, the network node considers that the corresponding NS message has been processed by the responder, and MUST NOT send any further NS messages for this session.

NC Message Processing The destination IP of the NC packet MUST be set to the Node-Local-IP field carried in the received NS message, and the destination port MUST be set to ASRP-PORT. Upon receiving an NC message, the network node considers that the session conflicts with an existing local session on the responder. If the F-CC flag is set, the conflict is on the client side; otherwise, it is on the server side. The network node SHOULD handle the conflict according to its local policy.

QS Message Processing The source IP of the QS packet MUST be set to the network node's local IP (obtainable from configuration), and the destination IP MUST be set to the client's or server's IP (obtainable from the forwarded packet as described in and , or derived via algorithmic mapping to the client or server as described in and ). The source port MUST be randomly generated, and the destination port MUST be set to ASRP-PORT. The backup holder MUST index the backed-up session by both the client-side and the server-side 5-tuples. When a client or server receives a QS packet, it MUST look up the backed-up session whose client-side or server-side 5-tuple matches the Session-Tuple in the QS message. If a match is found, it MUST return an RS message; otherwise, it MUST return an RX message. If a QS message is lost, subsequent packets will trigger the generation of new QS packets, continuing the attempt to recover the session. QS messages may be generated in both PSV and ACT modes. The handling procedures are described in , , and .

RS Message Processing The RS packet MUST reuse the protocol header of the QS packet, with the source and destination IP addresses swapped. When a network node receives an RS packet, it MUST recover the session and forward packets according to the session. If an RS message is lost, subsequent QS messages will continue the attempt to recover the session, thereby triggering retransmission of the RS message. RS messages may be generated in both PSV and ACT modes. The handling procedures are described in , , , , and .

RX Message Processing Upon receiving an RX message, the network node MUST follow the processing procedures described in the respective scenarios under PSV/ACT mode.

HS Message Processing When a network node receives an HS message, it MUST create a session, forward packets according to the session, and return an NS message to the client. HS messages are only generated in ACT mode. The handling procedure is described in .

PS Message Processing The source IP, destination IP, and source port of the PS packet MUST be copied from the packet sent by the server. When a network node receives a PS message, it MUST recover the session; it extracts the forwarded packet (if present) and forwards it according to the session. PS messages are only generated in PSV mode. The handling procedure is described in .

Security Considerations

Message Forgery Attacks The security design of the ASRP protocol is based on its typical deployment model. Deployment Boundaries and Access Control: ASRP recommends deploying network nodes and the clients or servers that back up sessions within the same trusted internal network domain. In this model, all ASRP protocol packets communicate within an internal address space. By implementing appropriate network segmentation (e.g., using firewall policies or security groups) and strictly checking the source addresses of packets, forged ASRP packets originating from untrusted external networks can be effectively prevented from reaching the target nodes. Session Legitimacy Verification: When processing ASRP packets that may establish new sessions (e.g., HS or RS packets), network nodes SHOULD perform basic validation according to the specific policies of the upper-layer application or service. For instance, in a load-balancing scenario, a node SHOULD verify whether the session points to a known and healthy server. In a NAT scenario, it SHOULD verify whether the address translation complies with predefined rules. This prevents the establishment of illegal sessions at the application layer. Internal Threat Assessment: Even if an attacker is located within the trusted network and can forge ASRP packets, the scope of impact is inherently limited. The attacker can only forge sessions where they themselves are the endpoint (e.g., masquerading as a client to request recovery of a non-existent connection). Such forged sessions are indistinguishable in form from sessions established through normal access. They do not directly jeopardize the security of other users or nodes, nor can they elevate the attacker's privileges or grant access to unauthorized resources.

QS Flood Attacks When a network node loses a session, it may generate a large volume of QS packets. If maliciously exploited or due to a malfunction, this could lead to a flood attack . To mitigate such risks, implementers SHOULD consider the following protective measures: Rate Limiting and Traffic Shaping: Each network node SHOULD implement monitoring and limiting of the rate at which QS packets are sent. A reasonable threshold (e.g., the number of QS packets allowed per second) SHOULD be set. When the rate exceeds this threshold, the node SHOULD adopt a packet drop policy, for example, discarding newly arriving forwarded packets that trigger queries. The parameters for rate limiting SHOULD be configurable to adapt to deployment environments of different scales.

IANA Considerations This document defines an application-layer protocol (ASRP). The protocol message types and internal identifiers are defined by this specification itself and constitute internal implementation details of the protocol. Therefore, there is no need to request registration of a separate protocol number or code point from IANA. However, for the implementation of this protocol, a UDP destination port requires allocation:

UDP Destination Port QS/RS/RX/NA/NC messages are encapsulated within UDP datagrams for transmission. A fixed UDP destination port number is required so that the receiving end can identify and process such encapsulated packets. Service Name: asrp Port Number: 51200 (proposed value for current experimentation) Transport Protocol: udp Description: Used for receiving UDP-encapsulated ASRP protocol messages. For experimental implementations and interoperability testing prior to IANA assignment, UDP port 51200 MAY be used as a temporary default. This port falls within the dynamic/private port range (49152-65535) reserved for local or temporary use and documentation examples . IANA is requested to assign a permanent port number in the "User Ports" range (1024-49151) for the "asrp" service in the "Service Name and Transport Protocol Port Number Registry", with a reference to this document.

&RFC0768; &RFC2119; &RFC8174; &RFC1034; &RFC1035; &RFC2991; &RFC2992; &RFC4787; &RFC4987; &RFC6335; &RFC9293;

Acknowledgments The authors would like to thank all individuals who have provided valuable feedback and contributions during the development of this document.