RTGWG Z. Hu
Internet-Draft Y. Zhu
Intended status: Standards Track China Telecom
Expires: 3 September 2025 X. Geng
Huawei
J. Hu
T. Pi
China Telecom
2 March 2025
Fast congestion notification for distributed RoCEv2 network based on
SRv6
draft-hu-rtgwg-rocev2-fcn-00
Abstract
AI services (e.g. distributed model training, separated storage and
model training) drive the need to transmit RMDA packets through SRv6
tunnels in WAN. RoCEv2 is the most popular open standard for
achieving RDMA and network offloads over ethernet, with its
congestion control based on the combination of PFC and ECN. The
document defines the fast congestion notification for distributed
RoCEv2 network based on SRv6 tunnels, and further extends PFC and ECN
to achieve precise flow control and end-to-end congestion control.
Status of This Memo
This Internet-Draft is submitted in full conformance with the
provisions of BCP 78 and BCP 79.
Internet-Drafts are working documents of the Internet Engineering
Task Force (IETF). Note that other groups may also distribute
working documents as Internet-Drafts. The list of current Internet-
Drafts is at https://datatracker.ietf.org/drafts/current/.
Internet-Drafts are draft documents valid for a maximum of six months
and may be updated, replaced, or obsoleted by other documents at any
time. It is inappropriate to use Internet-Drafts as reference
material or to cite them other than as "work in progress."
This Internet-Draft will expire on 3 September 2025.
Copyright Notice
Copyright (c) 2025 IETF Trust and the persons identified as the
document authors. All rights reserved.
Hu, et al. Expires 3 September 2025 [Page 1]
Internet-Draft Fast congestion notification for distrib March 2025
This document is subject to BCP 78 and the IETF Trust's Legal
Provisions Relating to IETF Documents (https://trustee.ietf.org/
license-info) in effect on the date of publication of this document.
Please review these documents carefully, as they describe your rights
and restrictions with respect to this document. Code Components
extracted from this document must include Revised BSD License text as
described in Section 4.e of the Trust Legal Provisions and are
provided without warranty as described in the Revised BSD License.
Table of Contents
1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2
2. Conventions Used in This Document . . . . . . . . . . . . . . 3
2.1. Abbreviations . . . . . . . . . . . . . . . . . . . . . . 3
2.2. Requirements Language . . . . . . . . . . . . . . . . . . 3
3. Scenarios for distributed RoCEv2 network . . . . . . . . . . 3
3.1. Scenario 1: distributed model training . . . . . . . . . 3
3.2. Scenario 2: separated storage and model training . . . . 4
4. Solution . . . . . . . . . . . . . . . . . . . . . . . . . . 4
4.1. Precise flow control . . . . . . . . . . . . . . . . . . 6
4.1.1. Illustration of Para-Type and Corresponding
Parameter . . . . . . . . . . . . . . . . . . . . . . 6
4.1.1.1. Para-Type Bit 2 . . . . . . . . . . . . . . . . . 7
4.1.1.2. Para-Type Bit 3 . . . . . . . . . . . . . . . . . 7
4.1.2. Process analysis of precise flow control . . . . . . 7
4.2. Extend ECN to WAN . . . . . . . . . . . . . . . . . . . . 8
4.2.1. Process analysis of CNP . . . . . . . . . . . . . . . 8
4.2.2. Fast CNP . . . . . . . . . . . . . . . . . . . . . . 10
5. Security Considerations . . . . . . . . . . . . . . . . . . . 10
6. IANA Considerations . . . . . . . . . . . . . . . . . . . . . 10
7. Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . 10
8. References . . . . . . . . . . . . . . . . . . . . . . . . . 10
8.1. Normative References . . . . . . . . . . . . . . . . . . 10
8.2. Informative References . . . . . . . . . . . . . . . . . 11
Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 11
1. Introduction
RDMA (Remote Direct Memory Access) enables direct access to memory
locations on remote machines, bypassing the need for CPU involvement
in data transfer processes. RDMA results in lower latency, reduced
CPU overhead, and increased network throughput, making RDMA
particularly beneficial for high-performance computing environments,
cloud infrastructure, and storage networks.
RoCEv2 is an open standard enabling RDMA over ethernet, with its
congestion control based on the combination of PFC and ECN.
Priority-based Flow Control (PFC) is a data link level flow control
Hu, et al. Expires 3 September 2025 [Page 2]
Internet-Draft Fast congestion notification for distrib March 2025
mechanism, which can selectively pause traffic according to its class
and eliminate packet loss caused by network congestion. Explicit
Congestion Notification (ECN) is an extension to network layer
protocol and transport layer protocol defined in RFC3168[RFC3168],
which enables the notification of network congestion.
AI services (e.g. distributed model training, separated storage and
model training) drive the demand for building distributed RoCEv2
networks based on WAN. Therefore, when congestion is detected in WAN
devices, it is essential to achieve fast congestion notification
based on the PFC and ECN mechanisms.
2. Conventions Used in This Document
2.1. Abbreviations
AIDC: Artificial Intelligence Data Center
CNP: Congestion Notification Packet
ECN: Explicit Congestion Notification
PFC: Priority-based Flow Control
RoCEv2: RDMA over Converged Ethernet version 2
WAN: wide area network
2.2. Requirements Language
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "NOT RECOMMENDED", "MAY", and
"OPTIONAL" in this document are to be interpreted as described in BCP
14 [RFC2119] [RFC8174] when, and only when, they appear in all
capitals, as shown here.
3. Scenarios for distributed RoCEv2 network
3.1. Scenario 1: distributed model training
The computational power growth of a single AIDC is limited by
multiple factors such as space and power consumption, making it
difficult to meet the fast-growing computational demands of large
models. Therefore, more and more enterprises are inclined to support
super-scale model training by coordinating distributed model training
across multiple AIDCs.
Hu, et al. Expires 3 September 2025 [Page 3]
Internet-Draft Fast congestion notification for distrib March 2025
During distributed model training, WAN needs to carry parameter
synchronization data between multiple AIDCs through tunnels, and this
data is transmitted using RDMA protocols such as RoCEv2.
3.2. Scenario 2: separated storage and model training
In scenarios such as computational power rental, computing clusters
and storage clusters are usually located in different DCs. To
prevent sensitive data from being stored in the compute cluster and
causing data leakage, clients request that this data be directly
transmitted to the memory of the compute cluster for model training.
During separated storage and model training, WAN needs to carry
sample data from storage clusters to compute clusters through
tunnels, and this data is transmitted using RDMA protocols such as
RoCEv2.
4. Solution
The scenarios mentioned in Chapter 3 can be summarized as the network
diagram shown in Figure 1. In these scenarios, the sender in DC
needs to transmit RoCEv2 packets to the receiver in another DC
through SRv6 tunnels in WAN, the process is as follows:
* The sender in DC sends RoCEv2 packets to WAN's ingress PE through
the leaf, spine, and gateway devices.
* At the WAN's ingress PE, the RoCEv2 packets are encapsulated
according to the SRv6 tunnel protocol.
* The WAN's P node transits the payload from ingress PE to egress PE
through SRv6 tunnels.
* At the WAN's egress PE, the payload are decapsulated to RoCEv2
packets and transmitted to the receiver in DC.
+--------------------------------------------------+
| H1...H4 H5...H8 H9...H12 H61..H64 |
| | | | | | | | | |
| ++-----++ ++-----++ ++-----++ ++----++ |
| | leaf1 | | leaf2 | | leaf3 |... |leaf16| |
| ++---+--+ +-+--+--+ +-+---+-+ +--+---+ |
| \ \ | \ / | / / |
| \ +----+----\--------/--+ | / / |
| \ | +------/---+--+--+ / / |
| \ | +--------+ | | | / / |
| \ | | +----------+--+--+-+ / |
| \ | | | | | | / |
Hu, et al. Expires 3 September 2025 [Page 4]
Internet-Draft Fast congestion notification for distrib March 2025
| \ | | | | | | / |
| +-+--+--+--+-+ +-+--+--+--+-+ |
| | Spine1 | | Spine2 | |
| +-------+----+ +---+--------+ |
| \ / |
| +-+-----------+-+ |
| DC1 | gateway | |
| +-------+-------+ |
+------------------------+-------------------------+
+------------------------+-------------------------+
| WAN +-----+----+ |
| +------+ingress PE+------+ |
| | +----------+ | |
| | | |
| +--+---+ +--+---+ |
| | R1 + + R2 | |
| +--+---+\ /+--+---+ |
| | \ / | |
| | \+---------+/ | |
| | + R2 + | |
| | /+---------+\ | |
| | / \ | |
| +--+---+/ \+--+---+ |
| | R3 + + R4 | |
| +--+---+ +--+---+ |
| | | |
| | +---------+ | |
| +-------+egress PE+------+ |
| +----+----+ |
+------------------------+-------------------------+
+------------------------+-------------------------+
| +-------+-------+ |
| | gateway | |
| DC2 +--+---------+--+ |
| / \ |
| +--------+---+ +--+---------+ |
| | Spine1 | | Spine2 | |
| +-+--+--+--+-+ +-+--+--+--+-+ |
| / | | | | | | \ |
| / | +--+----+ | | | \ |
| / | +-----\---+--+--+-+ \ |
| / +-+------------\--+ | | \ \ |
| / / | +-------\----+ | \ \ |
| / / | / \ | \ \ |
| / / | / \ | \ \ |
| ++----+-+ +-+--+--+ +-+----++ +-+---++ |
| | leaf1 | | leaf2 | | leaf3 |...|leaf16| |
| ++-----++ ++-----++ ++-----++ ++----++ |
Hu, et al. Expires 3 September 2025 [Page 5]
Internet-Draft Fast congestion notification for distrib March 2025
| | | | | | | | | |
| H1...H4 H5...H8 H9...H12 H61..H64 |
+--------------------------------------------------+
Figure 1: Network diagram
Within DC, RoCEv2 implements congestion control based on PFC and ECN.
In WAN, it is necessary to extend PFC and ECN so that fast
notifications can be achieved when congestion is detected by WAN
devices.
4.1. Precise flow control
PFC is the feature used in ethernet to prevent data loss due to
congestion. In PFC, traffic is classified into 8 priorities
according to the 802.1Q protocol, and each priority maintains its
queue. When the queue length of the router's receive port exceeds a
certain threshold, the router sends a PAUSE frame to upstream to stop
transmitting traffic. When the queue length of the router's receive
port drops below a certain threshold, the router sends a RESUME frame
to upstream to continue transmitting traffic.
As WAN devices often carry multiple services simultaneously, if PFC
is triggered due to the congestion in a specific service, it may
impact other services on the port and pose security risks.
Therefore, when carrying RoCEv2 packets through tunnel in WAN,
precise flow control needs to be implemented based on traffic
information such as inner IP header, outer IP header and priority.
This document proposes several parameters for the ARN mechanism
([I-D. draft-wh-rtgwg-adaptive-routing-
arn][I-D.wh-rtgwg-adaptive-routing-arn]), enabling fast notification
for congested flow in WAN.
4.1.1. Illustration of Para-Type and Corresponding Parameter
[I-D. draft-wh-rtgwg-adaptive-routing-
arn[I-D.wh-rtgwg-adaptive-routing-arn]] propose the ARN packet format
as follows, as well as two parameters: the Para-Type Bit 0 for flow
identifier and the Para-Type Bit 1 for path identifier.
Specifically, flow identifier is based on the five-tuple from packet
header to indicate the affected flow, path identifier is based on the
32-bit path ID to uniquely identify the affected path in the network.
Hu, et al. Expires 3 September 2025 [Page 6]
Internet-Draft Fast congestion notification for distrib March 2025
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Type |Version| Rvsd | Metric | Para-Type |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| |
| |
+ Parameters(Optional) +
| |
| |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Figure 2: ARN format
This document further defines the Para-Type Bit 2 for priority
identifier and the Para-Type Bit 3 for compression time.
4.1.1.1. Para-Type Bit 2
When bit2 of Para-Type is 1, the following parameter is concluded in
Parameters to indicate the priority of affected flows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Priority |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Priority ID: The 32-bit field is used to identify the priority of affected flows.
Figure 3: ARN format
4.1.1.2. Para-Type Bit 3
When bit3 of Para-Type is 1, the following parameter is concluded in
Parameters to indicate the compression time of affected flows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
| Time |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Time: The 32-bit field is used to uniquely identify the compression time of affected flows.
Figure 4: Compression time
4.1.2. Process analysis of precise flow control
Taking Figure 1 as an example, within DC1 and DC2, PFC is still used
to implement the flow control based on port priority.
Hu, et al. Expires 3 September 2025 [Page 7]
Internet-Draft Fast congestion notification for distrib March 2025
Within WAN, precise flow control can be used to achieve congestion
control at three levels (service level, priority level, and flow
level), effectively avoiding impact on other services. Specifically,
when the receiving port of device detects congestion, it will notify
upstream devices to pause the corresponding flow:
* When configured to stop the service in case of congestion, the
router will pass on ARN packet carrying flow identifier (identify
the corresponding service based on the five-tuple of outer IP
header) and compression time to upstream devices.
* When configured to stop specific flow in case of congestion, the
router will pass on ARN packet carrying flow identifier (identify
the corresponding flow based on the five-tuple of inner IP header)
and compression time to upstream devices.
* When configured to stop flow based on priority, the router will
pass on ARN packet carrying priority identifier and compression
time to upstream devices.
Upon receiving the ARN packet, upstream devices will halt the
corresponding flow based on the configured mode.
Between WAN and DC1 or DC2, considering the capabilities of the
gateway devices, congestion control can be achieved by PFC or precise
flow control.
4.2. Extend ECN to WAN
4.2.1. Process analysis of CNP
ECN enables a forwarding element (e.g., a router) to notify the
sender for congestion control without having to drop packets
[RFC3168[RFC3168]]. When a router detects congestion, instead of
dropping packets as a signal of congestion, it marks the packets with
an ECN codepoint in the IP header. The marked packets alert the
receiver that packet loss is imminent, and then the receiver alerts
the sender by sending Congestion Notification Packet (CNP). After
receiving the CNP, the sender knows to slow down the transmission
rate temporarily until the flow path is ready to handle a higher rate
of traffic.
Within WAN, devices should be able to notify the sender to reduce the
transmission rate when congestion is detected. [RFC6040[RFC6040]]
redefines how the explicit congestion notification (ECN) field of the
IP header should be constructed on entry to and exit from any IP-in-
IP tunnel (including the SRv6 tunnel). It defines the rules for
ingress encapsulation and egress decapsulation as follows:
Hu, et al. Expires 3 September 2025 [Page 8]
Internet-Draft Fast congestion notification for distrib March 2025
+-----------------+-------------------------------+
| Incoming Header | Departing Outer Header |
| (also equal to +---------------+---------------+
| departing Inner | Compatibility | Normal |
| Header) | Mode | Mode |
+-----------------+---------------+---------------+
| Not-ECT | Not-ECT | Not-ECT |
| ECT(0) | Not-ECT | ECT(0) |
| ECT(1) | Not-ECT | ECT(1) |
| CE | Not-ECT | CE |
+-----------------+---------------+---------------+
Figure 5: Tunnel ingress encapsulation behaviors
+---------+------------------------------------------------+
|Arriving | Arriving Outer Header |
| Inner +---------+------------+------------+------------+
| Header | Not-ECT | ECT(0) | ECT(1) | CE |
+---------+---------+------------+------------+------------+
| Not-ECT | Not-ECT |Not-ECT(!!!)|Not-ECT(!!!)| <drop>(!!!)|
| ECT(0) | ECT(0) | ECT(0) | ECT(1) | CE |
| ECT(1) | ECT(1) | ECT(1) (!) | ECT(1) | CE |
| CE | CE | CE | CE(!!!)| CE |
+---------+---------+------------+------------+------------+
Currently unused combinations are indicated by '(!!!)' or '(!)'
Figure 6: Tunnel egress decapsulation behaviors
At the ingress PE, there are two encapsulation modes: a REQUIRED
'normal mode' and a 'compatibility mode', which is for backward
compatibility with tunnel egress do not understand ECN. In normal
mode, the ingress PE constructs the outer encapsulating IP header by
copying the two-bit ECN field of the incoming IP header. In
compatibility mode, it clears the ECN field in the outer header to
the Not-ECT codepoint.
At the egress PE, to decapsulate the inner header at the tunnel
egress, The ECN field in the outgoing header is set to the codepoint
at the intersection of the appropriate arriving inner header (row)
and arriving outer header (column) in Figure 4, or the packet is
dropped where indicated.
To enable congestion control for WAN devices, the ingress PE MUST use
the normal mode to construct the outer encapsulating IP header. When
a WAN device detects congestion (e.g. R2), it sets the ECN field of
the outer IP header to CE. Then, the egress PE sets the ECN field in
the outgoing IP header to CE during decapsulation. When receiver
receives the marked packet, it sends CNP warning to the sender to
slow down the transmission rate of the corresponding flow, thereby
preventing congestion.
Hu, et al. Expires 3 September 2025 [Page 9]
Internet-Draft Fast congestion notification for distrib March 2025
4.2.2. Fast CNP
[RFC7514[RFC7514]] defines Really Explicit Congestion Notification
(RECN), also known as Fast Congestion Notification Packet (Fast CNP).
By extending the RoCEv2 CNP, Fast CNP can be sent by the intermediate
router directly to the sender, advising the sender to reduce the
transmission rate at which it sends the flow of RoCEv2 data traffic.
When transporting RoCEv2 packets through SRv6 tunnels in WAN,
intermediate devices are unable to send Fast CNP back to the ingress
PE based on the information in RoCEv2 packets. Additionally, the
ingress PE is unable to recognize the Fast CNP and therefore cannot
forward them to the sender. Therefore, this document proposes a
Congestion SID (END.CON) to address aforementioned issues.
END.CON is a 128-bit value configured on PE and disseminated to other
routers via IGP. The endpoint action associated with END.CON is to
decapsulate the packet and then look up the routing table for traffic
forwarding. The workflow of END.CON is as follows:
* When the ingress PE passes RoCEv2 packets through SRv6 tunnels,
END.CON is encapsulated as the source address of outer IP header.
* When WAN devices detect congestion, they encapsulate an outer IP
header (destination address is set to END.CON) outside the Fast
CNP and send it back to the ingress PE.
* When the ingress PE detects that the destination address of the
received packet hits END.CON, it will decapsulate the Fast CNP and
pass it to the sender in DC.
5. Security Considerations
TBD
6. IANA Considerations
TBD
7. Acknowledgments
TBD
8. References
8.1. Normative References
Hu, et al. Expires 3 September 2025 [Page 10]
Internet-Draft Fast congestion notification for distrib March 2025
[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate
Requirement Levels", BCP 14, RFC 2119,
DOI 10.17487/RFC2119, March 1997,
<https://www.rfc-editor.org/info/rfc2119>.
[RFC3688] Mealling, M., "The IETF XML Registry", BCP 81, RFC 3688,
DOI 10.17487/RFC3688, January 2004,
<https://www.rfc-editor.org/info/rfc3688>.
[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC
2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174,
May 2017, <https://www.rfc-editor.org/info/rfc8174>.
8.2. Informative References
[RFC3168] Ramakrishnan, K., Floyd, S., and D. Black, "The Addition
of Explicit Congestion Notification (ECN) to IP",
RFC 3168, DOI 10.17487/RFC3168, September 2001,
<https://www.rfc-editor.org/info/rfc3168>.
[RFC6040] Briscoe, B., "Tunnelling of Explicit Congestion
Notification", RFC 6040, DOI 10.17487/RFC6040, November
2010, <https://www.rfc-editor.org/info/rfc6040>.
[RFC7514] Luckie, M., "Really Explicit Congestion Notification
(RECN)", RFC 7514, DOI 10.17487/RFC7514, April 2015,
<https://www.rfc-editor.org/info/rfc7514>.
[I-D.wh-rtgwg-adaptive-routing-arn]
Wang, H., Huang, H., Geng, X., Xu, X., and Y. Xia,
"Adaptive Routing Notification", Work in Progress,
Internet-Draft, draft-wh-rtgwg-adaptive-routing-arn-03, 13
September 2024, <https://datatracker.ietf.org/doc/html/
draft-wh-rtgwg-adaptive-routing-arn-03>.
Authors' Addresses
Zehua Hu
China Telecom
Guangzhou
China
Email: huzh2@chinatelecom.cn
Yongqing Zhu
China Telecom
Guangzhou
China
Hu, et al. Expires 3 September 2025 [Page 11]
Internet-Draft Fast congestion notification for distrib March 2025
Email: zhuyq8@chinatelecom.cn
Xuesong Geng
Huawei
China
Email: gengxuesong@huawei.com
Jiayuan Hu
China Telecom
Guangzhou
China
Email: hujy5@chinatelecom.cn
Tanxin Pi
China Telecom
Guangzhou
China
Email: pitx1@chinatelecom.cn
Hu, et al. Expires 3 September 2025 [Page 12]