InfiniBand
TM
Architecture
Specification Volume 1
Release 1.4
Copyright © 2020 by InfiniBand
SM
Trade Association.
All rights reserved.
April 7, 2020
Final
InfiniBand
TM
Architecture Release 1.4 April 7, 2020
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL
InfiniBand
SM
Trade Association Page 2 Proprietary
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
LEGAL DISCLAIMER This IBTA specification provided “AS IS” and without
any warranty of any kind, including, without limita-
tion, any express or implied warranty of non-infringe-
ment, merchantability or fitness for a particular
purpose.
In no event shall IBTA or any member of IBTA be liable
for any direct, indirect, special, exemplary, punitive,
or consequential damages, including, without limita-
tion, lost profits, even if advised of the possibility of
such damages.
Table 1 Revision History
Revision Release Date
1.0 2000-09-26 Release 1.0
1.0.a 2001-06-19 Release 1.0 augmented with errata material. Updates only
correct errors - no additional features have been added.
1.1 2002-11-06 Release 1.0.a augmented with additional features. Revised
SA and CM Class with new version.
1.2 2004-09-07 Release 1.1 augmented with additional features (added
annexes A7 through A10). Incorporated errata.
1.2.1 2004-11-30 Release 1.2 augmented with additional features (added
annex A11-A13).
1.3 2015-03-03 Release 1.3 - added Annexes and integrated XRC into the
main body. Updated Chapters include new Change Bars
1.4 2020-04-07 Release 1.4 - added new material from the LWG, MgtWG
and SWG.
Corrected errors in Chapter 9 introduced in release 1.2.1
as per comments from Pat Thaler.
Added Virtualization Annex
Added RoCE-v1 and RoCE-v2 Annexes
Implemented comments approved by MgtWG
InfiniBand
TM
Architecture Release 1.4 April 7, 2020
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL
InfiniBand
SM
Trade Association Page 3 Proprietary
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
TABLE OF CONTENTS
Chapter 1: Introduction...........................................................................................................73
1.1 Acknowledgments.....................................................................................................................73
1.2 InfiniBand Conceptual Overview...............................................................................................77
1.2.1 The Problem ........................................................................................................................... 77
1.2.2 Features.................................................................................................................................. 78
1.2.3 Benefits................................................................................................................................... 78
1.3 Scope........................................................................................................................................79
1.4 Document Organization............................................................................................................80
1.4.1 Series of Volumes................................................................................................................... 80
1.4.2 Volume 1 Organization............................................................................................................ 81
1.5 Document Conventions ............................................................................................................81
1.5.1 Byte Ordering.......................................................................................................................... 81
1.5.2 Numeric Values....................................................................................................................... 82
1.6 Disclaimer .................................................................................................................................83
Chapter 2: vGlossary ..............................................................................................................84
Chapter 3: Architectural Overview.......................................................................................101
3.1 Architecture Scope .................................................................................................................102
3.1.1 Topologies & Components.................................................................................................... 103
3.2 Communication.......................................................................................................................105
3.2.1 Queuing ................................................................................................................................ 105
3.2.2 Connections.......................................................................................................................... 109
3.3 Communications Stack ...........................................................................................................109
3.4 IBA Components......................................................................................................................111
3.4.1 Links & Repeaters..................................................................................................................111
3.4.2 Channel Adapters ..................................................................................................................111
3.4.3 Switches ............................................................................................................................... 113
3.4.4 Routers ................................................................................................................................. 114
3.4.5 Management Components.................................................................................................... 115
3.4.5.1 Subnet Managers.................................................................................................. 115
3.4.5.2 Subnet Management Agents................................................................................. 115
3.4.5.3 General Service Agents......................................................................................... 116
3.5 IBA Features ........................................................................................................................... 116
3.5.1 Queue Pairs.......................................................................................................................... 116
3.5.2 Types of Service ................................................................................................................... 117
3.5.3 Keys...................................................................................................................................... 119
InfiniBand
TM
Architecture Release 1.4 April 7, 2020
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL
InfiniBand
SM
Trade Association Page 4 Proprietary
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
3.5.4 Virtual Memory Addresses .................................................................................................... 121
3.5.5 Protection Domains............................................................................................................... 121
3.5.6 Partitions............................................................................................................................... 121
3.5.7 Virtual Lanes......................................................................................................................... 122
3.5.8 Quality of Service.................................................................................................................. 123
3.5.8.1 Service Level......................................................................................................... 124
3.5.8.2 SL to VL mapping .................................................................................................. 124
3.5.8.3 Partitions ...............................................................................................................124
3.5.9 Injection Rate Control ........................................................................................................... 124
3.5.10 Addressing............................................................................................................................ 125
3.5.11 Multicast................................................................................................................................ 127
3.5.11.1 Multicast Example ................................................................................................. 128
3.5.11.2 Group Management .............................................................................................. 129
3.5.11.3 Multicast Prune...................................................................................................... 132
3.5.12 Verbs..................................................................................................................................... 132
3.6 Channel & Memory Semantics ...............................................................................................132
3.6.1 Communication Interface...................................................................................................... 133
3.6.2 IBA Transport Services ......................................................................................................... 135
3.7 IBA Layered Architecture ........................................................................................................139
3.7.1 Physical Layer....................................................................................................................... 140
3.7.2 Link Layer ............................................................................................................................. 141
3.7.3 Network Layer....................................................................................................................... 142
3.7.4 Transport Layer..................................................................................................................... 143
3.7.5 Upper Layer Protocols .......................................................................................................... 145
3.7.5.1 Subnet Management............................................................................................. 145
3.7.5.2 General Services................................................................................................... 146
3.8 IBA Transaction Flow ..............................................................................................................147
3.9 IBA Management Infrastructure ..............................................................................................148
3.9.1 Management Datagrams ...................................................................................................... 151
3.9.2 Management Methods .......................................................................................................... 152
3.9.2.1 Gets & Sets ........................................................................................................... 152
3.9.2.2 Traps and Notices ................................................................................................. 152
3.9.2.3 Sends ....................................................................................................................153
3.9.2.4 Reports.................................................................................................................. 153
3.9.3 Management Interfaces ........................................................................................................ 153
3.9.4 Subnet Management Interface.............................................................................................. 153
3.9.4.1 Fabric Initialization................................................................................................. 154
3.9.4.2 Directed Routes..................................................................................................... 154
3.9.5 General Service Interface ..................................................................................................... 154
3.9.5.1 Redirection ............................................................................................................ 155
InfiniBand
TM
Architecture Release 1.4 April 7, 2020
V
OLUME 1 - GENERAL SPECIFICATIONS FINAL
InfiniBand
SM
Trade Association Page 5 Proprietary
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
3.10 I/O Operation ..........................................................................................................................155
Chapter 4: bAddressing........................................................................................................158
4.1 Terminology And Concepts .....................................................................................................159
4.1.1 GID Usage and Properties....................................................................................................160
4.1.2 Channel Adapter, Switch, and Router Addressing Rules...................................................... 164
4.1.3 Local Identifiers..................................................................................................................... 164
Chapter 5: Data Packet Format ............................................................................................167
5.1 Packet Types ..........................................................................................................................167
5.2 Data Packet Format................................................................................................................168
5.2.1 Local Route Header (LRH) - 8 Bytes .................................................................................... 171
5.2.2 Global Route Header (GRH) - 40 Bytes................................................................................ 171
5.2.3 Base Transport Header (BTH) - 12 Bytes............................................................................. 172
5.2.4 Reliable Datagram Extended Transport Header (RDETH) - 4 Bytes.................................... 174
5.2.5 Datagram Extended Transport Header (DETH) - 8 Bytes.................................................... 174
5.2.6 RDMA Extended Transport Header (RETH) - 16 Bytes........................................................ 175
5.2.7 Atomic Extended Transport Header (AtomicETH) - 28 Bytes............................................... 176
5.2.8 XRC Extended Transport Header (XRCETH)....................................................................... 176
5.2.9 ACK Extended Transport Header (AETH) - 4 Bytes ............................................................. 177
5.2.10 Atomic ACK Extended Transport Header (AtomicAckETH) - 8 Bytes................................... 177
5.2.11 Immediate Data Extended Transport Header (ImmDt) - 4 Bytes .......................................... 178
5.2.12 INVALIDATE EXTENDED TRANSPORT HEADER (IETH) - 4 BYTES ................................ 178
5.2.13 Payload................................................................................................................................. 178
5.2.14 Invariant CRC ....................................................................................................................... 179
5.2.15 Variant CRC.......................................................................................................................... 179
5.3 Raw Packet Format ................................................................................................................179
5.4 Packet Examples ....................................................................................................................180
Chapter 6: Physical Layer Interface.....................................................................................182
6.1 Overview.................................................................................................................................182
6.2 Services provided by the Physical Layer. ...............................................................................182
6.3 Interface between physical and Link Layers...........................................................................183
6.3.1 Interface between physical receive and link receive............................................................. 183
6.3.1.1 Phy_link - Physical Link Status ............................................................................. 183
6.3.1.2 L_Init_Train - Link Initiate Retraining..................................................................... 183
6.3.1.3 rcv_stream - Receive Stream................................................................................ 184
6.3.2 Interface between physical Transmit and link Transmit. ....................................................... 184
6.3.2.1 Xmit_stream - Transmit Stream............................................................................. 184
6.3.2.2 Xmit_Ready - Physical Transmitter Ready............................................................ 184
