USB 3.x: A brief summary of packet types, flow control and handshakes

Scope

This page summarizes the main traffic elements on a SuperSpeed USB link. Being a summary, it’s by no means exhaustive or fully accurate, hence there’s no way around reading the actual spec for the full picture.

Another page shows a sample hex dump of data on wires, with some of the elements described below.

Packet groups

Except for idle symbols (which is just transmitted zeros), the traffic can be divided into three groups, which are distinct by their first four K-symbols:

• Header Packets, 20 symbols long, starting with SHP, SHP, SHP, EPF. These packets include four subgroups: Link Management Packets (LMP), Transaction Packets (TP), Data Packet Headers (DPH) and Isochronous Timestamp Packets (ITP), detailed below.
• Data Packet Payloads (DPP), 12 + n bytes long, where 1024 >= n >= 0 is the payload length, starting with SDP, SDP, SDP, EPF. These packets come immediately after DPH packets (no gap is allowed) and contain the data packet’s payload data. DPPs (and only DPPs) also have an end sequence: END, END, END, EPF.
• Link Commands, 8 byte long, starting with SLC, SLC, SLC, EPF. These packets consist of 11 bits of actual payload + 5 bits of CRC, which are both transmitted twice. These packets carry acknowledgement and flow control for Header Packets, link keepalive, and power management commands.

Breakdown of Header Packets (Per section 8.2):

• Link Management Packets (LMP): Non-routable packets going between two link partners for setting different attributes of the raw link (e.g. the timeout for transition from U1 to U2, etc)
• Transaction Packets (TP): Routable packets that travel all the way from host to device and vice versa. These packets don’t carry data, but assist with the end-to-end communication: ACK, NRDY, ERDY, STATUS, STALL, PING etc.
• Data Packet Headers (DPH): The precursor to packets carrying data, containing the information about them. Routed like TPs, and usually immediately followed by a Data Packet Payload (DPP). However the DPP isn’t considered part of the header packet, so if the DPP arrives faulty or is absent, the DPH portion is processed independently in terms of acknowledgment.
• Isochronous Timestamp Packets (ITP): Sent by the host, never responded to, and serves isochronous endpoints only.

Acknowledgment, flow control and other handshakes

There are two major mechanisms for controlling the flow of packets and acknowledging their proper arrival:

• Link layer: For Header Packets, there are LGOOD_n/LBAD/LRTY Link Command packets for acknowledging arrival, as well as a credit-based flow control mechanism consisting of LCRD_x Link Command packets.
• Protocol layer: For DPPs, there are ACK/NRDY/ERDY Transaction Packets (which are Header Packets), each relating to a specific IN/OUT endpoint. These packets acknowledge, report errors and control the flow of Data Packet Payloads.

Data Packet (DP) is the term used for a pair of DPH and DPP. The flow control for this pair involves two mechanisms: The Data Packet Header’s arrival is controlled by LGOOD_n/LBAD/LRTY. As for the Data Packet Payload, it’s ignored if the DPH didn’t arrive properly (see section 7.2.4.1.6) or with invalid data (see section 8.11.2). If the DPH arrived faulty, is has to be retransmitted like any Header Packet. Should this retransmission also include the DPP that followed it? Discussed on this page.

If the DPH is OK, it must be acknowledged immediately and regardless of the Data Packet that follows it, in order to meet the timing requirements (more on that below). It’s up to the Protocol Layer to verify the validity of the Data Packet Payload and respond accordingly. This means that if the Payload accumulates errors as it’s relayed through hubs, those errors are detected at the final destination.

There are also two other handshake mechanisms:

• Power management handshakes, consisting of certain Link Command Packets: LGO_Ux, LAU, LXU and LPMA. Discussed briefly on this page, and in more detail here.
• Port Configuration / Port Configuration Response handshake, consisting of Header Packets, which are subject to the Header Packet’s flow mechanism. However this is a two-way handshake between host and device that must take place soon after a device starts.

Header Packet Acknowledgement

Each header packet’s last two bytes is a Link Control Word, in which three bits are allocated for the Header Sequence Number (see section 7.2.1.1.3), which is a counter running from 0 to 7, and then rolls back to 0.

This number is incremented for each Header Packet that is inserted into one of the transmitter’s 4 buffers (in other words, a retransmitted packet doesn’t affect this counter). When the packet arrives properly at the receiver, it must respond with an LGOOD_n packet, where n is the Header Sequence Number in the respective packet. If the packet arrived with an error, it must respond with an LBAD packet.

The protocol is strict about ordering of both Header Sequence Numbers in the transmitted Header Packets as well as the LGOOD_n packets: Any diversion from the strict incremental ordering of these numbers (except for retransmission, see below) causes an immediate transition into Recovery (see sections 7.2.4.1.4 and 7.2.4.1.7). As the packets are protected by CRCs, there is no reason for this to ever happen with Header Packets, except that it may occur due to a corrupt LGOOD_n packet (hence the unexpected numbering of the following LGOOD_n packet).

This strict ordering requirements makes packet retransmission somewhat tricky, since the transmitter might have sent a few additional Header Packets until the LBAD packet arrived. This is resolved with the following mechanism:

• The receiver of the faulty Header Packet sends an LBAD and ignores all following arriving Header Packets until further notice (per section 7.2.4.1.4)
• The receiver of the LBAD packet responds with sending an LTRY packet, acknowledging the LBAD packet. It then retransmits the Header Packets that are left in its buffers, in (modulo-8) rising Header Sequence Number order. Retransmitted DHPs may include a retransmission of the DPP that followed it, or the DL bit can be set instead (section 7.2.4.1.9). This is discussed on this page. LINK
• The receiver of the LRTY packet stops ignoring Header Packets from that point on, and hence responds to these packets normally.

Section 7.2.4.1.1 says: “A header packet that is re-transmitted shall maintain its originally assigned Header Sequence Number”.

Note that the exchange of LCRD_x packets is not affected by the LBAD/LRTY handshake, as the receiver of Header Packets may continue to process the packets that have already arrived properly while waiting for retransmission.

A strict timeout is enforced on the time between the end of transmitting a Header Packet arrival of an LGOOD_n / LBAD packet, by virtue of the PENDING_HP_TIMER: 3 μs. (Or as put in section 7.2.4.1.10: “A port has a header packet transmitted but not acknowledged by its link partner”). This timeout isn’t particular for each Header Packet and its respective LGOOD_n packet, but rather enforces that there should be no more than 3 us of absence of LGOOD_n / LBAD packets as long as such packet is expected.

This means that LGOOD_n / LBAD packets must have high priority for transmission. For example, a DPH followed by a DPP can be up to 20 + 1036 = 1056 bytes long, which take ~2.11 μs to transmit on a USB 3.0 link. Hence if the receiver of a Header Packet just began to transmit a DPH, it can be up to ~2.11 μs until its transmission channel is vacant to send the LGOOD_n / LBAD packet.

As there are always some buffering and pipelining involved in handling data at high rates, there’s not much spare in the timing budget. This is why the DPH must be acknowledged immediately as has fully arrived, as mentioned earlier: Waiting for the Data Packet Payload is out of the question, as doing so can easily delay the LGOOD_n / LBAD response beyond the allowed timeout: For example, waiting for 1000 bytes to arrive takes 2 us, and then, in the worst case, wait another 2.11 μs for a transmitted Data Packet Payload to finish transmission.

Header Packet Flow Control

The underlying model is that the transmitter has 4 buffers for storing the Header Packets that have been transmitted, but not yet acknowledged (and hence might require retransmission) and that the receiver has 4 buffers for storing arriving packets before they are processed. Consequently, each Header Packet is acknowledged twice: Once when it arrives properly (with an LGOOD_n packet), and once when it has been processed and the buffer space it occupied has been freed (with an LCRD_x packet).

The Link Layer flow control mechanism for Header Packets is based upon a simple credit mechanism: When a port enters the U0 state (from Polling, Hot Reset, or Recovery) it’s not allowed to send any Header Packet to its like partner. Rather, it has to wait for permissions, credits, which take the form of LCRD_x packets: Each received LCRD_x packet is a permission to transmit one Header Packet. Hence each port maintains a credit counter, which is incremented every time an LCRD_x packet arrives, and decremented every time a Header Packet is transmitted. This counter will never exceed 4, as the underlying model is that each port has 4 buffers for receiving Header Packets (per section 7.2.4.1.1: “A port receiving LCRD_x from its link partner shall increment its Remote Rx Header Buffer Credit Count by one each time an LCRD_x is received up to four”). Curiously, the protocol doesn’t say how to react to an LCRD_x that would increment the counter beyond 4, even though switching to Recovery is the obvious choice.

A retransmitted packet doesn’t consume credits. Accordingly, no LCRD_x packet is sent on behalf of a packet that was discarded with an LBAD, nor on the packets that arrived after it and were ignored. Credit consumption, and hence LCRD_x packets, relate only to packets for which an LGOOD_n is transmitted.

An exception is upon invocation into U0, when the retransmission is a result of a Sequence Number Announcement that indicates one or more packets didn’t arrive. This credit consumption is vital, as the credit count has been reset back to 4 on entry to U0.

In order to ensure that no LCRD_x packets are lost due to bit errors, there are four packets available, LCRD_A, LCRD_B, LCRD_C and LCRD_D. The alphabetical designation has no meaning, except that these packets must be transmitted in alphabetical order, from A to D and back to A, starting from A upon entry into U0.

If the receiving port gets an LCRD_x packet that deviates from the expected order, it must switch to Recovery, as this indicates a loss of an LRCD_x packet. There’s also a timeout restriction for credit reclaiming, governed by the CREDIT_HP_TIMER: In essence, whenever a port expect to receive an LCRD_x packet (i.e. its credit count for transmission is less than the maximum of 4), it shouldn’t wait more than 5 ms for an LCRD_x packet. This applies to the time between the transmitted Header Packet that dropped the credit count to 3, and the first LCRD_x received after its transmission, and also applies to the time interval between LCRD_x packets as long as the credit count isn’t 4 (per section 7.2.4.1.10).

Announcements when entering U0

When a port enters U0 (from Polling or Recovery or Hot Reset), it must transmit an LGOOD_n packet as well as LCRD_x packets (as applicable) to announce its receivers state. This is meaningful only when returning to U0 from Recovery, in which case there’s a possible uncertainty on which Header Packets have arrived, and how many such packets the receiver is capable of receiving. Section 7.2.4.1.1 goes through the details of this topic.

The LGOOD_n announcement acknowledges the last Header Packet that was received properly prior to Recovery. If U0 was entered from Polling or Hot Reset, LGOOD_7 is always transmitted, signaling the link partner to start sending Header Packets with Sequence Number zero.

Both ports also transmit LCRD_x packets, starting from LCRD_A, in order to reflect their current capability to receive Header Packets, given that the link partner has just entered U0 with zero credits at its disposal. When entering U0 from Polling or Hot Reset, this means transmitting all four LCRD_x packets, from A to D, to reflect that all four reception buffers are vacant.

Both PENDING_HP_TIMER and CREDIT_HP_TIMER are in effect upon entry into U0, as if a Header Packet was just transmitted. In particular, this means that the LGOOD_x packet must arrive no later than 3 us into U0.

Once after the link initialization, and when each port has at least one credit, it should transmit a Port Capabilities LMP, announcing its link speed, OTG capability and possible port direction (up and/or down). Per section 8.4.5, this must be sent tPortConfiguration = 20 us after link initialization the ports go into Disabled / Inactive states (depending on which side).

Further on, the downstream ports sends a Port Configuration LMP, which must be acknowledged by the link partner with a Port Configuration Response. This handshake is practically meaningless, and was probably intended as a mechanism for upgrading the link speed and lane count in later USB 3.x revisions, but the follow-up USB 3.x revisions eventually chose another mechanism (SCD1/SCD2 encoding during Polling.LFPS).

Upon re-entry into U0 from Recovery, retransmission of Header Packets must take place before any other Header Packets (including Port Configuration packets), as retransmission must be done with the same Sequence Number as the original transmission.

LGOOD_n, LCRD_x and link stability

One unfortunate consequence of the strict requirements on ordering of LGOOD_n and LCRD_x packets is that a single bit error in such packet results in a transition into Recovery. The timeout requirements of the different LTSSM sub-states limit the Recovery phase to around ~19 ms, much longer than typically observed, which is a few microseconds.

This is however something to keep in mind when designing systems with real-time expectations that are based upon USB 3.x: Bandwidth outages of several milliseconds is a theoretically possible scenario, as no link is error free, even though these are expected to be no longer than a few microseconds, and very rare when proper hardware is involved.