Field notes
Cluster field notes
The tables worth having open during an incident: port states, link rates, error counters, Xid codes, NCCL debug lines, RoCE prerequisites and where Lustre writes fail. Free, no account. They are also built into the app, next to 150 practice questions and a simulated cluster that answers the commands below for real.
The single most useful table in the domain. Read State together with Physical state — neither means anything alone.
| State |
Physical state |
What it means |
First move |
| Down | Polling | No link partner: cable unplugged, bad cable/transceiver, remote port disabled or dead | Check the cable and the far end; iblinkinfo to see what should be there |
| Down | Disabled | Port administratively disabled | ibportstate enable, or check switch config |
| Initializing | LinkUp | Link trained, but no Subnet Manager has assigned a LID | sminfo — is an SM running? Start opensm |
| Armed | LinkUp | SM is mid-transition; transient | Re-check in a few seconds |
| Active | LinkUp | Healthy and passing traffic | Still check Rate and error counters |
Spot a degraded link by comparing the negotiated Rate against what the hardware should do. Width degradation (4X → 1X) is as damaging as speed degradation.
| Generation |
Per lane |
4x total |
Typical era |
| QDR | 10 Gb/s | 40 Gb/s | legacy |
| FDR | 14 Gb/s | 56 Gb/s | legacy |
| EDR | 25 Gb/s | 100 Gb/s | common |
| HDR | 50 Gb/s | 200 Gb/s | common |
| NDR | 100 Gb/s | 400 Gb/s | current |
| XDR | 200 Gb/s | 800 Gb/s | emerging |
Physical-layer counters point at cables. Congestion counters point at traffic. Clear with perfquery -R and re-read to get a rate, not a lifetime total.
| Counter |
Points at |
Reading |
| SymbolErrorCounter | Physical layer | Corrupted symbols on the wire: cable, transceiver, connector, port |
| LinkErrorRecoveryCounter | Physical layer | Link retrained after errors — marginal signal |
| LinkDownedCounter | Physical / remote | Link fully dropped; flapping if it climbs |
| PortRcvErrors | Physical / packet | Malformed or corrupted packets received |
| PortXmitDiscards | Congestion | Packets dropped waiting to transmit — downstream backpressure |
| PortXmitWait | Congestion | Cycles spent waiting for credits — the fabric is busy, not broken |
| VL15Dropped | SM / management | Management packets dropped — SM pressure or misconfiguration |
The split that matters operationally: is this the user's bug, or your hardware?
| Xid |
Meaning |
Whose problem |
Response |
| 13 | Graphics/compute engine exception — illegal address | Application | Send back to the user; check their kernel |
| 31 | GPU memory page fault (MMU) | Application | Usually an app bug; recurring across users means hardware |
| 43 | Reset channel verification failure | Application | App-level fault, job dies |
| 48 | Double-bit ECC error | Hardware | Drain, check row remap state, likely RMA |
| 63 / 64 | Row remap pending / failure | Hardware | 63: reset to apply. 64: remap failed — RMA |
| 74 | NVLink error | Hardware | Check nvidia-smi nvlink -s, fabric manager, reseat |
| 79 | GPU has fallen off the bus | Hardware | Drain immediately; power-cycle and inspect; validate before resuming |
| 94 / 95 | Contained / uncontained ECC error | Hardware | 94: app killed, GPU survives. 95: needs reset — drain |
| 119 / 120 | GSP RPC timeout | Driver/firmware | Check driver version; often fixed by driver update |
Set NCCL_DEBUG=INFO and read four things. Everything else is noise until these are right.
| Log line |
What it tells you |
| NET/IB : Using [0]mlx5_0:1/IB | Inter-node traffic is on InfiniBand. This is what you want. |
| NET/Socket : Using [0]eth0 | Silent fallback to TCP — expect a 10–20x slowdown. Find out why IB was rejected. |
| via NVL | Intra-node over NVLink. Good. |
| via P2P/IPC | Intra-node over PCIe peer-to-peer. Acceptable, slower than NVLink. |
| via SHM | Intra-node through host shared memory — P2P is unavailable. Check ACS/IOMMU. |
| [send] via NET/IB/0/GDRDMA | GPUDirect RDMA is active — the NIC reads GPU memory directly. |
| Call to ibv_reg_mr failed | Memory registration failed — check ulimit -l (memlock). |
RoCE is InfiniBand's transport on an Ethernet fabric you have to make lossless yourself. Every hop must agree.
| Item |
Check |
Failure symptom if wrong |
| RoCE version | show_gids — use a v2 GID for routed fabrics | Connections fail to establish across subnets |
| GID index | NCCL_IB_GID_INDEX / ib_write_bw -x matches a v2 entry | QP fails to move to RTR |
| Trust mode | mlnx_qos -i shows trust dscp for L3 fabrics | Priority lost at the first router hop; traffic unprotected |
| PFC | Enabled on the RoCE priority (commonly 3) on NIC and every switch | rx_prioN_discards climbing; throughput collapse |
| ECN / DCQCN | Thresholds set on switches; CNPs flowing | Pause storms, head-of-line blocking |
| MTU | Consistent end to end including switches | Silent drops, poor throughput |
df lies on Lustre. lfs df tells the truth.
| Symptom |
Likely cause |
Command |
| ENOSPC while df shows free space | One OST is full and the file is striped onto it | lfs df -h ; lfs getstripe |
| ENOSPC with plenty of OST space | MDT inodes exhausted | lfs df -i |
| Poor bandwidth on huge files | Stripe count 1 — confined to one OST | lfs getstripe ; lfs setstripe -c |
| Client hangs, load climbing | Client evicted, or an OSS is down | lctl dl ; dmesg | grep -i lustre |
| Writes fail for one user only | Quota exceeded (EDQUOT, not ENOSPC) | lfs quota -u /scratch |
What to run in the first sixty seconds. Interviewers listen for the sequence, not the trivia.
| Symptom |
First three commands |
| Multi-node job won't start / hangs at init | ibstat · sminfo · NCCL_DEBUG=INFO run |
| Multi-node job is slow but works | NCCL_DEBUG=INFO (check NET line) · perfquery · ibdiagnet |
| Job dies with CUDA error on one node | dmesg | grep -i xid · nvidia-smi · dcgmi diag -r 2 |
| Process vanishes with no traceback | dmesg | grep -i oom · free -h · top |
| Filesystem writes failing | lfs df -h · lfs df -i · dmesg | grep -i lustre |
| One node slower than its siblings | lspci -vv (LnkCap vs LnkSta) · nvidia-smi topo -m · perfquery |
| GPUs visible but CUDA won't init (HGX) | systemctl status nvidia-fabricmanager · nvidia-smi nvlink -s · journalctl -u nvidia-fabricmanager |
These are the notes. The practice is the product. 10 questions, one incident lab and the whole simulated cluster are free without an account — open ClusterDrill.
← Back to ClusterDrill