GPU Health

Unused ID. If you see an error emitted with this ID contact support.

This ID (4) is overloaded. It can be either invalid or corrupted push buffer stream or GPU semaphore timeout. In the latter case user app error is a potential cause.

Official NVIDIA documentation implicates only the driver as cause of error, thus recommended action is rebooting the system.

Official NVIDIA documentation implicates only the driver as cause of error, thus recommended action is rebooting the system.

Marked as non-critical, this error indicates GPU memory anomalies affecting code and data segments, arrays being out of their declared ranges, applications having illegal memory access issues, or instruction errors.

Restart applications and check whether the same Xid is returned. To debug, refer to cuda-memcheck or CUDA-GDB. In rare cases it can be caused by the hardware degradation. Please contact Modal support if the issue persists.

• The official NVIDIA documentation explains Xid 13 has many potential causes, including hardware, driver, user app, system memory corruption, bus, thermal issue, and fb corruption.
  • NVIDIA Xid 13: GR: SW Notify Error
  • "NVIDIA Xid Errors" (PDF)
  • NVIDIA Xid Errors
  • NVIDIA GPU debug guidelines
• DeepSeek AI paper explains Xid 13 indicates GPU memory anomalies, but can be a hardware error. Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning (Aug 2024)
• The official nvidia memory error doc does not mention Xid 13. NVIDIA GPU Memory Error Management
• The official nvidia k8s device plugin implements Xid 13 as an application error. NVIDIA/k8s-device-plugin health check
• The imbue-ai GPU health check) implements Xid 13 as a non-hardware error.
• The AWS support doc Troubleshoot Xid errors does not mention Xid 13.
• The Google Cloud doc Troubleshoot GPU VMs explains Xid 13 is returned when applications have illegal memory access issues, and recommends cuda-memcheck and CUDA-GDB for debugging.
• The Alibaba Cloud doc explains Xid 13 requires self-check by the user and users to resubmit the load to see if the issue goes away Diagnose GPU-accelerated nodes
• The Tencent Cloud doc How to handle common Xid events does not mention Xid 13.
• The [Azure HPC GPU node problem detector]((https://github.com/Azure/azurehpc/blob/master/experimental/aks_npd_draino/npd/deployment/node-problem-detector-config.yaml#L293) does not implement any health check for Xid 13.

Because this error is attributed only to the driver, the recommended operator action is rebooting the system.

Because this error is attributed only to the driver, the recommended operator action is rebooting the system.

Because this error is attributed only to the driver, the recommended operator action is rebooting the system.

It is expected that the timeout is recoverable.

Indicates framebuffer timeout, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Indicates a video processor exception, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Indicates video processor exception, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Indicates video processor exception, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Indicates GPU semaphore access error, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Debug the user application unless the issue is new and there have been no changes to the application but there has been changes to GPU driver or other GPU system software. Restart applications and check whether the same Xid is returned. To debug, refer to cuda-memcheck or CUDA-GDB. In rare cases it can be caused by the hardware degradation. If the issue persists, please contact support for hardware inspection and repair.

The official NVIDIA docs explains Xid 31 as a user application issue, but can also be driver bugs or hardware issues. This event is logged when MMU reports a fault when an illegal address access is made by an application unit on the chip.

• NVIDIA Xid 31: FIFO: MMU Error
• NVIDIA Xid Errors PDF (September 2024)
• NVIDIA Xid Errors
• The official NVIDIA debugging guideline explains Xid 31 requires user application debugging.
• The official NVIDIA memory error documentation does not mention Xid 31.
• The official NVIDIA Kubernetes device plugin (v0.170.8) implements Xid 31 as a user application issue.
• The imbue-ai GPU health check does not implement any Xid 31 health checks.
• The AWS support doc Troubleshoot Xid errors does not mention Xid 31.
• The Google Cloud troubleshooting guide explains Xid 31 is returned when applications have illegal memory access issues, and recommends cuda-memcheck and CUDA-GDB for debugging.
• The Alibaba Cloud guide Diagnose GPU-accelerated nodes explains Xid 31 requires self-check by users and users to resubmit the workload to see if the same Xid is returned.
• The Tencent Cloud doc How to handle common Xid events does not mention Xid 31.
• The Azure HPC GPU node problem detector does not implement any Xid 31 health checks.
• DeepSeek AI paper on Fire-Flyer explains Xid 31 is a user application issue, which may indicate the GPU memory anomalies, but can also be hardware issues.

The event is reported by the DMA controller of the PCIE bus that manages communication between the NVIDIA driver and GPU. In most cases, a PCI quality issue occurs.

• The official NVIDIA docs explains Xid 32 is a DMA controller error which manages the communication between the NVIDIA driver and GPU over the PCI-E bus. This indicates PCI quality issues, not user application issues.
• "NVIDIA Xid 32: PBDMA Error", https://docs.nvidia.com/deploy/xid-errors/index.html#xid-32-pbdma-error (accessed on Nov 3, 2024)

Indicates internal micro-controller error, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Indicates video processor exception, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Indicates video processor exception, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Indicates video processor exception, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Marked as critical, indicates NVIDIA driver firmware issues. Reboot the system to check whether the firmware issue persists.

• The official NVIDIA docs explains Xid 38 as a potential driver firmware error.
• The Alibaba Cloud guide Diagnose GPU-accelerated nodes explains Xid 38 as a driver firmware issue.

Indicates a video processor exception, and because a driver error is the only possible reason, the recommended operator action is rebooting the system.

Marked as non-critical, indicates GPU stopped processing, due to a user application encountering a software induced fault. Restart applications and check whether the same Xid is returned. And report if the issue persists.

• The official NVIDIA docs explains Xid 43 as a user application hitting a software induced faults.
  • NVIDIA Xid 43: Reset Channel Verif Error, https://docs.nvidia.com/deploy/xid-errors/index.html#xid-43-reset-channel-verif-error (accessed on Nov 3, 2024)
• The official NVIDIA k8s device plugin (v0.17.0) implements Xid 43 as a user application error, indicating GPU stopped processing. The Alibaba Cloud guide explains Xid 43 as a user application error encountering a software induced fault, as a result, GPU stopped processing.
• DeepSeek Fire-Flyer AI-HPC AI paper explains Xid 43 as a user application error, but may indicate GPU memory anomalies.

Marked as critical, indicates uncorrectable GPU errors. Stop existing workloads and reboot the system (or reset GPUs) to clear this error. Marked as critical, indicates uncorrectable GPU errors. If the uncorrectable GPU error persists after rebooting the system, contact provider for inspection and repair of the hardware.

The official NVIDIA docs explains Xid 44 as a potential driver issue. DeepSeek's Fire-Flyer AI paper explains Xid 44 indicates uncorrectable GPU errors, recommends GPU reset or node reboot.

Robust Channel Preemptive Removal. No action, informative only. Indicates channels affected by another failure. On A100, this error could be seen by itself due to unexpected Fabric Manager shutdown when FM is running in the same OS environment as the GPU. Otherwise, this error is safe to ignore as an informational message.

The official NVIDIA docs explains Xid 45 is returned when the kernel driver terminates a GPU application, as a result of a user of system action. NVIDIA Xid 45: OS: Preemptive Channel Removal

• The official NVIDIA debugging guideline explains Xid 45 indicates channel affected by another failure, or may indicate unexpected Fabric Manager shutdown.
• The official nvidia k8s device plugin implements Xid 45 as a user application error, being preemptive cleanup due to previous errors. NVIDIA/k8s-device-plugin health check, https://github.com/NVIDIA/k8s-device-plugin/blob/v0.17.0/internal/rm/health.go#L65-L71 (Aug 2024)
• The AWS Troubleshoot Xid guide does not mention Xid 45.
• The Google Cloud troubleshooting guide does not mention Xid 45.
• The Alibaba Cloud guide explains Xid 45 may indicate multiple cuda applications hitting a DBE, or the result of application being stopped due to another error.
• DeepSeek's Fire-Flyer AI paper explains Xid 45 indicates GPU memory anomalies, affecting code and data segments, but can be hardware-related.

Indicates GPU stopped processing, labeling a driver error as an only possible reason, thus we recommend rebooting the system.

Indicates a video processor exception, labeling a driver error as an only possible reason, thus we recommend rebooting the system.

This event is logged when the GPU detects that an uncorrectable error occurs on the GPU. A GPU reset or node reboot is needed to clear this error. Marked as critical, indicates uncorrectable double bit ECC errors (DBE), which also reports back to the user application. Stop existing workloads and reboot the system (or reset GPUs) to clear this error.

• If Xid 48 is followed by Xid 63 or 64: Drain/cordon the node, wait for all work to complete, and reset GPU(s) reporting the XID (refer to GPU reset capabilities/limitations section below).
• If Xid 48 is not followed by Xid 63 or 64: see Running Field Diagnostics to collect additional debug information. The error is also reported to your application. In most cases, you need to reset the GPU or node to fix this error.

Overloaded ID, could be unstable video memory interface detected or EDC error, clarified in printout (driver error=false)

As driver error is only potential cause, recommended operator action is rebooting the system.

As driver error is only potential cause, recommended operator action is rebooting the system.

PMU Breakpoint. Report a GPU Issue and Reset GPU(s) reporting the XID (refer GPU reset capabilities/limitations section below). Marked as critical, indicates internal micro-controller breakpoint/warning and GPU internal engine stops working. Stop existing workloads and reboot the system (or reset GPUs) to clear this error.

Internal micro-controller breakpoint/warning. The GPU internal engine stops working. Consequently, your businesses are affected.

• The official NVIDIA docs explains Xid 61 indicates internal micro-controller warning.
• The official NVIDIA debugging guidelines recommend resetting the GPU that reports the Xid 61.
• The Alibaba Cloud "Diagnose GPU-accelerated nodes" guide explains Xid 61 indicates GPU internal engine stops working, thus affecting the business.
• DeepSeek Fire-Flyer AI-HPC paper explains Xid 61 indicates uncorrectable GPU errors, which reports back to the user application, recommending GPU reset or node reboot to clear this error.

This event is similar to Xid 61. PMU Halt Error. Report a GPU Issue to support.

These events are logged when the GPU handles ECC memory errors on the GPU.

A100: Row-remapping recording event. This XID indicates successful recording of a row-remapping entry to the InfoROM. If associated with XID 94, the application that encountered the error needs to be restarted. All other applications on the system can keep running as is until there is a convenient time to reset the GPU (refer GPU reset capabilities/limitations section below) or reboot for row remapping to activate.

Legacy GPU: ECC page retirement recording event. If associated with XID 48, drain/cordon the node, wait for all work to complete, and reset GPU(s) reporting the XID (refer GPU reset capabilities/limitations section below). If not, it is from a single bit error and the system can keep running as is until there is a convenient time to reboot it. Xid 63 indicates that the retirement or remapping information is successfully recorded in infoROM.

These events are logged when the GPU handles ECC memory errors on the GPU.

A100 and later: Row-remapping recording failure. This XID indicates a failure in recording a row-remapping entry to the InfoROM. The node should be rebooted immediately since there is a recording failure. If the errors continue, drain, triage, and see Report a GPU Issue for further operator instructions.

Legacy GPU: ECC page retirement recording failure.

See above, however the node should be monitored closely. If there is no associated XID 48 error, then these are related to single bit-errors. The GPU(s) reporting the error must be reset (refer to GPU reset capabilities/limitations section below) immediately since there is a recording failure. If the errors continue, drain, triage, and see Report a GPU Issue. See guidelines on when to RMA GPUs based on excessive errors. ECC page retirement or row remapper recording failure. This event is similar to XID 63. However, Xid 63 indicates that the retirement or remapping information is successfully recorded in infoROM. Xid 64 indicates that the retirement or remapping information fails to be recorded.

Triggered when the GPU handles memory ECC errors on the GPU. Most instances can be resolved by simply resetting the GPU to retain optimal performance.

Video processor exception.

Xid 69 indicates uncorrectable GPU errors. Stop the workloads and reboot the system. If the same Xid is reported again after rebooting the system, the GPU hardware should be inspected and repaired. This error has zero observations on Modal GPUs.

• The official NVIDIA docs explains Xid 69 as a potential hardware/driver issue.
  • NVIDIA Xid Errors
  • NVIDIA Xid Errors PDF (Sep 2024)
• The official NVIDIA debugging guideline does not mention Xid 69.
• The official NVIDIA Memory Error Management doc does not mention Xid 69.
• The official NVIDIA k8s device plugin (v0.17.0) does not treat Xid 69 as an application error.
• The AWS support document does not mention Xid 69.
• The Google Cloud document does not mention Xid 69.
• The Alibaba Cloud doc Diagnose GPU-accelerated nodes does not mention Xid 69.
• The Tencent Cloud doc How to handle common Xid events does not mention Xid 69.
• The Azure HPC GPU node problem detector) implements Xid 69 detection.
• DeepSeek AI paper Fire-Flyer AI-HPC explains Xid 69 indicates GPU uncorrectable errors, recommending GPU reset or node reboot.

Indicates an unknown error. It is marked as a warning and does not require immediate action.

This event is logged when the GPU detects that a problem with a connection from the GPU to another GPU or NVSwitch over NVLink. A GPU reset or node reboot is needed to clear this error.

This event may indicate a hardware failure with the link itself, or may indicate a problem with the device at the remote end of the link. For example, if a GPU fails, another GPU connected to it over NVLink may report an Xid 74 simply because the link went down as a result. The nvidia-smi nvlink command can provide additional details on NVLink errors, and connection information on the links.

If this error is seen repeatedly and GPU reset or node reboot fails to clear the condition, contact your hardware vendor for support.

Extract the hex strings from the XID error message. eg: (0x12345678, 0x12345678, 0x12345678, 0x12345678, 0x12345678, 0x12345678, 0x12345678) Look at the bolded DWORD (the first) and take the following paths if the particular bits (counting from LSB side) are set.

• Bits 4 or 5: Likely HW issue with ECC/Parity. If seen more than 2 times on the same link, report a bug.
• Bits 21 or 22: Marginal channel SI issue. Check link mechanical connections. If other errors accompany, follow the resolution for those.
• Bits 8, 9, 12, 16, 17, 24, 28: Could possibly be a HW issue: Check link mechanical connections and re-seat if a field resolution is required. Run diags if issue persists.

"Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning" says that Xid 74 indicates errors in NVLink. For PCIe A100, it's mainly occurred on the NVLink Bridge between two GPUs. Its occurrence rate is several orders of magnitude higher than other hardware faults. Apart from stress testing to exclude those that are constantly repeating errors, there isn't a good way to avoid the occurrence of Xid 74 issues.

vGPU start error. Reboot the system.

This event is logged when the driver attempts to access the GPU over PCIe and finds it is not accessible. Often caused by hardware failures on the PCIe link bringing the link down. May also be failing GPU hardware or other driver issues. Review system and kernel PCI event logs for indications of link failures.

Example: "NVRM: Xid (PCI:0000:b1:00): 79, GPU has fallen off the bus.

Xid 81 indicates a VGA subsystem error, labeling a hardware failure as the only possible reason. It is recommended to contact support for hardware inspection. This error has zero observations on Modal GPUs.

A hardware or driver error occurs. This error is marked non-critical as single-bit errors can be handled by the GPU. See Running Field Diagnostics to collect additional debug information.

This XID indicates a contained ECC error has occurred. These events are logged when GPU drivers handle errors in GPUs that support error containment, starting with NVIDIA® A100 GPUs.

For Xid 94, these errors are contained to one application, and the application that encountered this error must be restarted.

All other applications running at the time of the Xid are unaffected. It is recommended to reset the GPU container when convenient. Applications can continue to be run until the reset can be performed.

NOTE: This XID is only expected on the NVIDIA Ampere A100s, Hopper, and later architectures. If observed on earlier architectures (e.g. Turing), contact support for investigation.

This XID indicates an uncontained ECC error has occurred.

These events are logged when GPU drivers handle errors in GPUs that support error containment, starting with NVIDIA® A100 GPUs.

For Xid 95, these errors affect multiple applications, and the affected GPU must be reset before applications can restart. Refer https://docs.nvidia.com/deploy/gpu-debug-guidelines/index.html for GPU Reset capabilities & limitations

A100 only:

• If MIG is enabled, drain any work on the other GPU instances, wait for all work to complete, and reset GPU(s) reporting the XID (refer to the GPU reset capabilities/limitations section below).
• If MIG is disabled, the node should be rebooted immediately since there is an uncorrectable uncontained ECC error.

(Modal does not support MIG.)

References:

https://docs.nvidia.com/deploy/a100-gpu-mem-error-mgmt/index.html#user-visible-statistics

This event is similar to Xid 94. However, Xid 94 indicates that the error is suppressed. Xid 95 indicates that the error fails to be suppressed. Other applications on the GPU-accelerated node are also affected.

This is a reserved Xid and is not expected to be observed. If encountered contact support for investigation.

An error with all possible causes, and usually recoverable.

This event should be uncommon unless there is a hardware failure. Modal will drain the worker and reset the GPU, and if the problem persists, contact the hardware vendor for support.

The official NVIDIA docs explains Xid 119 indicates GSP module failures to respond to RPC messages, recommending GPU reset or node power cycle if the issue persists.

The official NVIDIA docs explains Xid 120 indicates GSP module failures to respond to RPC messages, recommending GPU reset or node power cycle if the issue persists.

The official NVIDIA docs explains Xid 121 indicates corrected errors on the C2C NVLink connection to a Grace CPU, with no operational impact, recommending the GPU reset to retrain the link.

• "NVIDIA Xid 121: C2C Link corrected error"
• "NVIDIA Xid Errors"
• "NVIDIA Xid Errors (PDF)

Refer to GPU reset capabilities/limitations section provided in Section D.9 of the Fabric Manager User Guide.

This event is logged when a fault is reported by the remote memory management unit (MMU), such as when an illegal NVLink peer-to-peer access is made by an applicable unit on the chip. Typically these are application-level bugs, but can also be driver bugs or hardware bugs.

This event may occur when the GPU driver has observed uncorrectable errors in GPU memory, in such a way as to interrupt the GPU driver’s ability to mark the pages for dynamic page offlining or row remapping. Modal will drain the worker and reset the GPU, and if the problem persists, contact the hardware vendor for support.

GPU initialization failure. GPU_INIT_ERROR in driver (e.g. error status while polling for FSP boot complete).

Example error on NVIDIA B200: 'NVRM: Xid (PCI:0000:5b:00): 150, MSE Degraded Fatal XC0 i0 Link -1 (0x00000000 0x00000000 0x00000000 0x00000000 0x00000000 0x00000000)'

This error is not expected on non-Hopper GPU architectures.

Recovery action changed for GPU device. The following state transitions are observed:

• 0x0 (None) to 0x1 (GPU Reset Required)
• 0x0 (None) to 0x2 (Node Reboot Required)
• 0x0 (None) to 0x4 (Drain and Reset)

Unobserved transitions:

• 0x0 (None) to 0x3

Because all observed transitions require a reset or reboot, this XID is critical.