Timeout Detection & Recovery (TDR)

TDR stands for Timeout Detection and Recovery. This is a feature of the Windows operating system which detects response problems from a graphics card, and recovers to a functional desktop by resetting the card. If the operating system does not receive a response from a graphics card within a certain amount of time (default is 2 seconds), the operating system resets the graphics card. 

Before TDR existed, problems of this nature would have resulted in a system freeze and required a reboot of the operating system.  If TDR is enabled and you see the TDR error message, "Display driver stopped responding and has recovered," this means that the Windows operating system reset the display driver.

There are three different possible debugging configurations: 

• Local debugging with a single GPU,
• Local debugging with multiple GPUs, or
• Remote debugging.

Choose the one that most closely reflects your NVIDIA Nsight setup: 

Local Debugging with a Single GPU

Disabling TDR removes a valuable layer of protection, so it is generally recommended that you keep it enabled.

However, setting the TDR delay too low can cause the debugger to fail for one of two reasons:

• Debugging on some GPUs will fails with a TDR delay of less than 10 seconds.
• Debug builds of CUDA kernels run more slowly and may intrinsically require additional time to complete. With too low of a TDR delay, the kernels may not have enough time to complete.

Therefore, if you are using local debugging with a single GPU, it‘s recommended that you leave TDR enabled, and set the delay to 10 seconds.

To enable TDR and change the delay, do the following: 

1. Right-click the Nsight Monitor icon in the system tray.
2. Select Options. 
   
   
   
   
3. In the Options window on the General tab, set WDDM TDR enabled to True. 
   Change the WDDM TDR Delay from the default setting to 10. 

Local Debugging with Multiple GPUs or Remote Debugging

When using either a local debugging configuration with multiple GPUs, or a remote debugging configuration, it‘s important to disable TDR. This is because with most CUDA applications, a TDR means that any debugging operation after the TDR will fail. You will not be able to step, set breakpoints, view variables, etc. The application will receive a grid launch failure, and the CUcontext will begin to report errors.

Having TDR enabled can interfere with GPU debugging because the graphics card is perceived by the operating system as unresponsive when the execution of a target application is paused or when the debugger is performing certain operations.

To disable TDR, do the following: 

1. Right-click the Nsight Monitor icon in the system tray.
2. Select Options. 
   
   
   
   
3. In the Options window on the General tab, set WDDM TDR enabled to False. 

For more information about TDR, see:

http://www.microsoft.com/whdc/device/display/wddm_timeout.mspx