libobs-d3d11: Correctly handle duplicator failure

[ThrowTop]

Description

This change corrects error handling in the D3D11 desktop duplicator update path.
Previously, certain DXGI failures during AcquireNextFrame were not treated as fatal, causing the duplicator to remain in an invalid state and repeatedly return black frames.

The function now correctly returns false when frame acquisition fails, allowing higher-level logic to recreate the duplicator when required.

After applying this change and running it continuously for over four months across multiple systems, the persistent black screen issue has not reoccurred.
The error: gs_duplicator_update_frame: Failed to update frame (887A0001)
Occurred 3 times since applyting my patch and the duplicator has correctly recovered every time aka no permament black screen.

---

Motivation and Context

There have been long-standing reports of intermittent black screens when using DXGI desktop duplication capture in fullscreen games. Typically described as occurring very rarely and without a reliable reproduction path.

In practice, this issue manifests less than once per month and is therefore difficult to diagnose or reproduce on demand. When it does occur, capture remains black indefinitely until OBS or the capture pipeline is restarted (For example when clicking on the display capture source).

My testing shows that certain failure cases returned from AcquireNextFrame were not being handled as fatal errors when they should be. This leaves the duplicator in an invalid state instead of signaling failure and allowing higher-level logic to recreate the duplicator.

---

How Has This Been Tested?

• Windows 11 24h2 and 25h2
• DXGI Desktop Duplication capture
• Multi-monitor setup
• Verified capture recovers correctly after failure

---

Types of changes

• Bug fix (non-breaking change which fixes an issue)

[RytoEX]

Not sure why we originally returned true here. Requested some reviews to see if someone knows why or understands why this change is correct.

[ThrowTop]

This code is unchanged since 2015 (same year people started reporting blackscreen bug) And based on microsoft own documentation the duplicator should be recreated for any error EXCEPT timout. So returning true makes no sense there.

[RytoEX]

This code is unchanged since 2015 (same year people started reporting blackscreen bug) And based on microsoft own documentation the duplicator should be recreated for any error EXCEPT timout. So returning true makes no sense there.

Could you point me to the relevant documentation that states this? Just want to make sure I'm reading the same thing.

[ThrowTop]

Can also add that i've been using this patch for almost a year now with 0 blackscreens. which used to be a common occurance happening atleast once a month when recording fullscreen games like cs2. So it has definetly solved my issue and i havent noticed any issues.

[ThrowTop]

This code is unchanged since 2015 (same year people started reporting blackscreen bug) And based on microsoft own documentation the duplicator should be recreated for any error EXCEPT timout. So returning true makes no sense there.

Could you point me to the relevant documentation that states this? Just want to make sure I'm reading the same thing.

Its more that they dont define any recoverable states guarantee for random errors.
They only explicitly say to release the IDXGIOutputDuplication interface and create a new one for DXGI_ERROR_ACCESS_LOST

Any any error that isnt expected behaviour like DXGI_ERROR_WAIT_TIMEOUT if u call AccuireNextFrame multiple times without waiting for timeout is probably something that went wrong.

The lack of recovery documentation and the fact that the fix works as intended and fixes the issue is enough imo.

https://learn.microsoft.com/en-us/windows/win32/api/dxgi1_2/nf-dxgi1_2-idxgioutputduplication-acquirenextframe