Skip to content

tracing(csharp): RetryHttpHandler emits no telemetry — retries are invisible #479

@eric-wang-1990

Description

@eric-wang-1990

Severity: Medium

Observation

The RetryHttpHandler (csharp/src/Auth/RetryHttpHandler.cs) runs on every HTTP send — confirmed by stack-trace evidence in Traces.DriverResilienceTests.20260527_152945/:

at AdbcDrivers.Databricks.RetryHttpHandler.<>c__DisplayClass11_0.<<SendAsync>b__0>d.MoveNext()
   in .../RetryHttpHandler.cs:line 135

But the handler emits zero telemetry. No retry.attempt, retry.count, retry.delay_ms, retry.reason, retry.exhausted tags or events anywhere. Verified by:

  1. grep -ric "retry" *.log across all 19 trace files in DriverResilienceTests + the dedicated Traces.RetryCountTests.20260527_152556/ archive = 0 hits (other than the config tag adbc.spark.temporarily_unavailable_retry).
  2. Looking for the "retry shape" pattern of multiple sibling SendAsync spans under one parent — grep "SendAsync" shows 0 parents have more than 1 SendAsync child, even on error paths.

A customer asking "is my failure flaky (retried then succeeded) or hard (no retry available)?" cannot answer from the trace alone.

Caveat

hive2.thrift.poll_attempts event-tag DOES exist on HiveServer2Connection.PollForResponseAsync — but that's Thrift operation-status polling, NOT HTTP-level retry.

Suggested fix

In RetryHttpHandler.SendAsync, around the retry loop, call Activity.Current?.AddEvent("retry.attempt", attempt_n, reason, delay_ms) and set retry.total_attempts / retry.exhausted tags on the wrapping SendAsync span on completion.

Evidence

  • Traces.DriverResilienceTests.20260527_152945/ — exception stacktrace traverses RetryHttpHandler.cs:135 but no retry telemetry
  • Traces.RetryCountTests.20260527_152556/ — dedicated test for retry counts, contains no retry.* tag or event anywhere

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions