HTTP/3 support via OpenSSL 3.5#13186
Open
bneradt wants to merge 3 commits into
Open
Conversation
bneradt
force-pushed
the
openssl35-native-quic-support
branch
12 times, most recently
from
6901c2f to
0a42b65
Compare
bneradt
force-pushed
the
openssl35-native-quic-support
branch
2 times, most recently
from
841b545 to
f00cdf3
Compare
bneradt
changed the title
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
f00cdf3 to
0c99cfc
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates Apache Traffic Server’s HTTP/3 / QUIC support to work with upstream OpenSSL 3.5 by (1) adding a quiche-to-OpenSSL 3.5 QUIC TLS callback compatibility shim and (2) introducing an optional OpenSSL-native QUIC backend that is mutually exclusive with quiche. It also broadens and adds gold tests so core HTTP/3 scenarios can run across either QUIC backend where appropriate.
Changes:
- Add CMake feature detection + build options for OpenSSL 3.5 QUIC (native QUIC + QUIC TLS callbacks compatibility for quiche).
- Introduce an OpenSSL-native QUIC NetProcessor/NetVC/PacketHandler path and wire it into existing QUIC/HTTP3 plumbing.
- Expand/add gold tests and ATSReplay support for HTTP/3, plus minor HTTP/3 transaction / stream lifecycle fixes.
Reviewed changes
Copilot reviewed 59 out of 59 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| tests/gold_tests/timeout/quic_no_activity_timeout.test.py | Generalize QUIC gating to TS_USE_QUIC and keep quiche-specific assertions conditional. |
| tests/gold_tests/timeout/active_timeout.test.py | Run HTTP/3 active-timeout curl path under TS_USE_QUIC (still requires curl http3). |
| tests/gold_tests/headers/via.test.py | Run HTTP/3 portion of Via header test under TS_USE_QUIC. |
| tests/gold_tests/h3/replays/h3_stream_lifetime.replay.yaml | New Proxy Verifier replay covering concurrent stream lifetime behavior. |
| tests/gold_tests/h3/replays/h3_sni.replay.yaml | Adjust SNI/authority values used by the HTTP/3 SNI replay. |
| tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml | New replay covering multi-connection HTTP/3 proxy verifier interop. |
| tests/gold_tests/h3/replays/h3_active_timeout.replay.yaml | New replay exercising HTTP/3 active-timeout cleanup behavior. |
| tests/gold_tests/h3/h3_stream_lifetime.test.py | New ATSReplay-based test for stream lifetime handling. |
| tests/gold_tests/h3/h3_sni_check.test.py | Update QUIC gating + cert material for the SNI behavior test. |
| tests/gold_tests/h3/h3_session_ticket.test.py | New OpenSSL QUIC s_client-based session ticket test (OpenSSL 3.5+). |
| tests/gold_tests/h3/h3_proxy_verifier.test.py | New ATSReplay-based test for proxy verifier client interop. |
| tests/gold_tests/h3/h3_curl.test.py | New curl-based HTTP/3 interop test (requires curl http3 + --http3-only). |
| tests/gold_tests/h3/h3_active_timeout.test.py | New verifier-based HTTP/3 active-timeout cleanup test. |
| tests/gold_tests/autest-site/ats_replay.test.ext | Extend ATSReplay to pass http3_ports when QUIC is enabled in replay config. |
| src/traffic_server/traffic_server.cc | Switch QUIC include gate to TS_USE_QUIC. |
| src/traffic_layout/info.cc | Expose TS_HAS_OPENSSL_QUIC in feature reporting. |
| src/proxy/http3/test/test_Http3FrameDispatcher.cc | Adjust dispatcher test expectations for incremental feed behavior. |
| src/proxy/http3/Http3Transaction.cc | Improve stream-closure, timeout, and deferred read/write event handling; add safe self-delete gating. |
| src/proxy/http3/Http3StreamDataVIOAdaptor.cc | Fix reader lifecycle and make finalize() idempotent. |
| src/proxy/http3/Http3Session.cc | Avoid iterator invalidation while iterating transactions on session close. |
| src/proxy/http3/Http3HeaderVIOAdaptor.cc | Notify transaction when header decode completes (to schedule pending reads). |
| src/proxy/http3/Http3Frame.cc | Add DATA frame readiness parsing based on available payload bytes. |
| src/proxy/http3/Http3App.cc | Inform transaction on QUIC stream close for cleanup. |
| src/iocore/net/SSLSessionTicket.cc | Avoid hard-failing ticket callback when OpenSSL QUIC listener parses tickets before NetVC binding. |
| src/iocore/net/QUICNetVConnection.cc | Refactor stream send/recv to use QUICStreamIO interface and add stream I/O wrappers. |
| src/iocore/net/QUICMultiCertConfigLoader.cc | Add OpenSSL QUIC-specific SNI/cert selection + ALPN/curves hooks and TLS1.3-only enforcement for QUIC. |
| src/iocore/net/quic/QUICStreamVCAdapter.cc | Split read vs consume semantics and improve write reenable behavior. |
| src/iocore/net/quic/QUICStreamManager.cc | Implement local uni/bidi stream id allocation. |
| src/iocore/net/quic/QUICStreamAdapter.cc | Add consume() API and move on_read() notification there. |
| src/iocore/net/quic/QUICStream.cc | Abstract stream I/O via QUICStreamIO, support partial sends with pending blocks, and add on-write notification. |
| src/iocore/net/quic/QUICGlobals.cc | Adjust session handling compilation gates for new QUIC backends. |
| src/iocore/net/quic/QUICConfig.cc | Add QUIC server SSL_CTX creation via OpenSSL native QUIC method; gate quiche-only config APIs. |
| src/iocore/net/quic/OpenSSLQuicCompat.cc | New compatibility shim exporting quiche-expected legacy QUIC TLS callback symbols via OpenSSL 3.5 QUIC TLS callbacks API. |
| src/iocore/net/quic/CMakeLists.txt | Make quiche linkage conditional and add static compat shim library when required. |
| src/iocore/net/P_QUICPacketHandler.h | Add OpenSSL QUIC includes and OpenSSL-specific handler members/methods under feature guards. |
| src/iocore/net/P_QUICNetVConnection.h | Add OpenSSL QUIC init/stream plumbing, introduce QUICStreamIO, and add feature guards for quiche-only fields. |
| src/iocore/net/P_QUICNetProcessor.h | Make quiche-only members conditional and adjust signatures for new handler creation paths. |
| src/iocore/net/OpenSSLQUICPacketHandler.cc | New OpenSSL QUIC listener-based inbound packet handler implementation. |
| src/iocore/net/OpenSSLQUICNetVConnection.cc | New OpenSSL QUIC-based NetVC implementing stream accept/read/write via OpenSSL QUIC stream APIs. |
| src/iocore/net/OpenSSLQUICNetProcessor.cc | New OpenSSL QUIC NetProcessor wiring for inbound QUIC accept. |
| src/iocore/net/CMakeLists.txt | Select OpenSSL QUIC vs quiche QUIC sources and link dependencies based on feature flags. |
| include/tscore/ink_config.h.cmake.in | Add TS_HAS_OPENSSL_QUIC feature macro. |
| include/proxy/http3/Http3Transaction.h | Add stream close notification, new read scheduling helper, close-state tracking, and safe delete gating. |
| include/proxy/http3/Http3StreamDataVIOAdaptor.h | Track reader and finalized state for safe finalize/destruction. |
| include/proxy/http3/Http3HeaderVIOAdaptor.h | Minor formatting cleanup. |
| include/proxy/http3/Http3Frame.h | Allow DATA frame to override parsing readiness behavior. |
| include/iocore/net/QUICMultiCertConfigLoader.h | Add OpenSSL QUIC hook overrides behind TS_HAS_OPENSSL_QUIC. |
| include/iocore/net/quic/QUICStreamVCAdapter.h | Add _consume() override hook. |
| include/iocore/net/quic/QUICStreamManager.h | Make stream-creation methods virtual with out-params; track next local stream ids. |
| include/iocore/net/quic/QUICStreamAdapter.h | Add consume() API and require _consume() implementations. |
| include/iocore/net/quic/QUICStream.h | Introduce QUICStreamIO interface and pending-send tracking for partial writes. |
| include/iocore/net/quic/QUICConnection.h | Add optional on_stream_updated() hook. |
| include/iocore/net/quic/QUICConfig.h | Gate quiche-only APIs and declare quic_new_server_ssl_ctx(). |
| include/iocore/net/quic/Mock.h | Update mock stream adapter behavior to align with read/consume split. |
| CMakePresets.json | Add CI preset wiring for OpenSSL QUIC enablement and adjust Fedora preset inheritance. |
| CMakeLists.txt | Add ENABLE_OPENSSL_QUIC, OpenSSL QUIC feature checks, mutual exclusivity with quiche, and quiche static selection under compat mode. |
| cmake/Findquiche.cmake | Support selecting static quiche when required by compat shim linkage needs. |
| cmake/CheckOpenSSLHasQuicTlsCbs.cmake | New compile-check for OpenSSL QUIC TLS callback API availability. |
| cmake/CheckOpenSSLHasNativeQuic.cmake | New compile-check for OpenSSL native QUIC listener/stream APIs availability. |
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
0c99cfc to
1ecea18
Compare
Contributor
|
[approve ci autest 2] |
10 tasks
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
7b26164 to
8da4a30
Compare
Comment on lines
675
to
679
| UDPNetProcessorInternal::udp_read_from_net(UDPNetHandler *nh, UDPConnection *xuc) | ||
| { | ||
| #if HAVE_RECVMMSG | ||
| read_multiple_messages_from_net(nh, xuc); | ||
| while (read_multiple_messages_from_net(nh, xuc)) {} | ||
| #else |
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
8da4a30 to
c161a49
Compare
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
c161a49 to
006cd26
Compare
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
006cd26 to
a2e5c6e
Compare
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
a2e5c6e to
2342450
Compare
Comment on lines
674
to
679
| void | ||
| UDPNetProcessorInternal::udp_read_from_net(UDPNetHandler *nh, UDPConnection *xuc) | ||
| { | ||
| #if HAVE_RECVMMSG | ||
| read_multiple_messages_from_net(nh, xuc); | ||
| while (read_multiple_messages_from_net(nh, xuc)) {} | ||
| #else |
bneradt
force-pushed
the
openssl35-native-quic-support
branch
2 times, most recently
from
952254e to
ae8c052
Compare
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
ae8c052 to
43b7945
Compare
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
43b7945 to
45b6530
Compare
Comment on lines
+307
to
+309
| if (error && error->cls != Http3ErrorClass::UNDEFINED) { | ||
| this->_handle_error(*error); | ||
| } |
Comment on lines
30
to
+33
| return {Http3FrameType::DATA, Http3FrameType::HEADERS, Http3FrameType::X_RESERVED_1, Http3FrameType::CANCEL_PUSH, | ||
| Http3FrameType::SETTINGS, Http3FrameType::PUSH_PROMISE, Http3FrameType::X_RESERVED_2, Http3FrameType::GOAWAY, | ||
| Http3FrameType::X_RESERVED_3, Http3FrameType::X_RESERVED_4, Http3FrameType::MAX_PUSH_ID, Http3FrameType::X_MAX_DEFINED, | ||
| Http3FrameType::UNKNOWN}; | ||
| Http3FrameType::RESERVED, Http3FrameType::UNKNOWN}; |
added 3 commits
June 8, 2026 16:23
# Overview This patch extends the HTTP/3 autest coverage, using curl, Go, Python/aioquic, and Proxy Verifier HTTP/3 clients to generate their implementations of H3 traffic. It also adds request and response bodies of various sizes, including "large" 300k bodies to exercise multiple packet, buffer, and flow control ATS HTTP/3 implementations. It also exercises interesting requests and responses, such as HEAD, 204, PUT, DELETE, OPTIONS, H3-to-H2 origin forwarding, range responses over cached objects, and malformed HTTP/3 frame behavior. This patch also includes the various production fixes needed for these tests. # Issues Found and their Fixes ## UDP batches could stall large H3 transfers Large request and response bodies exposed a UDP receive starvation bug in the UDP read path. On systems using `recvmmsg()` with edge-triggered readiness, ATS could read one full batch of datagrams and then leave the rest queued in the kernel without another readable event to wake the QUIC stack. This changes `UDPNetProcessorInternal::read_multiple_messages_from_net()` in `src/iocore/net/UnixUDPNet.cc` to return whether the kernel supplied a full batch. `udp_read_from_net()` now processes a bounded number of full batches per event, preserving UDP batching for H3 while avoiding both unread UDP bursts and unbounded net-thread monopolization under sustained QUIC load. ## QUIC stream writes consumed data before quiche accepted it The stream write path consumed the `QUICStreamVCAdapter` write reader inside `_read()`, before `QUICStream::send_data()` knew whether `quiche_conn_stream_send()` had accepted the bytes. When quiche accepted only a partial write or returned a flow-control error, ATS could lose stream data and report write progress too early. This makes `QUICStream::send_data()` keep a pending `IOBufferBlock`/FIN pair until quiche reports successful consumption, and only then calls the new `QUICStreamAdapter::consume()` hook. The concrete reader accounting lives in `QUICStreamVCAdapter::_consume()`, while `QUICStream::has_data_to_send()`, `QUICStream::on_write()`, and `QUICNetVConnection::on_stream_updated()` make newly writable stream data schedule packet writes again. This also treats completed finite writes with only FIN left as writable stream state, so empty bodies and fully consumed bodies still close the H3 stream cleanly. ## QUIC stream reads could expose bytes beyond the VIO request The new H3-to-H2 and large-body tests exposed that `QUICStreamVCAdapter::_read()` could hand more data to the transaction than the read VIO requested. That was usually hidden by small bodies, but larger reads and protocol translation made finite request-body accounting fragile. This clamps cloned input blocks in `QUICStreamVCAdapter::_read()` to the requested and available byte count before filling the read VIO. The adapter now also checks for a missing reader before touching the read buffer, which makes late stream cleanup paths more defensive. ## H3 transaction cleanup raced with stream closure The timeout and stream lifetime tests exposed cases where an `HQTransaction` could be deleted while an event handler was still active, or while the QUIC stream adapter still had read/write cleanup to finish. That left later stream-close and timeout paths touching state that had already been torn down. This adds explicit transaction lifetime state in `HQTransaction`: `_closed`, `_stream_closed`, `_event_handler_active`, and `_is_write_buffer_flushed()`. `Http3App::on_stream_close()` now calls `HQTransaction::stream_closed()` while holding the transaction mutex, and `HQTransaction::_delete_if_possible()` waits until the transaction is done, the stream is closed or no longer readable, and pending writes have flushed before deleting the transaction. ## Malformed H3 streams could leave transactions behind The aioquic edge-case probes found malformed request streams that were correctly rejected at the H3 layer but still left partially constructed transactions attached to the session. Session teardown then either asserted because the transaction list was not empty or touched the H3 session after `Http3Session` had already nulled its network connection. This adds `HQSession::_close_transactions()` and drains any remaining transactions before destroying the H3 session-specific state. It also lets `Http3App::on_stream_close()` attach a cleanup callback to the transaction so the application stream map is erased when the transaction is actually destroyed, rather than when quiche first reports stream closure. ## H3 read completion could run before headers and DATA were settled The H3 request read path could signal completion before asynchronous QPACK header decode and buffered DATA delivery had finished updating the sink VIO. That showed up around HEAD, 204, and stream-close timing because the HTTP state machine needed a stable view of whether headers were decoded and whether a request body existed. This updates `Http3HeaderVIOAdaptor::_on_qpack_decode_complete()` to add the printed header length to the sink VIO and notify `Http3Transaction::on_header_decode_complete()`, which schedules the appropriate read event. `Http3StreamDataVIOAdaptor::finalize()` now uses a persistent reader, writes buffered DATA into the sink VIO exactly once, and updates `ndone`/`nbytes` consistently before the transaction is signaled. ## Malformed H3 frames were not consistently enforced The aioquic client can write raw QUIC stream data, which exposed gaps in ATS's HTTP/3 frame validation. Reserved frames on request streams, DATA-before-HEADERS, client-created push streams, and duplicate control streams did not all reliably close the QUIC connection with an H3 application error. This adds request-stream enforcement through `Http3ProtocolEnforcer` in `Http3Transaction`, recognizes reserved HTTP/3 frame types in `Http3Frame`, and routes connection-level errors through `Http3App::_handle_error()` and `Http3Transaction::_handle_error()` to close the QUIC connection. The transaction signal path now also avoids calling the HTTP state machine through closed transactions or the initial zero-byte write VIO created before the HTTP response handler is installed. ## H3-to-H2 origin traffic exposed H2 body accounting bugs The H3-to-H2 origin coverage found that HEAD and large request-body translation depended on HTTP/2 knowing both the original request method and the exact remaining write VIO byte count. Without that, an H2 origin stream could send DATA past the finite request body or mishandle no-body HEAD semantics. This records the sent request method in `Http2Stream` and uses it when validating response body framing. `Http2ConnectionState::send_a_data_frame()` now caps DATA payloads to the write VIO `ntodo()` value and sends END_STREAM when a finite body has been exhausted, even if the reader has additional buffered bytes. ## The QPACK static table had drifted from the standard table The HEAD, 204, and quic-go coverage exposed that ATS's static QPACK table was not the table used by external HTTP/3 implementations. The extra zstd entry and modified `accept-encoding` value in `src/proxy/http3/QPACK.cc` shifted later static indexes, so an externally encoded `:status 204` could decode as a different status. This restores the standard static table entries by using `accept-encoding: gzip, deflate, br` and removing the non-standard `content-encoding: zstd` entry. The new 204 cases in `tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml`, `tests/gold_tests/h3/replays/h3_server_for_go_client.replay.yaml`, and `tests/gold_tests/h3/replays/h3_server_for_python_client.replay.yaml` cover this interoperability point with Proxy Verifier, quic-go, and aioquic.
Fedora now ships OpenSSL 3.5 with the third-party QUIC TLS callback API, but quiche still links against the older quictls/BoringSSL symbols. ATS therefore could not use the system OpenSSL library for downstream HTTP/3 without dragging in a different TLS stack. This adds CMake detection for the OpenSSL callback API and provides a private compatibility layer that maps quiche's legacy hooks onto SSL_set_quic_tls_cbs. This requires static quiche in that mode so ATS resolves the shim symbols locally and links the final binaries against the system OpenSSL libraries. This also relaxes verifier-only HTTP/3 AuTest gates that do not execute curl, so those tests can run when ATS has QUIC support but the installed curl lacks HTTP/3.
OpenSSL 3.5 can terminate QUIC connections directly, but ATS only had a quiche-backed HTTP/3 listener. Operators who want to use the system OpenSSL QUIC stack needed a separate downstream backend without changing the existing quiche path or origin HTTP/3 scope. This adds an optional ENABLE_OPENSSL_QUIC backend that uses OpenSSL's native QUIC listener and stream APIs for downstream HTTP/3. This keeps the backend mutually exclusive with quiche, exposes TS_HAS_OPENSSL_QUIC, and shares ATS's existing HTTP/3 stream handling above the transport. This also installs native-QUIC TLS callbacks for ALPN and SNI certificate selection before ATS has a QUIC NetVC to bind. OpenSSL native QUIC does not make a selected SSL_CTX certificate active via SSL_set_SSL_CTX alone, so this applies the selected cert, key, and chain to the connection SSL. This also broadens client-side HTTP/3 tests to run with either backend and hardens transaction cleanup when OpenSSL closes stream state before ATS finishes teardown. This caches stream identifiers, declines listener-time QUIC tickets until a NetVC is bound, and adds focused H3 lifecycle and session-ticket coverage.
bneradt
force-pushed
the
openssl35-native-quic-support
branch
from
45b6530 to
7b994e9
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First Commit: HTTP/3 via OpenSSL 3.5 + quiche
Fedora now ships OpenSSL 3.5 with the third-party QUIC TLS
callback API, but quiche still links against the older
quictls/BoringSSL symbols. ATS therefore could not use the system
OpenSSL library for downstream HTTP/3 without dragging in a different
TLS stack.
This adds CMake detection for the OpenSSL callback API and provides a
private compatibility layer that maps quiche's legacy hooks onto
SSL_set_quic_tls_cbs. This requires static quiche in that mode so ATS
resolves the shim symbols locally and links the final binaries against
the system OpenSSL libraries.
This also relaxes verifier-only HTTP/3 AuTest gates that do not execute
curl, so those tests can run when ATS has QUIC support but the installed
curl lacks HTTP/3.
Second Commit: HTTP/3 via OpenSSL 3.5 QUIC
OpenSSL 3.5 can terminate QUIC connections directly, but ATS only had a
quiche-backed HTTP/3 listener. Operators who want to use the system
OpenSSL QUIC stack needed a separate downstream backend without changing
the existing quiche path or origin HTTP/3 scope.
This adds an optional ENABLE_OPENSSL_QUIC backend that uses OpenSSL's
native QUIC listener and stream APIs for downstream HTTP/3. This keeps
the backend mutually exclusive with quiche, exposes TS_HAS_OPENSSL_QUIC,
and shares ATS's existing HTTP/3 stream handling above the transport.
This also installs native-QUIC TLS callbacks for ALPN and SNI
certificate selection before ATS has a QUIC NetVC to bind. OpenSSL
native QUIC does not make a selected SSL_CTX certificate active via
SSL_set_SSL_CTX alone, so this applies the selected cert, key, and chain
to the connection SSL.
This also broadens client-side HTTP/3 tests to run with either backend
and hardens transaction cleanup when OpenSSL closes stream state before
ATS finishes teardown. This caches stream identifiers, declines
listener-time QUIC tickets until a NetVC is bound, and adds focused H3
lifecycle and session-ticket coverage.
Testing
Production Testing
This is currently running in docs for https://docs.trafficserver.apache.org/ targets via Alt-Svc header rewrite rules to negotiate
h3if it is supported.CI AuTests/Fedora 44 Note
This patch is built for openssl 3.5. It assumes CI is running the autests in a fedora:44 container which has that as the system openssl. The autests will fail to build as such until we switch over CI to fedora:44, which is hopefully soon.