From a261971455872a57aea13dba6d585c3d0196e9b7 Mon Sep 17 00:00:00 2001 From: bneradt Date: Thu, 28 May 2026 23:04:34 -0500 Subject: [PATCH 1/3] Add H3 quiche traffic handling tests and provide fixes # Overview This patch extends the HTTP/3 autest coverage, using curl, Go, Python/aioquic, and Proxy Verifier HTTP/3 clients to generate their implementations of H3 traffic. It also adds request and response bodies of various sizes, including "large" 300k bodies to exercise multiple packet, buffer, and flow control ATS HTTP/3 implementations. It also exercises interesting requests and responses, such as HEAD, 204, PUT, DELETE, OPTIONS, H3-to-H2 origin forwarding, range responses over cached objects, and malformed HTTP/3 frame behavior. This patch also includes the various production fixes needed for these tests. # Issues Found and their Fixes ## UDP batches could stall large H3 transfers Large request and response bodies exposed a UDP receive starvation bug in the UDP read path. On systems using `recvmmsg()` with edge-triggered readiness, ATS could read one full batch of datagrams and then leave the rest queued in the kernel without another readable event to wake the QUIC stack. This changes `UDPNetProcessorInternal::read_multiple_messages_from_net()` in `src/iocore/net/UnixUDPNet.cc` to return whether the kernel supplied a full batch. `udp_read_from_net()` now processes a bounded number of full batches per event, preserving UDP batching for H3 while avoiding both unread UDP bursts and unbounded net-thread monopolization under sustained QUIC load. ## QUIC stream writes consumed data before quiche accepted it The stream write path consumed the `QUICStreamVCAdapter` write reader inside `_read()`, before `QUICStream::send_data()` knew whether `quiche_conn_stream_send()` had accepted the bytes. When quiche accepted only a partial write or returned a flow-control error, ATS could lose stream data and report write progress too early. This makes `QUICStream::send_data()` keep a pending `IOBufferBlock`/FIN pair until quiche reports successful consumption, and only then calls the new `QUICStreamAdapter::consume()` hook. The concrete reader accounting lives in `QUICStreamVCAdapter::_consume()`, while `QUICStream::has_data_to_send()`, `QUICStream::on_write()`, and `QUICNetVConnection::on_stream_updated()` make newly writable stream data schedule packet writes again. This also treats completed finite writes with only FIN left as writable stream state, so empty bodies and fully consumed bodies still close the H3 stream cleanly. ## QUIC stream reads could expose bytes beyond the VIO request The new H3-to-H2 and large-body tests exposed that `QUICStreamVCAdapter::_read()` could hand more data to the transaction than the read VIO requested. That was usually hidden by small bodies, but larger reads and protocol translation made finite request-body accounting fragile. This clamps cloned input blocks in `QUICStreamVCAdapter::_read()` to the requested and available byte count before filling the read VIO. The adapter now also checks for a missing reader before touching the read buffer, which makes late stream cleanup paths more defensive. ## H3 transaction cleanup raced with stream closure The timeout and stream lifetime tests exposed cases where an `HQTransaction` could be deleted while an event handler was still active, or while the QUIC stream adapter still had read/write cleanup to finish. That left later stream-close and timeout paths touching state that had already been torn down. This adds explicit transaction lifetime state in `HQTransaction`: `_closed`, `_stream_closed`, `_event_handler_active`, and `_is_write_buffer_flushed()`. `Http3App::on_stream_close()` now calls `HQTransaction::stream_closed()` while holding the transaction mutex, and `HQTransaction::_delete_if_possible()` waits until the transaction is done, the stream is closed or no longer readable, and pending writes have flushed before deleting the transaction. ## Malformed H3 streams could leave transactions behind The aioquic edge-case probes found malformed request streams that were correctly rejected at the H3 layer but still left partially constructed transactions attached to the session. Session teardown then either asserted because the transaction list was not empty or touched the H3 session after `Http3Session` had already nulled its network connection. This adds `HQSession::_close_transactions()` and drains any remaining transactions before destroying the H3 session-specific state. It also lets `Http3App::on_stream_close()` attach a cleanup callback to the transaction so the application stream map is erased when the transaction is actually destroyed, rather than when quiche first reports stream closure. ## H3 read completion could run before headers and DATA were settled The H3 request read path could signal completion before asynchronous QPACK header decode and buffered DATA delivery had finished updating the sink VIO. That showed up around HEAD, 204, and stream-close timing because the HTTP state machine needed a stable view of whether headers were decoded and whether a request body existed. This updates `Http3HeaderVIOAdaptor::_on_qpack_decode_complete()` to add the printed header length to the sink VIO and notify `Http3Transaction::on_header_decode_complete()`, which schedules the appropriate read event. `Http3StreamDataVIOAdaptor::finalize()` now uses a persistent reader, writes buffered DATA into the sink VIO exactly once, and updates `ndone`/`nbytes` consistently before the transaction is signaled. ## Malformed H3 frames were not consistently enforced The aioquic client can write raw QUIC stream data, which exposed gaps in ATS's HTTP/3 frame validation. Reserved frames on request streams, DATA-before-HEADERS, client-created push streams, and duplicate control streams did not all reliably close the QUIC connection with an H3 application error. This adds request-stream enforcement through `Http3ProtocolEnforcer` in `Http3Transaction`, recognizes reserved HTTP/3 frame types in `Http3Frame`, and routes connection-level errors through `Http3App::_handle_error()` and `Http3Transaction::_handle_error()` to close the QUIC connection. The transaction signal path now also avoids calling the HTTP state machine through closed transactions or the initial zero-byte write VIO created before the HTTP response handler is installed. ## H3-to-H2 origin traffic exposed H2 body accounting bugs The H3-to-H2 origin coverage found that HEAD and large request-body translation depended on HTTP/2 knowing both the original request method and the exact remaining write VIO byte count. Without that, an H2 origin stream could send DATA past the finite request body or mishandle no-body HEAD semantics. This records the sent request method in `Http2Stream` and uses it when validating response body framing. `Http2ConnectionState::send_a_data_frame()` now caps DATA payloads to the write VIO `ntodo()` value and sends END_STREAM when a finite body has been exhausted, even if the reader has additional buffered bytes. ## The QPACK static table had drifted from the standard table The HEAD, 204, and quic-go coverage exposed that ATS's static QPACK table was not the table used by external HTTP/3 implementations. The extra zstd entry and modified `accept-encoding` value in `src/proxy/http3/QPACK.cc` shifted later static indexes, so an externally encoded `:status 204` could decode as a different status. This restores the standard static table entries by using `accept-encoding: gzip, deflate, br` and removing the non-standard `content-encoding: zstd` entry. The new 204 cases in `tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml`, `tests/gold_tests/h3/replays/h3_server_for_go_client.replay.yaml`, and `tests/gold_tests/h3/replays/h3_server_for_python_client.replay.yaml` cover this interoperability point with Proxy Verifier, quic-go, and aioquic. --- ci/rat-exclude.txt | 2 + include/iocore/net/quic/Mock.h | 22 +- include/iocore/net/quic/QUICConnection.h | 1 + include/iocore/net/quic/QUICStream.h | 6 + include/iocore/net/quic/QUICStreamAdapter.h | 4 +- include/iocore/net/quic/QUICStreamVCAdapter.h | 1 + include/iocore/net/quic/QUICTypes.h | 6 +- include/proxy/http2/Http2Stream.h | 21 +- include/proxy/http3/Http3App.h | 1 + include/proxy/http3/Http3ProtocolEnforcer.h | 1 + include/proxy/http3/Http3Session.h | 3 + .../proxy/http3/Http3StreamDataVIOAdaptor.h | 8 +- include/proxy/http3/Http3Transaction.h | 26 +- include/proxy/http3/Http3Types.h | 1 + include/proxy/logging/TransactionLogData.h | 2 + src/iocore/net/P_QUICNetVConnection.h | 1 + src/iocore/net/P_UDPNet.h | 2 +- src/iocore/net/QUICNetVConnection.cc | 30 +- src/iocore/net/UnixUDPNet.cc | 13 +- src/iocore/net/quic/QUICStream.cc | 97 +++- src/iocore/net/quic/QUICStreamAdapter.cc | 9 +- src/iocore/net/quic/QUICStreamVCAdapter.cc | 62 ++- src/proxy/http2/Http2ConnectionState.cc | 15 +- src/proxy/http3/Http3App.cc | 23 +- src/proxy/http3/Http3DebugNames.cc | 2 + src/proxy/http3/Http3Frame.cc | 15 +- src/proxy/http3/Http3HeaderVIOAdaptor.cc | 14 +- src/proxy/http3/Http3ProtocolEnforcer.cc | 17 +- src/proxy/http3/Http3Session.cc | 18 +- src/proxy/http3/Http3StreamDataVIOAdaptor.cc | 24 +- src/proxy/http3/Http3Transaction.cc | 195 +++++-- src/proxy/http3/QPACK.cc | 3 +- .../http3/test/test_Http3FrameDispatcher.cc | 30 +- src/proxy/http3/test/test_QPACK.cc | 4 +- .../autest-site/ats_replay.test.ext | 6 +- .../autest-site/conditions.test.ext | 51 +- .../early_hints/early_hints.test.py | 11 +- tests/gold_tests/h3/go_h3_client/go.mod | 13 + tests/gold_tests/h3/go_h3_client/go.sum | 38 ++ tests/gold_tests/h3/go_h3_client/main.go | 300 +++++++++++ tests/gold_tests/h3/h3_active_timeout.test.py | 26 + tests/gold_tests/h3/h3_curl.test.py | 143 ++++++ tests/gold_tests/h3/h3_flow_control.test.py | 27 + tests/gold_tests/h3/h3_go_client.test.py | 130 +++++ tests/gold_tests/h3/h3_h2_origin.test.py | 26 + tests/gold_tests/h3/h3_proxy_verifier.test.py | 27 + tests/gold_tests/h3/h3_python_client.test.py | 132 +++++ tests/gold_tests/h3/h3_range_cache.test.py | 140 +++++ tests/gold_tests/h3/h3_session_ticket.sh | 41 ++ tests/gold_tests/h3/h3_session_ticket.test.py | 110 ++++ tests/gold_tests/h3/h3_sni_check.test.py | 10 +- .../gold_tests/h3/h3_stream_lifetime.test.py | 27 + tests/gold_tests/h3/py_h3_client/h3_client.py | 332 ++++++++++++ .../h3/replays/h3_active_timeout.replay.yaml | 85 ++++ .../h3/replays/h3_flow_control.replay.yaml | 132 +++++ .../h3/replays/h3_h2_origin.replay.yaml | 352 +++++++++++++ .../h3/replays/h3_proxy_verifier.replay.yaml | 481 ++++++++++++++++++ .../h3_server_for_go_client.replay.yaml | 268 ++++++++++ .../h3_server_for_python_client.replay.yaml | 297 +++++++++++ .../gold_tests/h3/replays/h3_sni.replay.yaml | 6 +- .../h3/replays/h3_stream_lifetime.replay.yaml | 198 +++++++ .../gold_tests/timeout/active_timeout.test.py | 2 +- .../timeout/quic_no_activity_timeout.test.py | 33 +- tests/pyproject.toml | 2 +- 64 files changed, 3984 insertions(+), 141 deletions(-) create mode 100644 tests/gold_tests/h3/go_h3_client/go.mod create mode 100644 tests/gold_tests/h3/go_h3_client/go.sum create mode 100644 tests/gold_tests/h3/go_h3_client/main.go create mode 100644 tests/gold_tests/h3/h3_active_timeout.test.py create mode 100644 tests/gold_tests/h3/h3_curl.test.py create mode 100644 tests/gold_tests/h3/h3_flow_control.test.py create mode 100644 tests/gold_tests/h3/h3_go_client.test.py create mode 100644 tests/gold_tests/h3/h3_h2_origin.test.py create mode 100644 tests/gold_tests/h3/h3_proxy_verifier.test.py create mode 100644 tests/gold_tests/h3/h3_python_client.test.py create mode 100644 tests/gold_tests/h3/h3_range_cache.test.py create mode 100755 tests/gold_tests/h3/h3_session_ticket.sh create mode 100644 tests/gold_tests/h3/h3_session_ticket.test.py create mode 100644 tests/gold_tests/h3/h3_stream_lifetime.test.py create mode 100644 tests/gold_tests/h3/py_h3_client/h3_client.py create mode 100644 tests/gold_tests/h3/replays/h3_active_timeout.replay.yaml create mode 100644 tests/gold_tests/h3/replays/h3_flow_control.replay.yaml create mode 100644 tests/gold_tests/h3/replays/h3_h2_origin.replay.yaml create mode 100644 tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml create mode 100644 tests/gold_tests/h3/replays/h3_server_for_go_client.replay.yaml create mode 100644 tests/gold_tests/h3/replays/h3_server_for_python_client.replay.yaml create mode 100644 tests/gold_tests/h3/replays/h3_stream_lifetime.replay.yaml diff --git a/ci/rat-exclude.txt b/ci/rat-exclude.txt index 39ad1b11ca8..bb8926ef25c 100644 --- a/ci/rat-exclude.txt +++ b/ci/rat-exclude.txt @@ -24,6 +24,8 @@ blib/** **/*.default.in **/*.config **/*.gold +**/go.mod +**/go.sum **/*.hrw4u **/.gitignore **/.gitmodules diff --git a/include/iocore/net/quic/Mock.h b/include/iocore/net/quic/Mock.h index 95333ee2001..3332f82926f 100644 --- a/include/iocore/net/quic/Mock.h +++ b/include/iocore/net/quic/Mock.h @@ -28,6 +28,8 @@ #include "iocore/net/quic/QUICStreamAdapter.h" #include "iocore/net/quic/QUICStream.h" +#include + class MockQUICContext; using namespace std::literals; @@ -191,6 +193,11 @@ class MockQUICConnectionInfoProvider : public QUICConnectionInfoProvider { return negotiated_application_name_sv; } + + void + on_stream_updated() override + { + } }; class MockQUICStreamManager : public QUICStreamManager @@ -431,6 +438,11 @@ class MockQUICConnection : public QUICConnection return negotiated_application_name_sv; } + void + on_stream_updated() override + { + } + int _transmit_count = 0; int _retransmit_count = 0; Ptr _mutex; @@ -519,13 +531,19 @@ class MockQUICStreamAdapter : public QUICStreamAdapter Ptr _read(size_t len) override { - this->_sending_data_len -= len; - Ptr block = make_ptr(new_IOBufferBlock()); + len = std::min(len, this->_sending_data_len); + Ptr block = make_ptr(new_IOBufferBlock()); block->alloc(iobuffer_size_to_index(len, BUFFER_SIZE_INDEX_32K)); block->fill(len); return block; } + void + _consume(size_t len) override + { + this->_sending_data_len -= std::min(len, this->_sending_data_len); + } + private: size_t _sending_data_len = 0; size_t _total_sending_data_len = 0; diff --git a/include/iocore/net/quic/QUICConnection.h b/include/iocore/net/quic/QUICConnection.h index 96f8f25b895..5a91c10380d 100644 --- a/include/iocore/net/quic/QUICConnection.h +++ b/include/iocore/net/quic/QUICConnection.h @@ -60,6 +60,7 @@ class QUICConnectionInfoProvider virtual bool is_handshake_completed() const = 0; virtual QUICVersion negotiated_version() const = 0; virtual std::string_view negotiated_application_name() const = 0; + virtual void on_stream_updated() = 0; }; class QUICConnection : public QUICConnectionInfoProvider diff --git a/include/iocore/net/quic/QUICStream.h b/include/iocore/net/quic/QUICStream.h index e0cb94c8da0..ef84c9cb6aa 100644 --- a/include/iocore/net/quic/QUICStream.h +++ b/include/iocore/net/quic/QUICStream.h @@ -26,6 +26,7 @@ #include "tscore/List.h" #include "iocore/eventsystem/Event.h" +#include "iocore/eventsystem/IOBuffer.h" #include "iocore/net/quic/QUICConnection.h" #include "iocore/net/quic/QUICDebugNames.h" @@ -53,6 +54,7 @@ class QUICStream QUICStreamDirection direction() const; bool is_bidirectional() const; bool has_no_more_data() const; + bool has_data_to_send(); QUICOffset final_offset() const; @@ -66,6 +68,7 @@ class QUICStream * QUICApplication need to call one of these functions when it process VC_EVENT_* */ void on_read(); + void on_write(); void on_eos(); /** @@ -85,6 +88,9 @@ class QUICStream uint64_t _received_bytes = 0; uint64_t _sent_bytes = 0; bool _has_no_more_data = false; + Ptr _pending_send_block; + bool _pending_send_fin = false; + bool _sent_fin = false; }; class QUICStreamStateListener diff --git a/include/iocore/net/quic/QUICStreamAdapter.h b/include/iocore/net/quic/QUICStreamAdapter.h index 1bf36cbcc1a..87afe51a5b8 100644 --- a/include/iocore/net/quic/QUICStreamAdapter.h +++ b/include/iocore/net/quic/QUICStreamAdapter.h @@ -39,6 +39,7 @@ class QUICStreamAdapter virtual int64_t write(QUICOffset offset, const uint8_t *data, uint64_t data_length, bool fin) = 0; Ptr read(size_t len); + void consume(size_t len); virtual bool is_eos() = 0; virtual uint64_t unread_len() = 0; virtual uint64_t read_len() = 0; @@ -60,6 +61,7 @@ class QUICStreamAdapter virtual void notify_eos() = 0; protected: - virtual Ptr _read(size_t len) = 0; + virtual Ptr _read(size_t len) = 0; + virtual void _consume(size_t len) = 0; QUICStream &_stream; }; diff --git a/include/iocore/net/quic/QUICStreamVCAdapter.h b/include/iocore/net/quic/QUICStreamVCAdapter.h index ce27422560c..ce3e9327001 100644 --- a/include/iocore/net/quic/QUICStreamVCAdapter.h +++ b/include/iocore/net/quic/QUICStreamVCAdapter.h @@ -65,6 +65,7 @@ class QUICStreamVCAdapter : public VConnection, public QUICStreamAdapter protected: Ptr _read(size_t len) override; + void _consume(size_t len) override; VIO _read_vio; VIO _write_vio; diff --git a/include/iocore/net/quic/QUICTypes.h b/include/iocore/net/quic/QUICTypes.h index 706e807cadb..951226b054e 100644 --- a/include/iocore/net/quic/QUICTypes.h +++ b/include/iocore/net/quic/QUICTypes.h @@ -455,9 +455,9 @@ class QUICFiveTuple int protocol() const; private: - IpEndpoint _source; - IpEndpoint _destination; - int _protocol; + IpEndpoint _source{}; + IpEndpoint _destination{}; + int _protocol = 0; uint64_t _hash_code = 0; }; diff --git a/include/proxy/http2/Http2Stream.h b/include/proxy/http2/Http2Stream.h index bc4b0743c51..67e8525d5a3 100644 --- a/include/proxy/http2/Http2Stream.h +++ b/include/proxy/http2/Http2Stream.h @@ -163,6 +163,7 @@ class Http2Stream : public ProxyTransaction void increment_data_length(uint64_t length); bool payload_length_is_valid() const; bool is_write_vio_done() const; + int64_t write_vio_ntodo() const; void update_sent_count(unsigned num_bytes); Http2StreamId get_id() const; Http2StreamState get_state() const; @@ -174,6 +175,7 @@ class Http2Stream : public ProxyTransaction void set_receive_headers(HTTPHdr &h2_headers); void reset_receive_headers(); void reset_send_headers(); + void set_sent_request_method(int method); MIOBuffer *read_vio_writer() const; int64_t read_vio_read_avail(); bool is_read_enabled() const; @@ -214,6 +216,7 @@ class Http2Stream : public ProxyTransaction Http2StreamId _id = -1; Http2StreamState _state = Http2StreamState::HTTP2_STREAM_STATE_IDLE; int64_t _http_sm_id = -1; + int _sent_request_method{-1}; HTTPHdr _receive_header; #if TS_USE_MALLOC_ALLOCATOR @@ -314,6 +317,12 @@ Http2Stream::is_write_vio_done() const return this->write_vio.ntodo() == 0; } +inline int64_t +Http2Stream::write_vio_ntodo() const +{ + return this->write_vio.ntodo(); +} + inline void Http2Stream::update_sent_count(unsigned num_bytes) { @@ -389,6 +398,12 @@ Http2Stream::reset_send_headers() this->_send_header.create(HTTPType::RESPONSE); } +inline void +Http2Stream::set_sent_request_method(int method) +{ + _sent_request_method = method; +} + // Check entire DATA payload length if content-length: header exists inline void Http2Stream::increment_data_length(uint64_t length) @@ -405,9 +420,9 @@ Http2Stream::payload_length_is_valid() const // Skip Content-Length check on [RFC 7230] 3.3.2 conditions bool is_payload_precluded = - this->is_outbound_connection() && (_send_header.method_get_wksidx() == HTTP_WKSIDX_HEAD || - (_send_header.method_get_wksidx() == HTTP_WKSIDX_GET && _send_header.presence(mask) && - _receive_header.status_get() == HTTPStatus::NOT_MODIFIED)); + this->is_outbound_connection() && + (_sent_request_method == HTTP_WKSIDX_HEAD || (_sent_request_method == HTTP_WKSIDX_GET && _send_header.presence(mask) && + _receive_header.status_get() == HTTPStatus::NOT_MODIFIED)); if (content_length != 0 && !is_payload_precluded && content_length != data_length) { Warning("Bad payload length content_length=%d data_legnth=%d session_id=%" PRId64, content_length, diff --git a/include/proxy/http3/Http3App.h b/include/proxy/http3/Http3App.h index 80c05cf12c9..70e7377beff 100644 --- a/include/proxy/http3/Http3App.h +++ b/include/proxy/http3/Http3App.h @@ -79,6 +79,7 @@ class Http3App : public QUICApplication void _handle_bidi_stream_on_write_complete(int event, VIO *vio); void _handle_bidi_stream_on_eos(int event, VIO *vio); + void _handle_error(const Http3Error &error); void _set_qpack_stream(Http3StreamType type, QUICStreamVCAdapter *adapter); QUICStreamVCAdapter::IOInfo &_get_stream_info(QUICStreamId stream_id); diff --git a/include/proxy/http3/Http3ProtocolEnforcer.h b/include/proxy/http3/Http3ProtocolEnforcer.h index ee291bdeca1..e9801f6802b 100644 --- a/include/proxy/http3/Http3ProtocolEnforcer.h +++ b/include/proxy/http3/Http3ProtocolEnforcer.h @@ -37,4 +37,5 @@ class Http3ProtocolEnforcer : public Http3FrameHandler private: bool _is_first_frame_received_on_control = false; + bool _is_headers_frame_received = false; }; diff --git a/include/proxy/http3/Http3Session.h b/include/proxy/http3/Http3Session.h index 6528b1f0e6f..4785390eba7 100644 --- a/include/proxy/http3/Http3Session.h +++ b/include/proxy/http3/Http3Session.h @@ -56,6 +56,9 @@ class HQSession : public ProxySession void remove_transaction(HQTransaction *trans); HQTransaction *get_transaction(QUICStreamId); +protected: + void _close_transactions(); + private: // this should be unordered map? Queue _transaction_list; diff --git a/include/proxy/http3/Http3StreamDataVIOAdaptor.h b/include/proxy/http3/Http3StreamDataVIOAdaptor.h index 4eff081d9d7..8968ab65cc2 100644 --- a/include/proxy/http3/Http3StreamDataVIOAdaptor.h +++ b/include/proxy/http3/Http3StreamDataVIOAdaptor.h @@ -42,7 +42,9 @@ class Http3StreamDataVIOAdaptor : public Http3FrameHandler bool has_data(); private: - VIO *_sink_vio = nullptr; - int64_t _total_data_length = 0; - MIOBuffer *_buffer; + VIO *_sink_vio = nullptr; + int64_t _total_data_length = 0; + MIOBuffer *_buffer = nullptr; + IOBufferReader *_reader = nullptr; + bool _finalized = false; }; diff --git a/include/proxy/http3/Http3Transaction.h b/include/proxy/http3/Http3Transaction.h index 1b0eb48806c..a17eb6d4bf7 100644 --- a/include/proxy/http3/Http3Transaction.h +++ b/include/proxy/http3/Http3Transaction.h @@ -29,6 +29,8 @@ #include "proxy/http3/Http3FrameDispatcher.h" #include "proxy/http3/Http3FrameCollector.h" +#include + class QUICStreamIO; class HQSession; class Http09Session; @@ -36,6 +38,7 @@ class Http3Session; class Http3HeaderFramer; class Http3DataFramer; class Http3HeaderVIOAdaptor; +class Http3ProtocolEnforcer; class Http3StreamDataVIOAdaptor; class HQTransaction : public ProxyTransaction @@ -53,6 +56,8 @@ class HQTransaction : public ProxyTransaction void transaction_done() override; void release() override; int get_transaction_id() const override; + void stream_closed(); + void set_stream_cleanup(std::function cleanup); void increment_transactions_stat() override; void decrement_transactions_stat() override; @@ -81,6 +86,7 @@ class HQTransaction : public ProxyTransaction void _schedule_read_complete_event(); void _unschedule_read_complete_event(); void _close_read_complete_event(Event *e); + void _schedule_read_event(); void _schedule_write_ready_event(); void _unschedule_write_ready_event(); void _close_write_ready_event(Event *e); @@ -90,12 +96,14 @@ class HQTransaction : public ProxyTransaction void _signal_event(int event, Event *e); void _signal_read_event(); void _signal_write_event(); + bool _is_write_buffer_flushed(); void _delete_if_possible(); EThread *_thread = nullptr; MIOBuffer _read_vio_buf{BUFFER_SIZE_INDEX_4K}; QUICStreamVCAdapter::IOInfo &_info; + QUICStreamId _stream_id = 0; size_t _sent_bytes = 0; @@ -106,7 +114,12 @@ class HQTransaction : public ProxyTransaction Event *_write_ready_event = nullptr; Event *_write_complete_event = nullptr; - bool _transaction_done = false; + bool _transaction_done = false; + bool _event_handler_active = false; + bool _closed = false; + bool _stream_closed = false; + + std::function _stream_cleanup; }; class Http3Transaction : public HQTransaction @@ -121,6 +134,7 @@ class Http3Transaction : public HQTransaction int state_stream_closed(int event, Event *data) override; void do_io_close(int lerrno = -1) override; + void on_header_decode_complete(); bool is_response_header_sent() const; bool is_response_body_sent() const; @@ -131,14 +145,16 @@ class Http3Transaction : public HQTransaction private: int64_t _process_read_vio() override; int64_t _process_write_vio() override; + void _handle_error(const Http3Error &error); // These are for HTTP/3 Http3FrameDispatcher _frame_dispatcher; Http3FrameCollector _frame_collector; - Http3FrameGenerator *_header_framer = nullptr; - Http3FrameGenerator *_data_framer = nullptr; - Http3HeaderVIOAdaptor *_header_handler = nullptr; - Http3StreamDataVIOAdaptor *_data_handler = nullptr; + Http3ProtocolEnforcer *_protocol_enforcer = nullptr; + Http3FrameGenerator *_header_framer = nullptr; + Http3FrameGenerator *_data_framer = nullptr; + Http3HeaderVIOAdaptor *_header_handler = nullptr; + Http3StreamDataVIOAdaptor *_data_handler = nullptr; }; /** diff --git a/include/proxy/http3/Http3Types.h b/include/proxy/http3/Http3Types.h index 9d98578827c..5eb6eac2722 100644 --- a/include/proxy/http3/Http3Types.h +++ b/include/proxy/http3/Http3Types.h @@ -62,6 +62,7 @@ enum class Http3FrameType : uint64_t { X_RESERVED_4 = 0x09, MAX_PUSH_ID = 0x0D, X_MAX_DEFINED = 0x0D, + RESERVED = 0x21, UNKNOWN = 0x0E, }; diff --git a/include/proxy/logging/TransactionLogData.h b/include/proxy/logging/TransactionLogData.h index b65388d0a17..5649eb6265c 100644 --- a/include/proxy/logging/TransactionLogData.h +++ b/include/proxy/logging/TransactionLogData.h @@ -27,6 +27,8 @@ #include "proxy/hdrs/HTTP.h" #include "tscore/ink_inet.h" +#include +#include #include #include diff --git a/src/iocore/net/P_QUICNetVConnection.h b/src/iocore/net/P_QUICNetVConnection.h index e6fec2b8ea5..89af3685e4e 100644 --- a/src/iocore/net/P_QUICNetVConnection.h +++ b/src/iocore/net/P_QUICNetVConnection.h @@ -129,6 +129,7 @@ class QUICNetVConnection : public UnixNetVConnection, bool is_at_anti_amplification_limit() const override; bool is_address_validation_completed() const override; bool is_handshake_completed() const override; + void on_stream_updated() override; // QUICSupport QUICConnection *get_quic_connection() override; diff --git a/src/iocore/net/P_UDPNet.h b/src/iocore/net/P_UDPNet.h index 04af410c371..42438b2c3ba 100644 --- a/src/iocore/net/P_UDPNet.h +++ b/src/iocore/net/P_UDPNet.h @@ -56,7 +56,7 @@ class UDPNetProcessorInternal : public UDPNetProcessor private: void read_single_message_from_net(UDPNetHandler *nh, UDPConnection *uc); - void read_multiple_messages_from_net(UDPNetHandler *nh, UDPConnection *xuc); + bool read_multiple_messages_from_net(UDPNetHandler *nh, UDPConnection *xuc); }; extern UDPNetProcessorInternal udpNetInternal; diff --git a/src/iocore/net/QUICNetVConnection.cc b/src/iocore/net/QUICNetVConnection.cc index d113ae593e1..18a85913739 100644 --- a/src/iocore/net/QUICNetVConnection.cc +++ b/src/iocore/net/QUICNetVConnection.cc @@ -37,6 +37,7 @@ #include #include +#include namespace { @@ -390,8 +391,28 @@ QUICNetVConnection::stream_manager() } void -QUICNetVConnection::close_quic_connection(QUICConnectionErrorUPtr /* error ATS_UNUSED */) +QUICNetVConnection::close_quic_connection(QUICConnectionErrorUPtr error) { + if (this->_quiche_con == nullptr || quiche_conn_is_closed(this->_quiche_con) || quiche_conn_is_draining(this->_quiche_con)) { + return; + } + + const bool is_app_error = error != nullptr && error->cls == QUICErrorClass::APPLICATION; + const uint64_t code = error == nullptr ? static_cast(QUICTransErrorCode::NO_ERROR) : error->code; + const uint8_t *reason = nullptr; + size_t reason_len = 0; + + if (error != nullptr && error->msg != nullptr) { + reason = reinterpret_cast(error->msg); + reason_len = strlen(error->msg); + } + + if (quiche_conn_close(this->_quiche_con, is_app_error, code, reason, reason_len) != 0) { + QUICConDebug("failed to close QUIC connection with code %" PRIu64, code); + return; + } + + this->_schedule_packet_write_ready(false); } void @@ -500,6 +521,12 @@ QUICNetVConnection::negotiated_application_name() const return std::string_view(reinterpret_cast(name), name_len); } +void +QUICNetVConnection::on_stream_updated() +{ + this->_schedule_packet_write_ready(false); +} + bool QUICNetVConnection::is_closed() const { @@ -690,7 +717,6 @@ QUICNetVConnection::_handle_write_ready() while (written + max_udp_payload_size <= quantum) { res = quiche_conn_send(this->_quiche_con, reinterpret_cast(udp_payload->end()) + written, max_udp_payload_size, &send_info); - #ifdef HAVE_SO_TXTIME if (written == 0) { memcpy(&send_at_hint, &send_info.at, sizeof(struct timespec)); diff --git a/src/iocore/net/UnixUDPNet.cc b/src/iocore/net/UnixUDPNet.cc index 513b074c775..48f329be46c 100644 --- a/src/iocore/net/UnixUDPNet.cc +++ b/src/iocore/net/UnixUDPNet.cc @@ -70,7 +70,8 @@ EventType ET_UDP; namespace { #ifdef HAVE_RECVMMSG -const uint32_t MAX_RECEIVE_MSG_PER_CALL{16}; //< VLEN parameter for the recvmmsg call. +const uint32_t MAX_RECEIVE_MSG_PER_CALL{16}; //< VLEN parameter for the recvmmsg call. +const uint32_t MAX_RECEIVE_MSG_BATCHES_PER_EVENT{8}; //< Maximum number of full recvmmsg batches per event. #endif DbgCtl dbg_ctl_udpnet{"udpnet"}; @@ -516,7 +517,7 @@ UDPNetProcessorInternal::read_single_message_from_net(UDPNetHandler *nh, UDPConn } #ifdef HAVE_RECVMMSG -void +bool UDPNetProcessorInternal::read_multiple_messages_from_net(UDPNetHandler *nh, UDPConnection *xuc) { UnixUDPConnection *uc = static_cast(xuc); @@ -575,7 +576,7 @@ UDPNetProcessorInternal::read_multiple_messages_from_net(UDPNetHandler *nh, UDPC if (return_val <= 0) { Dbg(dbg_ctl_udp_read, "Done. recvmmsg() ret is %d, errno %s", return_val, strerror(errno)); - return; + return false; } Dbg(dbg_ctl_udp_read, "recvmmsg() read %d packets", return_val); @@ -593,7 +594,7 @@ UDPNetProcessorInternal::read_multiple_messages_from_net(UDPNetHandler *nh, UDPC if (mhdr.msg_namelen <= 0) { Dbg(dbg_ctl_udp_read, "Unable to get remote address from recvmmsg() for fd: %d", uc->getFd()); - return; + return false; } toaddr[packet_num].ss_family = AF_UNSPEC; @@ -666,6 +667,8 @@ UDPNetProcessorInternal::read_multiple_messages_from_net(UDPNetHandler *nh, UDPC nh->udp_callbacks.enqueue(uc); uc->onCallbackQueue = 1; } + + return return_val == static_cast(MAX_RECEIVE_MSG_PER_CALL); } #endif @@ -673,7 +676,7 @@ void UDPNetProcessorInternal::udp_read_from_net(UDPNetHandler *nh, UDPConnection *xuc) { #if HAVE_RECVMMSG - read_multiple_messages_from_net(nh, xuc); + for (uint32_t batch = 0; batch < MAX_RECEIVE_MSG_BATCHES_PER_EVENT && read_multiple_messages_from_net(nh, xuc); ++batch) {} #else read_single_message_from_net(nh, xuc); #endif diff --git a/src/iocore/net/quic/QUICStream.cc b/src/iocore/net/quic/QUICStream.cc index e221df25163..cdc427c9af8 100644 --- a/src/iocore/net/quic/QUICStream.cc +++ b/src/iocore/net/quic/QUICStream.cc @@ -24,7 +24,8 @@ #include "iocore/net/quic/QUICStream.h" #include "iocore/net/quic/QUICStreamAdapter.h" -constexpr uint32_t MAX_STREAM_FRAME_OVERHEAD = 24; +constexpr uint32_t MAX_STREAM_FRAME_OVERHEAD = 24; +constexpr size_t MAX_STREAM_SEND_BYTES_PER_EVENT = 16 * 1024; QUICStream::QUICStream(QUICConnectionInfoProvider *cinfo, QUICStreamId sid) : _connection_info(cinfo), _id(sid) {} @@ -60,6 +61,22 @@ QUICStream::has_no_more_data() const return this->_has_no_more_data; } +bool +QUICStream::has_data_to_send() +{ + if (this->_pending_send_block) { + return true; + } + if (this->_adapter == nullptr) { + return false; + } + + const bool has_buffered_data = this->_adapter->unread_len() > 0; + const bool needs_fin = !this->_sent_fin && this->_adapter->is_eos() && this->_adapter->total_len() == this->_sent_bytes; + + return has_buffered_data || needs_fin; +} + void QUICStream::set_io_adapter(QUICStreamAdapter *adapter) { @@ -87,6 +104,14 @@ QUICStream::on_read() { } +void +QUICStream::on_write() +{ + if (this->_connection_info != nullptr) { + this->_connection_info->on_stream_updated(); + } +} + void QUICStream::on_eos() { @@ -117,21 +142,63 @@ QUICStream::send_data(quiche_conn *quiche_con) ssize_t len = 0; [[maybe_unused]] ErrorCode error_code{0}; // Only set if QUICHE_ERR_STREAM_STOPPED(-15) or QUICHE_ERR_STREAM_RESET(-16) are // returned by quiche_conn_stream_send. + size_t written_this_event = 0; - len = quiche_conn_stream_capacity(quiche_con, this->_id); - if (len <= 0) { - return; - } - Ptr block = this->_adapter->read(len); - if (this->_adapter->total_len() == this->_sent_bytes + block->size()) { - fin = true; - } - if (block->size() > 0 || fin) { - ssize_t written_len = - quiche_conn_stream_send(quiche_con, this->_id, reinterpret_cast(block->start()), block->size(), fin, &error_code); - if (written_len >= 0) { - this->_sent_bytes += written_len; + while (written_this_event < MAX_STREAM_SEND_BYTES_PER_EVENT) { + len = quiche_conn_stream_capacity(quiche_con, this->_id); + if (len <= 0) { + return; + } + + if (!this->_pending_send_block) { + size_t read_len = std::min(static_cast(len), MAX_STREAM_SEND_BYTES_PER_EVENT - written_this_event); + this->_pending_send_block = this->_adapter->read(read_len); + if (!this->_pending_send_block) { + if (!this->_sent_fin && this->_adapter->is_eos() && this->_adapter->total_len() == this->_sent_bytes) { + static constexpr uint8_t empty_data = 0; + ssize_t written_len = quiche_conn_stream_send(quiche_con, this->_id, &empty_data, 0, true, &error_code); + if (written_len >= 0) { + this->_sent_fin = true; + } + } + this->_adapter->encourge_write(); + return; + } + this->_pending_send_fin = this->_adapter->total_len() == this->_sent_bytes + this->_pending_send_block->size(); } + + Ptr block = this->_pending_send_block; + fin = this->_pending_send_fin; + if (block->size() == 0 && !fin) { + this->_pending_send_block = nullptr; + this->_pending_send_fin = false; + this->_adapter->encourge_write(); + continue; + } + + if (block->size() > 0 || fin) { + ssize_t written_len = quiche_conn_stream_send(quiche_con, this->_id, reinterpret_cast(block->start()), + block->size(), fin, &error_code); + if (written_len >= 0) { + this->_adapter->consume(written_len); + this->_sent_bytes += written_len; + written_this_event += written_len; + if (written_len >= block->size()) { + this->_pending_send_block = nullptr; + this->_pending_send_fin = false; + this->_sent_fin = fin; + } else { + block->consume(written_len); + return; + } + if (!this->has_data_to_send()) { + this->_adapter->encourge_write(); + return; + } + continue; + } + } + this->_adapter->encourge_write(); + return; } - this->_adapter->encourge_write(); } diff --git a/src/iocore/net/quic/QUICStreamAdapter.cc b/src/iocore/net/quic/QUICStreamAdapter.cc index 9992b8a661e..2e49214ad50 100644 --- a/src/iocore/net/quic/QUICStreamAdapter.cc +++ b/src/iocore/net/quic/QUICStreamAdapter.cc @@ -26,7 +26,12 @@ Ptr QUICStreamAdapter::read(size_t len) { - auto ret = this->_read(len); + return this->_read(len); +} + +void +QUICStreamAdapter::consume(size_t len) +{ + this->_consume(len); this->_stream.on_read(); - return ret; } diff --git a/src/iocore/net/quic/QUICStreamVCAdapter.cc b/src/iocore/net/quic/QUICStreamVCAdapter.cc index d79abe51db9..225bef63c4a 100644 --- a/src/iocore/net/quic/QUICStreamVCAdapter.cc +++ b/src/iocore/net/quic/QUICStreamVCAdapter.cc @@ -24,6 +24,8 @@ #include "iocore/eventsystem/VConnection.h" #include "iocore/net/quic/QUICStreamVCAdapter.h" +#include + QUICStreamVCAdapter::QUICStreamVCAdapter(QUICStream &stream) : VConnection(new_ProxyMutex()), QUICStreamAdapter(stream) { SET_HANDLER(&QUICStreamVCAdapter::state_stream_open); @@ -84,18 +86,45 @@ QUICStreamVCAdapter::_read(size_t len) SCOPED_MUTEX_LOCK(lock, this->_write_vio.mutex, this_ethread()); IOBufferReader *reader = this->_write_vio.get_reader(); - block = make_ptr(reader->get_current_block()->clone()); + if (reader == nullptr || reader->get_current_block() == nullptr || reader->block_read_avail() <= 0) { + return block; + } + + const size_t read_len = std::min(len, static_cast(reader->block_read_avail())); + block = make_ptr(reader->get_current_block()->clone()); if (block->size()) { block->consume(reader->start_offset); - block->_end = std::min(block->start() + len, block->_buf_end); - this->_write_vio.ndone += block->size(); + block->_end = block->start() + read_len; + } + if (block->size() == 0) { + block = nullptr; } - reader->consume(block->size()); } return block; } +void +QUICStreamVCAdapter::_consume(size_t len) +{ + if (len == 0 || this->_write_vio.op != VIO::WRITE) { + return; + } + + SCOPED_MUTEX_LOCK(lock, this->_write_vio.mutex, this_ethread()); + + IOBufferReader *reader = this->_write_vio.get_reader(); + if (reader == nullptr) { + return; + } + + const size_t consume_len = std::min(len, static_cast(std::max(reader->read_avail(), 0))); + if (consume_len > 0) { + reader->consume(consume_len); + this->_write_vio.ndone += consume_len; + } +} + bool QUICStreamVCAdapter::is_eos() { @@ -119,7 +148,8 @@ QUICStreamVCAdapter::unread_len() { if (this->_write_vio.op == VIO::WRITE) { SCOPED_MUTEX_LOCK(lock, this->_write_vio.mutex, this_ethread()); - return this->_write_vio.get_reader()->block_read_avail(); + IOBufferReader *reader = this->_write_vio.get_reader(); + return reader == nullptr ? 0 : reader->block_read_avail(); } else { return 0; } @@ -321,23 +351,33 @@ QUICStreamVCAdapter::do_io_shutdown(ShutdownHowTo_t /* howto ATS_UNUSED */) } void -QUICStreamVCAdapter::reenable(VIO * /* vio ATS_UNUSED */) +QUICStreamVCAdapter::reenable(VIO *vio) { - // TODO We probably need to tell QUICStream that the application consumed received data - // to update receive window here. In other words, we should not update receive window - // until the application consume data. + if (vio == nullptr || vio->op != VIO::WRITE) { + // TODO We probably need to tell QUICStream that the application consumed received data + // to update receive window here. In other words, we should not update receive window + // until the application consume data. + return; + } + + const bool has_buffered_data = vio->get_reader() != nullptr && vio->get_reader()->read_avail() > 0; + const bool needs_fin = vio->nbytes != INT64_MAX && vio->ntodo() == 0; + if (has_buffered_data || needs_fin) { + this->stream().on_write(); + } } bool QUICStreamVCAdapter::is_readable() { - return this->stream().direction() != QUICStreamDirection::SEND && _read_vio.nbytes != _read_vio.ndone; + return this->stream().direction() != QUICStreamDirection::SEND && this->_read_vio.op == VIO::READ && + this->_read_vio.nbytes != this->_read_vio.ndone; } bool QUICStreamVCAdapter::is_writable() { - return this->stream().direction() != QUICStreamDirection::RECEIVE && _write_vio.nbytes != _read_vio.ndone; + return this->stream().direction() != QUICStreamDirection::RECEIVE; } int diff --git a/src/proxy/http2/Http2ConnectionState.cc b/src/proxy/http2/Http2ConnectionState.cc index 9c80aefe295..65cccf14604 100644 --- a/src/proxy/http2/Http2ConnectionState.cc +++ b/src/proxy/http2/Http2ConnectionState.cc @@ -2275,6 +2275,7 @@ Http2ConnectionState::send_a_data_frame(Http2Stream *stream, size_t &payload_len uint8_t flags = 0x00; IOBufferReader *resp_reader = stream->get_data_reader_for_send(); + bool last_write_vio_payload{false}; SCOPED_MUTEX_LOCK(stream_lock, stream->mutex, this_ethread()); @@ -2307,6 +2308,16 @@ Http2ConnectionState::send_a_data_frame(Http2Stream *stream, size_t &payload_len } else { payload_length = resp_reader->read_avail(); } + const int64_t remaining_write = stream->write_vio_ntodo(); + if (remaining_write != INT64_MAX) { + if (remaining_write > 0) { + last_write_vio_payload = payload_length >= static_cast(remaining_write); + payload_length = std::min(payload_length, static_cast(remaining_write)); + } else { + last_write_vio_payload = true; + payload_length = 0; + } + } } else { payload_length = 0; } @@ -2334,7 +2345,8 @@ Http2ConnectionState::send_a_data_frame(Http2Stream *stream, size_t &payload_len return Http2SendDataFrameResult::NO_PAYLOAD; } - if (stream->is_write_vio_done() && !resp_reader->is_read_avail_more_than(payload_length) && !stream->expect_send_trailer()) { + if (stream->is_write_vio_done() && (last_write_vio_payload || !resp_reader->is_read_avail_more_than(payload_length)) && + !stream->expect_send_trailer()) { Http2StreamDebug(this->session, stream->get_id(), "End of Data Frame"); flags |= HTTP2_FLAGS_DATA_END_STREAM; } @@ -2456,6 +2468,7 @@ Http2ConnectionState::send_headers_frame(Http2Stream *stream) flags |= HTTP2_FLAGS_HEADERS_END_HEADERS; if (stream->is_outbound_connection()) { // Will be sending a request_header int method = send_hdr->method_get_wksidx(); + stream->set_sent_request_method(method); // Set END_STREAM on request headers for POST, etc. methods combined with // an explicit length 0. Some origins RST on request headers with diff --git a/src/proxy/http3/Http3App.cc b/src/proxy/http3/Http3App.cc index 09d349a6a69..1f3d1a3172a 100644 --- a/src/proxy/http3/Http3App.cc +++ b/src/proxy/http3/Http3App.cc @@ -103,6 +103,15 @@ Http3App::start() // } } +void +Http3App::_handle_error(const Http3Error &error) +{ + if (error.cls == Http3ErrorClass::CONNECTION) { + this->_qc->close_quic_connection( + std::make_unique(QUICErrorClass::APPLICATION, static_cast(error.code))); + } +} + void Http3App::on_stream_open(QUICStream &stream) { @@ -131,7 +140,15 @@ Http3App::on_stream_open(QUICStream &stream) void Http3App::on_stream_close(QUICStream &stream) { - this->_streams.erase(stream.id()); + QUICStreamId const stream_id = stream.id(); + + if (auto *txn = this->_ssn->get_transaction(stream_id); txn != nullptr) { + SCOPED_MUTEX_LOCK(lock, txn->mutex, this_ethread()); + txn->set_stream_cleanup([this, stream_id]() { this->_streams.erase(stream_id); }); + txn->stream_closed(); + } else { + this->_streams.erase(stream_id); + } } int @@ -286,6 +303,10 @@ Http3App::_handle_uni_stream_on_read_ready(int /* event */, VIO *vio) default: break; } + + if (error && error->cls != Http3ErrorClass::UNDEFINED) { + this->_handle_error(*error); + } } void diff --git a/src/proxy/http3/Http3DebugNames.cc b/src/proxy/http3/Http3DebugNames.cc index 4208f6c4a43..1ea186fa19b 100644 --- a/src/proxy/http3/Http3DebugNames.cc +++ b/src/proxy/http3/Http3DebugNames.cc @@ -48,6 +48,8 @@ Http3DebugNames::frame_type(Http3FrameType type) return "X_RESERVED_3"; case Http3FrameType::X_RESERVED_4: return "X_RESERVED_4"; + case Http3FrameType::RESERVED: + return "RESERVED"; case Http3FrameType::UNKNOWN: default: return "UNKNOWN"; diff --git a/src/proxy/http3/Http3Frame.cc b/src/proxy/http3/Http3Frame.cc index 64b0a0a4651..6fc7d5b5441 100644 --- a/src/proxy/http3/Http3Frame.cc +++ b/src/proxy/http3/Http3Frame.cc @@ -42,6 +42,13 @@ constexpr int HEADER_OVERHEAD = 10; // This should work as long as a payloa DbgCtl dbg_ctl_http3_frame_factory{"http3_frame_factory"}; +bool +is_reserved_frame_type(uint64_t type) +{ + return type >= static_cast(Http3FrameType::RESERVED) && + (type - static_cast(Http3FrameType::RESERVED)) % 0x1f == 0; +} + } // end anonymous namespace // @@ -64,6 +71,8 @@ Http3Frame::type(const uint8_t *buf, size_t buf_len) ink_assert(ret != 1); if (type <= static_cast(Http3FrameType::X_MAX_DEFINED)) { return static_cast(type); + } else if (is_reserved_frame_type(type)) { + return Http3FrameType::RESERVED; } else { return Http3FrameType::UNKNOWN; } @@ -143,8 +152,12 @@ Http3Frame::length() const Http3FrameType Http3Frame::type() const { - if (static_cast(this->_type) <= static_cast(Http3FrameType::X_MAX_DEFINED)) { + const auto type = static_cast(this->_type); + + if (type <= static_cast(Http3FrameType::X_MAX_DEFINED)) { return this->_type; + } else if (is_reserved_frame_type(type)) { + return Http3FrameType::RESERVED; } else { return Http3FrameType::UNKNOWN; } diff --git a/src/proxy/http3/Http3HeaderVIOAdaptor.cc b/src/proxy/http3/Http3HeaderVIOAdaptor.cc index 9d3e256b7f6..7f489d4fa28 100644 --- a/src/proxy/http3/Http3HeaderVIOAdaptor.cc +++ b/src/proxy/http3/Http3HeaderVIOAdaptor.cc @@ -139,7 +139,8 @@ Http3HeaderVIOAdaptor::_on_qpack_decode_complete() // or // c). Add interface to HttpSM to handle HTTPHdr directly int bufindex; - int dumpoffset = 0; + int dumpoffset = 0; + int64_t header_length = 0; int done, tmp; IOBufferBlock *block; do { @@ -150,14 +151,19 @@ Http3HeaderVIOAdaptor::_on_qpack_decode_complete() writer->add_block(); block = writer->get_current_block(); } - done = this->_header.print(block->end(), block->write_avail(), &bufindex, &tmp); - dumpoffset += bufindex; + done = this->_header.print(block->end(), block->write_avail(), &bufindex, &tmp); + dumpoffset += bufindex; + header_length += bufindex; writer->fill(bufindex); if (!done) { writer->add_block(); } } while (!done); - this->_is_complete = true; + this->_sink_vio->ndone += header_length; + this->_is_complete = true; + if (auto *transaction = dynamic_cast(this->_txn); transaction != nullptr) { + transaction->on_header_decode_complete(); + } return 1; } diff --git a/src/proxy/http3/Http3ProtocolEnforcer.cc b/src/proxy/http3/Http3ProtocolEnforcer.cc index 9f7e8a8963a..a37cb543ae4 100644 --- a/src/proxy/http3/Http3ProtocolEnforcer.cc +++ b/src/proxy/http3/Http3ProtocolEnforcer.cc @@ -30,7 +30,7 @@ Http3ProtocolEnforcer::interests() return {Http3FrameType::DATA, Http3FrameType::HEADERS, Http3FrameType::X_RESERVED_1, Http3FrameType::CANCEL_PUSH, Http3FrameType::SETTINGS, Http3FrameType::PUSH_PROMISE, Http3FrameType::X_RESERVED_2, Http3FrameType::GOAWAY, Http3FrameType::X_RESERVED_3, Http3FrameType::X_RESERVED_4, Http3FrameType::MAX_PUSH_ID, Http3FrameType::X_MAX_DEFINED, - Http3FrameType::UNKNOWN}; + Http3FrameType::RESERVED, Http3FrameType::UNKNOWN}; } Http3ErrorUPtr @@ -47,9 +47,8 @@ Http3ProtocolEnforcer::handle_frame(std::shared_ptr frame, Htt "only one SETTINGS frame is allowed per the control stream"); } else if (f_type == Http3FrameType::DATA || f_type == Http3FrameType::HEADERS || f_type == Http3FrameType::X_RESERVED_1 || f_type == Http3FrameType::X_RESERVED_2 || f_type == Http3FrameType::X_RESERVED_3) { - std::string error_msg = Http3DebugNames::frame_type(f_type); - error_msg.append(" frame is not allowed on control stream"); - error = std::make_unique(Http3ErrorClass::CONNECTION, Http3ErrorCode::H3_FRAME_UNEXPECTED, error_msg.c_str()); + error = std::make_unique(Http3ErrorClass::CONNECTION, Http3ErrorCode::H3_FRAME_UNEXPECTED, + "frame is not allowed on control stream"); } if (!this->_is_first_frame_received_on_control) { this->_is_first_frame_received_on_control = true; @@ -57,9 +56,13 @@ Http3ProtocolEnforcer::handle_frame(std::shared_ptr frame, Htt } else { if (f_type == Http3FrameType::X_RESERVED_1 || f_type == Http3FrameType::X_RESERVED_2 || f_type == Http3FrameType::X_RESERVED_3) { - std::string error_msg = Http3DebugNames::frame_type(f_type); - error_msg.append(" frame is not allowed on any stream"); - error = std::make_unique(Http3ErrorClass::CONNECTION, Http3ErrorCode::H3_FRAME_UNEXPECTED, error_msg.c_str()); + error = std::make_unique(Http3ErrorClass::CONNECTION, Http3ErrorCode::H3_FRAME_UNEXPECTED, + "frame is not allowed on any stream"); + } else if (!this->_is_headers_frame_received && f_type == Http3FrameType::DATA) { + error = std::make_unique(Http3ErrorClass::CONNECTION, Http3ErrorCode::H3_FRAME_UNEXPECTED, + "DATA frame is not allowed before HEADERS"); + } else if (f_type == Http3FrameType::HEADERS) { + this->_is_headers_frame_received = true; } } diff --git a/src/proxy/http3/Http3Session.cc b/src/proxy/http3/Http3Session.cc index 0d33c9e63d8..c4569c16330 100644 --- a/src/proxy/http3/Http3Session.cc +++ b/src/proxy/http3/Http3Session.cc @@ -39,6 +39,8 @@ HQSession::HQSession(NetVConnection *vc) : ProxySession(vc) HQSession::~HQSession() { + this->_close_transactions(); + // Transactions should be deleted first before HQSession gets deleted. ink_assert(this->_transaction_list.head == nullptr); } @@ -59,6 +61,17 @@ HQSession::remove_transaction(HQTransaction *trans) return; } +void +HQSession::_close_transactions() +{ + while (this->_transaction_list.head != nullptr) { + auto *transaction = this->_transaction_list.head; + + transaction->do_io_close(); + delete transaction; + } +} + const char * HQSession::get_protocol_string() const { @@ -162,9 +175,11 @@ HQSession::main_event_handler(int event, void *edata) case VC_EVENT_ERROR: case VC_EVENT_EOS: this->do_io_close(); - for (HQTransaction *t = this->_transaction_list.head; t; t = static_cast(t->link.next)) { + for (HQTransaction *t = this->_transaction_list.head; t != nullptr;) { + HQTransaction *next = static_cast(t->link.next); SCOPED_MUTEX_LOCK(lock, t->mutex, this_ethread()); t->handleEvent(event, edata); + t = next; } break; } @@ -186,6 +201,7 @@ Http3Session::Http3Session(NetVConnection *vc) : HQSession(vc) Http3Session::~Http3Session() { + this->_close_transactions(); this->_vc = nullptr; delete this->_local_qpack; delete this->_remote_qpack; diff --git a/src/proxy/http3/Http3StreamDataVIOAdaptor.cc b/src/proxy/http3/Http3StreamDataVIOAdaptor.cc index 7f19c277d9a..296763972b8 100644 --- a/src/proxy/http3/Http3StreamDataVIOAdaptor.cc +++ b/src/proxy/http3/Http3StreamDataVIOAdaptor.cc @@ -24,10 +24,14 @@ #include "proxy/http3/Http3StreamDataVIOAdaptor.h" #include "iocore/eventsystem/VIO.h" -Http3StreamDataVIOAdaptor::Http3StreamDataVIOAdaptor(VIO *sink) : _sink_vio(sink), _buffer(new_MIOBuffer(BUFFER_SIZE_INDEX_4K)) {} +Http3StreamDataVIOAdaptor::Http3StreamDataVIOAdaptor(VIO *sink) : _sink_vio(sink), _buffer(new_MIOBuffer(BUFFER_SIZE_INDEX_4K)) +{ + this->_reader = this->_buffer->alloc_reader(); +} Http3StreamDataVIOAdaptor::~Http3StreamDataVIOAdaptor() { + this->_buffer->dealloc_reader(this->_reader); free_MIOBuffer(this->_buffer); } @@ -53,17 +57,17 @@ Http3StreamDataVIOAdaptor::handle_frame(std::shared_ptr frame, void Http3StreamDataVIOAdaptor::finalize() { - SCOPED_MUTEX_LOCK(lock, this->_sink_vio->mutex, this_ethread()); - MIOBuffer *writer = this->_sink_vio->get_writer(); - IOBufferReader *reader = this->_buffer->alloc_reader(); - IOBufferBlock *block; - while (reader->read_avail() > 0 && (block = reader->get_current_block()) != nullptr) { - writer->append_block(block); - reader->consume(block->size()); + if (this->_finalized) { + return; } - this->_buffer->dealloc_reader(reader); - this->_sink_vio->nbytes = this->_total_data_length; + SCOPED_MUTEX_LOCK(lock, this->_sink_vio->mutex, this_ethread()); + MIOBuffer *writer = this->_sink_vio->get_writer(); + int64_t delivered = writer->write(this->_reader, this->_reader->read_avail()); + this->_reader->consume(delivered); + this->_sink_vio->ndone += delivered; + this->_sink_vio->nbytes = this->_sink_vio->ndone; + this->_finalized = true; } bool diff --git a/src/proxy/http3/Http3Transaction.cc b/src/proxy/http3/Http3Transaction.cc index 8cdb7c3aead..9c437aea595 100644 --- a/src/proxy/http3/Http3Transaction.cc +++ b/src/proxy/http3/Http3Transaction.cc @@ -31,6 +31,7 @@ #include "proxy/http3/Http3HeaderVIOAdaptor.h" #include "proxy/http3/Http3HeaderFramer.h" #include "proxy/http3/Http3DataFramer.h" +#include "proxy/http3/Http3ProtocolEnforcer.h" #include "proxy/http/HttpSM.h" #define NetVC2QUICCon(netvc) netvc->get_service()->get_quic_connection() @@ -63,7 +64,8 @@ DbgCtl dbg_ctl_v_http3_trans{"v_http3_trans"}; // // HQTransaction // -HQTransaction::HQTransaction(HQSession *session, QUICStreamVCAdapter::IOInfo &info) : super(session), _info(info) +HQTransaction::HQTransaction(HQSession *session, QUICStreamVCAdapter::IOInfo &info) + : super(session), _info(info), _stream_id(info.adapter.stream().id()) { this->mutex = new_ProxyMutex(); this->_thread = this_ethread(); @@ -81,12 +83,16 @@ HQTransaction::~HQTransaction() this->_unschedule_write_complete_event(); static_cast(this->_proxy_ssn)->remove_transaction(this); + + if (this->_stream_cleanup) { + this->_stream_cleanup(); + } } void HQTransaction::set_active_timeout(ink_hrtime timeout_in) { - if (this->_proxy_ssn) { + if (!this->_closed && this->_proxy_ssn) { this->_proxy_ssn->set_active_timeout(timeout_in); } } @@ -94,7 +100,7 @@ HQTransaction::set_active_timeout(ink_hrtime timeout_in) void HQTransaction::set_inactivity_timeout(ink_hrtime timeout_in) { - if (this->_proxy_ssn) { + if (!this->_closed && this->_proxy_ssn) { this->_proxy_ssn->set_inactivity_timeout(timeout_in); } } @@ -102,7 +108,7 @@ HQTransaction::set_inactivity_timeout(ink_hrtime timeout_in) void HQTransaction::cancel_inactivity_timeout() { - if (this->_proxy_ssn) { + if (!this->_closed && this->_proxy_ssn) { this->_proxy_ssn->cancel_inactivity_timeout(); } } @@ -132,7 +138,7 @@ HQTransaction::do_io_read(Continuation *c, int64_t nbytes, MIOBuffer *buf) if (buf) { this->_process_read_vio(); - this->_schedule_read_ready_event(); + this->_schedule_read_event(); } return &this->_read_vio; @@ -166,6 +172,8 @@ HQTransaction::do_io_write(Continuation *c, int64_t nbytes, IOBufferReader *buf, void HQTransaction::do_io_close(int /* lerrno ATS_UNUSED */) { + this->_closed = true; + this->_read_vio.buffer.clear(); this->_read_vio.nbytes = 0; this->_read_vio.op = VIO::NONE; @@ -186,6 +194,10 @@ HQTransaction::do_io_shutdown(ShutdownHowTo_t /* howto ATS_UNUSED */) void HQTransaction::reenable(VIO *vio) { + if (this->_closed || this->_stream_closed) { + return; + } + if (vio->op == VIO::READ) { int64_t len = this->_process_read_vio(); this->_info.read_vio->reenable(); @@ -209,13 +221,28 @@ HQTransaction::transaction_done() // TODO: start closing transaction super::transaction_done(); this->_transaction_done = true; + this->_delete_if_possible(); return; } int HQTransaction::get_transaction_id() const { - return this->_info.adapter.stream().id(); + return this->_stream_id; +} + +void +HQTransaction::stream_closed() +{ + this->_stream_closed = true; + this->do_io_close(); + this->_delete_if_possible(); +} + +void +HQTransaction::set_stream_cleanup(std::function cleanup) +{ + this->_stream_cleanup = cleanup; } void @@ -276,6 +303,20 @@ HQTransaction::_schedule_read_complete_event() this->_read_complete_event = this->_thread->schedule_imm(this, VC_EVENT_READ_COMPLETE, &this->_read_vio); } +void +HQTransaction::_schedule_read_event() +{ + if (this->_read_vio.nbytes == 0) { + return; + } + + if (this->_info.read_vio->nbytes == INT64_MAX) { + this->_schedule_read_ready_event(); + } else { + this->_schedule_read_complete_event(); + } +} + void HQTransaction::_unschedule_read_complete_event() { @@ -353,17 +394,23 @@ HQTransaction::_close_write_complete_event(Event *e) } void -HQTransaction::_signal_event(int event, Event * /* edata ATS_UNUSED */) +HQTransaction::_signal_event(int event, Event *) { - // HttpSM::main_handler expects a VIO* as the event data for VC events so it - // can locate the vc_table entry. - if (this->_write_vio.cont) { - SCOPED_MUTEX_LOCK(lock, this->_write_vio.mutex, this_ethread()); - this->_write_vio.cont->handleEvent(event, &this->_write_vio); + if (this->_closed || this->_stream_closed) { + return; } - if (this->_read_vio.cont && this->_read_vio.cont != this->_write_vio.cont) { + + // HttpSM::main_handler expects a VIO* as the event data for VC events so it + // can locate the vc_table entry. Prefer the read side because H3 creates a + // zero-byte write VIO before HttpSM installs a client write handler. + if (this->_read_vio.cont && this->_read_vio.op != VIO::NONE) { SCOPED_MUTEX_LOCK(lock, this->_read_vio.mutex, this_ethread()); this->_read_vio.cont->handleEvent(event, &this->_read_vio); + return; + } + if (this->_write_vio.cont && this->_write_vio.op != VIO::NONE && this->_write_vio.nbytes > 0) { + SCOPED_MUTEX_LOCK(lock, this->_write_vio.mutex, this_ethread()); + this->_write_vio.cont->handleEvent(event, &this->_write_vio); } } @@ -373,10 +420,10 @@ HQTransaction::_signal_event(int event, Event * /* edata ATS_UNUSED */) void HQTransaction::_signal_read_event() { - if (this->_read_vio.cont == nullptr || this->_read_vio.op == VIO::NONE) { + if (this->_closed || this->_stream_closed || this->_read_vio.cont == nullptr || this->_read_vio.op == VIO::NONE) { return; } - int event = this->_read_vio.nbytes == INT64_MAX ? VC_EVENT_READ_READY : VC_EVENT_READ_COMPLETE; + int event = this->_read_vio.nbytes == INT64_MAX || this->_read_vio.ntodo() > 0 ? VC_EVENT_READ_READY : VC_EVENT_READ_COMPLETE; SCOPED_MUTEX_LOCK(lock, this->_read_vio.mutex, this_ethread()); this->_read_vio.cont->handleEvent(event, &this->_read_vio); @@ -390,9 +437,13 @@ HQTransaction::_signal_read_event() void HQTransaction::_signal_write_event() { - if (this->_write_vio.cont == nullptr || this->_write_vio.op == VIO::NONE) { + if (this->_closed || this->_stream_closed || this->_write_vio.cont == nullptr || this->_write_vio.op == VIO::NONE) { + return; + } + if (this->_write_vio.ntodo() == 0 && !this->_is_write_buffer_flushed()) { return; } + int event = this->_write_vio.ntodo() ? VC_EVENT_WRITE_READY : VC_EVENT_WRITE_COMPLETE; SCOPED_MUTEX_LOCK(lock, this->_write_vio.mutex, this_ethread()); @@ -401,6 +452,22 @@ HQTransaction::_signal_write_event() Http3TransVDebug("%s (%d)", get_vc_event_name(event), event); } +bool +HQTransaction::_is_write_buffer_flushed() +{ + if (this->_closed || this->_stream_closed) { + return true; + } + + if (this->_info.write_vio->op == VIO::NONE) { + return true; + } + + SCOPED_MUTEX_LOCK(lock, this->_info.write_vio->mutex, this_ethread()); + + return this->_info.write_vio->ntodo() == 0; +} + /** * Deletes this transaction itself. * This must be called only at the end of event handlers to avoid touching itself after deletion. @@ -408,7 +475,11 @@ HQTransaction::_signal_write_event() void HQTransaction::_delete_if_possible() { - if (this->_transaction_done) { + if (this->_event_handler_active) { + return; + } + + if (this->_transaction_done && this->_is_write_buffer_flushed() && (this->_stream_closed || !this->_info.adapter.is_readable())) { delete this; } } @@ -432,10 +503,12 @@ Http3Transaction::Http3Transaction(Http3Session *session, QUICStreamVCAdapter::I } else { http_type = HTTPType::REQUEST; } - this->_header_handler = new Http3HeaderVIOAdaptor(&this->_read_vio, http_type, session->remote_qpack(), stream_id, this); - this->_data_handler = new Http3StreamDataVIOAdaptor(&this->_read_vio); + this->_protocol_enforcer = new Http3ProtocolEnforcer(); + this->_header_handler = new Http3HeaderVIOAdaptor(&this->_read_vio, http_type, session->remote_qpack(), stream_id, this); + this->_data_handler = new Http3StreamDataVIOAdaptor(&this->_read_vio); this->_frame_dispatcher.add_handler(session->get_received_frame_counter()); + this->_frame_dispatcher.add_handler(this->_protocol_enforcer); this->_frame_dispatcher.add_handler(this->_header_handler); this->_frame_dispatcher.add_handler(this->_data_handler); @@ -453,6 +526,8 @@ Http3Transaction::~Http3Transaction() this->_header_framer = nullptr; delete this->_data_framer; this->_data_framer = nullptr; + delete this->_protocol_enforcer; + this->_protocol_enforcer = nullptr; delete this->_header_handler; this->_header_handler = nullptr; delete this->_data_handler; @@ -465,6 +540,7 @@ Http3Transaction::state_stream_open(int event, Event *edata) // TODO: should check recursive call? ink_release_assert(this->_thread == this_ethread()); SCOPED_MUTEX_LOCK(lock, this->mutex, this_ethread()); + this->_event_handler_active = true; switch (event) { case VC_EVENT_READ_READY: @@ -474,22 +550,29 @@ Http3Transaction::state_stream_open(int event, Event *edata) if (this->_process_read_vio() > 0) { this->_signal_read_event(); } - this->_info.read_vio->reenable(); + if (!this->_closed) { + this->_info.read_vio->reenable(); + } break; - case VC_EVENT_READ_COMPLETE: + case VC_EVENT_READ_COMPLETE: { Http3TransVDebug("%s (%d)", get_vc_event_name(event), event); this->_close_read_complete_event(edata); - this->_process_read_vio(); + int64_t nread = this->_process_read_vio(); if (!this->_header_handler->is_complete()) { - // Delay processing READ_COMPLETE - this->_schedule_read_complete_event(); + if (nread > 0) { + // Delay processing READ_COMPLETE until the header block can be fully decoded. + this->_schedule_read_complete_event(); + } break; } this->_data_handler->finalize(); // always signal regardless of progress this->_signal_read_event(); - this->_info.read_vio->reenable(); + if (!this->_closed) { + this->_info.read_vio->reenable(); + } break; + } case VC_EVENT_WRITE_READY: this->_close_write_ready_event(edata); Http3TransVDebug("%s (%d)", get_vc_event_name(event), event); @@ -497,7 +580,9 @@ Http3Transaction::state_stream_open(int event, Event *edata) if (this->_process_write_vio() > 0) { this->_signal_write_event(); } - this->_info.write_vio->reenable(); + if (!this->_closed) { + this->_info.write_vio->reenable(); + } break; case VC_EVENT_WRITE_COMPLETE: this->_close_write_complete_event(edata); @@ -505,7 +590,9 @@ Http3Transaction::state_stream_open(int event, Event *edata) this->_process_write_vio(); // always signal regardless of progress this->_signal_write_event(); - this->_info.write_vio->reenable(); + if (!this->_closed) { + this->_info.write_vio->reenable(); + } break; case VC_EVENT_EOS: case VC_EVENT_ERROR: @@ -513,12 +600,14 @@ Http3Transaction::state_stream_open(int event, Event *edata) case VC_EVENT_ACTIVE_TIMEOUT: { Http3TransVDebug("%s (%d)", get_vc_event_name(event), event); this->_signal_event(event, edata); + this->do_io_close(); break; } default: Http3TransDebug("Unknown event %d", event); } + this->_event_handler_active = false; this->_delete_if_possible(); return EVENT_DONE; } @@ -527,6 +616,7 @@ int Http3Transaction::state_stream_closed(int event, Event *data) { Http3TransVDebug("%s (%d)", get_vc_event_name(event), event); + this->_event_handler_active = true; switch (event) { case VC_EVENT_READ_READY: @@ -553,6 +643,7 @@ Http3Transaction::state_stream_closed(int event, Event *data) Http3TransDebug("Unknown event %d", event); } + this->_event_handler_active = false; this->_delete_if_possible(); return EVENT_DONE; } @@ -564,6 +655,12 @@ Http3Transaction::do_io_close(int lerrno) super::do_io_close(lerrno); } +void +Http3Transaction::on_header_decode_complete() +{ + this->_schedule_read_event(); +} + bool Http3Transaction::is_response_header_sent() const { @@ -576,9 +673,26 @@ Http3Transaction::is_response_body_sent() const return this->_data_framer->is_done(); } +void +Http3Transaction::_handle_error(const Http3Error &error) +{ + if (error.cls == Http3ErrorClass::CONNECTION) { + this->do_io_close(); + this->_transaction_done = true; + this->_stream_closed = true; + NetVC2QUICCon(this->_proxy_ssn->get_netvc()) + ->close_quic_connection( + std::make_unique(QUICErrorClass::APPLICATION, static_cast(error.code))); + } +} + int64_t Http3Transaction::_process_read_vio() { + if (this->_stream_closed) { + return 0; + } + if (this->_info.read_vio->cont == nullptr || this->_info.read_vio->op == VIO::NONE) { return 0; } @@ -590,7 +704,8 @@ Http3Transaction::_process_read_vio() auto error = this->_frame_dispatcher.on_read_ready(this->_info.adapter.stream().id(), Http3StreamType::UNKNOWN, *this->_info.read_vio->get_reader(), nread); if (error && error->cls != Http3ErrorClass::UNDEFINED) { - Http3TransDebug("Error occurred while processing read vio: %hu, %s", error->get_code(), error->msg); + Http3TransDebug("Error occurred while processing read vio: %hu", error->get_code()); + this->_handle_error(*error); return 0; } this->_info.read_vio->ndone += nread; @@ -600,6 +715,10 @@ Http3Transaction::_process_read_vio() int64_t Http3Transaction::_process_write_vio() { + if (this->_stream_closed) { + return 0; + } + if (this->_info.write_vio->cont == nullptr || this->_info.write_vio->op == VIO::NONE) { return 0; } @@ -612,7 +731,7 @@ Http3Transaction::_process_write_vio() auto error = this->_frame_collector.on_write_ready(this->_info.adapter.stream().id(), *this->_info.write_vio->get_writer(), nwritten, all_done); if (error && error->cls != Http3ErrorClass::UNDEFINED) { - Http3TransDebug("Error occured while processing write vio: %hu, %s", error->get_code(), error->msg); + Http3TransDebug("Error occurred while processing write vio: %hu", error->get_code()); return 0; } this->_sent_bytes += nwritten; @@ -636,6 +755,10 @@ Http3Transaction::has_request_body(int64_t content_length, bool /* is_chunked_se return true; } + if (this->_stream_closed) { + return false; + } + // No body if stream is already closed and DATA frame is not received yet if (this->_info.adapter.stream().has_no_more_data()) { return false; @@ -669,6 +792,7 @@ Http09Transaction::state_stream_open(int event, Event *edata) ink_release_assert(this->_thread == this_ethread()); SCOPED_MUTEX_LOCK(lock, this->mutex, this_ethread()); + this->_event_handler_active = true; switch (event) { case VC_EVENT_READ_READY: @@ -706,12 +830,15 @@ Http09Transaction::state_stream_open(int event, Event *edata) case VC_EVENT_INACTIVITY_TIMEOUT: case VC_EVENT_ACTIVE_TIMEOUT: { Http3TransDebug("%d", event); + this->_signal_event(event, edata); + this->do_io_close(); break; } default: Http3TransDebug("Unknown event %d", event); } + this->_event_handler_active = false; this->_delete_if_possible(); return EVENT_DONE; } @@ -727,6 +854,7 @@ int Http09Transaction::state_stream_closed(int event, Event *data) { Http3TransVDebug("%s (%d)", get_vc_event_name(event), event); + this->_event_handler_active = true; switch (event) { case VC_EVENT_READ_READY: @@ -752,6 +880,7 @@ Http09Transaction::state_stream_closed(int event, Event *data) Http3TransDebug("Unknown event %d", event); } + this->_event_handler_active = false; this->_delete_if_possible(); return EVENT_DONE; } @@ -760,6 +889,10 @@ Http09Transaction::state_stream_closed(int event, Event *data) int64_t Http09Transaction::_process_read_vio() { + if (this->_stream_closed) { + return 0; + } + if (this->_read_vio.cont == nullptr || this->_read_vio.op == VIO::NONE) { return 0; } @@ -840,6 +973,10 @@ static constexpr char http_1_1_version[] = "HTTP/1.1"; int64_t Http09Transaction::_process_write_vio() { + if (this->_stream_closed) { + return 0; + } + if (this->_write_vio.cont == nullptr || this->_write_vio.op == VIO::NONE) { return 0; } diff --git a/src/proxy/http3/QPACK.cc b/src/proxy/http3/QPACK.cc index dfdd2d278b3..bcfa4c70b8d 100644 --- a/src/proxy/http3/QPACK.cc +++ b/src/proxy/http3/QPACK.cc @@ -69,7 +69,7 @@ const QPACK::Header QPACK::StaticTable::STATIC_HEADER_FIELDS[] = { {":status", "503" }, {"accept", "*/*" }, {"accept", "application/dns-message" }, - {"accept-encoding", "gzip, deflate, br, zstd" }, + {"accept-encoding", "gzip, deflate, br" }, {"accept-ranges", "bytes" }, {"access-control-allow-headers", "cache-control" }, {"access-control-allow-headers", "content-type" }, @@ -80,7 +80,6 @@ const QPACK::Header QPACK::StaticTable::STATIC_HEADER_FIELDS[] = { {"cache-control", "no-cache" }, {"cache-control", "no-store" }, {"cache-control", "public, max-age=31536000" }, - {"content-encoding", "zstd" }, {"content-encoding", "br" }, {"content-encoding", "gzip" }, {"content-type", "application/dns-message" }, diff --git a/src/proxy/http3/test/test_Http3FrameDispatcher.cc b/src/proxy/http3/test/test_Http3FrameDispatcher.cc index 0c7a65b749c..37560e9a157 100644 --- a/src/proxy/http3/test/test_Http3FrameDispatcher.cc +++ b/src/proxy/http3/test/test_Http3FrameDispatcher.cc @@ -231,7 +231,7 @@ TEST_CASE("control stream tests", "[http3]") CHECK(nread == sizeof(input)); } - SECTION("RESERVED frame is not allowed on control stream") + SECTION("HTTP/2 reserved frame is not allowed on control stream") { uint8_t input[] = {0x04, // Type 0x08, // Length @@ -258,6 +258,32 @@ TEST_CASE("control stream tests", "[http3]") CHECK(nread == sizeof(input)); } + SECTION("GREASE reserved frame is ignored on control stream") + { + uint8_t input[] = {0x04, // Type + 0x08, // Length + 0x06, // Identifier + 0x44, 0x00, // Value + 0x09, // Identifier + 0x0f, // Value + 0x4a, 0x0a, // Identifier + 0x00, // Value + 0x21, // Type: reserved by the 0x21 + 0x1f * N pattern + 0x04, // Length + 0x11, 0x22, 0x33, 0x44}; + + buf->write(input, sizeof(input)); + + // Initial state + CHECK(handler.total_frame_received == 0); + CHECK(nread == 0); + + error = http3FrameDispatcher.on_read_ready(0, Http3StreamType::CONTROL, *reader, nread); + CHECK(!error); + CHECK(handler.total_frame_received == 1); + CHECK(nread == sizeof(input)); + } + SECTION("padding should not be interpreted as a DATA frame", "[http3]") { uint8_t input[] = { @@ -311,7 +337,7 @@ TEST_CASE("ignore unknown frames", "[http3]") } } -TEST_CASE("Reserved frame type not allowed", "[http3]") +TEST_CASE("HTTP/2 reserved frame type not allowed", "[http3]") { SECTION("Reject reserved frame type in non control stream") { diff --git a/src/proxy/http3/test/test_QPACK.cc b/src/proxy/http3/test/test_QPACK.cc index 438571d1149..338edeea583 100644 --- a/src/proxy/http3/test/test_QPACK.cc +++ b/src/proxy/http3/test/test_QPACK.cc @@ -85,7 +85,9 @@ class TestQUICStream : public QUICStream auto ibb = this->_adapter->read(buf_len); IOBufferReader reader; reader.block = ibb; - return reader.read(buf, buf_len); + auto nread = reader.read(buf, buf_len); + this->_adapter->consume(nread); + return nread; } }; diff --git a/tests/gold_tests/autest-site/ats_replay.test.ext b/tests/gold_tests/autest-site/ats_replay.test.ext index d6379025b20..4557cf2b082 100644 --- a/tests/gold_tests/autest-site/ats_replay.test.ext +++ b/tests/gold_tests/autest-site/ats_replay.test.ext @@ -35,6 +35,8 @@ def configure_ats(obj: 'TestRun', server: 'Process', ats_config: dict, dns: Opti name = ats_config.get('name', 'ts') process_config = ats_config.get('process_config', {}) ts = obj.MakeATSProcess(name, **process_config) + if 'startup_timeout' in ats_config: + ts.StartupTimeout = ats_config['startup_timeout'] # Configure records_config if specified. records_config = ats_config.get('records_config', {}) @@ -273,6 +275,7 @@ def ATSReplayTest(obj, replay_file: str): ats_config = autest_config['ats'] process_config = ats_config.get('process_config', {}) enable_tls = process_config.get('enable_tls', False) + enable_quic = process_config.get('enable_quic', False) metric_checks = ats_config.get('metric_checks', []) log_validation = ats_config.get('log_validation', None) @@ -286,8 +289,9 @@ def ATSReplayTest(obj, replay_file: str): name = client_config.get('name', 'client') process_config = client_config.get('process_config', {}) https_ports = [ts.Variables.ssl_port] if enable_tls else None + http3_ports = [ts.Variables.ssl_port] if enable_quic else None client = tr.AddVerifierClientProcess( - name, replay_file, http_ports=[ts.Variables.port], https_ports=https_ports, **process_config) + name, replay_file, http_ports=[ts.Variables.port], https_ports=https_ports, http3_ports=http3_ports, **process_config) # Set expected return code if specified. A list of codes is wrapped in # Any() so that any of the listed values is accepted. diff --git a/tests/gold_tests/autest-site/conditions.test.ext b/tests/gold_tests/autest-site/conditions.test.ext index 41c5a6cacdf..090075c4ff8 100644 --- a/tests/gold_tests/autest-site/conditions.test.ext +++ b/tests/gold_tests/autest-site/conditions.test.ext @@ -33,6 +33,12 @@ OPENSSL_TLS_FLAGS = { } +def _version_tuple(value, width=3): + parts = [int(part) for part in re.findall(r'\d+', value)[:width]] + parts.extend([0] * (width - len(parts))) + return tuple(parts) + + def _terminate_process(process): if process.poll() is not None: return @@ -130,10 +136,27 @@ def HasOpenSSLVersion(self, version): output = subprocess.check_output(os.path.join(self.Variables.BINDIR, "traffic_layout") + " info --versions --json", shell=True) json_data = output.decode('utf-8') openssl_str = json.loads(json_data)['openssl_str'] - exe_ver = re.search(r'\d\.\d\.\d', openssl_str).group(0) - if exe_ver == '': + match = re.search(r'\d+(?:\.\d+)+', openssl_str) + if match is None: raise ValueError("Error determining version of OpenSSL library needed by traffic_server executable") - return self.Condition(lambda: exe_ver >= version, "OpenSSL library version is " + exe_ver + ", must be at least " + version) + exe_ver = match.group(0) + return self.Condition( + lambda: _version_tuple(exe_ver) >= _version_tuple(version), + "OpenSSL library version is " + exe_ver + ", must be at least " + version) + + +def HasOpenSSLQuicClient(self): + """Check whether the openssl CLI supports s_client -quic.""" + + def check_openssl_quic_client(): + try: + result = subprocess.run(["openssl", "s_client", "-help"], capture_output=True, text=True, timeout=5) + except (OSError, subprocess.SubprocessError): + return False + + return "-quic" in result.stdout or "-quic" in result.stderr + + return self.Condition(check_openssl_quic_client, "OpenSSL CLI must support s_client -quic") def IsBoringSSL(self): @@ -217,6 +240,26 @@ def HasProxyVerifierVersion(self, version): return self.EnsureVersion([verifier_path, "--version"], min_version=version) +def HasGoVersion(self, version): + """Check whether the go command is available at the requested version.""" + + def check_go_version(): + try: + output = subprocess.check_output(["go", "version"], stderr=subprocess.STDOUT, text=True) + except (OSError, subprocess.SubprocessError): + return False + + match = re.search(r'\bgo(\d+\.\d+(?:\.\d+)?)\b', output) + if match is None: + return False + + found = _version_tuple(match.group(1)) + required = _version_tuple(version) + return found >= required + + return self.Condition(check_go_version, "Go must be installed and at least version " + version) + + def HasCurlFeature(self, feature): def default(output): @@ -325,7 +368,9 @@ def CurlUsingUnixDomainSocket(self): ExtendCondition(HasOpenSSLVersion) +ExtendCondition(HasOpenSSLQuicClient) ExtendCondition(HasProxyVerifierVersion) +ExtendCondition(HasGoVersion) ExtendCondition(IsBoringSSL) ExtendCondition(IsOpenSSL) ExtendCondition(HasLegacyTLSSupport) diff --git a/tests/gold_tests/early_hints/early_hints.test.py b/tests/gold_tests/early_hints/early_hints.test.py index 0f646624692..ddcc488be88 100644 --- a/tests/gold_tests/early_hints/early_hints.test.py +++ b/tests/gold_tests/early_hints/early_hints.test.py @@ -28,6 +28,7 @@ class Protocol(Enum): HTTP = auto() HTTPS = auto() HTTP2 = auto() + HTTP3 = auto() @classmethod def to_string(cls, protocol): @@ -37,6 +38,8 @@ def to_string(cls, protocol): return 'HTTPS' elif protocol == cls.HTTP2: return 'HTTP2' + elif protocol == cls.HTTP3: + return 'HTTP3' else: return None @@ -87,7 +90,7 @@ def _configure_ts(self, tr: 'TestRun'): :param tr: The TestRun for the traffic server. ''' - ts = Test.MakeATSProcess(f'ts_{self._protocol_str}', enable_tls=True) + ts = Test.MakeATSProcess(f'ts_{self._protocol_str}', enable_tls=True, enable_quic=self._protocol == Protocol.HTTP3) self._ts = ts ts.Disk.remap_config.AddLine(f'map / http://backend.server.com:{self._server.Variables.http_port}') ts.addDefaultSSLFiles() @@ -137,6 +140,10 @@ def _configure_client(self, tr: 'TestRun'): protocol_arg = '-k --http2' scheme = 'https' ts_port = self._ts.Variables.ssl_port + elif self._protocol == Protocol.HTTP3: + protocol_arg = '-k --http3-only' + scheme = 'https' + ts_port = self._ts.Variables.ssl_port tr.MakeCurlCommand( f'-v {protocol_arg} ' f'--resolve "server.com:{ts_port}:127.0.0.1" ' @@ -164,3 +171,5 @@ def _configure_client(self, tr: 'TestRun'): if not Condition.CurlUsingUnixDomainSocket(): TestEarlyHints(Protocol.HTTPS) TestEarlyHints(Protocol.HTTP2) + if Condition.HasATSFeature('TS_USE_QUIC') and Condition.HasCurlFeature('http3') and Condition.HasCurlOption('--http3-only'): + TestEarlyHints(Protocol.HTTP3) diff --git a/tests/gold_tests/h3/go_h3_client/go.mod b/tests/gold_tests/h3/go_h3_client/go.mod new file mode 100644 index 00000000000..9f8dd05a0be --- /dev/null +++ b/tests/gold_tests/h3/go_h3_client/go.mod @@ -0,0 +1,13 @@ +module trafficserver.apache.org/h3-go-client + +go 1.24 + +require github.com/quic-go/quic-go v0.59.1 + +require ( + github.com/quic-go/qpack v0.6.0 // indirect + golang.org/x/crypto v0.41.0 // indirect + golang.org/x/net v0.43.0 // indirect + golang.org/x/sys v0.35.0 // indirect + golang.org/x/text v0.28.0 // indirect +) diff --git a/tests/gold_tests/h3/go_h3_client/go.sum b/tests/gold_tests/h3/go_h3_client/go.sum new file mode 100644 index 00000000000..314cb107874 --- /dev/null +++ b/tests/gold_tests/h3/go_h3_client/go.sum @@ -0,0 +1,38 @@ +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/jordanlewis/gcassert v0.0.0-20250430164644-389ef753e22e h1:a+PGEeXb+exwBS3NboqXHyxarD9kaboBbrSp+7GuBuc= +github.com/jordanlewis/gcassert v0.0.0-20250430164644-389ef753e22e/go.mod h1:ZybsQk6DWyN5t7An1MuPm1gtSZ1xDaTXS9ZjIOxvQrk= +github.com/kr/pretty v0.3.1 h1:flRD4NNwYAUpkphVc1HcthR4KEIFJ65n8Mw5qdRn3LE= +github.com/kr/pretty v0.3.1/go.mod h1:hoEshYVHaxMs3cyo3Yncou5ZscifuDolrwPKZanG3xk= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/quic-go/qpack v0.6.0 h1:g7W+BMYynC1LbYLSqRt8PBg5Tgwxn214ZZR34VIOjz8= +github.com/quic-go/qpack v0.6.0/go.mod h1:lUpLKChi8njB4ty2bFLX2x4gzDqXwUpaO1DP9qMDZII= +github.com/quic-go/quic-go v0.59.1 h1:0Gmua0HW1Tv7ANR7hUYwRyD0MG5OJfgvYSZasGZzBic= +github.com/quic-go/quic-go v0.59.1/go.mod h1:upnsH4Ju1YkqpLXC305eW3yDZ4NfnNbmQRCMWS58IKU= +github.com/rogpeppe/go-internal v1.10.0 h1:TMyTOH3F/DB16zRVcYyreMH6GnZZrwQVAoYjRBZyWFQ= +github.com/rogpeppe/go-internal v1.10.0/go.mod h1:UQnix2H7Ngw/k4C5ijL5+65zddjncjaFoBhdsK/akog= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.uber.org/mock v0.5.2 h1:LbtPTcP8A5k9WPXj54PPPbjcI4Y6lhyOZXn+VS7wNko= +go.uber.org/mock v0.5.2/go.mod h1:wLlUxC2vVTPTaE3UD51E0BGOAElKrILxhVSDYQLld5o= +golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4= +golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc= +golang.org/x/mod v0.27.0 h1:kb+q2PyFnEADO2IEF935ehFUXlWiNjJWtRNgBLSfbxQ= +golang.org/x/mod v0.27.0/go.mod h1:rWI627Fq0DEoudcK+MBkNkCe0EetEaDSwJJkCcjpazc= +golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE= +golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg= +golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw= +golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA= +golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI= +golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k= +golang.org/x/term v0.34.0 h1:O/2T7POpk0ZZ7MAzMeWFSg6S5IpWd/RXDlM9hgM3DR4= +golang.org/x/term v0.34.0/go.mod h1:5jC53AEywhIVebHgPVeg0mj8OD3VO9OzclacVrqpaAw= +golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng= +golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU= +golang.org/x/tools v0.36.0 h1:kWS0uv/zsvHEle1LbV5LE8QujrxB3wfQyxHfhOk0Qkg= +golang.org/x/tools v0.36.0/go.mod h1:WBDiHKJK8YgLHlcQPYQzNCkUxUypCaa5ZegCVutKm+s= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c h1:Hei/4ADfdWqJk1ZMxUNpqntNwaWcugrBjAiHlqqRiVk= +gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c/go.mod h1:JHkPIbrfpd72SG/EVd6muEfDQjcINNoR0C8j2r3qZ4Q= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/tests/gold_tests/h3/go_h3_client/main.go b/tests/gold_tests/h3/go_h3_client/main.go new file mode 100644 index 00000000000..8e73127a333 --- /dev/null +++ b/tests/gold_tests/h3/go_h3_client/main.go @@ -0,0 +1,300 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor license agreements. See the NOTICE file +// distributed with this work for additional information regarding +// copyright ownership. The ASF licenses this file to you under +// the Apache License, Version 2.0 (the "License"); you may not +// use this file except in compliance with the License. You may +// obtain a copy of the License at +// +// http://www.apache.org/licenses/LICENSE-2.0 +// +// Unless required by applicable law or agreed to in writing, software +// distributed under the License is distributed on an "AS IS" BASIS, +// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +// See the License for the specific language governing permissions and +// limitations under the License. + +package main + +import ( + "bytes" + "context" + "crypto/tls" + "flag" + "fmt" + "io" + "net/http" + "os" + "sync" + "time" + + "github.com/quic-go/quic-go" + "github.com/quic-go/quic-go/http3" +) + +const ( + largeBodySize = 300000 + largeBodySuffix = "000927b " + reusedHeaderValue = "stable-qpack-value" +) + +type requestCase struct { + name string + method string + path string + requestSize int + responseSize int + status int +} + +func generatedBody(size int) []byte { + var body bytes.Buffer + for i := 0; body.Len() < size; i++ { + fmt.Fprintf(&body, "%07x ", i) + } + return body.Bytes()[:size] +} + +func newTLSConfig(serverName string) *tls.Config { + return &tls.Config{ + InsecureSkipVerify: true, + NextProtos: []string{http3.NextProtoH3}, + ServerName: serverName, + } +} + +func newQUICConfig() *quic.Config { + return &quic.Config{ + MaxIdleTimeout: 10 * time.Second, + } +} + +func newClient(serverName string) (*http.Client, *http3.Transport) { + transport := &http3.Transport{ + TLSClientConfig: newTLSConfig(serverName), + QUICConfig: newQUICConfig(), + DisableCompression: true, + } + client := &http.Client{ + Transport: transport, + Timeout: 30 * time.Second, + } + return client, transport +} + +func newRequest(ctx context.Context, baseURL string, authority string, tc requestCase) (*http.Request, error) { + var body io.Reader + if tc.requestSize > 0 { + body = bytes.NewReader(generatedBody(tc.requestSize)) + } + + req, err := http.NewRequestWithContext(ctx, tc.method, baseURL+tc.path, body) + if err != nil { + return nil, err + } + + req.Host = authority + req.Header.Set("User-Agent", "ats-h3-quic-go-autest") + req.Header.Set("X-H3-Go-Client", "quic-go") + req.Header.Set("X-H3-Reused-Header", reusedHeaderValue) + req.Header.Set("X-H3-Test-Case", tc.name) + req.Header.Set("uuid", tc.name) + if tc.requestSize > 0 { + req.Header.Set("Content-Type", "application/octet-stream") + } + + return req, nil +} + +func verifyResponse(tc requestCase, resp *http.Response) error { + defer resp.Body.Close() + + if resp.ProtoMajor != 3 { + return fmt.Errorf("%s: expected HTTP/3, got %s", tc.name, resp.Proto) + } + if resp.StatusCode != tc.status { + return fmt.Errorf("%s: expected status %d, got %d", tc.name, tc.status, resp.StatusCode) + } + + body, err := io.ReadAll(resp.Body) + if err != nil { + return fmt.Errorf("%s: read response body: %w", tc.name, err) + } + + if tc.method == http.MethodHead || tc.status == http.StatusNoContent { + if len(body) != 0 { + return fmt.Errorf("%s: expected no response body, got %d bytes", tc.name, len(body)) + } + return nil + } + + expected := generatedBody(tc.responseSize) + if !bytes.Equal(body, expected) { + return fmt.Errorf("%s: response body mismatch: got %d bytes, expected %d", tc.name, len(body), len(expected)) + } + if tc.responseSize == largeBodySize && !bytes.HasSuffix(body, []byte(largeBodySuffix)) { + return fmt.Errorf("%s: large response body does not end with %q", tc.name, largeBodySuffix) + } + + return nil +} + +func doRequest( + ctx context.Context, + roundTrip func(*http.Request) (*http.Response, error), + baseURL string, + authority string, + tc requestCase, +) error { + req, err := newRequest(ctx, baseURL, authority, tc) + if err != nil { + return err + } + + resp, err := roundTrip(req) + if err != nil { + return fmt.Errorf("%s: request failed: %w", tc.name, err) + } + + if err := verifyResponse(tc, resp); err != nil { + return err + } + + fmt.Printf("ok %s\n", tc.name) + return nil +} + +func runSequential(ctx context.Context, baseURL string, authority string, serverName string, cases []requestCase) error { + client, transport := newClient(serverName) + defer transport.Close() + + for _, tc := range cases { + if err := doRequest(ctx, client.Do, baseURL, authority, tc); err != nil { + return err + } + } + + return nil +} + +func runConcurrent(ctx context.Context, addr string, baseURL string, authority string, serverName string, cases []requestCase) error { + transport := &http3.Transport{} + defer transport.Close() + + conn, err := quic.DialAddr(ctx, addr, newTLSConfig(serverName), newQUICConfig()) + if err != nil { + return fmt.Errorf("dial concurrent HTTP/3 connection: %w", err) + } + clientConn := transport.NewClientConn(conn) + defer clientConn.CloseWithError(0, "done") + + var wg sync.WaitGroup + errs := make(chan error, len(cases)) + for _, tc := range cases { + tc := tc + wg.Add(1) + go func() { + defer wg.Done() + errs <- doRequest(ctx, clientConn.RoundTrip, baseURL, authority, tc) + }() + } + + wg.Wait() + close(errs) + for err := range errs { + if err != nil { + return err + } + } + + return nil +} + +func main() { + addr := flag.String("addr", "", "ATS HTTP/3 address in host:port form") + authority := flag.String("authority", "", "HTTP/3 request authority") + serverName := flag.String("server-name", "", "TLS SNI server name") + flag.Parse() + + if *addr == "" || *authority == "" || *serverName == "" { + flag.Usage() + os.Exit(2) + } + + baseURL := "https://" + *addr + ctx := context.Background() + sequentialCases := []requestCase{ + {name: "go-get-empty", method: http.MethodGet, path: "/go-get-empty", status: http.StatusOK}, + {name: "go-get-small", method: http.MethodGet, path: "/go-get-small", responseSize: 100, status: http.StatusOK}, + {name: "go-head-no-body", method: http.MethodHead, path: "/go-head-no-body", responseSize: 100, status: http.StatusOK}, + {name: "go-204-no-body", method: http.MethodGet, path: "/go-204-no-body", status: http.StatusNoContent}, + { + name: "go-post-small", + method: http.MethodPost, + path: "/go-post-small", + requestSize: 100, + responseSize: 100, + status: http.StatusOK, + }, + { + name: "go-put-small", + method: http.MethodPut, + path: "/go-put-small", + requestSize: 100, + responseSize: 100, + status: http.StatusOK, + }, + {name: "go-delete-empty", method: http.MethodDelete, path: "/go-delete-empty", status: http.StatusNoContent}, + {name: "go-options-small", method: http.MethodOptions, path: "/go-options-small", responseSize: 100, status: http.StatusOK}, + } + concurrentCases := []requestCase{ + { + name: "go-get-concurrent-large", + method: http.MethodGet, + path: "/go-get-concurrent-large", + responseSize: largeBodySize, + status: http.StatusOK, + }, + { + name: "go-get-concurrent-small", + method: http.MethodGet, + path: "/go-get-concurrent-small", + responseSize: 100, + status: http.StatusOK, + }, + } + largeCases := []requestCase{ + {name: "go-get-large", method: http.MethodGet, path: "/go-get-large", responseSize: largeBodySize, status: http.StatusOK}, + { + name: "go-post-large", + method: http.MethodPost, + path: "/go-post-large", + requestSize: largeBodySize, + responseSize: largeBodySize, + status: http.StatusOK, + }, + { + name: "go-put-large", + method: http.MethodPut, + path: "/go-put-large", + requestSize: largeBodySize, + responseSize: largeBodySize, + status: http.StatusOK, + }, + } + + if err := runSequential(ctx, baseURL, *authority, *serverName, sequentialCases); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + if err := runConcurrent(ctx, *addr, baseURL, *authority, *serverName, concurrentCases); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + if err := runSequential(ctx, baseURL, *authority, *serverName, largeCases); err != nil { + fmt.Fprintln(os.Stderr, err) + os.Exit(1) + } + + fmt.Println("completed 13 HTTP/3 requests") +} diff --git a/tests/gold_tests/h3/h3_active_timeout.test.py b/tests/gold_tests/h3/h3_active_timeout.test.py new file mode 100644 index 00000000000..9a159fb49c6 --- /dev/null +++ b/tests/gold_tests/h3/h3_active_timeout.test.py @@ -0,0 +1,26 @@ +''' +Verify HTTP/3 transaction cleanup on active timeout. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify HTTP/3 transactions are removed cleanly after transaction active timeout. +''' + +Test.SkipUnless(Condition.HasATSFeature('TS_USE_QUIC')) + +Test.ATSReplayTest(replay_file="replays/h3_active_timeout.replay.yaml") diff --git a/tests/gold_tests/h3/h3_curl.test.py b/tests/gold_tests/h3/h3_curl.test.py new file mode 100644 index 00000000000..0fffd6b3d10 --- /dev/null +++ b/tests/gold_tests/h3/h3_curl.test.py @@ -0,0 +1,143 @@ +''' +Verify HTTP/3 client interop with curl. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +Test.Summary = ''' +This test is written specifically to verify that an HTTP/3 curl client can +complete a request through ATS. +''' + +Test.SkipUnless( + Condition.HasATSFeature('TS_USE_QUIC'), + Condition.HasCurlFeature('http3'), + Condition.HasCurlOption('--http3-only'), +) +Test.SkipIf(Condition.CurlUsingUnixDomainSocket()) + + +class TestHttp3Curl: + """Configure a test to verify HTTP/3 curl client interoperability.""" + + response_body = "0123456789" * 30000 + + def __init__(self, name: str): + """Initialize the test. + + :param name: The name of the test. + """ + self.name = name + self._body_path = os.path.join(Test.RunDirectory, "h3_curl_body.txt") + self._configure_server() + self._configure_traffic_server() + self._configure_client() + + def _configure_server(self): + """Configure the origin server.""" + server = Test.MakeOriginServer("server") + server.addResponse( + "sessionlog.json", { + "headers": "GET /h3-curl HTTP/1.1\r\nHost: localhost\r\n\r\n", + "timestamp": "1469733493.993", + "body": "" + }, { + "headers": f"HTTP/1.1 200 OK\r\nConnection: close\r\nContent-Length: {len(self.response_body)}\r\n\r\n", + "timestamp": "1469733493.993", + "body": self.response_body + }) + + self._server = server + + def _configure_traffic_server(self): + """Configure Traffic Server.""" + ts = Test.MakeATSProcess("ts", enable_tls=True, enable_quic=True, enable_cache=False) + ts.StartupTimeout = 60 + ts.addDefaultSSLFiles() + ts.addSSLfile("../tls/ssl/signed-foo.pem") + ts.addSSLfile("../tls/ssl/signed-foo.key") + ts.Disk.ssl_multicert_yaml.AddLines( + ''' +ssl_multicert: + - ssl_cert_name: signed-foo.pem + ssl_key_name: signed-foo.key + - ssl_cert_name: server.pem + ssl_key_name: server.key + dest_ip: "*" +'''.split('\n')) + ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'quic|http3', + 'proxy.config.quic.server.stateless_retry_enabled': 0, + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + }) + ts.Disk.remap_config.AddLine(f'map / http://127.0.0.1:{self._server.Variables.Port}') + ts.Disk.logging_yaml.AddLines( + ''' +logging: + formats: + - name: h3_access + format: 'c_alpn=% client_version=% c_ssl_version=% c_method=% c_url=%' + + logs: + - filename: h3_access + format: h3_access +'''.split("\n")) + + self._access_log = Test.Disk.File(os.path.join(ts.Variables.LOGDIR, 'h3_access.log'), exists=True) + self._access_log.Content = Testers.ContainsExpression( + r'c_alpn=h3 client_version=http/3 c_ssl_version=[^ ]+ c_method=GET c_url=https://foo.com:[0-9]+/h3-curl', + "ATS should log the curl request as HTTP/3") + + self._ts = ts + + def _check_curl_response(self, tr): + """Verify that curl received the response over HTTP/3.""" + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.stdout = Testers.ContainsExpression( + f"size_download={len(self.response_body)}", "curl should receive the complete HTTP/3 response body") + tr.Processes.Default.Streams.stdout += Testers.ContainsExpression("http_version=3", "curl should report HTTP/3") + + def _configure_client(self): + """Configure the curl client test runs.""" + tr = Test.AddTestRun(self.name) + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._ts) + tr.MakeCurlCommand( + '--silent --show-error --fail --ipv4 --http3-only --insecure ' + f'--resolve "foo.com:{self._ts.Variables.ssl_port}:127.0.0.1" ' + f'--output "{self._body_path}" ' + '--write-out "\\nhttp_version=%{http_version}\\nsize_download=%{size_download}\\n" ' + f'https://foo.com:{self._ts.Variables.ssl_port}/h3-curl', + ts=self._ts) + self._check_curl_response(tr) + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + tr = Test.AddTestRun("Wait for HTTP/3 access log") + tr.Processes.Default.Command = ( + os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + + os.path.join(self._ts.Variables.LOGDIR, 'h3_access.log')) + tr.Processes.Default.ReturnCode = 0 + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + +TestHttp3Curl("curl forced HTTP/3 request") diff --git a/tests/gold_tests/h3/h3_flow_control.test.py b/tests/gold_tests/h3/h3_flow_control.test.py new file mode 100644 index 00000000000..85c4c5b5573 --- /dev/null +++ b/tests/gold_tests/h3/h3_flow_control.test.py @@ -0,0 +1,27 @@ +''' +Verify HTTP/3 traffic progresses with small QUIC flow-control windows. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify that large HTTP/3 request and response bodies complete when ATS +advertises small initial QUIC flow-control windows. +''' + +Test.SkipUnless(Condition.HasATSFeature('TS_USE_QUIC')) + +Test.ATSReplayTest(replay_file="replays/h3_flow_control.replay.yaml") diff --git a/tests/gold_tests/h3/h3_go_client.test.py b/tests/gold_tests/h3/h3_go_client.test.py new file mode 100644 index 00000000000..d99c1979e1e --- /dev/null +++ b/tests/gold_tests/h3/h3_go_client.test.py @@ -0,0 +1,130 @@ +''' +Verify HTTP/3 client interop with a quic-go client. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +Test.Summary = ''' +Verify that a quic-go HTTP/3 client can complete sequential and concurrent +transactions through ATS. +''' + +Test.SkipUnless( + Condition.HasATSFeature('TS_USE_QUIC'), + Condition.HasGoVersion('1.24'), +) + + +def add_default_ssl_multicert(ts): + """Configure the default server certificate.""" + ts.Disk.ssl_multicert_yaml.AddLines( + """ +ssl_multicert: + - dest_ip: "*" + ssl_cert_name: server.pem + ssl_key_name: server.key +""".split("\n")) + + +class TestHttp3GoClient: + """Configure a test to verify HTTP/3 quic-go client interoperability.""" + + replay_file = "replays/h3_server_for_go_client.replay.yaml" + + def __init__(self, name: str): + """Initialize the test.""" + self.name = name + self._configure_server() + self._configure_traffic_server() + self._configure_client() + + def _configure_server(self): + """Configure the Proxy Verifier origin server.""" + self._server = Test.MakeVerifierServerProcess( + "server-go-h3-client", self.replay_file, verbose=False, other_args="--poll-timeout 30000") + + def _configure_traffic_server(self): + """Configure Traffic Server.""" + ts = Test.MakeATSProcess("ts-go-h3-client", enable_tls=True, enable_quic=True, enable_cache=False) + ts.StartupTimeout = 60 + ts.addDefaultSSLFiles() + add_default_ssl_multicert(ts) + ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'quic|http3', + 'proxy.config.quic.initial_max_data_in': 1000000, + 'proxy.config.quic.initial_max_stream_data_bidi_remote_in': 1000000, + 'proxy.config.quic.server.stateless_retry_enabled': 0, + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + }) + ts.Disk.remap_config.AddLine(f'map / http://127.0.0.1:{self._server.Variables.http_port}') + ts.Disk.logging_yaml.AddLines( + ''' +logging: + formats: + - name: h3_go_access + format: 'c_alpn=% client_version=% c_method=% c_url=%' + + logs: + - filename: h3_go_access + format: h3_go_access +'''.split("\n")) + + self._access_log = Test.Disk.File(os.path.join(ts.Variables.LOGDIR, 'h3_go_access.log'), exists=True) + self._access_log.Content = Testers.ContainsExpression( + r'c_alpn=h3 client_version=http/3 c_method=GET c_url=https://go\.example\.com:[0-9]+/go-get-empty', + "ATS should log the quic-go request as HTTP/3") + self._access_log.Content += Testers.ContainsExpression( + r'c_alpn=h3 client_version=http/3 c_method=POST c_url=https://go\.example\.com:[0-9]+/go-post-large', + "ATS should log the quic-go large POST as HTTP/3") + + self._ts = ts + + def _configure_client(self): + """Configure the quic-go client test runs.""" + tr = Test.AddTestRun(self.name) + tr.Setup.Copy("go_h3_client") + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._ts) + tr.Processes.Default.Env['GOFLAGS'] = '-mod=readonly' + tr.Processes.Default.Env['GOCACHE'] = os.path.join(tr.RunDirectory, 'gocache') + tr.Processes.Default.Env['GOMODCACHE'] = os.path.join(tr.RunDirectory, 'gomodcache') + tr.Processes.Default.Env['GOTOOLCHAIN'] = 'local' + tr.Processes.Default.Command = ( + f'cd "{os.path.join(tr.RunDirectory, "go_h3_client")}" && ' + f'go run . --addr 127.0.0.1:{self._ts.Variables.ssl_port} ' + f'--authority go.example.com:{self._ts.Variables.ssl_port} ' + '--server-name go.example.com') + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.stdout = Testers.ContainsExpression( + "completed 13 HTTP/3 requests", "The quic-go client should complete all HTTP/3 requests.") + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + tr = Test.AddTestRun("Wait for quic-go HTTP/3 access log") + tr.Processes.Default.Command = ( + os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + + os.path.join(self._ts.Variables.LOGDIR, 'h3_go_access.log')) + tr.Processes.Default.ReturnCode = 0 + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + +TestHttp3GoClient("quic-go HTTP/3 client requests") diff --git a/tests/gold_tests/h3/h3_h2_origin.test.py b/tests/gold_tests/h3/h3_h2_origin.test.py new file mode 100644 index 00000000000..afa03799062 --- /dev/null +++ b/tests/gold_tests/h3/h3_h2_origin.test.py @@ -0,0 +1,26 @@ +''' +Verify HTTP/3 client traffic to an HTTP/2 origin. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify that HTTP/3 client requests are proxied correctly to an HTTP/2 origin. +''' + +Test.SkipUnless(Condition.HasATSFeature('TS_USE_QUIC')) + +Test.ATSReplayTest(replay_file="replays/h3_h2_origin.replay.yaml") diff --git a/tests/gold_tests/h3/h3_proxy_verifier.test.py b/tests/gold_tests/h3/h3_proxy_verifier.test.py new file mode 100644 index 00000000000..a32b21dfd59 --- /dev/null +++ b/tests/gold_tests/h3/h3_proxy_verifier.test.py @@ -0,0 +1,27 @@ +''' +Verify HTTP/3 client interop with Proxy Verifier. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify a real HTTP/3 Proxy Verifier client can complete multiple transactions +across multiple QUIC connections through ATS. +''' + +Test.SkipUnless(Condition.HasATSFeature('TS_USE_QUIC')) + +Test.ATSReplayTest(replay_file="replays/h3_proxy_verifier.replay.yaml") diff --git a/tests/gold_tests/h3/h3_python_client.test.py b/tests/gold_tests/h3/h3_python_client.test.py new file mode 100644 index 00000000000..2d809977ef4 --- /dev/null +++ b/tests/gold_tests/h3/h3_python_client.test.py @@ -0,0 +1,132 @@ +''' +Verify HTTP/3 client interop with an aioquic Python client. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import sys + +Test.Summary = ''' +Verify that an aioquic HTTP/3 client can complete normal requests and selected +HTTP/3 edge-case probes through ATS. +''' + +Test.SkipUnless( + Condition.HasATSFeature('TS_USE_QUIC'), + Condition.HasProgram("python3", "python3 is required for the aioquic HTTP/3 client"), +) + + +def add_default_ssl_multicert(ts): + """Configure the default server certificate.""" + if hasattr(ts.Disk, "ssl_multicert_yaml"): + ts.Disk.ssl_multicert_yaml.AddLines( + """ +ssl_multicert: + - dest_ip: "*" + ssl_cert_name: server.pem + ssl_key_name: server.key +""".split("\n")) + else: + ts.Disk.ssl_multicert_config.AddLine("dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key") + + +class TestHttp3PythonClient: + """Configure a test to verify HTTP/3 aioquic client interoperability.""" + + replay_file = "replays/h3_server_for_python_client.replay.yaml" + + def __init__(self, name: str): + """Initialize the test.""" + self.name = name + self._configure_server() + self._configure_traffic_server() + self._configure_client() + + def _configure_server(self): + """Configure the Proxy Verifier origin server.""" + self._server = Test.MakeVerifierServerProcess( + "server-python-h3-client", self.replay_file, verbose=False, other_args="--poll-timeout 30000") + + def _configure_traffic_server(self): + """Configure Traffic Server.""" + ts = Test.MakeATSProcess("ts-python-h3-client", enable_tls=True, enable_quic=True, enable_cache=False) + ts.StartupTimeout = 60 + ts.addDefaultSSLFiles() + add_default_ssl_multicert(ts) + ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'quic|http3', + 'proxy.config.quic.initial_max_data_in': 1000000, + 'proxy.config.quic.initial_max_stream_data_bidi_remote_in': 1000000, + 'proxy.config.quic.max_send_udp_payload_size_in': 1200, + 'proxy.config.quic.server.stateless_retry_enabled': 0, + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + }) + ts.Disk.remap_config.AddLine(f'map / http://127.0.0.1:{self._server.Variables.http_port}') + ts.Disk.logging_yaml.AddLines( + ''' +logging: + formats: + - name: h3_python_access + format: 'c_alpn=% client_version=% c_method=% c_url=%' + + logs: + - filename: h3_python_access + format: h3_python_access +'''.split("\n")) + + self._access_log = Test.Disk.File(os.path.join(ts.Variables.LOGDIR, 'h3_python_access.log'), exists=True) + self._access_log.Content = Testers.ContainsExpression( + r'c_alpn=h3 client_version=http/3 c_method=GET c_url=https://py\.example\.com:[0-9]+/py-get-empty', + "ATS should log the aioquic request as HTTP/3") + self._access_log.Content += Testers.ContainsExpression( + r'c_alpn=h3 client_version=http/3 c_method=PUT c_url=https://py\.example\.com:[0-9]+/py-put-large', + "ATS should log the aioquic large PUT as HTTP/3") + + self._ts = ts + + def _configure_client(self): + """Configure the aioquic client test runs.""" + tr = Test.AddTestRun(self.name) + tr.Setup.Copy("py_h3_client") + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._ts) + client_dir = os.path.join(tr.RunDirectory, "py_h3_client") + tr.Processes.Default.Command = ( + f'"{sys.executable}" "{os.path.join(client_dir, "h3_client.py")}" ' + f'--addr 127.0.0.1:{self._ts.Variables.ssl_port} ' + f'--authority py.example.com:{self._ts.Variables.ssl_port} ' + '--server-name py.example.com') + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.stdout = Testers.ContainsExpression( + "completed 18 Python HTTP/3 checks", "The aioquic client should complete all HTTP/3 checks.") + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + tr = Test.AddTestRun("Wait for aioquic HTTP/3 access log") + tr.Processes.Default.Command = ( + os.path.join(Test.Variables.AtsTestToolsDir, 'condwait') + ' 60 1 -f ' + + os.path.join(self._ts.Variables.LOGDIR, 'h3_python_access.log')) + tr.Processes.Default.ReturnCode = 0 + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + +TestHttp3PythonClient("aioquic HTTP/3 client requests") diff --git a/tests/gold_tests/h3/h3_range_cache.test.py b/tests/gold_tests/h3/h3_range_cache.test.py new file mode 100644 index 00000000000..2222e2e47dc --- /dev/null +++ b/tests/gold_tests/h3/h3_range_cache.test.py @@ -0,0 +1,140 @@ +''' +Verify HTTP/3 range requests over cached content. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os + +Test.Summary = ''' +Verify that HTTP/3 clients can populate cache and receive range responses from +cached objects. +''' + +Test.SkipUnless( + Condition.HasATSFeature('TS_USE_QUIC'), + Condition.HasCurlFeature('http3'), + Condition.HasCurlOption('--http3-only'), +) +Test.SkipIf(Condition.CurlUsingUnixDomainSocket()) + + +def add_default_ssl_multicert(ts): + """Configure the default server certificate.""" + if hasattr(ts.Disk, "ssl_multicert_yaml"): + ts.Disk.ssl_multicert_yaml.AddLines( + """ +ssl_multicert: + - dest_ip: "*" + ssl_cert_name: server.pem + ssl_key_name: server.key +""".split("\n")) + else: + ts.Disk.ssl_multicert_config.AddLine("dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key") + + +class TestHttp3RangeCache: + """Configure an HTTP/3 range-over-cache test.""" + + response_body = "0123456789" * 30000 + range_body = "6789012345678901" + + def __init__(self): + """Initialize the test.""" + self._configure_server() + self._configure_traffic_server() + self._configure_clients() + + def _configure_server(self): + """Configure the origin server.""" + server = Test.MakeOriginServer("server-h3-range-cache") + server.addResponse( + "sessionlog.json", { + "headers": "GET /h3-range-cache HTTP/1.1\r\nHost: localhost\r\n\r\n", + "timestamp": "1469733493.993", + "body": "" + }, { + "headers": + ( + "HTTP/1.1 200 OK\r\n" + "Connection: close\r\n" + "Cache-Control: public, max-age=60\r\n" + f"Content-Length: {len(self.response_body)}\r\n\r\n"), + "timestamp": "1469733493.993", + "body": self.response_body + }) + self._server = server + + def _configure_traffic_server(self): + """Configure Traffic Server.""" + ts = Test.MakeATSProcess("ts-h3-range-cache", enable_tls=True, enable_quic=True, enable_cache=True) + ts.StartupTimeout = 60 + ts.addDefaultSSLFiles() + add_default_ssl_multicert(ts) + ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'quic|http3|http', + 'proxy.config.quic.server.stateless_retry_enabled': 0, + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + }) + ts.Disk.remap_config.AddLine(f'map / http://127.0.0.1:{self._server.Variables.Port}') + self._ts = ts + + def _curl_base(self): + """Build the shared curl arguments.""" + return ( + '--silent --show-error --fail --ipv4 --http3-only --insecure ' + f'--resolve "range.example.com:{self._ts.Variables.ssl_port}:127.0.0.1" ' + f'https://range.example.com:{self._ts.Variables.ssl_port}/h3-range-cache') + + def _configure_clients(self): + """Configure the cache fill and range request clients.""" + full_body_path = os.path.join(Test.RunDirectory, "h3-range-full.txt") + range_body_path = os.path.join(Test.RunDirectory, "h3-range-part.txt") + + tr = Test.AddTestRun("HTTP/3 cache fill") + tr.Processes.Default.StartBefore(self._server) + tr.Processes.Default.StartBefore(self._ts) + tr.MakeCurlCommand( + f'{self._curl_base()} --output "{full_body_path}" ' + '--write-out "\\nhttp_code=%{http_code}\\nsize_download=%{size_download}\\n"', + ts=self._ts) + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.stdout = Testers.ContainsExpression("http_code=200", "The fill request should return 200.") + tr.Processes.Default.Streams.stdout += Testers.ContainsExpression( + f"size_download={len(self.response_body)}", "The fill request should receive the full object.") + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + tr = Test.AddTestRun("HTTP/3 cached range request") + tr.MakeCurlCommand( + f'{self._curl_base()} --header "Range: bytes=16-31" --output "{range_body_path}" ' + '--write-out "\\nhttp_code=%{http_code}\\nsize_download=%{size_download}\\n"', + ts=self._ts) + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.stdout = Testers.ContainsExpression("http_code=206", "The range request should return 206.") + tr.Processes.Default.Streams.stdout += Testers.ContainsExpression( + "size_download=16", "The range request should receive 16 bytes.") + Test.Disk.File( + range_body_path, exists=True).Content = Testers.ContainsExpression( + self.range_body, "The cached range response body should match the requested byte range.") + tr.StillRunningAfter = self._server + tr.StillRunningAfter = self._ts + + +TestHttp3RangeCache() diff --git a/tests/gold_tests/h3/h3_session_ticket.sh b/tests/gold_tests/h3/h3_session_ticket.sh new file mode 100755 index 00000000000..ca7ade26397 --- /dev/null +++ b/tests/gold_tests/h3/h3_session_ticket.sh @@ -0,0 +1,41 @@ +#!/usr/bin/env bash +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -euo pipefail + +session_option=$1 +session_file=$2 +port=$3 + +set +e +sleep 1 | timeout 5 openssl s_client \ + -quic \ + -alpn h3 \ + -connect "127.0.0.1:${port}" \ + -servername foo.com \ + "${session_option}" "${session_file}" \ + -brief \ + -ign_eof +status=${PIPESTATUS[1]} +set -e + +if [[ ${status} -ne 0 && ${status} -ne 1 && ${status} -ne 124 ]]; then + exit "${status}" +fi + +test -s "${session_file}" diff --git a/tests/gold_tests/h3/h3_session_ticket.test.py b/tests/gold_tests/h3/h3_session_ticket.test.py new file mode 100644 index 00000000000..fd839aad1cb --- /dev/null +++ b/tests/gold_tests/h3/h3_session_ticket.test.py @@ -0,0 +1,110 @@ +''' +Verify HTTP/3 QUIC TLS session ticket handling. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +import os +import shlex + +Test.Summary = ''' +Verify that HTTP/3 QUIC connections can receive and offer TLS session tickets. +''' + +Test.SkipUnless( + Condition.HasATSFeature('TS_USE_QUIC'), + Condition.HasOpenSSLVersion('3.5.0'), + Condition.HasOpenSSLQuicClient(), +) +Test.Setup.Copy('../tls/file.ticket') + + +def add_default_ssl_multicert(ts): + """Configure the default server certificate.""" + if hasattr(ts.Disk, "ssl_multicert_yaml"): + ts.Disk.ssl_multicert_yaml.AddLines( + """ +ssl_multicert: + - dest_ip: "*" + ssl_cert_name: server.pem + ssl_key_name: server.key +""".split("\n")) + else: + ts.Disk.ssl_multicert_config.AddLine("dest_ip=* ssl_cert_name=server.pem ssl_key_name=server.key") + + +class TestHttp3SessionTicket: + """Configure an HTTP/3 QUIC TLS session ticket test.""" + + def __init__(self, name: str): + """Initialize the test.""" + self.name = name + self.session_file = os.path.join(Test.RunDirectory, "h3-quic-session.pem") + self.ticket_file = os.path.join(Test.RunDirectory, "file.ticket") + self._configure_traffic_server() + self._configure_ticket_save() + self._configure_ticket_reuse() + + def _configure_traffic_server(self): + """Configure Traffic Server.""" + ts = Test.MakeATSProcess("ts", enable_tls=True, enable_quic=True, enable_cache=False) + ts.StartupTimeout = 60 + ts.addDefaultSSLFiles() + add_default_ssl_multicert(ts) + ts.Disk.records_config.update( + { + 'proxy.config.diags.debug.enabled': 1, + 'proxy.config.diags.debug.tags': 'quic|ssl', + 'proxy.config.quic.server.stateless_retry_enabled': 0, + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, + 'proxy.config.ssl.server.session_ticket.enable': 1, + 'proxy.config.ssl.server.session_ticket.number': 2, + 'proxy.config.ssl.server.ticket_key.filename': self.ticket_file, + }) + + self._ts = ts + + def _s_client_command(self, session_option: str): + """Build an OpenSSL QUIC client command for ticket save or reuse.""" + script = os.path.join(Test.TestDirectory, "h3_session_ticket.sh") + return f"{shlex.quote(script)} {session_option} {shlex.quote(self.session_file)} {self._ts.Variables.ssl_port}" + + def _check_s_client_handshake(self, tr): + """Verify that OpenSSL completed the QUIC handshake.""" + tr.Processes.Default.ReturnCode = 0 + tr.Processes.Default.Streams.All = Testers.ContainsExpression( + "CONNECTION ESTABLISHED", "OpenSSL should complete the QUIC handshake.") + tr.Processes.Default.Streams.All += Testers.ContainsExpression( + "Protocol version: QUICv1", "OpenSSL should negotiate QUICv1.") + + def _configure_ticket_save(self): + """Configure the ticket save test run.""" + tr = Test.AddTestRun(self.name) + tr.Processes.Default.StartBefore(self._ts) + tr.Processes.Default.Command = f"rm -f {shlex.quote(self.session_file)}; {self._s_client_command('-sess_out')}" + self._check_s_client_handshake(tr) + tr.StillRunningAfter = self._ts + + def _configure_ticket_reuse(self): + """Configure the ticket reuse test run.""" + tr = Test.AddTestRun("OpenSSL QUIC offers saved session ticket") + tr.Processes.Default.Command = self._s_client_command("-sess_in") + self._check_s_client_handshake(tr) + tr.StillRunningAfter = self._ts + + +TestHttp3SessionTicket("OpenSSL QUIC saves session ticket") diff --git a/tests/gold_tests/h3/h3_sni_check.test.py b/tests/gold_tests/h3/h3_sni_check.test.py index 185ef70236d..febbf4ab3b4 100644 --- a/tests/gold_tests/h3/h3_sni_check.test.py +++ b/tests/gold_tests/h3/h3_sni_check.test.py @@ -21,7 +21,7 @@ Verify h3 SNI checking behavior. ''' -Test.SkipUnless(Condition.HasATSFeature('TS_HAS_QUICHE'), Condition.HasCurlFeature('http3')) +Test.SkipUnless(Condition.HasATSFeature('TS_USE_QUIC')) Test.ContinueOnFail = True @@ -67,9 +67,13 @@ def _configure_traffic_server(self, tr: 'TestRun'): self._ts = ts # Configure TLS for Traffic Server. self._ts.addDefaultSSLFiles() + self._ts.addSSLfile("../tls/ssl/signed-foo.pem") + self._ts.addSSLfile("../tls/ssl/signed-foo.key") self._ts.Disk.ssl_multicert_yaml.AddLines( """ ssl_multicert: + - ssl_cert_name: signed-foo.pem + ssl_key_name: signed-foo.key - dest_ip: "*" ssl_cert_name: server.pem ssl_key_name: server.key @@ -78,9 +82,9 @@ def _configure_traffic_server(self, tr: 'TestRun'): { 'proxy.config.diags.debug.enabled': 1, 'proxy.config.diags.debug.tags': 'http', - 'proxy.config.ssl.server.cert.path': '{0}'.format(ts.Variables.SSLDir), + 'proxy.config.ssl.server.cert.path': ts.Variables.SSLDir, 'proxy.config.quic.no_activity_timeout_in': 0, - 'proxy.config.ssl.server.private_key.path': '{0}'.format(ts.Variables.SSLDir), + 'proxy.config.ssl.server.private_key.path': ts.Variables.SSLDir, 'proxy.config.ssl.client.verify.server.policy': 'PERMISSIVE', }) diff --git a/tests/gold_tests/h3/h3_stream_lifetime.test.py b/tests/gold_tests/h3/h3_stream_lifetime.test.py new file mode 100644 index 00000000000..9715392ea78 --- /dev/null +++ b/tests/gold_tests/h3/h3_stream_lifetime.test.py @@ -0,0 +1,27 @@ +''' +Verify HTTP/3 stream lifetime handling with concurrent streams. +''' +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +Test.Summary = ''' +Verify HTTP/3 transactions survive concurrent stream close and write-ready +events on the same QUIC connection. +''' + +Test.SkipUnless(Condition.HasATSFeature('TS_USE_QUIC')) + +Test.ATSReplayTest(replay_file="replays/h3_stream_lifetime.replay.yaml") diff --git a/tests/gold_tests/h3/py_h3_client/h3_client.py b/tests/gold_tests/h3/py_h3_client/h3_client.py new file mode 100644 index 00000000000..63af49f0c25 --- /dev/null +++ b/tests/gold_tests/h3/py_h3_client/h3_client.py @@ -0,0 +1,332 @@ +#!/usr/bin/env python3 +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +"""Exercise ATS HTTP/3 client-side behavior with aioquic.""" + +from __future__ import annotations + +import argparse +import asyncio +import ssl +from dataclasses import dataclass, field +from typing import Awaitable, Callable + +from aioquic.asyncio.client import connect +from aioquic.asyncio.protocol import QuicConnectionProtocol +from aioquic.buffer import encode_uint_var +from aioquic.h3.connection import H3Connection, H3_ALPN, FrameType, StreamType, encode_frame +from aioquic.h3.events import DataReceived, HeadersReceived +from aioquic.quic.configuration import QuicConfiguration +from aioquic.quic.events import ConnectionTerminated, QuicEvent, StreamDataReceived + +LARGE_BODY_SIZE = 300000 +LARGE_BODY_SUFFIX = b"000927b " +REUSED_HEADER_VALUE = b"stable-python-qpack-value" + + +@dataclass +class RequestCase: + """A single HTTP/3 request/response expectation.""" + + name: str + method: bytes + path: str + request_size: int = 0 + response_size: int = 0 + status: int = 200 + + +@dataclass +class ResponseState: + """Accumulate response headers and data for one HTTP/3 stream.""" + + header_blocks: list[list[tuple[bytes, bytes]]] = field(default_factory=list) + body: bytearray = field(default_factory=bytearray) + + @property + def status(self) -> int: + for header_block in reversed(self.header_blocks): + for name, value in header_block: + if name == b":status": + return int(value) + raise RuntimeError("response did not contain :status") + + +class H3ClientProtocol(QuicConnectionProtocol): + """Minimal HTTP/3 client protocol with raw QUIC stream helpers.""" + + def __init__(self, *args, **kwargs): + super().__init__(*args, **kwargs) + self._http = H3Connection(self._quic) + self._responses: dict[int, ResponseState] = {} + self._waiters: dict[int, asyncio.Future[ResponseState]] = {} + self._raw_response_bytes: dict[int, int] = {} + self._event_counts: dict[str, int] = {} + self._terminated: asyncio.Future[ConnectionTerminated] = asyncio.get_running_loop().create_future() + + def quic_event_received(self, event: QuicEvent) -> None: + event_name = type(event).__name__ + self._event_counts[event_name] = self._event_counts.get(event_name, 0) + 1 + + if isinstance(event, ConnectionTerminated) and not self._terminated.done(): + self._terminated.set_result(event) + for waiter in self._waiters.values(): + if not waiter.done(): + waiter.set_exception(RuntimeError(f"connection terminated: {event.error_code} {event.reason_phrase}")) + + for http_event in self._http.handle_event(event): + if isinstance(http_event, HeadersReceived): + response = self._responses.setdefault(http_event.stream_id, ResponseState()) + response.header_blocks.append(http_event.headers) + if http_event.stream_ended: + self._complete_response(http_event.stream_id) + elif isinstance(http_event, DataReceived): + response = self._responses.setdefault(http_event.stream_id, ResponseState()) + response.body.extend(http_event.data) + if http_event.stream_ended: + self._complete_response(http_event.stream_id) + + if isinstance(event, StreamDataReceived): + self._raw_response_bytes[event.stream_id] = self._raw_response_bytes.get(event.stream_id, 0) + len(event.data) + if event.end_stream and event.stream_id in self._responses: + self._complete_response(event.stream_id) + + self.transmit() + + def _complete_response(self, stream_id: int) -> None: + waiter = self._waiters.get(stream_id) + if waiter is not None and not waiter.done(): + waiter.set_result(self._responses[stream_id]) + + async def request(self, authority: str, request_case: RequestCase) -> ResponseState: + stream_id = self._quic.get_next_available_stream_id() + waiter: asyncio.Future[ResponseState] = asyncio.get_running_loop().create_future() + self._waiters[stream_id] = waiter + + headers = [ + (b":method", request_case.method), + (b":scheme", b"https"), + (b":authority", authority.encode()), + (b":path", request_case.path.encode()), + (b"user-agent", b"ats-h3-aioquic-autest"), + (b"x-h3-python-client", b"aioquic"), + (b"x-h3-reused-header", REUSED_HEADER_VALUE), + (b"x-h3-test-case", request_case.name.encode()), + (b"uuid", request_case.name.encode()), + ] + if request_case.request_size > 0: + headers.extend( + [ + (b"content-type", b"application/octet-stream"), + (b"content-length", str(request_case.request_size).encode()), + ]) + + request_body = generated_body(request_case.request_size) + self._http.send_headers(stream_id, headers, end_stream=not request_body) + if request_body: + self._http.send_data(stream_id, request_body, end_stream=True) + self.transmit() + + try: + return await asyncio.wait_for(waiter, timeout=30) + except TimeoutError as e: + response = self._responses.get(stream_id) + raw_response_bytes = self._raw_response_bytes.get(stream_id, 0) + event_summary = ", ".join(f"{name}={count}" for name, count in sorted(self._event_counts.items())) + if response is None: + raise TimeoutError( + f"{request_case.name}: timed out before receiving response headers; raw QUIC stream bytes={raw_response_bytes}; " + f"events=[{event_summary}]") from e + raise TimeoutError( + f"{request_case.name}: timed out after receiving {len(response.header_blocks)} header block(s) and " + f"{len(response.body)} response byte(s); raw QUIC stream bytes={raw_response_bytes}; events=[{event_summary}]" + ) from e + + async def wait_for_termination(self) -> ConnectionTerminated: + return await asyncio.wait_for(self._terminated, timeout=5) + + def send_unknown_unidirectional_stream(self) -> None: + stream_id = self._quic.get_next_available_stream_id(is_unidirectional=True) + self._quic.send_stream_data(stream_id, encode_uint_var(0x21) + b"ignored", end_stream=True) + self.transmit() + + def send_client_push_stream(self) -> None: + stream_id = self._quic.get_next_available_stream_id(is_unidirectional=True) + self._quic.send_stream_data(stream_id, encode_uint_var(StreamType.PUSH) + encode_uint_var(0), end_stream=False) + self.transmit() + + def send_duplicate_control_stream(self) -> None: + stream_id = self._quic.get_next_available_stream_id(is_unidirectional=True) + payload = encode_uint_var(StreamType.CONTROL) + encode_frame(FrameType.SETTINGS, b"") + self._quic.send_stream_data(stream_id, payload, end_stream=False) + self.transmit() + + def send_reserved_request_frame(self) -> None: + stream_id = self._quic.get_next_available_stream_id() + self._quic.send_stream_data(stream_id, encode_frame(0x21, b""), end_stream=True) + self.transmit() + + def send_data_before_headers(self) -> None: + stream_id = self._quic.get_next_available_stream_id() + self._quic.send_stream_data(stream_id, encode_frame(FrameType.DATA, b"bad"), end_stream=True) + self.transmit() + + +def generated_body(size: int) -> bytes: + """Generate deterministic content matching Proxy Verifier size bodies.""" + chunks: list[bytes] = [] + total = 0 + value = 0 + while total < size: + chunk = f"{value:07x} ".encode() + chunks.append(chunk) + total += len(chunk) + value += 1 + return b"".join(chunks)[:size] + + +def quic_configuration(server_name: str) -> QuicConfiguration: + """Create an insecure test-only HTTP/3 client configuration.""" + configuration = QuicConfiguration(is_client=True, alpn_protocols=H3_ALPN, server_name=server_name) + configuration.verify_mode = ssl.CERT_NONE + return configuration + + +async def connect_h3(host: str, port: int, server_name: str): + """Open an HTTP/3 connection using the test protocol.""" + return connect(host, port, configuration=quic_configuration(server_name), create_protocol=H3ClientProtocol) + + +def verify_response(request_case: RequestCase, response: ResponseState) -> None: + """Verify one response matches the expected status and body.""" + body = bytes(response.body) + if response.status != request_case.status: + raise AssertionError(f"{request_case.name}: expected status {request_case.status}, got {response.status}") + + if request_case.method == b"HEAD" or request_case.status == 204: + if body: + raise AssertionError(f"{request_case.name}: expected no response body, got {len(body)} bytes") + return + + expected = generated_body(request_case.response_size) + if body != expected: + raise AssertionError(f"{request_case.name}: response body mismatch: got {len(body)}, expected {len(expected)}") + if request_case.response_size == LARGE_BODY_SIZE and not body.endswith(LARGE_BODY_SUFFIX): + raise AssertionError(f"{request_case.name}: large body suffix mismatch") + + +async def run_requests(host: str, port: int, authority: str, server_name: str, request_cases: list[RequestCase]) -> None: + """Run a sequence of request cases on one HTTP/3 connection.""" + async with await connect_h3(host, port, server_name) as client: + for request_case in request_cases: + response = await client.request(authority, request_case) + verify_response(request_case, response) + print(f"ok {request_case.name}") + + +async def run_concurrent_requests(host: str, port: int, authority: str, server_name: str, request_cases: list[RequestCase]) -> None: + """Run request cases concurrently on one HTTP/3 connection.""" + async with await connect_h3(host, port, server_name) as client: + responses = await asyncio.gather(*(client.request(authority, request_case) for request_case in request_cases)) + for request_case, response in zip(request_cases, responses): + verify_response(request_case, response) + print(f"ok {request_case.name}") + + +async def expect_connection_error( + host: str, + port: int, + server_name: str, + name: str, + action: Callable[[H3ClientProtocol], None], +) -> None: + """Run a malformed action and require ATS to close the QUIC connection.""" + async with await connect_h3(host, port, server_name) as client: + action(client) + terminated = await client.wait_for_termination() + if terminated.error_code == 0: + raise AssertionError(f"{name}: expected non-zero H3/QUIC close error") + print(f"ok {name} error={terminated.error_code}") + + +async def run_edge_cases(host: str, port: int, authority: str, server_name: str) -> None: + """Exercise H3 control stream and frame behavior with raw stream writes.""" + async with await connect_h3(host, port, server_name) as client: + client.send_unknown_unidirectional_stream() + request_case = RequestCase("py-edge-after-unknown", b"GET", "/py-edge-after-unknown", response_size=100) + response = await client.request(authority, request_case) + verify_response(request_case, response) + print("ok py-unknown-unidirectional-stream") + + await expect_connection_error( + host, port, server_name, "py-client-push-stream-rejected", lambda client: client.send_client_push_stream()) + await expect_connection_error( + host, port, server_name, "py-duplicate-control-stream-rejected", lambda client: client.send_duplicate_control_stream()) + async with await connect_h3(host, port, server_name) as client: + client.send_reserved_request_frame() + request_case = RequestCase("py-edge-after-reserved", b"GET", "/py-edge-after-reserved", response_size=100) + response = await client.request(authority, request_case) + verify_response(request_case, response) + print("ok py-reserved-request-frame-ignored") + + await expect_connection_error( + host, port, server_name, "py-data-before-headers-rejected", lambda client: client.send_data_before_headers()) + + +async def async_main() -> None: + parser = argparse.ArgumentParser() + parser.add_argument("--addr", required=True, help="ATS HTTP/3 address in host:port form") + parser.add_argument("--authority", required=True, help="HTTP/3 request authority") + parser.add_argument("--server-name", required=True, help="TLS SNI server name") + args = parser.parse_args() + + host, port_text = args.addr.rsplit(":", 1) + port = int(port_text) + + sequential_cases = [ + RequestCase("py-get-empty", b"GET", "/py-get-empty"), + RequestCase("py-get-small", b"GET", "/py-get-small", response_size=100), + RequestCase("py-head-no-body", b"HEAD", "/py-head-no-body", response_size=100), + RequestCase("py-204-no-body", b"GET", "/py-204-no-body", status=204), + RequestCase("py-post-small", b"POST", "/py-post-small", request_size=100, response_size=100), + RequestCase("py-put-small", b"PUT", "/py-put-small", request_size=100, response_size=100), + RequestCase("py-delete-empty", b"DELETE", "/py-delete-empty", status=204), + RequestCase("py-options-small", b"OPTIONS", "/py-options-small", response_size=100), + ] + concurrent_cases = [ + RequestCase("py-get-concurrent-large", b"GET", "/py-get-concurrent-large", response_size=LARGE_BODY_SIZE), + RequestCase("py-get-concurrent-small", b"GET", "/py-get-concurrent-small", response_size=100), + ] + large_cases = [ + RequestCase("py-get-large", b"GET", "/py-get-large", response_size=LARGE_BODY_SIZE), + RequestCase("py-post-large", b"POST", "/py-post-large", request_size=LARGE_BODY_SIZE, response_size=LARGE_BODY_SIZE), + RequestCase("py-put-large", b"PUT", "/py-put-large", request_size=LARGE_BODY_SIZE, response_size=LARGE_BODY_SIZE), + ] + + await run_requests(host, port, args.authority, args.server_name, sequential_cases) + await run_requests(host, port, args.authority, args.server_name, large_cases) + await run_concurrent_requests(host, port, args.authority, args.server_name, concurrent_cases) + await run_edge_cases(host, port, args.authority, args.server_name) + print("completed 18 Python HTTP/3 checks") + + +def main() -> None: + asyncio.run(async_main()) + + +if __name__ == "__main__": + main() diff --git a/tests/gold_tests/h3/replays/h3_active_timeout.replay.yaml b/tests/gold_tests/h3/replays/h3_active_timeout.replay.yaml new file mode 100644 index 00000000000..f7a39324fd1 --- /dev/null +++ b/tests/gold_tests/h3/replays/h3_active_timeout.replay.yaml @@ -0,0 +1,85 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: "Verify HTTP/3 transaction cleanup on active timeout" + + server: + name: "server-h3-active-timeout" + + client: + name: "client-h3-active-timeout" + return_code: 1 + + ats: + name: "ts-h3-active-timeout" + startup_timeout: 60 + process_config: + enable_tls: true + enable_quic: true + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: "quic|http" + proxy.config.http.transaction_active_timeout_in: 1 + proxy.config.quic.no_activity_timeout_in: 0 + proxy.config.quic.server.stateless_retry_enabled: 0 + + remap_config: + - from: "https://example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + +sessions: +- protocol: + - name: http + version: 3 + - name: tls + sni: example.com + - name: udp + - name: ip + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-active-timeout ] + - [ uuid, h3-active-timeout ] + + server-response: + # Delay longer than the 1 second ATS active timeout configured by this replay. + delay: 2s + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "timeout-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrst" + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] diff --git a/tests/gold_tests/h3/replays/h3_flow_control.replay.yaml b/tests/gold_tests/h3/replays/h3_flow_control.replay.yaml new file mode 100644 index 00000000000..0444a72eaef --- /dev/null +++ b/tests/gold_tests/h3/replays/h3_flow_control.replay.yaml @@ -0,0 +1,132 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: "Verify HTTP/3 flow control with small QUIC windows" + + server: + name: "server-h3-flow-control" + process_config: + verbose: false + other_args: "--poll-timeout 30000" + + client: + name: "client-h3-flow-control" + process_config: + verbose: false + other_args: "--poll-timeout 30000" + + ats: + name: "ts-h3-flow-control" + startup_timeout: 60 + process_config: + enable_tls: true + enable_quic: true + enable_cache: false + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: "quic|http3" + proxy.config.quic.initial_max_data_in: 4096 + proxy.config.quic.initial_max_stream_data_bidi_remote_in: 4096 + proxy.config.quic.server.stateless_retry_enabled: 0 + + remap_config: + - from: "https://example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + + log_validation: + traffic_out: + contains: + - expression: 'start HTTP/3 app \(ALPN=h3\)' + description: "ATS should negotiate HTTP/3" + +sessions: +- protocol: + stack: http3 + tls: + sni: example.com + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", POST ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-flow-post-large ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, h3-flow-post-large ] + content: + size: 300000 + + proxy-request: + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-flow-get-large ] + - [ uuid, h3-flow-get-large ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 diff --git a/tests/gold_tests/h3/replays/h3_h2_origin.replay.yaml b/tests/gold_tests/h3/replays/h3_h2_origin.replay.yaml new file mode 100644 index 00000000000..16a4f6b5aa3 --- /dev/null +++ b/tests/gold_tests/h3/replays/h3_h2_origin.replay.yaml @@ -0,0 +1,352 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: "Verify HTTP/3 client traffic to an HTTP/2 origin" + + server: + name: "server-h3-h2-origin" + process_config: + verbose: false + other_args: "--poll-timeout 30000" + + client: + name: "client-h3-h2-origin" + process_config: + verbose: false + other_args: "--poll-timeout 30000" + + ats: + name: "ts-h3-h2-origin" + startup_timeout: 60 + process_config: + enable_tls: true + enable_quic: true + enable_cache: false + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: "quic|http3|http2" + proxy.config.quic.initial_max_data_in: 1000000 + proxy.config.quic.initial_max_stream_data_bidi_remote_in: 1000000 + proxy.config.quic.server.stateless_retry_enabled: 0 + proxy.config.ssl.client.alpn_protocols: "h2,http/1.1" + proxy.config.ssl.client.verify.server.policy: PERMISSIVE + + remap_config: + - from: "https://example.com/" + to: "https://127.0.0.1:{SERVER_HTTPS_PORT}/" + + log_validation: + traffic_out: + contains: + - expression: 'start HTTP/3 app \(ALPN=h3\)' + description: "ATS should negotiate HTTP/3 on the client side" + +sessions: +- protocol: + stack: http3 + tls: + sni: example.com + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-h2-get-empty ] + - [ uuid, h3-h2-get-empty ] + + proxy-request: + protocol: + stack: http2 + version: "2" + method: GET + scheme: https + url: /h3-h2-get-empty + + server-response: + version: "2" + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, "0" ] + content: + size: 0 + + proxy-response: + version: "3" + status: 200 + headers: + fields: + - [ Content-Length, { value: "0", as: equal } ] + + - client-request: + version: "3" + await: h3-h2-get-empty + headers: + fields: + - [ ":method", HEAD ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-h2-head-no-body ] + - [ uuid, h3-h2-head-no-body ] + + proxy-request: + protocol: + stack: http2 + version: "2" + method: HEAD + scheme: https + url: /h3-h2-head-no-body + + server-response: + version: "2" + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + + proxy-response: + version: "3" + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + + - client-request: + version: "3" + await: h3-h2-head-no-body + headers: + fields: + - [ ":method", OPTIONS ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-h2-options-small ] + - [ uuid, h3-h2-options-small ] + + proxy-request: + protocol: + stack: http2 + version: "2" + method: OPTIONS + scheme: https + url: /h3-h2-options-small + + server-response: + version: "2" + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + content: + size: 100 + + proxy-response: + version: "3" + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + size: 100 + + - client-request: + version: "3" + await: h3-h2-options-small + headers: + fields: + - [ ":method", DELETE ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-h2-delete-empty ] + - [ uuid, h3-h2-delete-empty ] + + proxy-request: + protocol: + stack: http2 + version: "2" + method: DELETE + scheme: https + url: /h3-h2-delete-empty + + server-response: + version: "2" + status: 204 + reason: No Content + headers: + fields: + - [ X-H3-Origin, h2-delete ] + + proxy-response: + version: "3" + status: 204 + +- protocol: + stack: http3 + tls: + sni: example.com + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", POST ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-h2-post-small ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + - [ uuid, h3-h2-post-small ] + content: + size: 100 + + proxy-request: + protocol: + stack: http2 + version: "2" + method: POST + scheme: https + url: /h3-h2-post-small + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + size: 100 + + server-response: + version: "2" + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + content: + size: 100 + + proxy-response: + version: "3" + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + size: 100 + + - client-request: + version: "3" + await: h3-h2-post-small + headers: + fields: + - [ ":method", PUT ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-h2-put-large ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, h3-h2-put-large ] + content: + size: 300000 + + proxy-request: + protocol: + stack: http2 + version: "2" + method: PUT + scheme: https + url: /h3-h2-put-large + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + + server-response: + version: "2" + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + + proxy-response: + version: "3" + status: 200 + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + +- protocol: + stack: http3 + tls: + sni: example.com + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-h2-get-large ] + - [ uuid, h3-h2-get-large ] + + proxy-request: + protocol: + stack: http2 + version: "2" + method: GET + scheme: https + url: /h3-h2-get-large + + server-response: + version: "2" + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + + proxy-response: + version: "3" + status: 200 + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 diff --git a/tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml b/tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml new file mode 100644 index 00000000000..817485859a5 --- /dev/null +++ b/tests/gold_tests/h3/replays/h3_proxy_verifier.replay.yaml @@ -0,0 +1,481 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: "Verify HTTP/3 client interop with Proxy Verifier across multiple connections" + + server: + name: "server-h3-proxy-verifier" + process_config: + verbose: false + other_args: "--poll-timeout 30000" + + client: + name: "client-h3-proxy-verifier" + process_config: + verbose: false + other_args: "--poll-timeout 30000" + + ats: + name: "ts-h3-proxy-verifier" + startup_timeout: 60 + process_config: + enable_tls: true + enable_quic: true + enable_cache: false + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: "quic|http3" + proxy.config.quic.initial_max_data_in: 1000000 + proxy.config.quic.initial_max_stream_data_bidi_remote_in: 1000000 + proxy.config.quic.server.stateless_retry_enabled: 0 + + remap_config: + - from: "https://example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + + log_validation: + traffic_out: + contains: + - expression: 'start HTTP/3 app \(ALPN=h3\)' + description: "ATS should negotiate HTTP/3" + +sessions: +- protocol: + - name: http + version: 3 + - name: tls + sni: example.com + - name: udp + - name: ip + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-get-empty ] + - [ uuid, h3-get-empty ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "0" ] + content: + size: 0 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "0", as: equal } ] + + - client-request: + version: "3" + await: h3-get-empty + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-get-small ] + - [ uuid, h3-get-small ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyzAB" + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvwxyzAB" + verify: { as: equal } + + - client-request: + version: "3" + await: h3-get-small + headers: + fields: + - [ ":method", HEAD ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-head-no-body ] + - [ uuid, h3-head-no-body ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + + - client-request: + version: "3" + await: h3-head-no-body + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-204-no-body ] + - [ uuid, h3-204-no-body ] + + server-response: + status: 204 + reason: No Content + headers: + fields: + - [ Content-Length, "0" ] + - [ X-H3-Status, no-content ] + + proxy-response: + status: 204 + + - client-request: + version: "3" + await: h3-204-no-body + headers: + fields: + - [ ":method", POST ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-post-small ] + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + - [ uuid, h3-post-small ] + content: + encoding: plain + data: "post-body-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqr" + + proxy-request: + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "post-body-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqr" + verify: { as: equal } + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "post-response-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmn" + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "post-response-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmn" + verify: { as: equal } + + - client-request: + version: "3" + await: h3-post-small + headers: + fields: + - [ ":method", PUT ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-put-small ] + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + - [ uuid, h3-put-small ] + content: + encoding: plain + data: "put-body-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrs" + + proxy-request: + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "put-body-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrs" + verify: { as: equal } + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "put-response-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmno" + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "put-response-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmno" + verify: { as: equal } + + - client-request: + version: "3" + await: h3-put-small + headers: + fields: + - [ ":method", DELETE ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-delete-empty ] + - [ uuid, h3-delete-empty ] + + server-response: + status: 204 + reason: No Content + headers: + fields: + - [ X-H3-Status, delete-no-content ] + + proxy-response: + status: 204 + + - client-request: + version: "3" + await: h3-delete-empty + headers: + fields: + - [ ":method", OPTIONS ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-options-small ] + - [ uuid, h3-options-small ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Allow, "GET, HEAD, POST, PUT, DELETE, OPTIONS" ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "options-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrst" + + proxy-response: + status: 200 + headers: + fields: + - [ Allow, { value: "GET, HEAD, POST, PUT, DELETE, OPTIONS", as: equal } ] + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "options-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrst" + verify: { as: equal } + +- protocol: + - name: http + version: 3 + - name: tls + sni: example.com + - name: udp + - name: ip + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", POST ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-post-empty ] + - [ Content-Type, text/plain ] + - [ Content-Length, "0" ] + - [ uuid, h3-post-empty ] + content: + size: 0 + + proxy-request: + headers: + fields: + - [ Content-Length, { value: "0", as: equal } ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "0" ] + content: + size: 0 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "0", as: equal } ] + + - client-request: + version: "3" + await: h3-post-empty + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-get-large ] + - [ uuid, h3-get-large ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + + - client-request: + version: "3" + await: h3-get-large + headers: + fields: + - [ ":method", PUT ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-put-large ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, h3-put-large ] + content: + size: 300000 + + proxy-request: + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + +- protocol: + - name: http + version: 3 + - name: tls + sni: example.com + - name: udp + - name: ip + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", POST ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-post-large ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, h3-post-large ] + content: + size: 300000 + + proxy-request: + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "300000", as: equal } ] + content: + size: 300000 diff --git a/tests/gold_tests/h3/replays/h3_server_for_go_client.replay.yaml b/tests/gold_tests/h3/replays/h3_server_for_go_client.replay.yaml new file mode 100644 index 00000000000..a7a475d8c3e --- /dev/null +++ b/tests/gold_tests/h3/replays/h3_server_for_go_client.replay.yaml @@ -0,0 +1,268 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - request_base: &request_base + version: "1.1" + - empty_response: &empty_response + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, "0" ] + content: + size: 0 + - generated_100_response: &generated_100_response + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + content: + size: 100 + - generated_300k_response: &generated_300k_response + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + +sessions: +- transactions: + - client-request: + <<: *request_base + method: GET + url: /go-get-empty + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-get-empty ] + - [ X-H3-Test-Case, go-get-empty ] + + server-response: + <<: *empty_response + + - client-request: + <<: *request_base + method: GET + url: /go-get-small + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-get-small ] + - [ X-H3-Test-Case, go-get-small ] + + server-response: + <<: *generated_100_response + + - client-request: + <<: *request_base + method: HEAD + url: /go-head-no-body + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-head-no-body ] + - [ X-H3-Test-Case, go-head-no-body ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + + - client-request: + <<: *request_base + method: GET + url: /go-204-no-body + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-204-no-body ] + - [ X-H3-Test-Case, go-204-no-body ] + + server-response: + status: 204 + reason: No Content + headers: + fields: + - [ Content-Length, "0" ] + - [ X-H3-Status, no-content ] + + - client-request: + <<: *request_base + method: POST + url: /go-post-small + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + - [ uuid, go-post-small ] + - [ X-H3-Test-Case, go-post-small ] + content: + size: 100 + verify: { as: equal } + + server-response: + <<: *generated_100_response + + - client-request: + <<: *request_base + method: PUT + url: /go-put-small + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + - [ uuid, go-put-small ] + - [ X-H3-Test-Case, go-put-small ] + content: + size: 100 + verify: { as: equal } + + server-response: + <<: *generated_100_response + + - client-request: + <<: *request_base + method: DELETE + url: /go-delete-empty + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-delete-empty ] + - [ X-H3-Test-Case, go-delete-empty ] + + server-response: + status: 204 + reason: No Content + headers: + fields: + - [ X-H3-Status, delete-no-content ] + + - client-request: + <<: *request_base + method: OPTIONS + url: /go-options-small + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-options-small ] + - [ X-H3-Test-Case, go-options-small ] + + server-response: + <<: *generated_100_response + +- transactions: + - client-request: + <<: *request_base + method: GET + url: /go-get-concurrent-large + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-get-concurrent-large ] + - [ X-H3-Test-Case, go-get-concurrent-large ] + + server-response: + <<: *generated_300k_response + + - client-request: + <<: *request_base + method: GET + url: /go-get-concurrent-small + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-get-concurrent-small ] + - [ X-H3-Test-Case, go-get-concurrent-small ] + + server-response: + <<: *generated_100_response + +- transactions: + - client-request: + <<: *request_base + method: GET + url: /go-get-large + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ uuid, go-get-large ] + - [ X-H3-Test-Case, go-get-large ] + + server-response: + <<: *generated_300k_response + + - client-request: + <<: *request_base + method: POST + url: /go-post-large + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, go-post-large ] + - [ X-H3-Test-Case, go-post-large ] + content: + size: 300000 + verify: { as: equal } + + server-response: + <<: *generated_300k_response + + - client-request: + <<: *request_base + method: PUT + url: /go-put-large + headers: + fields: + - [ X-H3-Go-Client, quic-go ] + - [ X-H3-Reused-Header, stable-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, go-put-large ] + - [ X-H3-Test-Case, go-put-large ] + content: + size: 300000 + verify: { as: equal } + + server-response: + <<: *generated_300k_response diff --git a/tests/gold_tests/h3/replays/h3_server_for_python_client.replay.yaml b/tests/gold_tests/h3/replays/h3_server_for_python_client.replay.yaml new file mode 100644 index 00000000000..2ccf00799bd --- /dev/null +++ b/tests/gold_tests/h3/replays/h3_server_for_python_client.replay.yaml @@ -0,0 +1,297 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + + blocks: + - request_base: &request_base + version: "1.1" + - empty_response: &empty_response + status: 200 + reason: OK + headers: + fields: + - [ Content-Length, "0" ] + content: + size: 0 + - generated_100_response: &generated_100_response + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + content: + size: 100 + - generated_300k_response: &generated_300k_response + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + content: + size: 300000 + +sessions: +- transactions: + - client-request: + <<: *request_base + method: GET + url: /py-get-empty + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-get-empty ] + - [ X-H3-Test-Case, py-get-empty ] + + server-response: + <<: *empty_response + + - client-request: + <<: *request_base + method: GET + url: /py-get-small + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-get-small ] + - [ X-H3-Test-Case, py-get-small ] + + server-response: + <<: *generated_100_response + + - client-request: + <<: *request_base + method: HEAD + url: /py-head-no-body + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-head-no-body ] + - [ X-H3-Test-Case, py-head-no-body ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "0" ] + + - client-request: + <<: *request_base + method: GET + url: /py-204-no-body + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-204-no-body ] + - [ X-H3-Test-Case, py-204-no-body ] + + server-response: + status: 204 + reason: No Content + headers: + fields: + - [ X-H3-Status, no-content ] + + - client-request: + <<: *request_base + method: POST + url: /py-post-small + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + - [ uuid, py-post-small ] + - [ X-H3-Test-Case, py-post-small ] + content: + size: 100 + verify: { as: equal } + + server-response: + <<: *generated_100_response + + - client-request: + <<: *request_base + method: PUT + url: /py-put-small + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "100" ] + - [ uuid, py-put-small ] + - [ X-H3-Test-Case, py-put-small ] + content: + size: 100 + verify: { as: equal } + + server-response: + <<: *generated_100_response + + - client-request: + <<: *request_base + method: DELETE + url: /py-delete-empty + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-delete-empty ] + - [ X-H3-Test-Case, py-delete-empty ] + + server-response: + status: 204 + reason: No Content + headers: + fields: + - [ X-H3-Status, delete-no-content ] + + - client-request: + <<: *request_base + method: OPTIONS + url: /py-options-small + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-options-small ] + - [ X-H3-Test-Case, py-options-small ] + + server-response: + <<: *generated_100_response + +- transactions: + - client-request: + <<: *request_base + method: GET + url: /py-get-concurrent-large + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-get-concurrent-large ] + - [ X-H3-Test-Case, py-get-concurrent-large ] + + server-response: + <<: *generated_300k_response + + - client-request: + <<: *request_base + method: GET + url: /py-get-concurrent-small + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-get-concurrent-small ] + - [ X-H3-Test-Case, py-get-concurrent-small ] + + server-response: + <<: *generated_100_response + +- transactions: + - client-request: + <<: *request_base + method: GET + url: /py-get-large + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-get-large ] + - [ X-H3-Test-Case, py-get-large ] + + server-response: + <<: *generated_300k_response + + - client-request: + <<: *request_base + method: POST + url: /py-post-large + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, py-post-large ] + - [ X-H3-Test-Case, py-post-large ] + content: + size: 300000 + verify: { as: equal } + + server-response: + <<: *generated_300k_response + + - client-request: + <<: *request_base + method: PUT + url: /py-put-large + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ Content-Type, application/octet-stream ] + - [ Content-Length, "300000" ] + - [ uuid, py-put-large ] + - [ X-H3-Test-Case, py-put-large ] + content: + size: 300000 + verify: { as: equal } + + server-response: + <<: *generated_300k_response + +- transactions: + - client-request: + <<: *request_base + method: GET + url: /py-edge-after-unknown + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-edge-after-unknown ] + - [ X-H3-Test-Case, py-edge-after-unknown ] + + server-response: + <<: *generated_100_response + +- transactions: + - client-request: + <<: *request_base + method: GET + url: /py-edge-after-reserved + headers: + fields: + - [ X-H3-Python-Client, aioquic ] + - [ X-H3-Reused-Header, stable-python-qpack-value ] + - [ uuid, py-edge-after-reserved ] + - [ X-H3-Test-Case, py-edge-after-reserved ] + + server-response: + <<: *generated_100_response diff --git a/tests/gold_tests/h3/replays/h3_sni.replay.yaml b/tests/gold_tests/h3/replays/h3_sni.replay.yaml index a4ca2bae174..55d23d08542 100644 --- a/tests/gold_tests/h3/replays/h3_sni.replay.yaml +++ b/tests/gold_tests/h3/replays/h3_sni.replay.yaml @@ -21,7 +21,7 @@ sessions: - protocol: stack: http3 tls: - sni: test_sni + sni: foo.com transactions: - client-request: @@ -31,7 +31,7 @@ sessions: - [ Content-Length, 0 ] - [:method, GET] - [:scheme, https] - - [:authority, example.com] + - [:authority, foo.com] - [:path, /path/test1] - [ uuid, has_sni ] server-response: @@ -55,7 +55,7 @@ sessions: - [ Content-Length, 0 ] - [:method, GET] - [:scheme, https] - - [:authority, example.com] + - [:authority, foo.com] - [:path, /path/test1] - [ uuid, no_sni ] server-response: diff --git a/tests/gold_tests/h3/replays/h3_stream_lifetime.replay.yaml b/tests/gold_tests/h3/replays/h3_stream_lifetime.replay.yaml new file mode 100644 index 00000000000..b263bea5277 --- /dev/null +++ b/tests/gold_tests/h3/replays/h3_stream_lifetime.replay.yaml @@ -0,0 +1,198 @@ +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under +# the Apache License, Version 2.0 (the "License"); you may not +# use this file except in compliance with the License. You may +# obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +meta: + version: "1.0" + +autest: + description: "Verify HTTP/3 stream lifetime handling with concurrent streams" + + server: + name: "server-h3-stream-lifetime" + process_config: + verbose: false + + client: + name: "client-h3-stream-lifetime" + process_config: + verbose: false + + ats: + name: "ts-h3-stream-lifetime" + startup_timeout: 60 + process_config: + enable_tls: true + enable_quic: true + enable_cache: false + + records_config: + proxy.config.diags.debug.enabled: 1 + proxy.config.diags.debug.tags: "quic|http3|v_http3_trans" + proxy.config.quic.server.stateless_retry_enabled: 0 + + remap_config: + - from: "https://example.com/" + to: "http://127.0.0.1:{SERVER_HTTP_PORT}/" + + log_validation: + traffic_out: + contains: + - expression: 'start HTTP/3 app \(ALPN=h3\)' + description: "ATS should negotiate HTTP/3" + +sessions: +- protocol: + - name: http + version: 3 + - name: tls + sni: example.com + - name: udp + - name: ip + + transactions: + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-slow ] + - [ uuid, h3-slow ] + + server-response: + delay: 500ms + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "slow-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvw" + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "slow-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvw" + verify: { as: equal } + + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-empty ] + - [ uuid, h3-empty ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "0" ] + content: + size: 0 + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "0", as: equal } ] + + - client-request: + version: "3" + headers: + fields: + - [ ":method", GET ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-small ] + - [ uuid, h3-small ] + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "small-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuv" + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "small-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuv" + verify: { as: equal } + + - client-request: + version: "3" + headers: + fields: + - [ ":method", POST ] + - [ ":scheme", https ] + - [ ":authority", example.com ] + - [ ":path", /h3-post ] + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + - [ uuid, h3-post ] + content: + encoding: plain + data: "post-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvw" + + proxy-request: + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "post-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmnopqrstuvw" + verify: { as: equal } + + server-response: + status: 200 + reason: OK + headers: + fields: + - [ Content-Type, text/plain ] + - [ Content-Length, "100" ] + content: + encoding: plain + data: "post-response-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmn" + + proxy-response: + status: 200 + headers: + fields: + - [ Content-Length, { value: "100", as: equal } ] + content: + encoding: plain + data: "post-response-0123456789abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789abcdefghijklmn" + verify: { as: equal } diff --git a/tests/gold_tests/timeout/active_timeout.test.py b/tests/gold_tests/timeout/active_timeout.test.py index 5227cc0e93c..f180340165a 100644 --- a/tests/gold_tests/timeout/active_timeout.test.py +++ b/tests/gold_tests/timeout/active_timeout.test.py @@ -66,7 +66,7 @@ tr3.MakeCurlCommand('-k -i --http2 https://127.0.0.1:{0}/file'.format(ts.Variables.ssl_port), ts=ts) tr3.Processes.Default.Streams.stdout = Testers.ContainsExpression("Activity Timeout", "Request should fail with active timeout") - if Condition.HasATSFeature('TS_HAS_QUICHE') and Condition.HasCurlFeature('http3'): + if Condition.HasATSFeature('TS_USE_QUIC') and Condition.HasCurlFeature('http3'): tr4 = Test.AddTestRun("tr") tr4.MakeCurlCommand('-k -i --http3 https://localhost:{0}/file'.format(ts.Variables.ssl_port), ts=ts) tr4.Processes.Default.Streams.stdout = Testers.ContainsExpression( diff --git a/tests/gold_tests/timeout/quic_no_activity_timeout.test.py b/tests/gold_tests/timeout/quic_no_activity_timeout.test.py index f50146f5bea..19b330097e2 100644 --- a/tests/gold_tests/timeout/quic_no_activity_timeout.test.py +++ b/tests/gold_tests/timeout/quic_no_activity_timeout.test.py @@ -16,7 +16,7 @@ Test.Summary = 'Basic checks on QUIC max_idle_timeout set by ts.quic.no_activity_timeout_in' -Test.SkipUnless(Condition.HasATSFeature('TS_HAS_QUICHE'), Condition.HasCurlFeature('http3')) +Test.SkipUnless(Condition.HasATSFeature('TS_USE_QUIC')) class Test_quic_no_activity_timeout: @@ -117,18 +117,19 @@ def run(self, check_for_max_idle_timeout=False): replay_keys="nodelays") test0.run() -test1 = Test_quic_no_activity_timeout( - "Test ts.quic.no_activity_timeout_in(quic max_idle_timeout) with a 5s delay", - no_activity_timeout_in=3000, # 3s `max_idle_timeout` - replay_keys="delay5s", - gold_file="gold/quic_no_activity_timeout.gold") -test1.run(check_for_max_idle_timeout=True) - -# QUIC Ignores the default_inactivity_timeout config, so the ts.quic.no_activity_timeout_in -# should be honor -test2 = Test_quic_no_activity_timeout( - "Ignoring default_inactivity_timeout and use the ts.quic.no_activity_timeout_in instead", - replay_keys="delay5s", - no_activity_timeout_in=3000, - extra_recs={'proxy.config.net.default_inactivity_timeout': 1}) -test2.run(check_for_max_idle_timeout=True) +if Condition.HasATSFeature('TS_HAS_QUICHE'): + test1 = Test_quic_no_activity_timeout( + "Test ts.quic.no_activity_timeout_in(quic max_idle_timeout) with a 5s delay", + no_activity_timeout_in=3000, # 3s `max_idle_timeout` + replay_keys="delay5s", + gold_file="gold/quic_no_activity_timeout.gold") + test1.run(check_for_max_idle_timeout=True) + + # QUIC Ignores the default_inactivity_timeout config, so the ts.quic.no_activity_timeout_in + # should be honored + test2 = Test_quic_no_activity_timeout( + "Ignoring default_inactivity_timeout and use the ts.quic.no_activity_timeout_in instead", + replay_keys="delay5s", + no_activity_timeout_in=3000, + extra_recs={'proxy.config.net.default_inactivity_timeout': 1}) + test2.run(check_for_max_idle_timeout=True) diff --git a/tests/pyproject.toml b/tests/pyproject.toml index d055f938c73..f976251df3b 100644 --- a/tests/pyproject.toml +++ b/tests/pyproject.toml @@ -52,6 +52,7 @@ dependencies = [ "pyOpenSSL", "eventlet", + "aioquic==1.3.0", # To test stats_over_http prometheus exporter. "prometheus_client", @@ -62,4 +63,3 @@ dev = [ "pyflakes", ] - From b81077623f29b44aefb99f9a4d1112d6c60dbc20 Mon Sep 17 00:00:00 2001 From: bneradt Date: Wed, 20 May 2026 15:30:31 -0500 Subject: [PATCH 2/3] HTTP/3 via OpenSSL 3.5 + quiche Fedora now ships OpenSSL 3.5 with the third-party QUIC TLS callback API, but quiche still links against the older quictls/BoringSSL symbols. ATS therefore could not use the system OpenSSL library for downstream HTTP/3 without dragging in a different TLS stack. This adds CMake detection for the OpenSSL callback API and provides a private compatibility layer that maps quiche's legacy hooks onto SSL_set_quic_tls_cbs. This requires static quiche in that mode so ATS resolves the shim symbols locally and links the final binaries against the system OpenSSL libraries. This also relaxes verifier-only HTTP/3 AuTest gates that do not execute curl, so those tests can run when ATS has QUIC support but the installed curl lacks HTTP/3. --- CMakeLists.txt | 25 +- cmake/CheckOpenSSLHasQuicTlsCbs.cmake | 55 +++ cmake/Findquiche.cmake | 27 ++ src/iocore/net/quic/CMakeLists.txt | 11 + src/iocore/net/quic/OpenSSLQuicCompat.cc | 493 +++++++++++++++++++++++ 5 files changed, 609 insertions(+), 2 deletions(-) create mode 100644 cmake/CheckOpenSSLHasQuicTlsCbs.cmake create mode 100644 src/iocore/net/quic/OpenSSLQuicCompat.cc diff --git a/CMakeLists.txt b/CMakeLists.txt index fcd8b2eef13..4a0112c76b6 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -280,6 +280,7 @@ pkg_check_modules(PCRE2 REQUIRED IMPORTED_TARGET libpcre2-8) include(CheckOpenSSLIsBoringSSL) include(CheckOpenSSLIsQuictls) include(CheckOpenSSLIsAwsLc) +include(CheckOpenSSLHasQuicTlsCbs) find_package(OpenSSL REQUIRED) check_openssl_is_boringssl(SSLLIB_IS_BORINGSSL BORINGSSL_VERSION "${OPENSSL_INCLUDE_DIR}") check_openssl_is_awslc(SSLLIB_IS_AWSLC AWSLC_VERSION "${OPENSSL_INCLUDE_DIR}") @@ -309,6 +310,16 @@ if(OPENSSL_VERSION VERSION_GREATER_EQUAL "3.0.0") add_compile_definitions(OPENSSL_API_COMPAT=10002 OPENSSL_IS_OPENSSL3) endif() +check_openssl_has_quic_tls_cbs(SSLLIB_HAS_QUIC_TLS_CBS "${OPENSSL_INCLUDE_DIR}") +if(SSLLIB_HAS_QUIC_TLS_CBS + AND NOT SSLLIB_IS_BORINGSSL + AND NOT SSLLIB_IS_AWSLC + AND NOT SSLLIB_IS_QUICTLS +) + set(TS_OPENSSL_QUIC_TLS_CBS_COMPAT TRUE) + add_compile_definitions(TS_OPENSSL_QUIC_TLS_CBS_COMPAT) +endif() + if(ENABLE_PROFILER) find_package(profiler REQUIRED) set(TS_HAS_PROFILER ${profiler_FOUND}) @@ -325,12 +336,20 @@ elseif(TS_HAS_MIMALLOC) endif() if(ENABLE_QUICHE) + if(TS_OPENSSL_QUIC_TLS_CBS_COMPAT) + set(quiche_USE_STATIC TRUE) + endif() find_package(quiche REQUIRED) set(TS_HAS_QUICHE ${quiche_FOUND}) set(TS_USE_QUIC ${TS_HAS_QUICHE}) - if(NOT SSLLIB_IS_BORINGSSL AND NOT SSLLIB_IS_QUICTLS) - message(FATAL_ERROR "Use of BoringSSL or OPENSSL/QUICTLS is required if quiche is used.") + if(NOT SSLLIB_IS_BORINGSSL + AND NOT SSLLIB_IS_QUICTLS + AND NOT TS_OPENSSL_QUIC_TLS_CBS_COMPAT + ) + message( + FATAL_ERROR "Use of BoringSSL, OPENSSL/QUICTLS, or OpenSSL QUIC TLS callbacks is required if quiche is used." + ) endif() if(SSLLIB_IS_QUICTLS) @@ -339,6 +358,8 @@ if(ENABLE_QUICHE) message( "WARNING - Using quictls requires using a special version of quiche. Make sure quictls is supported in quiche." ) + elseif(TS_OPENSSL_QUIC_TLS_CBS_COMPAT) + message(STATUS "Using OpenSSL QUIC TLS callbacks compatibility for quiche") endif() endif() diff --git a/cmake/CheckOpenSSLHasQuicTlsCbs.cmake b/cmake/CheckOpenSSLHasQuicTlsCbs.cmake new file mode 100644 index 00000000000..3aec5cf4cee --- /dev/null +++ b/cmake/CheckOpenSSLHasQuicTlsCbs.cmake @@ -0,0 +1,55 @@ +####################### +# +# Licensed to the Apache Software Foundation (ASF) under one or more contributor license +# agreements. See the NOTICE file distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License +# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +# or implied. See the License for the specific language governing permissions and limitations under +# the License. +# +####################### + +function(CHECK_OPENSSL_HAS_QUIC_TLS_CBS OUT_VAR OPENSSL_INCLUDE_DIR) + set(CHECK_PROGRAM + " + #include + #include + + int main() { + OSSL_FUNC_SSL_QUIC_TLS_crypto_send_fn *crypto_send = nullptr; + OSSL_FUNC_SSL_QUIC_TLS_crypto_recv_rcd_fn *crypto_recv = nullptr; + OSSL_FUNC_SSL_QUIC_TLS_crypto_release_rcd_fn *crypto_release = nullptr; + OSSL_FUNC_SSL_QUIC_TLS_yield_secret_fn *yield_secret = nullptr; + OSSL_FUNC_SSL_QUIC_TLS_got_transport_params_fn *got_transport_params = nullptr; + OSSL_FUNC_SSL_QUIC_TLS_alert_fn *alert = nullptr; + const OSSL_DISPATCH callbacks[] = { + {OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_SEND, reinterpret_cast(crypto_send)}, + {OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_RECV_RCD, reinterpret_cast(crypto_recv)}, + {OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_RELEASE_RCD, reinterpret_cast(crypto_release)}, + {OSSL_FUNC_SSL_QUIC_TLS_YIELD_SECRET, reinterpret_cast(yield_secret)}, + {OSSL_FUNC_SSL_QUIC_TLS_GOT_TRANSPORT_PARAMS, reinterpret_cast(got_transport_params)}, + {OSSL_FUNC_SSL_QUIC_TLS_ALERT, reinterpret_cast(alert)}, + {0, nullptr}, + }; + + return callbacks[0].function_id == 0 || + SSL_set_quic_tls_cbs == nullptr || + SSL_set_quic_tls_transport_params == nullptr; + } + " + ) + set(CMAKE_REQUIRED_INCLUDES "${OPENSSL_INCLUDE_DIR}") + set(CMAKE_REQUIRED_LIBRARIES OpenSSL::SSL OpenSSL::Crypto) + include(CheckCXXSourceCompiles) + check_cxx_source_compiles("${CHECK_PROGRAM}" ${OUT_VAR}) + set(${OUT_VAR} + ${${OUT_VAR}} + PARENT_SCOPE + ) +endfunction() diff --git a/cmake/Findquiche.cmake b/cmake/Findquiche.cmake index fcb3c61f276..4cead8bbdfb 100644 --- a/cmake/Findquiche.cmake +++ b/cmake/Findquiche.cmake @@ -28,7 +28,24 @@ # quiche::quiche # +if(quiche_USE_STATIC + AND quiche_LIBRARY + AND NOT quiche_LIBRARY MATCHES "\\${CMAKE_STATIC_LIBRARY_SUFFIX}$" +) + unset(quiche_LIBRARY CACHE) +endif() + +if(quiche_USE_STATIC) + set(_quiche_ORIGINAL_FIND_LIBRARY_SUFFIXES ${CMAKE_FIND_LIBRARY_SUFFIXES}) + set(CMAKE_FIND_LIBRARY_SUFFIXES ${CMAKE_STATIC_LIBRARY_SUFFIX}) +endif() + find_library(quiche_LIBRARY NAMES quiche) + +if(quiche_USE_STATIC) + set(CMAKE_FIND_LIBRARY_SUFFIXES ${_quiche_ORIGINAL_FIND_LIBRARY_SUFFIXES}) +endif() + find_path( quiche_INCLUDE_DIR NAMES quiche.h @@ -40,8 +57,18 @@ mark_as_advanced(quiche_FOUND quiche_LIBRARY quiche_INCLUDE_DIR) include(FindPackageHandleStandardArgs) find_package_handle_standard_args(quiche REQUIRED_VARS quiche_LIBRARY quiche_INCLUDE_DIR) +if(quiche_USE_STATIC + AND quiche_FOUND + AND NOT quiche_LIBRARY MATCHES "\\${CMAKE_STATIC_LIBRARY_SUFFIX}$" +) + message(FATAL_ERROR "Static quiche was requested, but ${quiche_LIBRARY} is not a static library") +endif() + if(quiche_FOUND) set(quiche_INCLUDE_DIRS "${quiche_INCLUDE_DIR}") + if(quiche_USE_STATIC) + message(STATUS "Using static quiche library: ${quiche_LIBRARY}") + endif() endif() if(quiche_FOUND AND NOT TARGET quiche::quiche) diff --git a/src/iocore/net/quic/CMakeLists.txt b/src/iocore/net/quic/CMakeLists.txt index 4151917c017..b3f1b7cb1d4 100644 --- a/src/iocore/net/quic/CMakeLists.txt +++ b/src/iocore/net/quic/CMakeLists.txt @@ -34,3 +34,14 @@ add_library( add_library(ts::quic ALIAS quic) target_link_libraries(quic PUBLIC ts::inkevent ts::inknet ts::tscore OpenSSL::Crypto OpenSSL::SSL quiche::quiche) + +if(TS_OPENSSL_QUIC_TLS_CBS_COMPAT) + add_library(openssl_quic_compat STATIC OpenSSLQuicCompat.cc) + add_library(ts::openssl_quic_compat ALIAS openssl_quic_compat) + target_link_libraries(openssl_quic_compat PUBLIC OpenSSL::SSL OpenSSL::Crypto) + set_property( + TARGET quiche::quiche + APPEND + PROPERTY INTERFACE_LINK_LIBRARIES ts::openssl_quic_compat + ) +endif() diff --git a/src/iocore/net/quic/OpenSSLQuicCompat.cc b/src/iocore/net/quic/OpenSSLQuicCompat.cc new file mode 100644 index 00000000000..7afafff280e --- /dev/null +++ b/src/iocore/net/quic/OpenSSLQuicCompat.cc @@ -0,0 +1,493 @@ +/** @file + + Bridges quiche's legacy QUIC TLS callback API to OpenSSL 3.5's third-party + QUIC TLS callback API. + + This compatibility layer exports the quictls/BoringSSL-style symbols that + quiche expects while ATS links against upstream OpenSSL 3.5. It stores + per-SSL callback state in OpenSSL ex-data and translates CRYPTO data, + encryption secrets, alerts, and transport parameters between the two APIs. + + Licensed to the Apache Software Foundation (ASF) under one or more contributor license + agreements. See the NOTICE file distributed with this work for additional information regarding + copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 + (the "License"); you may not use this file except in compliance with the License. You may obtain + a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software distributed under the License + is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express + or implied. See the License for the specific language governing permissions and limitations under + the License. + */ + +#include +#include +#include + +#include +#include +#include +#include +#include +#include + +enum ssl_encryption_level_t { + ssl_encryption_initial = 0, + ssl_encryption_early_data, + ssl_encryption_handshake, + ssl_encryption_application, +}; + +struct ssl_quic_method_st { + int (*set_encryption_secrets)(SSL *ssl, ssl_encryption_level_t level, uint8_t const *read_secret, uint8_t const *write_secret, + size_t secret_len); + int (*add_handshake_data)(SSL *ssl, ssl_encryption_level_t level, uint8_t const *data, size_t len); + int (*flush_flight)(SSL *ssl); + int (*send_alert)(SSL *ssl, ssl_encryption_level_t level, uint8_t alert); +}; + +using SSL_QUIC_METHOD = ssl_quic_method_st; + +namespace +{ + +constexpr auto read_secret_direction = 0; +constexpr auto write_secret_direction = 1; +constexpr auto quic_level_count = 4; + +struct PendingSecret { + std::vector read; + std::vector write; + bool have_read{false}; + bool have_write{false}; + bool delivered{false}; +}; + +struct QuicCompatState { + SSL_QUIC_METHOD const *method{nullptr}; + std::array>, quic_level_count> crypto_data; + std::array secrets; + ssl_encryption_level_t read_level{ssl_encryption_initial}; + ssl_encryption_level_t write_level{ssl_encryption_initial}; + ssl_encryption_level_t active_recv_level{ssl_encryption_initial}; + bool active_recv{false}; + std::vector local_transport_params; + std::vector peer_transport_params; +}; + +void +free_quic_ex_data(void *, void *ptr, CRYPTO_EX_DATA *, int, long, void *) +{ + delete static_cast(ptr); +} + +int +quic_ex_data_index() +{ + static int index = SSL_get_ex_new_index(0, nullptr, nullptr, nullptr, free_quic_ex_data); + + return index; +} + +bool +is_valid_level(ssl_encryption_level_t level) +{ + auto index = static_cast(level); + + return index >= 0 && index < quic_level_count; +} + +size_t +level_index(ssl_encryption_level_t level) +{ + return static_cast(level); +} + +uint8_t const * +data_or_null(std::vector const &data) +{ + return data.empty() ? nullptr : data.data(); +} + +bool +assign_bytes(std::vector &dst, uint8_t const *data, size_t len) +{ + if (len == 0) { + dst.clear(); + return true; + } + if (data == nullptr) { + return false; + } + + dst.assign(data, data + len); + return true; +} + +QuicCompatState * +get_state(SSL const *ssl) +{ + if (ssl == nullptr) { + return nullptr; + } + + int const index = quic_ex_data_index(); + return index < 0 ? nullptr : static_cast(SSL_get_ex_data(ssl, index)); +} + +QuicCompatState * +get_or_create_state(SSL *ssl) +{ + if (ssl == nullptr) { + return nullptr; + } + + if (auto *state = get_state(ssl); state != nullptr) { + return state; + } + + auto *state = new QuicCompatState; + int const index = quic_ex_data_index(); + if (index < 0 || SSL_set_ex_data(ssl, index, state) != 1) { + delete state; + return nullptr; + } + + return state; +} + +bool +compat_level_from_protection(uint32_t prot_level, ssl_encryption_level_t &level) +{ + switch (prot_level) { + case OSSL_RECORD_PROTECTION_LEVEL_NONE: + level = ssl_encryption_initial; + return true; + case OSSL_RECORD_PROTECTION_LEVEL_EARLY: + level = ssl_encryption_early_data; + return true; + case OSSL_RECORD_PROTECTION_LEVEL_HANDSHAKE: + level = ssl_encryption_handshake; + return true; + case OSSL_RECORD_PROTECTION_LEVEL_APPLICATION: + level = ssl_encryption_application; + return true; + default: + return false; + } +} + +bool +deliver_0rtt_secret(SSL *ssl, QuicCompatState &state, ssl_encryption_level_t level, int direction, + std::vector const &secret) +{ + if (state.method == nullptr || state.method->set_encryption_secrets == nullptr) { + return false; + } + + bool const is_server = SSL_is_server(ssl) == 1; + if (direction == read_secret_direction && is_server) { + return state.method->set_encryption_secrets(ssl, level, data_or_null(secret), nullptr, secret.size()) == 1; + } + + if (direction == write_secret_direction && !is_server) { + return state.method->set_encryption_secrets(ssl, level, nullptr, data_or_null(secret), secret.size()) == 1; + } + + return true; +} + +bool +deliver_paired_secrets(SSL *ssl, QuicCompatState &state, ssl_encryption_level_t level) +{ + if (state.method == nullptr || state.method->set_encryption_secrets == nullptr) { + return false; + } + + auto &pending = state.secrets[level_index(level)]; + if (!pending.have_read || !pending.have_write || pending.delivered) { + return true; + } + if (pending.read.size() != pending.write.size()) { + return false; + } + + int const result = + state.method->set_encryption_secrets(ssl, level, data_or_null(pending.read), data_or_null(pending.write), pending.read.size()); + if (result == 1) { + pending.delivered = true; + } + + return result == 1; +} + +int +crypto_send_cb(SSL *ssl, unsigned char const *buf, size_t buf_len, size_t *consumed, void *) +{ + auto *state = get_state(ssl); + if (state == nullptr || state->method == nullptr || state->method->add_handshake_data == nullptr) { + return 0; + } + + if (consumed != nullptr) { + *consumed = 0; + } + + static constexpr uint8_t empty_data = 0; + if (buf == nullptr && buf_len > 0) { + return 0; + } + + auto const *data = buf_len == 0 ? &empty_data : reinterpret_cast(buf); + if (state->method->add_handshake_data(ssl, state->write_level, data, buf_len) != 1) { + return 0; + } + if (state->method->flush_flight != nullptr && state->method->flush_flight(ssl) != 1) { + return 0; + } + + if (consumed != nullptr) { + *consumed = buf_len; + } + + return 1; +} + +int +crypto_recv_rcd_cb(SSL *ssl, unsigned char const **buf, size_t *bytes_read, void *) +{ + auto *state = get_state(ssl); + if (state == nullptr || buf == nullptr || bytes_read == nullptr) { + return 0; + } + + *buf = nullptr; + *bytes_read = 0; + + if (state->active_recv) { + auto &active_queue = state->crypto_data[level_index(state->active_recv_level)]; + if (!active_queue.empty()) { + *buf = active_queue.front().data(); + *bytes_read = active_queue.front().size(); + } + return 1; + } + + auto &queue = state->crypto_data[level_index(state->read_level)]; + if (queue.empty()) { + return 1; + } + + state->active_recv = true; + state->active_recv_level = state->read_level; + *buf = queue.front().data(); + *bytes_read = queue.front().size(); + + return 1; +} + +int +crypto_release_rcd_cb(SSL *ssl, size_t bytes_read, void *) +{ + auto *state = get_state(ssl); + if (state == nullptr || !state->active_recv) { + return 1; + } + + auto &queue = state->crypto_data[level_index(state->active_recv_level)]; + if (!queue.empty()) { + if (bytes_read >= queue.front().size()) { + queue.pop_front(); + } else { + queue.front().erase(queue.front().begin(), queue.front().begin() + bytes_read); + } + } + state->active_recv = false; + + return 1; +} + +int +yield_secret_cb(SSL *ssl, uint32_t prot_level, int direction, unsigned char const *secret, size_t secret_len, void *) +{ + auto *state = get_state(ssl); + if (state == nullptr) { + return 0; + } + + ssl_encryption_level_t level = ssl_encryption_initial; + if (!compat_level_from_protection(prot_level, level)) { + return 0; + } + + if (direction == read_secret_direction) { + state->read_level = level; + } else if (direction == write_secret_direction) { + state->write_level = level; + } else { + return 0; + } + + std::vector secret_copy; + if (!assign_bytes(secret_copy, reinterpret_cast(secret), secret_len)) { + return 0; + } + + if (level == ssl_encryption_early_data) { + return deliver_0rtt_secret(ssl, *state, level, direction, secret_copy) ? 1 : 0; + } + + auto &pending = state->secrets[level_index(level)]; + if (direction == read_secret_direction) { + pending.read = std::move(secret_copy); + pending.have_read = true; + } else { + pending.write = std::move(secret_copy); + pending.have_write = true; + } + pending.delivered = false; + + return deliver_paired_secrets(ssl, *state, level) ? 1 : 0; +} + +int +got_transport_params_cb(SSL *ssl, unsigned char const *params, size_t params_len, void *) +{ + auto *state = get_state(ssl); + if (state == nullptr) { + return 0; + } + + if (!assign_bytes(state->peer_transport_params, reinterpret_cast(params), params_len)) { + return 0; + } + + return 1; +} + +int +alert_cb(SSL *ssl, unsigned char alert_code, void *) +{ + auto *state = get_state(ssl); + if (state == nullptr || state->method == nullptr || state->method->send_alert == nullptr) { + return 0; + } + + return state->method->send_alert(ssl, state->write_level, alert_code); +} + +OSSL_DISPATCH const quic_tls_callbacks[] = { + {OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_SEND, reinterpret_cast(crypto_send_cb) }, + {OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_RECV_RCD, reinterpret_cast(crypto_recv_rcd_cb) }, + {OSSL_FUNC_SSL_QUIC_TLS_CRYPTO_RELEASE_RCD, reinterpret_cast(crypto_release_rcd_cb) }, + {OSSL_FUNC_SSL_QUIC_TLS_YIELD_SECRET, reinterpret_cast(yield_secret_cb) }, + {OSSL_FUNC_SSL_QUIC_TLS_GOT_TRANSPORT_PARAMS, reinterpret_cast(got_transport_params_cb)}, + {OSSL_FUNC_SSL_QUIC_TLS_ALERT, reinterpret_cast(alert_cb) }, + {0, nullptr }, +}; + +} // namespace + +extern "C" int +SSL_set_quic_method(SSL *ssl, SSL_QUIC_METHOD const *quic_method) +{ + auto *state = get_or_create_state(ssl); + if (state == nullptr) { + return 0; + } + + state->method = quic_method; + + return SSL_set_quic_tls_cbs(ssl, quic_tls_callbacks, nullptr); +} + +extern "C" int +SSL_set_quic_transport_params(SSL *ssl, uint8_t const *params, size_t params_len) +{ + auto *state = get_or_create_state(ssl); + if (state == nullptr) { + return 0; + } + + if (!assign_bytes(state->local_transport_params, params, params_len)) { + return 0; + } + return SSL_set_quic_tls_transport_params(ssl, data_or_null(state->local_transport_params), state->local_transport_params.size()); +} + +extern "C" void +SSL_get_peer_quic_transport_params(SSL const *ssl, uint8_t const **out_params, size_t *out_params_len) +{ + auto const *state = get_state(ssl); + if (out_params != nullptr) { + *out_params = state == nullptr ? nullptr : data_or_null(state->peer_transport_params); + } + if (out_params_len != nullptr) { + *out_params_len = state == nullptr ? 0 : state->peer_transport_params.size(); + } +} + +extern "C" ssl_encryption_level_t +SSL_quic_write_level(SSL const *ssl) +{ + auto const *state = get_state(ssl); + + return state == nullptr ? ssl_encryption_initial : state->write_level; +} + +extern "C" int +SSL_provide_quic_data(SSL *ssl, ssl_encryption_level_t level, uint8_t const *data, size_t len) +{ + auto *state = get_or_create_state(ssl); + if (state == nullptr || !is_valid_level(level)) { + return 0; + } + + auto &records = state->crypto_data[level_index(level)]; + records.emplace_back(); + if (!assign_bytes(records.back(), data, len)) { + records.pop_back(); + return 0; + } + + return 1; +} + +extern "C" int +SSL_process_quic_post_handshake(SSL *ssl) +{ + auto *state = get_state(ssl); + if (state == nullptr) { + return 0; + } + + bool has_crypto_data = false; + for (auto const &queue : state->crypto_data) { + if (!queue.empty()) { + has_crypto_data = true; + break; + } + } + if (!has_crypto_data) { + return 1; + } + + unsigned char data = 0; + size_t bytes_read = 0; + int const result = SSL_read_ex(ssl, &data, 0, &bytes_read); + + if (result == 1) { + return 1; + } + + int const error = SSL_get_error(ssl, result); + + return error == SSL_ERROR_WANT_READ || error == SSL_ERROR_WANT_WRITE; +} + +extern "C" void +SSL_set_quic_use_legacy_codepoint(SSL *, int) +{ +} From 276fef05eb4e37b4a60ee524e489c4ee8750d144 Mon Sep 17 00:00:00 2001 From: bneradt Date: Tue, 26 May 2026 15:43:32 -0500 Subject: [PATCH 3/3] HTTP/3 via OpenSSL QUIC OpenSSL 3.5 can terminate QUIC connections directly, but ATS only had a quiche-backed HTTP/3 listener. Operators who want to use the system OpenSSL QUIC stack needed a separate downstream backend without changing the existing quiche path or origin HTTP/3 scope. This adds an optional ENABLE_OPENSSL_QUIC backend that uses OpenSSL's native QUIC listener and stream APIs for downstream HTTP/3. This keeps the backend mutually exclusive with quiche, exposes TS_HAS_OPENSSL_QUIC, and shares ATS's existing HTTP/3 stream handling above the transport. This also installs native-QUIC TLS callbacks for ALPN and SNI certificate selection before ATS has a QUIC NetVC to bind. OpenSSL native QUIC does not make a selected SSL_CTX certificate active via SSL_set_SSL_CTX alone, so this applies the selected cert, key, and chain to the connection SSL. This also broadens client-side HTTP/3 tests to run with either backend and hardens transaction cleanup when OpenSSL closes stream state before ATS finishes teardown. This caches stream identifiers, declines listener-time QUIC tickets until a NetVC is bound, and adds focused H3 lifecycle and session-ticket coverage. --- CMakeLists.txt | 23 + CMakePresets.json | 13 +- cmake/CheckOpenSSLHasNativeQuic.cmake | 59 + .../iocore/net/QUICMultiCertConfigLoader.h | 14 +- include/iocore/net/TLSSNISupport.h | 1 + include/iocore/net/quic/QUICConfig.h | 7 + include/iocore/net/quic/QUICStream.h | 20 +- include/iocore/net/quic/QUICStreamManager.h | 10 +- include/proxy/http3/Http3Frame.h | 3 + include/proxy/http3/Http3HeaderVIOAdaptor.h | 1 - include/tscore/ink_config.h.cmake.in | 1 + src/iocore/net/CMakeLists.txt | 22 +- src/iocore/net/OpenSSLQUICNetProcessor.cc | 125 ++ src/iocore/net/OpenSSLQUICNetVConnection.cc | 1062 +++++++++++++++++ src/iocore/net/OpenSSLQUICPacketHandler.cc | 629 ++++++++++ src/iocore/net/P_QUICNetProcessor.h | 10 +- src/iocore/net/P_QUICNetVConnection.h | 56 +- src/iocore/net/P_QUICPacketHandler.h | 22 +- src/iocore/net/QUICMultiCertConfigLoader.cc | 252 +++- src/iocore/net/QUICNetVConnection.cc | 29 +- src/iocore/net/SSLSessionTicket.cc | 8 + src/iocore/net/TLSSNISupport.cc | 10 + src/iocore/net/quic/CMakeLists.txt | 9 +- src/iocore/net/quic/QUICConfig.cc | 23 + src/iocore/net/quic/QUICGlobals.cc | 3 +- src/iocore/net/quic/QUICStream.cc | 56 +- src/iocore/net/quic/QUICStreamManager.cc | 28 +- src/proxy/http3/Http3Frame.cc | 6 + .../http3/test/test_Http3FrameDispatcher.cc | 2 +- src/traffic_layout/info.cc | 1 + src/traffic_server/traffic_server.cc | 2 +- tests/gold_tests/headers/via.test.py | 2 +- 32 files changed, 2434 insertions(+), 75 deletions(-) create mode 100644 cmake/CheckOpenSSLHasNativeQuic.cmake create mode 100644 src/iocore/net/OpenSSLQUICNetProcessor.cc create mode 100644 src/iocore/net/OpenSSLQUICNetVConnection.cc create mode 100644 src/iocore/net/OpenSSLQUICPacketHandler.cc diff --git a/CMakeLists.txt b/CMakeLists.txt index 4a0112c76b6..527889b84ef 100644 --- a/CMakeLists.txt +++ b/CMakeLists.txt @@ -212,6 +212,7 @@ set(ENABLE_TPROXY 'X' where X is a number to use as the IP_TRANSPARENT sockopt, anything else to enable." ) +option(ENABLE_OPENSSL_QUIC "Use OpenSSL native QUIC (default OFF)") option(ENABLE_QUICHE "Use quiche (default OFF)") option(ENABLE_EXAMPLE "Build example directory (default OFF)") @@ -281,6 +282,7 @@ include(CheckOpenSSLIsBoringSSL) include(CheckOpenSSLIsQuictls) include(CheckOpenSSLIsAwsLc) include(CheckOpenSSLHasQuicTlsCbs) +include(CheckOpenSSLHasNativeQuic) find_package(OpenSSL REQUIRED) check_openssl_is_boringssl(SSLLIB_IS_BORINGSSL BORINGSSL_VERSION "${OPENSSL_INCLUDE_DIR}") check_openssl_is_awslc(SSLLIB_IS_AWSLC AWSLC_VERSION "${OPENSSL_INCLUDE_DIR}") @@ -320,6 +322,8 @@ if(SSLLIB_HAS_QUIC_TLS_CBS add_compile_definitions(TS_OPENSSL_QUIC_TLS_CBS_COMPAT) endif() +check_openssl_has_native_quic(SSLLIB_HAS_NATIVE_QUIC "${OPENSSL_INCLUDE_DIR}") + if(ENABLE_PROFILER) find_package(profiler REQUIRED) set(TS_HAS_PROFILER ${profiler_FOUND}) @@ -335,6 +339,25 @@ elseif(TS_HAS_MIMALLOC) link_libraries(mimalloc) endif() +if(ENABLE_OPENSSL_QUIC AND ENABLE_QUICHE) + message(FATAL_ERROR "ENABLE_OPENSSL_QUIC and ENABLE_QUICHE are mutually exclusive QUIC backends") +endif() + +if(ENABLE_OPENSSL_QUIC) + if(NOT SSLLIB_HAS_NATIVE_QUIC) + message(FATAL_ERROR "OpenSSL native QUIC support requires OpenSSL 3.5 or newer with OSSL_QUIC_server_method") + endif() + if(SSLLIB_IS_BORINGSSL + OR SSLLIB_IS_AWSLC + OR SSLLIB_IS_QUICTLS + ) + message(FATAL_ERROR "OpenSSL native QUIC support requires upstream OpenSSL 3.5 or newer") + endif() + set(TS_HAS_OPENSSL_QUIC TRUE) + set(TS_USE_QUIC TRUE) + message(STATUS "Using OpenSSL native QUIC") +endif() + if(ENABLE_QUICHE) if(TS_OPENSSL_QUIC_TLS_CBS_COMPAT) set(quiche_USE_STATIC TRUE) diff --git a/CMakePresets.json b/CMakePresets.json index 3fe7e010b85..41aaa8cf1b0 100644 --- a/CMakePresets.json +++ b/CMakePresets.json @@ -183,11 +183,10 @@ "inherits": ["ci"], "cacheVariables": { "ENABLE_PROBES": "ON", - "OPENSSL_ROOT_DIR": "/opt/openssl-quic", "opentelemetry_ROOT": "/opt", - "CURL_ROOT": "/opt", "wamr_ROOT": "/opt", - "ENABLE_CRIPTS": "ON" + "ENABLE_CRIPTS": "ON", + "ENABLE_OPENSSL_QUIC": "ON" } }, { @@ -207,20 +206,19 @@ "name": "ci-fedora-quiche", "displayName": "CI Fedora Quiche", "description": "CI Pipeline config for Fedora Linux (quiche build)", - "inherits": ["ci"], + "inherits": ["ci-fedora"], "cacheVariables": { "OPENSSL_ROOT_DIR": "/opt/h3-tools-boringssl/boringssl", "quiche_ROOT": "/opt/h3-tools-boringssl/quiche", - "opentelemetry_ROOT": "/opt", "CURL_ROOT": "/opt", - "wamr_ROOT": "/opt", "CMAKE_INSTALL_PREFIX": "/tmp/ats-quiche", + "ENABLE_OPENSSL_QUIC": "OFF", "ENABLE_QUICHE": "ON" } }, { "name": "ci-fedora-autest", - "displayName": "CI Fedora Quiche Autest", + "displayName": "CI Fedora Autest", "description": "CI Pipeline config for Fedora Linux (autest build)", "inherits": ["ci-fedora", "autest"] }, @@ -450,4 +448,3 @@ } ] } - diff --git a/cmake/CheckOpenSSLHasNativeQuic.cmake b/cmake/CheckOpenSSLHasNativeQuic.cmake new file mode 100644 index 00000000000..28c96acb71c --- /dev/null +++ b/cmake/CheckOpenSSLHasNativeQuic.cmake @@ -0,0 +1,59 @@ +####################### +# +# Licensed to the Apache Software Foundation (ASF) under one or more contributor license +# agreements. See the NOTICE file distributed with this work for additional information regarding +# copyright ownership. The ASF licenses this file to you under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with the License. You may obtain +# a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software distributed under the License +# is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express +# or implied. See the License for the specific language governing permissions and limitations under +# the License. +# +####################### + +function(CHECK_OPENSSL_HAS_NATIVE_QUIC OUT_VAR OPENSSL_INCLUDE_DIR) + set(CHECK_PROGRAM + " + #include + #include + #include + + int main() { + const SSL_METHOD *method = OSSL_QUIC_server_method(); + SSL *ssl = nullptr; + uint64_t value = 0; + return method == nullptr || + SSL_new_listener == nullptr || + SSL_listen == nullptr || + SSL_handle_events == nullptr || + SSL_accept_connection == nullptr || + SSL_accept_stream == nullptr || + SSL_new_stream == nullptr || + SSL_stream_conclude == nullptr || + SSL_get_stream_id == nullptr || + SSL_get_stream_type == nullptr || + SSL_get_stream_read_state == nullptr || + SSL_get_stream_write_buf_avail(ssl, &value) || + SSL_get_conn_close_info == nullptr || + SSL_shutdown_ex == nullptr || + SSL_set_default_stream_mode == nullptr || + SSL_set_blocking_mode == nullptr || + SSL_set_event_handling_mode(ssl, SSL_VALUE_EVENT_HANDLING_MODE_EXPLICIT) || + SSL_set_feature_request_uint(ssl, SSL_VALUE_QUIC_IDLE_TIMEOUT, value) || + SSL_set_incoming_stream_policy == nullptr; + } + " + ) + set(CMAKE_REQUIRED_INCLUDES "${OPENSSL_INCLUDE_DIR}") + set(CMAKE_REQUIRED_LIBRARIES OpenSSL::SSL OpenSSL::Crypto) + include(CheckCXXSourceCompiles) + check_cxx_source_compiles("${CHECK_PROGRAM}" ${OUT_VAR}) + set(${OUT_VAR} + ${${OUT_VAR}} + PARENT_SCOPE + ) +endfunction() diff --git a/include/iocore/net/QUICMultiCertConfigLoader.h b/include/iocore/net/QUICMultiCertConfigLoader.h index 4dad3313a63..6ae299a510f 100644 --- a/include/iocore/net/QUICMultiCertConfigLoader.h +++ b/include/iocore/net/QUICMultiCertConfigLoader.h @@ -25,6 +25,9 @@ #include "iocore/net/SSLMultiCertConfigLoader.h" #include "iocore/eventsystem/ConfigProcessor.h" +#include "tscore/ink_config.h" + +#include class QUICCertConfig { @@ -40,6 +43,10 @@ class QUICCertConfig static int _config_id; }; +#if TS_HAS_OPENSSL_QUIC +std::string_view quic_sni_server_name(SSL *ssl); +#endif + class QUICMultiCertConfigLoader : public SSLMultiCertConfigLoader { public: @@ -48,7 +55,12 @@ class QUICMultiCertConfigLoader : public SSLMultiCertConfigLoader virtual SSL_CTX *default_server_ssl_ctx() override; private: - const char *_debug_tag() const override; + const char *_debug_tag() const override; +#if TS_HAS_OPENSSL_QUIC + virtual void _set_handshake_callbacks(SSL_CTX *ctx) override; + virtual bool _set_alpn_callback(SSL_CTX *ctx) override; + virtual bool _set_curves(SSL_CTX *ctx) override; +#endif virtual bool _setup_session_cache(SSL_CTX *ctx) override; virtual bool _set_cipher_suites_for_legacy_versions(SSL_CTX *ctx) override; virtual bool _set_info_callback(SSL_CTX *ctx) override; diff --git a/include/iocore/net/TLSSNISupport.h b/include/iocore/net/TLSSNISupport.h index 2ce9556f431..43538333457 100644 --- a/include/iocore/net/TLSSNISupport.h +++ b/include/iocore/net/TLSSNISupport.h @@ -122,6 +122,7 @@ class TLSSNISupport * @return True if the servername was set successfully, false otherwise. */ bool set_sni_server_name(SSL *ssl, char const *name); + void set_sni_server_name(std::string_view name); /** * Get the server name in SNI diff --git a/include/iocore/net/quic/QUICConfig.h b/include/iocore/net/quic/QUICConfig.h index 1ebec4f6aa0..b0b1ea25b2f 100644 --- a/include/iocore/net/quic/QUICConfig.h +++ b/include/iocore/net/quic/QUICConfig.h @@ -24,7 +24,11 @@ #pragma once #include +#include "tscore/ink_config.h" + +#if TS_HAS_QUICHE #include +#endif #include "iocore/eventsystem/ConfigProcessor.h" #include "iocore/net/SSLTypes.h" @@ -89,7 +93,9 @@ class QUICConfigParams : public ConfigInfo bool disable_http_0_9() const; +#if TS_HAS_QUICHE quiche_cc_algorithm get_cc_algorithm() const; +#endif private: static int _connection_table_size; @@ -163,3 +169,4 @@ class QUICConfig }; SSL_CTX *quic_new_ssl_ctx(); +SSL_CTX *quic_new_server_ssl_ctx(); diff --git a/include/iocore/net/quic/QUICStream.h b/include/iocore/net/quic/QUICStream.h index ef84c9cb6aa..e57c49b6dc5 100644 --- a/include/iocore/net/quic/QUICStream.h +++ b/include/iocore/net/quic/QUICStream.h @@ -31,11 +31,25 @@ #include "iocore/net/quic/QUICConnection.h" #include "iocore/net/quic/QUICDebugNames.h" -#include +#include +#include class QUICStreamAdapter; class QUICStreamStateListener; +class QUICStreamIO +{ +public: + using ErrorCode = uint64_t; + + virtual ~QUICStreamIO() = default; + + virtual int64_t read_stream(QUICStreamId stream_id, uint8_t *buf, size_t len, bool &fin, ErrorCode &error_code) = 0; + virtual bool stream_read_finished(QUICStreamId stream_id) = 0; + virtual int64_t stream_write_capacity(QUICStreamId stream_id) = 0; + virtual int64_t write_stream(QUICStreamId stream_id, uint8_t const *buf, size_t len, bool fin, ErrorCode &error_code) = 0; +}; + /** * @brief QUIC Stream * TODO: This is similar to Http2Stream. Need to think some integration. @@ -61,8 +75,8 @@ class QUICStream void stop_sending(QUICStreamErrorUPtr error); void reset(QUICStreamErrorUPtr error); - void receive_data(quiche_conn *quiche_con); - void send_data(quiche_conn *quiche_con); + void receive_data(QUICStreamIO &stream_io); + int64_t send_data(QUICStreamIO &stream_io); /* * QUICApplication need to call one of these functions when it process VC_EVENT_* diff --git a/include/iocore/net/quic/QUICStreamManager.h b/include/iocore/net/quic/QUICStreamManager.h index 78985a7d3c4..c79704a9b73 100644 --- a/include/iocore/net/quic/QUICStreamManager.h +++ b/include/iocore/net/quic/QUICStreamManager.h @@ -48,10 +48,10 @@ class QUICStreamManager : public QUICStreamStateListener QUICStream *create_stream(QUICStreamId stream_id, QUICConnectionError &err); - QUICConnectionErrorUPtr create_uni_stream(QUICStreamId new_stream_id); - QUICConnectionErrorUPtr create_bidi_stream(QUICStreamId new_stream_id); - QUICConnectionErrorUPtr delete_stream(QUICStreamId new_stream_id); - void reset_stream(QUICStreamId stream_id, QUICStreamErrorUPtr error); + virtual QUICConnectionErrorUPtr create_uni_stream(QUICStreamId &new_stream_id); + virtual QUICConnectionErrorUPtr create_bidi_stream(QUICStreamId &new_stream_id); + QUICConnectionErrorUPtr delete_stream(QUICStreamId new_stream_id); + void reset_stream(QUICStreamId stream_id, QUICStreamErrorUPtr error); void set_default_application(QUICApplication *app); @@ -63,4 +63,6 @@ class QUICStreamManager : public QUICStreamStateListener protected: QUICContext *_context = nullptr; QUICApplicationMap *_app_map = nullptr; + QUICStreamId _next_local_bidi_stream_id{0}; + QUICStreamId _next_local_uni_stream_id{0}; }; diff --git a/include/proxy/http3/Http3Frame.h b/include/proxy/http3/Http3Frame.h index cd8d5e860e2..a0255496385 100644 --- a/include/proxy/http3/Http3Frame.h +++ b/include/proxy/http3/Http3Frame.h @@ -98,6 +98,9 @@ class Http3DataFrame : public Http3Frame // Head of IOBufferBlock chain to send Ptr _whole_frame; uint64_t _payload_len = 0; + +protected: + bool _parse() override; }; // diff --git a/include/proxy/http3/Http3HeaderVIOAdaptor.h b/include/proxy/http3/Http3HeaderVIOAdaptor.h index 59a02b9a15e..13b30f2314d 100644 --- a/include/proxy/http3/Http3HeaderVIOAdaptor.h +++ b/include/proxy/http3/Http3HeaderVIOAdaptor.h @@ -28,7 +28,6 @@ #include "proxy/http3/Http3FrameHandler.h" class HQTransaction; - class Http3HeaderVIOAdaptor : public Continuation, public Http3FrameHandler { public: diff --git a/include/tscore/ink_config.h.cmake.in b/include/tscore/ink_config.h.cmake.in index 73c8b860fb9..3898e3e7dc1 100644 --- a/include/tscore/ink_config.h.cmake.in +++ b/include/tscore/ink_config.h.cmake.in @@ -145,6 +145,7 @@ const int DEFAULT_STACKSIZE = @DEFAULT_STACK_SIZE@; #cmakedefine01 TS_HAS_JEMALLOC #cmakedefine01 TS_HAS_MIMALLOC #cmakedefine01 TS_HAS_PROFILER +#cmakedefine01 TS_HAS_OPENSSL_QUIC #cmakedefine01 TS_HAS_QUICHE #cmakedefine01 TS_HAS_SO_MARK #cmakedefine01 TS_HAS_SO_PEERCRED diff --git a/src/iocore/net/CMakeLists.txt b/src/iocore/net/CMakeLists.txt index c5b66d8468c..a3c1e4efbc9 100644 --- a/src/iocore/net/CMakeLists.txt +++ b/src/iocore/net/CMakeLists.txt @@ -80,18 +80,17 @@ if(TS_USE_QUIC) add_subdirectory(quic) target_sources( - inknet - PRIVATE QUICClosedConCollector.cc - QUICMultiCertConfigLoader.cc - QUICNet.cc - QUICNetProcessor.cc - QUICNetVConnection.cc - QUICNextProtocolAccept.cc - QUICPacketHandler.cc - QUICSupport.cc + inknet PRIVATE QUICClosedConCollector.cc QUICMultiCertConfigLoader.cc QUICNextProtocolAccept.cc QUICSupport.cc ) - target_link_libraries(inknet PUBLIC quiche::quiche ts::quic) + if(TS_HAS_OPENSSL_QUIC) + target_sources(inknet PRIVATE OpenSSLQUICNetProcessor.cc OpenSSLQUICNetVConnection.cc OpenSSLQUICPacketHandler.cc) + elseif(TS_HAS_QUICHE) + target_sources(inknet PRIVATE QUICNet.cc QUICNetProcessor.cc QUICNetVConnection.cc QUICPacketHandler.cc) + target_link_libraries(inknet PUBLIC quiche::quiche) + endif() + + target_link_libraries(inknet PUBLIC ts::quic) endif() if(BUILD_REGRESSION_TESTING OR BUILD_TESTING) @@ -155,6 +154,9 @@ if(BUILD_TESTING) ) if(TS_USE_QUIC) list(APPEND LINK_GROUP_LIBS quic http3) + if(TS_HAS_QUICHE) + list(APPEND LINK_GROUP_LIBS quiche::quiche) + endif() endif() if(CMAKE_LINK_GROUP_USING_RESCAN_SUPPORTED OR CMAKE_CXX_LINK_GROUP_USING_RESCAN_SUPPORTED) string(JOIN "," LINK_GROUP_LIBS_CSV ${LINK_GROUP_LIBS}) diff --git a/src/iocore/net/OpenSSLQUICNetProcessor.cc b/src/iocore/net/OpenSSLQUICNetProcessor.cc new file mode 100644 index 00000000000..bbaa5625516 --- /dev/null +++ b/src/iocore/net/OpenSSLQUICNetProcessor.cc @@ -0,0 +1,125 @@ +/** @file + + OpenSSL native QUIC NetProcessor support. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "P_Net.h" +#include "P_QUICNetProcessor.h" +#include "P_QUICPacketHandler.h" +#include "P_QUICNetVConnection.h" +#include "P_UnixNet.h" +#include "iocore/net/quic/QUICConfig.h" +#include "iocore/net/quic/QUICGlobals.h" +#include "iocore/net/QUICMultiCertConfigLoader.h" + +#include + +QUICNetProcessor quic_NetProcessor; + +namespace +{ +DbgCtl dbg_ctl_quic_ps{"quic_ps"}; +DbgCtl dbg_ctl_iocore_net_processor{"iocore_net_processor"}; +} // end anonymous namespace + +QUICNetProcessor::QUICNetProcessor() {} + +QUICNetProcessor::~QUICNetProcessor() {} + +void +QUICNetProcessor::init() +{ +} + +int +QUICNetProcessor::start(int, size_t /* stacksize ATS_UNUSED */) +{ + QUIC::init(); + QUICConfig::startup(); + QUICCertConfig::startup(); + + return 0; +} + +NetAccept * +QUICNetProcessor::createNetAccept(NetProcessor::AcceptOptions const &opt) +{ + return new QUICPacketHandlerIn(opt); +} + +NetVConnection * +QUICNetProcessor::allocate_vc(EThread *t) +{ + QUICNetVConnection *vc = nullptr; + + if (t) { + vc = THREAD_ALLOC(quicNetVCAllocator, t); + new (vc) QUICNetVConnection(); + } else if (likely(vc = quicNetVCAllocator.alloc())) { + new (vc) QUICNetVConnection(); + vc->from_accept_thread = true; + } + + if (vc != nullptr) { + vc->ep.syscall = false; + } + + return vc; +} + +Action * +QUICNetProcessor::connect_re(Continuation *cont, sockaddr const * /* remote_addr ATS_UNUSED */, + NetVCOptions const & /* opt ATS_UNUSED */) +{ + Dbg(dbg_ctl_quic_ps, "OpenSSL native QUIC origin connections are not supported"); + cont->handleEvent(NET_EVENT_OPEN_FAILED, reinterpret_cast(-ENOTSUP)); + return ACTION_IO_ERROR; +} + +Action * +QUICNetProcessor::main_accept(Continuation *cont, SOCKET fd, AcceptOptions const &opt) +{ + Dbg(dbg_ctl_iocore_net_processor, "NetProcessor::main_accept - port %d,recv_bufsize %d, send_bufsize %d, sockopt 0x%0x", + opt.local_port, opt.recv_bufsize, opt.send_bufsize, opt.sockopt_flags); + + IpEndpoint accept_ip; + NetAccept *na = createNetAccept(opt); + + Metrics::Gauge::increment(net_rsb.accepts_currently_open); + + if (opt.localhost_only) { + accept_ip.setToLoopback(opt.ip_family); + } else if (opt.local_ip.isValid()) { + accept_ip.assign(opt.local_ip); + } else { + accept_ip.setToAnyAddr(opt.ip_family); + } + ink_assert(0 < opt.local_port && opt.local_port < 65536); + accept_ip.network_order_port() = htons(opt.local_port); + + na->server.sock = UnixSocket{fd}; + ats_ip_copy(&na->server.accept_addr, &accept_ip); + + na->action_ = new NetAcceptAction(cont, &na->server); + na->init_accept(); + + return na->action_.get(); +} diff --git a/src/iocore/net/OpenSSLQUICNetVConnection.cc b/src/iocore/net/OpenSSLQUICNetVConnection.cc new file mode 100644 index 00000000000..9a06fb94d26 --- /dev/null +++ b/src/iocore/net/OpenSSLQUICNetVConnection.cc @@ -0,0 +1,1062 @@ +/** @file + + OpenSSL native QUIC NetVConnection support. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "P_SSLUtils.h" +#include "P_QUICNetVConnection.h" +#include "P_QUICPacketHandler.h" +#include "P_UnixNet.h" +#include "api/APIHook.h" +#include "iocore/eventsystem/EThread.h" +#include "iocore/net/QUICMultiCertConfigLoader.h" +#include "iocore/net/SSLAPIHooks.h" +#include "iocore/net/quic/QUICApplicationMap.h" +#include "iocore/net/quic/QUICEvents.h" +#include "iocore/net/quic/QUICGlobals.h" +#include "iocore/net/quic/QUICStream.h" +#include "iocore/net/quic/QUICStreamManager.h" +#include "tscore/ink_config.h" + +#include +#include +#include + +#include +#include +#include +#include + +namespace +{ +constexpr ink_hrtime OPENSSL_QUIC_EVENT_INTERVAL = HRTIME_MSECONDS(2); + +DbgCtl dbg_ctl_quic_net{"quic_net"}; +DbgCtl dbg_ctl_v_quic_net{"v_quic_net"}; + +class OpenSSLQUICStreamManager : public QUICStreamManager +{ +public: + OpenSSLQUICStreamManager(QUICContext *context, QUICApplicationMap *app_map, QUICNetVConnection *vc) + : QUICStreamManager(context, app_map), _vc(vc) + { + } + + QUICConnectionErrorUPtr + create_uni_stream(QUICStreamId &new_stream_id) override + { + return this->_vc->create_openssl_stream(SSL_STREAM_FLAG_UNI | SSL_STREAM_FLAG_NO_BLOCK, new_stream_id); + } + + QUICConnectionErrorUPtr + create_bidi_stream(QUICStreamId &new_stream_id) override + { + return this->_vc->create_openssl_stream(SSL_STREAM_FLAG_NO_BLOCK, new_stream_id); + } + +private: + QUICNetVConnection *_vc = nullptr; +}; + +} // end anonymous namespace + +#define QUICConDebug(fmt, ...) Dbg(dbg_ctl_quic_net, "[%s] " fmt, this->cids().data(), ##__VA_ARGS__) +#define QUICConVDebug(fmt, ...) Dbg(dbg_ctl_v_quic_net, "[%s] " fmt, this->cids().data(), ##__VA_ARGS__) + +ClassAllocator quicNetVCAllocator("quicNetVCAllocator"); + +QUICNetVConnection::QUICNetVConnection() +{ + this->_set_service(static_cast(this)); + this->_set_service(static_cast(this)); + this->_set_service(static_cast(this)); + this->_set_service(static_cast(this)); + this->_set_service(static_cast(this)); + this->_set_service(static_cast(this)); + this->_set_service(static_cast(this)); +} + +QUICNetVConnection::~QUICNetVConnection() {} + +void +QUICNetVConnection::init(SSL *ssl, QUICPacketHandler *packet_handler) +{ + SET_HANDLER((NetVConnHandler)&QUICNetVConnection::acceptEvent); + + this->_ssl = ssl; + this->_packet_handler = packet_handler; + this->_quic_connection_id.randomize(); + this->_initial_source_connection_id = this->_quic_connection_id; + this->_cid_text = this->_quic_connection_id.hex(); + + SSL_set_ex_data(ssl, QUIC::ssl_quic_qc_index, static_cast(this)); + SSL_set_blocking_mode(ssl, 0); + SSL_set_event_handling_mode(ssl, SSL_VALUE_EVENT_HANDLING_MODE_IMPLICIT); + SSL_set_incoming_stream_policy(ssl, SSL_INCOMING_STREAM_POLICY_ACCEPT, 0); + SSL_set_default_stream_mode(ssl, SSL_DEFAULT_STREAM_MODE_NONE); + + QUICConfig::scoped_config params; + SSL_set_feature_request_uint(ssl, SSL_VALUE_QUIC_IDLE_TIMEOUT, params->no_activity_timeout_in()); + SSL_set_feature_request_uint(ssl, SSL_VALUE_QUIC_STREAM_BIDI_LOCAL_AVAIL, params->initial_max_streams_bidi_in()); + SSL_set_feature_request_uint(ssl, SSL_VALUE_QUIC_STREAM_UNI_LOCAL_AVAIL, params->initial_max_streams_uni_in()); + + this->_bindSSLObject(); + if (auto const servername = quic_sni_server_name(ssl); !servername.empty()) { + this->set_sni_server_name(servername); + } +} + +void +QUICNetVConnection::set_quic_endpoints(IpEndpoint const &local, IpEndpoint const &remote) +{ + this->local_addr = local; + this->remote_addr = remote; + this->got_local_addr = true; + this->got_remote_addr = true; +} + +QUICConnectionErrorUPtr +QUICNetVConnection::create_openssl_stream(uint64_t flags, QUICStreamId &new_stream_id) +{ + if (this->_ssl == nullptr) { + return std::make_unique(QUICTransErrorCode::INTERNAL_ERROR, "QUIC connection is not initialized"); + } + + SSL *stream_ssl = SSL_new_stream(this->_ssl, flags); + if (stream_ssl == nullptr) { + return std::make_unique(QUICTransErrorCode::INTERNAL_ERROR, "Failed to create QUIC stream"); + } + + SSL_set_blocking_mode(stream_ssl, 0); + SSL_set_event_handling_mode(stream_ssl, SSL_VALUE_EVENT_HANDLING_MODE_IMPLICIT); + SSL_set_mode(stream_ssl, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER | SSL_MODE_ENABLE_PARTIAL_WRITE); + + new_stream_id = SSL_get_stream_id(stream_ssl); + this->_openssl_streams.emplace(new_stream_id, stream_ssl); + + [[maybe_unused]] QUICConnectionError err; + this->_stream_manager->create_stream(new_stream_id, err); + + return nullptr; +} + +void +QUICNetVConnection::free() +{ + this->free_thread(this_ethread()); +} + +void +QUICNetVConnection::remove_connection_ids() +{ +} + +void +QUICNetVConnection::destroy(EThread *t) +{ + QUICConDebug("Destroy connection"); + if (from_accept_thread) { + quicNetVCAllocator.free(this); + } else { + THREAD_FREE(this, quicNetVCAllocator, t); + } +} + +void +QUICNetVConnection::set_local_addr() +{ +} + +void +QUICNetVConnection::free_thread(EThread * /* t ATS_UNUSED */) +{ + QUICConDebug("Free connection"); + + this->_unschedule_openssl_event(); + + for (auto &[stream_id, stream_ssl] : this->_openssl_streams) { + SSL_free(stream_ssl); + } + this->_openssl_streams.clear(); + + if (this->_ssl != nullptr) { + this->_unbindSSLObject(); + SSL_free(this->_ssl); + this->_ssl = nullptr; + } + + this->_application_map.reset(); + this->_stream_manager.reset(); + + super::clear(); + ALPNSupport::clear(); + TLSBasicSupport::clear(); + TLSEventSupport::clear(); + TLSCertSwitchSupport::_clear(); + + if (this->_packet_handler != nullptr) { + this->_packet_handler->close_connection(this); + this->_packet_handler = nullptr; + } +} + +void +QUICNetVConnection::reenable(VIO * /* vio ATS_UNUSED */) +{ +} + +int +QUICNetVConnection::state_handshake(int event, Event *data) +{ + if (data == this->_packet_write_ready) { + this->_packet_write_ready = nullptr; + } + + switch (event) { + case EVENT_IMMEDIATE: + case EVENT_INTERVAL: + case QUIC_EVENT_PACKET_READ_READY: + case QUIC_EVENT_PACKET_WRITE_READY: + this->_handle_openssl_events(); + break; + case VC_EVENT_EOS: + case VC_EVENT_ERROR: + case VC_EVENT_ACTIVE_TIMEOUT: + case VC_EVENT_INACTIVITY_TIMEOUT: + this->_unschedule_openssl_event(); + this->_propagate_event(event); + this->closed = 1; + break; + default: + QUICConDebug("Unhandled event: %d", event); + break; + } + + if (this->closed != 1 && SSL_is_init_finished(this->_ssl)) { + this->_switch_to_established_state(); + this->_handle_openssl_events(); + } + + if (this->closed != 1 && this->_openssl_connection_closed()) { + this->_schedule_closing_event(); + } else if (this->closed != 1) { + this->_schedule_openssl_event(); + } + + return EVENT_DONE; +} + +int +QUICNetVConnection::state_established(int event, Event *data) +{ + if (this->_ssl == nullptr) { + return EVENT_DONE; + } + + if (data == this->_packet_write_ready) { + this->_packet_write_ready = nullptr; + } + + switch (event) { + case EVENT_IMMEDIATE: + case EVENT_INTERVAL: + case QUIC_EVENT_PACKET_READ_READY: + case QUIC_EVENT_PACKET_WRITE_READY: + this->_handle_openssl_events(); + break; + case VC_EVENT_EOS: + case VC_EVENT_ERROR: + case VC_EVENT_ACTIVE_TIMEOUT: + case VC_EVENT_INACTIVITY_TIMEOUT: + this->_unschedule_openssl_event(); + this->_propagate_event(event); + this->closed = 1; + break; + default: + QUICConDebug("Unhandled event: %d", event); + break; + } + + if (this->closed != 1 && this->_openssl_connection_closed()) { + this->_schedule_closing_event(); + } else if (this->closed != 1) { + this->_schedule_openssl_event(); + } + + return EVENT_DONE; +} + +void +QUICNetVConnection::_switch_to_established_state() +{ + QUICConDebug("Enter state_connection_established"); + this->_record_tls_handshake_end_time(); + this->_update_end_of_handshake_stats(); + SET_HANDLER((NetVConnHandler)&QUICNetVConnection::state_established); + this->_handshake_completed = true; + this->_start_application(); +} + +void +QUICNetVConnection::_start_application() +{ + if (this->_application_started) { + return; + } + + this->_application_started = true; + + unsigned char const *app_name = nullptr; + unsigned int app_name_len = 0; + SSL_get0_alpn_selected(this->_ssl, &app_name, &app_name_len); + + if (app_name == nullptr || app_name_len == 0) { + app_name = reinterpret_cast(IP_PROTO_TAG_HTTP_QUIC.data()); + app_name_len = IP_PROTO_TAG_HTTP_QUIC.size(); + } + + this->_negotiated_alpn.assign(reinterpret_cast(app_name), app_name_len); + this->set_negotiated_protocol_id(this->_negotiated_alpn); + + if (netvc_context == NET_VCONNECTION_IN) { + if (this->setSelectedProtocol(app_name, app_name_len)) { + this->endpoint()->handleEvent(NET_EVENT_ACCEPT, this); + } + } else { + this->action_.continuation->handleEvent(NET_EVENT_OPEN, this); + } +} + +void +QUICNetVConnection::_propagate_event(int event) +{ + QUICConVDebug("Propagating: %d", event); + if (this->read.vio.cont && this->read.vio.mutex == this->read.vio.cont->mutex) { + this->read.vio.cont->handleEvent(event, &this->read.vio); + } else if (this->write.vio.cont && this->write.vio.mutex == this->write.vio.cont->mutex) { + this->write.vio.cont->handleEvent(event, &this->write.vio); + } else { + QUICConVDebug("Session does not exist"); + } +} + +bool +QUICNetVConnection::shouldDestroy() +{ + return this->refcount() == 0; +} + +VIO * +QUICNetVConnection::do_io_read(Continuation *c, int64_t nbytes, MIOBuffer *buf) +{ + auto vio = super::do_io_read(c, nbytes, buf); + this->read.enabled = 1; + return vio; +} + +VIO * +QUICNetVConnection::do_io_write(Continuation *c, int64_t nbytes, IOBufferReader *buf, bool /* owner ATS_UNUSED */) +{ + auto vio = super::do_io_write(c, nbytes, buf); + this->write.enabled = 1; + this->_schedule_openssl_event(false); + return vio; +} + +int +QUICNetVConnection::acceptEvent(int event, Event *e) +{ + EThread *t = (e == nullptr) ? this_ethread() : e->ethread; + NetHandler *h = get_NetHandler(t); + + MUTEX_TRY_LOCK(lock, h->mutex, t); + if (!lock.is_locked()) { + if (event == EVENT_NONE) { + t->schedule_in(this, HRTIME_MSECONDS(net_retry_delay)); + return EVENT_DONE; + } else { + e->schedule_in(HRTIME_MSECONDS(net_retry_delay)); + return EVENT_CONT; + } + } + + this->_context = std::make_unique(this); + this->_application_map = std::make_unique(); + this->_stream_manager = std::make_unique(this->_context.get(), this->_application_map.get(), this); + + ink_assert(this->thread == this_ethread()); + + if (h->startIO(this) < 0) { + this->free_thread(t); + return EVENT_DONE; + } + + this->read.enabled = 1; + this->write.enabled = 1; + + SET_HANDLER((NetVConnHandler)&QUICNetVConnection::state_handshake); + + nh->startCop(this); + this->set_default_inactivity_timeout(0); + + if (inactivity_timeout_in) { + this->set_inactivity_timeout(inactivity_timeout_in); + } + + if (active_timeout_in) { + set_active_timeout(active_timeout_in); + } + + action_.continuation->handleEvent(NET_EVENT_ACCEPT, this); + this->_schedule_openssl_event(false); + + return EVENT_DONE; +} + +int +QUICNetVConnection::connectUp(EThread * /* t ATS_UNUSED */, int /* fd ATS_UNUSED */) +{ + return 0; +} + +QUICStreamManager * +QUICNetVConnection::stream_manager() +{ + return this->_stream_manager.get(); +} + +void +QUICNetVConnection::close_quic_connection(QUICConnectionErrorUPtr error) +{ + if (this->_ssl != nullptr) { + SSL_SHUTDOWN_EX_ARGS args = {}; + if (error != nullptr) { + args.quic_error_code = error->code; + args.quic_reason = error->msg; + } + SSL_shutdown_ex(this->_ssl, SSL_SHUTDOWN_FLAG_NO_BLOCK, &args, sizeof(args)); + } +} + +void +QUICNetVConnection::reset_quic_connection() +{ + if (this->_ssl != nullptr) { + SSL_shutdown_ex(this->_ssl, SSL_SHUTDOWN_FLAG_RAPID | SSL_SHUTDOWN_FLAG_NO_BLOCK, nullptr, 0); + } +} + +void +QUICNetVConnection::handle_received_packet(UDPPacket * /* packet ATS_UNUSED */) +{ +} + +void +QUICNetVConnection::ping() +{ +} + +QUICConnectionId +QUICNetVConnection::peer_connection_id() const +{ + return {}; +} + +QUICConnectionId +QUICNetVConnection::original_connection_id() const +{ + return {}; +} + +QUICConnectionId +QUICNetVConnection::first_connection_id() const +{ + return {}; +} + +QUICConnectionId +QUICNetVConnection::retry_source_connection_id() const +{ + return {}; +} + +QUICConnectionId +QUICNetVConnection::initial_source_connection_id() const +{ + return this->_initial_source_connection_id; +} + +QUICConnectionId +QUICNetVConnection::connection_id() const +{ + return this->_quic_connection_id; +} + +std::string_view +QUICNetVConnection::cids() const +{ + return this->_cid_text; +} + +QUICFiveTuple const +QUICNetVConnection::five_tuple() const +{ + return QUICFiveTuple(this->remote_addr, this->local_addr, IPPROTO_UDP); +} + +uint32_t +QUICNetVConnection::pmtu() const +{ + return 0; +} + +NetVConnectionContext_t +QUICNetVConnection::direction() const +{ + return NET_VCONNECTION_IN; +} + +QUICVersion +QUICNetVConnection::negotiated_version() const +{ + return QUIC_SUPPORTED_VERSIONS[0]; +} + +std::string_view +QUICNetVConnection::negotiated_application_name() const +{ + return this->_negotiated_alpn; +} + +void +QUICNetVConnection::on_stream_updated() +{ + this->_schedule_openssl_event(false); +} + +bool +QUICNetVConnection::is_closed() const +{ + return this->_openssl_connection_closed(); +} + +bool +QUICNetVConnection::is_at_anti_amplification_limit() const +{ + return false; +} + +bool +QUICNetVConnection::is_address_validation_completed() const +{ + return true; +} + +bool +QUICNetVConnection::is_handshake_completed() const +{ + return this->_handshake_completed; +} + +void +QUICNetVConnection::net_read_io(NetHandler * /* nh ATS_UNUSED */) +{ + SCOPED_MUTEX_LOCK(lock, this->mutex, this_ethread()); + this->handleEvent(QUIC_EVENT_PACKET_READ_READY, nullptr); +} + +int64_t +QUICNetVConnection::load_buffer_and_write(int64_t /* towrite ATS_UNUSED */, MIOBufferAccessor & /* buf ATS_UNUSED */, + int64_t & /* total_written ATS_UNUSED */, int & /* needs ATS_UNUSED */) +{ + return 0; +} + +void +QUICNetVConnection::_bindSSLObject() +{ + TLSBasicSupport::bind(this->_ssl, this); + TLSEventSupport::bind(this->_ssl, this); + ALPNSupport::bind(this->_ssl, this); + TLSSessionResumptionSupport::bind(this->_ssl, this); + TLSSNISupport::bind(this->_ssl, this); + TLSCertSwitchSupport::bind(this->_ssl, this); + QUICSupport::bind(this->_ssl, this); +} + +void +QUICNetVConnection::_unbindSSLObject() +{ + TLSBasicSupport::unbind(this->_ssl); + TLSEventSupport::unbind(this->_ssl); + ALPNSupport::unbind(this->_ssl); + TLSSessionResumptionSupport::unbind(this->_ssl); + TLSSNISupport::unbind(this->_ssl); + TLSCertSwitchSupport::unbind(this->_ssl); + QUICSupport::unbind(this->_ssl); +} + +void +QUICNetVConnection::_schedule_packet_write_ready(bool delay) +{ + this->_schedule_openssl_event(delay); +} + +void +QUICNetVConnection::_unschedule_packet_write_ready() +{ + this->_unschedule_openssl_event(); +} + +void +QUICNetVConnection::_close_packet_write_ready(Event *data) +{ + if (this->_packet_write_ready == data) { + this->_packet_write_ready = nullptr; + } +} + +void +QUICNetVConnection::_schedule_quiche_timeout() +{ +} + +void +QUICNetVConnection::_unschedule_quiche_timeout() +{ +} + +void +QUICNetVConnection::_close_quiche_timeout(Event * /* data ATS_UNUSED */) +{ +} + +void +QUICNetVConnection::_schedule_closing_event() +{ + QUICConDebug("Scheduling closing event"); + SSL_CONN_CLOSE_INFO info = {}; + if (this->_ssl != nullptr && SSL_get_conn_close_info(this->_ssl, &info, sizeof(info)) == 1) { + QUICConDebug("QUIC close info: error_code=%" PRIu64 " frame_type=%" PRIu64 " flags=0x%x reason=\"%.*s\"", info.error_code, + info.frame_type, info.flags, static_cast(info.reason_len), info.reason == nullptr ? "" : info.reason); + if (info.error_code == OSSL_QUIC_LOCAL_ERR_IDLE_TIMEOUT) { + QUICConDebug("QUIC Idle timeout detected"); + this->thread->schedule_imm(this, VC_EVENT_INACTIVITY_TIMEOUT); + return; + } + } + + this->thread->schedule_imm(this, VC_EVENT_EOS); +} + +void +QUICNetVConnection::_handle_read_ready() +{ + this->_handle_openssl_events(); +} + +void +QUICNetVConnection::_handle_write_ready() +{ + this->_handle_openssl_events(); +} + +void +QUICNetVConnection::_handle_interval() +{ + this->_handle_openssl_events(); +} + +void +QUICNetVConnection::_handle_openssl_events() +{ + if (this->_ssl == nullptr) { + return; + } + + if (SSL_handle_events(this->_ssl) != 1) { + QUICConDebug("SSL_handle_events failed: %s", ERR_error_string(ERR_peek_error(), nullptr)); + } + + if (this->_stream_manager != nullptr && this->_application_started) { + this->_accept_openssl_streams(); + this->_process_openssl_streams(); + } + + this->netActivity(); +} + +void +QUICNetVConnection::_accept_openssl_streams() +{ + while (SSL *stream_ssl = SSL_accept_stream(this->_ssl, SSL_ACCEPT_STREAM_NO_BLOCK)) { + SSL_set_blocking_mode(stream_ssl, 0); + SSL_set_event_handling_mode(stream_ssl, SSL_VALUE_EVENT_HANDLING_MODE_IMPLICIT); + SSL_set_mode(stream_ssl, SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER | SSL_MODE_ENABLE_PARTIAL_WRITE); + + QUICStreamId stream_id = SSL_get_stream_id(stream_ssl); + this->_openssl_streams.emplace(stream_id, stream_ssl); + + if (this->_stream_manager->find_stream(stream_id) == nullptr) { + [[maybe_unused]] QUICConnectionError err; + this->_stream_manager->create_stream(stream_id, err); + } + } +} + +void +QUICNetVConnection::_process_openssl_streams() +{ + std::vector stream_ids; + stream_ids.reserve(this->_openssl_streams.size()); + for (auto const &entry : this->_openssl_streams) { + stream_ids.push_back(entry.first); + } + + for (QUICStreamId stream_id : stream_ids) { + auto stream_it = this->_openssl_streams.find(stream_id); + if (stream_it == this->_openssl_streams.end()) { + continue; + } + + SSL *stream_ssl = stream_it->second; + QUICStream *stream = static_cast(this->_stream_manager->find_stream(stream_id)); + if (stream == nullptr) { + continue; + } + + int stream_type = SSL_get_stream_type(stream_ssl); + if ((stream_type & SSL_STREAM_TYPE_READ) != 0) { + stream->receive_data(*this); + } + if ((stream_type & SSL_STREAM_TYPE_WRITE) != 0 || stream->has_data_to_send()) { + if (stream->has_data_to_send()) { + while (stream->has_data_to_send() && stream->send_data(*this) > 0) {} + } else { + stream->send_data(*this); + } + } + } +} + +void +QUICNetVConnection::_schedule_openssl_event(bool delay) +{ + if (!delay && this->_packet_write_ready != nullptr) { + this->_packet_write_ready->cancel(); + this->_packet_write_ready = nullptr; + } + + if (this->_packet_write_ready == nullptr && this->thread != nullptr) { + if (delay) { + this->_packet_write_ready = this->thread->schedule_in(this, OPENSSL_QUIC_EVENT_INTERVAL); + } else { + this->_packet_write_ready = this->thread->schedule_imm(this, QUIC_EVENT_PACKET_WRITE_READY); + } + } +} + +void +QUICNetVConnection::_unschedule_openssl_event() +{ + if (this->_packet_write_ready != nullptr) { + this->_packet_write_ready->cancel(); + this->_packet_write_ready = nullptr; + } +} + +bool +QUICNetVConnection::_openssl_connection_closed() const +{ + if (this->_ssl == nullptr) { + return true; + } + + SSL_CONN_CLOSE_INFO info = {}; + return SSL_get_conn_close_info(this->_ssl, &info, sizeof(info)) == 1; +} + +int +QUICNetVConnection::populate_protocol(std::string_view *results, int n) const +{ + int retval = 0; + if (n > retval) { + results[retval++] = IP_PROTO_TAG_QUIC; + if (n > retval) { + results[retval++] = IP_PROTO_TAG_TLS_1_3; + if (n > retval) { + retval += super::populate_protocol(results + retval, n - retval); + } + } + } + return retval; +} + +char const * +QUICNetVConnection::protocol_contains(std::string_view prefix) const +{ + char const *retval = nullptr; + if (prefix.size() <= IP_PROTO_TAG_QUIC.size() && strncmp(IP_PROTO_TAG_QUIC.data(), prefix.data(), prefix.size()) == 0) { + retval = IP_PROTO_TAG_QUIC.data(); + } else if (prefix.size() <= IP_PROTO_TAG_TLS_1_3.size() && + strncmp(IP_PROTO_TAG_TLS_1_3.data(), prefix.data(), prefix.size()) == 0) { + retval = IP_PROTO_TAG_TLS_1_3.data(); + } else { + retval = super::protocol_contains(prefix); + } + return retval; +} + +QUICConnection * +QUICNetVConnection::get_quic_connection() +{ + return static_cast(this); +} + +int64_t +QUICNetVConnection::read_stream(QUICStreamId stream_id, uint8_t *buf, size_t len, bool &fin, QUICStreamIO::ErrorCode &error_code) +{ + auto it = this->_openssl_streams.find(stream_id); + if (it == this->_openssl_streams.end()) { + error_code = ENOENT; + return -1; + } + + SSL_handle_events(it->second); + + size_t read_len = 0; + fin = false; + if (SSL_read_ex(it->second, buf, len, &read_len) == 1) { + fin = this->stream_read_finished(stream_id); + return read_len; + } + + int ssl_error = SSL_get_error(it->second, 0); + if (ssl_error == SSL_ERROR_WANT_READ || ssl_error == SSL_ERROR_WANT_WRITE) { + return 0; + } + if (ssl_error == SSL_ERROR_ZERO_RETURN) { + fin = true; + return 0; + } + + error_code = ssl_error; + return -1; +} + +bool +QUICNetVConnection::stream_read_finished(QUICStreamId stream_id) +{ + auto it = this->_openssl_streams.find(stream_id); + if (it == this->_openssl_streams.end()) { + return true; + } + + int state = SSL_get_stream_read_state(it->second); + return state == SSL_STREAM_STATE_FINISHED || state == SSL_STREAM_STATE_RESET_REMOTE || state == SSL_STREAM_STATE_CONN_CLOSED; +} + +int64_t +QUICNetVConnection::stream_write_capacity(QUICStreamId stream_id) +{ + auto it = this->_openssl_streams.find(stream_id); + if (it == this->_openssl_streams.end()) { + return -1; + } + + uint64_t avail = 0; + if (SSL_get_stream_write_buf_avail(it->second, &avail) == 1) { + return std::min(avail, 16 * 1024); + } + + return 0; +} + +int64_t +QUICNetVConnection::write_stream(QUICStreamId stream_id, uint8_t const *buf, size_t len, bool fin, + QUICStreamIO::ErrorCode &error_code) +{ + auto it = this->_openssl_streams.find(stream_id); + if (it == this->_openssl_streams.end()) { + error_code = ENOENT; + return -1; + } + + SSL_handle_events(it->second); + + if (len == 0 && fin) { + SSL_stream_conclude(it->second, 0); + return 0; + } + + size_t written_len = 0; + const uint64_t flags = fin ? SSL_WRITE_FLAG_CONCLUDE : 0; + if (SSL_write_ex2(it->second, buf, len, flags, &written_len) == 1) { + return written_len; + } + + int ssl_error = SSL_get_error(it->second, 0); + if (ssl_error == SSL_ERROR_WANT_READ || ssl_error == SSL_ERROR_WANT_WRITE) { + return 0; + } + + error_code = ssl_error; + QUICConDebug("Failed to write %zu bytes on QUIC stream %" PRIu64 ": ssl_error=%d, openssl_error=%s", len, stream_id, ssl_error, + ERR_error_string(ERR_peek_error(), nullptr)); + return -1; +} + +void +QUICNetVConnection::reenable(int event) +{ + this->_is_verifying_cert = false; + + if (event == TS_EVENT_ERROR) { + this->_is_cert_verified = false; + } +} + +Continuation * +QUICNetVConnection::getContinuationForTLSEvents() +{ + return this; +} + +EThread * +QUICNetVConnection::getThreadForTLSEvents() +{ + return this->thread; +} + +Ptr +QUICNetVConnection::getMutexForTLSEvents() +{ + return this->nh->mutex; +} + +bool +QUICNetVConnection::_isReadyToTransferData() const +{ + return this->_handshake_completed; +} + +SSL * +QUICNetVConnection::_get_ssl_object() const +{ + return this->_ssl; +} + +ssl_curve_id +QUICNetVConnection::_get_tls_curve() const +{ + if (getIsResumedFromSessionCache()) { + return getSSLCurveNID(); + } else { + return SSLGetCurveNID(this->_ssl); + } +} + +std::string_view +QUICNetVConnection::_get_tls_group() const +{ + if (getIsResumedFromSessionCache()) { + return getSSLGroupName(); + } else { + return SSLGetGroupName(this->_ssl); + } +} + +int +QUICNetVConnection::_verify_certificate(X509_STORE_CTX * /* ctx ATS_UNUSED */) +{ + TSEvent eventId; + TSHttpHookID hookId; + APIHook *hook = nullptr; + + if (get_context() == NET_VCONNECTION_IN) { + eventId = TS_EVENT_SSL_VERIFY_CLIENT; + hookId = TS_SSL_VERIFY_CLIENT_HOOK; + } else { + eventId = TS_EVENT_SSL_VERIFY_SERVER; + hookId = TS_SSL_VERIFY_SERVER_HOOK; + } + hook = SSLAPIHooks::instance()->get(TSSslHookInternalID(hookId)); + if (hook != nullptr) { + this->_is_verifying_cert = true; + WEAK_SCOPED_MUTEX_LOCK(lock, hook->m_cont->mutex, this_ethread()); + hook->invoke(eventId, this); + } + + ink_assert(this->_is_verifying_cert == false); + if (this->_is_cert_verified) { + return 1; + } + + return 0; +} + +in_port_t +QUICNetVConnection::_get_local_port() +{ + return this->get_local_port(); +} + +IpEndpoint const & +QUICNetVConnection::_getLocalEndpoint() +{ + return this->local_addr; +} + +bool +QUICNetVConnection::_isTryingRenegotiation() const +{ + return this->_handshake_completed; +} + +shared_SSL_CTX +QUICNetVConnection::_lookupContextByName(std::string const &servername, SSLCertContextType ctxType) +{ + shared_SSL_CTX ctx = nullptr; + QUICCertConfig::scoped_config lookup; + SSLCertContext *cc = lookup->find(servername, ctxType); + + if (cc && cc->getCtx()) { + ctx = cc->getCtx(); + } + + return ctx; +} + +shared_SSL_CTX +QUICNetVConnection::_lookupContextByIP() +{ + shared_SSL_CTX ctx = nullptr; + QUICCertConfig::scoped_config lookup; + QUICFiveTuple five_tuple = this->five_tuple(); + IpEndpoint ip = five_tuple.destination(); + SSLCertContext *cc = lookup->find(ip); + + if (cc && cc->getCtx()) { + ctx = cc->getCtx(); + } + + return ctx; +} diff --git a/src/iocore/net/OpenSSLQUICPacketHandler.cc b/src/iocore/net/OpenSSLQUICPacketHandler.cc new file mode 100644 index 00000000000..1d04013eb3b --- /dev/null +++ b/src/iocore/net/OpenSSLQUICPacketHandler.cc @@ -0,0 +1,629 @@ +/** @file + + OpenSSL native QUIC listener support. + + @section license License + + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, software + distributed under the License is distributed on an "AS IS" BASIS, + WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + See the License for the specific language governing permissions and + limitations under the License. + */ + +#include "P_QUICPacketHandler.h" +#include "P_QUICNetProcessor.h" +#include "P_QUICNetVConnection.h" +#include "P_SSLCertLookup.h" +#include "P_UnixNet.h" +#include "iocore/net/QUICMultiCertConfigLoader.h" +#include "iocore/net/quic/QUICConfig.h" +#include "tscore/ink_atomic.h" +#include "tscore/ink_sock.h" + +#include +#include +#include +#include + +#include +#include + +namespace +{ +constexpr ink_hrtime OPENSSL_QUIC_EVENT_INTERVAL = HRTIME_MSECONDS(2); + +DbgCtl dbg_ctl_openssl_quic{"openssl_quic"}; + +struct PeerCaptureBioState { + BIO *inner{nullptr}; + IpEndpoint last_peer; + bool have_last_peer{false}; +}; + +thread_local PeerCaptureBioState *active_peer_capture = nullptr; + +void +free_quic_peer_ex_data(void *, void *ptr, CRYPTO_EX_DATA *, int, long, void *) +{ + delete static_cast(ptr); +} + +int +quic_peer_ex_data_index() +{ + static int index = SSL_get_ex_new_index(0, nullptr, nullptr, nullptr, free_quic_peer_ex_data); + + return index; +} + +bool +bio_addr_to_ip_endpoint(IpEndpoint &endpoint, BIO_ADDR const *bio_addr) +{ + if (bio_addr == nullptr) { + return false; + } + + in_port_t const port = BIO_ADDR_rawport(bio_addr); + if (BIO_ADDR_family(bio_addr) == AF_INET) { + in_addr_t addr = 0; + size_t len = sizeof(addr); + + if (BIO_ADDR_rawaddress(bio_addr, &addr, &len) != 1 || len != sizeof(addr)) { + return false; + } + + ats_ip4_set(&endpoint, addr, port); + return true; + } + + if (BIO_ADDR_family(bio_addr) == AF_INET6) { + in6_addr addr; + size_t len = sizeof(addr); + + if (BIO_ADDR_rawaddress(bio_addr, &addr, &len) != 1 || len != sizeof(addr)) { + return false; + } + + ats_ip6_set(&endpoint, addr, port); + return true; + } + + return false; +} + +PeerCaptureBioState * +peer_capture_state(BIO *bio) +{ + return static_cast(BIO_get_data(bio)); +} + +BIO_MSG * +bio_msg_at(BIO_MSG *msg, size_t stride, size_t index) +{ + return reinterpret_cast(reinterpret_cast(msg) + (stride * index)); +} + +void +update_last_peer(PeerCaptureBioState &state, BIO_MSG const &msg) +{ + IpEndpoint peer; + if (msg.peer != nullptr && bio_addr_to_ip_endpoint(peer, msg.peer)) { + state.last_peer = peer; + state.have_last_peer = true; + active_peer_capture = &state; + } +} + +void +copy_retry_flags(BIO *bio, BIO *inner) +{ + BIO_clear_retry_flags(bio); + BIO_set_flags(bio, BIO_get_retry_flags(inner)); + BIO_set_retry_reason(bio, BIO_get_retry_reason(inner)); +} + +int +peer_capture_bio_create(BIO *bio) +{ + BIO_set_init(bio, 0); + BIO_set_data(bio, nullptr); + BIO_set_shutdown(bio, 1); + + return 1; +} + +int +peer_capture_bio_destroy(BIO *bio) +{ + if (bio == nullptr) { + return 0; + } + + auto *state = peer_capture_state(bio); + if (state != nullptr) { + if (active_peer_capture == state) { + active_peer_capture = nullptr; + } + BIO_free(state->inner); + delete state; + BIO_set_data(bio, nullptr); + } + BIO_set_init(bio, 0); + + return 1; +} + +int +peer_capture_bio_write(BIO *bio, char const *data, int len) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + int const result = BIO_write(state->inner, data, len); + copy_retry_flags(bio, state->inner); + + return result; +} + +int +peer_capture_bio_write_ex(BIO *bio, char const *data, size_t len, size_t *written) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + int const result = BIO_write_ex(state->inner, data, len, written); + copy_retry_flags(bio, state->inner); + + return result; +} + +int +peer_capture_bio_sendmmsg(BIO *bio, BIO_MSG *msg, size_t stride, size_t num_msg, uint64_t flags, size_t *num_processed) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + int const result = BIO_sendmmsg(state->inner, msg, stride, num_msg, flags, num_processed); + copy_retry_flags(bio, state->inner); + + return result; +} + +int +peer_capture_bio_read(BIO *bio, char *data, int len) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + int const result = BIO_read(state->inner, data, len); + copy_retry_flags(bio, state->inner); + + return result; +} + +int +peer_capture_bio_read_ex(BIO *bio, char *data, size_t len, size_t *read_bytes) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + int const result = BIO_read_ex(state->inner, data, len, read_bytes); + copy_retry_flags(bio, state->inner); + + return result; +} + +int +peer_capture_bio_recvmmsg(BIO *bio, BIO_MSG *msg, size_t stride, size_t num_msg, uint64_t flags, size_t *num_processed) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + size_t const read_limit = std::min(num_msg, 1); + + int const result = BIO_recvmmsg(state->inner, msg, stride, read_limit, flags, num_processed); + copy_retry_flags(bio, state->inner); + + if (result == 1 && num_processed != nullptr && *num_processed > 0) { + update_last_peer(*state, *bio_msg_at(msg, stride, 0)); + } + + return result; +} + +long +peer_capture_bio_ctrl(BIO *bio, int cmd, long larg, void *parg) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + return BIO_ctrl(state->inner, cmd, larg, parg); +} + +long +peer_capture_bio_callback_ctrl(BIO *bio, int cmd, BIO_info_cb *cb) +{ + auto *state = peer_capture_state(bio); + if (state == nullptr || state->inner == nullptr) { + return 0; + } + + return BIO_callback_ctrl(state->inner, cmd, cb); +} + +BIO_METHOD * +peer_capture_bio_method() +{ + static BIO_METHOD *method = []() -> BIO_METHOD * { + BIO_METHOD *m = BIO_meth_new(BIO_TYPE_SOURCE_SINK | BIO_get_new_index(), "ATS OpenSSL QUIC peer capture"); + if (m == nullptr) { + return nullptr; + } + + BIO_meth_set_create(m, peer_capture_bio_create); + BIO_meth_set_destroy(m, peer_capture_bio_destroy); + BIO_meth_set_write(m, peer_capture_bio_write); + BIO_meth_set_write_ex(m, peer_capture_bio_write_ex); + BIO_meth_set_sendmmsg(m, peer_capture_bio_sendmmsg); + BIO_meth_set_read(m, peer_capture_bio_read); + BIO_meth_set_read_ex(m, peer_capture_bio_read_ex); + BIO_meth_set_recvmmsg(m, peer_capture_bio_recvmmsg); + BIO_meth_set_ctrl(m, peer_capture_bio_ctrl); + BIO_meth_set_callback_ctrl(m, peer_capture_bio_callback_ctrl); + + return m; + }(); + + return method; +} + +BIO * +new_peer_capture_bio(int fd) +{ + BIO *inner = BIO_new_dgram(fd, BIO_NOCLOSE); + if (inner == nullptr) { + return nullptr; + } + + BIO *bio = BIO_new(peer_capture_bio_method()); + if (bio == nullptr) { + BIO_free(inner); + return nullptr; + } + + auto *state = new PeerCaptureBioState; + state->inner = inner; + BIO_set_data(bio, state); + BIO_set_init(bio, 1); + + return bio; +} + +int +new_pending_quic_connection_cb(SSL_CTX *, SSL *new_ssl, void *) +{ + auto *capture = active_peer_capture; + if (capture == nullptr || !capture->have_last_peer) { + return 1; + } + + int const index = quic_peer_ex_data_index(); + if (index < 0) { + return 0; + } + + delete static_cast(SSL_get_ex_data(new_ssl, index)); + auto *peer = new IpEndpoint(capture->last_peer); + if (SSL_set_ex_data(new_ssl, index, peer) != 1) { + delete peer; + return 0; + } + + return 1; +} + +bool +get_bio_peer_addr(BIO *bio, IpEndpoint &remote_addr) +{ + if (bio == nullptr) { + return false; + } + + BIO_ADDR *peer_addr = BIO_ADDR_new(); + if (peer_addr == nullptr) { + return false; + } + + bool const success = BIO_dgram_get_peer(bio, peer_addr) == 1 && bio_addr_to_ip_endpoint(remote_addr, peer_addr); + BIO_ADDR_free(peer_addr); + + return success; +} + +bool +get_quic_peer_addr(SSL *ssl, IpEndpoint &remote_addr) +{ + int const index = quic_peer_ex_data_index(); + if (index >= 0) { + auto *peer = static_cast(SSL_get_ex_data(ssl, index)); + if (peer != nullptr) { + remote_addr = *peer; + return true; + } + } + + return get_bio_peer_addr(SSL_get_rbio(ssl), remote_addr) || get_bio_peer_addr(SSL_get_wbio(ssl), remote_addr); +} + +} // end anonymous namespace + +QUICPacketHandler::QUICPacketHandler() +{ + this->_closed_con_collector = std::make_unique(); + this->_closed_con_collector->mutex = new_ProxyMutex(); +} + +QUICPacketHandler::~QUICPacketHandler() +{ + if (this->_collector_event != nullptr) { + this->_collector_event->cancel(); + this->_collector_event = nullptr; + } +} + +void +QUICPacketHandler::close_connection(QUICNetVConnection *conn) +{ + int isin = ink_atomic_swap(&conn->in_closed_queue, 1); + if (!isin) { + this->_closed_con_collector->closedQueue.push(conn); + } +} + +void +QUICPacketHandler::send_packet(UDPConnection * /* udp_con ATS_UNUSED */, IpEndpoint & /* addr ATS_UNUSED */, + Ptr /* udp_payload ATS_UNUSED */, uint16_t /* segment_size ATS_UNUSED */, + struct timespec * /* send_at_hint ATS_UNUSED */) +{ +} + +QUICPacketHandlerIn::QUICPacketHandlerIn(NetProcessor::AcceptOptions const &opt) : NetAccept(opt), QUICPacketHandler() +{ + this->mutex = new_ProxyMutex(); +} + +QUICPacketHandlerIn::~QUICPacketHandlerIn() +{ + if (this->_event != nullptr) { + this->_event->cancel(); + this->_event = nullptr; + } + if (this->_listener != nullptr) { + SSL_free(this->_listener); + this->_listener = nullptr; + } +} + +NetProcessor * +QUICPacketHandlerIn::getNetProcessor() const +{ + return &quic_NetProcessor; +} + +NetAccept * +QUICPacketHandlerIn::clone() const +{ + return new QUICPacketHandlerIn(this->opt); +} + +int +QUICPacketHandlerIn::acceptEvent(int /* event ATS_UNUSED */, void * /* data ATS_UNUSED */) +{ + this->setThreadAffinity(this_ethread()); + + if (this->_collector_event == nullptr) { + this->_collector_event = this_ethread()->schedule_every(this->_closed_con_collector.get(), HRTIME_MSECONDS(100)); + } + + if (this->_listener == nullptr && !this->_start_listener()) { + return EVENT_DONE; + } + + this->_event = nullptr; + this->_service_listener(); + this->_schedule_event(); + + return EVENT_CONT; +} + +void +QUICPacketHandlerIn::init_accept(EThread *t) +{ + SET_HANDLER(&QUICPacketHandlerIn::acceptEvent); + + if (t == nullptr) { + t = eventProcessor.assign_thread(ET_NET); + } + + if (!this->action_->continuation->mutex) { + this->action_->continuation->mutex = t->mutex; + this->action_->mutex = t->mutex; + } + + this->mutex = get_NetHandler(t)->mutex; + t->schedule_imm(this); +} + +Continuation * +QUICPacketHandlerIn::_get_continuation() +{ + return this; +} + +void +QUICPacketHandlerIn::_recv_packet(int /* event ATS_UNUSED */, UDPPacket * /* udpPacket ATS_UNUSED */) +{ +} + +bool +QUICPacketHandlerIn::_start_listener() +{ + if (!this->server.sock.is_ok()) { + this->server.sock = UnixSocket{this->server.accept_addr.sa.sa_family, SOCK_DGRAM, 0}; + if (!this->server.sock.is_ok()) { + Error("failed to create OpenSSL QUIC UDP socket: %s", strerror(errno)); + return false; + } + + if (this->server.accept_addr.sa.sa_family == AF_INET6 && this->server.sock.enable_option(IPPROTO_IPV6, IPV6_V6ONLY) < 0) { + Error("failed to set IPV6_V6ONLY on OpenSSL QUIC socket: %s", strerror(errno)); + return false; + } + if (this->server.sock.enable_option(SOL_SOCKET, SO_REUSEADDR) < 0) { + Error("failed to set SO_REUSEADDR on OpenSSL QUIC socket: %s", strerror(errno)); + return false; + } + if (this->server.sock.bind(&this->server.accept_addr.sa, ats_ip_size(&this->server.accept_addr.sa)) < 0) { + Error("failed to bind OpenSSL QUIC socket: %s", strerror(errno)); + return false; + } + } + + if (this->server.sock.set_nonblocking() < 0) { + Error("failed to make OpenSSL QUIC socket nonblocking: %s", strerror(errno)); + return false; + } + if (this->opt.recv_bufsize > 0) { + this->server.sock.set_rcvbuf_size(this->opt.recv_bufsize); + } + if (this->opt.send_bufsize > 0) { + this->server.sock.set_sndbuf_size(this->opt.send_bufsize); + } + + QUICCertConfig::scoped_config server_cert; + QUICConfig::scoped_config params; + uint64_t listener_flags = 0; + if (!params->stateless_retry()) { + listener_flags |= SSL_LISTENER_FLAG_NO_VALIDATE; + } + + SSL_CTX_set_new_pending_conn_cb(server_cert->defaultContext(), new_pending_quic_connection_cb, nullptr); + + this->_listener = SSL_new_listener(server_cert->defaultContext(), listener_flags); + if (this->_listener == nullptr) { + Error("failed to create OpenSSL QUIC listener"); + return false; + } + + BIO *bio = new_peer_capture_bio(this->server.sock.get_fd()); + if (bio == nullptr) { + Error("failed to create OpenSSL QUIC peer capture BIO"); + return false; + } + SSL_set_bio(this->_listener, bio, bio); + if (SSL_get_rbio(this->_listener) == nullptr || SSL_get_wbio(this->_listener) == nullptr) { + Error("failed to configure OpenSSL QUIC listener socket"); + return false; + } + SSL_set_blocking_mode(this->_listener, 0); + SSL_set_event_handling_mode(this->_listener, SSL_VALUE_EVENT_HANDLING_MODE_EXPLICIT); + + SSL_set_feature_request_uint(this->_listener, SSL_VALUE_QUIC_IDLE_TIMEOUT, params->no_activity_timeout_in()); + SSL_set_feature_request_uint(this->_listener, SSL_VALUE_QUIC_STREAM_BIDI_LOCAL_AVAIL, params->initial_max_streams_bidi_in()); + SSL_set_feature_request_uint(this->_listener, SSL_VALUE_QUIC_STREAM_UNI_LOCAL_AVAIL, params->initial_max_streams_uni_in()); + SSL_set_incoming_stream_policy(this->_listener, SSL_INCOMING_STREAM_POLICY_ACCEPT, 0); + + if (SSL_listen(this->_listener) != 1) { + Error("failed to start OpenSSL QUIC listener"); + return false; + } + + Dbg(dbg_ctl_openssl_quic, "OpenSSL QUIC listener started on fd %d", this->server.sock.get_fd()); + return true; +} + +void +QUICPacketHandlerIn::_service_listener() +{ + SSL_handle_events(this->_listener); + + while (SSL *ssl = SSL_accept_connection(this->_listener, SSL_ACCEPT_CONNECTION_NO_BLOCK)) { + SSL_set_blocking_mode(ssl, 0); + SSL_set_event_handling_mode(ssl, SSL_VALUE_EVENT_HANDLING_MODE_IMPLICIT); + SSL_set_incoming_stream_policy(ssl, SSL_INCOMING_STREAM_POLICY_ACCEPT, 0); + SSL_set_default_stream_mode(ssl, SSL_DEFAULT_STREAM_MODE_NONE); + + auto *vc = static_cast(getNetProcessor()->allocate_vc(this_ethread())); + if (vc == nullptr) { + SSL_free(ssl); + continue; + } + + IpEndpoint remote_addr; + if (!get_quic_peer_addr(ssl, remote_addr)) { + remote_addr.setToAnyAddr(this->server.accept_addr.sa.sa_family); + } + + vc->init(ssl, this); + vc->id = net_next_connection_number(); + vc->set_quic_endpoints(this->server.accept_addr, remote_addr); + vc->submit_time = ink_get_hrtime(); + vc->thread = this_ethread(); + vc->mutex = get_NetHandler(this_ethread())->mutex; + vc->action_ = *this->action_; + vc->set_is_transparent(this->opt.f_inbound_transparent); + vc->set_context(NET_VCONNECTION_IN); + vc->options.ip_proto = NetVCOptions::USE_UDP; + vc->options.ip_family = this->server.accept_addr.sa.sa_family; + this_ethread()->schedule_imm(vc, EVENT_NONE, nullptr); + } +} + +void +QUICPacketHandlerIn::_schedule_event() +{ + if (this->_event == nullptr) { + this->_event = this_ethread()->schedule_in(this, OPENSSL_QUIC_EVENT_INTERVAL); + } +} + +void +QUICPacketHandlerOut::init(QUICNetVConnection * /* vc ATS_UNUSED */) +{ +} + +Continuation * +QUICPacketHandlerOut::_get_continuation() +{ + return this; +} + +void +QUICPacketHandlerOut::_recv_packet(int /* event ATS_UNUSED */, UDPPacket * /* udp_packet ATS_UNUSED */) +{ +} diff --git a/src/iocore/net/P_QUICNetProcessor.h b/src/iocore/net/P_QUICNetProcessor.h index 677b1fececb..21d2e91dc8a 100644 --- a/src/iocore/net/P_QUICNetProcessor.h +++ b/src/iocore/net/P_QUICNetProcessor.h @@ -43,7 +43,11 @@ #include "P_UnixNetProcessor.h" #include "iocore/net/quic/QUICConnectionTable.h" +#include "tscore/ink_config.h" + +#if TS_HAS_QUICHE #include +#endif class UnixNetVConnection; struct NetAccept; @@ -68,17 +72,21 @@ class QUICNetProcessor : public UnixNetProcessor Action *main_accept(Continuation *cont, SOCKET fd, AcceptOptions const &opt) override; +#if TS_HAS_QUICHE off_t quicPollCont_offset; +#endif protected: - NetAccept *createNetAccept(const NetProcessor::AcceptOptions &opt) override; + NetAccept *createNetAccept(NetProcessor::AcceptOptions const &opt) override; private: QUICNetProcessor(const QUICNetProcessor &); QUICNetProcessor &operator=(const QUICNetProcessor &); +#if TS_HAS_QUICHE QUICConnectionTable *_ctable = nullptr; quiche_config *_quiche_config = nullptr; +#endif }; extern QUICNetProcessor quic_NetProcessor; diff --git a/src/iocore/net/P_QUICNetVConnection.h b/src/iocore/net/P_QUICNetVConnection.h index 89af3685e4e..bb8064ecb49 100644 --- a/src/iocore/net/P_QUICNetVConnection.h +++ b/src/iocore/net/P_QUICNetVConnection.h @@ -46,11 +46,19 @@ #include "iocore/net/quic/QUICConnection.h" #include "iocore/net/quic/QUICConnectionTable.h" #include "iocore/net/quic/QUICContext.h" +#include "iocore/net/quic/QUICStream.h" #include "iocore/net/quic/QUICStreamManager.h" +#include "tscore/ink_config.h" #include "tscore/List.h" #include +#include +#include +#include + +#if TS_HAS_QUICHE #include +#endif class EThread; class QUICPacketHandler; @@ -66,16 +74,23 @@ class QUICNetVConnection : public UnixNetVConnection, public TLSCertSwitchSupport, public TLSEventSupport, public TLSBasicSupport, - public QUICSupport + public QUICSupport, + public QUICStreamIO { using super = UnixNetVConnection; ///< Parent type. public: QUICNetVConnection(); ~QUICNetVConnection(); +#if TS_HAS_OPENSSL_QUIC + void init(SSL *ssl, QUICPacketHandler *packet_handler); + void set_quic_endpoints(IpEndpoint const &local, IpEndpoint const &remote); + QUICConnectionErrorUPtr create_openssl_stream(uint64_t flags, QUICStreamId &new_stream_id); +#elif TS_HAS_QUICHE void init(QUICVersion version, QUICConnectionId peer_cid, QUICConnectionId original_cid, UDPConnection *, QUICPacketHandler *); void init(QUICVersion version, QUICConnectionId peer_cid, QUICConnectionId original_cid, QUICConnectionId first_cid, QUICConnectionId retry_cid, UDPConnection *, quiche_conn *, QUICPacketHandler *, QUICConnectionTable *ctable, SSL *); +#endif // Event handlers int acceptEvent(int event, Event *e); @@ -103,7 +118,7 @@ class QUICNetVConnection : public UnixNetVConnection, // NetVConnection int populate_protocol(std::string_view *results, int n) const override; - const char *protocol_contains(std::string_view tag) const override; + char const *protocol_contains(std::string_view tag) const override; // QUICConnection QUICStreamManager *stream_manager() override; @@ -120,20 +135,27 @@ class QUICNetVConnection : public UnixNetVConnection, QUICConnectionId initial_source_connection_id() const override; QUICConnectionId connection_id() const override; std::string_view cids() const override; - const QUICFiveTuple five_tuple() const override; + QUICFiveTuple const five_tuple() const override; uint32_t pmtu() const override; NetVConnectionContext_t direction() const override; QUICVersion negotiated_version() const override; std::string_view negotiated_application_name() const override; + void on_stream_updated() override; bool is_closed() const override; bool is_at_anti_amplification_limit() const override; bool is_address_validation_completed() const override; bool is_handshake_completed() const override; - void on_stream_updated() override; // QUICSupport QUICConnection *get_quic_connection() override; + // QUICStreamIO + int64_t read_stream(QUICStreamId stream_id, uint8_t *buf, size_t len, bool &fin, QUICStreamIO::ErrorCode &error_code) override; + bool stream_read_finished(QUICStreamId stream_id) override; + int64_t stream_write_capacity(QUICStreamId stream_id) override; + int64_t write_stream(QUICStreamId stream_id, uint8_t const *buf, size_t len, bool fin, + QUICStreamIO::ErrorCode &error_code) override; + // QUICNetVConnection int in_closed_queue = 0; @@ -167,11 +189,11 @@ class QUICNetVConnection : public UnixNetVConnection, in_port_t _get_local_port() override; // TLSSessionResumptionSupport - const IpEndpoint &_getLocalEndpoint() override; + IpEndpoint const &_getLocalEndpoint() override; // TLSCertSwitchSupport bool _isTryingRenegotiation() const override; - shared_SSL_CTX _lookupContextByName(const std::string &servername, SSLCertContextType ctxType) override; + shared_SSL_CTX _lookupContextByName(std::string const &servername, SSLCertContextType ctxType) override; shared_SSL_CTX _lookupContextByIP() override; // TLSEventSupport @@ -199,9 +221,16 @@ class QUICNetVConnection : public UnixNetVConnection, QUICConnectionId _initial_source_connection_id; // src cid used for Initial packet QUICConnectionId _quic_connection_id; // src cid in local - UDPConnection *_udp_con = nullptr; - quiche_conn *_quiche_con = nullptr; - QUICConnectionTable *_ctable = nullptr; + QUICConnectionTable *_ctable = nullptr; + +#if TS_HAS_OPENSSL_QUIC + std::unordered_map _openssl_streams; + std::string _cid_text; + std::string _negotiated_alpn; +#elif TS_HAS_QUICHE + UDPConnection *_udp_con = nullptr; + quiche_conn *_quiche_con = nullptr; +#endif void _bindSSLObject(); void _unbindSSLObject(); @@ -222,6 +251,15 @@ class QUICNetVConnection : public UnixNetVConnection, void _handle_write_ready(); void _handle_interval(); +#if TS_HAS_OPENSSL_QUIC + void _handle_openssl_events(); + void _accept_openssl_streams(); + void _process_openssl_streams(); + void _schedule_openssl_event(bool delay = true); + void _unschedule_openssl_event(); + bool _openssl_connection_closed() const; +#endif + void _propagate_event(int event); void _switch_to_established_state(); diff --git a/src/iocore/net/P_QUICPacketHandler.h b/src/iocore/net/P_QUICPacketHandler.h index f64cfc63105..47283c4058f 100644 --- a/src/iocore/net/P_QUICPacketHandler.h +++ b/src/iocore/net/P_QUICPacketHandler.h @@ -26,8 +26,13 @@ #include "P_NetAccept.h" #include "P_QUICClosedConCollector.h" #include "iocore/net/UDPConnection.h" +#include "tscore/ink_config.h" +#if TS_HAS_OPENSSL_QUIC +#include +#elif TS_HAS_QUICHE #include +#endif class QUICNetVConnection; class QUICConnectionTable; @@ -54,7 +59,11 @@ class QUICPacketHandler class QUICPacketHandlerIn : public NetAccept, public QUICPacketHandler { public: - QUICPacketHandlerIn(const NetProcessor::AcceptOptions &opt, QUICConnectionTable &ctable, quiche_config &config); +#if TS_HAS_OPENSSL_QUIC + QUICPacketHandlerIn(NetProcessor::AcceptOptions const &opt); +#elif TS_HAS_QUICHE + QUICPacketHandlerIn(NetProcessor::AcceptOptions const &opt, QUICConnectionTable &ctable, quiche_config &config); +#endif ~QUICPacketHandlerIn(); // NetAccept @@ -68,10 +77,21 @@ class QUICPacketHandlerIn : public NetAccept, public QUICPacketHandler Continuation *_get_continuation() override; private: +#if TS_HAS_OPENSSL_QUIC + SSL *_listener = nullptr; + Event *_event = nullptr; +#elif TS_HAS_QUICHE QUICConnectionTable &_ctable; quiche_config &_quiche_config; +#endif void _recv_packet(int event, UDPPacket *udpPacket) override; + +#if TS_HAS_OPENSSL_QUIC + bool _start_listener(); + void _service_listener(); + void _schedule_event(); +#endif }; class QUICPacketHandlerOut : public Continuation, public QUICPacketHandler diff --git a/src/iocore/net/QUICMultiCertConfigLoader.cc b/src/iocore/net/QUICMultiCertConfigLoader.cc index 2e7049643e8..c9c4483a1ed 100644 --- a/src/iocore/net/QUICMultiCertConfigLoader.cc +++ b/src/iocore/net/QUICMultiCertConfigLoader.cc @@ -24,9 +24,13 @@ #include "P_SSLCertLookup.h" #include "P_SSLConfig.h" #include "iocore/net/QUICMultiCertConfigLoader.h" +#include "iocore/net/TLSSNISupport.h" #include "iocore/net/quic/QUICConfig.h" #include "mgmt/config/ConfigContextDiags.h" +#include +#include + int QUICCertConfig::_config_id = 0; // @@ -88,12 +92,256 @@ QUICCertConfig::release(SSLCertLookup *lookup) SSL_CTX * QUICMultiCertConfigLoader::default_server_ssl_ctx() { - return quic_new_ssl_ctx(); + return quic_new_server_ssl_ctx(); +} + +#if TS_HAS_OPENSSL_QUIC +namespace +{ +DbgCtl dbg_ctl_quic{"quic"}; +} // end anonymous namespace + +namespace +{ +void +free_quic_sni_ex_data(void *, void *ptr, CRYPTO_EX_DATA *, int, long, void *) +{ + delete static_cast(ptr); +} + +int +quic_sni_ex_data_index() +{ + static int index = SSL_get_ex_new_index(0, nullptr, nullptr, nullptr, free_quic_sni_ex_data); + + return index; +} + +void +remember_quic_sni(SSL *ssl, std::string_view servername) +{ + if (ssl == nullptr || servername.empty()) { + return; + } + + auto const index = quic_sni_ex_data_index(); + if (index < 0) { + return; + } + + auto *stored = static_cast(SSL_get_ex_data(ssl, index)); + if (stored == nullptr) { + stored = new std::string; + if (SSL_set_ex_data(ssl, index, stored) != 1) { + delete stored; + return; + } + } + stored->assign(servername); + + if (auto *snis = TLSSNISupport::getInstance(ssl); snis != nullptr) { + snis->set_sni_server_name(servername); + } } bool -QUICMultiCertConfigLoader::_setup_session_cache(SSL_CTX * /* ctx ATS_UNUSED */) +apply_quic_sni_certificate(SSL *ssl, SSL_CTX *ctx) { + X509 *cert = SSL_CTX_get0_certificate(ctx); + EVP_PKEY *key = SSL_CTX_get0_privatekey(ctx); + + if (cert == nullptr && key == nullptr) { + return true; + } + if (cert == nullptr || key == nullptr) { + return false; + } + + if (SSL_use_certificate(ssl, cert) != 1 || SSL_use_PrivateKey(ssl, key) != 1) { + return false; + } + + STACK_OF(X509) *chain = nullptr; + if (SSL_CTX_get0_chain_certs(ctx, &chain) == 1 && chain != nullptr && SSL_set1_chain(ssl, chain) != 1) { + return false; + } + + return true; +} + +bool +select_quic_sni_context(SSL *ssl, std::string_view servername) +{ + if (servername.empty()) { + return true; + } + + QUICCertConfig::scoped_config lookup; + if (!lookup) { + Dbg(dbg_ctl_quic, "OpenSSL QUIC SNI context selection could not acquire cert lookup"); + return false; + } + + SSLCertContext *cert_context = lookup->find(std::string(servername)); + if (cert_context == nullptr) { + return true; + } + + shared_SSL_CTX const selected_ctx = cert_context->getCtx(); + if (selected_ctx == nullptr) { + Dbg(dbg_ctl_quic, "OpenSSL QUIC SNI context selection found null context for SNI '%.*s'", static_cast(servername.size()), + servername.data()); + return false; + } + + SSL_set_SSL_CTX(ssl, selected_ctx.get()); + if (!apply_quic_sni_certificate(ssl, selected_ctx.get())) { + Dbg(dbg_ctl_quic, "OpenSSL QUIC SNI context selection failed to apply SSL_CTX %p certificate to ssl=%p", selected_ctx.get(), + ssl); + return false; + } + + return true; +} + +std::string_view +servername_from_client_hello(SSL *ssl) +{ + unsigned char const *p = nullptr; + size_t remaining = 0; + if (SSL_client_hello_get0_ext(ssl, TLSEXT_TYPE_server_name, &p, &remaining) != 1 || remaining <= 2) { + return {}; + } + + size_t len = (static_cast(*p++) << 8); + len += *p++; + if (len + 2 != remaining) { + return {}; + } + remaining = len; + + if (remaining <= 3 || *p++ != TLSEXT_NAMETYPE_host_name) { + return {}; + } + --remaining; + + len = (static_cast(*p++) << 8); + len += *p++; + if (len + 2 > remaining) { + return {}; + } + + return {reinterpret_cast(p), len}; +} + +int +quic_client_hello_callback(SSL *ssl, int *, void *) +{ + auto const servername = servername_from_client_hello(ssl); + remember_quic_sni(ssl, servername); + + return select_quic_sni_context(ssl, servername) ? SSL_CLIENT_HELLO_SUCCESS : SSL_CLIENT_HELLO_ERROR; +} + +int +quic_servername_callback(SSL *ssl, int *, void *) +{ + char const *servername = SSL_get_servername(ssl, TLSEXT_NAMETYPE_host_name); + auto const sni = servername == nullptr ? std::string_view{} : std::string_view{servername}; + remember_quic_sni(ssl, sni); + + return select_quic_sni_context(ssl, sni) ? SSL_TLSEXT_ERR_OK : SSL_TLSEXT_ERR_ALERT_FATAL; +} +} // end anonymous namespace + +std::string_view +quic_sni_server_name(SSL *ssl) +{ + if (ssl == nullptr) { + return {}; + } + + auto const index = quic_sni_ex_data_index(); + if (index < 0) { + return {}; + } + + auto *stored = static_cast(SSL_get_ex_data(ssl, index)); + + return stored == nullptr ? std::string_view{} : std::string_view{*stored}; +} + +void +QUICMultiCertConfigLoader::_set_handshake_callbacks(SSL_CTX *ctx) +{ + Dbg(dbg_ctl_quic, "installing OpenSSL QUIC cert callback on SSL_CTX %p", ctx); + SSL_CTX_set_client_hello_cb(ctx, quic_client_hello_callback, nullptr); + SSL_CTX_set_tlsext_servername_callback(ctx, quic_servername_callback); + SSL_CTX_set_cert_cb( + ctx, + [](SSL *ssl, void *) -> int { + char const *servername = SSL_get_servername(ssl, TLSEXT_NAMETYPE_host_name); + auto const sni = servername == nullptr ? std::string_view{} : std::string_view{servername}; + remember_quic_sni(ssl, sni); + + return select_quic_sni_context(ssl, sni) ? 1 : 0; + }, + nullptr); +} + +namespace +{ +int +quic_alpn_select_callback(SSL * /* ssl ATS_UNUSED */, const unsigned char **out, unsigned char *outlen, const unsigned char *in, + unsigned int inlen, void * /* arg ATS_UNUSED */) +{ + static constexpr unsigned char server_protos[] = {2, 'h', '3', 5, 'h', '3', '-', '2', '9', 5, 'h', '3', '-', '2', '7'}; + + if (SSL_select_next_proto(const_cast(out), outlen, server_protos, sizeof(server_protos), in, inlen) == + OPENSSL_NPN_NEGOTIATED) { + return SSL_TLSEXT_ERR_OK; + } + + *out = nullptr; + *outlen = 0; + return SSL_TLSEXT_ERR_ALERT_FATAL; +} +} // end anonymous namespace + +bool +QUICMultiCertConfigLoader::_set_alpn_callback(SSL_CTX *ctx) +{ + SSL_CTX_set_alpn_select_cb(ctx, quic_alpn_select_callback, nullptr); + return true; +} + +bool +QUICMultiCertConfigLoader::_set_curves(SSL_CTX *ctx) +{ + QUICConfig::scoped_config params; + if (params->server_supported_groups() == nullptr) { + return true; + } + + if (!SSL_CTX_set1_groups_list(ctx, params->server_supported_groups())) { + Error("invalid QUIC groups list for server in records.config"); + return false; + } + + return true; +} +#endif + +bool +QUICMultiCertConfigLoader::_setup_session_cache(SSL_CTX *ctx) +{ +#if TS_HAS_OPENSSL_QUIC + SSL_CTX_set_min_proto_version(ctx, TLS1_3_VERSION); + SSL_CTX_set_max_proto_version(ctx, TLS1_3_VERSION); +#else + (void)ctx; +#endif + // Disabled for now // TODO Check if the logic in SSLMultiCertConfigLoader is reusable return true; diff --git a/src/iocore/net/QUICNetVConnection.cc b/src/iocore/net/QUICNetVConnection.cc index 18a85913739..451b1fc1243 100644 --- a/src/iocore/net/QUICNetVConnection.cc +++ b/src/iocore/net/QUICNetVConnection.cc @@ -679,7 +679,7 @@ QUICNetVConnection::_handle_read_ready() [[maybe_unused]] QUICConnectionError err; stream = this->_stream_manager->create_stream(s, err); } - stream->receive_data(this->_quiche_con); + stream->receive_data(*this); } quiche_stream_iter_free(readable); } @@ -696,7 +696,7 @@ QUICNetVConnection::_handle_write_ready() [[maybe_unused]] QUICConnectionError err; stream = this->_stream_manager->create_stream(s, err); } - stream->send_data(this->_quiche_con); + stream->send_data(*this); } quiche_stream_iter_free(writable); } @@ -791,6 +791,31 @@ QUICNetVConnection::get_quic_connection() return static_cast(this); } +int64_t +QUICNetVConnection::read_stream(QUICStreamId stream_id, uint8_t *buf, size_t len, bool &fin, QUICStreamIO::ErrorCode &error_code) +{ + return quiche_conn_stream_recv(this->_quiche_con, stream_id, buf, len, &fin, &error_code); +} + +bool +QUICNetVConnection::stream_read_finished(QUICStreamId stream_id) +{ + return quiche_conn_stream_finished(this->_quiche_con, stream_id); +} + +int64_t +QUICNetVConnection::stream_write_capacity(QUICStreamId stream_id) +{ + return quiche_conn_stream_capacity(this->_quiche_con, stream_id); +} + +int64_t +QUICNetVConnection::write_stream(QUICStreamId stream_id, const uint8_t *buf, size_t len, bool fin, + QUICStreamIO::ErrorCode &error_code) +{ + return quiche_conn_stream_send(this->_quiche_con, stream_id, const_cast(buf), len, fin, &error_code); +} + void QUICNetVConnection::reenable(int event) { diff --git a/src/iocore/net/SSLSessionTicket.cc b/src/iocore/net/SSLSessionTicket.cc index 7d6674bd64c..58ff5fe46a8 100644 --- a/src/iocore/net/SSLSessionTicket.cc +++ b/src/iocore/net/SSLSessionTicket.cc @@ -53,6 +53,14 @@ ssl_callback_session_ticket(SSL *ssl, unsigned char *keyname, unsigned char *iv, if (srs) { return srs->processSessionTicket(ssl, keyname, iv, cipher_ctx, hctx, enc); } else { +#if TS_HAS_OPENSSL_QUIC + if (SSL_is_quic(ssl) == 1) { + // OpenSSL native QUIC can parse client-offered tickets while servicing + // the listener, before ATS has accepted and bound a QUIC NetVC. + return 0; + } +#endif + // We could implement a default behavior that would have been done if this callback was not registered, but it's not necessary // at the moment because TLSSessionResumptionSupport is always available when the callback is registered. ink_assert(!"srs should be available"); diff --git a/src/iocore/net/TLSSNISupport.cc b/src/iocore/net/TLSSNISupport.cc index b6acd202b2b..55e0a673a4b 100644 --- a/src/iocore/net/TLSSNISupport.cc +++ b/src/iocore/net/TLSSNISupport.cc @@ -162,6 +162,16 @@ TLSSNISupport::set_sni_server_name(SSL *ssl, char const *name) return true; } +void +TLSSNISupport::set_sni_server_name(std::string_view name) +{ + if (name.empty()) { + this->_clear(); + } else { + this->_set_sni_server_name_buffer(name); + } +} + void TLSSNISupport::_clear() { diff --git a/src/iocore/net/quic/CMakeLists.txt b/src/iocore/net/quic/CMakeLists.txt index b3f1b7cb1d4..0d559c72dd5 100644 --- a/src/iocore/net/quic/CMakeLists.txt +++ b/src/iocore/net/quic/CMakeLists.txt @@ -26,16 +26,19 @@ add_library( QUICTypes.cc QUICIntUtil.cc QUICStream.cc - QUICStream.cc QUICStreamManager.cc QUICStreamAdapter.cc QUICStreamVCAdapter.cc ) add_library(ts::quic ALIAS quic) -target_link_libraries(quic PUBLIC ts::inkevent ts::inknet ts::tscore OpenSSL::Crypto OpenSSL::SSL quiche::quiche) +target_link_libraries(quic PUBLIC ts::inkevent ts::inknet ts::tscore OpenSSL::Crypto OpenSSL::SSL) + +if(TS_HAS_QUICHE) + target_link_libraries(quic PUBLIC quiche::quiche) +endif() -if(TS_OPENSSL_QUIC_TLS_CBS_COMPAT) +if(TS_OPENSSL_QUIC_TLS_CBS_COMPAT AND TS_HAS_QUICHE) add_library(openssl_quic_compat STATIC OpenSSLQuicCompat.cc) add_library(ts::openssl_quic_compat ALIAS openssl_quic_compat) target_link_libraries(openssl_quic_compat PUBLIC OpenSSL::SSL OpenSSL::Crypto) diff --git a/src/iocore/net/quic/QUICConfig.cc b/src/iocore/net/quic/QUICConfig.cc index 9905ae06889..7d9fe28927a 100644 --- a/src/iocore/net/quic/QUICConfig.cc +++ b/src/iocore/net/quic/QUICConfig.cc @@ -23,6 +23,9 @@ #include "iocore/net/quic/QUICConfig.h" +#if TS_HAS_OPENSSL_QUIC +#include +#endif #include #include "mgmt/config/ConfigContextDiags.h" @@ -54,6 +57,24 @@ quic_new_ssl_ctx() return ssl_ctx; } +SSL_CTX * +quic_new_server_ssl_ctx() +{ +#if TS_HAS_OPENSSL_QUIC + SSL_CTX *ssl_ctx = SSL_CTX_new(OSSL_QUIC_server_method()); + if (ssl_ctx == nullptr) { + return nullptr; + } + + SSL_CTX_set_min_proto_version(ssl_ctx, TLS1_3_VERSION); + SSL_CTX_set_max_proto_version(ssl_ctx, TLS1_3_VERSION); + + return ssl_ctx; +#else + return quic_new_ssl_ctx(); +#endif +} + /** ALPN and SNI should be set to SSL object with NETVC_OPTIONS **/ @@ -429,6 +450,7 @@ QUICConfigParams::disable_http_0_9() const return this->_disable_http_0_9; } +#if TS_HAS_QUICHE quiche_cc_algorithm QUICConfigParams::get_cc_algorithm() const { @@ -441,6 +463,7 @@ QUICConfigParams::get_cc_algorithm() const return QUICHE_CC_RENO; } } +#endif // // QUICConfig diff --git a/src/iocore/net/quic/QUICGlobals.cc b/src/iocore/net/quic/QUICGlobals.cc index b44366f8d2b..57d7b4d067d 100644 --- a/src/iocore/net/quic/QUICGlobals.cc +++ b/src/iocore/net/quic/QUICGlobals.cc @@ -54,8 +54,7 @@ QUIC::init() int QUIC::ssl_client_new_session([[maybe_unused]] SSL *ssl, [[maybe_unused]] SSL_SESSION *session) { -#if TS_HAS_QUICHE -#else +#if !TS_HAS_OPENSSL_QUIC && !TS_HAS_QUICHE QUICTLS *qtls = static_cast(SSL_get_ex_data(ssl, QUIC::ssl_quic_tls_index)); const char *session_file = qtls->session_file(); auto file = BIO_new_file(session_file, "w"); diff --git a/src/iocore/net/quic/QUICStream.cc b/src/iocore/net/quic/QUICStream.cc index cdc427c9af8..9f8de91fa0a 100644 --- a/src/iocore/net/quic/QUICStream.cc +++ b/src/iocore/net/quic/QUICStream.cc @@ -118,36 +118,43 @@ QUICStream::on_eos() } void -QUICStream::receive_data(quiche_conn *quiche_con) +QUICStream::receive_data(QUICStreamIO &stream_io) { uint8_t buf[4096]; - bool fin; - ssize_t read_len = 0; - [[maybe_unused]] ErrorCode error_code{0}; // Only set if QUICHE_ERR_STREAM_STOPPED(-15) or QUICHE_ERR_STREAM_RESET(-16) are - // returned by quiche_conn_stream_recv. + bool fin = false; + bool delivered = false; + ssize_t read_len = 0; + [[maybe_unused]] ErrorCode error_code{0}; - while ((read_len = quiche_conn_stream_recv(quiche_con, this->_id, buf, sizeof(buf), &fin, &error_code)) > 0) { + while ((read_len = stream_io.read_stream(this->_id, buf, sizeof(buf), fin, error_code)) > 0) { this->_adapter->write(this->_received_bytes, buf, read_len, fin); this->_received_bytes += read_len; + delivered = true; } - this->_has_no_more_data = quiche_conn_stream_finished(quiche_con, this->_id); - this->_adapter->encourge_read(); + if (read_len == 0 && fin && !this->_has_no_more_data) { + this->_adapter->write(this->_received_bytes, buf, 0, true); + delivered = true; + } + this->_has_no_more_data = stream_io.stream_read_finished(this->_id); + + if (delivered) { + this->_adapter->encourge_read(); + } } -void -QUICStream::send_data(quiche_conn *quiche_con) +int64_t +QUICStream::send_data(QUICStreamIO &stream_io) { bool fin = false; ssize_t len = 0; - [[maybe_unused]] ErrorCode error_code{0}; // Only set if QUICHE_ERR_STREAM_STOPPED(-15) or QUICHE_ERR_STREAM_RESET(-16) are - // returned by quiche_conn_stream_send. - size_t written_this_event = 0; + [[maybe_unused]] ErrorCode error_code{0}; + size_t written_this_event = 0; while (written_this_event < MAX_STREAM_SEND_BYTES_PER_EVENT) { - len = quiche_conn_stream_capacity(quiche_con, this->_id); + len = stream_io.stream_write_capacity(this->_id); if (len <= 0) { - return; + return written_this_event; } if (!this->_pending_send_block) { @@ -156,13 +163,14 @@ QUICStream::send_data(quiche_conn *quiche_con) if (!this->_pending_send_block) { if (!this->_sent_fin && this->_adapter->is_eos() && this->_adapter->total_len() == this->_sent_bytes) { static constexpr uint8_t empty_data = 0; - ssize_t written_len = quiche_conn_stream_send(quiche_con, this->_id, &empty_data, 0, true, &error_code); + ssize_t written_len = stream_io.write_stream(this->_id, &empty_data, 0, true, error_code); if (written_len >= 0) { this->_sent_fin = true; + return written_this_event + static_cast(written_len); } } this->_adapter->encourge_write(); - return; + return written_this_event; } this->_pending_send_fin = this->_adapter->total_len() == this->_sent_bytes + this->_pending_send_block->size(); } @@ -177,28 +185,30 @@ QUICStream::send_data(quiche_conn *quiche_con) } if (block->size() > 0 || fin) { - ssize_t written_len = quiche_conn_stream_send(quiche_con, this->_id, reinterpret_cast(block->start()), - block->size(), fin, &error_code); + ssize_t written_len = + stream_io.write_stream(this->_id, reinterpret_cast(block->start()), block->size(), fin, error_code); if (written_len >= 0) { this->_adapter->consume(written_len); this->_sent_bytes += written_len; - written_this_event += written_len; + written_this_event += static_cast(written_len); if (written_len >= block->size()) { this->_pending_send_block = nullptr; this->_pending_send_fin = false; this->_sent_fin = fin; } else { block->consume(written_len); - return; + return written_this_event; } if (!this->has_data_to_send()) { this->_adapter->encourge_write(); - return; + return written_this_event; } continue; } } this->_adapter->encourge_write(); - return; + return written_this_event; } + + return written_this_event; } diff --git a/src/iocore/net/quic/QUICStreamManager.cc b/src/iocore/net/quic/QUICStreamManager.cc index b3b82692fa4..4fbeb7b870f 100644 --- a/src/iocore/net/quic/QUICStreamManager.cc +++ b/src/iocore/net/quic/QUICStreamManager.cc @@ -113,14 +113,38 @@ QUICStreamManager::create_stream(QUICStreamId stream_id, QUICConnectionError & / } QUICConnectionErrorUPtr -QUICStreamManager::create_uni_stream(QUICStreamId /* new_stream_id ATS_UNUSED */) +QUICStreamManager::create_uni_stream(QUICStreamId &new_stream_id) { + if (this->_next_local_uni_stream_id == 0) { + this->_next_local_uni_stream_id = + static_cast(this->_context->connection_info()->direction() == NET_VCONNECTION_IN ? QUICStreamType::SERVER_UNI : + QUICStreamType::CLIENT_UNI); + } + + new_stream_id = this->_next_local_uni_stream_id; + this->_next_local_uni_stream_id += 4; + + [[maybe_unused]] QUICConnectionError err; + this->create_stream(new_stream_id, err); + return nullptr; } QUICConnectionErrorUPtr -QUICStreamManager::create_bidi_stream(QUICStreamId /* new_stream_id ATS_UNUSED */) +QUICStreamManager::create_bidi_stream(QUICStreamId &new_stream_id) { + if (this->_next_local_bidi_stream_id == 0) { + this->_next_local_bidi_stream_id = + static_cast(this->_context->connection_info()->direction() == NET_VCONNECTION_IN ? QUICStreamType::SERVER_BIDI : + QUICStreamType::CLIENT_BIDI); + } + + new_stream_id = this->_next_local_bidi_stream_id; + this->_next_local_bidi_stream_id += 4; + + [[maybe_unused]] QUICConnectionError err; + this->create_stream(new_stream_id, err); + return nullptr; } diff --git a/src/proxy/http3/Http3Frame.cc b/src/proxy/http3/Http3Frame.cc index 6fc7d5b5441..2c15ee03ef3 100644 --- a/src/proxy/http3/Http3Frame.cc +++ b/src/proxy/http3/Http3Frame.cc @@ -266,6 +266,12 @@ Http3DataFrame::payload_length() const return this->_length; } +bool +Http3DataFrame::_parse() +{ + return this->_reader->read_avail() >= static_cast(this->_length); +} + IOBufferReader * Http3DataFrame::data() const { diff --git a/src/proxy/http3/test/test_Http3FrameDispatcher.cc b/src/proxy/http3/test/test_Http3FrameDispatcher.cc index 37560e9a157..0e8e7324035 100644 --- a/src/proxy/http3/test/test_Http3FrameDispatcher.cc +++ b/src/proxy/http3/test/test_Http3FrameDispatcher.cc @@ -92,7 +92,7 @@ TEST_CASE("Http3FrameHandler dispatch", "[http3]") total_nread += nread; CHECK(!error); } - CHECK(handler.total_frame_received == 5); + CHECK(handler.total_frame_received == 1); CHECK(total_nread == 19); } } diff --git a/src/traffic_layout/info.cc b/src/traffic_layout/info.cc index e77c2926d3e..9b1e6c331c1 100644 --- a/src/traffic_layout/info.cc +++ b/src/traffic_layout/info.cc @@ -160,6 +160,7 @@ produce_features(bool json) print_feature("TS_USE_HWLOC", TS_USE_HWLOC, json); print_feature("TS_USE_TLS13", TS_USE_TLS13, json); print_feature("TS_USE_QUIC", TS_USE_QUIC, json); + print_feature("TS_HAS_OPENSSL_QUIC", TS_HAS_OPENSSL_QUIC, json); print_feature("TS_HAS_QUICHE", TS_HAS_QUICHE, json); print_feature("TS_HAS_SO_PEERCRED", TS_HAS_SO_PEERCRED, json); print_feature("TS_USE_REMOTE_UNWINDING", TS_USE_REMOTE_UNWINDING, json); diff --git a/src/traffic_server/traffic_server.cc b/src/traffic_server/traffic_server.cc index 0426cf8a1c2..8015e5db9c9 100644 --- a/src/traffic_server/traffic_server.cc +++ b/src/traffic_server/traffic_server.cc @@ -77,7 +77,7 @@ extern "C" int plock(int); #include "Crash.h" #include "tscore/signals.h" #include "../iocore/net/P_Net.h" -#if TS_HAS_QUICHE +#if TS_USE_QUIC == 1 #include "../iocore/net/P_QUICNetProcessor.h" #endif #include "../iocore/net/P_UDPNet.h" diff --git a/tests/gold_tests/headers/via.test.py b/tests/gold_tests/headers/via.test.py index 841736340ec..0a9f527cf07 100644 --- a/tests/gold_tests/headers/via.test.py +++ b/tests/gold_tests/headers/via.test.py @@ -115,7 +115,7 @@ tr.StillRunningAfter = ts # HTTP 3 - if Condition.HasATSFeature('TS_HAS_QUICHE') and Condition.HasCurlFeature('http3'): + if Condition.HasATSFeature('TS_USE_QUIC') and Condition.HasCurlFeature('http3'): tr = Test.AddTestRun() tr.MakeCurlCommand( '--verbose --ipv4 --http3 --insecure --header "Host: www.example.com" https://localhost:{}'.format(