[Enhancement]Extend SimulcastEncoderAdapter to support adaptive streaming

[ipavlidakis]

Summary

This fixes quality scaler interference during simulcast transitions and ensures
the SimulcastEncoderAdapter (SEA) correctly reports the active encoder's runtime
adaptation hints when only one spatial layer is active.

Without this fix, the quality scaler remains active during simulcast and
continuously degrades the input resolution, causing all simulcast layer
dimensions to be derived from scaled-down frames (e.g. f=268x480 instead of
f=720x1280 for 720p capture). Additionally, the SEA was reporting aggregated
encoder info with scaling_settings = kOff even when only one layer was active,
preventing the quality scaler from starting in single-active-layer mode (1:1 calls).

Problem

Three interrelated issues combined to produce broken quality adaptation behavior
during simulcast transitions:

1. Quality scaler active during simulcast. When transitioning from a 1:1
   call to a group call (simulcast), the quality scaler was not disabled. It
   continued to scale down the input resolution, and since simulcast layer
   dimensions are computed from the input frame size, all layers (q, h, f) were
   recomputed from the degraded input. This produced cascading downgrades
   (quality:1 → quality:2 → quality:3 → quality:4) and eventually triggered
   simulcast layer count reduction from 3 to 2.

2. SEA reporting aggregated info in single-active-layer mode. The
   SimulcastEncoderAdapter only forwarded the active encoder's info when
   stream_contexts_.size() == 1. At runtime, SEA can have multiple stream
   contexts while only one layer is unpaused. In that state, SEA reported
   scaling_settings = kOff, so the quality scaler could never start — even in
   1:1 mode where it's needed.

3. Conservative min_pixels_per_frame floor for iOS encoders. The ObjC
   encoder bridge used the 2-argument ScalingSettings constructor, inheriting
   kDefaultMinPixelsPerFrame = 320*180 = 57600. This prevented the quality
   scaler from reducing resolution below ~180x320 (the quarter-resolution
   simulcast layer for 720p), limiting adaptation range on constrained networks.

Root Cause

Quality scaler during simulcast: ConfigureQualityScaler() in
VideoStreamEncoderResourceManager had no awareness of whether simulcast was
active. It would start/keep the quality scaler running whenever the encoder
reported QP thresholds or is_quality_scaling_allowed was set, regardless of
how many simulcast layers were active.

SEA encoder info: The bug is in SimulcastEncoderAdapter::GetEncoderInfo().
The old logic treated "single stream context" as equivalent to "single active
layer", but those are not the same thing at runtime. Once SEA enters
multi-encoder mode, the number of stream contexts stays greater than one even if
only one layer is currently unpaused.

FindRequiredActiveLayers: FindRequiredActiveLayers() in
encoder_stream_factory.cc returned the position of the first active layer
instead of the highest, which could discard upper active layers when lower ones
were inactive.

min_pixels_per_frame: The ObjC bridge in objc_video_encoder_factory.mm
used ScalingSettings(low, high) which defaults min_pixels_per_frame to
57600 — too conservative for iOS VideoToolbox encoders that can encode well
below that threshold.

Fix

1. Quality scaler suppression during simulcast

• Added simulcast_active_ flag to VideoStreamEncoderResourceManager.
• SetSimulcastActive(bool) is called from ReconfigureEncoder() based on the
  number of active simulcast layers (not configured streams — the config may
  always specify 3 streams while the SFU controls activation).
• ConfigureQualityScaler() now checks !simulcast_active_ as an additional
  condition for quality_scaling_allowed. When simulcast is active, the quality
  scaler is stopped and removed.

2. Simulcast transition handling in VideoStreamEncoder

• ReconfigureEncoder() now detects transitions between single-stream and
  simulcast based on num_active_layers > 1.
• Transition to simulcast (e.g. 3rd participant joins): disables the quality
  scaler and calls ResetAdaptationsForSimulcastChange() to clear accumulated
  resolution restrictions. The video source then delivers full-resolution frames
  for correct layer dimension computation.
• Transition from simulcast (e.g. 3rd participant leaves): re-enables the
  quality scaler from a clean state (zero downgrades).
• Added diagnostic logging at every transition point for operational debugging.

3. SEA single-active-layer encoder info forwarding

• When exactly one stream is unpaused, SEA now forwards the active encoder's
  runtime-sensitive fields: scaling_settings, supports_native_handle,
  has_trusted_rate_controller, is_hardware_accelerated, is_qp_trusted,
  resolution_bitrate_limits, min_qp, preferred_pixel_formats.
• SEA still preserves its aggregated simulcast fields: implementation_name,
  fps_allocation, requested_resolution_alignment,
  apply_alignment_to_all_simulcast_layers.
• When more layers become active again, SEA returns to aggregated behavior.

4. FindRequiredActiveLayers fix

• Changed FindRequiredActiveLayers() to return the position after the highest
  active layer (not the first), ensuring all active layers are preserved in the
  stream count.

5. Lower min_pixels_per_frame for iOS encoders

• Changed the ObjC encoder bridge to use ScalingSettings(low, high, 90*160)
  instead of ScalingSettings(low, high). This lowers the quality scaler floor
  from 57600 to 14400 pixels, allowing adaptation below the quarter-resolution
  simulcast layer on constrained networks (e.g. 3G).

Why This Is Safe

• The quality scaler suppression is strictly scoped to the multi-active-layer
  state. Single-active-layer behavior (1:1 calls) is unchanged.
• Restriction clearing on simulcast transition is self-correcting: even if the
  first frame after clearing is still degraded, the source delivers
  full-resolution frames on the next frame, triggering a reconfigure with
  correct dimensions.
• The SEA change does not affect init-time encoder selection, bypass mode,
  pre-init GetEncoderInfo(), bitrate allocation, or true multi-layer
  aggregation.
• The min_pixels_per_frame change only affects ObjC-bridged encoders (iOS
  VideoToolbox). The global kDefaultMinPixelsPerFrame remains unchanged for
  software and Android encoders.

Tests

Added regression coverage in:

• video_stream_encoder_unittest.cc: quality scaler restrictions are cleared
  when active layers increase from 1 to multiple, and from 2 to 3.
• simulcast_encoder_adapter_unittest.cc: forwarding active encoder info when
  only one layer is unpaused; restoring aggregated SEA behavior when a second
  layer becomes active.
• encoder_stream_factory_unittest.cc: stream count preservation for sparse
  active patterns, highest-only active, lowest-only active.

Validation

Validated with on-device iOS testing:

• 1:1 call on 3G: quality scaler correctly adapts resolution down and back up.
• 3rd participant joins: simulcast activates, quality scaler disables, layer
  dimensions match configured ratios from full capture resolution.
• 3rd participant leaves: quality scaler re-enables from clean state.
• Repeated transitions: no accumulated degradation across cycles.

Summary by CodeRabbit

• Bug Fixes

  • Fixed simulcast layer count calculation to properly account for highest active layer, not just the first active layer.

• New Features

  • Enhanced video quality adaptation for simulcast scenarios by introducing intelligent suppression of quality scaling when multiple encoding layers are active.
  • Improved encoder information reporting and scaling behavior for single-layer encoding within simulcast configurations.

• Tests

  • Added comprehensive test coverage for quality-scaler behavior and encoder info forwarding during simulcast transitions.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6572e957-955a-4d23-8916-2f2aa75c2349

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch iliaspavlidakis/ios-1481-webrtcextend-simulcastencoderadapter-to-support-adaptive

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

🧹 Nitpick comments (1)

video/video_stream_encoder_unittest.cc (1)

6830-6844: Consider extracting a small helper for 3-layer VP8 config construction.

These repeated setup blocks differ mostly by active-layer flags; a helper would reduce duplication and keep future updates safer.

Also applies to: 6860-6873, 6900-6913, 6930-6943

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@video/video_stream_encoder_unittest.cc` around lines 6830 - 6844, Extract a
small helper (e.g., Build3LayerVp8Config or MakeVp8SimulcastConfig) that wraps
creating a VideoEncoderConfig via
test::FillEncoderConfiguration(PayloadStringToCodecType("VP8"), 3, &config),
sets video_stream_factory=nullptr, sets each
simulcast_layers[*].num_temporal_layers=1 and .max_framerate=kDefaultFramerate,
assigns max_bitrate_bps=kSimulcastTargetBitrate.bps() and
content_type=VideoEncoderConfig::ContentType::kRealtimeVideo, and accepts a
parameter to mark which of the three simulcast_layers is active (or a three-bool
vector); then replace the repeated blocks that manually toggle
config_1_active.simulcast_layers[i].active (occurrences around the references
shown) with calls to this helper to reduce duplication and centralize future
changes.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@media/engine/simulcast_encoder_adapter_unittest.cc`:
- Around line 1761-1768: The test's fps_allocation expectations are inconsistent
with the current adapter/mock contract: MockVideoEncoder::SetRates() does not
clear fps_allocation_ for paused layers and
SimulcastEncoderAdapter::GetEncoderInfo() copies
encoder_impl_info.fps_allocation[0] into each stream context, so update the
assertions to match that behavior (i.e., expect slot 0 to remain populated with
the low encoder's allocation rather than IsEmpty()). Locate the failing
assertions around info.fps_allocation in the test and change them to assert the
populated values (mirroring low_encoder->set_fps_allocation(...) ) and apply the
same change to the similar block at lines referenced (around 1825-1831), instead
of modifying production copying or mock SetRates behavior.

In `@video/adaptation/video_stream_encoder_resource_manager.cc`:
- Around line 562-598: ResetAdaptationsForSimulcastChange clears
bandwidth_quality_scaler_resource_ but the subsequent reconfigure path
(ConfigureBandwidthQualityScaler) can immediately re-add it for encoders with
is_qp_trusted == false because that function doesn't consider simulcast_active_.
Modify ConfigureBandwidthQualityScaler (or the place that adds
bandwidth_quality_scaler_resource_) to check simulcast_active_ (and/or
is_qp_trusted) and skip creating/adding bandwidth_quality_scaler_resource_ when
we're handling a simulcast active-layer transition (i.e., simulcast_active_
indicates the change) for untrusted-QP encoders; ensure the check uses the
existing members bandwidth_quality_scaler_resource_, simulcast_active_, and
is_qp_trusted to avoid re-adding the resource during this reset flow.

In `@video/video_stream_encoder.cc`:
- Around line 1062-1127: The code currently determines simulcast state from
encoder_config_.simulcast_layers which only updates on reconfigure; instead,
derive the active-layer count from the runtime EncoderInfo/state (the info
passed into ConfigureQualityScaler/EncodeVideoFrame or the current per-layer
sending state exposed by SetRates) so transitions reflect actual sending layers.
Change the logic that computes num_active_layers (and the subsequent
is_simulcast/was_simulcast decisions and calls to
stream_resource_manager_.SetSimulcastActive(...) and
ResetAdaptationsForSimulcastChange()) to use the runtime active-layer count from
the EncoderInfo/runtime layer state, and update
prev_num_active_simulcast_layers_ from that runtime count so the quality scaler
is enabled/disabled based on actual sending layers rather than only
encoder_config_.simulcast_layers.

---

Nitpick comments:
In `@video/video_stream_encoder_unittest.cc`:
- Around line 6830-6844: Extract a small helper (e.g., Build3LayerVp8Config or
MakeVp8SimulcastConfig) that wraps creating a VideoEncoderConfig via
test::FillEncoderConfiguration(PayloadStringToCodecType("VP8"), 3, &config),
sets video_stream_factory=nullptr, sets each
simulcast_layers[*].num_temporal_layers=1 and .max_framerate=kDefaultFramerate,
assigns max_bitrate_bps=kSimulcastTargetBitrate.bps() and
content_type=VideoEncoderConfig::ContentType::kRealtimeVideo, and accepts a
parameter to mark which of the three simulcast_layers is active (or a three-bool
vector); then replace the repeated blocks that manually toggle
config_1_active.simulcast_layers[i].active (occurrences around the references
shown) with calls to this helper to reduce duplication and centralize future
changes.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 33f64430-927d-4742-9af5-29b0e9aed35d

📥 Commits

Reviewing files that changed from the base of the PR and between 46a5408 and bbbb3bf.

📒 Files selected for processing (10)

• media/engine/simulcast_encoder_adapter.cc
• media/engine/simulcast_encoder_adapter_unittest.cc
• sdk/objc/native/src/objc_video_encoder_factory.mm
• video/adaptation/video_stream_encoder_resource_manager.cc
• video/adaptation/video_stream_encoder_resource_manager.h
• video/config/encoder_stream_factory.cc
• video/config/encoder_stream_factory_unittest.cc
• video/video_stream_encoder.cc
• video/video_stream_encoder.h
• video/video_stream_encoder_unittest.cc

[ipavlidakis]

Pushed 5c6789c5ed to address the review feedback.

I also folded in the review-summary nit by extracting Make3LayerVp8SimulcastConfig(...) in video_stream_encoder_unittest.cc, which removes the repeated 3-layer VP8 setup blocks used by the simulcast transition tests.

Verified locally with:

• ./rtc_media_unittests --gtest_filter="TestSimulcastEncoderAdapterFake.ForwardsRuntimeSensitiveEncoderInfoForSingleUnpausedLayer:TestSimulcastEncoderAdapterFake.RestoresAggregatedEncoderInfoWhenMultipleLayersUnpause"
• ./video_engine_tests --gtest_filter="VideoStreamEncoderTest.QualityScalerRestrictionsResetWhenActiveLayersIncrease:VideoStreamEncoderTest.QualityScalerRestrictionsResetOnTwoToThreeLayerIncrease:VideoStreamEncoderTest.BandwidthQualityScalerRemovedWhenTransitioningToSimulcast:VideoStreamEncoderTest.QualityScalerResourceTracksRuntimeLayerAllocation"