Add Picture-in-Picture support for video calls#6231
Add Picture-in-Picture support for video calls#6231gordon-to wants to merge 1 commit intosignalapp:mainfrom
Conversation
Enables system PiP during video calls so users can continue two-way video while using other apps. Remote video renders in the PiP window via AVSampleBufferDisplayLayer (using layerClass + IOSurface-backed buffers), and a local camera preview is shown as a small inset via AVCaptureVideoPreviewLayer. Multitasking camera access keeps the local camera alive during PiP so the remote caller continues receiving video. Key implementation details: - SampleBufferDisplayView uses layerClass override (required for the out-of-process PiP compositor) - IOSurface-backed pixel buffers (non-IOSurface silently dropped) - CIImage rotation for remote frames based on RTCVideoFrame.rotation - CMMotionManager for local preview orientation during background - kCMSampleAttachmentKey_DisplayImmediately for live frame display
|
Found another small bug. When video is off, and you enter pip you still get a black square for your own preview. |
|
Thanks for your interest in contributing! This is a neat feature, and it is something we're interested in integrating eventually. That said, transparently this is a complex feature and a complex PR, and we don't generally accept external contributions of this magnitude since the review and validation process ends up being as-or-more labor-intensive as implementing it internally. Consequently, I'm going to close this PR. That said, when we eventually do get this onto our roadmap we'll be sure to refer back to this PR, and if we end up reusing it we'll make sure to attribute it appropriately! Thanks again for the contribution, and as always for being a Signal user. |
2 participants
First time contributor checklist
Contributor checklist
Description
Enables system PiP during video calls so users can continue two-way video while using other apps. Remote video renders in the PiP window via AVSampleBufferDisplayLayer (using layerClass + IOSurface-backed buffers), and a local camera preview is shown as a small inset via AVCaptureVideoPreviewLayer. Multitasking camera access keeps the local camera alive during PiP so the remote caller continues receiving video.
These changes were tested on a local build in a two-way call over 20 minutes, there are some performance issues and it does not re-sync the frame buffer as fast as I'd like when it started to fall behind.
If the team is open to this contribution I will spend more time profiling and tracking down possible improvements, if this feature is not wanted that's ok too, it was fun to build just for myself, I will continue to envy android users.