feat(skills): add demo-video skill for narrated video assembly - #2404
Open
auyidi1 wants to merge 23 commits into
Open
feat(skills): add demo-video skill for narrated video assembly#2404auyidi1 wants to merge 23 commits into
auyidi1 wants to merge 23 commits into
Conversation
Add an experimental demo-video skill that assembles ordered frames or clips with per-segment narration into a narrated MP4 via FFmpeg. Closes the video-assembly gap in the Design Thinking demo workflow by bridging tts-voiceover narration and vscode-playwright frame capture. - Add SKILL.md with manifest schema, quick start, and reuse bridge - Add Python assembler (assemble_video.py) with segments.yml validation, ffprobe-derived durations, and safe list-arg subprocess calls - Add bash and PowerShell wrappers mirroring the video-to-gif convention - Add pytest suite, Atheris fuzz harness, corpus fixtures, and Pester helper tests - Register the skill in the experimental and hve-core-all collections and regenerate plugin outputs
Contributor
Dependency ReviewThe following issues were found:
License Issues.github/skills/experimental/demo-video/uv.lock
OpenSSF ScorecardScorecard details
Scanned Files
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #2404 +/- ##
==========================================
+ Coverage 82.76% 85.62% +2.85%
==========================================
Files 154 87 -67
Lines 20853 9132 -11721
Branches 13 13
==========================================
- Hits 17259 7819 -9440
+ Misses 3592 1311 -2281
Partials 2 2
Flags with carried forward coverage won't be shown. Click here to find out more.
🚀 New features to boost your workflow:
|
Contributor
Eval Execution✅ Status: Passed — no merge-blocking failures (6 advisory assertion failure(s) present)
|
Use the canonical "Copyright (c) 2026 Microsoft Corporation. All rights reserved." form so Copyright Header Validation passes on PR #2404.
… narration Recommend the tts-voiceover skill's Azure AI Speech (Azure AI Foundry) neural voices as the default for shareable narration and mark offline open-source TTS as a no-credential fallback only.
Add knowledge, tool-trigger, and bleed-detection stimuli for the demo-video skill so the eval stimulus-presence gate passes on PR #2404.
Replace the truncated 'narrat' stem with 'narration|narrated' so Spell Check passes on PR #2404.
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a new experimental demo-video skill to assemble ordered prototype frames/clips and per-segment narration WAVs into a single narrated MP4 using FFmpeg, and registers the skill across collections, plugins, and behavior evals.
Changes:
- Added the
demo-videoskill package (docs, Python assembler CLI, bash/PowerShell wrappers, examples, and tests including a fuzz harness). - Registered
demo-videoinexperimentalandhve-core-allcollections and regenerated plugin READMEs. - Extended skill behavior conformance eval coverage with
demo-videoknowledge/tool-trigger/bleed-detection stimuli.
Reviewed changes
Copilot reviewed 22 out of 23 changed files in this pull request and generated 6 comments.
Show a summary per file
| File | Description |
|---|---|
| plugins/hve-core-all/README.md | Adds demo-video to the packaged skills list |
| plugins/experimental/README.md | Adds demo-video to the experimental skills list |
| evals/behavior-conformance/skill-behavior.eval.yaml | Adds 3 new demo-video behavior-conformance stimuli and updates total count |
| collections/hve-core-all.collection.yml | Registers demo-video skill in the hve-core-all collection |
| collections/hve-core-all.collection.md | Lists demo-video in the hve-core-all collection documentation table |
| collections/experimental.collection.yml | Registers demo-video skill in the experimental collection |
| collections/experimental.collection.md | Lists demo-video in the experimental collection documentation table |
| .github/skills/experimental/demo-video/SKILL.md | Documents the skill purpose, manifest schema, usage, and prerequisites |
| .github/skills/experimental/demo-video/pyproject.toml | Declares Python dependencies + pytest/ruff config for the skill |
| .github/skills/experimental/demo-video/uv.lock | Pins the Python environment for the skill |
| .github/skills/experimental/demo-video/scripts/assemble_video.py | Implements the manifest-driven FFmpeg assembly CLI |
| .github/skills/experimental/demo-video/scripts/assemble-video.sh | Bash wrapper to uv sync and run the Python assembler |
| .github/skills/experimental/demo-video/scripts/Invoke-AssembleVideo.ps1 | PowerShell wrapper to uv sync and run the Python assembler |
| .github/skills/experimental/demo-video/tests/test_assemble_video.py | Pytest coverage for manifest parsing/validation and ffmpeg subprocess safety |
| .github/skills/experimental/demo-video/tests/fuzz_harness.py | Adds an Atheris/pytest-compatible fuzz harness for the manifest validator |
| .github/skills/experimental/demo-video/tests/DemoVideoHelpers.Tests.ps1 | Adds Pester tests for wrapper parameter/argument forwarding |
| .github/skills/experimental/demo-video/tests/corpus/valid-manifest.yml | Adds fuzz/test corpus fixture for a valid manifest shape |
| .github/skills/experimental/demo-video/tests/corpus/unknown-key.yml | Adds fuzz/test corpus fixture for unknown-key validation |
| .github/skills/experimental/demo-video/tests/corpus/empty-segments.yml | Adds fuzz/test corpus fixture for empty segments validation |
| .github/skills/experimental/demo-video/examples/segments.yml | Adds an example segments manifest |
| .github/skills/experimental/demo-video/examples/README.md | Documents how to set up local fixtures for the example manifest |
Comments suppressed due to low confidence (1)
.github/skills/experimental/demo-video/examples/README.md:30
- The file ends with a whitespace-only line (trailing spaces). This tends to fail markdownlint/formatters; remove the trailing spaces/extra blank line at EOF.
*🤖 Crafted with precision by ✨Copilot following brilliant human instruction, then carefully refined by our team of discerning human reviewers.*
rezatnoMsirhC
approved these changes
Jul 9, 2026
- Require PowerShell 7.4 in the wrapper to match repo conventions - Make uv install hint platform-neutral (winget/curl) - Remove dead subprocess.CalledProcessError handler - Emit debug logging so --verbose has observable effect - Drop unused manifest_path parameter from _validate_manifest - Use H2 heading in examples README (title in frontmatter) - Fix Pester fallback repo-root walk (5 levels) - Remove stale 'planned' qualifier in SKILL.md
The _validate_manifest signature no longer takes manifest_path; update the pytest and fuzz-harness call sites accordingly.
jkim323
approved these changes
Jul 11, 2026
# Conflicts: # collections/hve-core-all.collection.md # evals/behavior-conformance/skill-behavior.eval.yaml # plugins/hve-core-all/README.md
chaosdinosaur
previously requested changes
Jul 13, 2026
chaosdinosaur
left a comment
chaosdinosaur
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for the demo-video skill — the assembler/wrapper design is clean, the subprocess handling is injection-safe (list args, no shell=True, yaml.safe_load), and the repo Python-skill conventions (ruff/pytest/fuzz/uv.lock, dual wrappers, copyright headers) are all met. A few items I didn't see raised elsewhere on the PR:
- Blocking: the shipped
examples/segments.ymluses./examples/fixtures/...paths, but the assembler resolves relative paths against the manifest's own directory (examples/), so they expand toexamples/examples/fixtures/...and don't match the fixture layoutexamples/README.mddocuments — the flagship example fails on first run. Changing the manifest paths to./fixtures/...(andoutput: ./output/demo.mp4) fixes it; please verify end-to-end since the PR notes the smoke run was skipped. - The
npm run test:py 13 passedfigure in the PR body isn't reproducible — the two discoverable pytest files contain 11 test functions; please re-run and update the count. tests/DemoVideoHelpers.Tests.ps1doesn't mirror its system under test (Invoke-AssembleVideo.ps1); renaming toInvoke-AssembleVideo.Tests.ps1matches the Pester convention.- Minor:
SKILL.md's Manifest Schema omits the top-leveloutput/resolution/fpskeys the code and example use; an empty-stringnarrationskips thenarration_wavfallback; and_validate_manifestreturns the segments list redundantly.
Note
This review was AI-assisted and should be independently validated by a qualified human reviewer before acting on any finding.
added 2 commits
July 13, 2026 18:47
…z scope Validate effective frame rate is positive after merging CLI and manifest values, enforce that a declared segment type matches its visual or clip source, and narrow fuzz suppression to ManifestError so unexpected crashes surface. Adds tests for fps and type validations.
Coerce unexpected manifest keys to str when building error messages so non-string YAML keys raise ManifestError instead of a TypeError that crashes the Atheris fuzz target. Adds a regression test and applies ruff formatting.
bindsi
approved these changes
Jul 13, 2026
![github-actions[bot]](https://avatars.githubusercontent.com/in/15368?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Dependency Review — demo-video Skill
This PR introduces a new Python skill with the following dependencies. No existing workflow, devcontainer, or copilot-setup-steps.yml files are modified, so SHA-pinning and environment-sync checks are not applicable.
Dependencies Introduced
| Package | Resolved | Group | License | Notes |
|---|---|---|---|---|
pyyaml |
6.0.3 | runtime | MIT | YAML manifest parsing — necessary, no alternative in stdlib |
pytest |
9.1.1 | dev | MIT | Standard test runner |
pytest-cov |
7.1.0 | dev | MIT | Coverage reporting |
pytest-mock |
3.15.1 | dev | MIT | Mock support for pytest |
ruff |
0.15.20 | dev | MIT | Linter — consistent with other skills |
atheris |
3.1.0 | fuzz | Apache-2.0 | OSSF fuzz harness — consistent with repo convention |
Transitive (colorama, coverage, iniconfig, packaging, pluggy, pygments, tomli) |
— | indirect | MIT/BSD/Apache | All permissive, no concerns |
Safety Checks
| Check | Result |
|---|---|
| License compatibility (MIT project) | ✅ All permissive (MIT, Apache-2.0, BSD-3-Clause) |
| New dependencies justified | ✅ pyyaml is necessary for manifest parsing; dev/fuzz deps follow repo convention |
| Duplicates existing functionality | ✅ No duplication detected |
| GitHub Actions SHA pinning | ✅ No workflow files modified |
| Devcontainer / setup alignment | ✅ Neither .devcontainer/ nor copilot-setup-steps.yml touched |
| Known vulnerabilities | ✅ No known CVEs for these versions |
All dependency safety checks passed. No action required from this review.
Generated by Dependabot PR Review for #2404 · 39.3 AIC · ⌖ 7.05 AIC · ⊞ 8.5K
Contributor
There was a problem hiding this comment.
Dependency Review — demo-video skill
Scope: pyproject.toml + uv.lock added under .github/skills/experimental/demo-video/
Dependencies introduced
| Package | Version constraint | Group | License | Notes |
|---|---|---|---|---|
pyyaml |
>=6.0 |
runtime | MIT | Actively maintained; widely used |
pytest |
>=9.0 |
dev | MIT | Standard test framework |
pytest-cov |
>=5.0 |
dev | MIT | Coverage plugin |
pytest-mock |
>=3.14 |
dev | MIT | Mock plugin |
ruff |
>=0.15 |
dev | MIT | Linter/formatter |
atheris |
>=3.0 |
fuzz | Apache-2.0 | OSS fuzz harness; Apache-2.0 is MIT-compatible |
Indirect lock-file entries (colorama, coverage, iniconfig, packaging, pluggy, pygments, tomli) are all MIT, BSD, or Apache-2.0 — all compatible with the project's MIT license.
Safety check results
- ✅ License compatibility — all dependencies (runtime and dev) carry MIT, BSD, or Apache-2.0 licenses, compatible with the project's MIT license.
- ✅ No duplication —
pyyamlis the only runtime dependency; no existing repo-level dependency provides YAML parsing for this skill's use case. - ✅ SHA/hash pinning —
uv.lockpins every wheel and sdist by SHA-256 hash. No GitHub Actions references were modified in this PR. - ✅ Environment synchronization — no changes to
.devcontainer/orcopilot-setup-steps.yml; synchronization concern is N/A. - ✅ Vulnerability posture — all packages are well-established, actively maintained, and carry no known vulnerability signals.
atherisis correctly isolated in thefuzzdependency group (no macOS wheels) per repo convention. - ✅
uv.lockcommitted — lock file is present alongsidepyproject.toml, satisfying Dependabot resolution requirements.
All dependency safety checks passed. No findings require changes.
Generated by Dependabot PR Review for #2404 · 42.7 AIC · ⌖ 8.7 AIC · ⊞ 8.5K
… tts-voiceover The main merge left collections/hve-core-all.collection.md out of sync with its YAML source: the generated markdown retained the demo-video entry but dropped the tts-voiceover row that the YAML still lists. Regenerate via plugin:generate so the table includes both skills and prevent a merge-time regression that would remove tts-voiceover from main.
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Comment on lines
+1
to
+11
| output: ./examples/output/demo.mp4 | ||
| resolution: 1280x720 | ||
| fps: 24 | ||
| segments: | ||
| - type: frame | ||
| visual: ./examples/fixtures/intro.png | ||
| narration: ./examples/fixtures/intro.wav | ||
| duration: 3.0 | ||
| - type: clip | ||
| clip: ./examples/fixtures/interaction.mp4 | ||
| narration: ./examples/fixtures/interaction.wav |
Contributor
There was a problem hiding this comment.
The assembler resolves manifest-relative paths against manifest_path.parent (the examples/ directory). The paths in this file use the ./examples/... prefix, which causes them to resolve to <skill-root>/examples/examples/... — a location that does not exist. The example will fail on first run.
The examples/README.md documents fixtures at examples/fixtures/, which means the manifest (located at examples/segments.yml) should reference them as ./fixtures/....
Suggested fix:
output: ./output/demo.mp4
resolution: 1280x720
fps: 24
segments:
- type: frame
visual: ./fixtures/intro.png
narration: ./fixtures/intro.wav
duration: 3.0
- type: clip
clip: ./fixtures/interaction.mp4
narration: ./fixtures/interaction.wav
Contributor
There was a problem hiding this comment.
Nit: the repo Pester convention names test files after the script they cover. Since this file tests scripts/Invoke-AssembleVideo.ps1, consider renaming it to Invoke-AssembleVideo.Tests.ps1 to make the source-to-test mapping clear at a glance.
Comment on lines
+28
to
+50
| narration: ./audio/intro.wav | ||
| duration: 4.5 | ||
| - type: clip | ||
| clip: ./clips/interaction.mp4 | ||
| narration: ./audio/interaction.wav | ||
| ``` | ||
|
|
||
| ### Segment fields | ||
|
|
||
| * `type` identifies whether the segment is a still image (`frame`) or a motion clip (`clip`) | ||
| * `visual` points to an image file for a frame segment | ||
| * `clip` points to a motion clip file for a clip segment | ||
| * `narration` points to the WAV file generated from narration text (the script also accepts `narration_wav` as an alias) | ||
| * `duration` is optional and overrides the inferred duration when you want a fixed segment length | ||
|
|
||
| ## Quick Start | ||
|
|
||
| Use the bash or PowerShell wrappers to invoke the assembler from the skill directory. | ||
|
|
||
| ```bash | ||
| scripts/assemble-video.sh --manifest examples/segments.yml --output ./output/demo.mp4 | ||
| ``` | ||
|
|
Contributor
There was a problem hiding this comment.
The Manifest Schema section only shows the segments list. The three top-level keys that the assembler validates and uses (output, resolution, fps) are not documented, so a user reading SKILL.md alone would not know they exist or what their defaults are. The PR description's own schema example includes all three.
Consider expanding the schema example and field table to cover them:
output: ./output/demo.mp4 # optional; path for the assembled MP4
resolution: 1280x720 # optional; default 1280x720
fps: 24 # optional; default 24
segments:
- type: frame
...And add field descriptions alongside the existing segment-field bullet list:
outputsets the destination path for the assembled MP4 (resolved relative to the manifest; defaults to./demo.mp4)resolutioncontrols output width and height inWIDTHxHEIGHTform (default1280x720)fpssets the frame rate applied when rendering each segment (default24)
jkim323
approved these changes
Jul 16, 2026
chaosdinosaur
left a comment
chaosdinosaur
left a comment
Collaborator
There was a problem hiding this comment.
Re-reviewing at 2f2d912 — the type/source enforcement, --fps validation, non-string-key handling, and tighter fuzz scope from the follow-up commits all look good and are covered by new tests. I'm not repeating the items already open on the PR (the examples/segments.yml path-doubling thread and the Pester filename / SKILL.md top-level-keys notes from @rezatnoMsirhC still apply and remain unresolved — the example still fails on first run). A few net-new items I didn't see raised elsewhere:
- Please verify (from the concat change): the final concat now uses
-c copy(per the earlier suggestion) instead of re-encoding. With fractional per-segment durations and fixed-size AAC frames, stream-copy concat can drift A/V sync or leave gaps at segment seams that the previous re-encode masked. Since FFmpeg isn't in the dev container and the smoke run is still skipped, this primary output path is untested — please confirm the final MP4 stays in sync end-to-end, or keep a re-encode fallback for the concat step. - Add a shipped-example regression test: a test that loads
examples/segments.ymland asserts the resolved fixture paths match the README layout would guard the open path-doubling issue from recurring. - PR body count:
npm run test:py 13 passedisn't reproducible — there are now 14 discoverable test functions for this skill (12 intest_assemble_video.py+ 2 infuzz_harness.py). Please re-run and update the figure (or note it''s a repo-wide aggregate). - Minor: an empty-string
narration:still skips thenarration_wavfallback (onlyNone/absent falls back); and_validate_manifestreturns the segments list a second time even though the config dict it returns already containssegments.
Also note the branch is currently BEHIND main and the four approvals predate the still-open 07-15 findings, so a fresh look after the fixes would be worthwhile.
Note
This review was AI-assisted and should be independently validated by a qualified human reviewer before acting on any finding.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds an experimental
demo-videoskill that assembles ordered frames or clips with per-segment narration into a narrated MP4 via FFmpeg. It closes the video-assembly gap in the Design Thinking demo workflow by bridging existing skills:tts-voiceover(narration WAVs) andvscode-playwright(frame capture).What's included
SKILL.md— overview,segments.ymlmanifest schema, quick start, parameters reference, and the reuse bridge totts-voiceover/vscode-playwright, plus FFmpeg prerequisites.scripts/assemble_video.py— the assembler CLI (--manifest,--output,--fps,--resolution,-v/--verbose). Validates the manifest, resolves each segment's visual (frame image or clip) and narration WAV, derives per-segment duration from the narration length viaffprobe(overridable withduration:), renders normalized per-segment MP4s, and concatenates them. Usessubprocess.runwith list args and noshell=True.scripts/assemble-video.sh+scripts/Invoke-AssembleVideo.ps1— dual bash/PowerShell wrappers mirroring thevideo-to-gifconvention; the bash wrapper runsuv syncand forwards to the Python CLI.examples/— an examplesegments.ymlmanifest and a README describing the local fixture setup (fixtures are intentionally kept out of source control).pytestsuite (parse/validation, duration default+override, frame/clip branches, error paths, subprocess safety), an Atheris fuzz harness with corpus fixtures, and a Pester helper test.experimentalandhve-core-allcollections with regenerated plugin outputs.Manifest schema
Validation
npm run validate:skills— passnpm run test:py— 13 passednpm run lint:collections-metadata/plugin:validate— passnpm run lint:frontmatter— pass (demo-video files clean)npm run validate:copyright— passNotes / follow-ups
ffprobemust be on PATH. The end-to-end smoke run was not executed here because FFmpeg is not installed in the dev container.demo-setuporchestration (PR feat(skills): add demo-setup skill to design-thinking collection #1623) that would auto-generate the demo script and capture prototype frames is pending; until it merges, the DT → demo-video wiring is manual.Checklist