diff --git a/README.md b/README.md index 461e7f87..4c922634 100644 --- a/README.md +++ b/README.md @@ -35,7 +35,12 @@ For agent-friendly repos, install the Vizzly skill and add a short project vizzly init --agent-guidance ``` -Use `vizzly init --agent-skill` to install only the local skill, or +This installs the portable Agent Skills bundle in `.agents/skills/vizzly` and +adds a small managed block to `AGENTS.md`. Run the same command again after a +CLI upgrade to refresh both without replacing the rest of `AGENTS.md` or an +existing Vizzly config. + +Use `vizzly init --agent-skill` to install or refresh only the local skill, or `vizzly init --skip-agent-skill` when you want config without the agent prompt. ### Start Local TDD @@ -202,8 +207,8 @@ Generate a config file: vizzly init ``` -To teach project agents about Vizzly screenshot memory and the local visual TDD -loop, add the repo-local skill and AGENTS.md guidance: +To teach project agents the Vizzly evidence workflow, add the repo-local skill +and AGENTS.md guidance: ```bash vizzly init --agent-guidance diff --git a/docs/json-output.md b/docs/json-output.md index 6be1f57e..27ff4d26 100644 --- a/docs/json-output.md +++ b/docs/json-output.md @@ -1126,6 +1126,11 @@ vizzly init --agent-guidance --json } ``` +Run `vizzly init --agent-guidance --json` again after upgrading the CLI to +refresh the portable skill and Vizzly-owned `AGENTS.md` block. Existing output +then reports `"status": "refreshed"` for each updated artifact; unrelated +guidance and an existing Vizzly config remain unchanged. + ### `vizzly project link` ```bash diff --git a/skills/vizzly/SKILL.md b/skills/vizzly/SKILL.md index 6dc02569..b731ecee 100644 --- a/skills/vizzly/SKILL.md +++ b/skills/vizzly/SKILL.md @@ -1,74 +1,64 @@ --- name: vizzly -description: "Inspect, explain, and debug Vizzly visual regression evidence; use an existing Vizzly workflow before and after UI changes; add or adjust screenshot capture; troubleshoot local TDD or cloud builds; or configure Vizzly CI. Use when a task mentions Vizzly, screenshot baselines, comparisons, Honeydiff, visual review, dynamic regions, or a repository's Vizzly setup." +description: Inspect and explain Vizzly visual regression evidence, verify UI changes, troubleshoot local or cloud builds, adjust screenshot capture, or configure Vizzly CI. Use when a task mentions Vizzly, screenshot baselines, comparisons, Honeydiff, visual review, dynamic regions, or an existing Vizzly workflow. --- # Vizzly -Use Vizzly as visual evidence for user-facing changes. Keep the repository's -existing test workflow in charge of how the UI is exercised. +Use Vizzly as evidence for user-facing changes. Keep the repository's existing +test workflow in charge of how the UI is exercised. -## Start From The Repository +## Prepare -1. Confirm that the repository uses Vizzly by checking its configuration, - dependencies, scripts, or existing `.vizzly` data. -2. Use the repository's established CLI invocation. The examples in this skill - use `vizzly`; substitute its package script, `pnpm exec vizzly`, or - `npx vizzly` when that is how the repository runs local binaries. -3. Do not install, initialize, log in, or change credentials unless the task - explicitly includes setup. Never print or persist a token. +1. Confirm the repository uses Vizzly by checking its config, dependencies, + scripts, or `.vizzly` data. +2. Use the repository's established CLI invocation exactly. Otherwise, replace + `vizzly` in these examples with `pnpm exec vizzly`; use `npx vizzly` when the + repository uses npm. +3. Do not install, initialize, log in, or change credentials unless setup is in + scope. Never print or persist a token. -## Follow The Evidence Loop +## Inspect And Verify -1. Choose the most relevant source: - - Use a supplied cloud build or comparison ID when the task names one. - - Use existing local context when `.vizzly` contains the run under review. - - Generate fresh evidence through the repository's existing visual test - workflow when stored evidence may be stale or absent. -2. Request the bounded, machine-readable build context: +1. Choose the supplied cloud build or comparison when one is named. Otherwise, + use current local evidence or find the relevant cloud build. +2. Request bounded JSON: ```bash vizzly context build current --source local --agent --json vizzly context build --source cloud --agent --json ``` -3. Check the build identity, source, branch, timestamps, baseline selection, - review state, and truncation fields before interpreting the evidence. Treat - omitted fields as unknown. -4. Inspect the current, baseline, and diff images together using whatever - image, browser, URL, or local-file capability is available. If an image is - inaccessible, say so and do not infer its visual contents from metadata. -5. Read viewport, browser, screenshot metadata, review state, and Honeydiff - facts alongside the images. Follow the returned `suggested_commands` for - exact comparison or screenshot drill-downs. Request raw diff regions only - when the compact summary is insufficient. -6. Separate observations from explanations. State what the evidence shows, - then label any proposed cause with appropriate confidence. -7. Make the smallest justified UI or test change, rerun the owning user - workflow, and inspect the resulting evidence again. +3. Confirm the build, source, branch, timestamps, baseline, review state, and + pagination before drawing conclusions. If `has_more` is true, run the + returned next-page command before concluding. Missing fields remain unknown. +4. Follow `suggested_commands` to inspect a comparison. View its baseline, + current, and diff images together. If an image cannot be opened, label the + result metadata-only; do not call it visual verification. +5. Read image dimensions, viewport, browser, diff regions, fingerprint, and + relevant history alongside the images. A prior approval is supporting + evidence, not permission to approve the current comparison. +6. State observations before possible causes. Make the smallest justified + change, rerun the owning workflow, and inspect the new evidence. ## Guardrails -- Treat an approved baseline as the accepted reference for that comparison, - not infallible truth. Check its identity and selection reason. -- Do not invent progress, counts, review state, visual causes, or missing API +- Do not invent progress, ranking, review state, visual causes, or missing API values. -- Do not approve, reject, comment on, publish, or replace visual evidence - unless the user explicitly asks for that mutation. -- Preserve existing thresholds, cluster sizes, signature properties, and - dynamic-region behavior unless the task and evidence justify changing them. -- Prefer deterministic fixtures and existing end-to-end journeys over hiding a - diff with broader tolerances or a new screenshot-only test. -- Report the screenshot or comparison identity, the relevant build or link, - the observed evidence, any access limitations, and the command used. +- Do not approve, reject, comment on, publish, or replace evidence unless the + task explicitly asks for that mutation. +- Preserve thresholds, cluster sizes, signature properties, and dynamic-region + behavior unless repeated evidence justifies a change. +- Prefer deterministic fixtures and existing user journeys over hiding a diff + with broader tolerances or a new screenshot-only test. +- Report the comparison or screenshot identity, observed evidence, access + limitations, and the command or link used. -## Load References Only When Needed +## Load A Reference When Needed -- Read [references/cli-context.md](references/cli-context.md) for local and - cloud inspection commands, TDD lifecycle, and evidence drill-downs. -- Read [references/sdks.md](references/sdks.md) only when adding or changing - screenshot capture code. -- Read [references/dynamic-content.md](references/dynamic-content.md) when a - diff may involve unstable content, hotspots, or confirmed regions. -- Read [references/setup-ci.md](references/setup-ci.md) only when the task asks - to initialize Vizzly, change CI, or troubleshoot configuration. +- [CLI context](references/cli-context.md): local and cloud evidence, build + discovery, drill-downs, images, and TDD lifecycle. +- [SDK capture](references/sdks.md): add or change screenshot capture code. +- [Dynamic content](references/dynamic-content.md): investigate unstable + content, hotspots, or confirmed regions. +- [Setup and CI](references/setup-ci.md): initialize Vizzly or change CI. diff --git a/skills/vizzly/agents/openai.yaml b/skills/vizzly/agents/openai.yaml deleted file mode 100644 index 5b712e8f..00000000 --- a/skills/vizzly/agents/openai.yaml +++ /dev/null @@ -1,4 +0,0 @@ -interface: - display_name: "Vizzly" - short_description: "Inspect and debug visual changes with Vizzly" - default_prompt: "Use $vizzly to inspect the visual evidence for this UI change and explain what changed." diff --git a/skills/vizzly/references/cli-context.md b/skills/vizzly/references/cli-context.md index 9ac7259f..eb4c342b 100644 --- a/skills/vizzly/references/cli-context.md +++ b/skills/vizzly/references/cli-context.md @@ -1,16 +1,25 @@ -# CLI And Context +# CLI Context -Use the repository's existing CLI invocation. The examples use `vizzly` for -brevity; substitute the repository's package script or package-manager command -when needed. +Use the repository's established CLI invocation and existing authentication. +If cloud authentication is unavailable, report the blocker. Do not start an +interactive login unless setup is in scope. -Use existing authentication and project configuration. If cloud authentication -is missing, report the blocker. Do not start an interactive login or change -credentials unless the task includes setup. +## Choose The Evidence -## Inspect Existing Local Evidence +Use an ID supplied by the task. If no cloud build is supplied, list recent +builds and select the one matching the branch, commit, or pull request: -Request structured local evidence with both `--agent` and `--json`: +```bash +vizzly builds --branch --limit 5 --json +vizzly status --json +vizzly context build --source cloud --agent --json +``` + +Use status for lifecycle facts and build context for visual evidence. Do not +assume the first returned comparison is the most important; preserve API order +and inspect the records relevant to the task. + +For saved local evidence: ```bash vizzly context build current --source local --agent --json @@ -18,83 +27,71 @@ vizzly context screenshot "" --source local --json vizzly context review-queue --source local --json ``` -Pin hand-written local drill-downs with `--source local`. Without it, automatic -source resolution may fall back to cloud data when a local item is unavailable. - -Local context reads persisted `.vizzly` artifacts. Confirm that their build, -branch, timestamp, and baseline match the task before treating them as current. +Confirm the stored build, branch, timestamp, and baseline are current enough +for the task. -## Generate Fresh Local Evidence +## Generate Fresh Evidence -For a one-off run, let Vizzly own the complete local session: +For one run, let Vizzly own the local session: ```bash vizzly tdd run "" --no-open -vizzly context build current --source local --agent --json ``` -For repeated test runs, start the detached daemon once: +For repeated runs, start the detached daemon once: ```bash vizzly tdd start --json vizzly tdd status --json -vizzly context build current --source local --agent --json vizzly tdd stop --json ``` -Treat `tdd run` and `tdd start` as alternatives. Do not shell-background -`tdd start`; it already launches a detached daemon. Stop only a server started -for the current task, and reuse the printed port for status or stop commands -when Vizzly selects a non-default port. +`tdd run` and `tdd start` are alternatives. Stop only a daemon started for the +current task. -## Inspect Cloud Evidence - -When the task already has a build ID: +When a cloud build is in scope: ```bash -vizzly status --json +vizzly run "" --wait --json vizzly context build --source cloud --agent --json ``` -Use status for server-owned lifecycle, processing, comparison, and review -facts. Use build context for visual debugging. +## Inspect A Comparison -When creating a cloud build is in scope, wrap the repository's existing test -command: +Follow the build response's `suggested_commands`. The direct form is: ```bash -vizzly run "" --wait --json -vizzly context build --source cloud --agent --json +vizzly context comparison --source --agent --json +vizzly context comparison --source --agent --include diffs --json ``` -## Read And Drill Into Evidence +Open all three images together. Prefer `original_url` and fall back to `url`: -For each evidence record: +- Current: `comparison.screenshot.original_url` or + `comparison.screenshot.url` +- Baseline: `comparison.baseline.original_url` or `comparison.baseline.url` +- Diff: `comparison.analysis.diff_image_url` -1. Inspect the current, baseline, and diff images with an available harness - capability. State the limitation if an image cannot be accessed. -2. Read render metadata, review state, and compact Honeydiff diagnostics next - to the images. -3. Run the returned `suggested_commands` instead of reconstructing identifiers - or URLs. -4. Add `--include diffs` only when raw Honeydiff region geometry is needed. - Request comments only when human review context matters. +Then compare the visible change with diff regions, fingerprint, and the +separate `similar_by_fingerprint` and `recent_by_name` history streams. +Previous review decisions help explain recurring evidence but do not decide the +current review. -Useful manual drill-downs are: +Useful supporting commands: ```bash -vizzly context comparison --source --agent --json vizzly context screenshot "" --source --json vizzly context similar --source cloud --json vizzly context review-queue --source --json ``` -Use the source from the evidence you are inspecting in place of -``. `context similar` is cloud-only. Keep missing values -unknown, and do not turn metadata into a visual conclusion when the underlying -images are unavailable. +Use `--include diffs` only when compact diagnostics are insufficient. Request +comments only when human review context matters. + +## Continue Without Guessing -When a build has more than 10 actionable records, run its returned next-page -command. The command uses `--offset` to preserve API order without pulling the -full build context into the agent handoff. +Run returned `suggested_commands` rather than reconstructing IDs, sources, or +pagination. When more evidence exists, the next-page command carries the API's +opaque `--cursor`; do not edit or interpret it. Keep follow-up commands pinned +to the source that produced the evidence. diff --git a/skills/vizzly/references/dynamic-content.md b/skills/vizzly/references/dynamic-content.md index b245b7ec..1ce3230d 100644 --- a/skills/vizzly/references/dynamic-content.md +++ b/skills/vizzly/references/dynamic-content.md @@ -1,39 +1,28 @@ # Dynamic Content -Treat dates, timers, randomized data, API-backed content, generated images, and -responsive text as possible causes of change, not automatic explanations. +Dates, timers, random data, API content, generated images, and responsive text +can cause a diff. They are possible explanations, not conclusions. ## Diagnose Before Tuning -1. Inspect screenshot history and the actual image evidence: +1. Inspect the actual images and screenshot history: ```bash vizzly context screenshot "" --source --json ``` -2. Replace `` with the source of the evidence being diagnosed. - Record the observed region, recurrence, render metadata, and whether the - same change appears across builds or variants. -3. Check whether deterministic fixtures can remove irrelevant variation. -4. Treat hotspots and confirmed regions as server- or user-owned evidence. - They can affect comparison results, but this CLI exposes them for inspection - rather than authoring. -5. Preserve existing per-screenshot `threshold` and `minClusterSize` values - unless the task and repeated evidence justify a change. Do not invent new - tolerance values from one diff. - -## Avoid - -- Raising a global threshold for one unstable area. -- Masking a whole page when only a small region changes. -- Assuming a recurring change is harmless without inspecting it. -- Ignoring structural movement because a region contains dynamic content. -- Claiming a cause when only metadata, rather than images or history, is - available. - -## Report Findings - -State the observed change first. Then distinguish a likely explanation—such as -fixture drift, content disappearance, layout shift, capture timing, or baseline -mismatch—from unresolved alternatives. Include the screenshot identity and the -context command or link used. +2. Record the visible region, recurrence, render metadata, and whether the same + change appears across builds or variants. +3. Prefer deterministic fixtures when they can remove irrelevant variation. +4. Treat hotspots and confirmed regions as server- or user-authored evidence. + This CLI exposes them for inspection, not authoring. +5. Preserve `threshold` and `minClusterSize` unless repeated evidence + justifies a change. + +Avoid broad masks, global threshold changes for one region, and claims based on +metadata alone. A recurring change can still be a bug, especially when content +movement affects nearby layout. + +Report the observed change first. Label fixture drift, layout shift, capture +timing, or baseline mismatch as a possible cause until the evidence establishes +it. diff --git a/skills/vizzly/references/sdks.md b/skills/vizzly/references/sdks.md index 0501e8f2..a01edb20 100644 --- a/skills/vizzly/references/sdks.md +++ b/skills/vizzly/references/sdks.md @@ -1,9 +1,10 @@ -# SDK Capture Patterns +# SDK Capture -Prefer the repository's existing Vizzly integration and user journey. Add a new -capture path only when the task requires it. +Use the repository's existing Vizzly integration and user journey. The +JavaScript example applies only when no framework-specific capture path already +owns the workflow. -## JavaScript Client +## JavaScript ```javascript import { vizzlyScreenshot } from '@vizzly-testing/cli/client'; @@ -13,37 +14,28 @@ await vizzlyScreenshot('settings-profile-edit-mode', screenshot); ``` `vizzlyScreenshot(name, image, options)` accepts PNG bytes or a file path. -Available options include: +Options include `properties`, `threshold`, `minClusterSize`, and +`fullPage`. Preserve existing tuning unless visual evidence justifies changing +it. -- `properties`: metadata attached to the screenshot. -- `threshold`: per-screenshot CIEDE2000 Delta E tolerance. -- `minClusterSize`: minimum changed-pixel cluster size. -- `fullPage`: whether the screenshot represents a full-page capture. +## Keep Identity Stable -Do not add arbitrary tuning values. Preserve the repository's existing values -unless visual evidence justifies changing them. +Use a descriptive name. Identity always includes the name, viewport width, and +browser. Configured `signatureProperties` add custom identity fields; other +properties are metadata. -## Screenshot Identity +Do not assume theme, locale, or state creates a separate baseline unless the +configuration says so. Avoid generic names such as `screenshot1` and names +with slashes. -Use a stable, descriptive name. Vizzly's signature always includes the name, -viewport width, and browser. Only properties named in the project's -`signatureProperties` configuration participate in baseline identity. +## Follow Existing Integrations -Treat other `properties` as metadata. Do not assume values such as theme, -locale, or state create separate baselines unless the configuration says so. - -## Existing Integrations - -- Use the Vizzly Vitest matcher when the project already configures the plugin: +- Keep the Vizzly Vitest matcher when it is already configured: ```javascript await expect(page).toMatchScreenshot('hero-section.png'); ``` -- Prefer existing Storybook, static-site, or Ember capture flows over adding a - parallel Playwright path. -- For Swift/XCTest, inspect the repository's Vizzly Swift package documentation - and existing `VizzlyXCTest` usage before changing capture code. - -Use properties for searchable metadata and configured variants. Avoid generic -names such as `screenshot1` or `test`, and avoid names with slashes. +- Prefer existing Storybook, static-site, Ember, or Swift capture flows over a + parallel test path. +- Inspect the repository's integration setup before changing capture code. diff --git a/skills/vizzly/references/setup-ci.md b/skills/vizzly/references/setup-ci.md index 1818500d..84eee09a 100644 --- a/skills/vizzly/references/setup-ci.md +++ b/skills/vizzly/references/setup-ci.md @@ -1,10 +1,10 @@ # Setup And CI -Use this reference only when the user asks to initialize Vizzly, change CI, or -troubleshoot configuration. These workflows mutate project files or external -build state. +Use this reference only when the task includes initialization, CI changes, or +configuration troubleshooting. These workflows mutate files or external build +state. -## Local Setup +## Initialize Initialize only with authorization: @@ -12,57 +12,33 @@ Initialize only with authorization: vizzly init ``` -After setup, choose one local execution mode: - -```bash -vizzly tdd run "" --no-open -``` - -or: - -```bash -vizzly tdd start --json - -vizzly tdd stop --json -``` - -Do not run `tdd start` before `tdd run`; both own a local TDD server lifecycle. +Use [CLI context](cli-context.md) for local evidence and TDD lifecycle commands. ## Cloud CI -Use a project-scoped token supplied by the user's existing secret-management -system. Never print it, write it into the repository, or replace working -credentials while debugging. - -Wrap the CI job's existing visual test command: +Use a project-scoped token from the repository's existing secret system. Never +print it or write it into the repository. ```bash vizzly run "" --wait --json ``` -## Parallel CI - -Give every shard in one build the same parallel ID, unique to that workflow -attempt. Run all shards without `--wait`, then finalize that shared ID once -after every shard finishes: +For parallel CI, give every shard the same workflow-specific parallel ID. Run +shards without `--wait`, then finalize once after all shards finish: ```bash vizzly run "" --parallel-id "" --json vizzly finalize "" --json ``` -Do not create a different parallel ID per shard, and do not wait for a parallel -build to complete before it has been finalized. Use one finalizer rather than -having every shard race to finalize the build. - ## Troubleshoot -- Run `vizzly doctor` for local configuration checks. -- Run `vizzly tdd status --json` for a local daemon started by the task. -- Run `vizzly status --json` for cloud lifecycle facts. -- Run `vizzly context build --source cloud --agent --json` for visual - evidence. -- If screenshots are absent, verify the existing SDK or integration, the test - path that should capture them, and the active local or cloud session. -- If authentication is absent, report it rather than initiating login unless - setup is explicitly in scope. +- `vizzly doctor`: local configuration +- `vizzly tdd status --json`: a local daemon started by the task +- `vizzly status --json`: cloud lifecycle +- `vizzly context build --source cloud --agent --json`: visual + evidence + +If screenshots are absent, verify the existing integration, the test path that +should capture them, and the active session. If authentication is absent, +report it rather than logging in unless setup is in scope. diff --git a/src/cli.js b/src/cli.js index 4ebc0db0..ebb69a0f 100644 --- a/src/cli.js +++ b/src/cli.js @@ -594,10 +594,13 @@ program .command('init') .description('Initialize Vizzly in your project') .option('--force', 'Overwrite existing configuration') - .option('--agent-skill', 'Install the repo-local Vizzly agent skill') + .option( + '--agent-skill', + 'Install or refresh the repo-local Vizzly agent skill' + ) .option( '--agent-guidance', - 'Add Vizzly guidance to this project AGENTS.md and install the agent skill' + 'Install or refresh the repo-local Vizzly skill and AGENTS.md guidance' ) .option('--skip-agent-skill', 'Skip the Vizzly agent skill prompt') .action(async options => { diff --git a/src/commands/context.js b/src/commands/context.js index 21fbe693..029144bd 100644 --- a/src/commands/context.js +++ b/src/commands/context.js @@ -572,7 +572,7 @@ function buildCompactBuildCommands(context = {}, include = [], cursor = null) { }); } - if (buildTarget && evidence.length > 0) { + if (buildTarget && evidence.length > 0 && !include.includes('diffs')) { let cursorFlag = cursor ? ` --cursor ${quoteCommandArgument(cursor)}` : ''; commands.push({ label: 'Load raw diff diagnostics', @@ -625,6 +625,16 @@ function buildCompactComparisonCommands(context = {}, include = []) { ]; let includeFlag = include.length > 0 ? ` --include ${include.join(',')}` : ''; + if (comparisonId && !include.includes('diffs')) { + commands.push({ + label: 'Load raw diff diagnostics', + command: appendContextSource( + `vizzly --json context comparison ${quoteCommandArgument(comparisonId)} --agent --include diffs`, + context + ), + }); + } + for (let [stream, label] of streams) { let page = context.history?.[stream]?.page; if (!comparisonId || !page?.has_more || !page.next_cursor) { @@ -1143,6 +1153,7 @@ function displayScreenshotContext(output, context) { let confirmedRegionLabels = formatConfirmedRegionLabels( context.confirmed_regions ); + let hotspotAnalysis = context.hotspot_analysis; output.print(` ${colors.bold(context.screenshot.name)}`); output.print( @@ -1154,10 +1165,14 @@ function displayScreenshotContext(output, context) { 'Memory', `${context.history.recent_comparisons.length} recent comparisons · ${context.confirmed_regions.length} confirmed regions` ); - output.labelValue( - 'Hotspots', - `${context.hotspot_analysis.total_builds_analyzed} builds analyzed · ${context.hotspot_analysis.confidence}` - ); + if (hotspotAnalysis) { + output.labelValue( + 'Hotspots', + `${hotspotAnalysis.total_builds_analyzed ?? 'unknown'} builds analyzed · ${hotspotAnalysis.confidence ?? 'unknown'}` + ); + } else { + output.labelValue('Hotspots', 'unavailable'); + } if (confirmedRegionLabels) { output.labelValue('Known Regions', confirmedRegionLabels); diff --git a/src/commands/init.js b/src/commands/init.js index 772dfd0e..c983c8dd 100644 --- a/src/commands/init.js +++ b/src/commands/init.js @@ -37,6 +37,7 @@ function createInitDeps(deps = {}) { output: deps.output || output, promptAgentSkill: deps.promptAgentSkill || promptAgentSkill, readFile: deps.readFile || fs.readFile, + remove: deps.remove || fs.rm, skillSourcePath: deps.skillSourcePath || getPackagedAgentSkillPath(), writeFile: deps.writeFile || fs.writeFile, }; @@ -283,16 +284,20 @@ let agentGuidanceStart = ''; let agentGuidanceEnd = ''; function createAgentGuidanceContent() { + let skillPath = '.agents/skills/vizzly/SKILL.md'; return `${agentGuidanceStart} ## Visual Testing With Vizzly -Use Vizzly for user-facing UI changes when this repo has Vizzly configured. +For user-facing changes, use the repo-local Vizzly skill at +\`${skillPath}\`. -- Read visual history before changing UI: \`npx vizzly context build current --source local --agent\` -- Inspect named screenshot history when useful: \`npx vizzly context screenshot "" --source local --json\` -- Verify visual changes through the existing user workflow that owns the surface: \`npx vizzly tdd run "" --no-open\` +- Inspect existing visual evidence before and after the change. +- Use the repository's established Vizzly command and owning user workflow. +- Read bounded JSON with \`--agent --json\`, then inspect baseline, current, and + diff images together. -Prefer existing E2E/user journeys over narrow screenshot-only specs. Treat Vizzly diffs as review evidence; do not approve or reject changes unless asked. +Treat Vizzly diffs as review evidence. Do not approve, reject, or replace +evidence unless the task explicitly asks for that mutation. ${agentGuidanceEnd} `; } @@ -323,9 +328,36 @@ async function upsertProjectAgentGuidance({ } let existingContent = await readFile(agentsPath, 'utf8'); - if (existingContent.includes(agentGuidanceStart)) { + let startIndex = existingContent.indexOf(agentGuidanceStart); + let endIndex = existingContent.indexOf(agentGuidanceEnd); + let startCount = existingContent.split(agentGuidanceStart).length - 1; + let endCount = existingContent.split(agentGuidanceEnd).length - 1; + + if ( + startCount !== endCount || + startCount > 1 || + (startCount === 1 && endIndex < startIndex) + ) { return { - status: 'exists', + status: 'malformed', + agentsPath, + }; + } + + if (startIndex >= 0 && endIndex >= startIndex) { + let blockEnd = endIndex + agentGuidanceEnd.length; + let refreshedContent = `${existingContent.slice(0, startIndex)}${guidance.trimEnd()}${existingContent.slice(blockEnd)}`; + + if (refreshedContent === existingContent) { + return { + status: 'exists', + agentsPath, + }; + } + + await writeFile(agentsPath, refreshedContent, 'utf8'); + return { + status: 'refreshed', agentsPath, }; } @@ -340,9 +372,11 @@ async function upsertProjectAgentGuidance({ export async function installProjectAgentSkill({ cwd, sourcePath, + refresh = false, access = fs.access, copy = fs.cp, mkdir = fs.mkdir, + remove = fs.rm, }) { let targetPath = getProjectAgentSkillPath(cwd); @@ -354,7 +388,8 @@ export async function installProjectAgentSkill({ }; } - if (await fileExists(targetPath, access)) { + let targetExists = await fileExists(targetPath, access); + if (targetExists && !refresh) { return { status: 'exists', sourcePath, @@ -365,12 +400,18 @@ export async function installProjectAgentSkill({ await mkdir(path.dirname(targetPath), { recursive: true }); await copy(sourcePath, targetPath, { recursive: true, - errorOnExist: true, - force: false, + errorOnExist: !targetExists, + force: targetExists, }); + if (targetExists) { + await remove(path.join(targetPath, 'agents', 'openai.yaml'), { + force: true, + }); + } + return { - status: 'installed', + status: targetExists ? 'refreshed' : 'installed', sourcePath, targetPath, }; @@ -387,6 +428,12 @@ function writeAgentSkillOutput(output, result) { return; } + if (result.status === 'refreshed') { + output.complete('Refreshed Vizzly agent skill'); + output.hint(`Updated ${result.targetPath}`); + return; + } + if (result.status === 'exists') { output.hint('Vizzly agent skill already exists in this repo'); return; @@ -414,8 +461,20 @@ function writeAgentGuidanceOutput(output, result) { return; } + if (result.status === 'refreshed') { + output.complete('Refreshed Vizzly guidance in AGENTS.md'); + output.hint(`Updated ${result.agentsPath}`); + return; + } + if (result.status === 'exists') { output.hint('Vizzly guidance already exists in AGENTS.md'); + return; + } + + if (result.status === 'malformed') { + output.warn('Vizzly guidance markers in AGENTS.md are incomplete'); + output.hint(`Repair the managed block in ${result.agentsPath}`); } } @@ -480,6 +539,8 @@ export async function init(options = {}, deps = {}) { access: resolvedDeps.access, copy: resolvedDeps.copy, mkdir: resolvedDeps.mkdir, + refresh: options.agentGuidance || options.agentSkill, + remove: resolvedDeps.remove, }); } diff --git a/tests/commands/context-cli.test.js b/tests/commands/context-cli.test.js index 257bb0f5..8383816f 100644 --- a/tests/commands/context-cli.test.js +++ b/tests/commands/context-cli.test.js @@ -438,7 +438,14 @@ async function withBuildContextApi(callback) { JSON.stringify({ resource: 'screenshot_context', review_flow: 'legacy', + scope: { + organization: { slug: 'acme' }, + project: { slug: 'web', name: 'Web' }, + }, screenshot: { name: 'Screenshot 1' }, + history: { recent_comparisons: [] }, + confirmed_regions: [], + hotspot_analysis: null, }) ); return; @@ -681,6 +688,11 @@ describe('context CLI integration', () => { assert.deepStrictEqual(diffPayload.evidence.items[0].diff.regions, [ { x: 10, y: 20, width: 30, height: 40 }, ]); + assert.ok( + !diffPayload.suggested_commands.some( + command => command.label === 'Load raw diff diagnostics' + ) + ); let full = await runCLI( ['--json', 'context', 'build', 'build-123', '--agent', '--full'], @@ -812,6 +824,13 @@ describe('context CLI integration', () => { let payload = JSON.parse(result.stdout).data; assert.strictEqual(payload.resource, 'comparison_agent_context'); assert.strictEqual(payload.comparison.id, 'comparison-1'); + assert.ok( + payload.suggested_commands.some(command => + command.command.includes( + 'context comparison comparison-1 --agent --include diffs --source cloud' + ) + ) + ); assert.deepStrictEqual(payload.comparison.analysis.regions, [ { x: 10, y: 20, width: 30, height: 40 }, ]); @@ -866,6 +885,11 @@ describe('context CLI integration', () => { ) ) ); + assert.ok( + !withDiffsPayload.suggested_commands.some( + command => command.label === 'Load raw diff diagnostics' + ) + ); let full = await runCLI( [ @@ -924,6 +948,32 @@ describe('context CLI integration', () => { }); }); + it('renders human screenshot context when hotspot analysis is unavailable', async () => { + await withBuildContextApi(async ({ apiUrl }) => { + let cwd = mkdtempSync(join(tmpdir(), 'vizzly-context-human-')); + let result = await runCLI( + [ + '--no-color', + 'context', + 'screenshot', + 'Screenshot 1', + '--source', + 'cloud', + ], + { + cwd, + env: { + VIZZLY_API_URL: apiUrl, + VIZZLY_TOKEN: 'vzt_test_token', + }, + } + ); + + assert.strictEqual(result.code, 0, result.stderr); + assert.match(result.stdout, /Hotspots:\s+unavailable/); + }); + }); + it('does not let a cloud session relabel stale local evidence', async () => { await withBuildContextApi(async ({ apiUrl, requests }) => { let cwd = mkdtempSync(join(tmpdir(), 'vizzly-context-mixed-source-')); diff --git a/tests/commands/init.test.js b/tests/commands/init.test.js index 49f8888b..a9894dd9 100644 --- a/tests/commands/init.test.js +++ b/tests/commands/init.test.js @@ -1,6 +1,6 @@ import assert from 'node:assert/strict'; import { execFile } from 'node:child_process'; -import { mkdir, mkdtemp, readFile, writeFile } from 'node:fs/promises'; +import { access, mkdir, mkdtemp, readFile, writeFile } from 'node:fs/promises'; import { tmpdir } from 'node:os'; import path from 'node:path'; import { describe, it } from 'node:test'; @@ -382,14 +382,8 @@ describe('commands/init', () => { assert.strictEqual(result.status, 'created'); assert.match(installedSkill, /name: vizzly/); assert.match(agentsContent, /Visual Testing With Vizzly/); - assert.match( - agentsContent, - /npx vizzly context build current --source local --agent/ - ); - assert.match( - agentsContent, - /npx vizzly tdd run "" --no-open/ - ); + assert.match(agentsContent, /.agents\/skills\/vizzly\/SKILL.md/); + assert.match(agentsContent, /--agent --json/); assert.ok( output.calls.some( call => @@ -544,5 +538,114 @@ describe('commands/init', () => { assert.strictEqual(result.status, 'exists'); assert.strictEqual(existingSkill, 'local skill\n'); }); + + it('refreshes managed agent guidance when explicitly requested', async () => { + let cwd = await createTempProject(); + let skillSourcePath = path.join(await createTempProject(), 'vizzly'); + let targetPath = getProjectAgentSkillPath(cwd); + let agentsPath = getProjectAgentsPath(cwd); + let output = createMockOutput(); + + await mkdir(skillSourcePath, { recursive: true }); + await mkdir(path.join(targetPath, 'agents'), { recursive: true }); + await writeFile( + path.join(skillSourcePath, 'SKILL.md'), + '---\nname: vizzly\ndescription: Current guidance.\n---\n' + ); + await writeFile(path.join(targetPath, 'SKILL.md'), 'stale skill\n'); + await writeFile(path.join(targetPath, 'project-notes.md'), 'keep me\n'); + await writeFile( + path.join(targetPath, 'agents', 'openai.yaml'), + 'stale vendor metadata\n' + ); + await writeFile( + agentsPath, + '# Repo Guidance\n\n\nstale guidance\n\n' + ); + + await init( + { agentGuidance: true, plugins: [] }, + { + cwd: () => cwd, + output, + skillSourcePath, + } + ); + + let installedSkill = await readFile( + path.join(targetPath, 'SKILL.md'), + 'utf8' + ); + let projectNotes = await readFile( + path.join(targetPath, 'project-notes.md'), + 'utf8' + ); + let agentsContent = await readFile(agentsPath, 'utf8'); + + assert.match(installedSkill, /description: Current guidance/); + assert.strictEqual(projectNotes, 'keep me\n'); + assert.match(agentsContent, /# Repo Guidance/); + assert.match(agentsContent, /Inspect existing visual evidence/); + assert.doesNotMatch(agentsContent, /stale guidance/); + await assert.rejects( + access(path.join(targetPath, 'agents', 'openai.yaml')), + error => error.code === 'ENOENT' + ); + assert.ok( + output.calls.some( + call => + call.method === 'complete' && + call.args[0] === 'Refreshed Vizzly agent skill' + ) + ); + assert.ok( + output.calls.some( + call => + call.method === 'complete' && + call.args[0] === 'Refreshed Vizzly guidance in AGENTS.md' + ) + ); + }); + + it('leaves incomplete AGENTS.md markers for the user to repair', async () => { + let cwd = await createTempProject(); + let skillSourcePath = path.join(await createTempProject(), 'vizzly'); + let agentsPath = getProjectAgentsPath(cwd); + let output = createMockOutput(); + + await mkdir(skillSourcePath, { recursive: true }); + await writeFile( + path.join(skillSourcePath, 'SKILL.md'), + '---\nname: vizzly\ndescription: Current guidance.\n---\n' + ); + await writeFile( + agentsPath, + '# Repo Guidance\n\n\ncustom text\n' + ); + + await init( + { agentGuidance: true, plugins: [] }, + { + cwd: () => cwd, + output, + skillSourcePath, + } + ); + + let agentsContent = await readFile(agentsPath, 'utf8'); + assert.match(agentsContent, /custom text/); + assert.strictEqual( + agentsContent.split('vizzly-agent-guidance').length - 1, + 1 + ); + assert.ok( + output.calls.some( + call => + call.method === 'warn' && + call.args[0] === + 'Vizzly guidance markers in AGENTS.md are incomplete' + ) + ); + }); }); });