Conversation
Adds a "Speak text" PostGrabAction that reads OCR output aloud using Windows.Media.SpeechSynthesis. Speaks the final transformed text after all other actions run in FullscreenGrab; in GrabFrame, speaks only when the captured text changes to avoid repeating on every OCR tick. - ITtsEngine interface for future engine swappability - WindowsSpeechEngine wraps WinRT SpeechSynthesizer + MediaPlayer - TtsService queues utterances via Channel<string> so new text waits rather than interrupting in-progress speech - TtsSpeakWordLimit setting (default 100) truncates long captures; configurable in General Settings - PostGrabActionManager: new SpeakText_Click action at order 6.6 - GrabFrame: speaks on text change when action is checked - Tests: count updated to 6, "Speak text" assertion, fire-and-forget test Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Adds SpeakInsteadOfToast setting: when enabled, Text Grab speaks the captured text aloud rather than chiming a notification toast. Defaults to enabled. Includes: - Stop-speaking button in GrabFrame to cancel playback mid-sentence - TTS drain-on-shutdown fix so the queue empties cleanly on exit - Stop() method on TtsService to flush the queue and cancel speech
Adds a dedicated Voice Output page in Settings containing: - Voice picker populated from SpeechSynthesizer.AllVoices - Word limit setting - Speak-instead-of-notification toggle (moved from General Settings) - Preview button to hear the selected voice WindowsSpeechEngine now applies the saved TtsVoiceName before synthesising. TtsVoiceName setting added (empty = system default).
Introduce a new user-scoped boolean setting, WindowsAiDescriptionEnabled, to control Windows AI description features. The setting defaults to False and is added to App.config, Settings.settings, and Settings.Designer.cs.
Added WindowsAiDescriptionLang implementing ILanguage for the new "Windows AI Description" language kind. Introduced GeneratedOcrLinesWords, GeneratedOcrLine, and GeneratedOcrWord classes to generate OCR line/word models from text and bounding boxes. Updated HistoryInfo to support the new language kind.
Introduce WindowsAiDescriptionLang for image captioning via Windows AI. Update LanguageKind, LanguageService, and CaptureLanguageUtilities to support selection and caching. Route OCR/image description requests through new async methods in OcrUtilities and WindowsAiUtilities. Add device capability checks, content moderation, and async polling for image description. Exclude WindowsAiDescriptionLang from table output. Improve resource cleanup and error handling.
Improve test coverage for WindowsAiDescriptionLang integration: - Save/restore WindowsAiDescriptionEnabled for isolation - Test inclusion/exclusion in CaptureLanguageUtilities - Assert correct tag/kind handling in LanguageService - Verify fallback and HistoryInfo behavior with new language
Introduce WindowsAiDescriptionLang support in LanguagePicker and filter it from the language list. Add a toggle in LanguageSettings to enable/disable Windows AI Description as a language option, with explanatory text. Refactor settings initialization to use a unified loadingLanguageSettings flag and update event handlers accordingly. Implement methods to load and persist the new setting, and ensure language caches are invalidated when toggled.
Add the five Microsoft.Recognizers.Text.* packages (with BUILT-WITH
attribution) and a fixed BuiltInRecognizer catalog -- numbers, ordinals,
percentages, ages, currencies, dimensions, temperatures, dates/times, phone
numbers, emails, URLs, IPs, GUIDs, and booleans -- plus RecognizerExecutor for
matching and applying them (resolved value or matched text, first/last/all/nth).
GrabTemplate gains {r:Name:mode[:value|text][:separator]} placeholders
(TemplateRecognizerMatch), resolved by GrabTemplateExecutor alongside the
existing {p:} regex patterns.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Introduce PatternItem -- a single user-facing "pattern" that wraps either a saved
StoredRegex or a built-in recognizer -- and PatternExecutor, which dispatches
HasMatch/GetMatches/Apply to the regex or recognizer engine. This lets every UI
surface treat the two kinds uniformly. PatternItem also builds the grouped
("Saved Patterns" / "Smart Patterns") choice lists used by the combo pickers.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expose saved regexes and built-in recognizers together everywhere, split into
"Saved Patterns" and "Smart Patterns" subsections:
- Template inline {...} picker: one Patterns group (InlinePickerItem.Kind drives
whether selection emits {p:} or {r:}); {p:}/{r:} syntax unchanged.
- Edit Text Window: "Apply Pattern" / "Apply Pattern Per Line" list both kinds and
transform in place via PatternExecutor.
- Grab Frame: "Search by Pattern" submenu (saved regexes searchable for the first
time); Quick Simple Lookup and Find & Replace use a unified Patterns combo
(saved regex loads into the find box so replace still works; recognizers are
find-only).
Combo pickers use PatternChoice data items with DisplayMemberPath and
non-selectable subsection headers, fixing the selection-box width/height clipping
that came from using ComboBoxItem objects as data. User-visible "Recognizer"
wording is now "Pattern".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
A reusable search input used across the search surfaces: a text box with a placeholder and clear button, a removable smart-pattern chip, a WPF-UI-style accent focus underline on the whole control, a regex toggle rendered as a split-button whose chevron opens a scrollable pattern flyout (saved regexes + smart recognizers), and an optional exact-match toggle. Adds the ToggleSplitSegment and SplitSegmentButton styles the split-button uses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… Frame Replaces each window's hand-built search box, regex control, and pattern picker with the shared SearchBar and routes searching through its SearchChanged event: - Quick Simple Lookup: swaps its text box, regex toggle, and pattern combo for the control; regex validity now flows through SetRegexValidity. - Find & Replace: the find box becomes a SearchBar; the "Match Exactly" switch moves into the box as a toggle (wired via ExactMatchChanged), and a picked recognizer can be narrowed by typed text. Adds SetFindText for external callers. - Grab Frame: the regex, exact-match, and "search by pattern" options that were hidden in the text box context menu become visible controls, and the status bar grows to fit the taller control. Retargets RegexManager and EditTextWindow to SetFindText and drops the now-unused PatternItem combo helpers. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Registers a Ctrl+Shift+F key gesture in the Edit Text Window that opens Find & Replace, and shows the shortcut on the menu item. Ctrl+F stays mapped to Fullscreen Grab. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Expanded GetDefaultPatterns summary to clarify exclusion of patterns already handled by built-in recognizers (emails, phones, URLs, IPs, GUIDs, dates, times, currency, numbers). Removed these patterns from the defaults, retaining only credit card, hex color, SSN, and US zip code regexes.
Recognizers: unify saved regexes and smart recognizers as one "Patterns" concept
Introduce a new user-scoped boolean setting, WindowsAiDescriptionEnabled, to control Windows AI description features. The setting defaults to False and is added to App.config, Settings.settings, and Settings.Designer.cs.
Added WindowsAiDescriptionLang implementing ILanguage for the new "Windows AI Description" language kind. Introduced GeneratedOcrLinesWords, GeneratedOcrLine, and GeneratedOcrWord classes to generate OCR line/word models from text and bounding boxes. Updated HistoryInfo to support the new language kind.
Introduce WindowsAiDescriptionLang for image captioning via Windows AI. Update LanguageKind, LanguageService, and CaptureLanguageUtilities to support selection and caching. Route OCR/image description requests through new async methods in OcrUtilities and WindowsAiUtilities. Add device capability checks, content moderation, and async polling for image description. Exclude WindowsAiDescriptionLang from table output. Improve resource cleanup and error handling.
Improve test coverage for WindowsAiDescriptionLang integration: - Save/restore WindowsAiDescriptionEnabled for isolation - Test inclusion/exclusion in CaptureLanguageUtilities - Assert correct tag/kind handling in LanguageService - Verify fallback and HistoryInfo behavior with new language
Introduce WindowsAiDescriptionLang support in LanguagePicker and filter it from the language list. Add a toggle in LanguageSettings to enable/disable Windows AI Description as a language option, with explanatory text. Refactor settings initialization to use a unified loadingLanguageSettings flag and update event handlers accordingly. Implement methods to load and persist the new setting, and ensure language caches are invalidated when toggled.
…-Grab into WinAI-description
Add overloads to GetTextDescriptionWithWinAI for System.Drawing.Bitmap input and CancellationToken support. Use AsTask to enable cancellation of DescribeAsync, remove manual polling, and improve exception handling to propagate OperationCanceledException and log other errors. Temp files are now cleaned up after use.
Introduce a new user flow for Windows AI image description in fullscreen grab mode. The overlay now freezes the selected region as a background, shows a spinner, and after 2 seconds offers Cancel/Re-grab options. If the AI description fails or is empty, a "Send to Grab Frame" option is provided. Refactor PreviousGrabWindow and FullscreenGrab to support this asynchronous, user-driven workflow and improve code organization.
Replace the flat black strip with a centered rounded panel using the app background, teal border, and drop shadow like the fullscreen grab toolbar. Buttons now use the TealColor style with icon + label content, and the bar slides up from the bottom edge with a fade on first reveal. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
- Use C# collection expressions for English word array in WindowsAiUtilities - Return empty string (not error) if WinAI unavailable for image description - Ignore single clicks for AI Description in fullscreen grab, reset selection instead - Cancel grab if overlay closes without user choice to avoid orphaned tasks - Default TextContent to empty string if OCR result is null
Adds ShowSuccessAsync to PreviousGrabWindow for awaiting overlay close, ensuring text insertion occurs after focus is released. Refactors insertion logic in FullscreenGrab to prevent focus issues, threads cancellation tokens in WindowsAiUtilities, and cleans up redundant code.
Adds a ToggleButton to PreviousGrabWindow for sending OCR results to an Edit Text Window, with visibility and checked state managed during OCR. Introduces a SendToEditTextToggled event and syncs the toggle state across all fullscreen grab windows. Updates running/failed state logic to show or hide the toggle as appropriate.
WinAI Descriptions
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Upgraded CliWrap to 3.10.3, Magick.NET-Q16-AnyCPU to 14.16.0, Magick.NET.SystemDrawing to 8.0.25, and Magick.NET.SystemWindowsMedia to 8.0.25 for latest bug fixes and improvements.
Add AutomationProperties.AutomationId and Name to dialogs and controls for enhanced screen reader and UI automation support. Update ShortcutControl and WordBorder to set automation properties programmatically; implement custom AutomationPeer for WordBorder to support Value pattern. Replace Path.GetTempPath with AutomationProfile.GetTemporaryDirectory for temp files. Minor UI tweaks for accessibility and event handling. These changes improve assistive tech usability and test automation readiness.
Added AutomationProperties.AutomationId to nearly all interactive controls across all settings XAML pages to improve accessibility and enable automated UI testing. In select cases, also set AutomationProperties.Name for enhanced context. Updated file/folder pickers and temp file paths to use automation-aware or profile-based directories. No changes to UI logic or behavior; changes are strictly for accessibility, automation, and testability.
Added AutomationProperties.AutomationId (and Name where needed) to windows and controls in EditTextWindow, FirstRunWindow, FullscreenGrab, GrabFrame, QuickSimpleLookup, and SettingsWindow XAML. These changes improve accessibility and enable reliable automated UI testing by providing unique identifiers for screen readers and test frameworks. No behavioral or logic changes; all updates are declarative in XAML.
Centralize system integration and file paths via AutomationProfile. Gate system integration features (context menu, startup, hotkeys, etc.) on profile permissions. Prefer profile-specific directories for temp and output files. Improve hotkey registration error handling. Enhance modularity and testability for automation scenarios.
Introduce AutomationDiagnostics, AutomationProfile, and AutomationSettingsProvider in Text_Grab.Utilities. Enables event logging, automation-aware settings management, and reproducible configuration for CI and automated scenarios. AutomationProfile parses CLI/env for automation mode, manages directories, and supports settings seeding. Diagnostics and settings provider support robust traceability and override default settings when automation is active.
App now detects active AutomationProfile at startup, enabling automation diagnostics, event recording, and disabling system integration features as needed. SettingsService uses profile-specific paths for settings storage and seeding, ensuring isolated automation runs. Added OverrideAiArchCheck setting. Improved argument parsing, exception handling, and minor refactoring for automation scenarios.
Added AutomationProfileTests for TryCreate scenarios including env var and CLI overrides. Added UiAutomationContractTests to enforce AutomationId presence/uniqueness, WordBorder automation peer, and stable automation selectors in XAML and code. Validates source-level and config contracts to prevent UI automation regressions.
This adds interactive-ui.yml, a GitHub Actions workflow for running interactive UI tests on dedicated Windows desktop runners. The workflow supports manual and scheduled triggers, lane selection (smoke, system-integration, mixed-DPI, packaged, ARM64, Copilot+), and options for build type, recording, xUnit test execution, and dry run. It uses a matrix of specialized runners, ensures exclusive desktop access, runs PowerShell scripts for preflight and lane execution, summarizes UI test results, and uploads artifacts per lane.
Added new TextGrab.AutomationHost WPF project targeting .NET 10.0 and Windows 10 SDK 22621.0. Includes App.xaml/cs, MainWindow.xaml/cs for fixture hosting with selectable surfaces (text, multilingual, OCR, QR/barcode, table, empty, contrast, DPI grid), UI for text/metrics/input, and automation instrumentation. Added FixtureOptions for config parsing, FixtureStateWriter for JSON state output, app.manifest for DPI, and project file with test image content. Enables integration and automation testing with detailed state logging and UIA support.
Added TextGrab.SystemIntegrationHelper (.NET 10.0, WPF, unsafe) for system integration tasks. Program.cs provides CLI for mouse/keyboard simulation, clipboard, drag-and-drop, hotkeys, and window queries using user32.dll interop. Includes argument validation, error handling, and result serialization.
Added UiTests/TextGrab.AutomationHost to Text-Grab.sln for UI automation/testing. Registered project with Debug/Release configs for ARM64, x64, x86. No changes to existing projects.
Added comprehensive PowerShell-based UI automation and capability tests for Text Grab. Introduced helper modules for deterministic app launching, fixture management, and capability assertions. Added test scripts for EditText, GrabFrame, QuickLookup, Settings, Lifecycle, PatternsBulk, DisplayHardware, NotificationsTtsShare, OcrCapability, OsArchitecture, PackagedVm, SystemIntegration, Harness, and CliProtocol. Tests cover deterministic UI automation and system integration, with capability-based gating and environment-aware skips. Ensured test isolation, automation health, and resource cleanup.
Comprehensive UI automation and coverage validation for Text Grab: - Added coverage.json mapping 154 checklist items to tests, coverage, and manual exceptions. - Introduced PowerShell modules/scripts for UI automation, preflight checks, test harness, and result reporting. - Added release sign-off and coverage validation scripts with aggregation, enforcement, and Markdown/JSON reporting. - Supported safe, destructive, and capability-gated test lanes (system, VM, DPI, ARM64, Copilot+). - Updated README.md with prerequisites, runner safety, suite structure, and CI integration. - Added helper modules for parsing, summarizing, and enforcing UI test results and sign-off. - All scripts/modules are Windows-only, require .NET 10.0 and WinApp CLI 0.5.x.
.gitignore now excludes UiTests/artifacts/ to prevent tracking UI test artifacts. README.md adds a "UI automation" section, referencing UiTests/README.md for details on UI automation inventory, release sign-off, runner requirements, and opt-in lanes, improving discoverability of UI test infrastructure documentation.
Attach AutomationSettingsProvider to the generated Settings class via a hand-written partial so classic user.config reads/writes land in the active automation profile's isolated directory, deferring to the base LocalFileSettingsProvider when no profile is active. Implement IApplicationSettingsProvider so Upgrade/Reset/GetPreviousVersion are no-ops under a profile, and add a test seam to override the ambient profile plus unit coverage. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Add a FullscreenGrab suite with a deterministic always-on overlay smoke plus capability-gated real-input tests (toolbar reveal, region drag to clipboard). Add a --move cursor command to SystemIntegrationHelper that double-moves to guarantee a WM_MOUSEMOVE delta so the auto-hiding toolbar reveals itself, wire the suite into the system-integration CI lane and release signoff, and record its coverage entries. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move the lane catalog and selection logic into a select-lanes job that resolves lanes against the event, dispatch input, and repository variables, then emits a dynamic matrix the interactive-ui job consumes. This replaces the per-entry if: filter, which could not reference the matrix context, with a single has_lanes gate. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
OverrideCurrentForTests mutates process-global AutomationProfile state. Running in parallel with other classes that touch Settings.Default could redirect a concurrent Save into this test's temporary profile directory. Join the existing non-parallel "Settings isolation" collection. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Added logic to FullscreenGrab to enforce maximized overlay using ShouldForceMaximize, a DispatcherTimer, and LayoutUpdated handler. Cleaned up event/timer on close. Added unit tests for ShouldForceMaximize.
Automated UI Testing
The FullscreenGrab overlay is a transparent, owned WPF window whose UIA tree is unreachable by HWND, so the deterministic smoke (which waited on the window-root AutomationId and a bare Canvas, neither queryable) always timed out. Rework the suite to launch Fullscreen Grab from the command line and hook the overlay by process id, and exercise the real region drag + OCR against the known fixture text. Fixes uncovered along the way: - Harness: add process-scoped (--app) element targeting for the overlay; unwrap winapp 0.5 get-property's nested properties object (had silently broken every Get-SystemElementCenter/Assert-UiTestProperty); retry clipboard snapshot/restore on CLIPBRD_E_CANT_OPEN contention. - SystemIntegrationHelper: set PerMonitorV2 DPI awareness so synthetic input maps in physical pixels, matching UIA bounds (drags were 2x off at 200% scaling). - Tests: force-stop each dedicated overlay process between tests (a completed grab leaves it lingering, and only a foregrounded overlay shows its auto-hiding toolbar); the toolbar test forces foreground and moves the pointer to reveal the toolbar; readiness now keys on the always-present capture-surface background image. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Classification
New feature and major UI/UX enhancement: unified pattern/recognizer search, voice output, HDR capture correction, and improved settings.
PR Summary
This PR introduces unified "Smart Pattern" (recognizer) and regex search across the app, adds voice output (TTS) features, supports HDR-aware screen capture, and modernizes the settings UI. It also improves Grab Frame and Quick Simple Lookup workflows and adds .tggf file support.