Enhance TV connection with Claude API - #113
Conversation
Co-authored-by: djamesr23 <djamesr23@gmail.com>
|
Cursor Agent can help with this pull request. Just |
📊 PR Size AnalysisSize: Review Checklist
|
github-actions
Bot
added
documentation
🚨 CI/CD Failure DetectedThe Environment Variable Validation workflow failed for this PR. Action RequiredA tracking PR has been automatically created to help resolve this issue: Quick LinksPlease review the analysis and implement the suggested fixes. 🤖 Auto-generated by JARVIS CI/CD Manager |
🚨 CI/CD Failure DetectedThe Database Connection Validation workflow failed for this PR. Action RequiredA tracking PR has been automatically created to help resolve this issue: Quick LinksPlease review the analysis and implement the suggested fixes. 🤖 Auto-generated by JARVIS CI/CD Manager |
🚨 CI/CD Failure DetectedThe Validate Configuration workflow failed for this PR. Action RequiredA tracking PR has been automatically created to help resolve this issue: Quick LinksPlease review the analysis and implement the suggested fixes. 🤖 Auto-generated by JARVIS CI/CD Manager |
🚨 CI/CD Failure DetectedThe 🎨 Advanced Auto-Diagram Generator workflow failed for this PR. Action RequiredA tracking PR has been automatically created to help resolve this issue: Quick LinksPlease review the analysis and implement the suggested fixes. 🤖 Auto-generated by JARVIS CI/CD Manager |
🚨 CI/CD Failure DetectedThe PR Automation & Validation workflow failed for this PR. Action RequiredA tracking PR has been automatically created to help resolve this issue: Quick LinksPlease review the analysis and implement the suggested fixes. 🤖 Auto-generated by JARVIS CI/CD Manager |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| self._uae_clicker = get_uae_clicker( | ||
| vision_analyzer=self.vision_analyzer, | ||
| enable_uae=False, # Disable full UAE to avoid overhead | ||
| enable_verification=True, | ||
| enable_communication=False # We handle voice in this layer | ||
| ) |
Check failure
Code scanning / CodeQL
Wrong name for an argument in a call Error
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
General approach:
The fix consists of removing the unsupported keyword argument (enable_communication=False) from the call to get_uae_clicker on line 134. This will prevent the TypeError that would occur at runtime. If the exclusion of this argument is not functionally correct (i.e., if voice functionality needs to be disabled in this layer), further adjustment may be required elsewhere, but with the info given, we should only remove the problematic argument to match the function's signature precisely.
Detailed fix:
- Edit file
backend/display/jarvis_computer_use_integration.py. - In the block starting on line 130, modify the call to
get_uae_clickerto remove the line:
enable_communication=False # We handle voice in this layer - Leave all other keyword arguments unchanged.
- No new imports, methods, or definitions are needed.
| @@ -130,8 +130,7 @@ | ||
| self._uae_clicker = get_uae_clicker( | ||
| vision_analyzer=self.vision_analyzer, | ||
| enable_uae=False, # Disable full UAE to avoid overhead | ||
| enable_verification=True, | ||
| enable_communication=False # We handle voice in this layer | ||
| enable_verification=True | ||
| ) | ||
| logger.info("[JARVIS-COMPUTER-USE] ✅ UAE clicker initialized") | ||
| except Exception as e: |
| import base64 | ||
| import json | ||
| from typing import Dict, Any, Optional, List, Callable | ||
| from datetime import datetime |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
To fix the problem, simply delete the unused import statement from datetime import datetime from line 27 in backend/display/computer_use_display_connector.py. No changes to other import statements or any additional code sections are required since there is no indication of datetime being used anywhere in the snippet.
| @@ -24,7 +24,6 @@ | ||
| import base64 | ||
| import json | ||
| from typing import Dict, Any, Optional, List, Callable | ||
| from datetime import datetime | ||
| from pathlib import Path | ||
| from io import BytesIO | ||
|
|
| import json | ||
| from typing import Dict, Any, Optional, List, Callable | ||
| from datetime import datetime | ||
| from pathlib import Path |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
To fix an unused import, we should remove the import statement for Path, as it's not actually used in the displayed code. This avoids unnecessary dependencies and improves code readability. Specifically, in the file backend/display/computer_use_display_connector.py, line 28, the statement from pathlib import Path should be deleted. No additional code changes or dependencies are required.
| @@ -25,7 +25,6 @@ | ||
| import json | ||
| from typing import Dict, Any, Optional, List, Callable | ||
| from datetime import datetime | ||
| from pathlib import Path | ||
| from io import BytesIO | ||
|
|
||
| import anthropic |
| import logging | ||
| from typing import Dict, Any, Optional, Callable | ||
| from dataclasses import dataclass | ||
| from datetime import datetime |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
The best solution is to remove the unused import statement:
Delete the line from datetime import datetime (line 22) from the file backend/display/hybrid_display_connector.py.
This change is limited to a single line and will not affect any existing functionality since datetime is not referenced anywhere in this file. No additional methods, imports, or definitions are required.
| @@ -19,7 +19,6 @@ | ||
| import logging | ||
| from typing import Dict, Any, Optional, Callable | ||
| from dataclasses import dataclass | ||
| from datetime import datetime | ||
|
|
||
| logger = logging.getLogger(__name__) | ||
|
|
| if hasattr(self.uae_clicker, 'coordinate_cache'): | ||
| return target in self.uae_clicker.coordinate_cache | ||
| return False | ||
| except: |
Check notice
Code scanning / CodeQL
Except block handles 'BaseException' Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
The best way to fix this problem is to replace the bare except: on line 276 with an explicit except Exception: clause. This ensures that only standard runtime errors (all subclasses of Exception) are caught. This change allows KeyboardInterrupt and SystemExit to propagate as they should, while still handling regular error conditions gracefully. No extra imports or definitions are needed, as Exception is built into Python.
Only the code in backend/display/hybrid_display_connector.py, specifically the _uae_has_cached method (lines 270–278), needs editing. Replace except: with except Exception:.
| @@ -273,7 +273,7 @@ | ||
| if hasattr(self.uae_clicker, 'coordinate_cache'): | ||
| return target in self.uae_clicker.coordinate_cache | ||
| return False | ||
| except: | ||
| except Exception: | ||
| return False | ||
|
|
||
| async def _connect_with_uae( |
| import asyncio | ||
| import logging | ||
| from typing import Dict, Any, Optional, Callable | ||
| from pathlib import Path |
Check notice
Code scanning / CodeQL
Unused import Note
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 9 months ago
To fix the problem, we simply need to remove the unused import statement from pathlib import Path in file backend/display/jarvis_computer_use_integration.py on line 22. This is the most direct and safest fix, as it does not affect any other code. There are no references to Path elsewhere in the provided snippet, so removing this line does not change the functionality.
| @@ -19,7 +19,6 @@ | ||
| import asyncio | ||
| import logging | ||
| from typing import Dict, Any, Optional, Callable | ||
| from pathlib import Path | ||
|
|
||
| import pyautogui | ||
|
|
🤖 CI/CD Pipeline ResultsStatus: success Pipeline Stages
|
3 participants
Summary
This PR integrates the Claude Computer Use API into JARVIS to replace hardcoded, coordinate-based display connection workflows with a dynamic, vision-based approach. This significantly enhances robustness, adaptability to UI changes, and provides real-time voice transparency during execution.
Changes Made
backend/display/computer_use_display_connector.py: New module for vision-based display connections using the Claude Computer Use API.backend/display/hybrid_display_connector.py: New module implementing a hybrid strategy to intelligently choose between the existing UAE (fast, local) and the new Computer Use (robust, AI-driven) connectors.backend/display/jarvis_computer_use_integration.py: New integration layer that bridges JARVIS's voice system with the hybrid connector, providing real-time voice feedback and managing configuration.test_computer_use_integration.py: New comprehensive test script to demonstrate and verify the integration.COMPUTER_USE_README.md,COMPUTER_USE_QUICK_START.md,COMPUTER_USE_INTEGRATION.md, andCOMPUTER_USE_IMPLEMENTATION_SUMMARY.mdfor complete guidance.Type of Change
Test Plan
Testing Steps:
anthropicdependency:pip install anthropic>=0.8.0export ANTHROPIC_API_KEY="your-anthropic-api-key"python test_computer_use_integration.py "Living Room TV"python test_computer_use_integration.py "Living Room TV" --force-computer-useRelated Issues
Closes #
Relates to #
Screenshots (if applicable)
N/A
Deployment Notes
ANTHROPIC_API_KEYanthropic>=0.8.0Checklist
Additional Context
This integration addresses the "coordinate brittleness" problem by allowing JARVIS to visually understand and interact with the macOS UI. The hybrid approach ensures that common, stable connections remain fast and free via the existing UAE system, while complex or new scenarios leverage the robustness and intelligence of the Claude Computer Use API. JARVIS's voice transparency provides a clear, step-by-step narrative of the connection process.
Reviewer Guidelines:
Summary by cubic
Integrates the Claude Computer Use API to make TV/display connections vision-driven and resilient, replacing brittle coordinate clicks. Adds a hybrid connector that uses the fast UAE path first and falls back to Computer Use with real-time voice updates.
New Features
Migration
Written for commit 2e88d07. Summary will update automatically on new commits.