Feat/phase 9 monitoring#10
Conversation
Add RedactionFilter that scrubs API keys, JWT tokens, passwords, and other secrets from log messages before they are written. Patterns covered: - API keys (X-API-Key headers, api_key params) - JWT tokens (Bearer tokens, access_token values) - Passwords (password=, POSTGRES_PASSWORD) - Secrets (jwt_secret, secret_key) - Groq API keys (gsk_ prefix) - Database connection strings (postgresql://, redis://) The filter is installed on the root logger at app startup, after setup_logging(), so all log records pass through redaction before output. Filter is thread-safe and operates on record.msg and args.
Add cache hit/miss/eviction/size metrics as Prometheus gauges to the /metrics endpoint. Metrics are sourced from TTLCache.get_metrics() and exposed as: - hector_cache_size (current items in cache) - hector_cache_hits (total cache hits) - hector_cache_misses (total cache misses) - hector_cache_evictions (total evictions) - hector_cache_hit_rate_percent (hit rate percentage) These complement the existing cache metrics already exposed via the /status JSON endpoint.
Add Prometheus and Grafana as Docker services in docker-compose.prod.yml: Prometheus: - Scrapes /metrics endpoint from hector-api every 15s - Loads alerting rules from prometheus-alerts.yml - 30-day data retention - Read-only filesystem with persistent data volume - Resource limits: 512MB memory, 0.5 CPU Grafana: - Auto-provisions hector-main dashboard from grafana-dashboard.json - Admin credentials via GRAFANA_ADMIN_USER/GRAFANA_ADMIN_PASSWORD env vars - Connected to Prometheus as data source - Read-only filesystem with persistent data volume - Resource limits: 256MB memory, 0.5 CPU New prometheus.yml config file added to monitoring/ directory. New persistent volumes: prometheus_data, grafana_data.
Fix _redact_dict_value to be called from the dict args branch instead of _redact_value. The dict branch was only applying pattern-based redaction to values, which fails for standalone secrets (e.g., dict values without key= prefix). Add _redact_dict_value method that checks dict keys against a set of secret-suggesting key names (api_key, password, secret, authorization, token, etc.) and redacts values when the key matches. Normalizes key names for comparison.
Add 13 tests for api.log_redaction.RedactionFilter: TestRedactionFilter: - API key values redacted from log messages - X-API-Key header values redacted - Bearer JWT tokens redacted - Password values redacted - JWT secret values redacted - Groq API keys (gsk_ prefix) redacted - PostgreSQL connection strings redacted - Clean messages pass through unchanged - Tuple args with secrets are redacted - Dict args with secret keys are redacted TestInstallRedactionFilter: - Filter installed on root logger with correct name - Filter always returns True (passthrough) Tests use isolated loggers with null handlers to avoid polluting test output.
Update test_enhanced_ingestor.py to match the latest enhanced_ingestor.py changes pulled from main: - acts_found is now a list (not set), updated assertions accordingly - chunk_text now uses boundary-aware splitting with section detection instead of simple sliding window; tests verify all words are covered and chunks are non-empty rather than exact step counts - validate_pdf now exists; tests cover nonexistent/empty/too-small/ invalid-header/valid-PDF cases (invalid-header test uses >100 bytes to pass size check before header check) 847 tests passing.
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 34323629 | Triggered | Generic Password | 49720c9 | tests/test_log_redaction.py | View secret |
| 34010484 | Triggered | Generic Password | 15534c3 | docker-compose.prod.yml | View secret |
| 34323628 | Triggered | Generic Password | 49720c9 | tests/test_log_redaction.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
|
Warning Review limit reached
More reviews will be available in 6 minutes and 57 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (6)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
1 participant
No description provided.