Skip to content

feat(phase1): Karve controller skeleton with Prometheus integration - #8

Merged
diranged merged 15 commits into
mainfrom
matt/phase_1
Dec 6, 2025
Merged

feat(phase1): Karve controller skeleton with Prometheus integration#8
diranged merged 15 commits into
mainfrom
matt/phase_1

Conversation

@diranged

@diranged diranged commented Dec 5, 2025

Copy link
Copy Markdown
Contributor

Summary

Implements Phase 1 from RFC-0003: Basic controller skeleton with Prometheus integration to query Lumina metrics.

Changes

  • Controller Skeleton: Created basic controller using controller-runtime with leader election
  • Configuration Management: Viper-based config with environment variable overrides (KARVE_*)
  • Prometheus Client: Typed client for querying Lumina metrics (SP capacity, RI, spot/on-demand prices)
  • Metrics Reconciler: Background reconciler that queries Prometheus every 5 minutes and logs capacity data
  • E2E Test Framework: Native Kubernetes client-based E2E tests with real Prometheus and mock Lumina exporter
  • CI/CD: GitHub Actions workflows for unit tests and E2E tests with Kind cluster

Architecture

┌─────────────────┐      ┌──────────────┐      ┌─────────────┐
│ Karve Controller│─────▶│  Prometheus  │◀─────│ Mock Lumina │
│                 │      │              │      │  Exporter   │
└─────────────────┘      └──────────────┘      └─────────────┘
  Queries every 5min     Scrapes every 5s      Serves metrics

Test Coverage

  • Unit tests: 88.7% average coverage
  • E2E tests: 7 specs, all passing
  • Mock Lumina exporter serves Prometheus-format metrics
  • Real Prometheus deployment in E2E tests

Configuration

Default configuration:

  • prometheusURL: http://prometheus:9090 (can override with KARVE_PROMETHEUSURL)
  • logLevel: info (can override with KARVE_LOGLEVEL)

Next Steps (Phase 2)

  • Parse Karpenter NodeClasses and NodePools
  • Build pricing data structures
  • Integrate with Karpenter webhook to adjust pricing

Test plan

  • Unit tests pass (make test)
  • E2E tests pass (make test-e2e)
  • CI/CD workflows execute successfully

🤖 Generated with Claude Code

diranged and others added 15 commits December 5, 2025 12:48
@diranged @claude
feat(init): Phase 1 - Controller skeleton and configuration
8a7810d 
- Initialize Go module with controller-runtime, viper, and prometheus client
- Create configuration management package following Lumina patterns
- Add basic CLI with leader election support
- Set up health check endpoints (/healthz, /readyz)
- Add comprehensive test coverage (96%) for config package
- Create basic CI/CD workflows (test, lint, build)
- Add project documentation (README, CLAUDE.md)
- Follow RFC-0003 Phase 1 deliverables

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
feat(testutil): Add mock Prometheus server for testing
e57266a 
Create testing framework to simulate Lumina metrics without running
actual Lumina or Prometheus instances:

- MockPrometheusServer: In-memory HTTP server with Prometheus API
- Metric fixtures for common scenarios:
  - LuminaMetricsWithSPCapacity: Available RI/SP capacity
  - LuminaMetricsWithNoCapacity: Exhausted capacity
  - LuminaMetricsEmpty: No data (startup state)
  - LuminaMetricsWithSpotPrices: Spot pricing data
- 100% test coverage on testutil package
- Comprehensive README with usage examples

Design rationale:
- Tests actual HTTP code paths (vs config-based mocks)
- No test data pollution in production config
- Flexible for unit, integration, and E2E tests
- Follows Prometheus HTTP API format exactly

Related: RFC-0003 Phase 1 - Prometheus integration testing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
feat(prometheus): add Prometheus client with typed metric queries
c91e46d 
Add Prometheus client package for querying Lumina metrics with full test
coverage. The client provides typed methods for querying:
- Savings Plans remaining capacity
- Reserved Instances
- Spot pricing
- On-demand pricing
- Data freshness

Also enhanced the mock Prometheus server to handle both GET and POST
requests properly, matching the behavior of the real Prometheus Go client.

Test coverage:
- pkg/prometheus: 84.8%
- internal/testutil: 86.2% (updated)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
feat(reconciler): add metrics reconciler for Lumina data
6cb892d 
Add metrics reconciler that periodically queries Prometheus for Lumina
metrics and logs Savings Plans and Reserved Instances capacity. The
reconciler:

- Queries every 5 minutes (configurable)
- Logs data freshness from Lumina
- Logs SP capacity by instance family
- Logs RI availability by instance type
- Integrates with controller manager as a runnable

This completes Phase 1 core functionality for monitoring cost optimization
data from Lumina.

Test coverage:
- pkg/reconciler: 93.9%

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
test(e2e): add E2E test framework with native K8s clients
5b44f43 
Add comprehensive E2E test framework following Lumina patterns:

- Ginkgo/Gomega BDD-style tests with //go:build e2e tags
- Native Kubernetes client helpers (LogsClient, ResourceClient)
- No kubectl exec or curl - all interactions via client-go
- Mock Prometheus deployment for testing metrics reconciler
- Tests verify:
  - Controller startup and health
  - Metrics reconciler starts and queries Prometheus
  - Data freshness logging
  - Graceful handling of empty metrics
  - Continued periodic reconciliation

Test execution:
  make test-e2e  # Requires Kind cluster

This completes Phase 1 with both unit and E2E test coverage.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
refactor(e2e): use native K8s clients instead of kubectl
46caffd 
Refactor E2E test suite to use native Kubernetes client-go API instead
of kubectl exec commands, following Lumina's patterns:

- Replace all kubectl commands with K8s client calls
- Add CreateNamespace, DeleteNamespace methods
- Add CreateDeployment, CreateService, DeleteDeployment, DeleteService
- Add WaitForDeploymentReady helper
- Create all resources (Deployments, Services, ConfigMaps) via API
- Remove dependency on kubectl for resource management

This provides:
- Faster test execution (no subprocess overhead)
- Better error handling
- More idiomatic Go testing
- Consistent with Lumina's E2E patterns

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
ci(e2e): add E2E test workflow with Kind cluster
3330121 
Add dedicated GitHub Actions workflow for E2E tests:

- Runs on PRs and main branch pushes
- Sets up Kind cluster automatically
- Executes E2E tests via make test-e2e
- Collects pod logs and events on failure
- Runs in parallel with unit tests

This ensures E2E tests run in CI and catch integration issues before merge.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
feat(e2e): use real Prometheus with mock Lumina exporter
f9ce02b 
- Add Dockerfile for building Karve controller image
- Create mock Lumina exporter that serves Prometheus-format metrics
- Deploy real Prometheus server in E2E tests to scrape mock exporter
- Update E2E suite to wait for Prometheus to scrape metrics before starting controller
- All 7 E2E tests now passing

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
ci: use runs-on runners matching Lumina configuration
2b7cac7 
- Update test workflow to use 2cpu-linux-x64 runner with magic cache
- Update e2e workflow to use 4cpu-linux-x64 runner with magic cache
- Install kind manually instead of using helm/kind-action
- Pin action versions with commit SHAs for security
- Match Lumina's workflow patterns exactly

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
fix(ci): correct runs-on magic cache syntax
f416fdd 
- Change from cache=/var/cache/magic/extras=magic-cache to extras=s3-cache
- Remove unnecessary step name for runs-on/action@v2
- Follow runs-on.com/caching/magic-cache/ documentation

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
fix(lint): check error return values in prometheus_test.go
fbdef2f 
- Add error checking for json.Decoder.Decode() calls
- Fixes errcheck linter warnings

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
fix(ci): create Kind cluster before running E2E tests
2f5cbc4 
- Add step to create Kind cluster with 5 minute wait
- Add cluster verification step
- Tests were failing because no cluster existed

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
fix(ci): add golangci-lint config and update workflow
0dc8433 
- Add .golangci.yml with explicit linter configuration matching Lumina
- Set 5 minute timeout to prevent workflow timeouts
- Update lint workflow to use runs-on runner with s3-cache
- Upgrade to golangci-lint-action@v9 with version v2.5.0
- Pin action versions with commit SHAs

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
fix(lint): fix all errcheck and lll linter issues
6c00ba7 
- Add error checking for fmt.Fprintf/Fprint calls
- Add error checking for resp.Body.Close() calls using defer with blank identifier
- Add error checking for os.Setenv/Unsetenv in tests
- Fix long line in test/utils/utils.go by splitting fmt.Errorf across multiple lines

All linter issues resolved.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged @claude
fix(lint): refactor duplicate code in Prometheus client
a71824e 
Extract common query building and execution logic into helper functions
to eliminate duplication flagged by dupl linter. The QuerySpotPrice and
QueryOnDemandPrice methods now use shared buildInstanceTypeQuery and
executeQuery helpers.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <noreply@anthropic.com>
@diranged
diranged marked this pull request as ready for review December 6, 2025 00:11
@diranged
diranged merged commit 130cc68 into main Dec 6, 2025
5 checks passed
@diranged
diranged deleted the matt/phase_1 branch December 6, 2025 00:16
@diranged diranged mentioned this pull request Dec 6, 2025
Closed
12 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@diranged