Phase 2: NodePool Watching & Analysis

[diranged]

Goal

Query and understand available pre-paid AWS capacity (Compute SPs, EC2 Instance SPs, and RIs) from Lumina metrics to inform NodeOverlay creation decisions.

Background

Architecture Decision: Karve will create NodeOverlays for ALL instance families/types with available capacity, regardless of current NodePool configuration (Option 2 approach). This is simpler and more resilient than watching NodePools.

NodeOverlays are non-invasive - if no NodePool uses a particular instance family, the overlay is simply ignored by Karpenter.

Deliverables

1. Capacity Type Discovery

• Query Lumina Prometheus metrics for all capacity types:
  • Compute Savings Plans (global, all families)
  • EC2 Instance Savings Plans (per-family)
  • Reserved Instances (per-instance-type)
• Parse and structure capacity data for overlay decisions
• Handle missing/stale metrics gracefully

2. Utilization Tracking

• Query savings_plan_utilization_percent for each SP
• Query savings_plan_remaining_capacity for validation
• Query ec2_reserved_instance for RI availability
• Implement configurable utilization threshold (default: 95%)

3. Overlay Decision Logic

• Determine which overlays should exist based on:
  • Utilization < threshold (e.g., 95%)
  • Remaining capacity > 0
• Build data structures mapping capacity → overlay specs:

  type OverlayDecision struct {
      Name        string  // e.g., "cost-aware-compute-sp-global"
      Weight      int     // Precedence: RI=30, EC2-SP=20, Compute-SP=10
      Price       string  // "0.00" for 100% discount
      Requirements []Requirement
      ShouldExist bool    // true if utilization < threshold
  }

4. Logging & Observability

• Log all capacity sources discovered
• Log overlay decisions (create/delete/keep)
• Include utilization percentages in logs
• Emit metrics for monitoring (optional for Phase 2)

Implementation Details

Prometheus Queries

Compute Savings Plans (global):

savings_plan_utilization_percent{type="compute"}
sum(savings_plan_remaining_capacity{type="compute"})

EC2 Instance Savings Plans (per-family):

savings_plan_utilization_percent{type="ec2_instance"}
savings_plan_remaining_capacity{type="ec2_instance"}
# Labels include: instance_family, region

Reserved Instances (per-instance-type):

ec2_reserved_instance
# Labels include: instance_type, availability_zone, region

Overlay Decision Algorithm

// For each capacity source:
utilization := QueryUtilization(capacityType)
remaining := QueryRemaining(capacityType)
threshold := config.UtilizationThreshold  // default: 95

shouldExist := (utilization < threshold) && (remaining > 0)

decisions = append(decisions, OverlayDecision{
    Name: GenerateOverlayName(capacityType),
    Weight: GetWeight(capacityType),  // RI=30, EC2-SP=20, Compute-SP=10
    Price: "0.00",  // 100% discount
    ShouldExist: shouldExist,
})

Configuration Support

Add to config.yaml:

overlayManagement:
  utilizationThreshold: 95  # Delete overlays at this utilization %
  weights:
    reservedInstance: 30
    ec2InstanceSavingsPlan: 20
    computeSavingsPlan: 10

Example Log Output

{
  "timestamp": "2025-10-27T12:05:00Z",
  "message": "Capacity analysis complete",
  "compute_sp": {
    "utilization_percent": 87.5,
    "remaining_capacity_dollars_per_hour": 12.50,
    "overlay_decision": "create",
    "overlay_name": "cost-aware-compute-sp-global"
  },
  "ec2_instance_sp_m5": {
    "utilization_percent": 96.2,
    "remaining_capacity_dollars_per_hour": -0.80,
    "overlay_decision": "delete",
    "overlay_name": "cost-aware-ec2-sp-m5"
  },
  "reserved_instances": {
    "c5.xlarge": {
      "count": 5,
      "overlay_decision": "create",
      "overlay_name": "cost-aware-ri-c5-xlarge"
    }
  }
}

Success Criteria

• Correctly queries all Lumina capacity metrics
• Builds overlay decision list for all capacity types
• Logs show clear reasoning for each decision
• Configurable utilization threshold works
• Handles metric staleness/errors gracefully
• Unit tests for decision logic
• Integration tests with mock Prometheus responses

Testing Requirements

Unit Tests

• Overlay decision logic with various utilization percentages
• Threshold configuration (90%, 95%, 100%)
• Handling missing metrics (SP without utilization data)
• Weight assignment for different capacity types

Integration Tests

• Query mock Prometheus with sample Lumina metrics
• Parse responses into overlay decisions
• Verify correct overlay names/specs generated
• Test with multiple SPs, RIs simultaneously

Non-Goals (Deferred)

• ❌ Watching Karpenter NodePool CRs (not needed for Option 2)
• ❌ Parsing NodePool requirements (not needed for Option 2)
• ❌ Actually creating NodeOverlay CRs (that's Phase 4-5)
• ❌ Tracking per-instance cost allocation (Lumina handles this)

References

• RFC-0003 Phase 2 (lines 779-796) - NOTE: Approach changed to Option 2
• Lumina Metrics Documentation
• Karpenter Instance Family Labels
• Issue Testing: Validate Karpenter NodeOverlay Behavior #9: Testing Karpenter NodeOverlay behavior

[diranged]

Phase 2 is complete. All deliverables have been implemented:

Completed Items

1. Capacity Type Discovery ✅

• Prometheus client queries all Lumina metrics (Compute SPs, EC2 Instance SPs, RIs)
• Structured data types: SavingsPlanCapacity, ReservedInstance
• DataFreshness() method for staleness handling

2. Utilization Tracking ✅

• QuerySavingsPlanUtilization() for SP utilization
• QuerySavingsPlanCapacity() for remaining capacity
• QueryReservedInstances() for RI availability
• Configurable threshold (default 95%)

3. Overlay Decision Logic ✅

• Decision struct with all required fields
• DecisionEngine with analyze methods for each capacity type
• Aggregation functions for multiple SPs/RIs
• Configurable weights (RI=30, EC2-SP=20, Compute-SP=10)

4. Logging & Observability ✅

• Detailed logging of capacity sources and decisions
• Utilization percentages included in logs
• (Metrics emission marked optional, deferred)

5. Testing ✅

• Unit tests: decision_test.go, aggregation_test.go, client_test.go
• Integration tests: integration_test.go

Implemented in PR #10: feat: Phase 2 - Capacity Discovery and Decision Engine