[fix] Skip storage capacity/utilization gauges when capacity is None

[huniu20]

Background

Follow-up of #130, which made total_storage_size optional so that a storage unit can be configured with unlimited capacity.

Problem

When a storage unit is created with total_storage_size=None (unlimited), SimpleStorageUnit._handle_get_metrics reports capacity=None in its /metrics payload.

The metrics exporter in transfer_queue/metrics.py defines two prometheus_client.Gauge objects that consume this value:

• self.storage_capacity = Gauge("tq_storage_capacity_total", ...) — around L164
• self.storage_utilization = Gauge("tq_storage_utilization_ratio", ...) — around L170

Inside TQMetricsExporter._collect_storage_metrics, both gauges are updated unconditionally:

capacity = metrics.get("capacity", 0)
active   = metrics.get("active_keys", 0)
self.storage_capacity.labels(storage_unit_id=label).set(capacity)
...
self.storage_utilization.labels(storage_unit_id=label).set(
    active / capacity if capacity > 0 else 0.0
)

When capacity is None, Gauge.set() internally does float(value), and float(None) raises TypeError. Because the whole block is wrapped in try/except, this is caught by the surrounding handler and turned into:

Failed to collect metrics from storage unit <su_id>: float() argument must be a string or a real number, not 'NoneType'

This warning is emitted every TQ_METRICS_COLLECT_INTERVAL for every unlimited-capacity storage unit, flooding the logs and hiding real issues.

Fix

In TQMetricsExporter._collect_storage_metrics, treat capacity is None as "unlimited" and skip only the storage_capacity and storage_utilization gauges for that storage unit. All other metrics (storage_active_keys, storage_memory_rss, per-op request stats such as count / latency avg / p50 / p99) are still reported normally, so dashboards for unlimited-capacity units keep working — they simply no longer publish a (meaningless) capacity/utilization value.

Related

• [feat] Make total_storage_size optional to support unlimited storage capacity #130 Make total_storage_size optional to support unlimited storage capacity

[ascend-robot]

CLA Signature Pass

huniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[ascend-robot]

CLA Signature Pass

huniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍

[Copilot]

Pull request overview

This PR fixes Prometheus metrics collection for “unlimited capacity” storage units by avoiding publishing capacity/utilization gauge values when the storage unit reports capacity=None, preventing periodic float(None) errors and log flooding.

Changes:

• Skip setting tq_storage_capacity_total when capacity is None.
• Skip setting tq_storage_utilization_ratio when capacity is None (while continuing to export other storage metrics).

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[0oshowero0]

We can have an else here to deal with the remove logic. For example

     else:
         for gauge in (self.storage_capacity, self.storage_utilization):
             try:
                  gauge.remove(label)
             except (KeyError, ValueError):
                  pass

[huniu20]

Done. Added an else branch to prune the stale storage_capacity / storage_utilization series when a unit transitions to unlimited, mirroring the existing partition-level pruning. Also covered Copilot's suggestion with two unit tests in test_metrics.py.

[ascend-robot]

CLA Signature Pass

huniu20, thanks for your pull request. All authors of the commits have signed the CLA. 👍