PR 1: Detect undersized BlueStore DB devices - #1120
Conversation
There was a problem hiding this comment.
Pull request overview
Adds a new Ceph OSD health signal to detect undersized BlueStore DB devices on the local host by correlating locally discovered OSD IDs (from ceph-volume lvm list) with each OSD’s admin-socket perf dump (bluefs.db_total_bytes), and raises a scenario warning when the DB device is ≤ 5 GiB.
Changes:
- Introduces a
CephChecks.local_osds_with_small_bluestore_dbcheck with a 5 GiB threshold based onbluefs.db_total_bytes. - Extends CLI catalog support to read the “classic” sosreport admin-socket perf-dump filename for OSDs.
- Adds unit + scenario coverage and a new scenario raising
CephOSDWarningwhen undersized DB devices are detected.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tests/unit/storage/test_ceph_common.py | Unit test covering detection of local OSDs with small BlueStore DB devices. |
| hotsos/defs/tests/scenarios/storage/ceph/ceph-osd/small_bluestore_db.yaml | Scenario test validating the warning message is raised when the check reports bad OSDs. |
| hotsos/defs/scenarios/storage/ceph/ceph-osd/small_bluestore_db.yaml | New scenario definition that raises CephOSDWarning based on the new check property. |
| hotsos/core/plugins/storage/ceph/common.py | Implements the new CephChecks property and the 5 GiB threshold constant. |
| hotsos/core/host_helpers/cli/catalog.py | Adds sosreport file-source support for ceph daemon osd.N perf dump output. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Implement the first item of canonical#974 only: warn when a local OSD has a positive BlueStore DB capacity of 5 GiB or less. Discover local OSDs from ceph-volume lvm list and obtain each OSD's DB capacity from bluefs.db_total_bytes in its admin-socket perf dump. Support the current standard Ceph sosreport perf-dump filename, which avoids relying on ceph report: that cluster-wide command can fail on OSD-only hosts without an admin keyring. Add unit and scenario coverage for the warning. Other checks proposed by canonical#974, including BlueStore DB utilisation and spillover detection, are intentionally not included in this change. Signed-off-by: Zhang Hua <joshua.zhang@canonical.com>
f43a8c2 to
49a8780
Compare
|
hua@y9000p:/bak/work/hotsos$ .tox/py3/bin/python -m hotsos.cli --storage --scenario storage.ceph.ceph-osd.small_bluestore_db --short data/hotsos-small-db-test.tar.xz |
pponnuvel
left a comment
pponnuvel
left a comment
There was a problem hiding this comment.
Looks good overall but I do want to understand how 5GiB was chosen and whether it's good for different OSD sizes.
|
|
||
| @cached_property | ||
| def local_osds_with_small_bluestore_db(self): | ||
| """Return local OSDs with a BlueStore DB device at most 5 GiB.""" |
There was a problem hiding this comment.
Use BLUESTORE_DB_SIZE_MIN instead of 5 GiB
| """ Ceph Checks. """ | ||
| # A 1 GiB BlueStore DB is the ceph-volume default and is generally too | ||
| # small for production OSDs. | ||
| BLUESTORE_DB_SIZE_MIN = 5 * 1024 * 1024 * 1024 |
There was a problem hiding this comment.
wy 5 GiB? The typical recommendation is 4% of the block dev size. So it's possible that a DB dev is > 5GiB but still be "small" for that OSD.
Implement the first item of #974 only: warn when a local OSD has a positive BlueStore DB capacity of 5 GiB or less.
Discover local OSDs from ceph-volume lvm list and obtain each OSD's DB capacity from bluefs.db_total_bytes in its admin-socket perf dump. Support the current standard Ceph sosreport perf-dump filename, which avoids relying on ceph report: that cluster-wide command can fail on OSD-only hosts without an admin keyring.
Add unit and scenario coverage for the warning.
Other checks proposed by #974, including BlueStore DB utilisation and spillover detection, are intentionally not included in this change.