Skip to content

AAP-46484 Capture k8s pod diagnostics - #1332

Closed
arrestle wants to merge 67 commits into
ansible:develfrom
arrestle:capture-k8s-pod-diagnostics
Closed

AAP-46484 Capture k8s pod diagnostics#1332
arrestle wants to merge 67 commits into
ansible:develfrom
arrestle:capture-k8s-pod-diagnostics

Conversation

@arrestle

@arrestle arrestle commented Jun 2, 2025

Copy link
Copy Markdown

Detect a pod failure caused by OOMKilled or other infrastructure related error.

This one outputs:

ERROR: Remote unit failed: Pod diagnostics failed: pod default/kubeit-btlm7 failed due to OOMKilled:  reason OOMKilled

INFO 2025/06/03 17:06:57 Detected EOF for pod default/kubeit-btlm7. {"node_id":"kube-node"}
WARNING 2025/06/03 17:06:59 Pod diagnostics failed: pod default/kubeit-btlm7 failed due to OOMKilled:  reason OOMKilled {"node_id":"kube-node"}

Note we have a new PR 1368 for CapturePodStatus

@arrestle
arrestle marked this pull request as draft June 2, 2025 18:26
@arrestle
arrestle force-pushed the capture-k8s-pod-diagnostics branch from bc6839f to a56b3a7 Compare June 3, 2025 20:26

@lranjbar lranjbar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good! Just some feedback on these :)

Comment thread pkg/workceptor/kubernetes.go Outdated
Comment thread pkg/workceptor/kubernetes.go Outdated
Comment thread pkg/workceptor/kubernetes.go Outdated
Comment thread pkg/workceptor/kubernetes.go Outdated
@arrestle
arrestle force-pushed the capture-k8s-pod-diagnostics branch from 78448c9 to 4ad6fde Compare June 6, 2025 19:29
@arrestle
arrestle force-pushed the capture-k8s-pod-diagnostics branch from 4ad6fde to 55b3047 Compare June 6, 2025 19:38
@arrestle arrestle changed the title Capture k8s pod diagnostics AAP-46484 Capture k8s pod diagnostics Jun 9, 2025
@matoval

matoval commented Jun 9, 2025

Copy link
Copy Markdown
Collaborator

When a pod fails because the ephemeral-storage is over the limit I'm getting these error logs:
ERROR: Remote unit failed: Error in the terminal output and this in the logs:

WARNING 2025/06/09 14:34:02 Pod did not succeed: Error {"node_id":"kube-node"}
WARNING 2025/06/09 14:34:02 Pod did not succeed: Error {"node_id":"kube-node"}

I do see OOMKilled when the pod fails because of an OOMKill

@PabloHiro
PabloHiro force-pushed the capture-k8s-pod-diagnostics branch from cb9a50b to 49afc98 Compare June 10, 2025 10:59
Comment thread pkg/workceptor/mock_workceptor/workunitbase.go Outdated
Comment thread pkg/workceptor/kubernetes.go Outdated
Comment thread pkg/workceptor/kubernetes.go Outdated
@arrestle
arrestle marked this pull request as ready for review June 10, 2025 13:43
Comment thread pkg/workceptor/kubernetes.go Outdated
@codecov

codecov Bot commented Jun 10, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.30769% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 51.52%. Comparing base (55435c6) to head (2bb31a5).
⚠️ Report is 69 commits behind head on devel.

Files with missing lines Patch % Lines
pkg/workceptor/pod.go 94.73% 3 Missing and 1 partial ⚠️
pkg/workceptor/kubernetes.go 80.00% 3 Missing ⚠️
@@            Coverage Diff             @@
##            devel    #1332      +/-   ##
==========================================
- Coverage   51.60%   51.52%   -0.09%     
==========================================
  Files          59       63       +4     
  Lines        9826    10491     +665     
==========================================
+ Hits         5071     5405     +334     
- Misses       4451     4776     +325     
- Partials      304      310       +6     
Files with missing lines Coverage Δ
pkg/workceptor/kubernetes.go 39.14% <80.00%> (+0.42%) ⬆️
pkg/workceptor/pod.go 94.73% <94.73%> (ø)

... and 9 files with indirect coverage changes

Components Coverage Δ
Go 51.64% <92.30%> (+0.04%) ⬆️
Receptorctl 49.31% <ø> (∅)
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@arrestle
arrestle requested a review from PabloHiro June 11, 2025 14:31
@arrestle
arrestle marked this pull request as draft June 23, 2025 20:46
@lranjbar
lranjbar dismissed their stale review June 23, 2025 20:55

Lisa became collaborator on the PR

Comment thread pkg/workceptor/kubernetes.go
Comment thread pkg/workceptor/kubernetes.go
Comment thread pkg/workceptor/pod.go Outdated
Comment thread pkg/workceptor/pod.go Outdated
Comment thread pkg/workceptor/pod.go Outdated
Comment thread pkg/workceptor/pod.go Outdated
Comment thread pkg/workceptor/pod.go Outdated

@PabloHiro PabloHiro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you for the hard work!

@sonarqubecloud

sonarqubecloud Bot commented Jul 2, 2025

Copy link
Copy Markdown

@arrestle
arrestle marked this pull request as ready for review July 2, 2025 18:52

@davemulford davemulford left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed during a 1:1 call, talked through some of the testing methods. Looks great!

@AaronH88 AaronH88 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking right now

podName,
)

// timeout := int64(10)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Take out commented out code please,

The tests are enough for now

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AaronH88 I have a conflict here because @lranjbar would like to keep the comment in the code so we know where it goes in the future.

}

// only transition from WorkStateRunning to WorkStateSucceeded if WorkStateFailed is set we do not override
// pod, err := kw.KubeAPIWrapperInstance.Get(kw.GetContext(), kw.clientset, podNamespace, podName, metav1.GetOptions{})

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same as above
Take out commented out code please,

The tests are enough for now

@arrestle

Copy link
Copy Markdown
Author

Close this was merged because it's a duplicate of #1371 and #1372

@arrestle arrestle closed this Aug 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants