Skip to content

Add some logging to GPU detection #6992

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
davidpanderson wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Add some logging to GPU detection #6992

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
6861268
client: timestamp GPU detect warnings
davidpanderson Apr 12, 2026
c27eb8d
on my computer, cuMemGetInfo_v2() always return 201 (no context)
davidpanderson Apr 13, 2026
b663abb
client: logic tweak
davidpanderson Apr 13, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion client/gpu_detect.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -833,7 +833,7 @@ void COPROCS::bound_counts() {
}

void gpu_warning(vector<string> &warnings, const char* msg) {
fprintf(stderr, "%s\n", msg);
fprintf(stderr, "[%s] %s\n", time_to_string(dtime()), msg);
warnings.push_back(msg);
Comment on lines 835 to 837

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

gpu_warning() now calls time_to_string(dtime()), but this file doesn't include str_util.h where time_to_string() is declared. This will fail to compile; include str_util.h (or otherwise ensure time_to_string is declared) in this translation unit.

Copilot uses AI. Check for mistakes.
}

Expand Down
26 changes: 15 additions & 11 deletions client/gpu_nvidia.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -584,32 +584,36 @@ static void get_available_nvidia_ram(COPROC_NVIDIA &cc, vector<string>& warnings
retval = (*p_cuDeviceGet)(&device, cc.device_num);
if (retval) {
snprintf(buf, sizeof(buf),
"[coproc] cuDeviceGet(%d) returned %d", cc.device_num, retval
"cuDeviceGet(%d) returned %d", cc.device_num, retval
);
gpu_warning(warnings, buf);
return;
}
retval = (*p_cuCtxCreate)(&ctx, 0, device);
if (retval) {
snprintf(buf, sizeof(buf),
"[coproc] cuCtxCreate(%d) returned %d", cc.device_num, retval
"cuCtxCreate(%d) returned %d", cc.device_num, retval
);
gpu_warning(warnings, buf);
return;
}
if (p_cuMemGetInfo_v2) {
retval = (*p_cuMemGetInfo_v2)(&memfree, &memtotal);
}
else {
// cuMemGetInfo_v2() always returns 201 (no context)
if (p_cuMemGetInfo) {
retval = (*p_cuMemGetInfo)(&memfree, &memtotal);

@cubic-dev-ai cubic-dev-ai Bot Apr 13, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1: Handle cuMemGetInfo*() failures before updating available RAM; otherwise an error now turns available_ram into 0 instead of preserving the fallback total memory value.

Prompt for AI agents 
Check if this issue is valid — if so, understand the root cause and fix it. At client/gpu_nvidia.cpp, line 602:

<comment>Handle `cuMemGetInfo*()` failures before updating available RAM; otherwise an error now turns `available_ram` into 0 instead of preserving the fallback total memory value.</comment>

<file context>
@@ -598,19 +598,22 @@ static void get_available_nvidia_ram(COPROC_NVIDIA &cc, vector<string>& warnings
     // cuMemGetInfo_v2() always returns 201 (no context)
-    if (false && p_cuMemGetInfo_v2) {
+    if (p_cuMemGetInfo) {
+        retval = (*p_cuMemGetInfo)(&memfree, &memtotal);
+        snprintf(buf, sizeof(buf),
+            "cuMemGetInfo() returned %d; dev %d free %zu total %zu",
</file context>
Fix with Cubic

}
if (retval) {
snprintf(buf, sizeof(buf),
"[coproc] cuMemGetInfo(%d) returned %d", cc.device_num, retval
"cuMemGetInfo() returned %d; dev %d free %zu total %zu",
retval, cc.device_num, memfree, memtotal
Comment on lines +600 to +605

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The if (false && p_cuMemGetInfo_v2) branch disables cuMemGetInfo_v2() unconditionally. If only cuMemGetInfo_v2 is available (p_cuMemGetInfo is null), the else-branch will dereference a null function pointer and crash. Prefer selecting the available symbol (use v2 when v1 is missing), or try v2 first and fall back to v1 on CUDA_ERROR_INVALID_CONTEXT (201).

Copilot uses AI. Check for mistakes.
);
gpu_warning(warnings, buf);
(*p_cuCtxDestroy)(ctx);
return;
} else if (p_cuMemGetInfo_v2) {
retval = (*p_cuMemGetInfo_v2)(&memfree, &memtotal);
snprintf(buf, sizeof(buf),
"cuMemGetInfo_v2() returned %d; dev %d free %zu total %zu",
retval, cc.device_num, memfree, memtotal
);
gpu_warning(warnings, buf);
} else {
gpu_warning(warnings, "No MemGetInfo function available");
}
(*p_cuCtxDestroy)(ctx);
cc.available_ram = (double) memfree;
Expand Down
14 changes: 8 additions & 6 deletions client/gpu_opencl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -745,13 +745,15 @@ void COPROCS::get_opencl(
double freq = ((double)prop.max_clock_frequency) * MEGA;
prop.peak_flops = ((double)prop.max_compute_units) * freq;
}
char buf2[256];
snprintf(buf2, sizeof(buf2),
"OpenCL generic: peak FLOPS %f; Max units %u, max freq %u MHz",
prop.peak_flops,
prop.max_compute_units, prop.max_clock_frequency
);
gpu_warning(warnings, buf2);
if (prop.peak_flops <= 0 || prop.peak_flops > GPU_MAX_PEAK_FLOPS) {
char buf2[256];
snprintf(buf2, sizeof(buf2),
"OpenCL generic: bad peak FLOPS; Max units %u, max freq %u MHz",
prop.max_compute_units, prop.max_clock_frequency
);
gpu_warning(warnings, buf2);
gpu_warning(warnings, "bad peak flops value; using default");
Comment on lines +749 to +756

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This hunk introduces tab-indented lines (prop.peak_flops argument and the subsequent gpu_warning call), which are inconsistent with the surrounding space indentation and make the block harder to read/maintain. Replace the tabs with spaces to match the file’s existing indentation style.

Copilot uses AI. Check for mistakes.
prop.peak_flops = GPU_DEFAULT_PEAK_FLOPS;
Comment on lines 755 to 757

Copilot AI Apr 13, 2026

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The new warning message "bad peak flops value; using default" is ambiguous on its own (it doesn’t identify the device/vendor or the rejected value). Consider including at least the computed peak_flops (and/or vendor/name) so logs make sense when multiple devices are enumerated.

Copilot uses AI. Check for mistakes.
}

Expand Down