测试问题修复:🐛 修复 engineV4 终态日志 torch_error / paddle_cuda 重叠问题#678
Merged
Conversation
|
Thanks for your contribution! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🧭 背景
PaddleAPITest 的 accuracy_stable 测试中,
paddle.nn.functional.embedding在 0-size weight 场景下触发 CUDA illegal memory access。该错误发生在 torch backward 阶段(由_for_test_check_cuda_error()检测到),tester 层正确将其归为torch_error。但 engineV4 主进程收到 worker crash 后,对 exitcode=99 一律写入paddle_cuda,导致同一 config 同时出现在torch_error和paddle_cuda两份日志中,最终统计数据产生交集。engineV2 早已通过
FATAL_TORCH_EXIT_CODE(97)区分 torch 阶段与 paddle 阶段的 CUDA fatal error,engineV4 缺失这一逻辑。🛠️ 实现方案
将 engineV4 的 worker 异常处理对齐 engineV2 逻辑:先查询
terminal_log_type(tester 层已写入的终态类型),再结合错误消息中的 cuda/oom markers 决定 exit code。torch 阶段写入的torch_error+ CUDA error →FATAL_TORCH_EXIT_CODE(97);paddle 阶段 →FATAL_CUDA_EXIT_CODE(99)。主进程根据 exit code 分类写入对应终态日志。同时删除
base.py中report_runtime_error的强制覆盖逻辑(原来default_log_type == "torch_error"时会覆盖classify_runtime_error的结果),使分类结果始终以实际错误特征为准。在
aggregate_logs(end=True)结尾新增互斥性 assert,如有任何 config 出现在多种终态日志中立即暴露问题。🔧 主要变更
1. engineV4 CUDA 归因对齐 engineV2
引入
FATAL_CUDA_EXIT_CODE/FATAL_OOM_EXIT_CODE/FATAL_TORCH_EXIT_CODE常量。Worker 异常处理改为:读取terminal_log_type+ 匹配 cuda/oom markers → 选择对应 exit code → 写 checkpoint →os._exit。主进程根据 exit code 写入paddle_cuda/oom/torch_error。2. 移除 report_runtime_error 的强制覆盖
删除
elif default_log_type == "torch_error" and log_type != "oom": log_type = "torch_error",classify_runtime_error的明确结果不再被覆盖。3. 终态日志互斥性 assert
aggregate_logs(end=True)在返回log_counts前扫描所有终态日志文件,发现同一 config 出现在多种类型中时 assert 并打印诊断信息。不做任何去重——纯粹检测。4. 统计输出数字右对齐
print_log_info中所有数值使用{:>8}格式化,对齐更易读。📁 改动文件
✅ 验证
print_log_info输出格式示例如下: