Fix integer overflow in NHWC convolution kernel size calculation

[YellowJune]

Summary

This PR fixes a critical integer overflow vulnerability in the NHWC convolution operator. The overflow occurred during the calculation of the kernel size (kernel_height * kernel_width), leading to incorrect microkernel selection and undersized heap memory allocation.

Problem

In src/operators/convolution-nhwc.c, the kernel size was calculated using 32-bit integer arithmetic before being widened to size_t. When the product of kernel_height and kernel_width exceeds 2^32 (e.g., 65536 x 65536), the value wraps to zero.

This zero-value propagates through the system, causing:

1. Incorrect Microkernel Selection: find_dwconv_ukernel(0) matches any registered kernel, regardless of the actual filter size.
2. Undersized Allocation: The packed_weights buffer is allocated only for bias terms, leaving no space for filter weights.
3. Heap Buffer Overflow: During weight packing, the actual (large) dimensions are used to write into the undersized buffer, resulting in an out-of-bounds heap write.

Fix

1. Added Overflow Guard: Introduced a check in select_microkernel_type to reject kernel dimensions that would overflow size_t.
2. Size_t Arithmetic: Applied explicit size_t casting before multiplication in all critical paths (creation, reshape, and microkernel selection) to ensure 64-bit arithmetic is used.
3. Hardened Reshape Path: Updated effective_kernel_height/width calculations to prevent similar overflows when large dilations are applied.

This fix aligns with previous security patches applied to deconvolution (#9876) and pooling/NCHW convolution (commit 32e57d1).

Verification Results

• Verified that 65536 x 65536 is correctly calculated as 4,294,967,296 instead of 0.
• Confirmed the overflow guard correctly triggers for values exceeding SIZE_MAX.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[YellowJune]

Does the merge process automatically after copybara validation? Or, do you handle it manually after the full approval?

[qukhan]

It will process automatically.

[YellowJune]

It will process automatically.

However, I see that "1 workflow awaiting approval", which means task requires approval from a maintainer.. Can you please recheck about this? Best regards,