🐛 [BUG] - nnsmith crashes when saving a bug report for a TensorFlow model

[supermarkli]

Description

When the nnsmith fuzzer attempts to save a problematic model, it crashes during the call to the tf.saved_model.save function, causing the fuzzing loop to terminate prematurely.

This prevents the fuzzer from continuing to search for other potential bugs within the allocated time.

The crash occurs within the dump method in nnsmith/materialize/tensorflow/__init__.py. This method directly calls tf.saved_model.save to save the model.

The problematic code snippet is as follows:

# nnsmith/materialize/tensorflow/__init__.py:137
           tf.saved_model.save(
               self.net,
               os.path.join(path, TFModel.tfnet_dir_name()),
               signatures=concrete_net,
           )

Here, self.net is a plain Python object that contains state in the form of tf.Variables, but it is not a subclass of tf.Module. When tf.saved_model.save attempts to save such an object, it fails to correctly track and serialize the tf.Variables captured by the tf.function, leading to an internal error and program crash.

Installation

pip install "nnsmith[torch,onnx]" --upgrade

Reproduction

CUDA_VISIBLE_DEVICES=0 nnsmith.fuzz model.type=tensorflow fuzz.root=./tensorflow backend.type=xla cmp.with.type=xla fuzz.time=14400 backend.target=gpu cmp.with.target=gpu

Screenshots

![DESCRIPTION](LINK.png)

Logs

WARNING exec   - Saving bug report to tensorflow/bug-Symptom.INCONSISTENCY-Stage.VERIFICATION-10
Error executing job with overrides: ['model.type=tensorflow', 'fuzz.root=./tensorflow', 'backend.type=xla', 'cmp.with.type=xla', 'fuzz.time=14400', 'backend.target=gpu', 'cmp.with.target=gpu']
Traceback (most recent call last):
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/cli/fuzz.py", line 263, in main
    FuzzingLoop(cfg).run()
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/cli/fuzz.py", line 237, in run
    if not self.validate_and_report(testcase):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/cli/fuzz.py", line 201, in validate_and_report
    if not verify_testcase(
           ^^^^^^^^^^^^^^^^
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/cli/model_exec.py", line 78, in verify_testcase
    if not check_result(
           ^^^^^^^^^^^^^
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/cli/model_exec.py", line 67, in check_result
    bug_report.dump(odir)
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/materialize/__init__.py", line 330, in dump
    self.testcase.dump(root_folder)
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/materialize/__init__.py", line 275, in dump
    self.model.dump(
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/nnsmith/materialize/tensorflow/__init__.py", line 137, in dump
    tf.saved_model.save(
  ...
  File "/home/lzh/tools/miniforge3/envs/gputf/lib/python3.11/site-packages/tensorflow/python/eager/polymorphic_function/saved_model_exported_concrete.py", line 45, in __call__
    export_captures = _map_captures_to_created_tensors(

Front-end framework

tensorflow

Version of the front-end framework

No response

Back-end engine

xla

Version of the front-end engine

No response

Other context

I am new to this area, so I'm not sure if my understanding is correct. Please feel free to correct me if I'm wrong. Thank you!