[WIP] Code generator random Fortran programs

[dorchard]

This is a WIP PR to provide a technique for automatically enumerating valid Fortran programs using QuickCheck style generation. The aim is to provide a facility for easily generating sample programs which can be used for testing other tools, e.g., refactoring tools and compilers. This leverages the machinery of QuickCheck's Arbitrary class.

Typing and scoping are the main challenges. Looking into work such as Making Random Judgments: Automatically Generating
Well-Typed Terms from the Definition of a Type-System (ESOP 2015)
Also relevant: https://dl.acm.org/doi/pdf/10.1145/3363562
See also CSmith: https://users.cs.utah.edu/~regehr/papers/pldi11-preprint.pdf

Phase 1

• Simple value generator
• More complex values
• Source spans and positions (that do not need to be accureate)
• Well-typed expressions

Phase 2

• Statements
• Blocks

Phase 3

• Program units

[ksromanov]

@dorchard

Dominic, there is also a problem of an ecosystem for such a tool:

Suppose we write such a tool and target fortran-src only. Even at the very early stage of development the tool might reveal some problems, that are going to get fixed. However, to uncover more errors, the tool has to become more and more sophisticated. Since the errors will be fixed almost immediately (it is much easier to fix them than to find), the tool will become less useful with time, though getting more and more complex and hard to maintain. At some point we might end up with a tool, that is a state-of-art fuzzer that still can find nothing in fortran-src.

Therefore, we either should target other parsers/compilers as well, or CI to check for fortran-src regressions.

As an example, our simple fuzzy experiment was able to find a bug #251, however it was the only finding. So, at the moment it is completely useless.

[dorchard]

This is a good point @ksromanov - I was thinking about generating programs to test other compilers and tools, not just fortran-src itself. That would at least extend its lifetime / value, i.e., even if it turns up no issues with tool X, it might do in another new tool Y. But looking also at more 'fuzzing' based approaches might be good to, i.e., parsing in a program, then generating a lot of programs based from it. Do you think that goes beyond the problem you foresee, at least somewhat?