It looks like --force doesn't rerun cram tests. Should it?

[Kakadu]

It successfully reruns inline_tests, but not cram tests. I think it should

I tried to study 'dune trace cat' and indeed when .t file was "touched" I have +1 line in the trace. Demo: https://github.com/Kakadu/dune_test_force/blob/master/Makefile

[shonfeder]

I would agree that it should!

[Alizter]

Here is some recent discussion around --force: #15581 (comment)

[shonfeder]

Thanks, for the reference Ali. However, I don't see anything very illuminating to the point here. I see the two claims

--force only applies to anonymous actions attached to aliases. There are various internal actions that are not attached to aliases (e.g. ocamldep rules), which I presume we wouldn't want to rerun on --force

I think --force was already a bit arbitrary, so we shouldn't worry about keeping it's behaviour the same. If we think --force will be useful, which is already a bit dubious, it would be better to spend time giving it a better meaning at some later stage.

But these rather beg the point claimed here: whatever unclarity there may be in the semantics of --force in other parts of the build, in the context of running tests it seems patently obvious to me that dune test --force should rerun all the tests. Do you disagree on this point? If so, I'd be very interested to hear the grounds, as I don't have any foothold to entertain an opposing opinion at this point.

I can see how there may be edge cases where it can be difficult in principle to differentiate a build dependency from execution of a test, but surely that is not the case here, is it? Rerunning a cram test just means re-executing all of its commands, no?

[Alizter]

I wasn't making a point in either direction yet, but I can counter with another question:

Under which circumstances does it make sense to re-run a test?

I posit that most users will have a wide range of reasons for doing so, so I'm more interested in solving their concrete specific problems rather than giving them something like --force to rely on.

Generally if a test is required to be re-run, it is non-deterministic. There are some circumstances where non-determinism has its place, but I don't think we should be encouraging users to write their cram tests in this way.

On the contrary, I believe that we should be strict and encourage users to specify their dependencies correctly to make sure the execution of the test is as it claims. Otherwise, what's the point?

[Kakadu]

Under which circumstances does it make sense to re-run a test?

For example, when test create artifacts not in the build directory. In my case, instrumentalization with bisect_ppx creates some files in /tmp, which names are not statically known.

[shonfeder]

How can you tell if a test is truly non-deterministic in its logic if you can't rerun without changing the build (vs. e.g., some non-determinism in the build)?

Integration tests and E2E tests are a very common and compelling use case for cram-based tests, in which case what you are testing against may just not be under your control and its impossible to reach the ideal of a fully deterministic test.

I believe that we should be strict and encourage users to specify their dependencies correctly to make sure the execution of the test is as it claims. Otherwise, what's the point?

I'd posit the point is meeting real and common user needs, even if they are not able to achieve the ideal of complete purity :)

Edit: I should add, if we had an ergonomic way to guarantee determinism, I could agree that --force is just not necessary, but OCaml is an impure language and dune runs cram tests without strong isolation on impure OSs in an impure world, so it's hard for me to see a realistic near future where --force won't be useful!

[Kakadu]

Do we know a workaround? I tried running find regression -iname '*.t' -exec touch {} \; before coverage checking but it doesn't help

[Alizter]

@Kakadu touch won't work because dune digests the contents of the file rather than taking into account timestamps. If you make a trivial edit to the cram test it should rerun the test. Alternatively you can add a dependency on (universe) which will rerun the test every time.