#2341 Testing projects xUnit v2 to v3 migration

[methran1304]

Closes #2341

• Migrate testing projects to xUnit.net v3 #2341

This PR introduces xUnit v2 to v3 package upgrade for UnitTests and AcceptanceTests projects

Proposed Changes

• Added xunit.v3 package reference.
• Removed xunit v2.9.3.
• Removed Xunit.SkippableFact as xUnit v3 has native support for skipping tests.
• Replaced Skip.If() with Assert.SkipWhen().
• Replaced Skip.IfNot() with Assert.SkipUnless().
• Updated Xunit.Abstractions to Xunit namespace (moved in v3)
• Improved test clarity by adding required reason string to skip assertions.

Docs: Migration v2 to v3

[methran1304]

Test run (.NET 8, 9, 10) using xUnit.net VSTest Adapter v3.1.5

[methran1304]

I've been hitting intermittent "address already in use" failures on ConsulTests when running dotnet test.

The issue seems to be that xUnit v3 compiles test projects as executables instead of class libraries. So when tests run across multiple target frameworks (net8.0/net9.0/net10.0), they spin up separate processes that run in parallel. I can confirm this because I don't face the same issue when I run dotnet run --framework net8.0/net9.0/net10.0.

Edit: Same issue prior to upgrade. Is there a fix planned for this in roadmap?

Maybe we can use port 0 and let the system handle the port selection?

[coveralls]

coverage: 93.632%. remained the same
when pulling ef1e9cf on methran1304:mgs25/unittests-xunitv3-migration
into b8cc478 on ThreeMammals:develop.

[raman-m]

I've been hitting intermittent "address already in use" failures on ConsulTests when running dotnet test.

The issue seems to be that xUnit v3 compiles test projects as executables instead of class libraries. So when tests run across multiple target frameworks (net8.0/net9.0/net10.0), they spin up separate processes that run in parallel. I can confirm this because I don't face the same issue when I run dotnet run --framework net8.0/net9.0/net10.0.

I've seen similar issues before, but with xunit v2 and typically running tests through Visual Studio Test Explorer, the behavior has been stable in CI/CD. I suspect these failures happen when multiple TFMs run in parallel, whereas Visual Studio executes them separately and sequentially. Running test projects in parallel might be triggering the problem.

Edit: Same issue prior to upgrade. Is there a fix planned for this in roadmap?
Maybe we can use port 0 and let the system handle the port selection?

Avoid running testing projects in parallel and instead mimic Visual Studio Test Explorer’s behavior. In Ocelot CI/CD setup (GitHub Actions), I've written scripts to run one testing project and one TFM at a time.

[methran1304]

I've seen similar issues before, but with xunit v2 and typically running tests through Visual Studio Test Explorer, the behavior has been stable in CI/CD. I suspect these failures happen when multiple TFMs run in parallel, whereas Visual Studio executes them separately and sequentially. Running test projects in parallel might be triggering the problem.

Yes, I have also noticed an issue while running the test on a single TFM using xUnit v3 built-in runner. During the first run after a clean build, all the test cases seem to pass just fine. However, the subsequent runs fail due to port allocation (Ports are still being used). So, it could a process created during runtime that's non-terminated keeping the port still in use. Regarding your comment on stability when it comes to CI/CD test runner, I think it's because CI runner are ephemeral while our local machines are not. What are your thoughts on this?

Avoid running testing projects in parallel and instead mimic Visual Studio Test Explorer’s behavior. In Ocelot CI/CD setup (GitHub Actions), I've written scripts to run one testing project and one TFM at a time.

Yes. I'll do that.

[methran1304]

@raman-m All changes are ready for review.

[raman-m]

@raman-m All changes are ready for review.

Great! I will review after pushing an important commit to develop, hopefully in 2-3 days.

[methran1304]

@raman-m All changes are ready for review.

Great! I will review after pushing an important commit to develop, hopefully in 2-3 days.

Just a quick reminder on this @raman-m

[raman-m]

Do you see the merge conflicts below? They need to be resolved first. The latest build has also failed, so I need to fix that issue before anything else. You will have to wait until the build in develop is fixed.

[methran1304]

Do you see the merge conflicts below? They need to be resolved first. The latest build has also failed, so I need to fix that issue before anything else. You will have to wait until the build in develop is fixed.

I'll keep an eye on the develop build. I'll resolve the merge conflicts after the build succeeds.

[methran1304]

Should I sync this branch with develop? I can see the latest develop build has failed and changes haven't been made since then. What do you suggest @raman-m?

[raman-m]

@methran1304 yes, please.
Seems this PR is prepared well and it'll be merged first than my current work.

[raman-m]

There are some significant issues discovered in the test implementations and core library, but they fall outside the scope of this PR. These will be addressed in upcoming commits before the release.

Approved ✅

Just a note, more feedback on the user experience with the new testing v3 framework should be collected, as parallelization is a key focus for the team.