feat/damlc test improved output by roger-bosman-da · Pull Request #23139 · digital-asset/daml

roger-bosman-da · 2026-07-01T11:21:43Z

Summary

Improves daml test output to be more readable and actionable:

Buffer diagnostics for controlled output ordering — diagnostics are collected during execution and flushed after tests complete, allowing the summary to appear below trace output (diagnostics first, then summary at the end for visibility)
Add pass/fail counts to summary header
Hide passing tests when there are failures (focus on what needs fixing)
Auto-detect TTY for colored output
Only show coverage stats when --show-coverage is passed
Show package name in header, relative paths in summary, absolute paths in diagnostics
Group test results by file with cleaner formatting

Fixes #23132
Fixes #23185
Fixes #23188
Fixes #23189

Prints "N failed, M passed" (or just "M passed" when all green) directly under the Test Summary header so you can gauge the damage at a glance without scanning every line. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

When any test fails, the summary now only lists the failures — the count line already reports how many passed. This matches the convention of pytest, go test, and cargo test where passing detail is noise when you're triaging failures. Pass --verbose / -v to restore the full listing. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

When --color is not passed, check whether stdout is a TTY and enable color automatically. This matches the convention of grep, ls, cargo test, etc. — color on in interactive use, off when piped. --color still forces color on regardless of TTY detection. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

The coverage skeleton (~25 lines of "Modules internal/external..." counts) was printed unconditionally, even when all values were zero. Gate the entire block on --show-coverage so the default output stays focused on test results. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> --- PR title: Improve `daml test` output readability ## Summary `daml test` output was too verbose and didn't surface failures prominently. Users reported having to scroll through passing tests, coverage stats, and diagnostic traces just to find what broke. This PR makes the default output concise and failure-focused. ## Changes ### Buffer diagnostics so summary prints first Diagnostic detail blocks (the verbose `File: ... Severity: DsError ...` output) were appearing before the Test Summary because they wrote to stderr (unbuffered) while the summary wrote to stdout (line-buffered). Now diagnostics are buffered in an IORef and flushed after the summary, with an explicit `hFlush stdout` to guarantee ordering. ### Add pass/fail count line The Test Summary now shows `N failed, M passed` (or just `M passed` when all green) right under the header so you can gauge the damage at a glance. ### Hide passing tests when there are failures When any test fails, only the failures are listed in the summary — the count line already tells you how many passed. Pass `--verbose` / `-v` to restore the full listing. This matches pytest, go test, and cargo test conventions. ### Auto-detect color support Color is now enabled automatically when stdout is a TTY, instead of requiring `--color`. `--color` still forces color on regardless of TTY detection. ### Only show coverage stats when `--show-coverage` is passed The ~25 line "Modules internal/external to this package" coverage skeleton was printed unconditionally, even when all values were zero. It's now gated on `--show-coverage`. ## Before ``` File: CoinIssuance.daml Severity: DsError Message: Script execution failed: ... File: LockingByChangingState.daml Severity: DsError Message: Script execution failed: ... Test Summary CoinDelegation.daml:delegate: ok, 4 active contracts, 8 transactions. CoinIssuance.daml:coinIssuance: failed CoinTransferWithAuthorization.daml:tokenAccept: ok, 3 active contracts, 11 transactions. LimitedTimeCoinTransfer.daml:limitedTimeCoinTransfer: ok, 2 active contracts, 7 transactions. LockingByChangingState.daml:locking: failed LockingBySafekeeping.daml:locking: ok, 3 active contracts, 11 transactions. MultiplePartyAgreement.daml:multiplePartyAgreementTest: ok, 1 active contracts, 8 transactions. Utilities.daml:makeParties: ok, 0 active contracts, 0 transactions. Modules internal to this package: - Internal templates 21 defined 15 ( 71.4%) created - Internal template choices 56 defined 15 ( 26.8%) exercised ... (25+ more lines of coverage) ``` ## After ``` Test Summary 2 failed, 6 passed CoinIssuance.daml:coinIssuance: failed LockingByChangingState.daml:locking: failed File: CoinIssuance.daml Severity: DsError Message: Script execution failed: ... File: LockingByChangingState.daml Severity: DsError Message: Script execution failed: ... ``` Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

When running `daml test` across multiple subprojects (e.g. via sbt's `damlTest`), each test suite's output is now clearly bracketed with the project directory: - A "Running tests (/path/to/project) ..." header at the start - The existing "Test Summary (/path/to/project)" with pass/fail counts at the end This makes it easy to tell which project produced which output when many suites run sequentially. - Thread the project path from `runTestsInPackageOrFiles` through `execTest` → `runAndReport` → `printSummary` - Add `printTestSuiteBegin` to print a header before test execution - Update test assertions in `DamlcTest.hs` to verify the project path appears in both the begin and summary headers, and to pass `--show-coverage` where coverage output is checked Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com> Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…hen no project path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

This commit makes two improvements to the test output: 1. Combine test summary with counts on a single line: - Changed format from multi-line to: "Test summary for "path": X passed, Y failed" - Added "Failing tests:" section header before listing failures - Added blank line between summary and failing tests for readability 2. Display relative paths instead of absolute paths: - Project path in summary is relative to CWD where damlc is invoked - Test file paths are relative to the project path (when available) - Falls back to CWD-relative paths when no project path is set - Makes output more concise and readable Example output: Test summary for "docs/source/sdk/sdlc-howtos/smart-contracts/develop/patterns": 6 passed, 2 failed Failing tests: daml/CoinIssuance.daml:coinIssuance: failed daml/LockingByChangingState.daml:locking: failed Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

This commit addresses feedback from parallel daml test execution: 1. Use package name instead of relative path as identifier: - When daml.yaml is present, use the package name from daml.yaml - This provides a stable, unique identifier even when cwd == package-root - Falls back to path when no package config is available - Fixes the issue where all packages showed "." in parallel runs 2. Distinguish "no tests found" from "0 passed": - Empty test suites now show "Test summary: No tests found" - Non-empty suites show "Test summary: X passed, Y failed" - Makes it clear when a package has no test scripts vs ran tests Example output for parallel runs: Running tests (my-package-name) ... Test summary for "my-package-name": No tests found Running tests (other-package) ... Test summary for "other-package": 2 passed, 1 failed Note: Per-line tagging for fully demultiplexable parallel output would require more invasive changes to the logging infrastructure and is deferred. Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

This commit improves test output for readability: 1. Test summary uses short relative paths: - Paths are relative to package root for conciseness - Example: daml/Test.daml:main: ok - Makes output easy to scan 2. Reorder test output (failures last, most visible): - Summary header (X passed, Y failed) - All passed tests - All failed tests (always last, easier to read bottom-to-top) 3. Remove verbose flag functionality: - Previously: --verbose showed all tests, non-verbose showed only failures - Now: always shows all tests (passed then failed) - The --verbose flag is kept for CLI backward compatibility but ignored - Added comment in execTest documenting this Example output: Test summary for "my-package": 1 failed, 1 passed daml/Test.daml:main: ok, 1 active contracts, 2 transactions. daml/Test2.daml:main: failed Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

Convert relative file paths to absolute paths when printing diagnostics, making it easier to navigate to error locations from terminal output. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

Use a compact "Failure in: <path>" / "Message: <message>" format instead of the verbose multi-field format (File, Hidden, Range, Source, Severity, Message). Collapses whitespace in messages to keep output on minimal lines. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

This reverts commit 9916abd31511170ee83327ce80f589fca8e16e6e.

This reverts commit 9f3d0a6f29c05d07dd998000e0e16fee3d163bc6.

- Fix O(n²) list append in bufferingDiagnosticsLogger by prepending and reversing in flushDiagnostics - Clear IORef buffer in flushDiagnostics after reading to prevent duplicate output on repeated calls - Remove unused --verbose flag and Verbose type - Fix test summary format to use "Test Summary (name)" instead of "Test summary for \"name\"" to match test expectations - Hide passed tests when there are failures (show only failures) - Document makeRelative behavior: returns absolute path unchanged if paths have different roots (e.g., different drives on Windows) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

- Add script name to diagnostic _source field (e.g., "Script: main" instead of just "Script") so users can identify which test failed in verbose output - Fix prettyResult to only color "ok" green, not the entire result string Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

…plify summary - Add blank line after each diagnostic for better readability - Simplify test summary format: "file.daml: N tests failed" with indented list for multiple failures, or "file.daml: 1 test failed: name" for single failure - Remove unused prettyResult function and Data.Vector import - Update test expectations to match new output format Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

- Make "Failure in" text red when color is enabled - Change path format from "(Script: main)" to ":main" suffix - Remove "Message:" line - directly show the error message - Add blank line after each diagnostic for readability Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

The test output changes inadvertently changed printDiagnostics to use a simplified format, breaking the damlc lint output which expects the colored diagnostic format from showDiagnosticsColored. - Restore printDiagnostics to use showDiagnosticsColored (original) - Update DamlcTest to expect package names in headers (matching the behavior introduced in "Use package name as identifier") Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

dylant-da

Looks good with minor nitpicks, thank you for this!

dylant-da · 2026-07-22T08:48:48Z

+        (if colored then setSGRCode [SetConsoleIntensity BoldIntensity] else "")
+        <> "Running tests (" <> identifier <> ") ..."
+        <> (if colored then setSGRCode [] else "")


Instead, let's try a function like "renderColor"

Suggested change

(if colored then setSGRCode [SetConsoleIntensity BoldIntensity] else "")

<> "Running tests (" <> identifier <> ") ..."

<> (if colored then setSGRCode [] else "")

renderColor color [Important] ("Running tests (" <> identifier <> ") ...")

Which handles unsetting the color, and lets us add another backend in the future (Html, for IDE output)

dylant-da · 2026-07-22T08:48:57Z

+    if nTotal == 0
+      then do
+        putStrLn $
+          (if colored then setSGRCode [SetUnderlining SingleUnderline, SetConsoleIntensity BoldIntensity] else "")


dylant-da · 2026-07-22T08:49:04Z

+      else do
+        let countLine
+              | nFailed > 0 =
+                  (if colored then setSGRCode [SetColor Foreground Vivid Red] else "")


dylant-da · 2026-07-22T08:49:43Z

+                  <> (if colored then setSGRCode [] else "")
+                  <> ", " <> show nPassed <> " passed"
+              | otherwise =
+                  (if colored then setSGRCode [SetColor Foreground Vivid Green] else "")


Same here, and I let you find the rest

dylant-da · 2026-07-22T08:50:39Z

    loggerH <- getLogger opts "test"
+    color <- if getUseColor color then pure color else do
+        isTTY <- hIsTerminalDevice stdout
+        pure $ UseColor isTTY


Suggested change

pure $ UseColor isTTY

color <- if getUseColor color then pure color else fmap UseColor (hIsTerminalDevice stdout)

Dropping unnecessary use of do

dylant-da · 2026-07-22T08:54:47Z

+            printScriptResults color mbProjectPath res
+
+        -- Show failed tests last (most visible)
+        when (nFailed > 0) $


Make sure to report that these are only the failed tests, so that it's clear that the successes aren't included.

roger-bosman-da changed the title ~~Roger/feat/damlc test improved output~~ feat/damlc test improved output Jul 2, 2026

roger-bosman-da force-pushed the roger/feat/damlc-test-improved-output branch 3 times, most recently from 9281104 to acfd3a0 Compare July 2, 2026 13:52

This was referenced Jul 9, 2026

Show failed files first in Daml Script test summary #23185

Open

Sometimes the daml test summary is not provided, but only the lengthy output #23188

Open

daml test output is too verbose to jump to failing source files easily #23189

Open

roger-bosman-da force-pushed the roger/feat/damlc-test-improved-output branch from acfd3a0 to a88758c Compare July 20, 2026 11:22

github-actions Bot added the Standard-Change label Jul 20, 2026

roger-bosman-da and others added 21 commits July 20, 2026 18:17

[UNDO ME] break tests

8857188

Move test summary below diagnostics and hide 'Running tests' header w…

78e9b88

…hen no project path Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Signed-off-by: Roger Bosman <roger.bosman@digitalasset.com>

Revert "Simplify diagnostic output format for test failures"

679e7e3

This reverts commit 9916abd31511170ee83327ce80f589fca8e16e6e.

Revert "[UNDO ME] break tests"

7017e5b

This reverts commit 9f3d0a6f29c05d07dd998000e0e16fee3d163bc6.

add release notes

50815df

rm pr-description.md

04860b2

roger-bosman-da force-pushed the roger/feat/damlc-test-improved-output branch from a88758c to 30f9496 Compare July 20, 2026 16:17

github-actions Bot removed the Standard-Change label Jul 20, 2026

roger-bosman-da force-pushed the roger/feat/damlc-test-improved-output branch from 30f9496 to 04860b2 Compare July 20, 2026 16:17

roger-bosman-da marked this pull request as ready for review July 21, 2026 09:02

dylant-da approved these changes Jul 22, 2026

View reviewed changes

Navigation Menu

Uh oh!

feat/damlc test improved output - #23139

feat/damlc test improved output#23139
roger-bosman-da wants to merge 21 commits into
mainfrom
roger/feat/damlc-test-improved-output

roger-bosman-da commented Jul 1, 2026 •

edited

Loading

Uh oh!

dylant-da left a comment •

edited

Loading

Uh oh!

dylant-da Jul 22, 2026

Uh oh!

dylant-da Jul 22, 2026

Uh oh!

dylant-da Jul 22, 2026

Uh oh!

dylant-da Jul 22, 2026

Uh oh!

dylant-da Jul 22, 2026

Uh oh!

dylant-da Jul 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

	pure $ UseColor isTTY
	color <- if getUseColor color then pure color else fmap UseColor (hIsTerminalDevice stdout)

Uh oh!

Conversation

roger-bosman-da commented Jul 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Uh oh!

dylant-da left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dylant-da Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dylant-da Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dylant-da Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dylant-da Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dylant-da Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

dylant-da Jul 22, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

roger-bosman-da commented Jul 1, 2026 •

edited

Loading

dylant-da left a comment •

edited

Loading