Appropriately deal with files that don't end with a line terminator

[Aster89]

Both the given and expected files don't have a line terminator at EOF.

The new test does fail, demonstrating there is a bug, just like I observe in VSCode, see GIF attached to #5059.

However, Vim+YCM and Neovim seem to be immune to it. (As far as Vim+YCM goes, I know why it's immune because I fixed another bug, ycm-core/YouCompleteMe#4311.)

I think that the test failing shows that the bug is in HLS, not in VSCode. Vim+YCM and Neovim are probably just being smart and sidestepping the bug entirely.

---

Fixes #5059.

[Aster89]

A small experiment that I should probably turn into a test:

$ cabal  repl /home/enrico/haskell-language-server/hls-plugin-api/src/Ide/PluginUtils.hs
λ> import Language.LSP.Protocol.Types
λ> import Ide.PluginUtils
λ> :set -XOverloadedStrings
λ> uri = Uri {getUri = "file:///home/enrico/haskell-language-server/plugins/hls-class-plugin/test/testdata/T1W.hs"}
λ> verTxtDocId = VersionedTextDocumentIdentifier uri 0
λ> old = "module T1 where\n\ndata X = X\n\ninstance Eq X where\n"
λ> new = "module T1 where\n\ndata X = X\n\ninstance Eq X where\n  (==) = _\n"
λ> e1 = diffText' True (verTxtDocId, old) new IncludeDeletions
λ> e1
WorkspaceEdit {_changes = Nothing, _documentChanges = Just [InL (TextDocumentEdit {_textDocument = OptionalVersionedTextDocumentIdentifier {_uri = Uri {getUri = "file:///home/enrico/haskell-language-server/plugins/hls-class-plugin/test/tes
tdata/T1W.hs"}, _version = InL 0}, _edits = [InL (TextEdit {_range = Range {_start = Position {_line = 5, _character = 0}, _end = Position {_line = 5, _character = 0}}, _newText = "  (==) = _\n"})]})], _changeAnnotations = Nothing}

See that the WorkspaceEdit contains _newText = " (==) = _\n".

This is correct. But look what happens if we remove the trailing \n to both old and new:

λ> old = "module T1 where\n\ndata X = X\n\ninstance Eq X where"
λ> new = "module T1 where\n\ndata X = X\n\ninstance Eq X where\n  (==) = _"
λ> e2 = diffText' True (verTxtDocId, old) new IncludeDeletions
λ> e1 == e2
True

which is wrong! The _newText should be "\n (==) = _", not " (==) = _\n".

These are probably the shortest reproduction steps (but diffTextEdit is not currently exported):

λ> d1 = diffTextEdit "foo" "foo\nbar" IncludeDeletions 
λ> d2 = diffTextEdit "foo\n" "foo\nbar\n" IncludeDeletions 
λ> d1 == d2
True
λ> d1
[TextEdit {_range = Range {_start = Position {_line = 1, _character = 0}, _end = Position {_line = 1, _character = 0}}, _newText = "bar\n"}]

[Aster89]

Well, the bug is clearly on this line:

haskell-language-server/hls-plugin-api/src/Ide/PluginUtils.hs

Line 125 in d3464b6

d = getGroupedDiff (lines $ T.unpack fText) (lines $ T.unpack f2Text)

I mean, once you've done lines on the input texts, your linebreaks are long gone. Unless you re-inspect the input texts to see whether they ended with a line terminator, but that's not done, as fText and f2Text are only used on this line.

[Aster89]

Building on my earlier experience with text diff tools, the solution I've attempted consisted of

1. swapping Prelude's lines/unlines for these

   lines = split (dropFinalBlank $ keepDelimsR $ whenElt (== '\n'))
   unlines = concat

2. swapping getGroupedDiff for getGroupedDiffBy ((==) on` takeWhile (/= '\n'))``

The idea is that point 1 allows us not to throw away the line terminators, and point 2 preserves the equality used so far.

The drawback is that the resulting [Diff [String]] for foo vs foo\nbar is the following

[Both ["foo"] ["foo\n"],Second ["bar"]]

from which the current code deduces that "bar" is the only thing to be added to the Secondside, forgetting entirely that there was a\n` difference between the two first lines.

In the context of a full-fledged text diff tool, the direction I'd take is to perform a sub-comparison between the 2 sides of the Boths; eventually we'd get something like this,

[(Both ["foo"] ["foo\n"], Just (NonEmpty [Both "foo" "foo", Second "\n"])]),(Second ["bar"], Nothing)]

where the Maybe wraps the possibly absent/empty subcomparison.

If we were to diff foo\n vs foo\nbar\n then the above "diff+subdiff" would look like this:

[(Both ["foo\n"] ["foo\n"], Nothing]),(Second ["bar\n"], Nothing)]

---

Anyway, I've taken note of the above to avoid forgetting, but it sounds too much of a complication considering that the only time that Just would ever materialize is when we're adding a line at the end of a Windows/VSCode/windows-like-thingy--generated file.

And maybe it would break several tests.

Probably a simple hack is a better approach. Looking into it. But I also have to check what happens in case an action removes the last line of a file.

[Aster89]

This PR, in its current state, is

• a partial solution to the bug that affects diffTextEdit intended as a standalone function,
• but it's an actual solution to Class plugin breaks code when used at last line of a file that does not end with a line terminator #5059.

What do I mean?

1. From the perspective of text comparison, the solution is wrong, as demonstrated by the two tests that fail;

   • incidentally, it doesn't even update the the TextEdit's _range field, only the _newText field;

2. from the perspective of our usage of it, which is from call sites that guarantee (or don't they?) that we'll never get those inputs like in the files that cause those failures, it's good enough.

The point is that we never use this diffTextEdit function on two independent Text inputs. Those two inputs are

• the content of the source file on which HLS wants to do the change,
• the content after the change as computed via GHC's API.

As long as GHC (well, and ghc-exactprint after it) guarantees to honor the line-ending policy of a file, the scenario of the tests at point 1 above should never materialize.