Silver Ideas

[remexre]

Making a discussion for this because Slack #ideas disappears after a while...

[remexre]

It sounds like Dart puts tokens in a doubly-linked list, so that they can get the ignore terminals that came before any given terminal and the preceding non-ignore terminal. This seems pretty easy to implement; we'd add a couple mutable fields to every token:

public abstract class Terminal {
	@Nullable
	public Terminal prev, next;
	public boolean wasIgnored = true;
	...
}

The lexer then keeps around the last token it produced, to link in the next one. Finally, the parser sets wasIgnored = false; on any tokens that end up as children (or sets it to false on any that are ignored, whatever's easier to implement).

There are a few ways to expose this to Silver; the most direct one would be:

-- I guess this needs to be in core and have magic attached; users shouldn't be able to write instances, and the compiler should generate instances...
class Terminal<a> {
	nextTerminal :: (Maybe<AnyTerminal> ::= a);
	prevTerminal :: (Maybe<AnyTerminal> ::= a);
	terminalWasIgnored :: (Boolean ::= a);
}

nonterminal AnyTerminal; -- lol

abstract production anyTerminal
Terminal a =>
AnyTerminal ::= a
{}

instance Terminal<AnyTerminal> {
	...
}

Having this would make it easier to write an ignore-terminal-preserving unparse attribute; this would be somewhat useful in its own right for debugging and diagnostics, but is more important IMO as a prerequisite for writing refactors and other maintainable-source-to-maintainable-source transformations in Silver. It'd also let us get rid of the @{- -} syntax for doc comments and switch to {-- -} or plain {- -}.

[krame505]

I don't think you would need AnyTerminal, couldn't you just write

class Terminal<a> {
	nextTerminal :: Terminal<b> => (Maybe<b> ::= a);
	prevTerminal :: Terminal<b> => (Maybe<b> ::= a);
	terminalWasIgnored :: (Boolean ::= a);
}

Another option is putting this info on TerminalDescriptor, and add an easy way to get from a terminal to a TerminalDescriptor. Although, we could also probably now do away with TerminalDescriptor by adding a Terminal type class like this.

[krame505]

Or wait nevermind, we want it to be rank-2 since it can return any type of terminal. Yeah, having AnyTerminal is a tad annoying but there probably isn't a less-gross workaround, since I'm not adding general support for rank-n types :-P

[remexre]

I intended that to be like Maybe<exists b. Terminal<b> => b> ::= a, so the caller could distinguish between "no more terminals" and "a terminal of unknown type;" AnyTerminal is providing the existential.

TerminalDescriptor and Terminal/AnyTerminal are probably redundant, yeah.

[remexre]

Silver debuggability enhancement: kill -USR2 makes Silver dump a (Silver) stack trace, so you can tell where it's stuck if Silver is infinitely looping.

Further enhancement that's more work than it's worth, but might be a better fit for an optimizing compiler for a strict language: have a simple termination checker; when printing that stack trace, put a * next to anything that doesn't pass that termination checker.