Elixir port of Cucumber Expressions?

[aslakhellesoy]

Hi everyone,

I'm about to start porting Cucumber Expressions to Elixir, following the existing conventions and shared test data.

I'll implement it with Claude Code. I'm not sure what the current team's stance is wrt to agent-authored code. Based on the number of pull requests that have been rejected as "slop", I think maybe it's something you don't want. In that case I'll maintain it in a forked repo.

FYI - I intend to use it in https://var.oselvar.com.

Should I send a PR or not bother?

Aslak

[laeubi]

@aslakhellesoy I don't think it matter if you write it on paper or using vi or a IDE or an AI tool. What more matters is if you tell the agent "do this or that" it will do something in that direction but there is no guarantee it is correct (a bit like a non deterministic Turing machine always require a final deterministic step that tell "is the solution true or false").

So if you would supervise, understand and refine the result (what I have no doubt you would be able to do) then this is completely acceptable. What usually is qualified as slop is more when people don't do that or even don't understand what the tool has produced and just throwing it off hoping that other do the hard work for them!

A quite good comprehensive summary is given here: https://docs.github.com/en/copilot/tutorials/review-ai-generated-code

[mpkorstanje]

We recently added a section on AI usage to the contributing guide. I reckon that will broadly explain why those PRs were closed as slop.

Now I can't really say up front if the PR would be accepted because "implemented with Claude Code" doesn't convey anything about the quality on its own. Though I would prefer it if you scrutinized every line before you ask anyone else to do the same.

From a practical perspective though, since you'll be using this in Var, wouldn't you expect design pressures to push the expressions into a different direction?

[aslakhellesoy]

I don't really know Elixir, so my ability to review the code is limited.

That said, I've steered Claude to closely follow existing implementations (while keeping it Idiomatic Elixir) and of course to make sure it implements tests using the same test data. There is very little room for an agent to go out on a limb and do crazy stuff given all the patterns and constraints that are already in the repo.

There is already an Elixir port of Cucumber Expressions, but it hasn't been touched for four years and differs considerably from the official ones. I've instructed the agent to take inspiration from whatever good Elixir idioms this implementation uses, but to keep the implementation on par with the other official ones.

You can see it here: cucumber/cucumber-expressions@main...varar-dev:cucumber-expressions:elixir
It's using the Varar namespace for now since we're undecided whether this should be a PR against cucumber or stay as a fork. If you're OK with a PR, I'll change it to `Cucumber.

Based on my experience implementing Varar for 7 languages so far, I don't expect any new design pressures. Cucumber Expressions is a beautifully designed and small/closed system, and it does (and exposes) everything I need.

The only thing that might exercise pressure would be to align the official implementations with the Rust implementation. I think it might have some corectness and performance benefits over what's official, but this is just a hunch. And that's a different discussion. If we want to go down that route, I think it would be best to adopt the Rust implementation into the official repo and update the other implementations to align with it. Something I expect an agent would do very well.

So to sum it up - I think a single official Cucumber Expressions repo with many language ports would benefit both Cucumber and Varar.

You might wonder why I'm making an Elixir implementation if I don't know or use Elixir. It's for the same reasons we ported Cucumber to many languages - build a big community, increase adoption and mindshare.

[1]: I renamed Vár to Varar

[aslakhellesoy]

And while I'm haven't quite bought into dark factories yet, my experience is that with good steer and guardrails, it's possible to produce very good code without having to read it all.

I don't expect everyone to agree with this.

[mpkorstanje]

I don't really know Elixir, so my ability to review the code is limited.

So you haven't reviewed it at all? If so, I definitely don't want to review it.

Overall, I don't hear you speak positively about the code you'd submit.

You do speak highly of Claude, but I've seen others use Claude and the results always needed polishing and refinement. So that's not a convincing argument.

Furthermore, you say that your proces produces good results but you also lack the ability to verify that it actually does. So this loop isn't closed.

Funny thing is I had to learn some basic Elixir. It was not that difficult. And I do hold you high regard, so if you carefully review to the best of your ability what your agent produced, I'd consider that sufficient to close the loop.

Oh, and some test coverage report with a minum coverage gate would be good. It wouldn't be the first time an agent produces dead code.

[aslakhellesoy]

Fair pushback @mpkorstanje.

I've added code coverage, which is now at 100%, with a gate at 100%. This does not prove the absence of dead code though. In theory there may be code that's only exercised by tests. But by bringing the coverage up to this level, I did find and fix a couple of bugs with named regexp groups.

I've also added a guardrail that verifies all 115 tests from the test data corpus are run. If this number changes in the future, the Elixir build will fail.

I've reviewed all the code to my best ability. More details in cucumber/cucumber-expressions#439

You do speak highly of Claude, but I've seen others use Claude and the results always needed polishing and refinement. So that's not a convincing argument.

I've also seen others use agents to produce code of low quality, myself included. I still see it daily. In my experience, the quality of what comes out depends largely on three factors:

1. What assumptions the agent needs to make
2. What guardrails are in place
3. The quality of the LLM itself

For green-field code (new features), there are lots of assumptions. But this is not green-field. There are 6 pre-existing implementations that are used as context. All the decisions are already there. It's just a translation, with very solid guardrails (test data). This double-pressure pushes the code in a direction where the things to critique is mostly to what degree the code follows good idioms.

For me, the tipping point was when Opus came out in March. This is when I started seeing a significant drop in how much I needed to fix myself, given the assumptions were few (steered by existing code and prompts) and the guardrails were strong.

Furthermore, you say that your proces produces good results but you also lack the ability to verify that it actually does. So this loop isn't closed.

I disagree that I lack that ability. I have strong signals:

• The full acceptance test corpus is verified
• All tests are passing
• Test coverage is 100%
• The code looks similar to the Ruby code (my subjective assesment)
• I've added Styler - didn't find anything
• I've added property tests with StreamData - didn't find anything

Incidentally, I found a bug in the .NET implementation. It computes incorrect offsets for astral plane characters: 😀 {int}. I'll create a separate PR for that.

[mpkorstanje]

I disagree that I lack that ability.

That could have been worded better on my end. I see that you've reviewed the code in cucumber/cucumber-expressions#439.