Remove the deriving-aeson dependency

[Unisay]

plutus-core used deriving-aeson only for the CustomJSON deriving-via instances of the cost model types. This PR rewrites those as plain aeson generic instances (genericToJSON/genericToEncoding/genericParseJSON with explicit Options) and drops both the dependency and the allow-newer: deriving-aeson:aeson exception that #7820 had to add because deriving-aeson caps aeson < 2.3 on Hackage. The only public API change: PlutusCore.Evaluation.Machine.ExBudget no longer exports LowerInitialCharacter, which existed solely to support deriving-aeson.

The serialized JSON is unchanged, and the two commits are ordered so that this is checkable. The first adds tests comparing the serialized cost models against the checked-in cost-model/data files for every semantics variant, using those files as the expected output rather than a copy of them. The tests pass against the existing instances, and the second commit replaces the instances without touching them.

Closes #7870.

[zliu41]

What tests exist that confirm that the behavior didn't change?

[Unisay]

Added them in the first commit, so they run against the old instances and again unchanged against the new ones.

The serialized cost models are compared against the checked-in cost-model/data/*.json files for every semantics variant. Those files are the frozen format, so they serve as the expected output directly and there is no second copy to keep in sync. That covers renamed fields, renamed constructor tags, and fields appearing or disappearing, and it is stronger than what we had: the TH loading in ExBudgetingDefaults only checks that the files still parse, never that the encoder reproduces them. The remaining cases cover the field-omitting instances that generate-cost-model relies on.

Everything is compared as Values rather than bytes, since field order carries no meaning here: every consumer either parses the JSON or goes through CostModelParams, which is a sorted map. Differences are reported as a JSON patch via aeson-diff, so a failure names the offending entries by path instead of printing both cost models in full.

One thing worth being explicit about: the models under test are loaded from those same files by TH, so this is a fixed-point check on the encoder rather than on the cost model numbers, and anything that breaks decoding already fails at the splice. What it adds on top is the case where toJSON drifts away from parseJSON, which is newly possible now that the two are written out separately.

The rest was already covered elsewhere: CostModelInterface.Spec pins the flattened parameter keys against a checked-in params file, plutus-ledger-api has goldens for the ledger parameter names and counts, and the Agda conformance suite runs fromJSON @BuiltinCostKeyMap . toJSON over the builtin costs, cross-checking the generic encoder against the hand-written SimpleJSON parser.

[zliu41]

Also, allow-newer is bad if it is needed by a package released on CHaP, since all downstream repos must also declare allow-newer. Is this the case with deriving-aeson? If not, then I'm not sure what the advantage of removing it is, since it does make our code more complex.

[Unisay]

Also, allow-newer is bad if it is needed by a package released on CHaP, since all downstream repos must also declare allow-newer. Is this the case with deriving-aeson?

Indeed, this is the reason - deriving-aeson has incompatible constraints, so without this PR all downstream consumers that include it in their build plan (directly or transitively) would need to declare the same allow-newer exception.

[zliu41]

@kwxm please review

[kwxm]

This looks fine: it simplifies the dependencies and doesn't make things significantly more complicated: if fact making things a bit more explicit may be clearer.

I had a couple of comments where I wondered if things could be improved/simplified, but they're not urgent; I think @Unisay is off at the moment and I don't want to delay merging until he gets back. I made a ticket to look at those later.

[kwxm]

I was a bit worried that these might give different results, but if you look at the implementation in deriving-aeson it has

data CamelTo (separator :: Symbol)
...
type CamelToSnake = CamelTo  "_"
...
instance (KnownSymbol separator, NonEmptyString separator) => StringModifier (CamelTo separator) where
  getStringModifier = camelTo2 char
    where
      char = case symbolVal (Proxy @separator) of
        c : _ -> c
        _ -> error "Impossible"

so the previous code was secretly using camelTo2 anyway.

It does look as if you have to be a bit careful about what conversion functions like this do in general. For example we have a constructor called LinearInMaxYZ and there's a question of what happens if you have a sequence of multiple uppercase letters. In this case camelTo2 '_' gives linear_in_max_yz, but linear_in_max_y_z might be justifiable as well (and it looks as if there are functions in other libraries that would do that).

[kwxm]

The code here at least is at about the same level of complexity as the previous version, but maybe a bit more explicit, so it seems like a reasonable replacement. It's still very repetitive though (and that maybe makes it tempting to cut and paste when we introduce something new, which carries a risk of failing to update some of the pasted code). I wonder if we can simplify it by introducing a typeclass of our own to do some of the work? We probably want to merge this quite soon though, so we could come back to that later instead of trying to fix it now. I've made a ticket to think about that.

[kwxm]

Is this comment true? The ledger currently just uses an array of untagged numbers, although I think it maybe needed the tags in the past. It's probably best not to change anything though, especially if there's a risk of changing the ordering.

[kwxm]

We'll have to remember to update this if we ever add a new variant. Can we add a check that we've covered all of the variants here? It's possible that we need stuff like this elsewhere, so maybe we could implement a single function somewhere that returns all of the variants and their associated files and use that here.

[kwxm]

Maybe qualify this so we can see what we're using?

I tried import Barbies (ConstraintsB, TraversableB, FunctorB) but that mysteriously gave me several errors like this:

plutus-core/src/PlutusCore/Evaluation/Machine/BuiltinCostModel.hs:202:46: error: [GHC-18872]
    • Couldn't match representation of type: GHC.Generics.K1
                                               GHC.Generics.R (f ModelFourArguments)
                               with that of: barbies-2.1.1.0:Data.Generics.GenericN.Rec
                                               (barbies-2.1.1.0:Data.Generics.GenericN.Param
                                                  0 f ModelFourArguments)
                                               (f ModelFourArguments)
        arising from the 'deriving' clause of a data type declaration
        The data constructor ‘GHC.Generics.K1’
          of newtype ‘GHC.Generics.K1’ is not in scope
    • When deriving the instance for (ConstraintsB
                                        BuiltinCostModelBase)
    |
202 |   deriving anyclass (FunctorB, TraversableB, ConstraintsB)
    |                                              ^^^^^^^^^^^^

Not sure what's going on there.