Add upload simulation dataset feature#1852
Draft
remy-rabideau wants to merge 9 commits into
Draft
Conversation
… for creating simulation datasets with profiles
…plement in PostgresPlanRepository and InMemoryPlanRepository
…nstead of individual parameters Consolidate uploadSimulationDataset parameters into a single SimulationResults object. Update Hasura action definition, parsers, and repository implementations. Add SimulationResultsParser to deserialize JSON format produced by SimulationResultsWriter, including parsing of duration strings, timestamps, profile segments, activities, and events. Update PostgresPlanRepository to extract data from SimulationResults and write
…on dataset uploads
…oadSimulationDataset JSON input Add simulationArguments Map<String, SerializedValue> field to SimulationResults with constructor overload for backward compatibility. Update SimulationResultsParser to parse optional simulationArguments field from JSON. Modify PostgresPlanRepository to use parsed simulationArguments instead of empty map when creating simulation datasets. Add unit tests verifying parsing with and without simulationArguments fiel
The parser now handles optional "topics" and "events" sections in the upload JSON. Topics are parsed as a map of name -> schema and converted to List<Triple<Integer, String, ValueSchema>> with synthesized indices. Flat events are grouped by (realTime, transactionIndex), then reconstructed into EventGraph<EventRecord> objects via EventGraphUnflattener.unflatten() for insertion into the database. Previously these fields were ignored, resulting in uploaded simulation datasets always having 0 events.
This was referenced Jul 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
REQUIRES_GATEWAY_PR="148"
NASA-AMMOS/plandev-gateway#148
Description
Adds a
POST /uploadSimulationDatasetHasura action and merlin-server endpoint that accepts a simulation results JSON payload inSimulationResultsWriterformat and persists it to the database associated with an existing plan.The primary use case is transferring a simulation dataset from one plan to another — for example, a cloned or slightly modified plan using a compatible model — or between PlanDev instances. Ingesting results produced by external simulators is also supported. The
SimulationResultsWriterformat can be produced by aerie-cli viaplans download-simulation-full-results.The full pipeline: Hasura receives the mutation → forwards the action envelope to merlin-server →
SimulationResultsParserparses the JSON →PostgresPlanRepository.uploadSimulationDatasetwrites the dataset, profiles, segments, activity spans, topics, and events in a single transaction → returns the newsimulationDatasetId.Reviewers should be aware that the Javalin max request size was raised to 256 MB globally (not scoped to this route) to support large uploads.
Changes:
uploadSimulationDatasetmutation andSimulationResultsInputscalar type; adds missing custom scalar declarationsaerie_adminanduserpermissionsSimulationResultsWriterJSON format (DOY timestamps, real/discrete profiles with named segments, simulated activities, optional topics/events)POST /uploadSimulationDatasetroute; parses Hasura action envelope, checksinsert_ext_datasetpermission, delegates to plan servicehasuraUploadSimulationDatasetActionPparser for the action input envelopeserializeCreatedSimulationDatasetIdmethod returning a JSON object withsimulationDatasetIdinsert_ext_datasetpermission andUploadSimulationDatasetInputrecorduploadSimulationDatasetto the service interface and implementationuploadSimulationDatasetto the repository interface; PostgreSQL implementation writes the full dataset in a single transactionVerification
UploadSimulationDatasetParserTestcovering real profiles, discrete profiles, activities, simulation arguments, topics, and events (including absence defaults). Run locally and pass.UploadSimulationDatasetTestscovering: successful upload with empty profiles, discrete profiles, real profiles, and mixed profiles; invalid plan ID (404); missing required field (400); unauthorized user (403).Documentation
No existing documentation covers this endpoint. The
SimulationResultsWriterformat it accepts is defined in theaerie-clirepository (see NASA-AMMOS/aerie-cli#187).Future work
uploadSimulationDatasetroute only rather than applying it globally