Gaia Epoch Ingest JSON Files#683
Conversation
Current WIP Gaia Epoch IGN
ISSUE SIMPLE-AstroDB#667 Attempting to ingest all 1265 rows with epochs of the Gaia DR3 table, which was successful for all 1265 rows. No errors occurred.
Attempted to ingest 1265 sources and successfully ingested 1265 sources. Modified the ingest script in order to prevent null overwrites in json files where data already existed.
…MPLE-db into Gaia-Epoch-Ingest
|
Exciting!!! Next step is to modify the tests. |
Updated the json files to keep original references and add GaiaDR3 to other_references. Updated conftest.py to fix the error with windows file in use error. Updated the ingest script for the other_references.
Using build db from json instead of load astrodb now.
|
Great! Next step is to add some tests which verify the modifications that you made. Focus on the 1265 epochs. The test should go at the bottom of this module: https://github.com/SIMPLE-AstroDB/SIMPLE-db/blob/main/tests/test_data_astrometry.py. Reading the other tests in that module should give you inspiration for how to write a new test that verifies the number of sources with and without an epoch. |
|
I also have to say, having the 587 other modified files with the boolean fix is very distracting. Is it possible to open a new PR that only does the epoch ingest. Save the boolean fix for it's own PR? |
|
In order to do that, you'd basically start over and re-run your script. Copy your script somewhere temporary (like your desktop), make a new fresh branch from an updated main, move your script into the new branch, modify the script to not touch the booleans (but save this code for later), run the script and save the JSONs, open a new PR with only 1265 modified JSON files. |
2 participants
Short description: Adds the 1265 json files from the successful Gaia ingest. The other 597 modified json files had their adopted values set to booleans instead of 0/1s. The ingest script was edited to be less destructive, not setting values to None anymore.
Link to relevant issue: Closes #667