Skip to content

Test custom job results #512

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
EmileSonneveld wants to merge 8 commits into
base: master
Choose a base branch
from
Open

Test custom job results #512

Show file tree
Hide file tree
Changes from 5 commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
30e3269
Add test_get_job_results_100_server_test
EmileSonneveld Jun 11, 2026
a9063cd
test_get_job_results_100_server_test
EmileSonneveld Jun 11, 2026
009d66f
test_get_job_results_100_from_cwl test_get_job_results_110_from_cwl
EmileSonneveld Jun 11, 2026
9bdd0a3
Fix for job_id
EmileSonneveld Jun 11, 2026
fdcd45a
j-job_metadata_from_cwl
EmileSonneveld Jun 11, 2026
b6da14b
Fix for if collection links are used. Added example_stac_catalog_11 s…
EmileSonneveld Jun 12, 2026
e7fc9a7
Cleaner names in stac examle. Checked with `stac-check example_stac_c…
EmileSonneveld Jun 15, 2026
6e93707
Respond to review
EmileSonneveld Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion openeo_driver/views.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1850,7 +1850,7 @@ def _asset_object(
if filename.endswith(".model"):
# Machine learning models.
return result_dict
bands = asset_metadata.get("bands")
bands = asset_metadata.get("bands") or asset_metadata.get("eo:bands")

@EmileSonneveld EmileSonneveld Jun 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@soxofaan An issue for CWL here was that the results can be stac1.0, while the code assumed the was 1.1.
Would wrapping this or with TREAT_JOB_RESULTS_V100_LIKE_V110 make more sense?

@EmileSonneveld EmileSonneveld Jun 12, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For example with

        bands = asset_metadata.get("bands")
        TREAT_JOB_RESULTS_V100_LIKE_V110 = smart_bool(os.environ.get("TREAT_JOB_RESULTS_V100_LIKE_V110", "0"))
        if not bands and stac11 and TREAT_JOB_RESULTS_V100_LIKE_V110:
            bands = asset_metadata.get("eo:bands")

@soxofaan soxofaan Jun 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

as far as I understand, TREAT_JOB_RESULTS_V100_LIKE_V110 is a temporary toggle/hack, which I would avoid here

I would just use bands = asset_metadata.get("bands") or asset_metadata.get("eo:bands")

There is a possible problem however that "bands" and "eo:bands" have subtly different sub-structure: e.g. in "bands" you could find "center_wavelength", while in "eo:bands" you would just have "center_wavelength" (without prefix). But I guess the most (only?) important field is just "name", which is the same in both.

@EmileSonneveld EmileSonneveld Jun 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed the conditional, and now only keeps name property
bands = [{"name": b["name"]} for b in eo_bands]


if bands:
# TODO: #298 this is a quick stop-gap solution for lack of clear API
Expand Down
308 changes: 308 additions & 0 deletions tests/data/job_metadata_from_cwl.json
View file
Open in desktop

@soxofaan soxofaan Jun 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what is this document exactly? It doens't have an id or type as far as I see

@EmileSonneveld EmileSonneveld Jun 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is the batch job metadata that gets stored in the results folder. It is used to create stac content with that is returned to the client.

Original file line number Diff line number Diff line change
@@ -0,0 +1,308 @@
{
"geometry": null,
"bbox": null,
"area": null,
"start_datetime": "2023-06-01T00:00:00Z",
"end_datetime": "2023-06-06T00:00:00Z",
"links": [
{
"href": ".../collection.json",

@soxofaan soxofaan Jun 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it intentional to have leading tripple dots here (and in some other places)?

@EmileSonneveld EmileSonneveld Jun 15, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed this and other examples to .../non/existing/example/path/collection.json

"rel": "original",
"title": "Link to original STAC catalog.",
"type": "application/json"
}
],
"items": [
{
"type": "Feature",
"stac_version": "1.0.0",
"id": "openEO_2023-06-01Z.tif",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[5.069685009564396, 51.21481793134414],
[5.069685009564396, 51.22021524164682],
[5.080364860947119, 51.22021524164682],
[5.080364860947119, 51.21481793134414],
[5.069685009564396, 51.21481793134414]
]
]
},
"bbox": [
5.069685009564396,
51.21481793134414,
5.080364860947119,
51.22021524164682
],
"properties": {
"testing_custom_property": "https://github.com/cloudinsar/s1-workflows/issues/70",
"datetime": "2023-06-01T00:00:00Z"
},
"links": [],
"assets": {
"openEO_2023-06-01Z.tif": {
"href": ".../openEO_2023-06-01Z.tif",
"roles": ["data"],
"type": "image/tiff; application=geotiff",
"eo:bands": [
{
"name": "B04",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
},
{
"name": "B03",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
},
{
"name": "B02",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
}
],
"raster:bands": [
{
"name": "B04",
"statistics": {
"minimum": 180,
"maximum": 1766,
"mean": 401.07085498347,
"stddev": 162.43588375276,
"valid_percent": 100
}
},
{
"name": "B03",
"statistics": {
"minimum": 278,
"maximum": 1336,
"mean": 598.11572980633,
"stddev": 133.74416960473,
"valid_percent": 100
}
},
{
"name": "B02",
"statistics": {
"minimum": 216,
"maximum": 1072,
"mean": 368.17264997638,
"stddev": 99.2670146459,
"valid_percent": 100
}
}
]
}
}
},
{
"type": "Feature",
"stac_version": "1.0.0",
"id": "openEO_2023-06-04Z.tif",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[5.069685009564396, 51.21481793134414],
[5.069685009564396, 51.22021524164682],
[5.080364860947119, 51.22021524164682],
[5.080364860947119, 51.21481793134414],
[5.069685009564396, 51.21481793134414]
]
]
},
"bbox": [
5.069685009564396,
51.21481793134414,
5.080364860947119,
51.22021524164682
],
"properties": {"datetime": "2023-06-04T00:00:00Z"},
"links": [],
"assets": {
"openEO_2023-06-04Z.tif": {
"href": ".../openEO_2023-06-04Z.tif",
"roles": ["data"],
"type": "image/tiff; application=geotiff",
"eo:bands": [
{
"name": "B04",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
},
{
"name": "B03",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
},
{
"name": "B02",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
}
],
"raster:bands": [
{
"name": "B04",
"statistics": {
"minimum": 174,
"maximum": 1628,
"mean": 360.10746339159,
"stddev": 160.47737013663,
"valid_percent": 100
}
},
{
"name": "B03",
"statistics": {
"minimum": 263,
"maximum": 1358,
"mean": 546.93197921587,
"stddev": 129.15120454283,
"valid_percent": 100
}
},
{
"name": "B02",
"statistics": {
"minimum": 200,
"maximum": 1116,
"mean": 344.86820973075,
"stddev": 95.975778884601,
"valid_percent": 100
}
}
]
}
}
},
{
"type": "Feature",
"stac_version": "1.0.0",
"id": "openEO_2023-06-06Z.tif",
"geometry": {
"type": "Polygon",
"coordinates": [
[
[5.069685009564396, 51.21481793134414],
[5.069685009564396, 51.22021524164682],
[5.080364860947119, 51.22021524164682],
[5.080364860947119, 51.21481793134414],
[5.069685009564396, 51.21481793134414]
]
]
},
"bbox": [
5.069685009564396,
51.21481793134414,
5.080364860947119,
51.22021524164682
],
"properties": {"datetime": "2023-06-06T00:00:00Z"},
"links": [],
"assets": {
"openEO_2023-06-06Z.tif": {
"href": ".../openEO_2023-06-06Z.tif",
"roles": ["data"],
"type": "image/tiff; application=geotiff",
"eo:bands": [
{
"name": "B04",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
},
{
"name": "B03",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
},
{
"name": "B02",
"common_name": null,
"wavelength_um": null,
"aliases": null,
"gsd": null
}
],
"raster:bands": [
{
"name": "B04",
"statistics": {
"minimum": 171,
"maximum": 1774,
"mean": 380.50094473311,
"stddev": 176.19790726983,
"valid_percent": 100
}
},
{
"name": "B03",
"statistics": {
"minimum": 282,
"maximum": 1470,
"mean": 592.27586206896,
"stddev": 138.20284171799,
"valid_percent": 100
}
},
{
"name": "B02",
"statistics": {
"minimum": 270,
"maximum": 1190,
"mean": 412.22508266415,
"stddev": 103.22080164099,
"valid_percent": 100
}
}
]
}
}
}
],
"epsg": null,
"instruments": [],
"processing:facility": "VITO - SPARK",
"processing:software": "openeo-geotrellis-0.72.0a12",
"unique_process_ids": ["run_cwl_to_stac"],
"providers": [
{
"name": "VITO",
"description": "This data was processed on an openEO backend maintained by VITO.",
"roles": ["processor"],
"processing:facility": "openEO Geotrellis backend",
"processing:software": {"Geotrellis backend": "0.72.0a12"},
"processing:expression": {
"format": "openeo",
"expression": {
"runcwltostac1": {
"process_id": "run_cwl_to_stac",
"arguments": {
"context": {},
"cwl": "https://raw.githubusercontent.com/Open-EO/openeo-geotrellis-kubernetes/master/openeo-geopyspark-k8s-custom-processes/src/openeo_geopyspark_k8s_custom_processes/cwl/dummy_stac.cwl"
},
"result": true
}
}
}
}
]
}
Loading