GDAL Support for creating dataframes from geospatial files and data

[phayes]

I've created a new crate called "geopolars_gdal" here: https://github.com/phayes/geopolars_gdal

It is not yet published.

Is there any desire to include this as part of geopolars itself (under a feature flag?), or should I publish it as a standalone crate?

[kylebarron]

This is similar in spirit to #48, though we can make this thread specific to a Rust implementation.

My ideal way to implement this would be to write a PR for this issue in the gdal crate georust/gdal#280. Essentially in GDAL 3.6+ GDAL has implemented the transpose to Arrow, so we don't need to do any sort of loop on our side. Any GDAL format can be exposed as an Arrow table that we can read without a copy. See also the original RFC https://gdal.org/development/rfc/rfc86_column_oriented_api.html

[phayes]

Thanks @kylebarron,

It makes sense to use GDAL's internal transformation machinery. Let me see what I can do to use that.

My code for doing it in rust might still be useful as a backup for when the local GDAL lib doesn't have arrow support.

[kylebarron]

for when the local GDAL lib doesn't have arrow support

Since geopolars is all-around bleeding edge, and OGR's Arrow API is coming in GDAL 3.6, it would be nice if we bring in only the super clean GDAL Arrow API into geopolars. By the time geopolars is more stable, ideally GDAL 3.6+ will be widespread

[metasim]

I'm excited about the prospects here. Let me know if I can help on the GDAL side. :-)

[kylebarron]

Me too @metasim! I'd really love to see georust/gdal#280 implemented. It looks like GDAL is currently releasing release candidates for 3.6.0 (and based on the release cadence a November release would be expected), so georust/gdal might be able to update to 3.6 pretty soon.

I don't think I'll have bandwidth to work on that myself before 2023, but would be more than happy to discuss it. I would assume we could roughly follow the pyogrio implementation

[kylebarron]

GDAL 3.6 was released today, so the stage is set to make progress on this issue!

[jorisvandenbossche]

I would assume we could roughly follow the pyogrio implementation

Related to pyogrio, I am trying to finalize that work and get a new pyogrio release out. One thing I was wondering: I am now only exposing it through the read_dataframe (which gives the data converted to (geo)pandas.(Geo)DataFrame) or the read (giving a pyarrow.Table) function, but would it be interesting for geopolars to also expose raw access to the C stream data (pointer)? Then pyogrio would still handle the interaction with GDAL (and packaging etc), but you can import that data even without a dependency on pyarrow.

(I don't know if it would still be useful to have one of pyogrio/fiona to rely on (instead of georust/gdal), and whether you care about potentially not requiring pyarrow (assuming that polars / rust' arrow can import from c data interface))

[kylebarron]

I expect we'll use pyogrio for the python geopolars bindings and georust/gdal for the rust geopolars bindings. That means we won't need to do any gdal packaging for python ourselves.

potentially not requiring pyarrow

It's an interesting thought. The only time I can think of when I wouldn't want to use pyarrow for IO would be in constrained environments like Lambda where the pyarrow package is too large. I'm not sure how much effort to put in to supporting this.

So it might be interesting for pyogrio to expose raw access, but if it's difficult/time intensive don't do it just for me