Everyone asking why this exists when DuckDB or PostGIS or the JVM based Sedona already exists, clearly has not run into the painful experience of working on these large geospatial workloads when the legacy options are either not viable or not an option for other reasons, which happens more often than you might expect! And the CRS awareness!!! Incredible! This is such a huge source of error when you throw folks that are doing their best, but don't have a lot of experience with GIS workloads. Very expensive queries have had to be rerun with drastic changes to the results, because someone got their CRS mixed up.
I don't get to do geospatial work as much anymore, but I would have killed for this just a year ago.
Somehow I dont see this applicable for 90% of all current spatial needs, where PostGIS does just right, and same IMHO goes for DuckDB. There perhaps exists 10% of business where data is so immense you want to hit it with Rust & whatnot, but all others do just fine im Postgre.
My bet is most of actually useful spatial ST_ functions are not implemented in this one, as they are not in the DuckDB offering.
I wrote a book on PostGIS and used it for years and these single node analytical tools make sense when PostGIS performance starts to break down. For many tasks PostGIS works great, but again you are limited by the fact that your tables have to live in the DB and can only scale as much as the computing resources you have allocated.
In terms of number of functions PostGIS is still the leader, but for analytical functions (spatial relationships, distances, etc) having those in place in these systems is important. DuckDB started this but this has a spatial focused engine. You can use the two together, PostGIS for transactional processing and queries, and then SedonaDB for processing and data prep.
A combination of tools makes a lot of sense here especially as the data starts to grow.
I think this is a great perspective in my professional experience it was very common to be using multiple tools. ESRI for some things, GDAL for others, and then some hacks here and there like most complex analytical systems. Some of it vendor shenanigans but some of it is specific features.
SedonaDB can decode PROJJSON and authority:code CRSes at the moment, although the underlying representation is just a string. In this case you might want something like CZBOND:999 or
Baller references and customization. Thank you for taking the time to craft that, I really appreciate it. Looking now because that was a main requirement of mine)
Rust is a good language for performant computing in general, but especially for data projects because there are so many great OSS data libraries like DataFusion and Arrow.
SedonaDB currently supports SQL, Python, R, and Rust APIs. We can support APIs for other languages in the future. That's another nice part about Rust. There are lots of libraries to expose other language bindings to Rust projects.
While DuckDB is excellent, I've found the spatial extension still has some rough edges compared to more mature solutions like PostGIS.
1. The latitude/longitude ordering for points differs from PostGIS and most standard geospatial libraries, which creates friction due to muscle memory.
2. Anecdotal: spatial joins haven't matched PostGIS performance for similar operations, though this may vary by use case and data size.
3. The spatial extension has a backlog of long-standing GitHub issues.
Currently, lazier GeoParquet reads, a K-nearest neigbours join, Coordinate Reference System tracking, and built-in GeoPandas IO. These aren't things that DuckDB spatial can't or won't do, but they are things that DuckDB hasn't prioritized over the last year that are essential to a lot of spatial pipelines.
I thought Apache Sedona is implemented in Java/Scala for distributed runtimes like Spark and Flink. Wouldn't Rust tooling for interactive use be built atop a completely different stack?
I’ve been out of the geo loop for a while. I’m struggling to understand why I’d use this over postgis. There used to be the argument that installing extensions was painful, but now that docker exists pulling the postgis image is just as easy as normal Postgres. And RDS has supported it for a while.
PostGIS is great when your data is already in a Postgres table! SedonaDB and DuckDB are much faster when your data starts elsewhere (e.g., GeoParquet files).
> Update (August 2024): GeoPolars is blocked on Polars supporting Arrow extension types, which would allow GeoPolars to persist geometry type information and coordinate reference system (CRS) metadata. It's not feasible to create a geopolars. GeoDataFrame as a subclass of a polars. DataFrame (similar to how the geopandas. GeoDataFrame is a subclass of pandas.DataFrame) because polars explicitly does not support subclassing of core data types.
SedonaDB builds on libraries in the Rust ecosystem, like Apache DataFusion, to provide users with a nice geospatial DataFrame experience. It has functions like ST_Intersects that are common in spatial libraries, but not standard in most DataFrame implementations.
There are other good alternatives, such as GeoPandas and DuckDB Spatial. SedonaDB has Python/SQL APIs and is very fast. New features like full raster support and compatibility with lakehouse formats are coming soon!
Everyone asking why this exists when DuckDB or PostGIS or the JVM based Sedona already exists, clearly has not run into the painful experience of working on these large geospatial workloads when the legacy options are either not viable or not an option for other reasons, which happens more often than you might expect! And the CRS awareness!!! Incredible! This is such a huge source of error when you throw folks that are doing their best, but don't have a lot of experience with GIS workloads. Very expensive queries have had to be rerun with drastic changes to the results, because someone got their CRS mixed up.
I don't get to do geospatial work as much anymore, but I would have killed for this just a year ago.
well for one, it's not crashing at some larger use-cases when duckdb does. according to the graph unless I'm mis-reading
I'd like to know the details of the errors -- because it could have been as simple as running out of memory.
You can generate the dataset with the instructions in this readme: https://github.com/apache/sedona-spatialbench/tree/main
Here are the queries: https://github.com/apache/sedona-spatialbench/blob/main/prin...
They should be fairly easy to replicate!
Crashing when running out of memory is not acceptable software behavior in my opinion.
Right, but all it says is that an error was thrown.
Somehow I dont see this applicable for 90% of all current spatial needs, where PostGIS does just right, and same IMHO goes for DuckDB. There perhaps exists 10% of business where data is so immense you want to hit it with Rust & whatnot, but all others do just fine im Postgre.
My bet is most of actually useful spatial ST_ functions are not implemented in this one, as they are not in the DuckDB offering.
I wrote a book on PostGIS and used it for years and these single node analytical tools make sense when PostGIS performance starts to break down. For many tasks PostGIS works great, but again you are limited by the fact that your tables have to live in the DB and can only scale as much as the computing resources you have allocated.
In terms of number of functions PostGIS is still the leader, but for analytical functions (spatial relationships, distances, etc) having those in place in these systems is important. DuckDB started this but this has a spatial focused engine. You can use the two together, PostGIS for transactional processing and queries, and then SedonaDB for processing and data prep.
A combination of tools makes a lot of sense here especially as the data starts to grow.
I think this is a great perspective in my professional experience it was very common to be using multiple tools. ESRI for some things, GDAL for others, and then some hacks here and there like most complex analytical systems. Some of it vendor shenanigans but some of it is specific features.
Question: Does SedonaDB support custom / alternative coordinate systems?
For example, if i wanted to define a 4d region called (fish, towel, mouse, alien) and there were floats for each of fish/towel/mouse/alien?
SedonaDB can decode PROJJSON and authority:code CRSes at the moment, although the underlying representation is just a string. In this case you might want something like CZBOND:999 or
{ "type": "EngineeringCRS", "name": "Fish, Towel, Mouse", "datum": {"name": "Wet Kitty + Mouse In Peril"}, "coordinate_system": { "subtype": "Cartesian", "axis": [ {"name": "Fish", "abbreviation": "F", "direction": "east"}, {"name": "Towel", "abbreviation": "T", "direction": "north"}, {"name": "Mouse", "abbreviation": "M", "direction": "up"}, ] } }
(Subject to the limitations of PROJJSON, such as a 4D CRS having a temporal axis and a limited set of acceptable "direction" values)
Baller references and customization. Thank you for taking the time to craft that, I really appreciate it. Looking now because that was a main requirement of mine)
is written in Rust a value add ? vs say Accessible not only via Python but first party support in Ruby, C#, Javascript ?
Rust is a good language for performant computing in general, but especially for data projects because there are so many great OSS data libraries like DataFusion and Arrow.
SedonaDB currently supports SQL, Python, R, and Rust APIs. We can support APIs for other languages in the future. That's another nice part about Rust. There are lots of libraries to expose other language bindings to Rust projects.
What is the advantage over Duckdb with Spatial Extension.
While DuckDB is excellent, I've found the spatial extension still has some rough edges compared to more mature solutions like PostGIS.
1. The latitude/longitude ordering for points differs from PostGIS and most standard geospatial libraries, which creates friction due to muscle memory.
2. Anecdotal: spatial joins haven't matched PostGIS performance for similar operations, though this may vary by use case and data size.
3. The spatial extension has a backlog of long-standing GitHub issues.
Currently, lazier GeoParquet reads, a K-nearest neigbours join, Coordinate Reference System tracking, and built-in GeoPandas IO. These aren't things that DuckDB spatial can't or won't do, but they are things that DuckDB hasn't prioritized over the last year that are essential to a lot of spatial pipelines.
Interesting, but why share the Sedona name?
I thought Apache Sedona is implemented in Java/Scala for distributed runtimes like Spark and Flink. Wouldn't Rust tooling for interactive use be built atop a completely different stack?
Apache Sedona, not so well loved in the GIS space in my experience, so I don't think it's a huge issue even if it is a bit confusing.
Wait is this a sifferent.apache sedona to the spark based apache sedona GIS dataframe engine I've cone into contact with before?
Surely they're the same? Two sedona projects is one thing, but two apache sedona projects is sheer madness?
Yes, this is Apache SedonaDB and the other is just Apache Sedona (Spark)
There is another great lib built on Apache Arrow - polars dataframe, which has amazing DSL.
It comes a disappointment for me that SedonaDB hasn’t adopted a similar approach.
Apache stack provides everything needed, but for small things I would not prefer SQL exactly
I’ve been out of the geo loop for a while. I’m struggling to understand why I’d use this over postgis. There used to be the argument that installing extensions was painful, but now that docker exists pulling the postgis image is just as easy as normal Postgres. And RDS has supported it for a while.
What am I missing? The api even looks the same.
PostGIS is great when your data is already in a Postgres table! SedonaDB and DuckDB are much faster when your data starts elsewhere (e.g., GeoParquet files).
The "DuckDB is probably the most important geospatial software of the last decade" post has a nice related discussion: https://news.ycombinator.com/item?id=43881468
Oh I see. So if you have some kind of a pipeline in which you don’t need or want to load the data into a DB first. That makes total sense, thanks!
Please don't use emojis in titles. Immediately looks like written by AI.
Looks interesting for more efficient geospatial operations. Congratulations!
Whats the point of this over polars?
There is a project called GeoPolars: https://github.com/geopolars/geopolars
From the README:
> Update (August 2024): GeoPolars is blocked on Polars supporting Arrow extension types, which would allow GeoPolars to persist geometry type information and coordinate reference system (CRS) metadata. It's not feasible to create a geopolars. GeoDataFrame as a subclass of a polars. DataFrame (similar to how the geopandas. GeoDataFrame is a subclass of pandas.DataFrame) because polars explicitly does not support subclassing of core data types.
I'm working on implementing extension types in Polars. Stay tuned.
Was "written in Rust" really necessary?
What does it do besides being written in Rust?
Look, this article is absolutely excellent, and answers your questions. Please read the article before commenting this sort of thing.
As someone who has had to use geopandas a lot, having something which is up to an order of magnitude faster is a real dream come true.
SedonaDB builds on libraries in the Rust ecosystem, like Apache DataFusion, to provide users with a nice geospatial DataFrame experience. It has functions like ST_Intersects that are common in spatial libraries, but not standard in most DataFrame implementations.
There are other good alternatives, such as GeoPandas and DuckDB Spatial. SedonaDB has Python/SQL APIs and is very fast. New features like full raster support and compatibility with lakehouse formats are coming soon!