Planetary-scale answers, unlocked.
A Hands-On Guide for Working with Large-Scale Spatial Data. Learn more.
Authors
We’re excited to share that Databricks users can now use Wherobots – a product optimized for transforming data from the physical world into spatial intelligence – with Databricks Unity Catalog, bringing powerful capabilities for managing and querying spatial data in Databricks.
With Wherobots, Databricks users can now:
Customers like Dotlas, Leaf Agriculture, and Overture are realizing step-function improvements in performance, cost efficiency, scale, and innovation capability by choosing Wherobots as their spatial intelligence engine.
Wherobots Data Federation with Unity Catalog allows you to read from and write to Iceberg or Delta tables governed by Unity Catalog, using Wherobots with a Databricks service principal and OAuth or PAT tokens for authentication.
Get started or schedule a call with Wherobots today.
Spatial intelligence is the understanding of features of interest, and their relationships across space and time in multi-dimensional environments. With it, bridges are formed between digital and physical worlds. Decision making can improve, and you can create better products or services with higher returns.
In a simple form, spatial intelligence can be customer visits grouped for a location or area. But simplified formats like aggregations inherently lack precision, and you need to make tradeoffs between cost and resolution, all of which limit their usefulness. These aggregations are typically grouped by cells in a grid (like H3) and most commonly used to create visualizations. While interesting to look at, visualizations can be used to support a decision, intuition, or analysis, but they are generally less actionable because precision or other context is missing. In a more complete form, it can be a forecast, or a composite risk, opportunity, or value score associated with potentially millions of specific assets or features across a continent derived from any valuable combination of IoT, location, building, weather, road network, terrain, crops, parcel, aerial imagery, BI, or mobility datasets. By processing perspectives about features of interest from multiple valued aspects, the complete picture forms, which becomes highly actionable, and extremely useful intelligence. But even the most popular data platforms still don’t make it easy to create.
There are many cloud data engines and warehouses that support the simple form described above, including BigQuery, Snowflake, and now Databricks with its Spatial SQL support in preview. However they still lack feature and data type support, reasonable query price-performance at scale, and the solution expertise you may need to create a complete form of spatial intelligence. Here’s why.
Shaped by demand, most data platforms were first designed to handle the structured data exhaust from the web and devices connected to it, not data collected from or about the physical world.
Physical world data is inherently complex, unstructured, and doesn’t fit neatly into key-based joins. It takes a specialized compute engine to make it easy to create solutions from spatial data. Easy means it’s capable of fusing and transforming various spatial and non spatial data types with high accuracy, scale, performance, and low cost, while ensuring development is productive with the teams you have.
The spatial extensions and APIs for today’s big data engines and warehouses provide limited support for simple workloads. But because of design bottlenecks, missing features, and limited technical support, spatial solutions on these platforms can be expensive and difficult to build, while ideas remain far-fetched.
Ideally the solution for creating complete spatial intelligence just fits into your existing software development workflows, already supports your future needs and the data you want to utilize, is accessible to the teams you have, and just performs at the right scale — on-demand, at a cost that encourages innovation. It’s lakehouse ready, so you don’t need to move your data or utilize proprietary formats or data types to use it. You also have dedicated expertise in reach to unblock innovation.
With this capability at your fingertips, ideas can flow and innovation takes place. Your business can reach higher levels of efficiency, reducing costs, carbon footprint, and risk. You can speed up deliveries or pickups, increase the effectiveness of CAPEX, improve consistency, grow revenue, and build in ways that were thought to be impossible.
This capability is Wherobots, and it’s directly available to Databricks users via data federation with Databricks Unity Catalog.
Using Wherobots you can easily build a complete picture of what’s happened, over space and time, and integrate this intelligence into your Databricks data platform to drive growth – faster and at a lower cost than ever. Our mission is to make spatial data easy to utilize, and it’s all we are focused on. The results of our focus speak for themselves.
Wherobots makes it easy and economical to produce local to planetary scale data solutions that rely on any combination of aerial and overhead imagery, IoT and mobility data, ground truth datasets, and your own business context. And using Wherobots Data Federation with Unity Catalog, you can easily integrate the data products you build with Wherobots, into your Databricks data platform while retaining custody and governance of data.
There are two main classes of spatial data supported by Wherobots: raster and vector data. You also get the support and scale you’d expect for tabular data operations from Wherobots’ Spark compatible engine.
Raster data is typically a collection of sensor or imagery data, where each pixel in the image represents information about what is being captured, like temperature, elevation, infrared spectrum, etc. File formats include GeoTIFFs, Zarr, and NetCDF and more. Raster datasets are commonly GBs to TBs in scale.
Vector data is a collection of multi-dimensional geometries or geographies that represent the trajectory, shape, elevation, and location of things. They can be trips, points, and outlines of features like buildings or parcel and crop boundaries. File formats include GeoParquet (soon to be Parquet), Shapefiles, and GeoJSON.
Customers like Leaf Agriculture, Dotlas, Overture and others have compared the price-performance of using Wherobots for their spatial data workloads vs other managed Spark or other leading data platforms. Subscribed to the Professional Edition, they are self-reporting up to 20x better performance (5x-20x is typical) with on-demand savings reaching as high as 60%, with even higher savings from the Enterprise Edition.
Data teams are equally less limited by scaling bottlenecks. This becomes apparent after workloads finish faster on smaller WherobotsDB runtimes, and after customers realize they have significant headroom to scale well past their existing needs.
“Previously, our data volumes and processing requirements were increasing faster than we could keep up with, burdening our team with costly rebuilds. Now with Wherobots, not only can we easily scale to millions of acres, we also can rest assured that our costs won’t spiral out of control.” – G. Bailey Stockdale, CEO Leaf Agriculture
“Previously, our data volumes and processing requirements were increasing faster than we could keep up with, burdening our team with costly rebuilds. Now with Wherobots, not only can we easily scale to millions of acres, we also can rest assured that our costs won’t spiral out of control.”
– G. Bailey Stockdale, CEO Leaf Agriculture
These results are a function of specialization and a company-wide focus; WherobotsDB was built first for processing spatial data. This intentional design obviates the typical performance bottlenecks and complexities now alive in leading data platforms and warehouses, which were first designed for purposes unrelated to processing spatial data.
While the quotes from our customers matter the most, we also know how important open performance benchmarks are. But currently there are no spatial query benchmarks (or at least reputable ones), which makes query performance hard to compare across platforms without trials. It’s also hard to claim progress was made on performance when standards have not been established. We’re working on this too, and soon we will release a new open source spatial query benchmarking framework for Apache Sedona, and we will release spatial query performance results for query engines, data warehouses, and data platforms.
We already have preliminary results that compare WherobotsDB to Apache Sedona on various managed Spark engines along with query performance from engines with Spatial SQL APIs. Feel free to reach out and we can share these results when you contact us.
Wherobots is serverless and built for data security first. There’s no infrastructure to manage, although customers can also choose to run Wherobots in their AWS VPC for maximum control.
Wherobots was founded by the original creators of Apache Sedona, and Sedona is the most widely used geospatial extension for Apache Spark and in Databricks. With decades of research and experience with spatial data, open source, and cloud-scale systems, our product and team are ready to support Databricks customers’ solutions on the lakehouse.
We’re also a team leading geospatial modernization efforts in open source. Wherobots has supported GEO types for years with our Havasu table format. But rather than keeping this support in-house, we decided these types would better serve the physical world in the open so we proactively drove support for them in Apache Iceberg and Parquet.
The lakehouse gives you the ability to choose the product best suited for the job. Don’t settle for the simple form of spatial intelligence or what the default provider offers, when complete is in reach with better economics, scale, capability, performance, and support.
By integrating Wherobots into your Databricks workflows, organizations can reduce costs, improve operations, and realize new innovations powered by data from the physical world.
Go to wherobots.com/databricks to get started, schedule a meeting with experts, and enhance your spatial intelligence today.
Introducing RasterFlow: a planetary scale inference engine for Earth Intelligence
RasterFlow takes insights and embeddings from satellite and overhead imagery datasets into Apache Iceberg tables, with ease and efficiency at any scale.
The Medallion Architecture for Geospatial Data: Why Spatial Intelligence Demands a Different Approach
When most data engineers hear “medallion architecture,” they think of the traditional multi-hop layering pattern that powers countless analytics pipelines. The concept is sound: progressively refine raw data into analytical data and products. But geospatial data breaks conventional data engineering in ways that demand we rethink the entire pipeline. This isn’t about just storing location […]
Wherobots and Taylor Geospatial Engine Bring Fields-of-the-World Models to Production Scale
Agriculture depends on timely, reliable insight into what’s happening on the ground—what’s being planted, what’s being harvested, and how fields evolve over time. The Fields of The World (FTW) project was created to support exactly this mission, by building a fully open ecosystem of labeled data, software, standards and models to create a reliable global […]
share this article
Awesome that you’d like to share our articles. Where would you like to share it to:
