Geospatial Big Data: Why Analytics Is Hard to Scale

Teams struggle to scale geospatial analytics for five reasons. Location data is large. It comes in two forms: vector data, shapes such as buildings and roads, and raster data, grids of pixels such as satellite images. It is also recorded in many coordinate systems, the different ways of turning a place on Earth into numbers.

Spatial joins are the fourth reason. A spatial join matches rows by location, such as each building to the county it sits in. A filter or a sum splits easily across many machines. A spatial join does not. The fifth reason is upkeep: each new data source adds pipeline steps to convert it, move it into a shared coordinate system, and reconcile it with the rest.

The examples below measure each reason on open data in Wherobots: buildings, roads and satellite imagery for Washington State and the lower 48 US states.

Key takeaways

• Geospatial analytics is hard to scale for five separate reasons: data volume, mixed formats, many coordinate systems, spatial joins, and pipeline upkeep.
• Volume: Overture Maps, an open map dataset, lists 2,533,842,612 buildings in 253.2 GB and 352,054,710 road segments in 68.5 GB.
• Coordinate systems: Sentinel-2 satellite imagery of the lower 48 states sits in 10 Universal Transverse Mercator (UTM) zones. UTM splits Earth into narrow north-south strips called zones, each with its own coordinates. 43 of the 48 states span two or three zones.
• Joins: matching each building in Washington to its county returned 3,824,072 building-county pairs across 39 counties. The query plan, the list of steps the engine will run, is the same for the lower 48 states. It copies the county shapes to every machine with an R-tree, an index that finds the shapes near a location without checking them all.
• A distributed spatial engine runs one query across many machines. It reads only the files that can match and splits the work by location. The same query plan covers a county and a country.

Why do teams struggle to scale geospatial analytics workloads?

1. Data volume. A national or global layer, one map dataset such as all buildings, can hold billions of rows. A tool that runs on one computer can load only what fits in that computer's memory and disk. That is a hard limit.
2. Mixed formats. Vector data stores shapes as points, lines and polygons. Raster data stores a grid of pixel values. Each data model has its own file formats and functions. A tool built for one model needs an export step to use the other.
3. Many coordinate systems. A coordinate reference system (CRS) is the rule that ties coordinates to places on Earth. Satellite imagery, survey data and web maps often use different ones. Data must be reprojected, converted from one CRS to another, before the layers line up.
4. Spatial joins. A spatial join pairs rows whose shapes meet a condition, such as a building inside a county. There is no shared key, such as an ID column, to group the rows by. Splitting the work across machines takes spatial partitioning: grouping rows by where they sit.
5. Pipeline upkeep. Each source must be downloaded, converted, reprojected, indexed and checked for quality. The steps multiply with each source and each refresh of a source.

The causes compound. A team running a national flood study joins every building polygon to elevation rasters in a different CRS. The team repeats the work when either source updates.

What is geospatial big data?

Geospatial big data is location data that is too large, too varied or too fast-changing for one computer and the usual geographic information system (GIS) tools. GIS tools are the software analysts use to work with maps. Common examples are building footprints (outlines) for a country, years of satellite scenes, and Global Positioning System (GPS) tracks from vehicle fleets.

Reviews of the field appeared in the mid-2010s:

• Jae-Gil Lee and Minseo Kang reviewed the challenges and opportunities of geospatial big data in Big Data Research (2015).
• The International Society for Photogrammetry and Remote Sensing (ISPRS) publishes the ISPRS Journal of Photogrammetry and Remote Sensing. There, Songnian Li and colleagues published a review of geospatial big data handling theory and methods (2016).
• Chaowei Yang and colleagues covered big data and cloud computing for Earth science in the International Journal of Digital Earth (2017).

Property	Spatial data on one machine	Geospatial big data
Typical size	One county or one project's layers	A country or the planet, billions of rows
Storage	One file or one database server	Many files in object storage, split by location
Formats	Usually one data model per tool	Vector and raster together
Coordinate systems	One CRS for the project	Several, one per source or tile
Joins	Run in memory	Need spatial partitioning across machines
Typical tools	Desktop GIS, GeoPandas, one database	A distributed spatial engine

The geospatial data guide covers the data types themselves. The geospatial analysis guide covers the operations.

Data too large for a single machine

Open map data alone reaches billions of rows. Query 1 below reads the table metadata of four Overture Maps tables in Wherobots. Table metadata is the record of each table's files, with their row counts and sizes.

Query 1 returned these sizes:

• The buildings table holds 2,533,842,612 rows in 5,141 files and 253.2 GB.
• The road segments table holds 352,054,710 rows in 68.5 GB.
• Places add 81,455,423 rows.
• Boundary polygons add 1,080,667.

A tool that runs on one machine loads the data it works on into memory. Apache Sedona is an open-source engine for spatial data. Its benchmark, SpatialBench, sets its data size with a scale factor (SF), and a larger SF means more data. The SpatialBench single-node results show where the memory limit falls.

GeoPandas is a Python library for map data. On one machine with 8 CPUs and 32 GB of memory, "GeoPandas completes SF 1 but not SF 10." DuckDB, another single-machine database, "handles some join queries well but encounters scaling issues in certain cases."

Volume also sets how much a query reads. The Overture tables in Wherobots store data as many files. A bounding box is the smallest rectangle around a shape.

For each file, the table metadata records the range of bounding-box values inside it. A query for one area can therefore skip files whose boxes miss that area. The geospatial database guide reports a test of that skipping on the same buildings table.

Vector and raster data in different file formats

Analysts use vector and raster data for different questions. Vector shapes give precise boundaries, such as a land parcel or a road. Raster grids give a value for each cell, or pixel, such as elevation or one band (light range) of a satellite image. The raster vs vector guide compares them in detail.

Each data model has its own file format for cloud storage:

• GeoParquet. GeoParquet adds geometry types, column types that hold shapes, to Apache Parquet. Parquet is a file format that stores a table column by column. The GeoParquet project describes it as an incubating Open Geospatial Consortium (OGC) standard, meaning one still in development. Details are on the GeoParquet website.
• Cloud Optimized GeoTIFF (COG). A COG stores an image inside a Tagged Image File Format (TIFF) file as tiles, small square blocks of pixels. The file also holds lower-resolution copies of the image. A program reading the file uses HTTP range requests, which fetch only part of a file from a web server, to download only the part of the image it requests. The OGC COG standard sets out this layout.
• Catalogs. A SpatioTemporal Asset Catalog (STAC) item is a record for one scene, a single image capture. It lists the scene's files, its time and its footprint (the ground it covers). Details are on the STAC specification website.

A tool that reads only one data model needs the other exported first. Imagery gets converted to polygons, or polygons get drawn onto a pixel grid. Only then can the two be compared.

The Copernicus 30 m elevation table in Wherobots shows how a raster table can stay small. The table is a grid of ground heights with 30-meter pixels. It holds 3,961,635 rows in 0.1 GB.

Query 5 below sampled 1,000 rows, and all 1,000 were out-of-database (out-db) rasters. An out-db raster row stores a reference to an image file, plus its georeferencing: the numbers that tie each pixel to a place on Earth. The RS_FromPath documentation states that pixel data is not loaded until a function such as RS_Value reads it. The out-of-database architecture post describes the design.

Many coordinate systems within a single study area

Each data producer picks a coordinate system that suits its own data. The UTM system splits Earth into narrow north-south zones. Each zone is 6° of longitude wide.

The European Space Agency (ESA) Sentinel-2 product guide describes the Level-2A tiles as 100 km by 100 km images in UTM. With the overlap between neighbouring tiles, each covers about 110 km by 110 km. Level-2A images are corrected for the atmosphere. The guide says UTM "divides Earth into 60 longitudinal zones."

An analysis of a whole country therefore covers many zones. Query 2 counts them for Sentinel-2 scenes from July 2025 over the lower 48 states. The scenes fall in 995 tiles, fixed squares of the Sentinel-2 grid, across 10 UTM zones.

Each zone has a code from the European Petroleum Survey Group (EPSG), the standard numbers for coordinate systems. These EPSG codes run from 32610 to 32619. 43 of the 48 states touch tiles from two or three zones. 17 span three zones, 26 span two, and 5 fit in one.

The other layers in this article use different systems again. The Sentinel-2 footprints and the Overture boundaries are stored in longitude and latitude. The Copernicus elevation tiles are 256 by 256 pixels in EPSG:4326, also longitude and latitude.

Measuring areas or distances across these sources means choosing one projected CRS and reprojecting the others to it. A projected CRS lays the curved Earth flat, with distances in units such as meters. The coordinate reference system guide covers how to pick one. The map projection guide covers what each projection distorts.

Spatial joins have no shared key to split the work

A filter or a sum splits cleanly. Each row goes to a worker, one of the machines in a cluster. The partial results then add up.

A spatial join pairs rows from two tables by location, such as each building with the county that contains it. There is no shared ID column to group rows by. So the simplest plan tests every building against every county.

Spatial engines avoid that cost with a broadcast index join or a partitioned join. The spatial join guide describes both:

• Broadcast index join. A small table, such as 3,108 county polygons, is copied to every worker with an R-tree index. Each worker joins its share of the large table on its own machine.
• Partitioned join. Two large tables are split into the same spatial cells, squares of the map. Each worker joins one cell.

On one machine, the usual workaround is tiling by hand: cutting the map into pieces and running each piece in turn. The partitioning tax article has measurements of what that costs: loops over every tile, errors at tile edges, and reruns.

Query 3 measures how the input to the same building query grows with its area. An extent is the rectangle around a study area. Overture's building files record their bounding boxes, so the count of files that can match depends on the extent:

• King County, Washington, needs 7 files with 3,702,731 rows (0.46 GB).
• The state of Washington needs 18 files with 8,559,044 rows (1.05 GB).
• The lower 48 states need 444 files with 185,495,559 rows (22.41 GB).

Query 4 runs the join for Washington. It returned 3,824,072 building-county pairs across all 39 counties. Its plan for the lower 48 states is the same. The plan is a BroadcastIndexJoin with an R-tree on the county polygons. The box filter is pushed into the table scan, so files are skipped as the table is read.

Pipeline steps that multiply with each source

Every source in this article comes in its own format:

• Overture comes as vector tables.
• Sentinel-2 comes as STAC items that point to COG files in UTM.
• The elevation model comes as raster tiles in longitude and latitude.

Before one analysis, a team takes these steps:

1. Download each source.
2. Convert it to a shared format.
3. Reproject it to a shared CRS.
4. Tile or partition the result.
5. Build indexes.
6. Check the output.

Each refresh of a source repeats the chain. A team with a pipeline built on one machine also has to add tiling once the data outgrows that machine. Tiling brings the edge errors and reruns described in the partitioning tax article.

Two Wherobots posts describe this pattern. One covers why traditional spatial systems break down on mobility data, the location records of moving vehicles and people. The other covers a pipeline design that pairs PostGIS with Wherobots. PostGIS adds spatial functions to the PostgreSQL database.

Causes, symptoms and fixes compared

Cause	Symptom	Number measured here	What helps
Data volume	Out-of-memory errors, jobs that fail or stall	2.53 billion buildings in 253.2 GB	Distributed compute; read only the files that can match
Mixed formats	Export and convert before every comparison	Elevation as 3,961,635 out-db raster rows in 0.1 GB	One engine with vector and raster functions
Many coordinate systems	Layers that do not line up; wrong areas	10 UTM zones over the lower 48	Reproject inside the query to one CRS
Spatial joins	Joins that run on a county and stall on a country	3,824,072 pairs in Washington	Spatial partitioning, indexes, broadcast joins
Pipeline upkeep	Fragile scripts, manual reruns	Three sources, three formats, two coordinate conventions	Queries that run on the full extent with no tiling

Geospatial big data in Wherobots

Expected result. If the five causes are real, each should show up as a number in open data. The expected numbers are billions of rows, out-db rasters, several UTM zones per state, and a join whose input grows with its area.

Method. Six queries in Structured Query Language (SQL) ran in a Wherobots SQL session on October 13, 2026. WherobotsDB, the engine in Wherobots, comes from the original creators of Apache Sedona.

The queries read these tables:

• Overture Maps buildings, road segments and boundaries.
• The Sentinel-2 Level-2A catalog.
• The Copernicus 30 m elevation model.

All are tables in the Havasu catalog. Havasu tables use the Apache Iceberg table format, with column types for geometry and rasters.

Table sizes from file metadata (query 1)

An Iceberg table keeps a .files metadata table. It has one row per data file, with the file's row count and size. Reading it touches no data files.

SELECT 'buildings_building' AS tbl, COUNT(*) AS files, SUM(record_count) AS row_count,
       ROUND(SUM(file_size_in_bytes)/1e9, 1) AS gb
FROM wherobots_open_data.overture_maps_foundation.buildings_building.files
UNION ALL SELECT 'places_place', COUNT(*), SUM(record_count), ROUND(SUM(file_size_in_bytes)/1e9, 1)
FROM wherobots_open_data.overture_maps_foundation.places_place.files
UNION ALL SELECT 'transportation_segment', COUNT(*), SUM(record_count), ROUND(SUM(file_size_in_bytes)/1e9, 1)
FROM wherobots_open_data.overture_maps_foundation.transportation_segment.files
UNION ALL SELECT 'divisions_division_area', COUNT(*), SUM(record_count), ROUND(SUM(file_size_in_bytes)/1e9, 1)
FROM wherobots_open_data.overture_maps_foundation.divisions_division_area.files
UNION ALL SELECT 'copernicus_dem_glo_30m', COUNT(*), SUM(record_count), ROUND(SUM(file_size_in_bytes)/1e9, 1)
FROM wherobots_open_data.copernicus_dem.glo_30m.files

Table	Files	Rows	GB
Buildings	5,141	2,533,842,612	253.2
Road segments	1,393	352,054,710	68.5
Places	189	81,455,423	9.5
Boundaries	86	1,080,667	3.6
Copernicus elevation, 30 m	2	3,961,635	0.1

UTM zones per state (query 2)

The Sentinel-2 catalog stores each scene's EPSG code in proj:epsg. Query 2a joins July 2025 scenes to the state polygons, copied to every worker. It returns one row per state and a total.

WITH states AS (
  SELECT region, geometry FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country = 'US' AND subtype = 'region' AND is_land = true
    AND region NOT IN ('US-AK','US-HI','US-PR','US-GU','US-VI','US-AS','US-MP','US-UM')
),
scenes AS (
  SELECT mgrs_tile, `proj:epsg` AS epsg, geometry FROM wherobots_open_data.sentinel2.l2a_source_items
  WHERE year = 2025 AND month = 7
    AND bbox.xmax >= -125 AND bbox.xmin <= -66.9 AND bbox.ymax >= 24.5 AND bbox.ymin <= 49.4
),
pairs AS (
  SELECT /*+ BROADCAST(s) */ DISTINCT s.region, c.mgrs_tile, c.epsg
  FROM scenes c JOIN states s ON ST_Intersects(s.geometry, c.geometry)
)
SELECT region, COUNT(DISTINCT epsg) AS utm_zones, COUNT(DISTINCT mgrs_tile) AS tiles,
       MIN(epsg) AS min_epsg, MAX(epsg) AS max_epsg
FROM pairs GROUP BY region
UNION ALL
SELECT 'ALL', COUNT(DISTINCT epsg), COUNT(DISTINCT mgrs_tile), MIN(epsg), MAX(epsg) FROM pairs
ORDER BY region

The filter on bbox keeps every scene whose box overlaps the lower 48 box from query 3, rounded outward. The join with the state polygons sets which tiles count. The total row reads 995 tiles in 10 UTM zones, EPSG:32610 to EPSG:32619.

California touched three zones across 74 tiles, and Texas three zones across 106 tiles. The states query includes the District of Columbia, which touched one zone.

Query 2b leaves out the District of Columbia and groups the 48 states by zone count. It also counts the county polygons used in query 4's plan and the July 2025 scenes in the box.

WITH states AS (
  SELECT region, geometry FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country = 'US' AND subtype = 'region' AND is_land = true
    AND region NOT IN ('US-AK','US-HI','US-PR','US-GU','US-VI','US-AS','US-MP','US-UM','US-DC')
),
scenes AS (
  SELECT mgrs_tile, `proj:epsg` AS epsg, geometry FROM wherobots_open_data.sentinel2.l2a_source_items
  WHERE year = 2025 AND month = 7
    AND bbox.xmax >= -125 AND bbox.xmin <= -66.9 AND bbox.ymax >= 24.5 AND bbox.ymin <= 49.4
),
per_state AS (
  SELECT /*+ BROADCAST(s) */ s.region, COUNT(DISTINCT c.epsg) AS zones
  FROM scenes c JOIN states s ON ST_Intersects(s.geometry, c.geometry)
  GROUP BY s.region
)
SELECT 'states_with_' || zones || '_zones' AS k, COUNT(*) AS v FROM per_state GROUP BY zones
UNION ALL
SELECT 'lower48_counties', COUNT(*) FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country='US' AND subtype='county' AND is_land = true
    AND region NOT IN ('US-AK','US-HI','US-PR','US-GU','US-VI','US-AS','US-MP','US-UM')
UNION ALL
SELECT 'scenes_july_2025', COUNT(*) FROM scenes

Result	Value
States in three UTM zones	17
States in two UTM zones	26
States in one UTM zone	5
County polygons in the lower 48 states	3,108
July 2025 scenes in the box	20,848

Building files that can match each extent (query 3)

Each extent's box comes from the Overture boundary polygons. The file bounds come from the Iceberg metadata for the bbox column.

WITH ext AS (
  SELECT 'King County' AS extent, ST_XMin(geometry) xmin, ST_XMax(geometry) xmax, ST_YMin(geometry) ymin, ST_YMax(geometry) ymax
  FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country='US' AND region='US-WA' AND subtype='county' AND is_land = true AND names.primary = 'King County'
  UNION ALL
  SELECT 'Washington', ST_XMin(geometry), ST_XMax(geometry), ST_YMin(geometry), ST_YMax(geometry)
  FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country='US' AND region='US-WA' AND subtype='region' AND is_land = true
  UNION ALL
  SELECT 'Lower 48 states', MIN(ST_XMin(geometry)), MAX(ST_XMax(geometry)), MIN(ST_YMin(geometry)), MAX(ST_YMax(geometry))
  FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country='US' AND subtype='region' AND is_land = true
    AND region NOT IN ('US-AK','US-HI','US-PR','US-GU','US-VI','US-AS','US-MP','US-UM')
),
f AS (
  SELECT record_count, file_size_in_bytes,
    readable_metrics.`bbox.xmin`.lower_bound AS fxmin, readable_metrics.`bbox.xmax`.upper_bound AS fxmax,
    readable_metrics.`bbox.ymin`.lower_bound AS fymin, readable_metrics.`bbox.ymax`.upper_bound AS fymax
  FROM wherobots_open_data.overture_maps_foundation.buildings_building.files
)
SELECT /*+ BROADCAST(e) */ e.extent, ROUND(e.xmin,3) xmin, ROUND(e.xmax,3) xmax, ROUND(e.ymin,3) ymin, ROUND(e.ymax,3) ymax,
  COUNT(*) AS files, SUM(f.record_count) AS rows_in_files, ROUND(SUM(f.file_size_in_bytes)/1e9, 2) AS gb
FROM f JOIN ext e ON f.fxmax >= e.xmin AND f.fxmin <= e.xmax AND f.fymax >= e.ymin AND f.fymin <= e.ymax
GROUP BY e.extent, e.xmin, e.xmax, e.ymin, e.ymax ORDER BY files

Extent	Box (longitude, latitude)	Files	Rows in those files	GB
King County	-122.543 to -121.066, 47.084 to 47.780	7	3,702,731	0.46
Washington	-124.754 to -116.916, 45.544 to 49.002	18	8,559,044	1.05
Lower 48 states	-124.754 to -66.95, 24.523 to 49.384	444	185,495,559	22.41

The whole table, from query 1, is 5,141 files with 2,533,842,612 rows in 253.2 GB.

Joining buildings to counties (query 4)

The filter keeps every building whose box overlaps the Washington extent that query 3 returned. The county polygons are the small side of the join. The BROADCAST hint is an instruction in the SQL. With it, the county polygons are copied to every worker.

WITH counties AS (
  SELECT names.primary AS county, geometry FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country='US' AND region='US-WA' AND subtype='county' AND is_land = true
),
b AS (
  SELECT geometry FROM wherobots_open_data.overture_maps_foundation.buildings_building
  WHERE bbox.xmax >= -124.754 AND bbox.xmin <= -116.916 AND bbox.ymax >= 45.544 AND bbox.ymin <= 49.002
)
SELECT /*+ BROADCAST(c) */ COUNT(DISTINCT c.county) AS counties, COUNT(*) AS building_county_pairs
FROM b JOIN counties c ON ST_Intersects(c.geometry, b.geometry)

The join returned 3,824,072 pairs across 39 counties. A building that crosses a county line counts once per county.

The national version uses all 3,108 county polygons in the lower 48 states and the lower 48 box from query 3. It was planned with EXPLAIN FORMATTED, which prints the plan without running the query. It was not run, to keep the shared session free.

EXPLAIN FORMATTED
WITH counties AS (
  SELECT region, names.primary AS county, geometry FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE country='US' AND subtype='county' AND is_land = true
    AND region NOT IN ('US-AK','US-HI','US-PR','US-GU','US-VI','US-AS','US-MP','US-UM')
),
b AS (
  SELECT geometry FROM wherobots_open_data.overture_maps_foundation.buildings_building
  WHERE bbox.xmax >= -124.754 AND bbox.xmin <= -66.95 AND bbox.ymax >= 24.523 AND bbox.ymin <= 49.384
)
SELECT /*+ BROADCAST(c) */ c.region, c.county, COUNT(*) AS buildings
FROM b JOIN counties c ON ST_Intersects(c.geometry, b.geometry)
GROUP BY c.region, c.county

EXPLAIN printed a BroadcastIndexJoin over a SpatialIndex of type RTREE on the county polygons. The Iceberg scan of the buildings table carried the four bbox filters. That is the Washington plan with a larger input.

Out-of-database rasters in the elevation table (query 5)

RS_BandPath returns the file path of an out-db raster. It returns NULL, no value, for a raster stored in the row.

SELECT COUNT(*) AS sampled,
  SUM(CASE WHEN RS_BandPath(rast) IS NOT NULL THEN 1 ELSE 0 END) AS out_db,
  FIRST(RS_Width(rast)) AS width_px, FIRST(RS_Height(rast)) AS height_px, FIRST(RS_SRID(rast)) AS srid
FROM (SELECT rast FROM wherobots_open_data.copernicus_dem.glo_30m LIMIT 1000)

All 1,000 sampled rows were out-db rasters, each 256 by 256 pixels in EPSG:4326.

Limits of the test

• The UTM count covers one month of Sentinel-2 scenes. A tile with no July 2025 scene would be missing. So 995 is a floor: the true count can only be higher.
• Query 3 counts files whose boxes overlap a rectangle. The lower 48 rectangle also covers parts of Canada, Mexico and the ocean.
• The queries measure data and plans. They do not time a single-machine tool on the same data, because all compute for this article ran in Wherobots.

Verdict

Each cause showed up as a number:

• The buildings table holds 2.53 billion rows.
• The elevation model is stored as out-db rasters, and the map data is vector.
• The imagery spans 10 UTM zones, and most states span more than one.
• A building join's input grows from 7 files for a county to 444 for the lower 48 states. The plan stays a broadcast R-tree join.

Every test above ran as plain SQL in WherobotsDB, a cloud spatial engine for vector and raster data, with no tiling step.

Research history and what changes at scale

General-purpose cluster computing came first. A cluster is a group of networked machines that share one job. The Association for Computing Machinery (ACM) published these papers in its journal, Communications of the ACM:

• Jeffrey Dean and Sanjay Ghemawat described MapReduce (2008). In their model, users write a map function and a reduce function. The runtime system runs the work in parallel across a cluster.
• Matei Zaharia and colleagues described Apache Spark (2016) as one engine for batch, streaming and interactive work.

Spatial systems followed:

• The Very Large Data Bases (VLDB) Endowment publishes the Proceedings of the VLDB Endowment. There, Ablimit Aji and colleagues described Hadoop-GIS (2013), which they built on MapReduce.
• Hadoop is an open-source MapReduce system. Ahmed Eldawy and Mohamed Mokbel added a global spatial index, one index over the whole dataset, to Hadoop. They called it SpatialHadoop (International Conference on Data Engineering, 2015).
• Jia Yu, Jinxuan Wu and Mohamed Sarwat brought spatial partitioning and indexes to Spark with GeoSpark (ACM SIGSPATIAL, 2015).
• Yu, Zongsi Zhang and Sarwat described its partitioners, the methods that split data by location, in a 2019 GeoInformatica paper.

GeoSpark became Apache Sedona.

Varun Pandey and colleagues compared these systems in How good are modern spatial analytics systems? (Proceedings of the VLDB Endowment, 2018). On SpatialBench at scale factor 1000, the current WherobotsDB runs spatial queries and joins up to 2.5 times faster than the previous generation. The mean speedup is 1.9 times. The WherobotsDB performance post reports these results.

What changes at scale is where the work happens. Data moves from one file to many files in object storage, the cloud storage that holds files. Table metadata records each file's range of bounding-box values, so a query can skip files outside its area.

Rasters stay in their image files until a function reads pixels. Joins run as partitioned or broadcast plans. The analyst's SQL stays the same from a county to a country.

Conclusions

Wherobots is built to remove these five burdens, from one county to the whole planet. Taylor Geospatial worked with Wherobots to produce 8.2 billion field boundaries worldwide for the Fields of the World program. Developers building applications for the physical world are better off on one engine that handles all of their workloads from the first day. Who knows what question you will need to answer tomorrow?

1. The five causes arrive together. Each one showed up as a number in open data. Overture lists 2.53 billion buildings, the elevation model sits beside vector maps as rasters, Sentinel-2 spans 10 UTM zones, and a join's input grows from 7 files to 444.
2. One engine covers vector and raster. WherobotsDB stores both in Havasu tables and keeps raster pixels in their image files until a function reads them. No export step sits between the two data models.
3. The same SQL runs from a county to a country. WherobotsDB reads only the files whose bounding boxes overlap the query area and splits the join by location. The Washington join and the lower 48 plan used the same broadcast R-tree join. A larger area takes a larger runtime and the same SQL.
4. Choose the platform for the questions you have not asked yet. A team that starts on a single-machine tool has to move to another one when a question outgrows the machine. GeoPandas did not complete SpatialBench scale factor 10 on a 32 GB machine. Starting on an engine that already scales skips that move.
5. Two checks hold on any engine. If a study area crosses UTM zones, pick one projected CRS early: 43 of the lower 48 states span two or three. If a join covers a state or a country, check its plan before running it. The plan should narrow the candidate pairs before the exact geometry test, with a spatial index, spatial partitioning or matching grid cells. In WherobotsDB that shows as a BroadcastIndexJoin or a partitioned join over a spatial index. A nested loop over both tables tests every pair.

Related terms

• Partitioning tax: the cost of splitting spatial data into tiles by hand
• Spatial join: matching rows of two tables by location
• Raster vs vector data: the two data models a pipeline has to combine
• Coordinate reference system: picking one CRS across sources
• Geospatial database: geometry types, spatial indexes and spatial SQL
• Geospatial analysis: the operations that run at scale
• Havasu catalog: the Iceberg tables behind the test

Read more from Wherobots

• Raster processing at scale: the out-of-database architecture in WherobotsDB
• Mobility data processing at scale: why traditional spatial systems break down
• Spatial data pipeline architecture: PostGIS and Wherobots together
• WherobotsDB is 3x faster with up to 45% better price performance