What is Spatial Intelligence? Definition, AI, Earth Scale

Spatial intelligence is the ability to understand and act in three-dimensional space: where things are, how they relate, how they move and how they change. Fei-Fei Li, the Stanford computer scientist who led ImageNet, calls it AI's next frontier. Her company World Labs builds models that generate 3D worlds, and in September 2026 the chipmaker AMD agreed to acquire it. The same ability at Earth scale needs measured data about real places, and that data is uneven.

Satellites photograph Nairobi as often as San Francisco, but open map data records the height of almost every building in one city and almost none in the other. In Overture Maps data read in November 2026, 96.2% of San Francisco's 162,538 buildings carry a height, against 0.13% of Nairobi's 731,946. In 2025, Sentinel-2 satellites took images over Nairobi on 100 days and over San Francisco on 86. The height count covers building records in open map data. It leaves out private data and modeled height maps such as Google's Open Buildings 2.5D Temporal dataset.

The word also has a human meaning. In psychology, spatial intelligence is a person's skill at picturing and turning shapes in the mind, one of the intelligences in Howard Gardner's 1983 theory. Apart from the questions at the end, this article is about machines.

Key takeaways

• Spatial intelligence is the ability to perceive, reason about and act in 3D space. Stanford's Institute for Human-Centered Artificial Intelligence and Fei-Fei Li use the term for AI that understands the physical world.
• Li argues for world models, which predict how a scene changes after an action. She writes that they learn from images and video and must create scenes, accept many kinds of input and respond to actions.
• At Earth scale, spatial intelligence needs measured maps and repeated satellite images of real places.
• Open map data holds a height for 96.2% of San Francisco's 162,538 buildings and for 0.13% of Nairobi's 731,946 (Overture Maps, read November 2026). The count leaves out private data and modeled height maps.
• Sentinel-2 satellite images show where land changed. In Nairobi, 173 km2, a quarter of the city, sits in hexagons where at least a tenth of the land became roofs or pavement between 2020 and 2025. Those hexagons are the places to fly first for sharper images or lidar (laser scans from aircraft). With a stricter or looser threshold, the flagged area ranges from 120 to 229 km2, and a 10 m pixel cannot show a single small house. In Nairobi National Park, where little was built, the same rule flagged 2 of 911 hexagons.
• How Wherobots measured it: one structured query language (SQL) query counted 894,484 buildings from Overture Maps open data inside two city boundaries in under a minute. One batch job in Wherobots compared 32 Sentinel-2 images from 2020 and 2025 in about 8 minutes.

What is spatial intelligence?

Stanford University's Institute for Human-Centered Artificial Intelligence (HAI) publishes a definition of spatial intelligence. It reads: "Spatial Intelligence is the ability to understand and reason about the three-dimensional physical world, including how objects relate to each other in space, how they move, and how they interact." In AI, the definition continues, it means systems that "perceive, interpret, and navigate physical environments." Stanford HAI names robotics, autonomous vehicles and augmented reality as the uses.

Fei-Fei Li sets out the AI meaning in her essay From Words to Worlds: Spatial Intelligence is AI's Next Frontier (November 10, 2025). She writes that "Spatial Intelligence is the scaffolding upon which our cognition is built." Without spatial intelligence, she writes, AI "is disconnected from the physical reality it seeks to understand." Her example from history is Eratosthenes. He measured the angle of a shadow in Alexandria to calculate the Earth's circumference.

With Jia Deng and colleagues, Li built ImageNet (2009), a large database of labeled photos for computer vision research. Her essay treats spatial intelligence as the next step after seeing: from naming what is in a picture to understanding the 3D world the picture comes from.

The term has an older AI use too. Andrew Davison's FutureMapping: The Computational Structure of Spatial AI Systems (2018) used "spatial AI" for robots and devices that build and update a map of their surroundings as they move.

How spatial intelligence works in AI

Li writes that spatial intelligence needs world models, models that predict what a scene looks like and how it changes after an action. She lists three capabilities they need:

1. Generative. A world model creates worlds that stay consistent in geometry and physics, "whether representing real or virtual spaces."
2. Multimodal. It takes images, video, depth maps, text, gestures or actions as input.
3. Interactive. Given an action, it outputs the next state of the world.

David Ha and Jürgen Schmidhuber's World Models (2018) trained an agent inside a learned model of a video game. Neural radiance fields (NeRF) train a neural network on a few dozen photos of a scene so it can draw the scene from new angles. Ben Mildenhall and colleagues introduced them in NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis (2020). Bernhard Kerbl and colleagues' 3D Gaussian Splatting (2023) stores a scene as millions of small colored blobs and draws it in real time.

Products now build on that work. World Labs' Marble creates 3D worlds from text, images, video or rough 3D layouts and exports them as Gaussian splats, 3D surface models (meshes) or video. Google DeepMind's Genie 3 (August 2025) generates worlds a person can move through in real time. NVIDIA's Cosmos (2025) offers world models whose trained parameters anyone can download (open weights), for training robots and vehicles. The physical AI article covers how robots and vehicles use them.

The field is drawing large investment. On September 28, 2026, the chipmaker AMD agreed to acquire World Labs in an all-stock deal valued at about $8.2 billion, CNBC reported the same day. AMD said it expects the deal to close by the end of 2026, subject to regulatory approval. AMD also said Li will join it as chief scientist.

What data spatial intelligence needs

"Internet-scale collections of images and videos," she writes, are "abundant, accessible training material." The challenge she names is pulling 3D structure out of those flat frames. She adds depth sensors, touch and synthetic data as supplements.

Spatial intelligence at Earth scale starts from measurements tied to coordinates on the planet. These include satellite images taken on a fixed schedule, elevation models, and maps of buildings, roads and land cover. Earth observation data covers those sources. Waldo Tobler's first law of geography (1970), "everything is related to everything else, but near things are more related than distant things," is the working rule for joining them.

How the two kinds of spatial intelligence compare:

• Question. In a scene: what is around me, and what happens if I act? At Earth scale: where is everything, and what changed?
• Main input. In a scene: photos, video, depth sensors and simulation. At Earth scale: satellite and aerial images, lidar and maps.
• Unit. In a scene: a room, a street or a generated world. At Earth scale: every building, road and field on a coordinate grid.
• How truth is checked. In a scene: the 3D scene stays consistent. At Earth scale: comparison with surveys and official records.
• How often it updates. In a scene: when new photos or video arrive. At Earth scale: satellites revisit every few days, and maps and heights update far less often.
• Typical output. In a scene: a 3D scene or a robot action. At Earth scale: a map layer, a count or a list of places that changed.

Scene models and Earth-scale maps meet in world models of real places. A robot that delivers in a real city, or a digital twin a utility uses to plan repairs, needs a model that matches the actual streets and buildings. Geospatial AI models such as NASA and IBM's Prithvi and Google DeepMind's AlphaEarth Foundations already learn from years of satellite images of the whole planet. Li herself points this way: her essay says spatially intelligent systems can help climate science by "integrating multi-dimensional simulation with real-world data collection."

Spatial intelligence in Wherobots: San Francisco and Nairobi

The question. A world model of a real city needs the city's 3D shape. Does open data supply it everywhere, and can satellites show where it has gone out of date?

What would support the claim that the data is uneven. Two big cities photographed by satellite on a similar number of days, where one has heights for far more of its buildings than the other.

The places. San Francisco, where World Labs is based, and Nairobi, Kenya's capital, where Google's and Microsoft's models traced 27.6% of the buildings in open map data. Both are read with the same data, the same rules and the same years.

Buildings and heights

Overture Maps Foundation's buildings data merges OpenStreetMap, building outlines traced from imagery by Microsoft and Google machine learning models, and other sources. It carries a height when a source supplies one. One query in WherobotsDB, the spatial database in Wherobots Cloud, counted every Overture building whose center lies inside each city's land boundary:

WITH bnd AS (
  SELECT 'San Francisco' AS city, geometry AS g
  FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE id = '273bc9a0-96a1-402c-992c-84f5c2f212cb'
  UNION ALL
  SELECT 'Nairobi', geometry
  FROM wherobots_open_data.overture_maps_foundation.divisions_division_area
  WHERE id = '288a59f2-163f-48f2-a942-1d5096088685'
)
SELECT bnd.city, b.sources[0].dataset AS source,
       COUNT(*) AS buildings,
       SUM(CASE WHEN b.height IS NOT NULL THEN 1 ELSE 0 END) AS with_height
FROM wherobots_open_data.overture_maps_foundation.buildings_building b
JOIN bnd ON ST_Intersects(bnd.g, ST_Centroid(b.geometry))
-- Bounding-box filter on each city's extent: lets the engine skip files outside the two cities
WHERE (b.bbox.xmin > -122.53 AND b.bbox.xmax < -122.32 AND b.bbox.ymin > 37.70 AND b.bbox.ymax < 37.87)
   OR (b.bbox.xmin > 36.66 AND b.bbox.xmax < 37.11 AND b.bbox.ymin > -1.45 AND b.bbox.ymax < -1.16)
GROUP BY ROLLUP(bnd.city, b.sources[0].dataset)

The result:

• San Francisco: 162,538 buildings, 156,434 with a height (96.2%). Almost all outlines come from OpenStreetMap; 1,339 were traced by Microsoft's models.
• Nairobi: 731,946 buildings, 972 with a height (0.13%). Counting buildings with a floor count as well raises the share to 0.57%.
• Traced from imagery: in Nairobi, 201,901 outlines (27.6%) were traced from satellite images by Google's and Microsoft's models. None of them has a height.

San Francisco's heights trace back to measured sources. The city's own building footprints dataset comes from a 2010 3D model of the city. Overture adds heights from US Geological Survey lidar (laser scans flown from aircraft) for 12,913 buildings. That city dataset lists 177,023 footprints, 9% more than Overture's 162,538, because the city splits buildings at property lines.

Satellite coverage and change

Sentinel-2 is the European Union's Copernicus mission of twin imaging satellites. Together they photograph land every five days at the equator, and each pixel covers 10 m by 10 m of ground (the European Space Agency's Sentinel-2 mission page). Matthias Drusch and colleagues describe the design in Sentinel-2: ESA's Optical High-Resolution Mission for GMES Operational Services (2012). Each image covers a 110 km square and reports its share of cloud. On 14 days in 2025 in each city, every image over the city reported under 10% cloud.

At 10 m detail, Sentinel-2 cannot measure a building's height. It can show where land changed, which tells a mapping team where to send aircraft for sharper images or lidar. The batch job compared a dry-season image of each city in 2020 with one in 2025:

1. Composite. For each year, it averaged the clear pixels of the four least cloudy dates in the same months: January and February in Nairobi, June to September in San Francisco.
2. Indices. It computed two standard ratios of the satellite's color bands for each 10 m pixel. The normalized difference vegetation index (NDVI) is high over plants; the NDVI article explains it. The normalized difference built-up index (NDBI), from Yong Zha and colleagues' 2003 paper on the built-up index, is high over roofs and pavement.
3. Built-up or not. In each year, a pixel counts as built-up when its NDVI is under 0.25 and its NDBI is over 0. New built-up land is a pixel that was not built-up in 2020 and was in 2025. Comparing two classified maps this way is one of the methods Ashbindu Singh compared in his 1989 review of change detection.
4. List of places to photograph again. The job grouped pixels into hexagons of about 0.1 km2 from the H3 grid system. It flagged a hexagon when at least 20 pixels became built-up and they made up at least 10% of its clear pixels.

A first version of the rule flagged pixels where NDVI fell and NDBI rose. It marked 75% of Nairobi as changed because the 2020 season was greener everywhere, so we replaced it with the rule in step 3 before looking at that rule's results.

The result:

• Nairobi: 56.5 km2 of 10 m pixels became built-up between 2020 and 2025. Over the same years, 14.6 km2 changed from built-up to not built-up, which gives a rough size of the method's errors. The job flagged 1,546 hexagons covering 173.3 km2, a quarter of the city's 695.5 km2 of land.
• Buildings in the flagged area: those hexagons hold 265,735 of Nairobi's buildings in Overture Maps. 378 of them have a height.
• San Francisco: 8.1 km2 of pixels became built-up and 7.3 km2 changed back, so the city shows almost no net change. The job flagged 136 hexagons covering 11.3 km2. Many sit along the shore, where sand and water edges shift between images.
• Control parks: in Nairobi National Park, where little was built, the rule flagged 2 of 911 hexagons. In Golden Gate Park it flagged 1 of 36.

The flagged area depends on where the line between built-up and not built-up is drawn. With a stricter line (NDVI under 0.20 and NDBI over 0.05), Nairobi's flagged area is 120.1 km2. With a looser one (NDVI under 0.30 and NDBI over -0.05), it is 228.7 km2.

Nairobi's 2020 images were also greener than its 2025 images: the mean vegetation index fell from 0.50 to 0.42. Drier land can read as built-up, and the low count in the national park suggests the effect is small. The batch job took about 8 minutes for both cities.

Verdict. The data is uneven, as the claim said. Satellites photographed both cities on a similar number of days, but open map data holds heights for one city and almost none of the other. Sentinel-2 also narrowed where to measure next in Nairobi: by this rule, three quarters of the city was not flagged for new aircraft images or lidar. In San Francisco the rule found no net change, so most of its flags there are likely false changes to check.

What the run leaves out. Overture's height field counts open data only. A 10 m pixel cannot show a single small house, a new floor on an old building, or change under tree cover. A flagged hexagon says where to collect new images or lidar, and those finer measurements show what changed.

Spatial intelligence examples

• Robots in homes and warehouses. A robot that grasps a cup it has never seen uses depth and shape. Li names robotics as one of the main uses.
• Generated 3D worlds. World Labs' Marble and Google DeepMind's Genie 3 turn a prompt or photo into a 3D world a person can explore.
• Self-driving cars. A car places nearby objects in 3D and predicts their paths. Stanford HAI lists autonomous vehicles among the uses of spatial intelligence.
• Building outlines from imagery. Google's Open Buildings and Microsoft's ML Buildings trace building footprints from satellite images with machine learning models. In Nairobi they supply 27.6% of Overture's outlines.
• Change detection. Change detection compares images of the same place over time to find new roads, buildings or cleared land.
• Digital twins of cities. A city model that pairs footprints with heights from measured sources, as San Francisco's does, is the base layer for a world model of a real place.

Spatial intelligence research and what changes at scale

From seeing to mapping

The AI line runs from image recognition (ImageNet, 2009) to 3D scenes rebuilt from photos (NeRF, 2020; Gaussian splatting, 2023). It continues to world models that generate and respond (Genie 3, Marble, Cosmos). The mapping line runs in parallel. Michael Goodchild's Citizens as sensors (2007) described how volunteers, as in OpenStreetMap, became a main source of map data. Krzysztof Janowicz and colleagues' GeoAI (2020) argued that AI for geography must handle place, scale and the first law of geography.

What scale changes

A scene model handles one room or street. Earth-scale work joins hundreds of millions of buildings to boundaries, rasters and other layers, and repeats the join as new images arrive. That needs spatial joins, which match each building to the boundary or pixel it falls in, split across many computers. Jia Yu, Jinxuan Wu and Mohamed Sarwat's GeoSpark (2015), now Apache Sedona, spreads spatial joins across a computer cluster; Spatial data management in Apache Spark: the GeoSpark perspective and beyond (2019) describes the design. WherobotsDB builds on it and reads satellite images in place from cloud storage, as the out-of-database raster post explains.

Open problems

• Heights where nobody flew lidar. In Nairobi, 99.87% of the buildings in open map data have no height. Estimating heights from satellite images is one route. Google's Open Buildings 2.5D Temporal dataset does this from Sentinel-2 images across Africa, Latin America and South and Southeast Asia, and each estimate needs checking against measured buildings.
• Training world models on real places. A world model of a real city can be checked against its measured buildings only where those measurements exist.
• Keeping models current. A world model trained on 2020 images shows 2020. Change detection says where it went stale.

Conclusions

1. Spatial intelligence in AI means perceiving and acting in 3D space. Stanford HAI and Fei-Fei Li define it that way, and world models are the main way to build it.
2. At Earth scale, spatial intelligence depends on measured data that is uneven. Open map data holds a height for 96.2% of San Francisco's buildings and 0.13% of Nairobi's, with satellite coverage about equal.
3. Satellite images repeat every few days; heights need aircraft or lidar. If you need a city in 3D, use Sentinel-2 to find where it changed, and send aircraft for sharper images or lidar only there.
4. Sentinel-2 can shrink a re-mapping job where a city is growing. In Nairobi it cut the area to photograph again to 173 km2 of 696. In San Francisco it found no net change, so in San Francisco, choosing places to photograph again needs a sharper sensor than Sentinel-2. The flagged area moves with the built-up threshold, from 120 to 229 km2 in Nairobi.
5. A world model of a real place inherits its map's gaps. If you train or test one on a city, check the share of buildings with a measured height first. Where it is near zero, the model's 3D shape of that city is a guess.

Related terms

• World model: a model that predicts how a scene changes after an action
• Physical AI: AI that perceives, reasons about and acts in the physical world
• Geospatial AI: machine learning and AI agents applied to maps and satellite images
• Geospatial intelligence: analysis of imagery and map data to describe activity on Earth
• Context engine: the system that computes and serves the context AI models receive
• Earth observation data: satellite and aerial images of the planet's surface
• Change detection: finding what changed between images of the same place
• Building footprint: the outline of a building on a map

Read more from Wherobots

• How Wherobots builds with NVIDIA to let AI see the physical world
• RasterFlow is now available in Public Preview
• Spatial Graph RAG for the Physical World

Run spatial SQL on Overture Maps and Sentinel-2 data from your AI agent through the Wherobots Model Context Protocol (MCP) server. Start a free trial of Wherobots Cloud.