What is Physical AI? Definition, Examples, How It Works

Physical AI is artificial intelligence that perceives, reasons about, and acts in the physical world. The term covers robots, autonomous vehicles, drones, and the world models used to train them. It also covers models that read satellite, aerial, and sensor data to describe the ground, from one street to a continent. Each one depends on geometry, location, motion, and physics. Language models learn from text, documents, databases, and the internet, which describe those only in part.

Key takeaways

• Physical AI is AI that perceives, reasons about, and acts in the physical world, with or without a body.
• It runs a loop: sense, model, decide, act. Developers often train it in simulation, with world models and synthetic data.
• A physical AI system needs spatial context: where things are, what surrounds them, and how the ground is shaped.
• That context comes from joining buildings, roads, places, elevation, and imagery around a location.

What is physical AI?

A physical AI system takes in sensor data and builds a model of its surroundings. It then produces an output tied to a location: a steering command, a robot grasp, a flight path, or a map of damaged roofs.

In robotics research, Aslan Miriyev and Mirko Kovač used "physical artificial intelligence" in Skills for physical artificial intelligence (Nature Machine Intelligence, 2020). They applied it to robots whose bodies, materials, and control are designed together. Their approach draws on materials science, mechanical engineering, computer science, biology, and chemistry. In industry, NVIDIA popularized the term at CES, the consumer electronics trade show, on 6 January 2025. That day it announced Cosmos, a platform of world foundation models for training robots and autonomous vehicles.

Generative AI produces text, images, code, and video. Agentic AI plans and runs multistep tasks in software. Physical AI produces outputs that must match physics and geography. A generated paragraph can contain an error and still read well. A robot path or a flood map with an error can cause damage in the real world.

How physical AI works

1. Sense. Cameras, lidar (laser distance scanners), radar, inertial sensors, GPS, and satellites produce images, point clouds (sets of 3D points), traces, and time series.
2. Model. Perception models turn pixels and points into objects and measurements. A map or world model places them in space and time.
3. Decide. A planner or policy (a rule that maps a situation to an action) predicts what happens next and selects an action.
4. Act. A machine moves, or a system delivers a decision that a person acts on, and the loop repeats.

Maps and localization

A robot or vehicle has to know where it is before it can decide anything. Simultaneous localization and mapping (SLAM) builds a map of unknown surroundings while tracking the machine's position in it. Hugh Durrant-Whyte and Tim Bailey's two-part tutorial Simultaneous localization and mapping: part I (IEEE Robotics and Automation Magazine, 2006) is the standard introduction. Self-driving cars also use prebuilt maps. Liu, Wang, and Zhang's review High Definition Map for Automated Driving (Journal of Navigation, 2020) covers high definition (HD) maps, which record roads lane by lane. Vehicles match sensor readings to them to find their position.

World models and world foundation models

A world model predicts how a scene changes after an action, so a system can test actions before taking them. David Ha and Jürgen Schmidhuber's World Models (2018) trained a compact model of the game VizDoom and trained a player inside its predictions. Danijar Hafner and colleagues' DreamerV3, published as Mastering diverse control tasks through world models (Nature, 2025), used one configuration across more than 150 tasks.

A world foundation model is a world model pretrained (first trained) on large video collections, then fine-tuned (trained further) for one robot, vehicle, or scene. NVIDIA's Cosmos World Foundation Model Platform for Physical AI (2025) includes video tokenizers and pretrained models with open weights. Google DeepMind's Genie 3 (August 2025) generates interactive environments from a text prompt at 720p and 24 frames per second. The Waymo World Model (February 2026) adapts Genie 3 to generate camera and lidar driving scenes, and World Labs' Atlas (September 2026) generates and reconstructs 3D worlds.

Vision-language-action models

A vision-language-action (VLA) model maps camera images and a text instruction directly to robot actions. Google DeepMind's RT-2 (2023) introduced the term. It showed that pretraining on web images and text helps a robot handle objects and commands absent from its robot data. Open X-Embodiment (International Conference on Robotics and Automation, 2024) pooled more than one million recorded robot trajectories from 22 robot types across 21 institutions. Physical Intelligence's π0 (Robotics: Science and Systems, 2025) applies the approach to dexterous tasks such as folding laundry. Google DeepMind's Gemini Robotics (2025) extends Gemini 2.0 to robot control.

Simulation, digital twins, and synthetic data

A robot cannot crash millions of times to learn, so developers train in simulation and transfer the result to hardware. Josh Tobin and colleagues wrote the domain randomization paper (International Conference on Intelligent Robots and Systems, 2017). In it, they trained an object detector only on simulated images with random textures, then transferred it to a real robot. The CARLA urban driving simulator (Conference on Robot Learning, 2017) and Habitat (ICCV 2019) for indoor robots became shared test beds. A digital twin is a simulation of one real factory, warehouse, or city.

The physical AI stack

1. Data: satellite imagery, maps, sensor feeds, and GPS traces.
2. Compute: graphics processing unit (GPU) chips, cloud, edge devices, and spatial engines.
3. Spatial context: where things are, what is around them, and how the physical world is structured.
4. Perception: computer vision, lidar processing, and sensor fusion.
5. Planning and decisions: world models, simulation, and path planning.
6. Actuation: robots, vehicles, and drones acting in the world.

Spatial context, which the layers above depend on, answers these questions about a place:

• Where is this? Coordinates, geometry, and spatial relationships between objects.
• What is around it? Proximity, neighborhood, and joins between datasets by location.
• What does the world look like here? Imagery, terrain, and land use.
• What has changed? Change detection and comparison over time.
• How are things connected? Road and sidewalk networks, supply chains, and infrastructure topology.

Assembling those answers for a model or agent is context engineering for physical AI. An ontology, a shared list of entity types and their relations, keeps the answers consistent. Text search for a language model returns passages of text. Joining geometries, processing imagery, and routing over networks need a spatial engine.

Physical AI vs. embodied AI vs. spatial AI

Physical AI is the broadest term: any AI that reasons about or interacts with the physical world, with or without a body.

Embodied AI is the subset of physical AI that runs inside a body, such as a humanoid robot, a drone, or a self-driving car. Duan and colleagues wrote A Survey of Embodied AI: From Simulators to Research Tasks (IEEE Transactions on Emerging Topics in Computational Intelligence, 2022). It groups the field's research into visual exploration, visual navigation, and embodied question answering.

Spatial AI, also called geospatial AI, is the capability to process spatial data and reason about spatial relationships. Andrew Davison's FutureMapping (2018) used the term for SLAM extended to map both the shape and the identity of objects around a device.

Spatial foundation models are pretrained models for Earth data. SatMAE (Cong and colleagues, NeurIPS 2022) handles multispectral imagery taken on several dates. NASA and IBM's Prithvi (Jakubik and colleagues, 2023) was pretrained on more than 1 TB of Harmonized Landsat Sentinel-2 imagery. Google DeepMind's AlphaEarth Foundations (Brown and colleagues, 2025) produces annual embedding layers, numeric summaries of each location, that cover the planet from 2017 through 2024.

	Physical AI	Embodied AI	Spatial AI	Generative AI
Scope	Any AI for the physical world	AI with a physical body	Spatial reasoning capability	Content generation
Examples	Satellite analytics, risk models, robots, vehicles	Humanoids, drones, autonomous vehicles	Spatial joins, SLAM, Earth observation models	Chat assistants, image and video generators
Physical body	Not required	Required	Not required	Not required
Output	An action or a decision tied to a place	Motion and manipulation	A map, relation, or measurement	Text, images, code, video

Physical AI examples

Autonomous vehicles. SAE International's J3016 standard defines six levels of driving automation, from Level 0 (no automation) to Level 5 (full automation). Stanford's Stanley won the 2005 Grand Challenge, a self-driving race run by the Defense Advanced Research Projects Agency (DARPA). Sebastian Thrun and colleagues describe it in Stanley: The robot that won the DARPA Grand Challenge (Journal of Field Robotics, 2006). Two driving datasets followed: KITTI (Geiger, Lenz, and Urtasun, CVPR 2012) and the Waymo Open Dataset (Sun and colleagues, CVPR 2020). The Waymo dataset holds 1,150 scenes of synchronized lidar and camera data.

Robotics. Warehouse and humanoid robots grasp and move objects, and VLA models let them follow plain-language instructions.

Drones. Delivery and inspection drones plan flights around buildings, terrain, and airspace rules. In the United States, Code of Federal Regulations (CFR) rule 14 CFR 107.51 caps small drones at 400 feet above ground level. A drone may fly higher if it stays within a 400-foot radius of a structure and no higher than 400 feet above the structure's top. Applying that rule takes building heights and ground elevation.

Earth-scale models. Agriculture platforms track crop health from Sentinel-2 imagery and join it to parcels and weather. Insurers detect roofs, pools, and solar panels in aerial imagery across a portfolio and join each property to flood and wildfire layers.

Physical AI companies

• Models and simulation: NVIDIA builds Cosmos world foundation models, Omniverse for digital twins, and Isaac for robotics. Google DeepMind and Physical Intelligence build robot foundation models.
• Robots and vehicles: Waymo operates autonomous vehicles. Tesla builds the Optimus humanoid robot.
• Sensors and imagery: satellite operators such as Planet and Maxar capture Earth imagery.
• Open map data: the Overture Maps Foundation, founded in December 2022 under the Linux Foundation, publishes open buildings, places, transportation, and boundary data.
• Spatial context: Wherobots, described below.

The data challenge in physical AI

Physical AI runs on spatial data. Raster data is a grid of pixels, such as satellite imagery or a digital elevation model. Vector data is points, lines, and polygons, such as roads, building footprints, and parcel boundaries.

Ask a chat model how many homes along a coastline sit in a flood zone, and it can describe a method. Without a tool connection it cannot compute the number. That takes a spatial join, which matches each parcel to the flood zone polygons it falls inside. The data is hard to work with:

• Scale. Whole countries, years of imagery.
• Complexity. Coordinate reference systems, spatial indexes, and raster and vector formats.
• Fusion. Imagery with parcels, GPS traces with road networks.
• Change. Roads open and crops grow, so the data needs reprocessing.

Physical AI in Wherobots

Wherobots is a cloud platform for context engineering for physical AI: it gives AI models and agents the spatial data and compute to work with real places at scale. It covers the spatial context layer of the stack.

• RasterFlow (Public Preview) builds mosaics from imagery, runs computer vision inference on them, and writes the results as vectorized outputs on the same platform. Running Meta's Segment Anything Model (SAM3) on 30 cm imagery costs less than $25 per 1,000 square kilometers.
• WherobotsDB runs spatial SQL and Python on vector and raster data, so an agent can join a detected object to its parcel in one query. WherobotsDB is built by the original creators of Apache Sedona, 100% code compatible across all spatial functions.
• The Wherobots Model Context Protocol (MCP) server, command-line interface (CLI), and Visual Studio Code extension connect AI coding tools such as Claude Code and Cursor to that compute. An agent can find the data, write the query, and run it from a plain-language request.
• The Havasu Catalog holds open datasets ready to query. They include Overture buildings, Overture places, the US Department of Agriculture's National Agriculture Imagery Program (NAIP) aerial imagery, Sentinel-2, and the Copernicus 30 m elevation model.

Context for a delivery robot at the University of Maryland

A sidewalk robot or drone bound for a building needs facts about the drop-off. It needs nearby buildings and heights, sidewalks and crosswalks, paths and roads to cross, what sits at the destination, and the shape of the ground. This example builds that context for Theodore R. McKeldin Library on the University of Maryland campus in College Park.

The Tile2Net segmentation model, which labels each pixel of an image, produced polygons labeled sidewalk, crosswalk, or road. It ran with RasterFlow on 30 cm NAIP aerial imagery of College Park, as described in Detecting sidewalks with RasterFlow. Surface height comes from the Copernicus GLO-30 surface model, a global elevation model with 30 m pixels. It includes the tops of buildings and vegetation. The queries read these tables in wherobots_open_data:

Source	Table	Data type	Rows (October 2026)	What the queries read
Overture buildings	overture_maps_foundation.buildings_building	Vector polygons	2,533,842,612	The library footprint and height, buildings and heights within 122 m, and buildings on the straight line
Tile2Net output (RasterFlow on NAIP; the queries do not read NAIP)	rasterflow_output_samples.tile2net_vector	Vector polygons	5,225	Sidewalk and crosswalk polygons within 122 m and on the line
Overture transportation	overture_maps_foundation.transportation_segment	Vector lines	352,054,710	Road-subtype segments, including footways, paths, steps, and cycleways, within 122 m and on the line
Overture places	overture_maps_foundation.places_place	Vector points	81,455,423	Places within 122 m and inside the library footprint
Copernicus GLO-30	copernicus_dem.glo_30m	Raster tiles, 30 m pixels	3,961,635 tiles	The tile under the library: surface height at the library, the range within 122 m, and samples every 10 m along the line

The first query uses a 122 m radius, the 400 feet in the Federal Aviation Administration (FAA) drone rule. Each table read is limited to a box around the library. Distances are in meters, in Universal Transverse Mercator (UTM) zone 18N, a flat map grid:

WITH c AS (
  SELECT ST_Transform(ST_SetSRID(ST_Point(-76.94512, 38.98596), 4326), 'EPSG:4326', 'EPSG:32618') AS p
),
lib AS (
  SELECT id, names.primary AS name, height, geometry
  FROM wherobots_open_data.overture_maps_foundation.buildings_building
  WHERE bbox.xmin > -76.947 AND bbox.xmax < -76.943 AND bbox.ymin > 38.985 AND bbox.ymax < 38.987
    AND names.primary = 'Theodore R. McKeldin Library'
),
bld AS (
  SELECT /*+ BROADCAST(c) */ COUNT(*) AS buildings, COUNT(o.height) AS with_height,
         MAX_BY(o.names.primary, o.height) AS tallest, MAX(o.height) AS tallest_m
  FROM wherobots_open_data.overture_maps_foundation.buildings_building o CROSS JOIN c
  WHERE o.bbox.xmin < -76.943 AND o.bbox.xmax > -76.947 AND o.bbox.ymin < 38.9875 AND o.bbox.ymax > 38.9845
    AND ST_DWithin(ST_Transform(ST_SetSRID(o.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122)
),
walk AS (
  SELECT /*+ BROADCAST(c) */
         SUM(CASE WHEN t.layer = 'sidewalk' THEN 1 ELSE 0 END) AS sidewalk_polygons,
         SUM(CASE WHEN t.layer = 'sidewalk' THEN ST_Area(ST_Intersection(
             ST_Transform(ST_SetSRID(t.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), ST_Buffer(c.p, 122))) ELSE 0 END) AS sidewalk_m2,
         SUM(CASE WHEN t.layer = 'crosswalk' THEN 1 ELSE 0 END) AS crosswalks
  FROM wherobots_open_data.rasterflow_output_samples.tile2net_vector t CROSS JOIN c
  WHERE t.xmin < -76.943 AND t.xmax > -76.947 AND t.ymin < 38.9875 AND t.ymax > 38.9845
    AND ST_DWithin(ST_Transform(ST_SetSRID(t.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122)
),
roads AS (
  SELECT /*+ BROADCAST(c) */ COUNT(*) AS segments,
         SUM(CASE WHEN s.class IN ('footway', 'pedestrian', 'path', 'steps', 'cycleway') THEN 1 ELSE 0 END) AS path_segments,
         SUM(ST_Length(ST_Intersection(
             ST_Transform(ST_SetSRID(s.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), ST_Buffer(c.p, 122)))) AS segment_m
  FROM wherobots_open_data.overture_maps_foundation.transportation_segment s CROSS JOIN c
  WHERE s.bbox.xmin < -76.943 AND s.bbox.xmax > -76.947 AND s.bbox.ymin < 38.9875 AND s.bbox.ymax > 38.9845
    AND s.subtype = 'road'
    AND ST_DWithin(ST_Transform(ST_SetSRID(s.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122)
),
pl AS (
  SELECT /*+ BROADCAST(c, lib) */ COUNT(*) AS places,
         SUM(CASE WHEN ST_Contains(lib.geometry, p.geometry) THEN 1 ELSE 0 END) AS places_in_library
  FROM wherobots_open_data.overture_maps_foundation.places_place p CROSS JOIN c CROSS JOIN lib
  WHERE p.bbox.xmin < -76.943 AND p.bbox.xmax > -76.947 AND p.bbox.ymin < 38.9875 AND p.bbox.ymax > 38.9845
    AND ST_DWithin(ST_Transform(ST_SetSRID(p.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122)
),
dem AS (
  SELECT /*+ BROADCAST(c) */ t.name AS dem_tile,
         RS_Value(t.rast, ST_Point(-76.94512, 38.98596)) AS surface_m,
         RS_ZonalStats(t.rast, ST_Transform(ST_SetSRID(ST_Buffer(c.p, 122), 32618), 'EPSG:32618', 'EPSG:4326'), 1, 'min', true) AS dem_min_m,
         RS_ZonalStats(t.rast, ST_Transform(ST_SetSRID(ST_Buffer(c.p, 122), 32618), 'EPSG:32618', 'EPSG:4326'), 1, 'max', true) AS dem_max_m
  FROM wherobots_open_data.copernicus_dem.glo_30m t CROSS JOIN c
  WHERE ST_Intersects(t.footprint, ST_Point(-76.94512, 38.98596))
)
SELECT /*+ BROADCAST(bld, walk, roads, pl, dem) */
       lib.id, lib.name, ROUND(lib.height, 1) AS height_m,
       bld.buildings, bld.with_height, bld.tallest, ROUND(bld.tallest_m, 1) AS tallest_m,
       walk.sidewalk_polygons, ROUND(walk.sidewalk_m2) AS sidewalk_m2, walk.crosswalks,
       roads.segments, roads.path_segments, ROUND(roads.segment_m) AS segment_m,
       pl.places, pl.places_in_library,
       dem.dem_tile, ROUND(dem.surface_m, 1) AS surface_m, ROUND(dem.dem_min_m, 1) AS dem_min_m, ROUND(dem.dem_max_m, 1) AS dem_max_m
FROM lib CROSS JOIN bld CROSS JOIN walk CROSS JOIN roads CROSS JOIN pl CROSS JOIN dem

The query reads the raster surface model and four vector tables in WherobotsDB: raster and vector in one engine; most stacks pick one. It returns one row in about 36 seconds, the context packet a planner needs for the drop-off:

Field	Value
Overture Global Entity Reference System (GERS) building ID	45837714-76fa-43b7-8e93-3988558a9e68
Library height	12.9 m
Buildings within 122 m (with a height value)	9 (8)
Tallest building within 122 m	Jimenez Hall, 17.3 m
Tile2Net sidewalk area within 122 m	8,240 m² in 5 polygons
Tile2Net crosswalks	3
Overture segments within 122 m (of which footways, paths, steps, cycleways)	133 (130), 3,253 m in all
Places within 122 m (inside the library)	14 (2)
Surface height at the library centroid, and range within 122 m	52.8 m, 37.4 to 59.6 m

Jimenez Hall, at 17.3 m, is the tallest structure within 400 feet. That height is the input to the Part 107 structure rule for a drone over the drop-off. The one building with no height stays an unknown. The 130 path segments (unfiltered, steps included) and 3 crosswalks are the inventory a sidewalk robot's route planner starts from. The 2 places inside the footprint identify where in the building the order goes.

A second query, on four of the same tables, checks the straight line from Adele H. Stamp Student Union, the campus student center, to the library. It returns the buildings and segments the line crosses, its length on Tile2Net sidewalk, and the surface model every 10 m:

WITH r AS (
  SELECT ST_MakeLine(ST_Point(-76.94474, 38.98814), ST_Point(-76.94512, 38.98596)) AS ll,
         ST_Transform(ST_SetSRID(ST_MakeLine(ST_Point(-76.94474, 38.98814), ST_Point(-76.94512, 38.98596)), 4326),
                      'EPSG:4326', 'EPSG:32618') AS utm
),
bld AS (
  SELECT /*+ BROADCAST(r) */ COUNT(*) AS buildings_crossed,
         concat_ws(', ', collect_list(concat(coalesce(o.names.primary, 'unnamed'), ' ',
                   coalesce(CAST(ROUND(o.height, 1) AS STRING), 'no height'), ' m'))) AS crossed
  FROM wherobots_open_data.overture_maps_foundation.buildings_building o CROSS JOIN r
  WHERE o.bbox.xmin < -76.9445 AND o.bbox.xmax > -76.9455 AND o.bbox.ymin < 38.9883 AND o.bbox.ymax > 38.9858
    AND ST_Intersects(o.geometry, r.ll)
),
walk AS (
  SELECT /*+ BROADCAST(r) */
         ST_Length(ST_Intersection(r.utm, ST_Union_Aggr(CASE WHEN t.layer = 'sidewalk'
             THEN ST_Transform(ST_SetSRID(t.geometry, 4326), 'EPSG:4326', 'EPSG:32618') END))) AS on_sidewalk_m,
         SUM(CASE WHEN t.layer = 'crosswalk' THEN 1 ELSE 0 END) AS crosswalks_crossed
  FROM wherobots_open_data.rasterflow_output_samples.tile2net_vector t CROSS JOIN r
  WHERE t.xmin < -76.9445 AND t.xmax > -76.9455 AND t.ymin < 38.9883 AND t.ymax > 38.9858
    AND ST_Intersects(t.geometry, r.ll)
  GROUP BY r.utm
),
roads AS (
  SELECT /*+ BROADCAST(r) */ COUNT(*) AS segments_crossed,
         concat_ws(', ', collect_list(concat(coalesce(s.names.primary, 'unnamed'), ' (', s.class, ')'))) AS segment_list
  FROM wherobots_open_data.overture_maps_foundation.transportation_segment s CROSS JOIN r
  WHERE s.bbox.xmin < -76.9445 AND s.bbox.xmax > -76.9455 AND s.bbox.ymin < 38.9883 AND s.bbox.ymax > 38.9858
    AND s.subtype = 'road' AND ST_Intersects(s.geometry, r.ll)
),
pts AS (
  SELECT d * 10 AS dist_m,
         ST_Transform(ST_SetSRID(ST_LineInterpolatePoint(r.utm, LEAST(d * 10 / ST_Length(r.utm), 1.0)), 32618),
                      'EPSG:32618', 'EPSG:4326') AS pt
  FROM r LATERAL VIEW explode(sequence(0, CAST(FLOOR(ST_Length(r.utm) / 10) AS INT))) e AS d
),
prof AS (
  SELECT /*+ BROADCAST(pts) */ collect_list(struct(pts.dist_m, ROUND(RS_Value(t.rast, pts.pt), 1) AS dsm_m)) AS profile
  FROM wherobots_open_data.copernicus_dem.glo_30m t CROSS JOIN pts
  WHERE ST_Intersects(t.footprint, ST_Point(-76.945, 38.987))
)
SELECT ROUND(ST_Length(r.utm), 1) AS route_m, bld.buildings_crossed, bld.crossed,
       ROUND(walk.on_sidewalk_m, 1) AS on_sidewalk_m, walk.crosswalks_crossed,
       roads.segments_crossed, roads.segment_list, array_sort(prof.profile) AS profile
FROM r CROSS JOIN bld CROSS JOIN walk CROSS JOIN roads CROSS JOIN prof

The line is 244.2 m long and returns in about 32 seconds. It crosses 3 buildings: the student union (15.6 m), the University of Maryland (UMD) Health Center (10.6 m), and the library (12.9 m). It crosses 11 Overture segments: Campus Drive, a tertiary road, plus 8 footways, 1 cycleway, and 1 flight of steps. Only 41.5 m of it lies on Tile2Net sidewalk, and it crosses none of the Tile2Net crosswalks.

A drone can fly the straight line if it clears the roofs. A wheeled robot cannot. The line runs through three buildings, across a road at a point with no Tile2Net crosswalk, and down a flight of steps. The robot has to route on the footway network. The 30 m surface model mixes ground, roofs, and tree canopy. Slope checks for a sidewalk robot need finer elevation, such as lidar.

Physical AI research and what changes at scale

Physical AI research runs through four threads: embodied robotics, autonomous driving, learned world models, and AI for geographic data.

From behavior-based robots to robot foundation models

Rodney Brooks argued in Intelligence without representation (Artificial Intelligence, 1991) that intelligence comes from layers of simple behaviors coupled to the world through sensors, with no central symbolic model. SLAM and Stanley took the opposite approach, with an explicit, uncertain map of the surroundings.

Robot learning then added pretraining on web data and pooled robot data, as in RT-2 and Open X-Embodiment. The Cosmos platform paper frames the current step. It says a physical AI system needs two digital twins. One is a twin of the system itself (the policy). The other is a twin of the world (the world model).

AI for geographic data

Krzysztof Janowicz and colleagues argued that models for geographic problems need location, distance, and spatial relations built in. They made this argument in GeoAI: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond (IJGIS, 2020). The GeoLLM study (Manvi and colleagues, ICLR 2024) found that prompting with coordinates alone gives poor estimates. Adding nearby OpenStreetMap features to the prompt improved results by 70% over baselines.

Vision models now build part of the context layer. Maryam Hosseini and colleagues' Mapping the walk (Computers, Environment and Urban Systems, 2023) introduced Tile2Net. It detects roads, sidewalks, and crosswalks in aerial image tiles and converts them into a connected sidewalk network. The College Park sidewalks above come from Tile2Net.

Computing spatial context at scale

One delivery takes a handful of joins. A fleet serving a metro area repeats them for millions of locations over tables with billions of rows, and again when the map changes. Jia Yu, Jinxuan Wu, and Mohamed Sarwat introduced GeoSpark (ACM SIGSPATIAL 2015) to run spatial joins across a cluster. Yu, Zongsi Zhang, and Sarwat described its spatial partitioning, indexing, and join design in Spatial data management in Apache Spark: the GeoSpark perspective and beyond (GeoInformatica, 2019). GeoSpark became Apache Sedona, and Yu and Sarwat went on to found Wherobots.

A detector such as Tile2Net runs over every tile of an area before any join can use its output. Change detection on Earth observation data marks the tiles that need reprocessing.

Open problems

• The sim-to-real gap. Policies trained in simulation still fail on textures, lighting, and physics the simulator never produced.
• The long tail. Rare events such as an unmapped construction zone or a fallen tree carry much of the safety risk and appear rarely in any dataset.
• Map freshness. HD maps, sidewalk networks, and building heights age, so every fact needs a date and a source.
• Data gaps. Open building heights are uneven, and aerial sidewalk extraction misses paths under tree canopy.
• Evaluation. The benchmarks cited here test perception or manipulation in isolation. None of them tests whether a system used the right spatial context for a real place.

Conclusions

The walkthrough shows what physical AI needs before a machine acts at one real place: facts joined from several datasets around it.

1. Physical AI needs spatial context built per place, from joins. Every value in the McKeldin Library packet came from joins around one point.
2. The same joins show what each machine can do. A drone can use the straight line only if it clears the roofs. A wheeled robot meets three buildings, a road crossing with no Tile2Net crosswalk, and steps. If the output is an action, the context has to carry the constraints that forbid it.
3. Physical AI needs elevation at the resolution of the decision. A 30 m surface model mixes ground, roofs, and canopy. Sidewalk slope needs finer elevation.
4. A physical AI planner has to carry unknowns as unknowns. One of 9 buildings has no height, and aerial extraction can miss sidewalks under canopy.
5. Physical AI at fleet scale needs distributed joins. A fleet repeats one delivery's joins for millions of locations. Run them across a cluster, and reprocess only changed tiles.
6. Map context may help physical AI plan. That is unproven. Nearby map features improved language model estimates by 70% in the GeoLLM study, on its benchmarks. Whether that gain carries to action planning is open.

Related terms

• Context engineering: assembling the spatial facts a model or agent needs about a place
• Ontology: the classes and relations that let AI agents reason about places
• Geospatial AI: machine learning and AI agents applied to spatial data
• Earth observation data: satellite imagery that many physical AI models use
• Change detection: finding what changed in imagery over time
• LiDAR data: 3D point clouds robots and vehicles use to sense their surroundings
• POI (point of interest) data: the places where a delivery or trip ends
• Havasu Catalog: the Wherobots knowledge lake of physical world data, models, and apps

Read more from Wherobots

• How Wherobots builds with NVIDIA to let AI see the physical world
• Introducing developer tools that let AI build with physical world data
• Detecting objects from text prompts with RasterFlow and Segment Anything 3
• Spatial Graph RAG for the Physical World

Connect your AI agent to physical world data through the Wherobots MCP server. Start a free trial at cloud.wherobots.com.