--- title: "What is Physical AI? Definition, Examples, How It Works" url: "https://wherobots.com/discover/physical-ai/" description: "Physical AI is AI that perceives, reasons, and acts in the physical world. How it works, robots, world models, examples, and the spatial data it runs on." published: "2026-10-02" modified: "2026-10-21" --- # What is Physical AI? Definition, Examples, How It Works Physical AI is artificial intelligence that perceives, reasons about, and acts in the physical world. The term covers robots, autonomous vehicles, drones, and the world models used to train them. It also covers models that read satellite, aerial, and sensor data to describe the ground, from one street to a continent. Each one depends on geometry, location, motion, and physics. Language models learn from text, documents, databases, and the internet, which describe those only in part. ## Key takeaways - Physical AI is AI that perceives, reasons about, and acts in the physical world, with or without a body. - It runs a loop: sense, model, decide, act. Developers often train it in simulation, with world models and synthetic data. - A physical AI system needs spatial context: where things are, what surrounds them, and how the ground is shaped. - That context comes from joining buildings, roads, places, elevation, and imagery around a location. ## What is physical AI? A physical AI system takes in sensor data and builds a model of its surroundings. It then produces an output tied to a location: a steering command, a robot grasp, a flight path, or a map of damaged roofs. In robotics research, Aslan Miriyev and Mirko Kovač used "physical artificial intelligence" in [Skills for physical artificial intelligence](https://doi.org/10.1038/s42256-020-00258-y) (Nature Machine Intelligence, 2020). They applied it to robots whose bodies, materials, and control are designed together. Their approach draws on materials science, mechanical engineering, computer science, biology, and chemistry. In industry, NVIDIA popularized the term at CES, the consumer electronics trade show, on 6 January 2025. That day it [announced Cosmos](https://investor.nvidia.com/news/press-release-details/2025/NVIDIA-Launches-Cosmos-World-Foundation-Model-Platform-to-Accelerate-Physical-AI-Development/default.aspx), a platform of world foundation models for training robots and autonomous vehicles. Generative AI produces text, images, code, and video. Agentic AI plans and runs multistep tasks in software. Physical AI produces outputs that must match physics and geography. A generated paragraph can contain an error and still read well. A robot path or a flood map with an error can cause damage in the real world. ## How physical AI works 1. **Sense.** Cameras, lidar (laser distance scanners), radar, inertial sensors, GPS, and satellites produce images, point clouds (sets of 3D points), traces, and time series. 2. **Model.** Perception models turn pixels and points into objects and measurements. A map or world model places them in space and time. 3. **Decide.** A planner or policy (a rule that maps a situation to an action) predicts what happens next and selects an action. 4. **Act.** A machine moves, or a system delivers a decision that a person acts on, and the loop repeats. ![Diagram of the physical AI loop: sense, model, decide, and act in a cycle. A spatial context panel below feeds footprints, heights, roads, sidewalks, places, elevation, imagery, and rules into the model and decide steps. A training panel on the right lists simulation and digital twins, world foundation models, vision-language-action models, real-world logs, and spatial foundation models](https://wherobots.com/wp-content/themes/wherobots/assets/img/discover/physical-ai-loop.webp) *The physical AI loop. Live sensors cover what the machine can see. Spatial context supplies what it cannot see yet, such as the building at the end of a route or the slope beyond the next corner.* ### Maps and localization A robot or vehicle has to know where it is before it can decide anything. Simultaneous localization and mapping (SLAM) builds a map of unknown surroundings while tracking the machine's position in it. Hugh Durrant-Whyte and Tim Bailey's two-part tutorial [Simultaneous localization and mapping: part I](https://doi.org/10.1109/MRA.2006.1638022) (IEEE Robotics and Automation Magazine, 2006) is the standard introduction. Self-driving cars also use prebuilt maps. Liu, Wang, and Zhang's review [High Definition Map for Automated Driving](https://doi.org/10.1017/S0373463319000638) (Journal of Navigation, 2020) covers high definition (HD) maps, which record roads lane by lane. Vehicles match sensor readings to them to find their position. ### World models and world foundation models A world model predicts how a scene changes after an action, so a system can test actions before taking them. David Ha and Jürgen Schmidhuber's [World Models](https://arxiv.org/abs/1803.10122) (2018) trained a compact model of the game VizDoom and trained a player inside its predictions. Danijar Hafner and colleagues' DreamerV3, published as [Mastering diverse control tasks through world models](https://doi.org/10.1038/s41586-025-08744-2) (Nature, 2025), used one configuration across more than 150 tasks. A world foundation model is a world model pretrained (first trained) on large video collections, then fine-tuned (trained further) for one robot, vehicle, or scene. NVIDIA's [Cosmos World Foundation Model Platform for Physical AI](https://arxiv.org/abs/2501.03575) (2025) includes video tokenizers and pretrained models with open weights. Google DeepMind's [Genie 3](https://deepmind.google/discover/blog/genie-3-a-new-frontier-for-world-models/) (August 2025) generates interactive environments from a text prompt at 720p and 24 frames per second. The [Waymo World Model](https://waymo.com/blog/2026/02/the-waymo-world-model-a-new-frontier-for-autonomous-driving-simulation) (February 2026) adapts Genie 3 to generate camera and lidar driving scenes, and World Labs' [Atlas](https://www.worldlabs.ai/blog/atlas) (September 2026) generates and reconstructs 3D worlds. ### Vision-language-action models A vision-language-action (VLA) model maps camera images and a text instruction directly to robot actions. Google DeepMind's [RT-2](https://arxiv.org/abs/2307.15818) (2023) introduced the term. It showed that pretraining on web images and text helps a robot handle objects and commands absent from its robot data. [Open X-Embodiment](https://arxiv.org/abs/2310.08864) (International Conference on Robotics and Automation, 2024) pooled more than one million recorded robot trajectories from 22 robot types across 21 institutions. Physical Intelligence's [π0](https://arxiv.org/abs/2410.24164) (Robotics: Science and Systems, 2025) applies the approach to dexterous tasks such as folding laundry. Google DeepMind's [Gemini Robotics](https://arxiv.org/abs/2503.20020) (2025) extends Gemini 2.0 to robot control. ### Simulation, digital twins, and synthetic data A robot cannot crash millions of times to learn, so developers train in simulation and transfer the result to hardware. Josh Tobin and colleagues wrote the [domain randomization](https://arxiv.org/abs/1703.06907) paper (International Conference on Intelligent Robots and Systems, 2017). In it, they trained an object detector only on simulated images with random textures, then transferred it to a real robot. The [CARLA](https://proceedings.mlr.press/v78/dosovitskiy17a.html) urban driving simulator (Conference on Robot Learning, 2017) and [Habitat](https://arxiv.org/abs/1904.01201) (ICCV 2019) for indoor robots became shared test beds. A digital twin is a simulation of one real factory, warehouse, or city. ## The physical AI stack ![The physical AI stack, from bottom to top: data, compute, spatial context, perception, planning and decisions, and actuation](https://wherobots.com/wp-content/themes/wherobots/assets/img/discover/physical-ai-stack.webp) 1. **Data:** satellite imagery, maps, sensor feeds, and GPS traces. 2. **Compute:** graphics processing unit (GPU) chips, cloud, edge devices, and spatial engines. 3. **Spatial context:** where things are, what is around them, and how the physical world is structured. 4. **Perception:** computer vision, lidar processing, and sensor fusion. 5. **Planning and decisions:** world models, simulation, and path planning. 6. **Actuation:** robots, vehicles, and drones acting in the world. Spatial context, which the layers above depend on, answers these questions about a place: - **Where is this?** Coordinates, geometry, and spatial relationships between objects. - **What is around it?** Proximity, neighborhood, and joins between datasets by location. - **What does the world look like here?** Imagery, terrain, and land use. - **What has changed?** Change detection and comparison over time. - **How are things connected?** Road and sidewalk networks, supply chains, and infrastructure topology. Assembling those answers for a model or agent is [context engineering for physical AI](https://wherobots.com/discover/context-engineering/). An [ontology](https://wherobots.com/discover/ontology/), a shared list of entity types and their relations, keeps the answers consistent. Text search for a language model returns passages of text. Joining geometries, processing imagery, and routing over networks need a spatial engine. ## Physical AI vs. embodied AI vs. spatial AI **Physical AI** is the broadest term: any AI that reasons about or interacts with the physical world, with or without a body. **Embodied AI** is the subset of physical AI that runs inside a body, such as a humanoid robot, a drone, or a self-driving car. Duan and colleagues wrote [A Survey of Embodied AI: From Simulators to Research Tasks](https://doi.org/10.1109/TETCI.2022.3141105) (IEEE Transactions on Emerging Topics in Computational Intelligence, 2022). It groups the field's research into visual exploration, visual navigation, and embodied question answering. **Spatial AI**, also called [geospatial AI](https://wherobots.com/discover/geospatial-ai/), is the capability to process spatial data and reason about spatial relationships. Andrew Davison's [FutureMapping](https://arxiv.org/abs/1803.11288) (2018) used the term for SLAM extended to map both the shape and the identity of objects around a device. **Spatial foundation models** are pretrained models for Earth data. [SatMAE](https://arxiv.org/abs/2207.08051) (Cong and colleagues, NeurIPS 2022) handles multispectral imagery taken on several dates. NASA and IBM's [Prithvi](https://arxiv.org/abs/2310.18660) (Jakubik and colleagues, 2023) was pretrained on more than 1 TB of Harmonized Landsat Sentinel-2 imagery. Google DeepMind's [AlphaEarth Foundations](https://arxiv.org/abs/2507.22291) (Brown and colleagues, 2025) produces annual embedding layers, numeric summaries of each location, that cover the planet from 2017 through 2024. | | Physical AI | Embodied AI | Spatial AI | Generative AI | | --- | --- | --- | --- | --- | | Scope | Any AI for the physical world | AI with a physical body | Spatial reasoning capability | Content generation | | Examples | Satellite analytics, risk models, robots, vehicles | Humanoids, drones, autonomous vehicles | Spatial joins, SLAM, Earth observation models | Chat assistants, image and video generators | | Physical body | Not required | Required | Not required | Not required | | Output | An action or a decision tied to a place | Motion and manipulation | A map, relation, or measurement | Text, images, code, video | ## Physical AI examples **Autonomous vehicles.** SAE International's [J3016](https://saemobilus.sae.org/standards/j3016_202104-taxonomy-definitions-terms-related-driving-automation-systems-road-motor-vehicles) standard defines six levels of driving automation, from Level 0 (no automation) to Level 5 (full automation). Stanford's Stanley won the 2005 Grand Challenge, a self-driving race run by the Defense Advanced Research Projects Agency (DARPA). Sebastian Thrun and colleagues describe it in [Stanley: The robot that won the DARPA Grand Challenge](https://doi.org/10.1002/rob.20147) (Journal of Field Robotics, 2006). Two driving datasets followed: [KITTI](https://doi.org/10.1109/CVPR.2012.6248074) (Geiger, Lenz, and Urtasun, CVPR 2012) and the [Waymo Open Dataset](https://openaccess.thecvf.com/content_CVPR_2020/html/Sun_Scalability_in_Perception_for_Autonomous_Driving_Waymo_Open_Dataset_CVPR_2020_paper.html) (Sun and colleagues, CVPR 2020). The Waymo dataset holds 1,150 scenes of synchronized lidar and camera data. **Robotics.** Warehouse and humanoid robots grasp and move objects, and VLA models let them follow plain-language instructions. **Drones.** Delivery and inspection drones plan flights around buildings, terrain, and airspace rules. In the United States, Code of Federal Regulations (CFR) rule [14 CFR 107.51](https://www.ecfr.gov/current/title-14/chapter-I/subchapter-F/part-107/subpart-B/section-107.51) caps small drones at 400 feet above ground level. A drone may fly higher if it stays within a 400-foot radius of a structure and no higher than 400 feet above the structure's top. Applying that rule takes building heights and ground elevation. **Earth-scale models.** Agriculture platforms track crop health from [Sentinel-2](https://wherobots.com/discover/sentinel-2/) imagery and join it to parcels and weather. Insurers detect roofs, pools, and solar panels in aerial imagery across a portfolio and join each property to flood and wildfire layers. ## Physical AI companies - **Models and simulation:** NVIDIA builds Cosmos world foundation models, Omniverse for digital twins, and Isaac for robotics. Google DeepMind and Physical Intelligence build robot foundation models. - **Robots and vehicles:** Waymo operates autonomous vehicles. Tesla builds the Optimus humanoid robot. - **Sensors and imagery:** satellite operators such as Planet and Maxar capture Earth imagery. - **Open map data:** the [Overture Maps Foundation](https://overturemaps.org/), founded in December 2022 under the Linux Foundation, publishes open buildings, places, transportation, and boundary data. - **Spatial context:** Wherobots, described below. ## The data challenge in physical AI Physical AI runs on spatial data. Raster data is a grid of pixels, such as satellite imagery or a [digital elevation model](https://wherobots.com/discover/digital-elevation-model/). Vector data is points, lines, and polygons, such as roads, [building footprints](https://wherobots.com/discover/building-footprint/), and [parcel boundaries](https://wherobots.com/discover/parcel-data/). Ask a chat model how many homes along a coastline sit in a flood zone, and it can describe a method. Without a tool connection it cannot compute the number. That takes a [spatial join](https://wherobots.com/discover/spatial-join/), which matches each parcel to the flood zone polygons it falls inside. The data is hard to work with: - **Scale.** Whole countries, years of imagery. - **Complexity.** [Coordinate reference systems](https://wherobots.com/discover/coordinate-reference-system/), spatial indexes, and raster and vector formats. - **Fusion.** Imagery with parcels, GPS traces with road networks. - **Change.** Roads open and crops grow, so the data needs reprocessing. ## Physical AI in Wherobots Wherobots is a cloud platform for [context engineering for physical AI](https://wherobots.com/): it gives AI models and agents the spatial data and compute to work with real places at scale. It covers the spatial context layer of the stack. - **[RasterFlow](https://wherobots.com/products/ai-for-earth/) (Public Preview)** builds mosaics from imagery, runs computer vision inference on them, and writes the results as vectorized outputs on the same platform. Running Meta's Segment Anything Model (SAM3) on 30 cm imagery costs less than [$25 per 1,000 square kilometers](https://wherobots.com/blog/wherobots-nvidia-geospatial-ai-physical-world/). - **[WherobotsDB](https://wherobots.com/products/compute/)** runs spatial SQL and Python on [vector and raster data](https://wherobots.com/discover/raster-vs-vector-data/), so an agent can join a detected object to its parcel in one query. WherobotsDB is built by the original creators of [Apache Sedona](https://wherobots.com/apache-sedona/), [100% code compatible](https://wherobots.com/apache-sedona/wherobots-vs-apache-sedona/) across all spatial functions. - **The [Wherobots Model Context Protocol (MCP) server](https://docs.wherobots.com/develop/mcp/mcp-server-overview), command-line interface (CLI), and Visual Studio Code extension** connect AI coding tools such as Claude Code and Cursor to that compute. An agent can find the data, write the query, and run it from a plain-language request. - **The [Havasu Catalog](https://wherobots.com/havasu/)** holds open datasets ready to query. They include [Overture buildings](https://wherobots.com/havasu/#buildings_building), [Overture places](https://wherobots.com/havasu/#places_place), the US Department of Agriculture's [National Agriculture Imagery Program (NAIP) aerial imagery](https://wherobots.com/havasu/#naip), Sentinel-2, and the [Copernicus 30 m elevation model](https://wherobots.com/havasu/#glo_30m). ### Context for a delivery robot at the University of Maryland A sidewalk robot or drone bound for a building needs facts about the drop-off. It needs nearby buildings and heights, sidewalks and crosswalks, paths and roads to cross, what sits at the destination, and the shape of the ground. This example builds that context for Theodore R. McKeldin Library on the University of Maryland campus in College Park. The Tile2Net segmentation model, which labels each pixel of an image, produced polygons labeled sidewalk, crosswalk, or road. It ran with RasterFlow on 30 cm NAIP aerial imagery of College Park, as described in [Detecting sidewalks with RasterFlow](https://wherobots.com/solution-gallery/rasterflow-tile2net/). Surface height comes from the [Copernicus GLO-30 surface model](https://registry.opendata.aws/copernicus-dem/), a global elevation model with 30 m pixels. It includes the tops of buildings and vegetation. The queries read these tables in `wherobots_open_data`: | Source | Table | Data type | Rows (October 2026) | What the queries read | | --- | --- | --- | --- | --- | | Overture buildings | `overture_maps_foundation.buildings_building` | Vector polygons | 2,533,842,612 | The library footprint and height, buildings and heights within 122 m, and buildings on the straight line | | Tile2Net output (RasterFlow on NAIP; the queries do not read NAIP) | `rasterflow_output_samples.tile2net_vector` | Vector polygons | 5,225 | Sidewalk and crosswalk polygons within 122 m and on the line | | Overture transportation | `overture_maps_foundation.transportation_segment` | Vector lines | 352,054,710 | Road-subtype segments, including footways, paths, steps, and cycleways, within 122 m and on the line | | Overture [places](https://wherobots.com/discover/poi-data/) | `overture_maps_foundation.places_place` | Vector points | 81,455,423 | Places within 122 m and inside the library footprint | | Copernicus GLO-30 | `copernicus_dem.glo_30m` | Raster tiles, 30 m pixels | 3,961,635 tiles | The tile under the library: surface height at the library, the range within 122 m, and samples every 10 m along the line | The first query uses a 122 m radius, the 400 feet in the Federal Aviation Administration (FAA) drone rule. Each table read is limited to a box around the library. Distances are in meters, in Universal Transverse Mercator (UTM) zone 18N, a flat map grid: ``` WITH c AS ( SELECT ST_Transform(ST_SetSRID(ST_Point(-76.94512, 38.98596), 4326), 'EPSG:4326', 'EPSG:32618') AS p ), lib AS ( SELECT id, names.primary AS name, height, geometry FROM wherobots_open_data.overture_maps_foundation.buildings_building WHERE bbox.xmin > -76.947 AND bbox.xmax < -76.943 AND bbox.ymin > 38.985 AND bbox.ymax < 38.987 AND names.primary = 'Theodore R. McKeldin Library' ), bld AS ( SELECT /*+ BROADCAST(c) */ COUNT(*) AS buildings, COUNT(o.height) AS with_height, MAX_BY(o.names.primary, o.height) AS tallest, MAX(o.height) AS tallest_m FROM wherobots_open_data.overture_maps_foundation.buildings_building o CROSS JOIN c WHERE o.bbox.xmin < -76.943 AND o.bbox.xmax > -76.947 AND o.bbox.ymin < 38.9875 AND o.bbox.ymax > 38.9845 AND ST_DWithin(ST_Transform(ST_SetSRID(o.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122) ), walk AS ( SELECT /*+ BROADCAST(c) */ SUM(CASE WHEN t.layer = 'sidewalk' THEN 1 ELSE 0 END) AS sidewalk_polygons, SUM(CASE WHEN t.layer = 'sidewalk' THEN ST_Area(ST_Intersection( ST_Transform(ST_SetSRID(t.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), ST_Buffer(c.p, 122))) ELSE 0 END) AS sidewalk_m2, SUM(CASE WHEN t.layer = 'crosswalk' THEN 1 ELSE 0 END) AS crosswalks FROM wherobots_open_data.rasterflow_output_samples.tile2net_vector t CROSS JOIN c WHERE t.xmin < -76.943 AND t.xmax > -76.947 AND t.ymin < 38.9875 AND t.ymax > 38.9845 AND ST_DWithin(ST_Transform(ST_SetSRID(t.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122) ), roads AS ( SELECT /*+ BROADCAST(c) */ COUNT(*) AS segments, SUM(CASE WHEN s.class IN ('footway', 'pedestrian', 'path', 'steps', 'cycleway') THEN 1 ELSE 0 END) AS path_segments, SUM(ST_Length(ST_Intersection( ST_Transform(ST_SetSRID(s.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), ST_Buffer(c.p, 122)))) AS segment_m FROM wherobots_open_data.overture_maps_foundation.transportation_segment s CROSS JOIN c WHERE s.bbox.xmin < -76.943 AND s.bbox.xmax > -76.947 AND s.bbox.ymin < 38.9875 AND s.bbox.ymax > 38.9845 AND s.subtype = 'road' AND ST_DWithin(ST_Transform(ST_SetSRID(s.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122) ), pl AS ( SELECT /*+ BROADCAST(c, lib) */ COUNT(*) AS places, SUM(CASE WHEN ST_Contains(lib.geometry, p.geometry) THEN 1 ELSE 0 END) AS places_in_library FROM wherobots_open_data.overture_maps_foundation.places_place p CROSS JOIN c CROSS JOIN lib WHERE p.bbox.xmin < -76.943 AND p.bbox.xmax > -76.947 AND p.bbox.ymin < 38.9875 AND p.bbox.ymax > 38.9845 AND ST_DWithin(ST_Transform(ST_SetSRID(p.geometry, 4326), 'EPSG:4326', 'EPSG:32618'), c.p, 122) ), dem AS ( SELECT /*+ BROADCAST(c) */ t.name AS dem_tile, RS_Value(t.rast, ST_Point(-76.94512, 38.98596)) AS surface_m, RS_ZonalStats(t.rast, ST_Transform(ST_SetSRID(ST_Buffer(c.p, 122), 32618), 'EPSG:32618', 'EPSG:4326'), 1, 'min', true) AS dem_min_m, RS_ZonalStats(t.rast, ST_Transform(ST_SetSRID(ST_Buffer(c.p, 122), 32618), 'EPSG:32618', 'EPSG:4326'), 1, 'max', true) AS dem_max_m FROM wherobots_open_data.copernicus_dem.glo_30m t CROSS JOIN c WHERE ST_Intersects(t.footprint, ST_Point(-76.94512, 38.98596)) ) SELECT /*+ BROADCAST(bld, walk, roads, pl, dem) */ lib.id, lib.name, ROUND(lib.height, 1) AS height_m, bld.buildings, bld.with_height, bld.tallest, ROUND(bld.tallest_m, 1) AS tallest_m, walk.sidewalk_polygons, ROUND(walk.sidewalk_m2) AS sidewalk_m2, walk.crosswalks, roads.segments, roads.path_segments, ROUND(roads.segment_m) AS segment_m, pl.places, pl.places_in_library, dem.dem_tile, ROUND(dem.surface_m, 1) AS surface_m, ROUND(dem.dem_min_m, 1) AS dem_min_m, ROUND(dem.dem_max_m, 1) AS dem_max_m FROM lib CROSS JOIN bld CROSS JOIN walk CROSS JOIN roads CROSS JOIN pl CROSS JOIN dem ``` The query reads the raster surface model and four vector tables in WherobotsDB: raster and vector in one engine; most stacks pick one. It returns one row in about 36 seconds, the context packet a planner needs for the drop-off: | Field | Value | | --- | --- | | Overture Global Entity Reference System (GERS) building ID | 45837714-76fa-43b7-8e93-3988558a9e68 | | Library height | 12.9 m | | Buildings within 122 m (with a height value) | 9 (8) | | Tallest building within 122 m | Jimenez Hall, 17.3 m | | Tile2Net sidewalk area within 122 m | 8,240 m² in 5 polygons | | Tile2Net crosswalks | 3 | | Overture segments within 122 m (of which footways, paths, steps, cycleways) | 133 (130), 3,253 m in all | | Places within 122 m (inside the library) | 14 (2) | | Surface height at the library centroid, and range within 122 m | 52.8 m, 37.4 to 59.6 m | ![Map of the University of Maryland campus between Stamp Student Union and McKeldin Library. Overture building footprints are shaded by height. Tile2Net sidewalk polygons are teal and Tile2Net crosswalks yellow. Overture footways and Campus Drive are drawn as lines, and place points are pink. A dashed 122 meter circle surrounds the library, and a dotted straight line runs from the student union to the library. A card lists 9 buildings within 122 meters, with Jimenez Hall the tallest at 17.3 meters. It also lists 8,240 square meters of sidewalk, 3 crosswalks, 130 path segments, and 14 places. For the straight line, it lists 244.2 meters, 3 buildings crossed, and 11 segments crossed](https://wherobots.com/wp-content/themes/wherobots/assets/img/discover/physical-ai-delivery-context-map.webp) *Context for a delivery to McKeldin Library: 9 buildings, 3 crosswalks, and 14 places within 122 m. A 244.2 m straight line crosses 3 buildings and 11 path and road segments. Source: Overture Maps, Tile2Net on NAIP imagery, and Copernicus GLO-30 in wherobots_open_data.* Jimenez Hall, at 17.3 m, is the tallest structure within 400 feet. That height is the input to the Part 107 structure rule for a drone over the drop-off. The one building with no height stays an unknown. The 130 path segments (unfiltered, steps included) and 3 crosswalks are the inventory a sidewalk robot's route planner starts from. The 2 places inside the footprint identify where in the building the order goes. A second query, on four of the same tables, checks the straight line from Adele H. Stamp Student Union, the campus student center, to the library. It returns the buildings and segments the line crosses, its length on Tile2Net sidewalk, and the surface model every 10 m: ``` WITH r AS ( SELECT ST_MakeLine(ST_Point(-76.94474, 38.98814), ST_Point(-76.94512, 38.98596)) AS ll, ST_Transform(ST_SetSRID(ST_MakeLine(ST_Point(-76.94474, 38.98814), ST_Point(-76.94512, 38.98596)), 4326), 'EPSG:4326', 'EPSG:32618') AS utm ), bld AS ( SELECT /*+ BROADCAST(r) */ COUNT(*) AS buildings_crossed, concat_ws(', ', collect_list(concat(coalesce(o.names.primary, 'unnamed'), ' ', coalesce(CAST(ROUND(o.height, 1) AS STRING), 'no height'), ' m'))) AS crossed FROM wherobots_open_data.overture_maps_foundation.buildings_building o CROSS JOIN r WHERE o.bbox.xmin < -76.9445 AND o.bbox.xmax > -76.9455 AND o.bbox.ymin < 38.9883 AND o.bbox.ymax > 38.9858 AND ST_Intersects(o.geometry, r.ll) ), walk AS ( SELECT /*+ BROADCAST(r) */ ST_Length(ST_Intersection(r.utm, ST_Union_Aggr(CASE WHEN t.layer = 'sidewalk' THEN ST_Transform(ST_SetSRID(t.geometry, 4326), 'EPSG:4326', 'EPSG:32618') END))) AS on_sidewalk_m, SUM(CASE WHEN t.layer = 'crosswalk' THEN 1 ELSE 0 END) AS crosswalks_crossed FROM wherobots_open_data.rasterflow_output_samples.tile2net_vector t CROSS JOIN r WHERE t.xmin < -76.9445 AND t.xmax > -76.9455 AND t.ymin < 38.9883 AND t.ymax > 38.9858 AND ST_Intersects(t.geometry, r.ll) GROUP BY r.utm ), roads AS ( SELECT /*+ BROADCAST(r) */ COUNT(*) AS segments_crossed, concat_ws(', ', collect_list(concat(coalesce(s.names.primary, 'unnamed'), ' (', s.class, ')'))) AS segment_list FROM wherobots_open_data.overture_maps_foundation.transportation_segment s CROSS JOIN r WHERE s.bbox.xmin < -76.9445 AND s.bbox.xmax > -76.9455 AND s.bbox.ymin < 38.9883 AND s.bbox.ymax > 38.9858 AND s.subtype = 'road' AND ST_Intersects(s.geometry, r.ll) ), pts AS ( SELECT d * 10 AS dist_m, ST_Transform(ST_SetSRID(ST_LineInterpolatePoint(r.utm, LEAST(d * 10 / ST_Length(r.utm), 1.0)), 32618), 'EPSG:32618', 'EPSG:4326') AS pt FROM r LATERAL VIEW explode(sequence(0, CAST(FLOOR(ST_Length(r.utm) / 10) AS INT))) e AS d ), prof AS ( SELECT /*+ BROADCAST(pts) */ collect_list(struct(pts.dist_m, ROUND(RS_Value(t.rast, pts.pt), 1) AS dsm_m)) AS profile FROM wherobots_open_data.copernicus_dem.glo_30m t CROSS JOIN pts WHERE ST_Intersects(t.footprint, ST_Point(-76.945, 38.987)) ) SELECT ROUND(ST_Length(r.utm), 1) AS route_m, bld.buildings_crossed, bld.crossed, ROUND(walk.on_sidewalk_m, 1) AS on_sidewalk_m, walk.crosswalks_crossed, roads.segments_crossed, roads.segment_list, array_sort(prof.profile) AS profile FROM r CROSS JOIN bld CROSS JOIN walk CROSS JOIN roads CROSS JOIN prof ``` The line is 244.2 m long and returns in about 32 seconds. It crosses 3 buildings: the student union (15.6 m), the University of Maryland (UMD) Health Center (10.6 m), and the library (12.9 m). It crosses 11 Overture segments: Campus Drive, a tertiary road, plus 8 footways, 1 cycleway, and 1 flight of steps. Only 41.5 m of it lies on Tile2Net sidewalk, and it crosses none of the Tile2Net crosswalks. ![Step chart of Copernicus surface height every 10 meters along the 244.2 meter straight line from Stamp Student Union to McKeldin Library. Height starts at 59.5 meters, drops to a low of 46.5 meters near 200 meters, and jumps to 59.6 meters over the library footprint. Shaded bands mark where the line crosses the student union, the UMD Health Center, and the library. Tick marks show where it crosses Campus Drive, footways, a cycleway, and steps](https://wherobots.com/wp-content/themes/wherobots/assets/img/discover/physical-ai-route-surface-profile.webp) *Surface height along the straight line, from the Copernicus GLO-30 surface model sampled every 10 m. It reads 59.5 m at the start, 46.5 m at the lowest sample, and 59.6 m over the library footprint.* A drone can fly the straight line if it clears the roofs. A wheeled robot cannot. The line runs through three buildings, across a road at a point with no Tile2Net crosswalk, and down a flight of steps. The robot has to route on the footway network. The 30 m surface model mixes ground, roofs, and tree canopy. Slope checks for a sidewalk robot need finer elevation, such as [lidar](https://wherobots.com/discover/lidar-data/). ## Physical AI research and what changes at scale Physical AI research runs through four threads: embodied robotics, autonomous driving, learned world models, and AI for geographic data. ![Timeline with four lanes. Robotics and embodied AI: Brooks 1991, Miriyev and Kovač 2020, RT-2 and Open X-Embodiment 2023, pi zero and Gemini Robotics 2025. Autonomous driving: Stanley and the SLAM tutorial 2005 to 2006, KITTI 2012, the Waymo Open Dataset and an HD map review 2020. World models and simulation: domain randomization and CARLA 2017, World Models and Habitat 2018 to 2019, Cosmos, DreamerV3 and Genie 3 2025. Geographic AI: GeoSpark 2015, GeoAI 2020, Prithvi and Tile2Net 2023, AlphaEarth Foundations 2025](https://wherobots.com/wp-content/themes/wherobots/assets/img/discover/physical-ai-research-timeline.webp) *Four research threads that meet in physical AI, with the landmark papers and releases cited in this article.* ### From behavior-based robots to robot foundation models Rodney Brooks argued in [Intelligence without representation](https://doi.org/10.1016/0004-3702%2891%2990053-M) (Artificial Intelligence, 1991) that intelligence comes from layers of simple behaviors coupled to the world through sensors, with no central symbolic model. SLAM and Stanley took the opposite approach, with an explicit, uncertain map of the surroundings. Robot learning then added pretraining on web data and pooled robot data, as in RT-2 and Open X-Embodiment. The [Cosmos platform paper](https://arxiv.org/abs/2501.03575) frames the current step. It says a physical AI system needs two digital twins. One is a twin of the system itself (the policy). The other is a twin of the world (the world model). ### AI for geographic data Krzysztof Janowicz and colleagues argued that models for geographic problems need location, distance, and spatial relations built in. They made this argument in [GeoAI: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond](https://doi.org/10.1080/13658816.2019.1684500) (IJGIS, 2020). The [GeoLLM study](https://arxiv.org/abs/2310.06213) (Manvi and colleagues, ICLR 2024) found that prompting with coordinates alone gives poor estimates. Adding nearby OpenStreetMap features to the prompt improved results by 70% over baselines. Vision models now build part of the context layer. Maryam Hosseini and colleagues' [Mapping the walk](https://doi.org/10.1016/j.compenvurbsys.2023.101950) (Computers, Environment and Urban Systems, 2023) introduced Tile2Net. It detects roads, sidewalks, and crosswalks in aerial image tiles and converts them into a connected sidewalk network. The College Park sidewalks above come from Tile2Net. ### Computing spatial context at scale One delivery takes a handful of joins. A fleet serving a metro area repeats them for millions of locations over tables with billions of rows, and again when the map changes. Jia Yu, Jinxuan Wu, and Mohamed Sarwat introduced [GeoSpark](https://doi.org/10.1145/2820783.2820860) (ACM SIGSPATIAL 2015) to run spatial joins across a cluster. Yu, Zongsi Zhang, and Sarwat described its spatial partitioning, indexing, and join design in [Spatial data management in Apache Spark: the GeoSpark perspective and beyond](https://doi.org/10.1007/s10707-018-0330-9) (GeoInformatica, 2019). GeoSpark became Apache Sedona, and Yu and Sarwat went on to found Wherobots. A detector such as Tile2Net runs over every tile of an area before any join can use its output. [Change detection](https://wherobots.com/discover/change-detection/) on [Earth observation data](https://wherobots.com/discover/earth-observation-data/) marks the tiles that need reprocessing. ### Open problems - **The sim-to-real gap.** Policies trained in simulation still fail on textures, lighting, and physics the simulator never produced. - **The long tail.** Rare events such as an unmapped construction zone or a fallen tree carry much of the safety risk and appear rarely in any dataset. - **Map freshness.** HD maps, sidewalk networks, and building heights age, so every fact needs a date and a source. - **Data gaps.** Open building heights are uneven, and aerial sidewalk extraction misses paths under tree canopy. - **Evaluation.** The benchmarks cited here test perception or manipulation in isolation. None of them tests whether a system used the right spatial context for a real place. ## Conclusions The walkthrough shows what physical AI needs before a machine acts at one real place: facts joined from several datasets around it. 1. **Physical AI needs spatial context built per place, from joins.** Every value in the McKeldin Library packet came from joins around one point. 2. **The same joins show what each machine can do.** A drone can use the straight line only if it clears the roofs. A wheeled robot meets three buildings, a road crossing with no Tile2Net crosswalk, and steps. If the output is an action, the context has to carry the constraints that forbid it. 3. **Physical AI needs elevation at the resolution of the decision.** A 30 m surface model mixes ground, roofs, and canopy. Sidewalk slope needs finer elevation. 4. **A physical AI planner has to carry unknowns as unknowns.** One of 9 buildings has no height, and aerial extraction can miss sidewalks under canopy. 5. **Physical AI at fleet scale needs distributed joins.** A fleet repeats one delivery's joins for millions of locations. Run them across a cluster, and reprocess only changed tiles. 6. **Map context may help physical AI plan. That is unproven.** Nearby map features improved language model estimates by 70% in the GeoLLM study, on its benchmarks. Whether that gain carries to action planning is open. ## Related terms - [Context engineering](https://wherobots.com/discover/context-engineering/): assembling the spatial facts a model or agent needs about a place - [Ontology](https://wherobots.com/discover/ontology/): the classes and relations that let AI agents reason about places - [Geospatial AI](https://wherobots.com/discover/geospatial-ai/): machine learning and AI agents applied to spatial data - [Earth observation data](https://wherobots.com/discover/earth-observation-data/): satellite imagery that many physical AI models use - [Change detection](https://wherobots.com/discover/change-detection/): finding what changed in imagery over time - [LiDAR data](https://wherobots.com/discover/lidar-data/): 3D point clouds robots and vehicles use to sense their surroundings - [POI (point of interest) data](https://wherobots.com/discover/poi-data/): the places where a delivery or trip ends - [Havasu Catalog](https://wherobots.com/discover/havasu-catalog/): the Wherobots knowledge lake of physical world data, models, and apps ## Read more from Wherobots - [How Wherobots builds with NVIDIA to let AI see the physical world](https://wherobots.com/blog/wherobots-nvidia-geospatial-ai-physical-world/) - [Introducing developer tools that let AI build with physical world data](https://wherobots.com/blog/spatial-ai-coding-tools/) - [Detecting objects from text prompts with RasterFlow and Segment Anything 3](https://wherobots.com/blog/sam3-earth-observation-rasterflow/) - [Spatial Graph RAG for the Physical World](https://wherobots.com/blog/spatial-graph-rag/) Connect your AI agent to physical world data through the Wherobots MCP server. Start a free trial at [cloud.wherobots.com](https://cloud.wherobots.com). ## Frequently asked questions ### What is an example of physical AI? A self-driving car is physical AI: it senses the road with cameras and lidar, predicts what other road users will do, and steers. Other examples are warehouse robots that pick objects of different shapes, sidewalk delivery robots, delivery drones, and models that map roof damage across a city from aerial imagery. ### What companies are working on physical AI? NVIDIA builds Cosmos world foundation models, Omniverse, and Isaac. Google DeepMind and Physical Intelligence build robot foundation models. Waymo operates autonomous vehicles, and Tesla builds the Optimus humanoid robot. Satellite operators such as Planet and Maxar capture Earth imagery. The Overture Maps Foundation publishes open map data. Wherobots provides the spatial data and compute that AI models and agents use to work with real places. ### What is the difference between physical AI and generative AI? Generative AI produces text, images, code, and video. Physical AI perceives, reasons about, and acts in the physical world. Its outputs must match physics and geography: a robot path, a steering command, or a flood map tied to real locations. Many physical AI systems use generative models inside them, for example world foundation models that generate synthetic training video. ### What is the difference between physical AI and agentic AI? Agentic AI plans and carries out multistep tasks, usually in software: it calls tools, queries data, and writes files. Physical AI acts in the physical world or makes decisions about it. The two meet when an AI agent queries physical world data, for example to plan a delivery route or check which buildings sit in a flood zone. ### Is physical AI the same as robotics? No. Robots, drones, and autonomous vehicles are embodied AI, the subset of physical AI that runs inside a body. Physical AI also covers models without a body, such as a platform that maps crop health or roof damage from satellite imagery. ### What data does physical AI use? Physical AI runs on spatial data: raster data such as satellite imagery and elevation models, and vector data such as roads, sidewalks, building footprints, and parcels. Sensor feeds, lidar point clouds, and GPS traces record current conditions at a place. Driving and robot datasets such as KITTI, the Waymo Open Dataset, and Open X-Embodiment supply training examples. ### What is a world foundation model? A world foundation model predicts how a scene changes after an action. It is first trained on large collections of video, then trained further for a specific robot, vehicle, or environment. Developers use world foundation models such as NVIDIA Cosmos to generate physics-based synthetic data for training robots and autonomous vehicles in simulation. For how they differ from the world models used in reinforcement learning and weather forecasting, see what is a world model. ### What is NVIDIA physical AI? NVIDIA uses the term physical AI for its platforms that train and run robots and autonomous vehicles. The platforms are Cosmos world foundation models (announced at CES on 6 January 2025), Omniverse for digital twins and simulation, and Isaac for robotics. The Cosmos paper describes a video curation pipeline, video tokenizers, and pretrained world models released with open weights. ### What is a vision-language-action model? A vision-language-action (VLA) model takes camera images and a text instruction and outputs robot actions. Google DeepMind introduced the term with RT-2 in 2023. Later models apply it to dexterous work: Physical Intelligence’s π0 folds laundry, and Gemini Robotics extends Google’s Gemini 2.0 model to robot control. ### Is ChatGPT physical AI? No. A chat assistant is generative AI. It produces text from text, documents, databases, and the internet it was trained on. It has no sensors or actuators (the motors and other parts that make a machine move). Language and vision-language models do appear inside physical AI systems, for example as the backbone of a vision-language-action model that controls a robot. ### How do I connect an AI agent to physical world data? The Wherobots Model Context Protocol (MCP) server connects AI coding tools such as Claude Code and Cursor to spatial compute and open datasets. An agent can then find data, write a spatial query, and run it from a plain-language request. Start a free trial at [cloud.wherobots.com](https://cloud.wherobots.com).