What is Context Engineering for Physical AI?

Context engineering is the practice of choosing, formatting, and maintaining the information a large language model receives when it runs: instructions, tool definitions, retrieved data, memory, and conversation history. Anthropic describes it in Effective context engineering for AI agents as the set of strategies for curating and maintaining the optimal set of tokens while the model runs. Tokens are the words and word pieces a model reads. Context engineering for physical AI applies the same practice to questions about real places, where the context is geometry, location, imagery, and the spatial relations between them.

Key takeaways

• Context engineering sets which tokens a model reads at each step of its work.
• Prompt engineering writes the instructions. Context engineering also covers retrieval, tools, memory, and compression across an agent's whole run.
• Model accuracy grows less reliable with longer input, so send only the tokens the task needs.
• For physical AI, the context comes from spatial joins, which match records by location, across buildings, places, roads, boundaries, hazards, and imagery. Each layer has its own scale, update cycle, coordinate system, and identifiers.
• A spatial engine computes that context in a query and returns a short, sourced summary of one place.
• Distances belong in meters. At the Ferry Building, a fixed degree-to-meter conversion overstates the gap to the nearest road by about 15%.

What is context engineering?

A model's context is the set of tokens it reads when it generates a response. Anthropic's Applied AI team calls context engineering "the natural progression of prompt engineering." The work moves from writing one good prompt to managing everything that enters the context window, the limited number of tokens a model can read at once. In an agent, that window changes with every tool call. Anthropic's guiding principle is to find "the smallest possible set of high-signal tokens that maximize the likelihood of some desired outcome."

Andrej Karpathy backed the term on 25 June 2025. He described context engineering as "the delicate art and science of filling the context window" with the right information for the next step. Phil Schmid's breakdown of context lists the parts that fill that window:

• Instructions: the system prompt, rules, and examples.
• The user prompt: the task at hand.
• State and history: the current conversation.
• Long-term memory: facts and preferences kept across sessions.
• Retrieved information: documents, database rows, and API results.
• Available tools: the functions the model can call, with their descriptions.
• Structured output: the format the answer must follow.

IBM's explainer frames context engineering as the framework that brings prompt engineering and retrieval-augmented generation (RAG) together. It also counts data from structured sources as context.

Why longer context can hurt

Context windows now hold hundreds of thousands of tokens, and filling them has a cost. Liu and colleagues published Lost in the Middle (Transactions of the Association for Computational Linguistics, 2024). They showed that accuracy is highest when the relevant passage sits at the start or end of a long input. Accuracy drops when the passage sits in the middle. Chroma's Context Rot report (Hong, Troynikov, and Huber, 2025) tested 18 models. It held task difficulty constant and varied only input length. Performance grew less reliable as input grew, even on simple tasks. Anthropic calls this a finite attention budget. Every extra row or paragraph uses part of that budget, so the context should hold only what the task needs.

Context engineering vs prompt engineering vs RAG

Context engineering contains the other two. Prompt engineering is one input to it, and RAG is one way to fill it.

	Prompt engineering	RAG	Context engineering
What it shapes	The instructions	Retrieved passages or rows	Every token in the window, at every step
When it happens	Written once, before the run	At query time	At each step of an agent loop
Main techniques	Role, rules, examples, output format	Embedding search, reranking	Retrieval, tool design, memory, compaction, sub-agents
Typical failure	Ambiguous instructions	Relevant passage missed	Too much, stale, or conflicting context

RAG comes from Lewis and colleagues' Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (NeurIPS 2020). It paired a language model with a neural retriever that searched Wikipedia passages, stored as numeric vectors, for text close in meaning to the question. Context engineering keeps retrieval and adds the decisions around it: what to keep, what to drop, and when to fetch more.

How context engineering works

LangChain's context engineering guide groups the techniques into four strategies: write, select, compress, and isolate. Anthropic's article describes the same moves:

1. Retrieve on demand. The agent keeps short identifiers, such as file paths, stored queries, or links. When a step needs the data behind one, the agent loads it with a tool call. Claude Code, Anthropic's coding agent, loads its CLAUDE.md instruction files at the start. It finds other files as it works with glob and grep, two file search commands.
2. Compact. When a conversation nears the window limit, the agent summarizes it and continues from the summary.
3. Write notes. The agent saves progress and open questions to memory outside the window and reads them back later.
4. Isolate with sub-agents. A sub-agent runs one subtask in its own clean window and returns a short result.

Tools are how an agent selects context. Anthropic open-sourced the Model Context Protocol (MCP) on 25 November 2024. MCP is a standard for connecting AI assistants to the systems where data lives. Any application that supports MCP can call the tools a data source offers.

Context engineering examples

• Coding agents. An agent reads the project's instruction file, searches the repository with grep, opens only the matching files, and runs tests to check its change.
• Customer support. A support agent retrieves the customer's account, the matching help articles, and the ticket history.
• Deep research. A lead agent sends sub-agents to read sources in parallel, and each returns a short summary with citations.
• Property questions. An agent asked about one building needs its footprint, the businesses inside it, the road that reaches it, and the boundaries and hazards around it. Those facts rarely sit in one document, so a spatial query has to compute them.

Context engineering for physical AI

Physical AI covers AI that perceives, reasons about, and acts in the physical world, from robots to models that read satellite imagery. Language models were trained on text, documents, databases, and the internet. Those sources rarely state which building sits in which flood zone. The GeoLLM study (Manvi and colleagues, ICLR 2024) found that prompting with coordinates alone gives poor estimates. Adding nearby OpenStreetMap features to the prompt improved results by 70% over baselines.

Location is the join key. A text agent matches context by keyword or meaning. An agent answering a place question matches it by position: the places inside a building, the nearest road, the county around it. Each relation comes from a spatial join, which matches rows from two tables by their geometry. The join uses tests such as ST_Contains (one shape inside another), ST_Intersects (shapes overlap or touch), and ST_KNN (nearest neighbors).

The layers come in different shapes. One building draws on polygons (footprints, parcels, flood zones, boundaries), lines (roads), points (businesses), and rasters (elevation, imagery). Geospatial data mixes these types. The context step reduces each one to a few facts a model can read.

Freshness. Each layer updates on its own cycle. Overture publishes a map release every month, parcel records follow county schedules, flood maps carry an effective date, and weather warnings carry an expiration time. A context packet is the set of facts handed to the model for one place. When the packet records each source's date, the reader can judge what is current. The ORATOR comparison below shows two dated sources for one building.

Coordinate systems. A latitude and longitude only have meaning in a stated coordinate reference system (CRS), the definition that ties coordinates to places on Earth. Overture stores geometry in longitude and latitude (EPSG:4326). A flat-plane distance on those coordinates comes out in degrees. A degree of longitude is about 111 km at the Equator and 88.1 km at the Ferry Building's 37.8° N, where a degree of latitude is 111.0 km. The gap between the Ferry Building and The Embarcadero, its nearest primary road, is 0.00021 degrees. Multiplying by a fixed 111.32 km per degree gives 23.4 m. In Universal Transverse Mercator (UTM) zone 10N, a projected CRS laid out as a flat grid in meters, the gap between the closest points is 20.4 m. The distance between those points on the World Geodetic System 1984 (WGS 84) ellipsoid, the model of Earth's shape that GPS uses, is also 20.4 m. The fixed conversion overstates the gap by about 15%, and the error grows toward the poles. The context step should do this math in the engine, with ST_DistanceSphere, ST_DistanceSpheroid, or a projected CRS, and pass the model meters.

Stable IDs. Each fact in the context needs an ID that a reader can cite and an agent can use to fetch more. Overture's Global Entity Reference System gives each feature an ID that stays stable across releases, so a building ID in the context can be fetched again next month. Grid systems such as H3 divide the Earth into cells, each with an ID. Two datasets with no common key can be joined on the cell ID.

A spatial knowledge graph stores these relations ahead of time as typed edges: links between two features, labeled with the relation, such as located_in. An ontology defines what each edge type means. Spatial Graph RAG for the physical world works through that approach. Spatial SQL builds the edges. A graph query then follows edges out from one feature and returns the linked features as context.

Context engineering in Wherobots

Wherobots is the Context Engine for Physical World Data, built for context engineering for physical AI. WherobotsDB runs the spatial joins. The Wherobots MCP server lets an AI agent find tables, write spatial SQL, and run it. The Havasu catalog holds the layers, such as Overture buildings, Overture places, and the Copernicus 30 m elevation model.

The two queries below assemble context for the San Francisco Ferry Building. They read six tables in wherobots_open_data. Row counts come from COUNT(*) on each table in October 2026.

Source	Table	Format	Rows	What the query read
Overture buildings	overture_maps_foundation.buildings_building	Vector polygons	2,533,842,612	The Ferry Building's footprint, height, and footprint source
Overture places	overture_maps_foundation.places_place	Vector points	81,455,423	The count of places inside the footprint
Overture transportation segments	overture_maps_foundation.transportation_segment	Vector lines	352,054,710	Road segments of six classes near the building, and the nearest one
Overture division areas	overture_maps_foundation.divisions_division_area	Vector polygons	1,080,667	The land division areas that contain the building's centroid
Copernicus GLO-30 elevation model	copernicus_dem.glo_30m	Raster tiles from Cloud Optimized GeoTIFF files	3,961,635 tiles	The surface height at the centroid
National Weather Service (NWS) watches and warnings	noaa.nws_watch_warnings	Vector polygons	944,342	Alerts from the MTR office that cover the centroid (second query)

The first query measures road distance in meters in UTM zone 10N and orders the divisions from country down. Bounding box filters, checks on each row's minimum and maximum coordinates, keep each table read to the blocks around the building. One query reads the raster and joins the vector layers: raster and vector in one engine; most stacks pick one.

WITH b AS (
  SELECT id, names.primary AS name, height, geometry, ST_Centroid(geometry) AS c,
         ST_Transform(ST_SetSRID(geometry, 4326), 'EPSG:4326', 'EPSG:32610') AS utm,
         sources[0].dataset AS footprint_source, sources[0].update_time AS footprint_updated
  FROM wherobots_open_data.overture_maps_foundation.buildings_building
  WHERE bbox.xmin > -122.396 AND bbox.xmax < -122.391
    AND bbox.ymin > 37.794 AND bbox.ymax < 37.797
    AND names.primary = 'San Francisco Ferry Building'
),
places AS (
  SELECT COUNT(*) AS n
  FROM wherobots_open_data.overture_maps_foundation.places_place p
  JOIN b ON ST_Contains(b.geometry, p.geometry)
  WHERE p.bbox.xmin > -122.396 AND p.bbox.xmax < -122.391
    AND p.bbox.ymin > 37.794 AND p.bbox.ymax < 37.797
),
road AS (
  SELECT s.id AS road_id, s.names.primary AS road_name, s.class AS road_class,
         ST_Distance(b.utm, ST_Transform(ST_SetSRID(s.geometry, 4326), 'EPSG:4326', 'EPSG:32610')) AS road_m
  FROM wherobots_open_data.overture_maps_foundation.transportation_segment s
  CROSS JOIN b
  WHERE s.bbox.xmin > -122.400 AND s.bbox.xmax < -122.387
    AND s.bbox.ymin > 37.792 AND s.bbox.ymax < 37.799
    AND s.subtype = 'road'
    AND s.class IN ('motorway', 'trunk', 'primary', 'secondary', 'tertiary', 'residential')
  ORDER BY road_m
  LIMIT 1
),
divs AS (
  SELECT array_sort(collect_list(struct(
           CASE d.subtype WHEN 'country' THEN 1 WHEN 'region' THEN 2
                WHEN 'county' THEN 3 ELSE 4 END AS rank,
           d.names.primary AS name, d.division_id AS division_id))) AS levels
  FROM wherobots_open_data.overture_maps_foundation.divisions_division_area d
  JOIN b ON ST_Contains(d.geometry, b.c)
  WHERE d.bbox.xmin < -122.393 AND d.bbox.xmax > -122.394
    AND d.bbox.ymin < 37.795 AND d.bbox.ymax > 37.796
    AND d.class = 'land'
),
dem AS (
  SELECT t.name AS dem_tile, RS_Value(t.rast, b.c) AS surface_m
  FROM wherobots_open_data.copernicus_dem.glo_30m t
  JOIN b ON ST_Intersects(t.footprint, b.c)
)
SELECT b.id, b.name, b.height, ROUND(ST_AreaSpheroid(b.geometry)) AS footprint_m2,
       b.footprint_source, b.footprint_updated,
       places.n AS places_inside,
       road.road_id, road.road_name, road.road_class, ROUND(road.road_m, 1) AS road_m,
       concat_ws(' > ', transform(divs.levels, x -> x.name)) AS divisions,
       transform(divs.levels, x -> x.division_id) AS division_ids,
       dem.dem_tile, ROUND(dem.surface_m, 1) AS surface_m
FROM b CROSS JOIN places CROSS JOIN road CROSS JOIN divs CROSS JOIN dem

It returns one row, the context packet for the building:

Field	Value
id	68b6339c-54c7-4736-a228-01345e78bcdd
height	15 m
footprint_m2	9,900
footprint_source, footprint_updated	OpenStreetMap, 31 August 2026
places_inside	205
road_id	3bbd1042-26cc-4251-b785-0f610be864d8
road_name, road_class, road_m	The Embarcadero, primary, 20.4 m
divisions	United States > California > San Francisco
division_ids	f39eb4af-5206-481b-b19e-bd784ded3f05, 4c555166-326b-48b4-8a33-155eb7c23e8d, 713e9077-600c-4f06-84b9-6bae6e559c34
dem_tile, surface_m	Copernicus_DSM_COG_10_N37_00_W123_00_DEM.tif, 10.8

The 205 places span 49 categories, led by financial services (30), law firms (24), restaurants (19), and legal services (19). The Copernicus elevation model is a digital surface model, so the surface value includes the roof as well as the ground. A second query adds a hazard layer from NWS watches and warnings. In all, 102 National Weather Service alert polygons from the San Francisco Bay Area office (MTR) cover the building's centroid. They include 76 areal flood advisories, coded FA.Y in the Valid Time Event Code (VTEC) system, issued between February 2009 and November 2025:

SELECT PHENOM, SIG, COUNT(*) AS warnings, MIN(ISSUED) AS first_issued, MAX(ISSUED) AS last_issued
FROM wherobots_open_data.noaa.nws_watch_warnings
WHERE WFO = 'MTR'
  AND ST_Intersects(geometry, ST_Point(-122.39344, 37.79553))
GROUP BY PHENOM, SIG
ORDER BY warnings DESC

Within 400 m of the building's centroid sit 79 buildings, 1,469 places, and 429 road segments, 1,977 rows in all. Sending all of them would spend the attention budget on records the question does not need. Each feature fact carries its Overture ID, and the footprint and elevation values carry their source, so the agent can cite them and fetch more. The 205 places are a count over the building's geometry, so the building ID is the key that fetches them. The same pattern extends to parcels and flood zones, each one more join on the same geometry.

Radius and freshness on the Ferry Building

The packet above reduced 1,977 rows to 8 facts. Two more choices shape a packet like that: how far from the building to search, and how current each source is. The radius counts below come from Overture Maps release 2026-09-23.1, read from Overture's public bucket, with distances computed in UTM zone 10N. The graph counts come from Wherobots queries on wherobots_open_data.

How far to search

The count of nearby features grows fast with distance from the footprint.

Distance from the footprint	Places	Road segments	Other buildings
50 m	242	75	4
100 m	267	133	11
200 m	685	247	28
300 m	1,090	360	68

The growth is uneven: places jump from 335 to 554 between 150 m and 175 m. The radius is a property of the question. Road access needs a few tens of meters, foot traffic needs a few hundred, and a flood or wildfire question needs the extent of the hazard. Tobler's first law of geography (1970) holds that everything is related to everything else, and near things more than distant things. That supports starting small and widening only when the question calls for it. The sources cited here give no general rule for the distance, and choosing it remains an open problem.

How current each source is

Each fact in a packet has its own date, and derived data ages too. Overture places carry their own source records and confidence scores. The ORATOR knowledge graph that Wherobots publishes as wherobots_open_data.spatial_knowledge_graph carries an update time of 18 June 2026 on its Ferry Building node. It links 112 places to the building through its located_in relation: 86 inside the building polygon at confidence 1.0 and 26 matched by proximity, as What is an ontology? describes. The September 2026 Overture release has 205 place points inside the same footprint. The counts differ in date and in method. The building's GERS ID, 68b6339c-54c7-4736-a228-01345e78bcdd, is the same in both, so an agent holding the graph's ID can fetch the current record.

Context engineering research

Context engineering for physical AI draws on three research threads: retrieval for language models, GeoAI, and distributed spatial computing.

From retrieval to context engineering

The retriever in RAG came from Karpukhin and colleagues' Dense Passage Retrieval (EMNLP 2020). It beat BM25, a keyword ranking method, by 9 to 19 points of top-20 passage accuracy on open-domain question answering. As windows grew, the Lost in the Middle and Context Rot studies measured where the relevant passage sits and how much of the input helps. Edge and colleagues at Microsoft Research introduced GraphRAG (2024). It builds a knowledge graph from a text corpus and summarizes clusters of related entities. It then answers questions about the whole corpus from those summaries. The Model Context Protocol (2024) then standardized how agents call data sources, and Anthropic's 2025 article set out on-demand retrieval, compaction, notes, and sub-agents as the working methods.

Each step moved more of the work from the model to the system around it.

Why place questions need computed context

Geography research reached the same conclusion: location carries information that text alone does not. Janowicz, Gao, McKenzie, Hu, and Bhaduri argued that models for geographic problems need location, distance, and spatial relations built in. They made this argument in GeoAI: spatially explicit artificial intelligence techniques for geographic knowledge discovery and beyond (IJGIS, 2020).

Geographic knowledge in language models is uneven. Yamada and colleagues' Evaluating Spatial Understanding of Large Language Models (Transactions on Machine Learning Research, 2024) tested navigation over grids, rings, and trees. They found that accuracy varied with the structure and with how it was described. Mai and colleagues wrote a vision paper, On the Opportunities and Challenges of Foundation Models for GeoAI (ACM Transactions on Spatial Algorithms and Systems, 2024). It surveys where general models fall short on geospatial tasks. It calls for multimodal models that combine text, imagery, and vector data.

Spatial retrieval and geo-enrichment

KnowWhereGraph, described by Janowicz and colleagues in Know, Know Where, KnowWhereGraph (AI Magazine, 2022), pairs a cross-domain geographic knowledge graph with a geo-enrichment service. For a region of interest, the service returns linked data on natural hazards, soils, climate, and demographics: a context packet for a place, served as a graph. Yu and colleagues' Spatial-RAG (2025) answers geospatial questions by pairing a spatial database with a language model. A spatial filter finds candidates by location. A semantic match ranks them by meaning. The final answer weighs both.

What scale changes

One address takes a handful of joins. A portfolio of a million addresses takes the same joins a million times over tables with billions of rows. The joins have to run in a distributed engine, with results cached by stable ID. The radius multiplies the cost. For the Ferry Building, widening the search from 50 m to 300 m raises the place count from 242 to 1,090. A portfolio repeats that choice for every building.

Jia Yu, Jinxuan Wu, and Mohamed Sarwat introduced GeoSpark (ACM SIGSPATIAL 2015) to run spatial joins like these across a cluster. Yu, Zongsi Zhang, and Sarwat described its spatial partitioning, indexing, and join design in Spatial data management in Apache Spark: the GeoSpark perspective and beyond (GeoInformatica, 2019). GeoSpark became Apache Sedona, and Yu and Sarwat went on to found Wherobots.

Open problems

• Conflicting sources. Overture, parcel records, and a county's own data can disagree on a building's height or use. A packet that keeps each value with its source puts the conflict in plain view.
• Time. Most of the layers above hold one snapshot. Questions about change need versioned sources and a date on every fact.
• Evaluation. Benchmarks for spatial reasoning test models on puzzles and map questions. Few test whether an agent fetched the right context for a real place and used it correctly.

Conclusions

The Ferry Building walkthrough and the cited research give these rules for context engineering for physical AI.

1. Context engineering for physical AI reduces many nearby rows to a few facts. The Ferry Building packet reduced 1,977 nearby rows to 8 facts. The Context Rot tests found 18 models less reliable as input grew.
2. Place context needs distances computed in the engine, in meters. A fixed degree-to-meter conversion overstated the Ferry Building's gap to The Embarcadero by about 15% (23.4 m against 20.4 m). Use a projected CRS or a spheroid distance function.
3. Every fact in place context needs an ID. The Ferry Building's GERS ID is the same in the September 2026 Overture release and the June 2026 ORATOR graph. An agent holding either can fetch the current record.
4. The search radius is a context engineering choice, so set it from the question and state it. Places near the Ferry Building grew from 242 at 50 m to 1,090 at 300 m, and a portfolio repeats that choice for every building.
5. Place context needs a date on every fact. ORATOR's 112 linked places and Overture's 205 differ by date and method. A packet that carries each source's date lets the reader judge which is current.

Related terms

• Context engine: the system that assembles physical world data into context for AI
• Physical AI: AI that perceives, reasons about, and acts in the physical world
• Geospatial AI: machine learning and AI agents applied to spatial data
• Ontology: the classes and relations that give spatial context its meaning
• Spatial join: the operation that computes spatial relations between datasets
• Building footprint: the polygon at the center of most building context
• POI (point of interest) data: the businesses and services inside buildings
• Coordinate reference system: what gives a coordinate its location and units

Read more from Wherobots

• Spatial Graph RAG for the physical world
• Introducing developer tools that let AI build with physical world data
• Tutorial: How to use GERS IDs in Wherobots for geospatial data integration

Connect an AI agent to physical world data through the Wherobots MCP server with a Wherobots free trial at cloud.wherobots.com.