Detecting Objects From Text Prompts with RasterFlow

This notebook will guide you through detecting objects in aerial imagery using text prompts, powered by Wherobots RasterFlow and Meta’s Segment Anything Model 3 (SAM3). You will gain a hands-on understanding of how to run geometry inference on your selected area of interest, work with the detected geometries, and visualize the results in WherobotsDB.

SAM3

SAM3 is a text-prompted geometry inference model that detects objects in imagery based on natural language descriptions. Given a text prompt like "building" or "roofs", the model produces georeferenced vector geometries (bounding boxes or polygons) for each detected object, along with a confidence score.

Unlike segmentation models that produce raster outputs, SAM3 directly outputs vector geometries, making it straightforward to integrate results into geospatial workflows. We will demonstrate results using 30cm resolution data from the National Agriculture Imagery Program (NAIP).

Preview: model inputs and outputs

The interactive map linked below is an example run over Marion County, Oregon, showing the input imagery and model output at county scale. This notebook runs over a smaller AOI, College Park, Maryland, so your own output will cover a different area. Toggle layers in the sidebar to compare the input imagery with the model’s PMTiles output side-by-side.

Layers:

• SAM3 input mosaic: RGB NAIP aerial imagery the model runs on
• SAM3 PM Tiles: polygons output directly by the text-prompted model, delivered as PMTiles for fast rendering at scale (SAM3 is a geometry inference model, so there is no intermediate raster output)

View the interactive map here.

Selecting an Area of Interest (AOI)

To start, we will choose an Area of Interest (AOI) for our analysis where 30cm resolution NAIP data is available: College Park, Maryland.

import wkls
import geopandas as gpd
import os

# Generate a geometry for College Park, Maryland using Well-Known Locations (https://github.com/wherobots/wkls)
gdf = gpd.read_file(wkls.us.md.collegepark.geojson())

# Save the geometry to a parquet file in the user's S3 path
aoi_path = os.getenv("USER_S3_PATH") + "collegepark.parquet"
gdf.to_parquet(aoi_path)

Selecting a time range and verifying NAIP coverage

Because NAIP collects state-level imagery at mixed resolutions, SAM3’s required 30cm data may not exist for your selected AOI or timeframe. The USDA’s NAIP coverage map (PDF, 2002-2025) shows what is available where.

from datetime import datetime

# Date range for imagery to be used by the model
start_date = datetime(2023, 1, 1)
end_date = datetime(2024, 1, 1)

# The SAM3 recipes run on 30cm NAIP imagery. The index holds one row per NAIP scene, with its
# footprint, resolution (res) and acquisition time.
MODEL_RES = 0.3
naip_index = gpd.read_parquet(
    "s3://wherobots-examples/rasterflow/indexes/naip_index.parquet",
    columns=["geometry", "res", "year", "time"],
    storage_options={"anon": True},
)

# Scenes at the model's resolution touching the AOI, compared in the index's CRS
aoi_geom = gdf.geometry.to_crs(naip_index.crs).union_all()
at_res = naip_index.query(f"res == {MODEL_RES}")
nearby = at_res.iloc[at_res.sindex.query(aoi_geom, predicate="intersects")]

# Narrowed to the requested date range
covering = nearby.query(f"time >= '{start_date:%Y-%m-%d}' and time <= '{end_date:%Y-%m-%d}'")

# The recipe needs the AOI fully inside the matching scenes
if not covering.geometry.union_all().contains(aoi_geom):
    # How much of the AOI is covered, measured in an equal-area CRS
    area_crs = gdf.estimate_utm_crs()
    aoi_area = gdf.geometry.to_crs(area_crs).union_all()
    tiles = covering.geometry.to_crs(area_crs).union_all()
    fraction = max(0.0, 1.0 - aoi_area.difference(tiles).area / aoi_area.area)

    # Years that would work, applying the same full-coverage test as the gate above
    available = sorted(
        year
        for year, scenes in nearby.groupby("year")
        if scenes.geometry.union_all().contains(aoi_geom)
    )
    raise ValueError(
        f"{MODEL_RES:g}m NAIP covers {fraction:.1%} of this AOI between "
        f"{start_date:%Y-%m-%d} and {end_date:%Y-%m-%d}; the recipe needs the AOI fully covered. "
        + (
            f"Years with complete {MODEL_RES:g}m coverage over this AOI: {available}."
            if available
            else f"No year has complete {MODEL_RES:g}m NAIP coverage over this AOI."
        )
    )

print(f"NAIP coverage is available. Found {len(covering)} NAIP scenes at {MODEL_RES:g}m covering the AOI")
print(f"Acquisition years: {sorted(covering['year'].unique().tolist())}")

Initializing the RasterFlow client

from rasterflow_remote import RasterflowClient
from rasterflow_remote.data_models import GeometryModelRecipes

rf_client = RasterflowClient()

Running geometry inference

RasterFlow has pre-defined recipes that simplify orchestration of the processing steps for geometry inference. These steps include:

• Ingesting imagery for the specified Area of Interest (AOI)
• Generating a seamless mosaic from multiple image tiles
• Running text-prompted geometry inference with the SAM3 model

The output is a GeoDataFrame of detected geometries with confidence scores.

Note: The patch_size configured by InferenceConfig is always resized to 1008×1008 by the GeometryModelRecipes.SAM3_TEXT_GEOMETRY and GeometryModelRecipes.SAM3_TEXT_BBOX recipes. This means you can control the amount of spatial context passed to SAM3 in each pass, but selecting patch sizes larger than 1008×1008 will upsample the resolution.

Note: This step will take approximately 22 minutes to complete the first time it is run.

model_output = rf_client.predict_mosaic_geometries_recipe(
    # Path to our AOI in GeoParquet or GeoJSON format
    aoi=aoi_path,

    # Date range for imagery to be used by the model (set in the coverage check above)
    start=start_date,
    end=end_date,

    # Coordinate Reference System EPSG code for the output
    target_crs=3857,

    # The model recipe and text prompt for object detection
    model_recipe=GeometryModelRecipes.SAM3_TEXT_GEOMETRY,
    # You can also pass multiple prompts to detect several object types at once,
    # e.g. text_prompt=["roofs", "roads"]
    text_prompt="roofs",
    # Lower confidence thresholds surface more (but noisier) detections; you may
    # want to adjust this to trade off recall against precision for your AOI.
    confidence_threshold=0.3,
)

detections_gdf = gpd.read_parquet(model_output.uri)
detections_gdf

Explore the detected geometries

The geometry inference output is a GeoDataFrame where each row is a detected object. The columns include:

• geometry: the georeferenced polygon for the detection
• label: the text prompt category (e.g. "roofs")
• bbox_score: confidence score for the detection
• bbox: bounding box coordinates
• time: timestamp of the source imagery

print(f"Total detections: {len(detections_gdf)}")
print(f"Columns: {list(detections_gdf.columns)}")
print(f"nConfidence score stats:")
detections_gdf["bbox_score"].describe()

Save the results to the catalog

We can store these geometry outputs in the catalog using WherobotsDB to persist the GeoParquet results.

from sedona.spark import *
from pyspark.sql.functions import expr

config = SedonaContext.builder().getOrCreate()
sedona = SedonaContext.create(config)

sedona.sql("CREATE DATABASE IF NOT EXISTS examples_temp.sam3_db")

df = sedona.read.format("geoparquet").load(model_output.uri)
df = df.withColumnRenamed("label", "layer")
df.writeTo("examples_temp.sam3_db.sam3_roofs").createOrReplace()

Visualize the detected geometries

We can filter the detections by area to remove noise, then visualize the results.

df_filtered = df.withColumn(
    "area_m2",
    expr("ST_AreaSpheroid(geometry)")
).filter("area_m2 > 10")

df_filtered.show()

from wherobots_gl import Map

# Write the filtered detections to GeoParquet so Wherobots-GL Map can load them by URL
filtered_path = os.getenv("USER_S3_PATH") + "sam3_roofs_filtered.parquet"
df_filtered.write.format("geoparquet").mode("overwrite").save(filtered_path)

Map(layers=[{"type": "geoparquet", "source": filtered_path, "name": "SAM3 roof detections"}])

Generate PM Tiles for visualization

To improve visualization performance of a large number of geometries, we can use the Wherobots built-in high performance PM tile generator.

from wherobots import vtiles

full_tiles_path = os.getenv("USER_S3_PATH") + "sam3_roofs_tiles.pmtiles"
vtiles.generate_pmtiles(df_filtered, full_tiles_path)

vtiles.show_pmtiles(full_tiles_path)