WherobotsAI Raster Inference is GA with Support for Bring Your Own Model

Introduction

UPDATE: Raster inference is included in Wherobots RasterFlow. See Wherobots Get Started with RasterFlow – Wherobots for the most up-to-date workflows.

We are excited to announce that WherobotsAI Raster Inference is now generally available! Raster Inference is a serverless, planetary-scale computer vision solution that enables data teams to extract meaningful insights from aerial imagery (raster data) sources, such as satellites or drones, and puts these insights at the fingertips of data scientists and developers.

During the preview period, customers analyzed raster data by running inference with a limited set of Wherobots-hosted open-source computer vision models. Now, you can bring your own model into Raster Inference, offload inference pipeline management effort, and apply this capability to a much broader set of use cases. We have also made significant enhancements to our compute service to accelerate inference performance.

Typical Use Cases

Data teams use WherobotsAI Raster Inference to identify information in complex, large-scale overhead imagery data. Some common use cases include:

• Agriculture: Satellite and drone imagery are critical for monitoring land use, predicting crop yields, and improving sustainable farming practices.
• Environment and Conservation: Raster imagery is essential for monitoring ecosystem and biodiversity changes, such as tracking glacier melting, sea level rise, temperature fluctuations, and environmental degradation (e.g., oil spills, deforestation).
• Energy: Renewable energy developers analyze satellite data to assess land suitability, solar radiation, and wind patterns, to determine optimal locations for renewable energy projects.
• Insurance: Insurers use raster data to calculate risk assessments based on environmental and local factors to conduct and improve damage assessments at scale.
• Map Creation & Maintinence: Global digital, high fidelity map producers and maintainers extract features such as buildings, road networks, landcover, shipping lanes, etc., and their respective changes in order to maintain truthful connection between global map data products and the real world.

Traditional challenges with computer vision pipelines

We’ve met with many businesses struggling to get critical insights from raster data. Some are manually sifting through imagery. This method doesn’t scale, it’s expensive, error prone, and time intensive. Others are utilizing complex computer vision solutions that are not designed for overhead raster imagery. These computer visions solutions:

• Take time to build, and require significant management to efficiently load, store, and process large raster datasets.
• Are difficult to scale to accommodate increasing workload sizes.
• Are fragile with multiple components involved and challenges maintaining compatibility with an evolving modeling stack.
• Require effort to experiment, test, and integrate new models and inference runs.

Benefits of WherobotsAI Raster Inference

With WherobotsAI Raster Inference, you can:

• Use data pipelines that are ready for small to planetary-scale raster data.
• Deploy an on-demand solution in seconds that scales to meet workload needs without having to manage infrastructure.
• Easily import your model or utilize any model hosted by Wherobots.
• Experiment and generate critical insights faster.

In the following sections, we’ll provide a brief overview of Wherobots-hosted models, how to bring your own model, recent performance improvements, and how to use this feature effectively.

Choosing a Model for Raster Inference

Wherobots-Hosted Models

Wherobots-hosted models are precompiled and optimized for raster inference, enabling them to scale effortlessly and execute inference on large datasets. Below is a brief overview of the initial set of Wherobots-hosted models. We plan to host additional models based on customer feedback. For more information on these models and their performance metrics, see our docs page here.

Below is a code snippet on how to use landcover-eurosat-sentinel2 in WherobotsAI Raster Inference:

# set the model name

model_id = "landcover-eurosat-sentinel2"

# call the model in raster inference
df_predictions = df_raster_input.withColumn("preds", rs_classify(model_id, "outdb_raster"))

The STAC Machine Learning Model Extension Specification

The model import function of Raster Inference is built on the STAC Machine Learning Model (MLM) Extension Specification, an open community standard for model sharing we co-developed with CRIM and other collaborators. The MLM specification enables model portability, making it easier to use models across teams and compute platforms.

Before the MLM specification, data scientists and modelers sharing geospatial computer vision models often had to adapt existing standards, such as HuggingFace model cards, to store relevant information. This involved repurposing the cards to specify details like required raster bands, necessary data preprocessing, and post-processing functions. Without a community standard for organizing this information, these model cards were often inconsistent and documented in varying formats, making model sharing and reproducibility across different compute platforms cumbersome.

The STAC MLM extension introduces a community standard designed to simplify storing and sharing geospatial computer vision models. It achieves this by providing a comprehensive schema to:

• Describe critical geospatial model attributes, such as geolocation and temporal range.
• Include key model inference reproducibility details, such as required bands, model artifact locations, and pre- and post- processing steps.
• Enable model collections to be searched alongside associated spatiotemporal datasets.

The MLM specification has already been adopted in key modeling efforts at Terradue and is proudly supported by Radiant Earth. We’ll share more in a series of blog posts and a panel discussion with our collaborators on the MLM in late January — register here on the interest form to receive an invite when the date is finalized.

Bring Your own Model

The MLM specification enables users to quickly and easily use many geospatial, deep learning-based computer vision models with Raster Inference. We leverage a model’s MLM specification to specify and integrate the required data preprocessing, model, and post processing into the larger raster inference pipeline.

To bring your own model to Raster Inference:

1. Fill out our MLM form for your model. This will create a MLM formatted JSON file (MLM JSON) describing your model. Download the file.
2. Upload the MLM JSON file from Step 1 to your AWS S3 bucket.
3. Copy and save the AWS S3 URI to your MLM JSON.
4. During runtime, use your model for inference by calling the raster inference function with your MLM’s S3 URI link.

You can find a full walkthrough in our documentation on using the MLM form to bring your own model.

Performance Improvements

In addition to enabling bring your own model, we’ve accelerated asynchronous data loading in the inference engine to boost performance. Below, you can see how performance has evolved with experiments conducted using WherobotsAI Raster Inference and a Tiny GPU Runtime.

Example: Raster Inference with Bring Your own Model

For a full tutorial example on how to bring your own model to segment solar farms in Sentinel-2 imagery, see our documentation here.

Example: Run Raster Inference with a Wherobots-hosted model

We’ll walk through an example on how to identify solar infrastructure in raster imagery using a Wherobots-hosted model. In our example, we will be using the model:

solar-satlas-sentinel2

The full Python notebook file can be found on our GitHub.

1. Set up the WherobotsDB context

import warnings
warnings.filterwarnings('ignore')

from wherobots.inference.data.io import read_raster_table
from sedona.spark import SedonaContext
from pyspark.sql.functions import expr

from wherobots.inference.engine.register import create_semantic_segmentation_udfs
from pyspark.sql.functions import col

config = SedonaContext.builder().appName('segmentation-batch-inference')
    .getOrCreate()

sedona = SedonaContext.create(config)

2. Load Satellite Imagery

tif_folder_path = "s3a://wherobots-benchmark-prod/data/ml/satlas"
files_df = read_raster_table(tif_folder_path, sedona, limit=400)
df_raster_input = files_df.withColumn(
        "outdb_raster", expr("RS_FromPath(path)")
    )

df_raster_input.cache().count()
df_raster_input.show(truncate=False)
df_raster_input.createOrReplaceTempView("df_raster_input")

3. Run WherobotsAI Raster Inference

Specify a Wherobots-hosted model to run inference

model_id = "solar-satlas-sentinel2"

You can run WherobotsAI Raster Inference using either the Wherobot’s SQL API or Python API.

Using the SQL API

predictions_df = sedona.sql("""
SELECT
  outdb_raster,
  segment_result.*
FROM (
  SELECT
    outdb_raster,
    RS_SEGMENT('{model_id}', outdb_raster) AS segment_result
  FROM
    df_raster_input
) AS segment_fields
""")

predictions_df.cache().count()
predictions_df.show()
predictions_df.createOrReplaceTempView("predictions")

Using the Python API

rs_segment =  create_semantic_segmentation_udfs(batch_size = 10, sedona=sedona)
df = df_raster_input.withColumn("segment_result", rs_segment(model_id, col("outdb_raster"))).select(
                               "outdb_raster",
                               col("segment_result.confidence_array").alias("confidence_array"),
                               col("segment_result.class_map").alias("class_map")
                           )
df.show(3)

4. Extract predicted geometries (continued from step 4 using the SQL API)

df_multipolys = sedona.sql("""
    WITH t AS (
        SELECT RS_SEGMENT_TO_GEOMS(outdb_raster, confidence_array, array(1), class_map, 0.65) result
        FROM predictions
    )
    SELECT result.* FROM t
""")

df_multipolys.cache().count()
df_multipolys.show()
df_multipolys.createOrReplaceTempView("multipolygon_predictions")

df_merged_predictions = sedona.sql("""
    SELECT
        element_at(class_name, 1) AS class_name,
        cast(element_at(average_pixel_confidence_score, 1) AS double) AS average_pixel_confidence_score,
        ST_Collect(geometry) AS merged_geom
    FROM
        multipolygon_predictions
""")
df_filtered_predictions = df_merged_predictions.filter("ST_IsEmpty(merged_geom) = False")
df_filtered_predictions.cache().count()
df_filtered_predictions.show()

5. Visualize results

from sedona.maps.SedonaKepler import SedonaKepler
config = {
    'version': 'v1',
    'config': {
        'mapStyle': {
            'styleType': 'dark',
            'topLayerGroups': {},
            'visibleLayerGroups': {},
            'mapStyles': {}
        },
    }
}
map = SedonaKepler.create_map(config=config)

SedonaKepler.add_df(map, df=df_filtered_predictions, name="Solar Farm Detections")
map

Get started with WherobotsAI Raster Inference

Professional Edition Users

If you’re a Wherobots Professional Edition or Enterprise user, you have access to all capabilities of WherobotsAI Raster Inference! If you have access to GPU runtimes, sign in to your account now to launch a Wherobots Notebook and explore the feature. If you don’t yet have access, request it today and start using Raster Inference as soon as tomorrow. Explore how easy it is to bring your own model using our guided example on GitHub or within a Wherobots notebook instance. Start integrating WherobotsAI Raster Inference into your workflow today!

Community Edition Users

Although Raster Inference is not available in Wherobots Community Edition, we are currently offering a free trial for the Wherobots Professional Edition. You can either sign up through AWS Marketplace or upgrade your account to get started for free and integrate Raster Inference into your workflow today.

What’s next

We’re eager to hear about the models you’d like us to support and any features you’d like to see added. For product feedback, feel free to email us at feedback@wherobots.com (no request is too small). To see how others are using Raster Inference, ask questions, and share your own experiences, join the Wherobots community. We look forward to your ideas and creations!

Missed our panel discussion with our collaborators CRIM, Terradue and Radiant Earth on the MLM STAC extension? Watch the recording below.