Orchestrating Wherobots Jobs from AWS Step Functions: A Reference Architecture

When the execution engine is loosely coupled with orchestration, the pipeline is blind while waiting for one fact: did the job finish, fail, or die? This was the case with jobs running inside Wherobots Cloud and the AWS pipeline launching it… That was not cool with me, so we designed a reference implementation using AWS Step Functions, their callback pattern, and the Wherobots Python SDK.

The reference implementation is intentionally small and the accompanying (vibe coded) interactive app is designed to walk you through the implementation for learning purposes. There are four Lambda functions, two state machines, one API Gateway endpoint, and a simple Wherobots job script. Every AWS resource is created by plain CLI calls you can read, and a guided app drives the whole lifecycle: upload, deploy, run, teardown. The demo job counts Overture Maps buildings within one kilometer of downtown Seattle (geodesic buffer, EPSG:4326) and returns 1,084.

What the Reference Architecture Does

The pipeline submits a Wherobots job to our distributed engine through the wherobots-python-sdk, pauses for the callback, and resumes the moment the job reports its own result. The submit handler Lambda calls WherobotsJob(...).submit() with the task token passed as plain job arguments. The state machine then waits. There is no polling loop on this happy path. When the job finishes, the state machine wakes within seconds and the next state receives the job’s output as its input.

Authentication stays minimal on both sides. The Lambdas hold one Wherobots API key. The SDK uploads the job script to managed storage over short-lived STS credentials, so no long-lived AWS keys appear anywhere. Wherobots runs the job on a managed Apache Sedona runtime, built by the original creators of Apache Sedona and 100% code compatible across all spatial functions.

How the Callback Pattern Works

Step Functions supports a task type that pauses until something calls SendTaskSuccess or SendTaskFailure with a one-time task token. The catch: the job runs in Wherobots’ AWS account and cannot sign calls to your Step Functions API. The reference bridges that gap with a relay. The job POSTs plain JSON over HTTPS to an API Gateway endpoint, and a small Lambda behind it relays the payload into the Step Functions API.

The single-use task token is the security capability. A request without a live token can do nothing, which is what makes a public endpoint acceptable for a reference implementation. Production hardening adds an API key or IAM auth at the gateway. The reference uses API Gateway instead of a Lambda Function URL as blocking anonymous Function URL invocation by service control policy is a general best practice, and API Gateway is the sanctioned front door.

Failure Mode Drives the Design

In a decoupled orchestration and compute engine system, one major question to consider is “What happens when the job fails?”. How the job fails drives the resolution and code path, one layer cannot cover all cases.

Soft failures report themselves

The Wherobots job logic is wrapped in a typical try/except loop so the Python-level failures, from bad SQL to bad data, are caught and posted as a failure callback carrying the full traceback. The execution fails and within seconds the traceback appears in the AWS console with the trace. In live verification, a forced RuntimeError is surfaced in the execution history with the exact line that raised it.

A Heartbeat catches the hard death

A driver OOM or a cluster kill ends the Python process mid-instruction. No except or finally runs and no callback is ever sent. There are no silver bullets to report a death like this, which is why the job also runs a daemon thread that posts a heartbeat every 60 seconds. The task sets HeartbeatSeconds: 240; the window counts from task entry, and the job’s first heartbeat cannot arrive until the runtime finishes provisioning, so the window sits above typical provisioning time. On a rare slower cold start the timeout routes to the fallback poller, which finds the run healthy and still delivers COMPLETED. When the process dies, heartbeats stop with it, and the timeout fires within four minutes initiating the failure tracing process.

The fallback poller finds the truth

The heartbeat timeout routes to a recovery path. A FindRun Lambda recovers the run ID from the deterministic job name using WherobotsJob.list_runs, then a reusable poller state machine checks the run every 30 seconds until Wherobots returns a terminal status. The poller takes {run_id, poll_seconds} as input and carries no reference to the pipeline that submitted the job, so any other pipeline can reuse it as-is. In live verification, the simulated OOM (os._exit(137), no callback, no goodbye) reached a confirmed FAILED verdict in 4 minutes and 46 seconds, end to end: the heartbeat window plus one poll cycle (adjust HeartbeatSeconds and the polling interval to your desired detection latency).

The Guided Dashboard

The repository ships a local app to walk you through the architecture. Five stages build on each other in order:

1. PREFLIGHT: reads a masked .env
2. UPLOAD: pushes the job script to Wherobots storage
3. BUILD streams every AWS CLI call as it creates the infrastructure
4. RUN: kicks off the step function chain
5. TEARDOWN: removes resources created in step 3

The test-mode dropdown makes the failure contract demonstrable through the live execution graph:

How to Run It

Minimal setup required to play with the dashboard and implementation. Currently the repo is private, but if you wan to run it contact us and we can give you access. We expect this to be live as an open repo very soon in Wherobots Labs.

git clone https://github.com/wherobots/labs-reference-implementations.git
cd labs-reference-implementations/aws-step-functions-python-sdk
python3 -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env   # set WHEROBOTS_API_KEY; blank AWS keys use your default chain
python3 dashboard/server.py

Open the dashboard, click through the five stages, and run the three test modes. The demo costs a few cents of Lambda and Step Functions time plus one tiny Wherobots runtime per run. Don’t forget to run (and confirm) the Teardown cell, it removes every AWS resource the build created, (each resource carries ManagedBy and TeardownBy tags for auditing) .

Design Decisions Worth Stealing

Four steps in this reference transfer to production systems directly.

Deterministic job names. The execution input carries the job name, so the fallback path can recover the run ID after the callback path dies. Recovery depends on nothing the failed job was supposed to send.

A reusable poller. Polling logic lives in its own state machine with a two-field input contract. The callback version of the pipeline demoted it to a fallback without changing a line of it.

Failure evidence over failure flags. The soft-failure callback carries the traceback, and the fallback poller reports the run’s terminal status from the Wherobots API. Whoever reads the failed execution reads the cause.

Symmetric teardown. Every create call in the deploy script has a delete call in the teardown script, and both are readable CLI commands. Reference infrastructure that cannot be removed cleanly teaches the wrong habit.

Where It Fits

This reference covers job runs: script-shaped and batch-shaped work submitted to Wherobots compute. Interactive SQL over the Spatial SQL API and Airflow-based orchestration with the Wherobots provider are separate surfaces with their own patterns. If your pipelines live in Step Functions today and your spatial workloads need managed jobs at scale, this architecture is the shortest path between them.

Get Started with Wherobots

Try Now