What Is PySpark? Definition, Tutorial, and Spatial Data

PySpark is the Python application programming interface (API) for Apache Spark, an open-source engine that runs one program across many machines. You write Python that reads, filters, joins and groups tables. Spark splits the tables into parts and runs the steps on a cluster.

Joining two tables by location is harder. On its own, Spark joins them by testing every row of one table against every row of the other. Apache Sedona, an open-source spatial add-on for Spark, adds join planning: rules that run the join with a spatial index instead. In Wherobots, we counted which county each of 391,703 Washington places falls in, using open map data from the Overture Maps Foundation. On the same data and machines, the join took under a second with Sedona's join planning and 6 minutes without it.

Key takeaways

• PySpark is the Python interface to Apache Spark. The DataFrame API and Spark SQL, Spark's module for Structured Query Language (SQL), run on the same engine. For a query of the top five place categories in Washington, both produced one optimized plan. Their median times over six runs each were 0.197 and 0.192 seconds. That 0.005-second gap is smaller than the spread between runs of one version: the DataFrame runs took 0.14 to 0.84 seconds.
• Spark computes nothing until an action, such as count() or show(), requests a result. Before that, each step only adds to a plan.
• Spark 4.2 added geometry column types with five conversion functions. They include no function that tests how two shapes relate, such as whether a county contains a point, and no spatial index for finding nearby shapes. Apache Sedona adds spatial SQL, spatial joins and raster functions to PySpark.
• Finding the county for each of Washington's 391,703 Overture places took 0.85 seconds with Sedona's join planning and 359 seconds without it. Those were first runs on one tiny Wherobots runtime on November 2, 2026, and times depend on the data and cluster. Without the planning, Spark tested all 15.3 million place-county pairs. Both ways returned the same counts, and later runs with the planning took 0.06 seconds.
• How we measured it: Wherobots Job Runs used one tiny runtime with Spark 4.1.4 and Sedona. They joined the places to 39 counties and 305 towns, with and without Sedona's join planning. They also read the ground elevation under 894 places Overture tags as hospitals, in 57.1 seconds, from the Copernicus GLO-30 grid of 30 m cells. All code in this article ran as printed. The place counts measure Overture listings.

What is PySpark?

PySpark is "the Python API for Apache Spark," in the words of the PySpark documentation. Spark is a cluster engine: one program, run across many computers. According to the Spark project history, it began as a research project at the University of California, Berkeley's AMPLab in 2009. It moved to the Apache Software Foundation in 2013 and became a top-level Apache project in February 2014.

The current release is Spark 4.3.0. It was published to the Python Package Index (PyPI) on October 30, 2026, and needs Python 3.11 or later. PySpark gives Python access to these parts of Spark:

• Spark SQL and DataFrames: a DataFrame is a table with named columns. You query it with Python methods or with SQL.
• Pandas API on Spark: pandas syntax on a cluster, added in Spark 3.2.
• Structured Streaming: the same table operations on data that keeps arriving.
• MLlib: Spark's machine learning library.
• Spark Connect: a client and server design, so a laptop can send work to a remote cluster.

Databricks' glossary describes PySpark as the Python API for Apache Spark that "lets Python users run distributed data processing and analytics on large datasets."

How PySpark works

A PySpark program has one driver and many executors. The driver is the process that runs your Python script. It connects to Spark's engine, which runs on the Java Virtual Machine (JVM), through a bridge library called Py4J. Executors are worker processes, usually on other machines. Each one holds partitions, the slices of a table, and runs one task, a unit of work, per partition.

Spark works in two phases. Transformations, such as filter, select, join and groupBy, return a new DataFrame and compute nothing. The Spark programming guide says: "All transformations in Spark are lazy." Actions, such as count, show and collect, make Spark compute a result.

At the first action, Spark's optimizer, called Catalyst, turns the chain of transformations into a physical plan. The plan lists each step Spark will run, such as a table scan, a filter or a join method, and explain() prints it. Armbrust and coauthors introduced Catalyst and the DataFrame API in Spark SQL: Relational Data Processing in Spark, at the 2015 SIGMOD database conference of the Association for Computing Machinery (ACM).

PySpark tutorial: a first job

This tutorial uses real data: Washington State's boundary, its counties and the places of interest inside it. All three come from the Overture Maps Foundation, which publishes open map data. They are stored in Wherobots Open Data as Havasu tables. Havasu is Wherobots' version of Apache Iceberg, an open table format for data lakes, with geometry and raster columns.

Each block ran on November 2, 2026, in a Wherobots Job Run, a Python script run on a cloud cluster. The runs used Spark 4.1.4 on a Wherobots runtime of the "tiny" size, the second-smallest of the eight general-purpose runtime sizes.

Start a session

Every PySpark program starts with a SparkSession, the object a script creates first; every table read and query goes through it. In plain PySpark you write SparkSession.builder.getOrCreate(). In Wherobots, SedonaContext returns a SparkSession that also has Sedona's spatial types and functions. On your own Spark installation, Sedona also needs its Java packages: the Sedona setup guide adds them to the builder with spark.jars.packages.

from pyspark.sql import SparkSession, functions as F
from sedona.spark import SedonaContext

config = SedonaContext.builder().getOrCreate()   # a SparkSession with Sedona's types and functions
sedona = SedonaContext.create(config)

Read, filter and select

sedona.table() reads a catalog table into a DataFrame. filter keeps rows, and select keeps and renames columns. The block below finds Washington's boundary and its 39 counties. Then it narrows Overture's 88 million places to those inside the state's bounding box, the smallest rectangle around the state.

areas = sedona.table("wherobots_open_data.overture_maps_foundation.divisions_division_area")
wa = areas.filter((F.col("country") == "US") & (F.col("region") == "US-WA") & F.col("is_land"))
state = wa.filter(F.col("subtype") == "region").select(F.col("geometry").alias("state_geom"))
counties = wa.filter(F.col("subtype") == "county").select(
    F.col("names.primary").alias("name"), F.col("geometry").alias("geom"))

env = state.selectExpr("ST_XMin(state_geom) AS xmin", "ST_XMax(state_geom) AS xmax",
                       "ST_YMin(state_geom) AS ymin", "ST_YMax(state_geom) AS ymax").first()

places = sedona.table("wherobots_open_data.overture_maps_foundation.places_place")
in_box = places.filter(
    (F.col("bbox.xmin") >= env.xmin) & (F.col("bbox.xmax") <= env.xmax)
    & (F.col("bbox.ymin") >= env.ymin) & (F.col("bbox.ymax") <= env.ymax)
).select("id", F.col("names.primary").alias("name"),
         F.col("taxonomy.primary").alias("category"), "geometry")

The box filter compares plain numbers, so plain PySpark can run it. It keeps 454,192 places. The rectangle also covers land beyond the state line, so the next step tests each place against the state polygon. That test needs ST_Intersects, a spatial function from Sedona. cache() keeps the result in executor memory for later steps, and count() is the action that runs it.

places_wa = (in_box
             .join(F.broadcast(state), F.expr("ST_Intersects(state_geom, geometry)"))
             .drop("state_geom")
             .cache())
places_wa.count()

The count is 391,703 places, loaded and cached in 5.2 seconds.

Group and count: PySpark vs Spark SQL

The next block answers one question two ways: which five place categories are most common in Washington? The first way uses DataFrame methods. The second registers the DataFrame as a view, a named table for SQL, and runs Spark SQL.

top_df = (places_wa.filter(F.col("category").isNotNull())
          .groupBy("category").count()
          .orderBy(F.desc("count"))
          .limit(5))

places_wa.createOrReplaceTempView("places_wa")
top_sql = sedona.sql("""
    SELECT category, COUNT(*) AS count
    FROM places_wa
    WHERE category IS NOT NULL
    GROUP BY category
    ORDER BY count DESC
    LIMIT 5
""")
top_df.show()

Both return the same five rows: real estate agents (10,312), health care (6,956), Christian places of worship (5,803), real estate services (5,294) and contractors (5,009). Spark's sameResult check confirmed that both produce the same optimized plan. We ran each version six times, in alternating order. The median times were 0.197 seconds for the DataFrame API and 0.192 seconds for SQL.

So in PySpark, the DataFrame API ran no faster or slower than the same query in Spark SQL. The 0.005-second gap between medians is smaller than the 0.14 to 0.84 second spread of the DataFrame runs. The Spark SQL guide says "the same execution engine is used, independent of which API/language you are using." DataFrame methods suit code that builds a query step by step. SQL suits queries that analysts share and read.

PySpark vs pandas, and Apache Spark vs PySpark

pandas holds a table in the memory of one computer and runs each step right away. PySpark splits the table across executors and runs nothing until an action. A table that fits in memory can be faster in pandas, because nothing is planned or sent between machines. A table that does not fit needs PySpark, or the pandas API on Spark, which keeps pandas syntax on the cluster.

PySpark is one of several front ends to the Spark engine, which runs on the JVM. PySpark builds the plan in Python and passes it to the JVM. Python code runs on the executors mainly for a user-defined function (UDF), your own Python function applied to each row.

To run a Python UDF, Spark sends rows from the JVM to a Python process and back. The PySpark user guide says scalar Python UDFs "encounter performance bottlenecks" from this transfer. Since Spark 4.2, Python UDFs use Apache Arrow, a columnar memory format, by default, which the guide says significantly improves performance. A built-in function, when one exists, avoids the transfer.

PySpark examples

Teams use PySpark for jobs like these:

• ETL (extract, transform and load): read raw files, clean and join them, and write tables to a data lake.
• Feature engineering: build the columns a machine learning model trains on, across billions of rows.
• Machine learning: train models with MLlib, and apply a trained model to each row to get a prediction.
• Streaming: count, filter and join events as they arrive with Structured Streaming.
• Spatial analysis: with Apache Sedona, join places to boundaries, find nearest neighbors and read raster values such as elevation.

What changes for spatial data

Spatial data needs three things: a type for shapes, functions that test shapes, and a join method that avoids testing every pair of rows. Spark 4.2 has the first, and Sedona adds the other two.

A geometry type. Spark 4.2, released in July 2026, added GEOMETRY and GEOGRAPHY column types. GEOMETRY holds shapes on a flat plane, and GEOGRAPHY holds shapes on the Earth's curved surface. Both follow the Open Geospatial Consortium (OGC) standards.

The 4.2 reference lists five functions for them: ST_AsBinary, ST_GeomFromWKB, ST_GeogFromWKB, ST_Srid and ST_SetSrid. These convert shapes to and from well-known binary (WKB), a standard way to store a shape as bytes. They also read or set a shape's coordinate reference system, the code that says how its coordinates map to places on Earth. None of them tests how two shapes relate. PySpark 4.3.0 offers the same five.

Spatial functions. Apache Sedona adds functions that test how shapes relate, such as ST_Contains, and functions that measure them, such as ST_Distance. Its vector function names start with ST_, for spatial type. They follow SQL/MM Part 3, a published standard for spatial SQL.

Raster functions start with RS_. The What is Apache Sedona? post covers the project, and pip install apache-sedona installs it.

A spatial join method. A normal join matches rows that share a value, such as the same county ID. Spark can sort or group both tables by that value and match them in one read of each. A spatial test such as ST_Contains has no shared value to sort by. Without spatial join planning, Spark tests every row of one table against every row of the other.

A spatial index is a tree of bounding boxes, so each shape is tested only against nearby shapes. Sedona's join planning, rules that Sedona adds to Spark's query optimizer, replaces the every-pair test with an indexed join. A broadcast index join copies one table, with a spatial index, to every machine. A range join splits both tables by area first.

The spatial join article explains the join types. The partitioning tax article explains what goes wrong when people split space into tiles by hand.

Other platforms add spatial functions too. Databricks documents ST geospatial functions on Databricks Runtime 17.1 and above. Support for GEOMETRY is generally available there, and GEOGRAPHY is in Public Preview. Databricks describes its spatial join speedups in a December 2025 blog post.

PySpark with Apache Sedona in Wherobots

WherobotsDB, the engine in Wherobots, comes from the creators of Apache Sedona and runs Sedona's PySpark API. The tutorial above ran there.

The claim. A point-in-polygon join matches each point to the polygon that contains it. On its own, Spark plans this join by testing every point against every polygon. With Sedona's join planning, the same code runs as an indexed join and returns the same answer much faster.

What would break it. The claim fails if Spark finds a better plan on its own, if the two plans return different counts, or if the indexed join is less than 10 times faster.

The method. We joined the 391,703 cached places to Washington's 39 counties, then to its 305 towns. Each join ran with Sedona's default join planning, then with that planning turned off by one setting, sedona.join.optimizationmode = none. The spatial test, Sedona's ST_Contains, stayed the same in both runs; only the join method changed. The timed runs called collect(), the action that returns all the counts; the block below uses show() and explain() to print them.

per_county = (counties.alias("c")
              .join(places_wa.alias("p"), F.expr("ST_Contains(c.geom, p.geometry)"))
              .groupBy("c.name").count()
              .orderBy(F.desc("count")))
per_county.show(5)
per_county.explain()

ST_Contains(c.geom, p.geometry) is true when the county polygon contains the place point. The aliases c and p name each table, so c.name and p.name refer to different columns. With Sedona's join planning on, explain() shows a BroadcastIndexJoin. Sedona built an R-tree, one kind of spatial index, over the places, and copied it to every executor. For each county, Spark then tested only the places inside the county's bounding box.

This block turns the planning off and prints the plan again:

sedona.conf.set("sedona.join.optimizationmode", "none")
per_county_plain = (counties.alias("c")
                    .join(places_wa.alias("p"), F.expr("ST_Contains(c.geom, p.geometry)"))
                    .groupBy("c.name").count())
per_county_plain.explain()
sedona.conf.set("sedona.join.optimizationmode", "nonequi")   # the default

Now explain() prints a CartesianProduct, which pairs every place with every county. When the job ran, Spark carried it out as a BroadcastNestedLoopJoin, a loop over the same pairs. That is 391,703 places times 39 counties, about 15.3 million tests.

The result.

• Counties: 0.85 seconds on the first run with Sedona's join planning, and a median of 0.061 seconds over five more. Without it, 359 seconds, 6 minutes. Both matched all 391,703 places to a county, with identical counts per county.
• Towns: 1.07 seconds on the first run with Sedona's join planning, and a median of 0.051 seconds over five more. Without it, 425 seconds, 7 minutes. Both matched 321,485 places, 82% of the total, to 300 of the 305 towns. The other places lie outside any town boundary.

The verdict. The claim holds: Spark alone tested every pair, both methods gave the same answer, and testing every pair took about 400 times as long as the first indexed run. The time without Sedona did not grow in step with the number of tests. The town join meant about 119 million tests, eight times as many as the county join, but took only 18% longer: 425 seconds against 359. Our runs did not measure the reason.

How we measured. All runs used one Wherobots tiny runtime and the same cached places. In a first job, each join with Sedona's join planning ran once untimed and then once timed: 0.1 seconds for counties and for towns. The same job's runs with the planning off took 361 seconds for counties and 430 seconds for towns. A second job repeated the joins without the planning and took 359 and 425 seconds, close to the first job.

The article reports the second job's times. That job timed the very first run with the planning separately from five later runs.

The map shows the county counts. King County, home to Seattle, has 136,674 places, more than the next three counties combined. Seattle leads the towns with 57,313, followed by Bellevue, Spokane, Tacoma and Vancouver. These are Overture records, so the counts measure listed places. A county with poor listing coverage looks emptier than it is.

Rasters in the same DataFrame

Map data comes in two forms. Vector data stores points, lines and polygons, like the places and counties above. Raster data stores a grid of values, such as elevation. Sedona reads both into PySpark DataFrames.

The next block reads the elevation under every place Overture tags as a hospital. It uses the Copernicus GLO-30 digital elevation model, a global elevation grid with 30 m cells built from TanDEM-X radar satellite data. The table stores it as tiles, each with a footprint, the polygon outlining the area the tile covers. RS_Value reads the elevation in meters at each hospital point.

dem = (sedona.table("wherobots_open_data.copernicus_dem.glo_30m")
       .filter(F.expr(f"ST_Intersects(footprint, ST_PolygonFromEnvelope("
                      f"{env.xmin}, {env.ymin}, {env.xmax}, {env.ymax}))")))
hospitals = places_wa.filter(F.col("category") == "hospital")
elev = (hospitals.alias("h")
        .join(dem.alias("d"), F.expr("ST_Intersects(d.footprint, h.geometry)"))
        .groupBy("h.id", "h.name")
        .agg(F.max(F.expr("RS_Value(d.rast, h.geometry)")).alias("elev_m")))
elev.orderBy(F.desc("elev_m")).show(3, truncate=False)

All 894 hospital-tagged places got an elevation, in 57.1 seconds. The median is 96.4 m, and 100 of them sit above 500 m. The highest is Ferry County Memorial Hospital, at 827.3 m. Overture's hospital category also holds some clinics and medical offices. The plan used a BroadcastIndexJoin, the same indexed join as the counties.

The digital elevation model article explains elevation grids.

PySpark research and what changes at scale

Spark began with a paper on keeping data in memory between steps. Zaharia and coauthors described it in Spark: Cluster Computing with Working Sets, at the USENIX HotCloud workshop in 2010. In 2012 they defined the Resilient Distributed Dataset (RDD) in Resilient Distributed Datasets, which won the best paper award at USENIX's networked systems conference (NSDI). An RDD is a set of records split into partitions across machines. If a machine fails, Spark rebuilds the lost partitions by rerunning the steps that made them.

The DataFrame API and the Catalyst optimizer followed in 2015, in the Spark SQL paper. Zaharia and coauthors summarized the design in Apache Spark: A Unified Engine for Big Data Processing, in Communications of the ACM in 2016.

Spatial support began outside Spark. Jia Yu, Jinxuan Wu and Mohamed Sarwat introduced GeoSpark at ACM SIGSPATIAL 2015, adding spatial types, indexes and partitioning to Spark. Yu, Zhang and Sarwat extended it in Spatial data management in Apache Spark: the GeoSpark perspective and beyond, in GeoInformatica in 2019. That paper compares four ways to split space across machines: a uniform grid, an R-tree, a quadtree, which splits crowded cells into four, and a KDB-tree. A KDB-tree splits space so that each cell holds about the same number of rows.

GeoSpark became Apache Sedona, which the Apache Software Foundation made a top-level project in 2023.

Three problems remain open at scale:

• Skew: one partition can hold far more of the work than the others. If the places were split by county, King County's partition would hold over a third of them: 136,674 of 391,703.
• Python UDFs: they still move rows between the JVM and Python, so spatial code written as Python UDFs runs slower than built-in functions.
• Exchange: geometry types now exist in Spark 4.2, Sedona, Databricks and the Parquet and Iceberg formats. Moving data between them depends on a shared way to write shapes as bytes, such as WKB.

The SpatialBench post compares spatial engines on one benchmark. The scaling GeoPandas post shows how to move GeoPandas code to Sedona.

Conclusions

1. PySpark and Spark SQL are one engine. The DataFrame API and SQL produced the same optimized plan, with median times of 0.197 and 0.192 seconds over six runs each. Choose whichever reads better.
2. Without spatial join planning, Spark joins on a spatial test by testing every pair. With Sedona's join planning off, the county join printed a CartesianProduct, ran as a nested loop and took 6 minutes. Run explain() before a large spatial join. If the plan shows CartesianProduct or BroadcastNestedLoopJoin, Spark is testing every pair, so turn on spatial join planning such as Sedona's.
3. Sedona's join planning returned identical counts in about a second on the first run, and in 0.05 to 0.06 seconds after that. That held for 39 counties and for 305 towns, on 391,703 points and one tiny runtime. The size of the gap depends on the data, the polygons and the cluster.
4. Spark 4.2's geometry types store shapes and convert them. They have no spatial tests or index, so a point-in-polygon join still needs Sedona or another spatial engine.
5. Raster and vector data can share one PySpark job. With Sedona, elevation from a raster table joined to 894 places Overture tags as hospitals, in one DataFrame.
6. Overture counts measure listings. Treat places per county as a measure of map coverage, and check them against a second source before using them as a measure of activity.

Related terms

• Spatial join: the join types and spatial tests behind the county count
• Partitioning tax: the cost of splitting spatial data into tiles by hand
• Geospatial database: geometry types, a spatial index and spatial SQL in one system
• Raster vs. vector data: the two data models this article joins
• Digital elevation model: the elevation grid read under each hospital
• Points of interest data: the place records counted per county
• Geospatial analytics at scale: why spatial work slows down as data grows

Read more from Wherobots

• What is Apache Sedona?
• Apache Sedona: the project, its APIs and its community
• Scaling GeoPandas with Apache Sedona
• SedonaDB and SpatialBench
• Wherobots, Sedona and GeoPandas are better together
• Spatial joins in Wherobots: the Getting Started notebook, Part 4

Run the code above with a Wherobots free trial at cloud.wherobots.com.