Rasterflow, Earth Intelligence & inference engine now in public preview Learn More

This Month in Apache Sedona: November Edition

Apache Sedona

Apache Sedona has reached ⭐ 2M+ downloads ⭐ in the past month!

What’s New in Apache Sedona 1.7.0

Our newest features include KNN Join, GeoStats and DataFrame-based readers. Learn more about the latest release in our most recent office hour.

You can can access the presentation slides here and the kNN Join slides here. Be sure to save the date for the next office hour to stay up-to-date with the latest releases!

Latest Content

What is Apache Sedona?

Apache Sedona

Spatial data should be treated as a first class citizen. Here’s an overview of Apache Sedona and some of its common use cases. Learn more.

Introducing GeoStats

Introducting GeoStats

A machine learning and statistical toolbox for WherobotsAI and Apache Sedona users. Learn about its use cases and what challenges it helps solve. Read more.

O’Reilly: Cloud-Native Geospatial Analytics with Apache Sedona

O'Reilly Cloud Native Geospatial Analytics with Apache Sedona

Ideal for developers, data scientists and engineers, this hands-on guide provides practical solutions for challenges in working with various types of geospatial data. Get access here.

Past Event

Spatial Data Science Conference Workshop

Spatial Data Science Conference

  • Wherobots attended the Spatial Data Science Conference in New York last month. Head of Marketing Ben Pruden and Senior Solution Architect Daniel Smith led a workshop on insurance risk analysis using Wherobots and CARTO.
  • We also took the stage with GeoPostcodes to showcase how Apache Sedona on Wherobots Cloud can be leveraged to better understand population data, reducing processing times from weeks to days.
  • During the meetup, we enjoyed some drinks and light bites while having a great time connecting with members of the Apache Sedona Community.
  • We’re grateful for the opportunity to meet and learn from others in the geospatial data space, hearing about their use cases and how they’re leveraging these technologies.

Upcoming Events

Cloud Native Geospatial Meetup: GIS Week – November 19

Hear from an amazing lineup of speakers from the Google Maps and Google Earth teams, former ESRI and Wherobots. Topics include kNN Join, the origins of GIS Day, the history and future of cloud-native geospatial technology, and current work within the open source community. More details here.

AWS re:Invent – December 2-6

  • Expo booth (all week): Swing by the booth #1865 to learn more about Wherobots and Apache Sedona. Grab some free SWAG and participate in our trivia to win some exciting raffle prizes!
  • GeoParty (December 4): Join us for a geoparty at the Venetian Pool Deck! Enjoy drinks, light snacks, good music and networking with other professionals in the field. Space is limited and you must register to attend. RSVP here.
  • Session: Extract insights from satellite imagery at scale with WherobotsAI (December 5 12:30PM-12:50PM) at Venetian Hall B Expo, Theater 4

    Learn>Learn how WherobotsAI Raster Inference enables data platform and science teams to analyze our planet with satellite imagery faster, more reliably, and with a zero carbon footprint—using SQL and Python. This fully managed, high-performance, carbon-neutral planetary-scale computer vision solution makes AI/ML on satellite imagery accessible to most developers and data scientists.

  • Connect at re:Invent: If you’ll be at re:Invent and would like to set up a time to meet, feel free email us at reinvent@wherobots.com and we can find a time to connect.

Apache Sedona Office Hour – December 10

We host monthly office hours as a way to engage with the community, share the latest updates and releases, along with future plans. If you’re working on something exciting with Apache Sedona, we’d love to hear about it. Save the date for the next office hour.

Key takeaways

  • This November 2024 Apache Sedona newsletter is a community roundup, not a standalone technical deep dive.
  • The issue says Apache Sedona reached 2 million+ downloads in the past month.
  • Sedona 1.7.0 features highlighted here are kNN Join, GeoStats, and DataFrame-based readers, with links to office-hour slides.
  • At the Spatial Data Science Conference in New York, Wherobots ran an insurance-risk workshop with CARTO and a GeoPostcodes session on population data that cut processing from weeks to days.
  • Upcoming items: Cloud Native Geospatial Meetup on November 19; AWS re:Invent December 2–6 (booth #1865, Raster Inference lightning talk December 5 12:30–12:50 PM at Venetian Hall B Expo Theater 4, GeoParty December 4); Sedona office hour December 10.

Wherobots Joins Overture, Winning The Taco Wars, Spatial SQL API, Geospatial Index Podcast – This Month In Wherobots

Welcome to This Month In Wherobots the monthly developer newsletter for the Wherobots & Apache Sedona community! This month we have news about Wherobots and the Overture Maps Foundation, a deep dive on new Wherobots Cloud features like raster inference, generating vector tiles, and the Spatial SQL API, plus a look at retail cannibalization analysis for the commercial real estate industry.

Wherobots Joins Overture Maps Foundation

Wherobots joins Overture Maps Foundation

Wherobots has officially joined Overture Maps Foundation to support the next generation of planetary-scale open map data. Wherobots has supported the development of Overture datasets through Overture Maps Foundation’s use of the open-source Apache Sedona project to develop and distribute global data, enabling Overture to embrace modern cloud-native geospatial technologies like GeoParquet. By joining Overture as Contributing Members Wherobots will continue to support the ongoing development, distribution, and evolution of this critical open dataset that enables developers and data practitioners to make sense of the world around us.

Read the announcement blog post

This month’s featured community members is Ilya Marchenko from YuzuData where he focuses on AI and location intelligence for the commercial real estate industry. Ilya recently wrote a blog post showing how to use Wherobots for a retail cannibalization study. Thanks Sean and Ilya for being a part of the community and sharing how you’re building geospatial products using Wherobots!

Comparing Taco Chains: A Consumer Retail Cannibalization Study With Isochrones

Retail cannibalization analysis with Wherobots

Understanding the impact of opening a new retail location on existing locations is an important analysis in the commercial real estate industry. In this code-heavy blog post the YuzuData team detail a retail cannibalization analysis using WherobotsDB, Overture Maps point of interest data, drive-time isochrones using the Valhalla API, and visualization with SedonaKepler. Sean also presented this analysis earlier this week in a live webinar.

Read the blog post or watch the video recording

Unlock Satellite Imagery Insights With WherobotsAI Raster Inference

Raster inference with WherobotsAI

One of the most exciting features in Wherobots’ latest release is WherobotsAI Raster Inference which enables running machine learning models on satellite imagery for object detection, segmentation, and classification. This post gives a detailed look at the types of models supported by WherobotsAI and an overview of the SQL and Python APIs for raster inference with an example of identifying solar farms for the purpose of mapping electricity infrastructure.

Read the blog post to learn more about WherobotsAI Raster Inference

Generating Global PMTiles In 26 Minutes With WherobotsDB VTiles

Generating PMTiles with Wherobots VTiles vector tiles generator

WherobotsDB VTiles is a highly scalable vector tile generator capable of generating vector tiles from small to planetary scale datasets quickly and cost-efficiently and supports the PMTiles format. In this post we see how to generate vector tiles of the entire planet using three Overture layers. Using Wherobots Cloud to generate PMTiles of the Overture buildings layer takes 26 minutes. The post includes all code necessary to recreate these tile generation operations and a discussion of performance considerations.

Read the blog post to learn more about WherobotsDB VTiles

Spatial SQL API Brings Performance Of WherobotsDB To Your Favorite Data Applications

Using Apache Airflow with WherobotsDB

The Wherobots Spatial SQL API enables integration with Wherobots Cloud via Python and Java client drivers. In addition to enabling integrations with your favorite data applications via the client drivers, Wherobots has released an Apache Airflow provider for orchestrating data pipelines and an integration with Harlequin, a popular SQL IDE.

Read the blog post to learn more about the Wherobots Spatial SQL API

Wherobots On The Geospatial Index Podcast

Wherobots On The Geospatial Index Podcast

William Lyon from Wherobots was recently a guest on The Geospatial Index podcast. In this episode he discusses the origins of Apache Sedona, the open-source technology behind Wherobots, how users are building spatial data products at massive scale with Wherobots, how Wherobots is improving the developer experience around geospatial analytics, and much more.

Watch the video recording

Upcoming Events

  • Apache Sedona Community Office Hours (Online – August 6th) – Join the Apache Sedona community for updates on the state of Apache Sedona, presentation and demo of recent features, and provide your input into the roadmap, future plans, and contribution opportunities.
  • GeoMeetup: Cloud Native Spatial Data Stack (San Francisco – September 5th) – Join us on September 5th for an exciting GeoMeetup featuring talks from industry leaders with Wherobots and Felt.com. In this meetup we will be exploring the elements of the cloud native spatial data stack.
  • FOSS4G NA 2024 (St Louis – September 9th-11th) – FOSS4G North America is the premier open geospatial technology and business conference. Join the Wherobots team for a pre-conference workshop or come by and chat with us at the Wherobots booth to learn about the latest developments in Apache Sedona.

Want to receive this monthly update in your inbox? Sign up for the The Spatial Intelligence Newsletter:

Key takeaways

  • This August 2024 “This Month In Wherobots” newsletter roundups existing posts and events rather than original benchmarks.
  • Wherobots joined the Overture Maps Foundation as a Contributing Member after already supporting Overture’s use of Apache Sedona and GeoParquet.
  • Featured community work: Ilya Marchenko and Sean Knight of YuzuData on a retail cannibalization study with Overture POIs, Valhalla isochrones, and SedonaKepler.
  • Product posts recapped here include WherobotsAI Raster Inference, generating global PMTiles of Overture buildings in 26 minutes with VTiles, and the Spatial SQL API (Python/Java drivers, Airflow provider, Harlequin).
  • Upcoming events listed: Apache Sedona office hours August 6, GeoMeetup in San Francisco September 5, and FOSS4G NA in St. Louis September 9–11.

Raster Data Analysis, Processing Petabytes of Agronomic Data, Overview of Sedona 1.5, and Unlocking The Spatial Frontier – This Month In Wherobots

Welcome to This Month In Wherobots, the monthly newsletter for data practitioners in the Apache Sedona and Wherobots community. This month we’re exploring raster data analysis with Spatial SQL, processing petabytes of agronomic data with Apache Sedona, a deep dive on new features added in the 1.5 release series, and an overview of working with files in Wherobots Cloud.

Raster Data Analysis With Spatial SQL & Apache Sedona

One of the strengths of Apache Sedona and Wherobots Cloud is the ability to work with large scale vector and raster geospatial data together using Spatial SQL. This post (and video) takes a look at how to get started working with raster data in Sedona using Spatial SQL and some of the use cases for raster data analysis including vector / raster join operations, zonal statistics, and using raster map algebra.

Read The Article: Raster Data Analysis With Spatial SQL & Apache Sedona

This month’s Wherobots & Apache Sedona featured community member is Luiz Santana. Luiz is Co-Founder and CTO of Leaf Agriculture. He has extensive experience as a former data architect and developer. Luiz has a PhD in Computer Science from Universidade Federal de Santa Catarina and spent time researching data processing and integration in highly scalable environments. Leaf Agriculture is building the unified API for food and agriculture by leveraging large-scale agronomic data. Luiz has given several presentations at conferences such as Apache Sedona: How To Process Petabytes of agrnomic data with Spark, and Perspectives on the use of data in Agriculture, which covers how Leaf uses Apache Sedona to analyze large-scale agricultural data and how Sedona fits into their stack alongside other technologies. Thank you Luiz for your work with the Apache Sedona and Wherobots community and sharing your knowledge and experience!

Apache Sedona: How To Process Petabytes of Agronomic Data With Spark

In this presentation from The Developer’s Conference Luiz Santana shares the experience of using Apache Sedona at Leaf Agriculture to process petabytes of agronomic data from satellites, agricultural machines, drones and other sensors. He discusses how Leaf uses Sedona for tasks such as geometry intersections, geographic searches, and polygon transformations with high performance and speed. Luiz also presented Perspectives On The Use of Data in Agriculture which covers some of the data challenges that Leaf handles and an overview of the technologies used to address these challenges, including Apache Sedona.

See The Slides From The Presentation

Working With Files – Getting Started With Wherobots Cloud

https://wherobots.com/wp-content/uploads/2024/02/Screenshot_2024-02-16_at_9.44.20_AM-1024x842.png?ver=1790894673

This post takes a look at loading and working with our own data in Wherobots Cloud as well as creating and saving data as the result of our analysis, such as the end result of a data pipeline. It covers importing files in various formats including CSV, GeoJSON, Shapefile, and GeoTIFF in Wherobots Cloud, working with AWS S3 cloud object storage, and creating GeoParquet files using Apache Sedona.

Read The Post: Working With Files – Getting Started With Wherobots Cloud

Introducing Sedona 1.5: Making Sedona the most comprehensive & scalable spatial data processing and ETL engine for both raster and vector data

https://wherobots.com/wp-content/uploads/2024/02/Screenshot-2024-02-08-at-2.13.52-PM.png?ver=1790894673

The 1.5 series of Apache Sedona represents a leap forward in geospatial processing that adds essential features and enhancements to make Sedona a comprehensive, all-in-one cluster computing engine for geospatial vector and raster data analysis. This post covers XYZM coordinates and SRID, vector and raster joins, raster data manipulation, visualization with SedonaKepler and SedonaPyDeck, GeoParquet reading and writing, H3 hexagons, and new cluster compute engine support.

Read The Post: Introducing Sedona 1.5 – Making Sedona The Most Comprehensive & Scalable Spatial Data Processing and ETL Engine For Both Raster and Vector Data

Unlocking the Spatial Frontier: The Evolution and Potential of spatial technology in Apple Vision Pro and Augmented Reality Apps

https://wherobots.com/wp-content/uploads/2024/02/VisionPro.001.png?ver=1790894673

Apple adopted the term “spatial computing” when announcing the Apple Vision Pro to describe its new augmented reality platform. This post from Wherobots CEO Mo Sarwat examines spatial computing in the context of augmented reality experiences to explore spatial object localization and presentation and the role of spatial query processing and spatial data analytics in Apple Vision Pro.

Read The Post: Unlocking The Spatial Frontier

Upcoming Events

Subscribe to the Spatial Intelligence Newsletter to get the latest updates on Wherobots, Apache Sedona and geospatial data:

Key takeaways

  • This March 2024 “This Month In Wherobots” newsletter points at other posts rather than introducing new product numbers.
  • Featured member Luiz Santana (Leaf Agriculture) describes using Apache Sedona with Spark to process petabytes of agronomic data from satellites, machines, drones, and sensors.
  • Linked tutorials cover raster analysis with Spatial SQL (vector/raster joins, zonal stats, map algebra), working with files in Wherobots Cloud, Sedona 1.5, and Mo Sarwat’s Apple Vision Pro spatial-computing essay.
  • Upcoming events listed: Sedona office hour April 9, a Neo4j + Sedona training on March 26, Havasu at Subsurface Live on May 2, and a cloud-native geospatial talk at SW2 in Denver on May 15.
  • Leaf’s TDC talk is titled Apache Sedona: How To Process Petabytes of agronomic data with Spark, alongside a second talk on data in agriculture and how Sedona sits in Leaf’s stack.

SedonaSnow: Apache Sedona On Snowflake, Accelerating Your GIS Pipeline, Exploring Global Fishing Watch Data With GeoParquet, and Apache Sedona 1.5.1 Release

Welcome to This Month In Wherobots the monthly developer newsletter for the Wherobots & Apache Sedona community! In this edition we cover SedonaSnow: Apache Sedona on Snowflake, accelerating your GIS pipeline with Apache Sedona, exploring Global Fishing Watch public data with SedonaDB and GeoParquet, and a look at new features and updates in the 1.5.1 release of Apache Sedona.

Apache Sedona Now Available In Snowflake Marketplace: SedonaSnow

SedonaSnow

Apache Sedona, the scalable open-source geospatial compute engine is now available on Snowflake via the Snowflake Marketplace or via manual installation. The SedonaSnow plugin brings Apache Sedona’s Spatial SQL functionality to Snowflake via 130+ Sedona SQL “ST” SQL functions that can be used alongside Snowflake SQL.

Read More About Using SedonaSnow In Snowflake In This Tutorial

This month’s Wherobots & Apache Sedona featured community members are Alihan Zihna, Lead Data Scientist at CKDelta and Fernando Palacios, Director of Data Science & Data Engineering also at CKDelta. Alihan and Fernando presented “GIS Pipeline Acceleration With Apache Sedona” at the Data + AI Summit where they share how they were able to improve the performance and innovation of their geospatial analysis pipelines, going from a pipeline that took 48 hours to complete down to 10 minutes using Apache Sedona. Thank you Fernando and Alihan for being a part of the community and sharing your work!

GIS Pipeline Acceleration With Apache Sedona

GIS pipeline accelerations with Apache Sedona

In this presentation from Data + AI Summit, Fernando and Alihan discuss some of the various usecases for working with large-scale geospatial data at conglomerate CKDelta, part of the Hutchinson Group which operates ports, utility networks, retail stores and mobile telecom networks with hundreds of millions of users across dozens of countries. They discuss how geospatial analytics at scale is important for identifying water leakage in their utility network, understanding customer satisfaction, identifying sites for electric vehicle charging station installation, and forecasting the supply and demand of energy. They provide a technical overview of Apache Sedona and share the results of improving and extending their geospatial analytics pipelines including one process that reduced running time from 48 hours to 10 minutes using Apache Sedona.

Watch the recording of “GIS Pipeline Acceleration With Apache Sedona”

The Wherobots Notebook Environment – Getting Started With Wherobots Cloud & SedonaDB Part 2

Wherobots Initial Notebook

In Part 2 of our Getting Started With Wherobots Cloud & SedonaDB series we dive into the Wherobots Notebook Environment including how to configure and start notebook runtimes, an overview of the sample notebooks included in Wherobots Cloud, and how to use version control like git with notebooks. If you missed it check out Part 1: An Overview of Wherobots Cloud or sign up for a free Wherobots Cloud account to get started directly.

Read More About The Wherobots Notebooks Environment

Exploring Global Fishing Watch Public Data With SedonaDB & GeoParquet

Matched vs unmatched vessels

This post is a hands-on look at offshore ocean infrastructure and industrial vessel activity with SedonaDB using data from Global Fishing Watch. We also see how GeoParquet can be used with this data to improve the efficiency of data retrieval and enable large-scale geospatial visualization using GeoArrow and the Lonboard Python visualization library.

Read “Exploring Global Fishing Watch Public Data With SedonaDB & GeoParquet”

Apache Sedona 1.5.1 Release

Apache Sedona 1.5.1 Release Notes

The most recent release of Apache Sedona introduces some exciting new updates including support for Spark 3.5, 20+ new raster functions, 7 new vector functions, support for running Sedona in Snowflake with SedonaSnow, updates to Sedona’s GeoParquet reader and writer, and more! The updated raster functions include RS_ZonalStats for computing zonal statistics, RS_Tile and RS_TileExplode to enable tiling large rasters, and updates to RS_MapAlgebra to enable user defined raster functions that can work across multiple rasters. Updated vector functions include ST_IsValidReason which exposes the reason geometries might not be valid, and ST_LineLocatePoint which can be useful for map matching and snapping data to road networks.

Read More About Apache Sedona 1.5.1 In The Release Notes

Hands-On With Havasu and GeoParquet

GeoParquet and Iceberg Havasu

Each month you can find a new livestream tutorial on the Wherobots YouTube channel. January’s livestream was all about working with GeoParquet and Havasu tables in SedonaDB. We dig in to understanding some of the optimizations built into the Apache Parquet format to learn how Parquet delivers efficient data storage and data retrieval before exploring the GeoParquet specification for storing geospatial data in Parquet. We cover loading, analyzing, and creating GeoParquet files using SedonaDB with a focus on comparing performance of various GeoParquet partitioning strategies. Finally, we see how the Havasu extension to the Apache Iceberg table format enables working with both vector and raster geospatial data backed by GeoParquet but with the familiar developer experience of SQL tables.

Watch The Recording: Hands-On With Havasu And GeoParquet

Upcoming Events

  • Apache Sedona Community Office Hour (Online Zoom Call – February 13, 2024) – Join the Apache Sedona community for updates on the state of Apache Sedona, presentation and demo of recent features, and provide your input into the roadmap, future plans, and contribution opportunities.
  • Raster Data Analysis With Spatial SQL & SedonaDB (Online Livestream – February 29th, 2024) – This month our livestream is focused on raster data analysis. We’ll see how to load raster data in SedonaDB and perform raster operations like map algebra and zonal statistics using Spatial SQL. Be sure to subscribe to the Wherobots YouTube channel to keep up to date with more Wherobots livestreams and videos!

Want to receive this monthly update in your inbox? Sign up for the Spatial Intelligence Newsletter:

Key takeaways

  • This February 2024 “This Month In Wherobots” newsletter recaps SedonaSnow, a GIS pipeline talk, a Global Fishing Watch tutorial, notebooks part 2, and Apache Sedona 1.5.1.
  • Apache Sedona is available on the Snowflake Marketplace as SedonaSnow, exposing 130+ Sedona ST_ SQL functions alongside Snowflake SQL.
  • Featured CKDelta speakers Alihan Zihna and Fernando Ayuso Palacios describe a geospatial pipeline that went from 48 hours to 10 minutes with Apache Sedona (Data + AI Summit).
  • Sedona 1.5.1 notes in this issue: Spark 3.5 support, 20+ new raster functions, 7 new vector functions, SedonaSnow, GeoParquet reader/writer updates, RS_ZonalStats, RS_Tile/RS_TileExplode, RS_MapAlgebra across rasters, ST_IsValidReason, and ST_LineLocatePoint.
  • Upcoming: Sedona office hour February 13 and a raster Spatial SQL livestream on February 29.