dayjournal | Wherobots #004 - Overture MapsのデータをPMTilesに変換

Overture MapsのデータをPMTilesに変換するメモ。

WherobotsDBを利用

WherobotsDBを利用するために「SedonaContext」オブジェクトを作成します。

from sedona.spark import *

config = SedonaContext.builder().getOrCreate()
sedona = SedonaContext.create(config)

範囲指定したデータを抽出

東京周辺の範囲を指定します。

from sedona.sql.st_constructors import ST_GeomFromText
from sedona.sql.st_predicates import ST_Intersects

region_wkt = "POLYGON ((139.7543 35.7044, 139.7371 35.6921, 139.7377 35.6758, 139.7498 35.6600, 139.7734 35.6658, 139.7841 35.6793, 139.7845 35.7023, 139.7543 35.7044))"

Overture Mapsの「buildings」テーブルのDataFrameを作成します。

import pyspark.sql.functions as f

buildings_df = (
    sedona.table("wherobots_open_data.overture_2024_02_15.buildings_building")
    .select(
        f.col("geometry"),
        f.lit("buildings").alias("layer"),
        f.element_at(f.col("sources"), 1).dataset.alias("source")
    )
)
buildings_df.show()

+--------------------+---------+--------------------+
|            geometry|    layer|              source|
+--------------------+---------+--------------------+
|POLYGON ((-49.438...|buildings|Microsoft ML Buil...|
|POLYGON ((-49.438...|buildings|Google Open Build...|
|POLYGON ((-49.438...|buildings|Microsoft ML Buil...|
|POLYGON ((-49.438...|buildings|Google Open Build...|
|POLYGON ((-49.441...|buildings|Google Open Build...|
|POLYGON ((-49.440...|buildings|Google Open Build...|
|POLYGON ((-49.441...|buildings|Microsoft ML Buil...|
|POLYGON ((-49.441...|buildings|Google Open Build...|
|POLYGON ((-49.440...|buildings|Microsoft ML Buil...|
|POLYGON ((-49.442...|buildings|Google Open Build...|
|POLYGON ((-49.442...|buildings|Google Open Build...|
|POLYGON ((-49.442...|buildings|Google Open Build...|
|POLYGON ((-49.442...|buildings|Google Open Build...|
|POLYGON ((-49.442...|buildings|Microsoft ML Buil...|
|POLYGON ((-49.442...|buildings|Google Open Build...|
|POLYGON ((-49.438...|buildings|Google Open Build...|
|POLYGON ((-49.438...|buildings|Google Open Build...|
|POLYGON ((-49.439...|buildings|Google Open Build...|
|POLYGON ((-49.439...|buildings|Google Open Build...|
|POLYGON ((-49.438...|buildings|Microsoft ML Buil...|
+--------------------+---------+--------------------+
only showing top 20 rows

Overture Mapsの「roads」テーブルのDataFrameを作成します。

roads_df = (
    sedona.table("wherobots_open_data.overture_2024_02_15.transportation_segment")
    .select(
        f.col("geometry"),
        f.lit("roads").alias("layer"),
        f.element_at(f.col("sources"), 1).dataset.alias("source")
    )
)
roads_df.show()

+--------------------+-----+-------------+
|            geometry|layer|       source|
+--------------------+-----+-------------+
|LINESTRING (7.034...|roads|OpenStreetMap|
|LINESTRING (7.037...|roads|OpenStreetMap|
|LINESTRING (7.032...|roads|OpenStreetMap|
|LINESTRING (7.033...|roads|OpenStreetMap|
|LINESTRING (7.031...|roads|OpenStreetMap|
|LINESTRING (7.031...|roads|OpenStreetMap|
|LINESTRING (7.031...|roads|OpenStreetMap|
|LINESTRING (7.033...|roads|OpenStreetMap|
|LINESTRING (7.034...|roads|OpenStreetMap|
|LINESTRING (7.030...|roads|OpenStreetMap|
|LINESTRING (7.037...|roads|OpenStreetMap|
|LINESTRING (7.037...|roads|OpenStreetMap|
|LINESTRING (7.041...|roads|OpenStreetMap|
|LINESTRING (7.051...|roads|OpenStreetMap|
|LINESTRING (7.037...|roads|OpenStreetMap|
|LINESTRING (7.050...|roads|OpenStreetMap|
|LINESTRING (7.054...|roads|OpenStreetMap|
|LINESTRING (7.051...|roads|OpenStreetMap|
|LINESTRING (7.052...|roads|OpenStreetMap|
|LINESTRING (7.052...|roads|OpenStreetMap|
+--------------------+-----+-------------+
only showing top 20 rows

Overture Mapsの「buildings」と「roads」を組み合わせた東京周辺のDataFrameを作成します。

features_df = roads_df.union(buildings_df)

if filter:
    features_df = features_df.filter(ST_Intersects(f.col("geometry"), ST_GeomFromText(f.lit(region_wkt))))

features_df.count()

ベクトルタイルを作成

抽出したデータのベクトルタイルを作成します。

from wherobots import vtiles

tiles_df = vtiles.generate(features_df)
tiles_df.show(3, 150, True)

-RECORD 0----------------------------------------------------------------------------------------------------------------------------------------------------------
 tile     | {56, 25, 6}                                                                                                                                            
 features | [1A DA B8 02 0A 05 72 6F 61 64 73 12 16 12 02 00 00 18 02 22 0E 09 A4 36 82 0D 22 01 01 01 00 01 00 01 01 12 10 12 02 00 00 18 02 22 08 09 98 36 F2... 
-RECORD 1----------------------------------------------------------------------------------------------------------------------------------------------------------
 tile     | {14552, 6453, 14}                                                                                                                                      
 features | [1A 8E 01 0A 05 72 6F 61 64 73 12 3E 12 02 00 00 18 02 22 36 09 FE 0A 8A 08 AA 01 01 0F 03 27 1E 8F 03 04 2D 06 27 22 83 02 14 9D 01 06 31 06 35 1E... 
-RECORD 2----------------------------------------------------------------------------------------------------------------------------------------------------------
 tile     | {29104, 12900, 15}                                                                                                                                     
 features | [1A 90 20 0A 05 72 6F 61 64 73 12 14 12 02 00 00 18 02 22 0C 09 D4 23 D6 3D 1A 05 56 03 38 29 0E 12 43 12 02 00 00 18 02 22 3B 09 88 21 D0 3A C2 01... 
only showing top 3 rows

ベクトルタイルをPMTilesに変換

ベクトルタイルをPMTilesに変換します。

import os

full_tiles_path = os.getenv("USER_S3_PATH") + "tiles.pmtiles"
vtiles.write_pmtiles(tiles_df, full_tiles_path, features_df=features_df)

PMTilesのデータを可視化

LeafmapでPMTilesのデータを可視化します。

vtiles.show_pmtiles(full_tiles_path)

簡素化したPMTilesのデータを可視化

大量なデータを簡素化して可視化します。

sample_tiles_path = os.getenv("USER_S3_PATH") + "sampleTiles.pmtiles"
vtiles.generate_quick_pmtiles(features_df, sample_tiles_path)

参考文献
Wherobots

dayjournal memo