Generating geoparquet for OSM planet PBF

[laurentpellegrino]

Hey there!

We're trying to use quackosm at OpenPlanetData to convert a full‑planet OSM PBF into GeoParquet, but it’s consuming more than 128 GB of RAM, beyond our system’s capacity.

Are there configuration options that reduce memory usage (at the cost of longer runtimes) while still producing an identical result?

[RaczeQ]

Hello @laurentpellegrino

Thank you for this question. I have to admit that I didn't have the resources to test the whole planet file on my machine. But I have an idea what could be improved to make it possible to parse a whole planet file.

I suspect that having many CPUs and working in a multiprocessing environment might explode RAM in DuckDB or Polars beyond available resources. I will test it locally with a smaller example, but will scale resources accordingly.

Could you share some information that could help me pinpoint the problem?

• Number of available CPUs
• Operating Software (Windows / Linux / macOS)
• If possible, the step at which the script failed? It should be logged to stdout with verbosity equal to transient (default in Python) or verbose (default in CLI).

[laurentpellegrino]

@RaczeQ Thanks for your answer.

• Number of available CPUs: 32 cores
• Memory: tested on a 64GB and a 128GB memory nodes
• Operating Software (Windows / Linux / macOS): Fedora 42
• If possible, the step at which the script failed? The crash is caused by a lack of memory on the node. The node freeze at that time and had to reboot.

For your information, as a workaround I tried ohsome‑planet which works on a planet OSM PBF with 46GB at peak using the same nodes.

[onnimonni]

I also tried to convert the whole planet:

# download the whole dataset from openplanetdata.com
$ rclone copy \
    --http-url https://download.openplanetdata.com \
    :http:osm/planet/pbf/planet-latest.osm.pbf . \
    --multi-thread-cutoff 0 \
    --multi-thread-streams 64 \
    --multi-thread-chunk-size 512M \
    --transfers 1 --progress
# try to convert it to parquet
$ quackosm planet-latest.osm.pbf --output planet-latest.osm.parquet

And this fails on my server with 128gb of ram available.

But also doing this for "just" Europe fails with SIGKILL:

$ quackosm --osm-extract-query 'europe' --output europe.parquet"
100%|██████████████████████████████████████| 32.8G/32.8G [00:00<00:00, 369TB/s]
⠏ [   1/33] Reading nodes • 0:01:51
⠋ [   2/33] Filtering nodes - intersection • 0:00:00
⠏ [   3/33] Filtering nodes - tags • 0:00:11
⠇ [   4/33] Calculating distinct filtered nodes ids • 0:00:02
⠏ [   5/33] Reading ways • 0:02:15
⠦ [   6/33] Unnesting ways • 0:01:15
⠏ [   7/33] Filtering ways - valid refs • 0:01:15⏎
~ [SIGKILL]>

I assume both of them fail because it runs out of memory.

[RaczeQ]

The changes with automatic CPU limiting for DuckDB have been added to version 0.16.0.