PyPSA-Earth-Status: validating any PyPSA network on Earth

Track the data quality status of PyPSA-Earth globally. Contributions are wanted both on the coding side and by sharing needs and insights on regional energy system modelling.

Development Status: under development

Enabling accurate and reproducible energy system modelling for every region of the world

💡 Any modeling is only as good as the underlying data and assumptions allow. For energy systems modelling it translates into the need to quantify accuracy of modelling inputs and validating optimisation outputs against real-world data.

PyPSA-Earth-Status is a collaborative project aims to make that process faster, easier, and more consistent across PyPSA-meets-Earth community. The workflow is designed to validate PyPSA networks against available energy system statistics in an automated way. It provides automated procedures to compare user-provided PyPSA networks with state-of-the-art databases from authoritative sources such as IRENA, IEA, and others.

By streamlining the validation process, PyPSA-Earth-Status helps modelers enhance the credibility, transparency, and quality of their modeling results saving time and allowing them to focus on what matters most: developing better energy system models.

Contributing

🤝 We warmly welcome contributions to expand the capabilities of PyPSA-Earth-Status and to build a shared foundation for model validation in open energy system research worldwide.

Regional and domain expertise is crucial for interpreting energy statistics and ensure reference validation procedures make sense. We are very interested in experience in energy system modelling and the use of openly available energy system data sources for countries worldwide.

Ways to contribute

1. Flag existing validation needs by opening issues and joining discussions in PyPSA-meets-Earth Discord
2. Suggest improvements by opening issues and pull requests to this repository
3. Contribute non-code insights filling out this form

Functionality

🔍 Automated validation covers the following points for which accurate representation is crucial to ensure energy system optimisation outputs are relevant:

• Installed capacities: p_nom of generators and storage units
• Optimized capacities: p_nom_opt of generators and storage units
• Transmission capacities: s_nom of power lines
• Demand: comparison with real-world consumption data. Currently, electricity only is covered while in future we are planning to also include sector-coupled capabilities

Additionally, we are working to add validation of the energy mix modelled for a reference state of an energy system, such past years.

Features

• Automated validation of PyPSA networks against real-world data
• Creation of reference statistics from leading reference databases
• Generation of tables and visualizations for easy interpretation of results

... and we are aiming for more features with your help!

Currently, the following datasets are supported:

• Installed capacities from IRENA and IEA
• Cross-border line capacities from Global Transmission Database
• Comparison of network data with reference statistics
• Demand data from Our World in Data

The following repositories provide more reference data:

• Awesome-Electrical-Grid-Mapping on datasets and documents related to power grid topology, capacity and development plans
• Awesome-Electricity-Demand collects datasets on electricity demand in hourly resolution

Improving coverage with regionally focused datasets is of high interest for this project. If you see any datasets relevant for your study, feel free to open an issue to flag them, and we can develop an approach to accommodate it into PyPSA-Earth-Status together.

Installation

To install PyPSA-Earth-Status, simply clone the repository:

    git clone https://github.com/pypsa-meets-earth/pypsa-earth-status

The Python dependencies required to run the scripts are defined in this repository. We recommend using the provided Conda environment file.

Create the environment as follows:

    cd pypsa-earth-status
    conda env create -f envs/environment.yaml

Tutorial

To validate a first PyPSA network, you can run the following command in your terminal from the pypsa-earth-status folder:

    conda activate pypsa-earth
    snakemake -j 1 visualize_data

This command will:

1. Create the sample network scigrid_de from PyPSA
2. Save it as resources/example_DE.nc with minimal changes
3. Execute the typical validation procedure to produre tables and visualizations in the folder results/ for easy inspection

Execute your first custom validation

If you want to validate your own PyPSA network, you can:

1. Open the file config.yaml in the pypsa-earth-status folder

2. Specify the path to the network you want to validate in the field network_path under network_validation

3. Adapt the list of countries you want to validate in the field countries under network_validation using 2-letter code naming convention; please keep at least two neighbouring countries in the list, e.g. ['DE', 'FR'] for the tutorial case for Germany.

4. Execute:

       snakemake -j 1 visualize_data

5. Check the results in the results/ folder

Compiling Health Status Reports

If you want to validate multiple solved networks (scenarios) across multiple countries and compare them against different configurable reference sources (e.g., Ember, IRENA, Our World in Data) in a single long-format tidy table:

1. Configure Scenarios: Open the file config.yaml and add your solved PyPSA network files (.nc) under the networks: block. You can add them in two formats:

   • Format A: Explicit Country List (Dictionary format) Use this for multi-country networks (e.g. SAPP) or to restrict validation to a subset of countries. List the file path and the target country codes:

     SAPP:
       path: "results/SAPP/networks/elec_s_40flex_ec_lc1.0_1H.nc"
       countries: ["ZA", "BW", "LS", "MW", "MZ"]

   • Format B: Dynamic Auto-detection (Simple string format) Use this when you want the validation script to automatically discover and validate all countries present in the network buses (n.buses.country):

     NG_2021: "results/NG_2021/networks/elec_s_40flex_ec_lc1.0_1H.nc"

   (Note: Set fallback_pypsa_earth_version under the network_validation block in config.yaml to specify the default version when metadata is absent in the network file).

2. Select Reference Sources: Under the datasets: block, specify the list of sources you want to compare against for demand, capacity, and generation:

   datasets:
     demand: ["ourworldindata", "ember"]
     installed_capacity: ["irena", "ember"]
     electricity_generation: ["ember", "irena"]

3. Execute the rule: Run the following Snakemake command:

   snakemake -j 1 results/health_status.csv

4. Review Results: The tidy long-format comparison is exported to results/health_status.csv. Each row compares a single scenario, country, and metric against a single reference source, detailing the relative error deviation and a validation grade.

   Grades are assigned from the absolute relative deviation between the PyPSA value and the reference value:

   Grade	Deviation from reference
   A	below 5%
   B	5% to below 10%
   C	10% to below 20%
   D	20% or above
   (blank)	metric is not graded

   The mean absolute error metrics (capacity_mae_pct and generation_share_mae) are deliberately left ungraded. They report a deviation, but it summarises the average per-carrier spread rather than a deviation from one reference figure, so their reference_value is empty and the grade thresholds above do not apply to them.

   This output supports two distinct applications:

   1. Validating a custom set of scenarios. Any collection of networks listed under networks: is validated in a single run, so an arbitrary set of countries, regions, or model configurations can be checked together rather than one network at a time.
   2. Tracking the accuracy status of the PyPSA-Earth workflow. Because results persist across runs, the file builds up into an overall picture of how model output compares against reference data over time and across scenarios.

   To serve the second purpose, and unlike the other validation outputs, this file is not written to a per-validation subfolder. It is a global tracker kept at a single stable path so that results accumulate across scenarios and validation configurations: each run replaces only the rows for the scenario-country pairs it just validated and leaves all other rows untouched. Running the rule with no networks: configured therefore leaves any existing results unchanged.