Importing a classic runtime directory

A first and experimental version of import_directory is available from pyltm release 0.19.0. It is not complete and will not cover all edge cases.

Note

Prerequisites:

  1. The dataset must be compatible with the LTM version 10.

  2. The LTM-application konvtiltser.exe must be run before importing loads and market_steps. The application converts the .ENMD-files to the supported format and creates the timeseries.h5 file which contains timeseries for loads and market_steps.

  3. The LTM-application konv_cap_file must be run to convert the the MASKENETT.DATA and cap_file.h5 into a format supported by LTM-API.

  4. The dataset must have been run successfully through the simulation phase after the conversion applications konvtiltser.exe and konv_cap_file.exe.

Currently implemented features:

There are bugs and uncovered edge cases.

EMPS and EOPS are implemented, both are experimental.

Import of time periods

The API will parse LtmSystem.XML when importing a directory. From the time periods defined in the XML file, the API will calculate historical period, data period and simulation period. If FBMC is active, it will open the corresponding ptdf_constraints.h5 file to reconstruct number of active weeks with FBMC.

Note

Intra week timesteps is hardcoded to 168. PRISAVSNITT.DATA is not read when importing a runtime folder.

Import DETD with all restrictions

Import of DETD (detailed hydro specification) is complete. All modules, plants, hydraulic couplings and pumps with restrictions and timeseries is complete. Bugs are still present. Can be disabled.

Import of DCLines

DCLines are read from a file named MASKENETT.DATA.

Supported inputs are:

  • Capacities in capfile (Cap_file.h5)

  • Constant capacities in maskenett.

If there are multiple capacities present in maskenett, the API will throw an error message and refuse to continue. Use convert capfile functionality with konv_cap_file to convert it into a format supported by LTM-API.

Note

Care must be taken when transferring the dataset from Windows to Linux. Casing of filenames are important. The correct casing for maskenett is MASKENETT.DATA. If the file is lower-case, the API will not be able to find it and will throw an error.

Rename the file to MASKENETT.DATA.

Import of market description(.ENMD-files)

The importer parses the .ENMD-files of the dataset. Only areas specified in the LtmSystem.XML file is imported.

Supported imports are:

  • Water value calculation parameters from group 3.

  • Market steps from group 4 of category 10,11,12,13,14,20 and 40 are supported. Can be disabled.

  • Aggregated hydro description is supported. Groups 5, 7, 8, 10

  • Loads from group 16 of category 6 and 7 are supported. Other categories are not. Can be disabled.

  • VRES (wind, solar, etc) from group 18 is supported. Can be disabled.

  • Import of exogenous price series (Prisrekker). Group 17 and group 4

Importer options

Optional importer options include:

  • import_market_steps

  • import_loads

  • recalculate_average_yearly_inflow

  • import_detailed_hydro

  • import_wind

They can be disabled by setting them to False. All options are True by default.

session.import_directory(path,import_market_steps=True,import_loads=True,
    recalculate_average_yearly_inflow=True,import_detailed_hydro=True,import_wind=True)

TODO

A list of items not imported in alphabetical order.

  • Import of Battery

  • Import of FBMC (Flytbasert markedsklarering)

  • Import of Outage and Breakdown (Havari)

  • Import of Price Segments (Prisavsnitt)

  • Anything not mentioned.

Example code

This is a simple import-and-dump example. Given a path to an existing EMPS runtime folder, /path/to/ltm/runtime/directory, it will read that directory, import data, and write the API model to the relative path dump_output_path. The path is relative to the current working directory for the invoking process.

Output files are model.json and timeseries_api.h5. They contain the complete API model with topology and references to timeseries in model.h5. The timeseries are API-compatible, and not LTM-compatible.

import pyltm
from pyltmapi import LtmSession, LtmDot

# Create a session
session = pyltm.LtmApiModule("import_session_name")

# Path to read from
path = "/path/to/ltm/runtime/directory"
session.import_directory(path)
# or with import options
# session.import_directory(path,import_market_steps=True,import_loads=True,
#    recalculate_average_yearly_inflow=True,import_detailed_hydro=True,import_wind=True)

session.dump_model(
    "dump_output_path",
    model_filename="model.json",
    timeseries_filename="timeseries_api.h5",
)

print("areas:")
for bb in session.model.areas():
    print(f"\n{bb}:\n")
    # Uncomment to display detailed hydro topology for this area
    # LtmDot.display_dot_image(bb.build_connection_tree())

print("DCLines:")
for dc in session.model.dclines():
    print(
        f" - {dc.name.strip()}: {dc.from_area.strip()} -> {dc.to_area.strip()}"
    )

Troubleshooting

The marketstep ‘Salg_spotpris’ in area ‘FANTASI’ has water value calculation type ‘VANN’, which is not supported yet. The calculation will be treated as TAPP [unit test logger] LtmApiModel: The marketstep ‘salg_spotpris’ in area ‘FANTASI’ has water value calculation type ‘VANN’, which is not supported yet. The calculation will be treated as TAPP

Automatiske vannverdivekter / manuelle vannverdivekter

Invalid input data for PQ curves for DETD-module ‘GA1’. Continuation/repeat with ‘-1’ is only supported when all defined PQ-curves are within weeks 1-52 inclusive. Remove continuation marker or remove PQ curves. Offending end weeks/SLUTTUKER: 156

Got exception while loading timeseries data for loads and/or market steps. Could not open /home/sintef/gitlab/ltmio/pyltm/test/data/LTM_v10_datasets/EOPS_Fantasi/TidsserieData.h5. Run ‘konvtiltser’ before import. Error message: H5 error encountered: H5Fopen failed. Could not open file ‘/home/sintef/gitlab/ltmio/pyltm/test/data/LTM_v10_datasets/EOPS_Fantasi/TidsserieData.h5’

$ ./konvtiltser ./konvtiltser: error while loading shared libraries: libimf.so: cannot open shared object file: No such file or directory

patchelf –force-rpath –set-rpath “$ORIGIN/.” konvtiltser

patchelf –force-rpath –set-rpath “$ORIGIN/.” konvtiltser

$ patchelf –print-rpath konvtiltser $ORIGIN/.

Bypass restriction curves for ‘HARTEVANN’ have inconsistent data. max_bypass_curve is below min_bypass_curve. check_min_max_linearized: non-overlapping time periods for linearized series. start=Mon 2021.01.04 00:00:00.000, end=Mon 1703.09.17 00:00:00.000

Total buyback percentage of load with id 4 exceeds 100%. Found 100.000005%.

Input error. Scenario number mismatch for price series, name=par-7p-pris.csv, sum of price and historical scenarios= 0. Number of historical years = 50 and number of price scenarios = 1 and the expected number of scenarios = 50