.. _external-data-label:

=============
External data
=============

Some datasets are large. To avoid having large JSON-documents with all the data for all scenarios, data can be sideloaded in an external data store. Currently CSV, HDF5, and SQLite are supported. The easiest way to create a dataset with either method is to use the helper methods in ``create_csv_timeseries.py``, ``create_hdf5_timeseries.py``, or ``create_sqlite_timeseries.py``. See the complete listings below.

CSV
===

The structure for CSV-files is a 2D-array, where each line is starting with one timestamp in milliseconds (UNIX epoch), and all values for all scenarios are comma-separated.

.. code-block:: text

  [timestamp_ms],[value_scenario_0],...,[scenario_n]


.. code-block:: text

  # timestamp_ms, scenario_0, scenario_1, scenario_2, scenario_3, scenario_4, scenario_5, scenario_6, scenario_7, scenario_8, scenario_9, scenario_10
  1,1, 1, 101, 201, 301, 401, 501, 601, 701, 801, 901
  2,2, 2, 102, 202, 302, 402, 502, 602, 702, 802, 902
  3,3, 3, 103, 203, 303, 403, 503, 603, 703, 803, 903

.. warning:: Lines starting with '#' are ignored. The parser is very bare-bones and will choke on regular CSV-headers unless starting with '#'. It is adviced to not use headers in CSV-files.

Usage with ``txy``:

.. code-block:: json

  "external_reference": {
    "type": "csv",
    "filename": "example_dataset.csv"
  },

HDF5
====


.. code-block:: json

  "external_reference": {
    "type": "hdf5",
    "filename": "example_dataset.h5",
    "path": "/global/calc_water_value_period"
  },

``name`` : optional, string
---------------------------

SQLite
======

Usage with ``txy``:

.. code-block:: json

  "external_reference": {
    "type": "sqlite",
    "filename": "example_dataset.sqlite"
    "path": "table_name"
  },

CSV Example
===========

.. collapse:: Create CSV timeseries

  .. literalinclude:: ../../../tests/create_csv_timeseries.py
    :language: python

H5 Example
==========

.. collapse:: Create HDF5 timeseries

  .. literalinclude:: ../../../tests/create_hdf5_timeseries.py
    :language: python

SQLite Example
==============

.. collapse:: Create SQLite timeseries

  .. literalinclude:: ../../../tests/create_sqlite_timeseries.py
    :language: python

Example JSON file with external references:
===========================================

.. collapse:: JSON model

  .. literalinclude:: ../../../tests/busbar_external.json
    :language: JSON