Reporting

FlexMeasures feeds upon raw measurement data (e.g. solar generation) and data from third parties (e.g. weather forecasts).

However, there are use cases for enriching these raw data by combining them:

Pre-calculations: For example, from a tariff and some tax rules we compute the real financial impact of price data.
Post-calculations: To be able to show the customer value, we regularly want to compute things like money or CO₂ saved.

These calculations can be done with code, but there’ll be many repetitions.

We added an infrastructure that allows us to define computation pipelines and CLI commands for developers to list available reporters and trigger their computations regularly:

flexmeasures show reporters
flexmeasures add report

The reporter classes we are designing are using pandas under the hood and can be sub-classed, allowing us to build new reporters from stable simpler ones, and even pipelines. Remember: re-use is developer power!

We believe this infrastructure will become very powerful and enable FlexMeasures hosts and plugin developers to implement exciting new features.

Below are two quick examples, but you can also dive deeper in Toy example V: Computing reports.

Example: solar feed-in / self-consumption delta

So here is a glimpse into a reporter we made - it is based on the AggregatorReporter (which is for the combination of any two sensors). This simplified example reporter basically calculates pv - consumption at grid connection point. This tells us how much solar power we fed back to the grid (positive values) and/or the amount of grid power within the overall consumption that did not come from local solar panels (negative values).

This is the configuration of how the computation works:

{
    "method" : "sum",
    "weights" : {
        "pv" : 1.0,
        "consumption" : -1.0
    }
}

This parameterizes the computation (from which sensors does data come from, which range & where does it go):

{
    "input": [
        {
            "name" : "pv",
            "sensor": 1,
            "source" : 1,
        },
        {
            "name" : "consumption",
            "sensor": 1,
            "source" : 2,
        }
    ],
    "output": [
        {
            "sensor": 3,
        }
    ],
    "start" : "2023-01-01T00:00:00+00:00",
    "end" : "2023-01-03T00:00:00+00:00",
}

Note

In addition to filtering by specific data source IDs (source / sources), reporter input data can be filtered using:

source_types: list of source type names to include (e.g. ["forecaster", "scheduler"])
exclude_source_types: list of source type names to exclude
account_id: list of account IDs to include only data from sources belonging to those accounts (note: only matches user-type data sources — DataSources created by reporters, schedulers, and forecasters have no account and will not be matched by this filter)

These correspond to the same filters available on Sensor.search_beliefs.

Example: Profits & losses

A report that should cover a use case right off the shelf for almost everyone using FlexMeasures is the ProfitOrLossReporter ― a reporter to compute how profitable your operation has been. Showing the results of your optimization is a crucial feature, and now easier than ever.

First, reporters can be stored as data sources, so they are easy to be used repeatedly and the data they generate can reference them. Our data source has ProfitOrLossReporter as model attribute and these configuration information stored on its attribute defines the reporter further (the least a ProfitOrLossReporter needs to know is a price):

{
  "data_generator": {
    "config": {
      "consumption_price_sensor": 1
    }
  }
}

And here are more excerpts from the tutorial mentioned above. Here we configure the input and output:

$ echo "
  {
      'input' : [{'sensor' : 4}],
      'output' : [{'sensor' : 9}]
  }" > profitorloss-parameters.json

The input sensor stores the power/energy flow, and the output sensor will store the report. Recall that we already provided the price sensor to use in the reporter’s data source.

$ flexmeasures add report\
  --source 6 \
  --parameters profitorloss-parameters.json \
  --start-offset DB,1D --end-offset DB,2D

Here, the ProfitOrLossReporter used as source (with Id 6) is the one we configured above. With the offsets, we control the timing ― we indicate that we want the new report to encompass the day of tomorrow (see Pandas offset strings).

The report sensor will now store all costs which we know will be made tomorrow by the schedule.

Automating reports

Reports can be queued as background jobs (add --as-job to flexmeasures add report, and let a worker process the reporting queue, see Redis Queues), and computed on a recurring basis by an automation defined on the asset (see Automations for the full concept, including how to manage and run automations).

One-off reporting jobs can also be triggered via the API, mirroring how forecasts and schedules are triggered. POST /api/v3_0/assets/<id>/reports/trigger takes the reporter class, its configuration and the report parameters (which must reference at least one output sensor, assumed to belong to the asset or one of its (grand)children):

{
    "reporter": "PandasReporter",
    "config": {
        "required_input": [{"name": "sensor_1"}, {"name": "sensor_2"}],
        "required_output": [{"name": "df_agg"}],
        "transformations": [
            {"df_input": "sensor_1", "method": "add", "args": ["@sensor_2"], "df_output": "df_agg"},
            {"method": "resample_events", "args": ["2h"]}
        ]
    },
    "parameters": {
        "input": [{"name": "sensor_1", "sensor": 1}, {"name": "sensor_2", "sensor": 2}],
        "output": [{"name": "df_agg", "sensor": 3}],
        "start": "2023-04-10T00:00:00+00:00",
        "end": "2023-04-10T10:00:00+00:00"
    }
}

The response contains the id of the queued reporting job, whose status can be polled via the generic job status endpoint (GET /api/v3_0/jobs/<uuid>).

The reporter and its configuration are stored on a data source (steady across runs, so all report results attribute to the same source), while the report parameters are stored on the automation itself and their timing is resolved freshly on each run:

Use start-offset and/or end-offset fields (comma-separated Pandas offsets, like the CLI options above) for a rolling window relative to the run time, in the timezone of the first output sensor. For instance, "start-offset": "-1D,DB" with "end-offset": "DB" reports on the whole previous day.
Omit timing fields entirely to report on the period since the automation’s actual last run (falling back to the last cron period — from the previous cron fire time until the run time — when no last run is known, e.g. on the first run).
Absolute start/end fields are also accepted, but draw a warning, as each run would then compute the same period.

For example, this automation computes a report over each past day, every morning at 1 AM:

flexmeasures add automation --asset 3 --name "Daily aggregation report" --cron "0 1 * * *" --type reports \
  --reporter PandasReporter --config reporter-config.yml --parameters report-parameters.yml