.. _progress_monitor:

# Progress statistics

When RiskScape runs a model, it displays real-time progress statistics on the CLI.
These numbers give you an indication as to how long the model will take to run.

## Example output

The progress statistics output focuses on the number of tuples (rows of data) that RiskScape processes.
Example output from a wizard model might look something like this.

```none
Progress:
  995917 /  2025135      49.18%  exposures_input.complete
  995727 total, 20122.089/s avg: exposures.in
  995727 total, 20122.091/s avg: exposures.out
  995921 total, 20140.616/s avg: exposures_input.out
  994823 total, 19947.583/s avg: exposures_join_areas.in
  994823 total, 19947.583/s avg: exposures_join_areas.out
  995727 total, 20121.692/s avg: exposures_join_hazards.in
  995727 total, 20121.692/s avg: exposures_join_hazards.out
  995242 total, 20058.763/s avg: sample_hazard_layer.in
  995239 total, 20058.474/s avg: sample_hazard_layer.out
  993367 total, 19928.925/s avg: steps-select_3~>report_event-impact-sink.in
```

The first line shows you how far through the input data RiskScape is (i.e. 49.18%
complete for the exposure-layer). Often you can use this as a rough guide to how far through
the overall model RiskScape is.

.. note::
    The percentage complete does not include any invalid tuples or tuples that get removed by
    a bookmark filter. The reported progress could be inaccurate if your bookmark has a ``filter``
    that removes a large number of features. You could potentially move the ``filter`` from your
    bookmark into your model for more accurate progress.

The remaining lines show you a breakdown of the pipeline steps that are currently in progress.
Each line shows you:
- The `total` tuples processed so far.
- The `avg` tuples processed per second.
- The name of the pipeline step(s) that are doing the work.

The name of the pipeline steps are further broken into `in` and `out`.
This is because some pipeline steps can emit more tuples than they consume (e.g. `unnest` and `join` steps),
and others can emit fewer tuples (e.g. `filter` steps).

.. note::
    By default, a summary of these statistics are also saved to a ``model-run-stats.csv`` file in the output directory.
    Although viewing the statistics in real-time makes it easier to see what is happening.

## Pipeline processing

In general, RiskScape tries to 'stream' your input data so that it is spread out through the entire pipeline.

Model processing can involve a large number of data-points.
Potentially this is more data than it is practical to hold in memory all at once (especially true of probabilistic models).
So RiskScape tries to get rid of the data as quickly as it can, by getting the data from one end of the pipeline to the other.

.. mermaid::

    graph LR

      EL("Exposure-layer<br/>Input");
      EL --> GS("Geospatial<br/>matching");
      GS --> CA("Analyse<br/>consequences");
      CA --> REP("Save<br/>results");

      %% Styling
      class EL rs-box-navy
      style EL color:#FFF
      class CA,GS rs-box
      class REP rs-box-green

In a 'waterfall' approach, *all* the input data would be read before moving on to the next step (geospatial matching), and so on.
RiskScape does not do this.
Instead, RiskScape will read just enough input data to keep the rest of the pipeline steps busy.
When the 'geospatial matching' step starts to run out of data, then RiskScape will read some more input data.

The goal of this approach is to make maximal use of your CPU cores, by parallelizing work,
while using your available RAM efficiently, by holding minimal data in memory at once.

What this generally means is that the percentage complete metric for the input step (i.e. `exposures_input.complete`)
is often a good indication for the progress through the model itself.

.. _model_run_stats:

## Troubleshooting performance bottlenecks

The CLI progress will highlight when a model is running slow, but it can still be hard to pinpoint what the specific bottleneck is.
This is partly due to the 'streaming' approach RiskScape takes through the data - if one step in the pipeline is particularly slow,
all the pipeline steps prior to that also slow down, because data cannot move through the pipeline as quickly.

.. tip::
    Before trouble-shooting, check the :ref:`slow_model_tips` section below to see if anything obvious applies to your model.

The simplest way to troubleshoot performance is to look at the `model-run-stats.csv` produced once the model completes.

### Analysing performance
    
Once each model run completes, a `model-run-stats.csv` file is always written in the output directory.
This file contains some basic metrics about how much data each pipeline step processed and how long it took.

The following is a example of output from a simple pipeline that had a performance issue (fixed by issue GL1517).

| `name`                                    | `runtime-ms` | `runtime-average-ms` | `tuples-in` | `tuples-in-per-sec` | `tuples-out` | `tuples-out-per-sec` | `context-switches` |
| ----------------------------------------- | ------------ | -------------------- | ----------- | ------------------- | ------------ | -------------------- | ------------------ |
| `steps-sample_region~>sample_region-sink` | 416360       | 27045.07             | 50487       | 765.21              |              |                      | 16                 |
| `exposures_input`                         | 1342         | 130.31               |             |                     | 50487        | 1117.37              | 9                  |
| `aggregate_by_region`                     | 3            | 3                    |             |                     | 2            | 0                    | 1                  |
| `aggregate_by_region-capped-sink`         | 0            | 0                    | 2           | 0                   |              |                      | 1                  |

The `model-run-stats.csv` contains basic runtime information for both pipeline steps and CPython functions.
Entries for CPython functions will always have a name in the format of `Cpython_Function_<function-id>`.

CPython functions are recorded separately because they can often be a processing bottleneck, particularly
if they are called a lot.  CPython functions are _always_ called by a pipeline step, and so the Python run-time
will be reported twice - once for the CPython function, and once for the pipeline step that calls the function.

The `model-run-stats.csv` file contains the following columns:

- `name`: This is the name of a CPython function, a pipeline step, or a related group of pipeline steps, from the underlying model pipeline code.
  Where multiple pipeline steps have been grouped together, the name is in the format `steps-FIRST_STEP~>LAST_STEP`.
  This lets you map the performance metrics back to a particular step in your pipeline code.
  Note that explicitly naming each pipeline step may help you to debug the performance bottleneck.
- `runtime-ms`: This is the total execution time in milliseconds for the pipeline steps, across all CPU cores.
  Some pipeline steps will use multi-threaded processing, whereas others (for example, `input()`) are single-threaded.
  In this case the regional sampling was multi-threaded across 8 CPU cores, so `(416360 / 8) / 1000` is approximately 52 seconds.
  This particular model run took 60 seconds of elapsed real time to run (this includes typically 5 seconds of initialization and Java start-up time,
  which is not captured in the `model-run-stats.csv`).
- `context-switches`: This is the number of times that a 'worker thread' started processing this particular pipeline task.
  When a worker thread needs to wait for either upstream or downstream pipeline steps to catch up, it "switches context" and processes data
  for a different pipeline step.
- `runtime-average-ms`: This is the average time a worker thread spent processing data for this particular task in one go,
  i.e. the average time before a context switch was needed.
- `tuples-in`, `tuples-out`: The total rows of data either consumed (`in`) or produced (`out`) by the pipeline steps.
  Note that it is normal for some pipeline steps to consume no data (e.g. `input()` steps) or produce no data (e.g. `save()` steps).
- `tuples-in-per-sec`, `tuples-out-per-sec`: The approximate number of rows of data, on average, produced or consumed by the pipeline steps per second.
  Note that this value is averaged across time where the task is idle and waiting for upstream or downstream pipeline steps to catch up.

The `model-run-stats.csv` file is sorted by `runtime-ms`, which means that the slowest pipeline steps will typically always be at the top of the file.

In this example, the GL1517 performance issue occurred when sampling the region.
The model-run stats also highlight that the `sample_region` pipeline step is the bottleneck, accounting for the vast majority of the CPU runtime (i.e. `runtime-ms`).

.. note::
    Be careful when examining the average metrics, as slow upstream or downstream pipeline steps can skew these values.
    The best metric to rely on would be the ``runtime-ms`` value, as that is the *total* CPU runtime consumed by a pipeline step,
    so it is harder to misinterpret.

### Temporary speed-up for debugging

If your model takes several hours to run, or is not completing at all, it is going to be very painful to troubleshoot.
You can see if you can reduce the model run-time down to 5-10 minutes
by reducing the number of exposure-layer features used in the model.
For example, if your model takes two hours to run with 100,000 buildings, then try reducing that down to 5,000 buildings.

One way to reduce your exposure dataset is via a RiskScape bookmark with a `filter` statement.
For example, you could something like the following bookmark specifically for testing performance:

```ini
[bookmark Reduced-Exposure-layer]
location = YOUR_EXPOSURE_DATA.gpkg
description = Selects a random 5% of the data to include in the model
filter = random_uniform(0, 100) <= 5
```

The `filter` statement only includes rows of data when the filter condition is true.
In this case, the filter condition is generating a random number between 0 and 100 and checking if that number is less than or equal to 5,
which should happen roughly 5% of the time. You can adjust the `5` in the `filter` line to include more or less data.

.. tip::
    If your model is still just as slow, it may mean the problem is with the exposure-layer itself.
    For example, loading all the exposure data via WFS could be very slow if it is a large dataset.
    Another problem may be that the model is still trying to consume more RAM than is available on your system.
    For example, cutting a road into 1cm segments is still going to be too much work, even if you only have one road.

## FAQ

### Why do I not see a percentage complete?

Not all input formats support a percentage complete metric, e.g. WFS input data does not.
Some formats, like Shapefile, may not report a percentage complete when a `filter` is configured
for the bookmark.

Furthermore, some model pipelines are organized such that it is hard to report their progress.
For example, you might have an exposure-layer that contains a few hundred roads, but your model then cuts these
roads into small (e.g. 1-metre) pieces.
In this case, RiskScape may read *all* the input data before it gets to the slow part of the model (the cutting).

Unfortunately, it is not practical to report a percentage complete for *every* step in the pipeline.
Due to the nature of pipelines, most steps can easily change the total number of tuples in the model,
which makes it hard to know exactly how many tuples the next step can expect to process.

.. _slow_model_tips:

### Why is my model slow?

Typically, the slowest processing in a model pipeline involves geometry operations.
If your model is running particularly slow, here are some things to check.

- Are you loading remote input data? E.g. your model uses WFS bookmarks or HTTP links.
  If so, try saving a local copy of the data to your file system and using that instead.
- Are you geoprocessing your input data? E.g. cutting or buffering the input geometry.
  If so, you could try:
  - Filtering the input data *before* your cut or buffer.
    This means you do the expensive geometry processing for less of the input data (i.e. only the data you care about).
  - If cutting by distance, try using a larger distance.
  - Avoid doing the geoprocessing *every* time you run the model.
    Instead, you could cut or buffer the geometry once, save the result, and then use that as the input layer to your models.
- Do you have a large (e.g. national-level) dataset and a localized hazard?
  If so, you could try filtering your input data by the bounds of your hazard.
  Refer to the geoprocessing options in the wizard.
- Do you have large polygons in your exposure-layer and a small grid distance for your hazard-layer?
  And are you using *all-intersections* or *closest* sampling?
  E.g. if you had farmland polygons and a 1-metre resolution hazard grid, then the sampling operation
  may still end up cutting each farm polygon into 1-metre segments.
  You could use centroid sampling instead.
  For better sampling accuracy, you could cut the polygons into more reasonable size first (e.g. 100-metre segments),
  and then use centroid sampling.
- Are you using shapefiles for the hazard-layer or area-layer?
  These can hold complex geometry, which can slow down spatial sampling operations.
  Here are a few things to check:
  - Does your area-layer contain a marine or oceanic feature that encloses the other features?
    If so, we recommend using bookmark :ref:`bookmark_filter` to remove this feature, as it will slow down spatial matching.
  - Try running the `riskscape bookmark info` command for your shapefile bookmarks.
    If this command seems to hang for a minute or more, then it may indicate something is wrong with your geometry input data.
    You can use `Ctrl c` to stop the command. Try setting `validate-geometry = off` in your bookmark and repeat.
    Refer to :ref:`invalid-geometry` on how to fix invalid geometry.
  - If you have large hazard-layer or area-layer shapes, try cutting the geometry into smaller pieces (e.g. 1km by 1km polygon segments)
    using the geoprocessing features in the wizard.
    This should not make any difference to the results, but it can mean that the geospatial matching is quicker
    because it is matching against smaller geometry.
- Processing vector-layer multi-polygon data can be slow if the multi-polygons span distant locations.
  For example, if a single multi-polygon contains a polygon sub-component in Christchurch, and another polygon sub-component in Auckland.
  In this case, segmenting the multi-polygons may improve performance (you should be able to use a large cut-distance).
  This can also be an issue when using a region-layer that is a single complex multi-polygon for the entire country.
- Are you filtering your *output* results when you could be filtering your *input*?
  It's more efficient to filter out any data as early in the model as you can.
  So if you were filtering by region say, you would want to filter in the *geoprocessing* phase
  of your model, rather than in the *reporting* phase (which is after the bulk of the processing work has been done).
- If you have lots of rows of data and are aggregating it, some aggregation operations are more efficient than others.
  - For example, `count()`, `sum()`, and `mean()` should be fairly efficient, whereas `stddev()`, `percentile()`, and others
  can consume a lot more memory. You could try temporarily replacing aggregation functions with `sum()` or `count()`
  as a sanity-check, and see if performance improves. You could also try filtering out unnecessary data
  before you aggregate it.
  - The `to_list()` and `to_lookup_table()` functions can also consume a large amount of RAM.  Try reducing
  the number of attributes being stored, especially if they are geometry or text.  Large amounts of
  floating point numbers can be stored more efficiently using the :ref:`smallfloat`
- By default, the memory that RiskScape can consume is capped by Java.
  If you have plenty of free RAM available on your system, you could try increasing this Java limit and see if RiskScape runs more efficiently.
  For more details on how to do this, see :ref:`java_memory`.
- If your exposure-layer and hazard-layer are in different CRS, then it can be more efficient to reproject
  the exposure-layer to the hazard-layer's CRS. In particular, if you have large exposure-layer features and a small hazard grid.
  You should be able to reproject geometry by adding a line similar to this to your bookmark:
  ```
  set-attribute.geom = reproject(*, 'EPSG:2197')
  ```

For advanced users who have written their own pipeline code, you could also check:

- Are you running a probabilistic model with a large number of data-points?
  The total data-points will be your number of elements-at-risk multiplied by the number of events.
  If this seems like a big number, try thinking of ways to process the data more efficiently.
  For example, you could filter (i.e. remove) data you are not interested in,
  or use interpolation (refer to the `create_continuous()` function) so that your Python code gets called less.
- Are you manually joining two different datasets or layers together, e.g. based on a common attribute?
  If so, make sure that the *smaller* dataset is on the `rhs` of the `join` step.
- Are you doing any unnecessary spatial operations?
  E.g. if you are filtering your data, then do any geospatial matching to region *after* the filter, not before.
- Do you have large polygons (i.e. land area) that you are sampling repeatedly against different GeoTIFFs?
  The repeated sampling operations will repeatedly cut your polygons, so it can be quicker to cut your polygons
  *once* up front, to match the GeoTIFF grid resolution (use the `segment_by_grid()` function).
