.. _changelog:

# Release Changelog

## v1.11.0

*Built 13 October 2025*

### Enhancements

- The :ref:`beta-plugin` can now save raster outputs in the GeoTIFF format.
  See :ref:`geotiff_output_example` for how to use the new feature. (GL#1541)

- The :ref:`beta-plugin` now supports a `subpipeline()` pipeline step, which allows
  pipeline code to be more easily reused by other models.
  Refer to :ref:`subpipelines` for more details. (GL#1458)

- RiskScape now supports loading multiple *vector* hazard-layers dynamically.
  Previously, only raster data could be loaded dynamically.
  Refer to :ref:`multi_file_hazard` for more details on how to do this.
  More specifically, the `bookmark()` function no longer needs a `type` argument in some situations when loading
  "dynamic" data. (GL#1464)

- RiskScape now produces improved diagnostics to aid in troubleshooting performance bottlenecks.
  A `model-run-stats.csv` file is now included in the output directory when a model run finishes
  (which replaces the `stats.txt` file generated previously).
  Refer to :ref:`model_run_stats` for more details on using these performance statistics. (GL#1497)

- Various performance improvements have been made, particularly around spatial sampling. These include:
  - Optimizations to the algorithm used by the `sample()` function (i.e. 'all intersections' sampling) to avoid
    unnecessary computations. (GL#1510)
  - Optimizations for sparse GeoTIFFs to improve sampling performance with layers
    that contain many empty tiles. For example, flood layers. (GL#1511)
  - Improved the performance of `sample_one()` function (i.e. 'closest' sampling) when sampling
    line-string geometries against a coverage of polygons. (GL#1517)
  - Small performance improvements (15-20%) when sampling all-intersections of an exposure, especially
    for large features across a small grid. (GL#1508, GL#1509)

- Better support has been added for managing sensitive API keys and access tokens within project INI files.
  Refer to :ref:`secrets_ini` for more details. This includes:
  - Remote data from Koordinates services can now be accessed without embedding API keys/tokens in URLs.
    This lets you share WFS bookmarks for LINZ or Stats NZ data without revealing your API token. (GL#1503)
  - Bookmarks now support a `requires-secret` parameter to indicate that a 'secret', such as a LINZ API key,
    will be required in order to access the bookmarked data. (GL#1502)
  - Data from the RiskScape Platform can be accessed directly from CLI projects, without having to manually download the files first.
    Refer to the :ref:`RiskScape Platform secrets example<secrets_platform>` for how to configure this. (GL#961)

- When the :ref:`beta-plugin` is enabled, the `to_relation()` function now supports converting a GeoTIFF into
  relational data. This will treat each pixel in the GeoTIFF as a separate row of data.
  Previously, this was only supported by the `input()` pipeline step, but now RiskScape also supports
  converting raster to vector data when the data is loaded dynamically.
  In addition, the `unnest()` pipeline step can now unnest a relation directly, whereas
  previously you had to use `to_list()` to turn the relation into a list first. (GL#1527)

- New RiskScape functions have been added to the :ref:`beta-plugin`, including:
  - `assert()` and `warning()` functions, which let you terminate a model or log
    a warning when unexpected conditions are encountered. (GL#1488)
  - A `get_attr()` function, which lets you safely check whether or not a given attribute is present
    in the input data, and use an appropriate default if absent. (GL#1529)

- The `python()` step now accepts parameters which are passed from the pipeline through to the `def function` Python code.
  This makes it easier to pass model parameters directly to your Python script. (GL#1484)

### Fixes

- Updated INI parser to remove trailing white space. Previously any white space characters would be
  added to filepaths which would cause unhandled exception on Windows. (GL#1548)

- Fixed a potential null pointer exception when sampling from relation-backed coverages
  (for example, when `to_coverage()` was used with vector data). (GL#1547)

- Empty CSV, GeoJSON and KML outputs (i.e. containing no features) now produce valid files. (GL#1522)

- Fixed an unhandled exception when the `python()` step is executed with input with different, but equivalent,
  struct types. (GL#1487)

- Identified types can now be used as the `result-type` in the `python()` step. (GL#1466)

- Fixed an unhandled error when the `python()` step is called with input containing unsupported data types. (GL#1465)

- A better error message is now displayed when trying to use an input geospatial file that is
  in an incompatible or unsupported format. (GL#1530) 

### Minor

- Model parameters would appear in the RiskScape Platform in a random order. This has been resolved
  so that the order that the parameters appear in the pipeline code is used. (platform#718)

- Coverages no longer report a 'referenced' type, as it is unnecessary for coverage types. (GL#1489)

- The `centroid()` function now preserves the 'referenced' type of the geometry. (GL#1541)

- The `bounds()` function now correctly reports the bounding polygon for dynamic arguments
  (i.e. where the given layer argument is potentially changing for each row of input data). (GL#1489)

- The `to_coverage()` function no longer needs a referenced geometry type in order to work.  (GL#1462).
  
## v1.10.0

*Built 30 June 2025*

### Enhancements

- The `intersection_cut` option for `to_coverage` now defaults to true for polygon geometries. In most cases this
  should improve the performance when sampling the produced coverage. (GL#1392)

- `list_to_columns` can now accept zero as the columns number. This can be used to remove all attributes
  from the result. (GL#1427)

- A `assert_not_null` function has been added to remove the nullable wrapping from the given input. (GL#1446)

- An experimental `compress` option has been added to `create_continuous_2d`.  Using this feature
  can reduce the amount of RAM needed by this function by storing all the known data points
  in a less precise, but more compact, format. (GL1440)

- A `nullable` option is added to `list_to_columns` that can be set to false to prevent
  output columns having a nullable type. (GL#1433)

- Added a `filter` function to filter list contents. (GL#1431)

- The `to_lookup_table` can now be used with lists (outside of a `group` step) (GL#1404).

- CSV outputs are saved with a Byte Order Mark (BOM) for better compatibility with MS Excel. (GL#1419)

- The current project directory is now added to CPython's `sys.path`, allowing you to import 
  code into inline functions. (GL#723)
- `apply_continuous` now accepts nullable curves which allows it to be called with curves
  from a lookup table. (GL#1403)
- Adds support for a new `smallfloat` type.  This allows for the collection of larger floating
  point datasets with less RAM (#1402)
- Aggregating tuples to lists (via `to_list`) now stores them far more efficiently, allowing
  larger datasets to be processed with less memory (GL#1406).

### Fixes

- Fixed a bug that could cause an unexpected exception if a function was defined with a bad
  type definition. (GL#1445)

- Fixed a bug in `map` and `zip` where the resulting lists could contain the wrong results when
  closing over, or using, the entire scope outside of the function. (GL#1414)

- Fixed bug where CPython processes were left running when used in a lambda expression as part
  of a `map`, `zip` or an expression function call. (GL#1439, GL#1443)

- Fixed bug where functions that reference missing (or erroneous) user defined types would cause
  unhandled exceptions. (GL#1438)

- Fixed bug when returning lists of structs from `jython` functions causing unhandled exceptions. (GL#1434)

- Fixed a bug with the CPython Reaper that would sometimes produce spurious error messages. (GL#866) 

- Fixed issue with expression functions, with inferred argument types, where the function would
  return null rather than the expected value when one of the inputs was null. (GL#1426)

- Fixed exception when transforming null vector data into a coverage. (GL#1409)

- Fixed issue with expression functions where an expression that returned a nullable type would
  result in a function with a return type of anything. (GL#1423)

- Fixed issue where the CPython step would hang if running without enough pipeline threads. (GL#1400)

- Fixed exception when sampling an empty coverage. (GL#1405)

- Various fixes for the new project file (`.ini`) parser. (GL#1985, GL#1393, GL#1399, GL#1408, GL#1412)

- Fixed unhandled exception in `bucket_range` when `range` is of mixed floating and integer
  values and `add_bounds` is false (GL#1398).

### Breaking

- The new project file (`.ini`) parser has been enabled by default. You can switch to the legacy parser
  by adding:

  ```ini
  [engine]
  use-legacy-ini = true
  ```
  
  to your `settings.ini`.
  For more details, see :ref:`multiline-values`

## v1.9.0

*Built 12 March 2025*

### Enhancements

- RiskScape functions can now be defined by using the RiskScape expression language.  See
  :ref:`expression_functions` for more details.

- RiskScape functions can now be written "in-line" within your project's INI files.  Use
  the `source` parameter to define your function without having to create a new project
  file, e.g:

  ```
  [function echo]
  framework = cpython
  source = '''
  def function(message):
      return message + ' back at you'
  '''
  ...
  ```

  *Note*: This feature works best when used in tandem with the new INI parser and
  multi-line strings (as shown in the example above).

- CPython functions called in the `python()` pipeline step can now register
  arbitrary output files to RiskScape. Registered files will be moved to the
  configured output directory and reported along with other outputs on model
  completion. (GL#1250)

- Updated documentation on the difference between the new Python step and Python functions. (GL#1372)

- A new `seed` argument has been added to the RiskScape functions `random_uniform()`,
  `random_norm()`, and `random_choice()`. The seed lets you reproducibly control random
  number generation for a specific feature - for example, based on a building's `ID` attribute.
  The `random_uniform()` and `random_norm()` functions now also accept a `samples` argument,
  which lets you generate a list of random numbers.  (GL#748)

- Various errors with RiskScape expression now include a more precise source code location,
  showing both the line and the column. (GL1381)

- A new experimental parser has been added for project configuration files (`.ini` files).  This
  feature is off by default and is enabled by setting `use-ini-ng = true` to the `[engine]`
  section of your RiskScape settings. (GL#647)

  This new parser supports multi-line values (GL#1382) using triple-quote syntax, e.g.:

  ```
  [model count-buildings]
  pipeline = """
  input('buildings')
  ->
  filter(roof_height > 1)
  ->
  group(by: use_type, select: count(*))
  """
  ```

### Fixes

- Use the step name in scheduler log messages. (GL#1390)

- Better error messages for the python step if the result-type is specified incorrectly. (GL#1372)

- Corrected help text for 'group by' step. (GL#1364)

- Fixed unhandled exception in `to_list` when null number types are collected to a list. (GL#1377)

## v1.8.0

*Built 11 December 2024*

### Enhancements

- Beta support for processing a whole pipeline via a CPython function. This makes it easier
  to use libraries like pandas within your pipeline. (GL#1362)

- A new function `remove_attr` has been added to the :ref:`beta-plugin`. This function allows you to remove attributes
  from a struct without having to specify everything that you want to keep. (GL#1359)

- Slight improvement for error messages when a pipeline fails to realize - the error message now shows
  the line number in the pipeline source for the step that has failed. (GL#1355)

- RiskScape docker images are now based off `Ubuntu 24.04`. This upgrades the Python version used in the CLI image
  to 3.12. This will not affect anyone that installs RiskScape from `.zip` file. (GL#1213)

- The `aal_trapz()` function can now be used outside of `group` aggregation steps.
  To use it like this, a list of the losses and a list of exceedance probabilities
  should be passed to the function. (GL#1062).

- A worked example has been added to the documentation on how to use RiskScape with Jupyter Notebook. (GL#291)

- New `settings.ini` configuration has been added that allows you to refine the scheduler performance.
  The new settings are `page-size` and `tuples-per-task`, under a new `[engine]` section. (GL#1367)

### Fixes

- The `--print` option for the `riskscape model run` command will now display details about a RiskScape pipeline model
  that contains realization errors. This can be helpful in tracking down the cause of the realization problem. (GL#720)

- Memory utilization has been improved for complex pipelines that involve large `join` or `group` steps. (GL#1361)

### Breaking

- The docker image version of RiskScape replaces the `ubuntu` user with a `riskscape` user and defaults
  to running the container as that user.

- How 'pipeline' framework models get loaded from the `project.ini` file has been reworked slightly.
  As such, now the `riskscape model list` command will only show you *syntactic* errors in the pipeline code.
  Whereas previously the command would also show you errors that would prevent the pipeline from running
  (for example, forgetting to chain two steps together). (GL#1327)

### Minor

- The Beta plugin can now be controlled using the CLI flags `--beta` and `--no-beta`.

## v1.7.0

*Built 17 September 2024*

### Enhancements

- A `list_to_columns` function has been added. This function is similar to the beta `list_to_tuple` function but
  has improved options. The `list_to_tuple` function has been deprecated and will be removed in a future release. (GL#1347)

- The `create_continuous_2d` function now has a `zero_loss` option. When set zero losses (from the loss function)
  are tracked and later uses of the curve will return the zero loss if the x and y values are below a previously
  found zero loss. This can improve performance. (GL#1340)

- Default values can be specified for parameter templates. Templates with the same name as parameters will be used
  automatically, unless manually overridden (GL#1309)

- The wizard now supports aggregating into buckets (GL#1313)

- The `if_then_else` function has been updated to:
  - accept named arguments.
  - allow the then/else arguments to accept lambda expressions. This improves performance as
    lambda expressions are only evaluated if required.
  - alias this function as `if`.
  (GL#1159)

- The [GeoTools](https://www.geotools.org/) library used by RiskScape for geometry processing has been updated
  to version 31.2.

  The GeoTools update includes security fixes for
  [Remote Code Execution vulnerability in evaluating XPath expressions](https://github.com/geotools/geotools/security/advisories/GHSA-w3pj-wh35-fq8w). (GL#1314)

- The `to_coverage` function has a new optimization available - see the function's
  documentation for more details (GL#1303).

- A new "ancestry rule" system has been added to RiskScape's type system to allow
  a wider variety of situations where types can be combined in lists, while
  preserving as much of the common type information as possible. (GL#1298)

- Added the "file" parameter property, which is like a text parameter, but in the
  RiskScape platform will display a file chooser. Useful when you want a user to 
  specify a file location, but not control other bookmark attributes. (GL1307)

### Fixes

- Fixed the handling of HDF5 arrays. (GL#352)

- Fixed encoding of boolean attributes in GeoPackage output files. (GL#1354)

- Shapefile output files generated by RiskScape will now load in ArcGIS with the correct CRS projection.
  Previously, the `.prj` files for Shapefile outputs were generated with newlines,
  which caused compatibility issues with ArcGIS. (GL#1346)

- Fixed error with bookmark identifiers that begin with an apostrophe. (GL#1342)

- Updated `layer_intersection` function so the `merge_attributes` and `return_difference`
  are both optional. That means you can now call the function using none, either, or both
  of these arguments. (GL#1344)

- Fixed issue preventing bookmarks working where the location contained spaces.
  This includes projects loaded remotely via the internet (GL#1341)

- Bucket range expression no longer generates attribute names with dots,
  e.g. `range_10_0_20_0` instead of `range_10.0_20.0` (GL#1337)

- Fixed null pointer exceptions in equality expressions when null values are used. (GL#1297)

- When RiskScape tried to read a GeoPackage input file from a read-only file system
  and the GeoPackage had `wal-mode` enabled, RiskScape would exit with an unexpected error:
  ```
  Unable to obtain connection: [SQLITE_CANTOPEN] Unable to open the database file
  ```
  This problem has now been resolved. (GL#1291)

- Update Mac installation instructions in the documentation (GL#1339)

### Breaking

- The `if_then_else` function now returns the else value when the 'if' condition is null. Previously a
  null value would have been returned. (GL#1353)

### Non-functional changes

Some changes were made to the underlying codebase that should not produce any noticeable
difference in functionality or engine behaviour. This includes:

- Some refactors to the CLI wizard to make the code more reusable. (GL1254)

## v1.6.0

*Built 8 April 2024*

### Breaking

- The behaviour of the bookmark parameter property has changed slightly.
  This will only affect expert users who have added `param.NAME.properties = bookmark`
  to their RiskScape model manually. (GL973)

### Enhancements

- The [GeoTools](https://www.geotools.org/) library used by RiskScape for geometry processing
  has been updated from v28.2 to v30.2. (GL1245)

- Upgraded many of the libraries that RiskScape uses. (GL1243)

- RiskScape supports loading bookmarks from
  [data URIs](https://datatracker.ietf.org/doc/html/rfc2397).  This is primarily an
  integration feature for developers to allow smaller datasets to be passed directly
  to RiskScape without having to store them. (GL1092)

- RiskScape now accepts `bookmark()` expressions for model parameters that expect bookmarks.
  A `bookmark-template` parameter property has been added that means expert users can
  now apply template bookmark settings to a model parameter. For example, this lets you swap out a
  CSV file location whilst still applying bookmark settings to the input data. (GL973)

- Model parameters can now be given a `bookmark` and a `type` property to support the platform
  user interface.  Modellers can state that a parameter is a bookmark and what type it wants
  to allow the platform to show a tailored user interface for this parameter.  Note that only
  identified types are supported currently, i.e. one that is listed when you run `riskscape type list`
  (GL1180)

- The beta `losses_by_period` function now defaults to percentile based interpolation. This
  can be changed to `ranked_closest` using the mode option.

- User supplied functions written in Java can now be loaded remotely over secure HTTP. (GL1184)

- A `create_continuous_2d` function has been added. `apply_continuous` and `stack_continuous` have
  been updated to work with both one and two dimensional functions. For now `create_continuous_2d` is
  only available when the beta plugin is enabled. (GL1182)

- List expressions can now accept a mix of integers and floating values.
  For example, previously the expression `[1, 3.14]` would result in a list of type `Anything`,
  whereas now it will result in a list of floating values. (GL445)

- Bookmark identifiers and locations may now be quoted when passed to RiskScape on the command line
  and in model definitions. (GL1216)

- On Linux, RiskScape now attempts to use `/usr/bin/python3` to run CPython functions if the
  Python binary location is not set. (GL1222)

- The HDF5 bookmark resolver now supports remote files. (GL1238)

- A new `riskscape_prompt.bat` is now distributed with the RiskScape software to make
  the shortcut setup simpler for Windows users. (GL1129)

- Features moved from the :ref:`beta-plugin` into defaults to make them available to all users (GL1252):
  - The `aal_trapz` function, which simplifies the process of calculating the
    Average Annual Loss (AAL) for :ref:`hazard_based_probabilistic`.

  - `to_list` can now be used on relations.

  - Automatic generation of the `project.qgs` file. As before, this can be disabled
    with the `output.qgis-project` configuration option if required.

- RiskScape can now load CSV files encoded in UTF-16 and UTF-32 with a Byte Order Marker. (GL1247)

- The sort step now has an optional extra parameter `delta-type`. If specified,
  the sort step will pass the delta from the previous step into the lambda
  expression (as `prev.delta`). This provides basic support for cascading hazards
  (but will not scale well to large datasets). (GL1241)

### Fixes

- The `bookmark` function is now more tolerant when given non-string types (GL1253).

- Can now "splat" a struct that is nullable, e.g. `select({foo.*})` where foo can be null
  (GL989).

- Improved the error handling of bookmarks when they attempt to access data over the
  internet that is missing. (GL1215)

- The `apply_continuous` function now ensures that the x and y values cannot be null.
  Previously if these values were null then an unexpected error was encountered. (GL1234)

- The error handling of missing remote pipeline and Jython function files has been improved.
  Previously this would have caused an unexpected exception. (`riskscape-platform#121`)

- RiskScape now correctly loads raster data (e.g. GeoTIFF) as an exposure layer in
  a bookmark expression (i.e. `input(bookmark('foo.tif'))`) (GL1251)

### Non-functional changes

Some changes were made to the underlying codebase that should not produce any noticeable
difference in functionality or engine behaviour. This includes:

- Some refactors to the main object used to store pipeline data (`Tuple.java`). (GL1240)

## v1.5.0

*Built 6 July 2023*

### Breaking

- RiskScape now requires a Java Runtime Environment of at least version 17.
  If RiskScape fails to run with an error message like:

  `Exception in thread "main" java.lang.UnsupportedClassVersionError: nz/org/riskscape/engine/cli/Main has been compiled by a more recent version of the Java Runtime`

  Then you will need to upgrade your Java Runtime.
  Refer to the [Java documentation](https://www.oracle.com/java/technologies/downloads/)
  for more details on how to upgrade Java. (GL634)

### Enhancements

- The `union` step can now retain the CRS information associated with the input geometry,
  providing that all the inputs to the `union` step are in the same CRS. This means that
  more geometry-based operations, such as `to_coverage()` can now be performed on the
  output of the `union` step. (GL824)

- A new `aal_trapz()` aggregation function has been added to the :ref:`beta-plugin`.
  This new function simplifies the process of calculating the Average Annual Loss (AAL)
  for :ref:`hazard_based_probabilistic`. (GL993)

- Added a new file output: QGIS project file.
  Model runs now produce a QGIS project file named `project.qgs` which can be opened in
  QGIS and contains all output data. This feature currently supports file system output
  files as GeoJSON, KML, shapefile, and GeoPackage. The new `project.qgs` will automatically
  be generated whenever the :ref:`beta-plugin` is enabled, although it can be disabled with the
  `output.qgis-project` configuration option if required.

### Fixes

- Various improvements to error handling, including:
  - An untidy `ResultOrProblems.logIfWarningsIgnored()` stack-trace was sometimes displayed
    when saving results, particularly for GeoPackage format files.
  - An `UnsupportedOperationException` when saving SQL Date attributes to a GeoPackage file has now been fixed.
  - More contextual information is now displayed if an error occurs writing to a GeoJSON file.
  - Previously, RiskScape could exit unexpectedly if a `group` pipeline step tried to
    use non-unique attribute names. Now, the error message is correctly displayed.
  - Occasionally, the user-friendly error message would not be displayed when an exception
    occurred, and instead a message like `Problem(ERROR, STOPPED)` would be shown.
    The full error message is now correctly displayed. (GL964)

- An issue was resolved when models use raster data as relational input data for a model, for example, using a
  population grid as the exposure-layer. Some raster files may have a large extent and a
  small grid resolution (i.e. a large number of total pixels), even if the size of the raster file
  itself is not particularly large. If the total number of pixels was over 2.1 billion, RiskScape
  would incorrectly return no relational input data for the file. Now, the data will be read, however,
  the performance will still be very slow due to the sheer number of pixels. (GL964)

- Previously, RiskScape would write an empty shapefile in a manner that would cause
  an exception when RiskScape tried to read the shapefile again later. This has been resolved
  so that RiskScape writes empty shapefiles in a format that it can successfully read again. (GL990)

## v1.4.0

*Built 14 April 2023*

### Breaking

- Support for the legacy `template = table` models has now been dropped.
  Table models were useful if the exposure and hazard data had already been
  combined into a *single* data source. However, you can achieve the same
  functionality through a pipeline model - follow the :ref:`table_pipelines` guide
  for a worked example of how to do this. (GL813)

- The next RiskScape release (1.5.0) will require Java 17 to run. Users are advised to upgrade
  their Java environment now, if necessary, to ensure the next RiskScape version will run.
  A warning has been added to RiskScape to warn users if they are using a Java version that will
  not run RiskScape 1.5.0. (GL903)

- The `if_null(test, else)` function will now report an error if the `else` argument is a nullable type.
  The previous behaviour when the `else` argument was null was that the value returned would *always*
  be `null` (i.e. `Nothing` or `None`), even if the `test` value was non-null. (GL940)

- The `bucket_range()` function now names the lowest possible range as `range_<_0`, whereas previously it would have
  been `range_-_0`.  Some models may now produce slightly different `bucket_range` outputs,
  as the ranges will now automatically include the highest (i.e. `range_X_+`) and
  lowest (i.e. `range_<_Y`) values possible, by default. (GL942)

- Structs could be defined without a comma separating the struct attributes. E.g
  `{ 'foo' 'bar' }` and `{ 'foo' as foo 'bar' as bar }` should have a comma separating
  the `foo` and `bar` attributes but RiskScape was treating these expressions as valid.

  Now RiskScape will treat these expressions as invalid and produce an error that the
  comma is missing. (GL925)

### Enhancements

- A new, more efficient implementation is now used when cutting geometry into smaller pieces.
  Models that involve a lot of cutting operations, including the `segment()`, `segment_by_grid()`,
  and `combine_coverages()` functions, should now run faster.
  Existing models may emit minutely different geometry using this new method, but only at the
  smallest of resolutions (coordinates seem consistent within 1 x 10^<superscript>-15</superscript> of
  precision when cutting WGS-84 polygons by a 1000² metric grid). (GL433)

- The [GeoTools](https://www.geotools.org/) and [JTS](https://www.geotools.org/) libraries used
  by RiskScape for geometry processing have been updated. GeoTools has been updated from v26.0
  to v28.2, and JTS has been updated from v1.18.2 to v1.19.0.

  The GeoTools update includes security fixes for
  [OGC Filter SQL Injection Vulnerabilities](https://github.com/geotools/geotools/security/advisories/GHSA-99c3-qc2q-p94m)
  which could have affected `postgis` bookmarks in RiskScape. Refer to this
  [community post](https://community.riskscape.org.nz/t/geotools-vulnerability-may-affect-riskscape-users-who-use-postgis-bookmarks/118) for more detail about how it affects RiskScape. (GL937)

- When sharing models, you can now constrain what values users can enter for pipeline model
  parameters. For example, you can require that the user always enters a numeric value within
  a certain range. For more details, refer to :ref:`parameter_properties`. (GL441)

- The :ref:`beta-plugin` is no longer required in order to use the `union` pipeline step.
  For more on the `union` step, refer to :ref:`union_step`.

- New string utility functions `ends_with` and `str_find` have been added.
  These can be useful for filtering bookmark input data. (GL941)

- The `bucket_range()` function will now automatically add the lower-most and upper-most bounds
  to the given `range` argument automatically. For example, if the given `range` argument were `[ 1, 1000, 100000 ]`,
  it will now behave as if `[ minint(), 1, 1000, 100000, maxint() ]` had been specified.
  This ensures that any values that might fall outside the specified range will *not* be ignored by default.
  You can avoid this new behaviour and revert to the old behaviour by specifying a new `options` argument,
  e.g. `bucket_range(pick: loss, select: { count(*) }, range: [ 1, 1000, 100000 ], options: { add_bounds: false })`.
  (GL942)

- A new `tile-cache-percent` configuration setting has been added to the `[global]` section of the :ref:`settings_ini`.
  This lets you configure the size of the tile cache used when sampling raster data, such as a GeoTIFF,
  as a percentage of the total Java memory.
  Using a larger cache can improve the performance of `riskscape model run`, as it means the raster pixels
  need to be swapped in and out of memory less frequently. The default tile cache size is now 5%
  of the total Java memory. (GL949)

### Fixes

- When a raster bookmark used `map-value` or the `map_coverage()` function was used in a pipeline, sampling the coverage
  could sometimes produce inconsistent results. This issue has now been resolved. If you have used `map-value`
  in a bookmark or `map_coverage()` in a pipeline, we recommend upgrading RiskScape and re-running your model(s). (GL958)

- Previously, running RiskScape on certain systems may have resulted in an exception
  `RuntimeException: Waiting thread received a null tile.` This issue should now be resolved. (GL830)

- The `riskscape model list` command now displays any warnings that a model might have.
  Previously warnings would only be displayed when the model was actually run. (GL910)

- The CLI wizard now lets you select *any* attribute for the `group-by` parameter.
  This includes geometry and struct attributes. For example, you can now aggregate by
  the entire area-layer struct, which will report the results broken down by region
  as a geospatial output (previously, only aggregated CSV outputs were supported
  in the CLI wizard). (GL884)

- The :ref:`hazard_based_probabilistic` documentation has been updated to exclude
  duplicated zero loss values from the AAL calculation. When the `continuous` fitter
  is used in the `fit_curve` function, it will now result in a model run error if the input
  data contains duplicate values. (GL911)

- When RiskScape wrote GeoJSON output files it would save the first geometry as the GeoJSON
  geometry, as well as a text version of the same geometry. This would increase the size of the file
  needlessly. The text version of the geometry is no longer included. (GL948)

- RiskScape would throw a null pointer exception for expressions that did an equality
  operation with a integer to float conversion on a null input. E.g `null_of('floating') = 4`
  would fail. This has been resolved now. (GL943)

## v1.3.0

*Built 7 December 2022*

### Breaking

- If you have manually specified a `crs-name` for a ESRI Grid (`.asc` file) bookmark,
  then the default `crs-longitude-first` setting has changed. Previously, the default was
  `false`, which meant the axis order defined in the EPSG CRS was used (typically `lat,long`).
  The default value is now `true`, which means RiskScape will read the data as `long,lat` coordinates.
  This *only* applies when a `crs-name` is set explicitly for a `.asc` bookmark.
  In general, you should not need to set the `crs-name` for a `.asc` bookmark unless the `.prj`
  file is missing or unreadable. (GL807)

- The `--parameters` option has been removed from the `riskscape pipeline evaluate` command.
  Previously this allowed you to override low-level parameters for a given pipeline step.
  If this change affects you, we now recommend that you use :ref:`parameterized_pipelines` instead.
  The `riskscape pipeline evaluate` and `riskscape wizard` commands now both save a `pipeline.txt`
  file in the output directory. (GL821)

### Enhancements

- The `to_coverage()` function now accepts raster data as an argument-type.
  This means that advanced users can now write a single pipeline that samples geospatial data from *either*
  a vector source (i.e. shapefile) or a raster source (i.e. a GeoTIFF). Previously, this would require
  writing two separate pipelines, one for each source. (GL809).

- The performance of 'all-intersections' and 'closest' spatial sampling has been improved when dealing with
  large or complex polygons in the exposure-layer and a GeoTIFF or `.asc` hazard-layer.
  Previously, the entire polygon would be cut by the grid resolution of the hazard-layer, which could be very
  computationally expensive. Now, cutting will be skipped for parts of the polygon that overlap a 'no-data'
  cell in the hazard-layer. (GL820)

- A new `union` step has been added to the beta plugin for advanced pipeline models.
  This means you can now combine multiple different exposure-layers (e.g. water pipes *and* pump stations) into a single model.
  For more details, see :ref:`union_step`. (GL436)

- When the data types in your exposure-layer do not match the types that your Python function expects,
  the error message will now provide more detail on exactly which attributes are missing or are the wrong
  data type. (GL823)

- Support has been added to the beta plugin to dynamically load relational data from multiple sources (GL837).
  This allows a pipeline to load CSV input that contains the locations of other CSV files to read.
  Advanced users can now do this in a pipeline using the following approach:
  ```
  input('list-of-csv-files.csv', name: 'csv')
  -> select({*, to_list(bookmark('template-bookmark,
                                 {location: csv.filename},
                                 type: 'relation(struct(foo: floating))
                        ) as csv_data)
  -> unnest(csv_data)
  ```

- A new `losses_by_period()` function has been added to the beta plugin. This function can be used in
  a probabilistic pipeline to calculate loss metrics such as Probable Maximum Loss, Occurrence Exceedance
  Probability, and Aggregated Exceedance Probability. (GL864)

- Support for reading and writing GeoPackage files is now enabled by default.
  Previously it required that the beta plugin was enabled. (GL853)

### Fixes

- Previously, using the `exposed_ratio` attribute or the `scale-losses` option in wizard models
  could occasionally calculate the incorrect ratio when sampling a polygon exposure-layer against
  a GeoTIFF or `.asc` hazard file. This meant that scaling the loss could calculate the incorrect result.
  If the input geometry aligned exactly with the hazard grid, the `sample()` operation could potentially
  return a *line-string* rather than a polygon geometry match. The `exposed_ratio` calculation would then
  `measure()` the sampled geometry, which returns the *length* of the line-string, rather than the area,
  as expected. This issue has now been fixed. Note that a work-around on previous versions of RiskScape
  would be to take the pipeline code and change the `measure()` function calls to `measure_area()` instead.
  This problem could also manifest itself as a `ClassCastException` when the `analysis.save-raw-results`\
  parameter was set for the wizard model. The `measure()` function will also now return an error if
  given a collection of different geometry types (i.e. polygons and line-strings). (GL817)

- Previously the `crs-name` was silently ignored for GeoTIFF bookmarks. Now the setting will take effect.
  Note that you should not normally need to specify `crs-name` explicitly for a GeoTIFF bookmark. (GL807)

- Previously, if warnings were present for a model parameter, these warnings would either be silently
  ignored by the `riskscape model run` command, or a stack-trace would be logged along with the message
  `Non-error problems are present and have not been consumed...`. Now, RiskScape will display any warnings
  associated with the model when it is run. (GL763)

- Performance and memory usage have been improved for aggregation pipeline steps. In particular, you
  may notice an improvement if your model pipeline contains many `group` steps (e.g. probabilistic models),
  or the model aggregation produces a large number of rows (e.g. aggregating by the original exposure). (GL808)

- Previously, RiskScape could sometimes exit unexpectedly whilst running a model.
  When this problem occurred, the message 'Deadlock detected!' would be logged.
  This issue has now been fixed. (GL816)

- Support has been improved when a file is specified for the `--progress-indicator` option for the
  `riskscape model run` command. Previously, if an intermittent file error occurred, for example the progress file
  was temporarily locked, then RiskScape would stop writing statistics completely for the duration of the model run.
  Now, RiskScape will continue to retry writing the progress statistics to file, even if file errors are encountered. (GL806)

- Some :ref:`arcgis_workarounds` users having shapefile projection issues have now been documented. (GL855)

## v1.2.0

*Built 29 July 2022*

### Breaking

- Duplicate bookmark settings now show an error, rather than being silently ignored.  Previously,
  the last defined setting was used, and any earlier values were ignored.  This change brings
  bookmarks in to line line with other project configuration, such as model definitions where
  duplicates are not expected.  You can fix this error by simply removing any surplus settings
  and leaving only the one your project needs (usually this is the last one).  (GL769)

- Previously, when RiskScape read an HDF5 dataset that contained a simple type (e.g. integer or
  floating values), it produced a relation containing a single attributed called `value`.
  Now, the name of the attribute will match the name of the dataset instead. For example, reading
  the `/gmf_data/gmv_0` dataset will now produce a relation containing a `gmv_0` attribute,
  instead of a `value` attribute. (GL718)

- The `--no-progress-indicator` CLI option has been removed for the `riskscape model run`,
  `riskscape model batch`, and `riskscape pipeline eval` commands. Instead, this has been replaced
  with a new `--progress-indicator=none` option.

### Enhancements

- Loading plugins in the settings file has been simplified.
  E.g you can now load the beta plugin with a `settings.ini` containing:
  `load-plugin = beta`. (GL757)

- The RiskScape expression language now supports floating point numbers in
  scientific notation, e.g `5.27e-10`. (GL754)

- RiskScape now supports GeoPackage as an output (e.g `--output results.gpkg`). This will save
  all result layers to the same output GeoPackage file, rather than separate files when using the
  `--format geopackage` option. (GL758)

- Manifest information is now saved when writing output to GeoPackage and PostGIS outputs. (GL759)

- RiskScape has been updated to support the v1.12.2 [HDF5 library](https://www.hdfgroup.org/solutions/hdf5/).
  Linux users may need to change their `settings.ini`'s `[hdf5] lib_path` to point to `/opt/hdf5-1.12.2/lib/libhdf5_java.so` (GL760)

- HDF5 bookmarks can now read multiple datasets at once and combine them into a single relation.
  This is only supported when the datasets all contain the exact same number of elements. (GL718)

- Coverage bookmarks now accept a `map-value` parameter that lets you manipulate the *value* that
  gets returned when the coverage is sampled. For example, you could convert a hazard intensity measure from centimetres to metres
  by using `map-value = value / 100`. A new `map_coverage()` function has also been added, which
  lets you do the same thing directly from pipeline code. (GL782)

- The `round()` function now lets you specify the number of decimal places to round to. (GL452)

- New `norm_ppf` and `lognorm_ppf` functions have been added to return the Percent Point Function
  (inverse Cumulative Distribution Function) from a normal or log-normal curve. (GL744)

- For USGS ShakeMaps, you can now specify a specific dataset to read. Previously RiskScape would
  only read the PGA data. Refer to :ref:`shakemaps` for more details. (GL744)

- Running a model with unknown parameters will now produce a warning. Unknown parameters are likely
  to be caused by mistyping the parameter name. (GL768)

- The progress statistics, displayed on the terminal when a model runs, have been improved.
  They now include a percentage complete metric for some pipeline steps. For long-running models,
  the percentage complete for the exposure-layer (e.g. `exposures_input.complete`) can often be
  used as a guide to the overall progress of the model. Refer to :ref:`progress_monitor` for more details. (GL792)

- Commands that support progress statistics now support a `--progress-indicator` CLI option.
  Instead of displaying the progress statistics on the CLI, you can specify a file that progress
  updates will be written to, e.g. `--progress-indicator=output/progress.txt`. This may be useful
  when running RiskScape as part of a script. (GL792)

- More improvements have been made to the RiskScape documentation. Of note are:
  - :ref:`bookmark_examples` for many different bookmark data sources.
  - Some :ref:`performance tips<slow_model_tips>` for models that run slow.

### Fixes

- When RiskScape accessed files over HTTP, some websites could mistake RiskScape for a bot and reject
  the HTTP request with a 403 error. RiskScape now sets the 'User-Agent' in HTTP requests to avoid
  this problem (GL751).

- Various issues with using WFS input data have now been resolved, including:
  - Connection errors now show the cause of the connection error. Previously these
    were obscured by an unhelpful `Failed to create data store` error. (GL461)
  - Previously RiskScape was only loading the first 100,000 features/records for a WFS layer
    and was ignoring any subsequent data. (GL753)
  - If the WFS server had a configuration setting (e.g. `maxRecordCount`) that limited how many features
    are returned in each WFS request, then RiskScape would only load that many features.
    The number of features per WFS request can now also be lowered using the `wfs-page-size` bookmark parameter. (GL761)
  - Trying to use a WFS layer that contained a large number of records (e.g. 500,000) could
    result in RiskScape exiting with an error, generally due to a connection timeout. (GL573)
  - Depending on the WFS server, RiskScape could exit with an exception trying to read the WFS data. (GL573)
    When the `--show-stack-trace` CLI option was used, the following error would be displayed in this case:

    ```
    java.lang.ClassCastException: java.util.HashMap cannot be cast to org.opengis.feature.simple.SimpleFeature
    ```
  - Some examples on how to use WFS input data have now been added to the RiskScape documentation.
    Refer to the new :ref:`wfs` page.

- Previously RiskScape could exit with an exception when the `--show-project-errors` CLI option was used,
  and the project contained a bookmark with an error. This has now been fixed. (GL764)

- When input data was read in GeoJSON format, floating-point values could sometimes be truncated and turned
  into whole numbers. For example, the value `20.5` would be turned into `20.0`. This would *only* occur if the
  same attribute in the input data contained a mixture of integer (i.e. whole number) and floating-point values.
  When this mixture of integer and floating-point values were present in the input data, RiskScape could
  also exit with an exception:

  ```none
  IllegalStateException: Found conflicting types Double and Long for property XYZ
  ```

  These issues have now been fixed. (GL788)

- For advanced users that write their own pipelines, RiskScape could sometimes complain about a
  pipeline cycle error when there was no cycle. The pipeline cycle checking has now been improved. (GL772)

- Various minor improvements have been made to error handling, for example how errors
  and exceptions are displayed, particularly when the `--show-stacktrace` CLI option is used. (GL803)

## v1.1.0

*Built 5 May 2022*

### Enhancements

- An optional :ref:`beta-plugin` has been added to make new/experimental features available.
  The beta plugin is not enabled by default, and any beta plugin features may still change
  significantly in future releases of RiskScape. (GL716)

- Preliminary support has been added to the beta plugin for:
  - Saving results to a :ref:`PostGIS <postgis>` database. (GL705)
  - Reading and writing vector data in the GeoPackage (`.gpkg`) file format (GL706, GL728)

- New functions have been added to the beta plugin, including:
  - `list_to_tuple`, which makes it easier to save the contents of a list to CSV or shapefile.
  Each item in the list will be written as a separate column or attribute.
  (GL719)
  - `switch`, which is useful when you have a range of possible input values that you
  want to map to a smaller subset of values. E.g. you want to map any of the value codes
  `0, 12, 44, 22` to simply `'Timber'`.

- Using raster data as your exposure-layer now results in polygon geometries that cover the
  extent of each raster point. Previously the geometry produced was a point rather than a polygon. (GL620)

- Type expressions can now use hyphens in struct attribute names as well as quoted
  identifiers, e.g `struct(value_0-100: integer, "Total Cost": floating)` (GL686)

- More tutorials have been added to the RiskScape documentation, including an
  'Intermediate Concepts' section that covers:
  - How the various RiskScape spatial sampling and consequence analysis techniques work. (GL649)
  - Improved CPython setup instructions and an example damage state function.
  - How the `riskscape model batch` lets you run a model repeatedly by varying the hazard-layer. (GL168)
  - How the various RiskScape geoprocessing features operate. (GL722)

- A new `riskscape bookmark evaluate` command has been added. This lets you visualize the input data,
  typically in shapefile form, *after* it has been manipulated by any RiskScape bookmark settings.
  This is useful to check the CRS axis-order is correct when you need to construct geometry manually,
  such as from CSV input data. (GL608)

### Fixes

- Table rendering on Windows has been improved for `info` commands. Previously, commands
  such as `riskscape format info shapefile`, could result in poor rendering if the content
  being displayed contained line-breaks. (GL703)

- The default `validate-geometry` setting is `error`, which means RiskScape detects and
  automatically fixes any invalid geometry in the input data. However, the default
  `validate-geometry` project setting was not being applied correctly when geometry
  was reprojected. This meant that geometry could become invalid as a result of reprojection
  and RiskScape would not detect or fix it (unless `validate-geometry = error` was explicitly
  set in the `project.ini` file). This issue has now been fixed. (GL724)

- When setting up CPython support in RiskScape, you no longer need double backslashes in the
  `python3-bin` file path in `settings.ini`. For example, you can now use `C:\Users\Ronnie\Anaconda3\python.exe`
  instead of `C:\\Users\\Ronnie\\Anaconda3\\python.exe`. Existing `settings.ini` files that contain
  double backslashes for the Python path should still continue to work fine. Previously, exceptions or
  other errors could occur when single backslashes were used, but this has now been fixed. (GL-735)

- Setting the `output-base-location` in the project would result in an extra `output` directory
  appearing in the output path. This has been fixed now. (GL743)

- Running RiskScape without a `project.ini` or home directory would result in unusual errors if a
  relative file was accessed, e.g `riskscape bookmark info FILE.csv`. This has now been fixed. (GL710)

- Spatial sampling has been improved for the `sample_one()` function, which is used by `CLOSEST`
  sampling in the wizard. For complex line and polygon geometry, such as shapes that look like a 'C' or multi-polygons,
  the geometric centroid of the shape can fall outside the shape itself. Previously, RiskScape would
  always attempt to use the geometric centroid first when sampling the hazard value, even if this centroid
  fell outside the shape itself. Now `sample_one` will always ensure that the sampled value falls inside the input geometry. (GL737)

- When CPython functions were enabled on your system, sometimes the `riskscape model run` command
  could exit with a `java.lang.NullPointerException` error. This only occurred if the shapefile
  input data for the exposure-layer contained `NULL` values. This issue has now been fixed. (GL-739)

- When all-intersections sampling is used, the wizard will now produce a single value for the `exposed_ratio` attribute.
  Previously `exposed_ratio` was a `List` type, which made it harder to use. (GL741)

- Model run `--parameter` values may now contain white space around the equals sign. E.g
  `--parameter "input-exposures.layer = Roads_SE_Upolu"`. (GL741)

## v1.0.0

*Built 4 February 2022*

Version 1.0.0 is the first official public release of RiskScape.

If you have been using older, evaluation versions of RiskScape, you can view a record of the
changes in functionality between releases :ref:`here <old_changelog>`.
