.. _arcgis:

# ArcGIS Online, ArcGIS Enterprise, and ArcGIS Hub

The simplest way to use ArcGIS data in your RiskScape model is to download the data,
rather than using the HTTP APIs to access the data.
You can then use the downloaded file in your model as per normal.

Manually downloading the data has a few downsides:

- Your local copy might be out of date if the data in ArcGIS changes.
- It makes it harder to share your model with others, as they have to download the exact same set of files manually.

RiskScape can download data directly from any HTTP link.
However, it is not always easy to find the right URL to access ArcGIS data.
The following page describes how to read data directly from sources like ArcGIS Online and ArcGIS Enterprise.

.. tip::
    Generally the formats that work best with RiskScape over HTTP are WFS, GeoJSON, and KML.

.. _argcgis_wfs:

## WFS

WFS is a standard way of accessing geospatial data across the internet,
and is generally the simplest way to download data from an ArcGIS source.
You can read more about WFS :ref:`here <wfs>`.

Here is an example bookmark that accesses an ArcGIS Online WFS server.
This example was found through the https://data.govt.nz website,
and accesses a dataset containing the recreation centres in Canterbury.

```ini
[bookmark Canterbury_rec_centres]
location = https://gis.ecan.govt.nz/arcgis/services/Public/Canterbury_Maps/MapServer/WFSServer
format = wfs
layer = Canterbury_Maps:Recreation_Centre
crs-name = EPSG:2193
crs-longitude-first = true
```

Note that we have to set `crs-longitude-first`, as this particular WFS server returns the coordinates in _long, lat_ order.

The WFS address for an ArcGIS Online server will typically look something like:

```none
https://WEBSITE/arcgis/services/DATASET_CONTAINER/MapServer/WFSServer?request=GetCapabilities&service=WFS
```

Note that it typically has `WFSServer` after `MapServer` in the URL.
The `?request=GetCapabilities&service=WFS` bit is optional in your RiskScape bookmark, but can be useful for testing in your web browser.

ArcGIS Online usually also provides a [REST API](https://en.wikipedia.org/wiki/Representational_state_transfer)
that can help you navigate the datasets in your web browser.
For our example bookmark, you can view the ArcGIS Online REST API
[here](https://gis.ecan.govt.nz/arcgis/rest/services/Public/Canterbury_Maps/MapServer).

When you view the REST API, at the top of the page there will be links for (optionally)
JSON, SOAP, WFS, and WMS. Click on the WFS link - that is the URL you should use in your RiskScape bookmark.

.. tip::
    Typically, the WFS link will only appear on REST API pages that end with ``MapServer``,
    i.e. if you click on the individual datasets, you will no longer see the WFS link at the top of the page.

## Accessing private data

The previous WFS example accessed data that was publicly available.
If the data you want to use is *not* public, then you will need to use an
[access token](https://developers.arcgis.com/documentation/mapping-apis-and-services/security/#arcgis-identity)
in your URL.

Typically, this is just a matter of including `token=YOUR_TOKEN` in your bookmark URL, e.g.

```none
https://WEBSITE/arcgis/services/DATASET_CONTAINER/MapServer/WFSServer?request=GetCapabilities&service=WFS&token=YOUR_TOKEN
```

The hard part is obtaining a suitable ArcGIS access token to use (i.e. replacing `YOUR_TOKEN` with a valid token).
The simplest way is to use [generate-token](https://developers.arcgis.com/rest/users-groups-and-items/generate-token.htm)
in the ArcGIS REST API.

For example, the following URL should generate an access token valid for one week,
if you replace `WEBSITE`, `USERNAME`, and `PASSWORD` with your actual details.

```none
https://WEBSITE/server/rest/generateToken?username=USERNAME&password=PASSWORD&client=requestip&ip=&referer=&expiration=10080&f=json
```

.. note::
    Be careful to keep your password details secure here.
    For example, delete the URL from your browser history once you are done.

### Access token limitations

Access tokens will last for up to two weeks.
So you will regularly need to generate a new token, and update your bookmarks.

In order to use the `generateToken` REST API, you *must* use a
[built-in account](https://enterprise.arcgis.com/en/portal/latest/administer/windows/managing-access-to-your-portal.htm)
in ArcGIS. Otherwise you will likely get an error code response, e.g. 405 (not allowed).

The `generateToken` API is not supported if your organization uses
[OAuth](https://developers.arcgis.com/documentation/mapping-apis-and-services/security/oauth-2.0/) to authenticate ArcGIS users,
i.e. you login to ArcGIS with the same LDAP username and password you use to access other systems within your organization.

The ArcGIS administrator for your organization may be able to create a new built-in account for you to use.
However, this may incur additional licensing costs depending on your ArcGIS setup, as it will require an extra 'Viewer' license.

If your organization uses OAuth and you cannot easily create new built-in accounts,
then you may still be able to retrieve an access token via OAuth.
For example, you may be able to use the 'Web Developer' tools in your browser to view your token
when you login to ArcGIS normally.

Try contacting your ArcGIS support person for more details on obtaining an access token.

## ArcGIS Hub

Anyone can access data that is available in [ArcGIS Hub](https://hub.arcgis.com).
If your organization uses ArcGIS Online, then it can be
[configured](https://doc.arcgis.com/en/arcgis-online/administer/configure-open-data.htm)
to make public data available in ArcGIS Hub.

The URL to load ArcGIS Hub data into RiskScape will look something like this:

```none
https://opendata.arcgis.com/api/v3/datasets/DATASET_ID/downloads/data?format=geojson&spatialRefId=4326
```

However, you need to replace `DATASET_ID` with the actual ID of the data you are interested in.
The hard part is finding out the correct dataset ID to use.
You can follow these steps to find a suitable download link:

1. Go to https://hub.arcgis.com/

2. Search for the data you are interested in, and then click on the correct search result.

3. Click on the 'Download' icon on the left-hand side of the map.
   Then click on either the GeoJSON or KML buttons.

4. Use your web browser's 'downloads' manager to find the URL that was used.
   E.g. in Firefox you can right-click on the downloaded file and select 'Copy Download Link'.

5. You can then use the copied link as the location in a RiskScape bookmark.
   You will also need to specify either `format = geojson` or `format = kml` in your bookmark,
   based on the download button you used.

.. warning::
    It is unclear whether these ArcGIS Hub download links will pull in the latest data if the
    underlying dataset is updated. In other words, updates to the underlying dataset may result
    in a new ``DATASET_ID``, and so your bookmark may continue pointing to an *older* copy of the data.

Here is an example of a bookmark that uses data downloaded from ArcGIS Hub.

```none
[bookmark Canterbury_stormwater_pipes]
description = Over 400,000 pipes managed by Canterbury regional Council. Refer \
    https://hub.arcgis.com/datasets/ecan::canterbury-stormwater-pipelines
location = https://opendata.arcgis.com/api/v3/datasets/268050a3ebee4fc3861b88122e5b6771_3/downloads/data?format=geojson&spatialRefId=4326
format = geojson
```

## REST API

The final option available is to load data into RiskScape directly from the ArcGIS Online REST API.

.. warning::
    The ArcGIS REST API should only be used in RiskScape for small datasets, i.e. 1000-2000 records or fewer.
    ArcGIS feature services have a ``maxRecordCount`` configured, which defaults to 1000 or 2000,
    and *only* that many records will be returned.
    When using the REST API, you should always check that your bookmark returns *all* the expected features.
    You can use ``riskscape bookmark info`` or ``riskscape bookmark evaluate`` to check this.

Typically, an ArcGIS Online REST API address will look something like:

```none
https://WEBSITE/arcgis/rest/services/DATASET_CONTAINER/MapServer/LAYER_ID
```

For example, Greater Wellington Regional Council has a dataset containing segments of the Te Araroa trail.
You can view this dataset in the ArcGIS REST API [here](https://mapping.gw.govt.nz/arcgis/rest/services/Tracks/Regional_Tracks/MapServer).

We can download this data in GeoJSON format by using the REST 'query' API as the bookmark's `location`, e.g.

```ini
[bookmark Te_Araroa]
location = https://mapping1.gw.govt.nz/arcgis/rest/services/Tracks/Regional_Tracks/MapServer/2/query?where=1%3D1&\
    outFields=*&f=geojson
format = geojson
```

All we have done here is add `query?where=1%3D1&outFields=*&f=geojson` onto the end of the ArcGIS REST API
for the layer we are interested in.
Breaking the URL down, it uses the [query](https://developers.arcgis.com/rest/services-reference/enterprise/query-feature-service-layer-.htm)
REST API to return all attributes (`outFields=*`) for all records (`where=1%3D1`, or `1=1`) in GeoJSON format (`f=geojson`).

