Remote and Cloud Stores

Coming from xarray or fsspec

In Python you would hand that catalog entry to xarray and let fsspec sort out the connection:

import xarray as xr

ds = xr.open_dataset(
    "s3://mdmf/gdp/gridMET.zarr/",
    engine="zarr",
    backend_kwargs={"storage_options": {
        "anon": True,
        "client_kwargs": {"endpoint_url": "https://usgs.osn.mghpcc.org/"},
    }},
)

However, pizzarr has no storage_options argument. Connection settings come from environment variables that the Rust object_store client reads when it first opens a store. The reason is mechanical rather than principled: store handles are cached by URL on the Rust side and shared across calls, so there is no per-call place to pass options. The translation is:

Set them with Sys.setenv() for the session, or in .Renviron to persist them. Because the handle is cached, changing a variable after a store is open does nothing until you call zarrs_close_store() on that URL.

Two routes to the same data

An S3 bucket read anonymously is, underneath, an HTTPS server that returns objects at predictable paths. pizzarr can take either view of it, and which one you want depends less on the data than on which build you installed (CRAN or r-universe).

The HTTPS route addresses objects directly. That is, an s3://bucket/key address served by endpoint https://host is reachable at https://host/bucket/key, so gridMET becomes:

https://usgs.osn.mghpcc.org/mdmf/gdp/gridMET.zarr

This https route works on both distribution tiers — the pure-R CRAN build and the r-universe build with the zarrs backend. It needs no credentials and no configuration, and it is the only route the CRAN build has, so it is the one to reach for unless something stops you.

The S3 route uses the s3:// URL and the real S3 API by way of object_store. It requires the r-universe build with the s3 feature compiled in, and it is what you need for buckets that are not anonymously readable over plain HTTPS, or that require signed requests. Check what you have:

pizzarr_compiled_features()
#> zarrs backend not available (pure R install).
#> See ?pizzarr_upgrade for the r-universe install.
#> character(0)

Reading over HTTPS

HttpStore takes the URL, and zarr_open() gives back a group:

url <- "https://usgs.osn.mghpcc.org/mdmf/gdp/gridMET.zarr"

store <- HttpStore$new(url)
g <- zarr_open(store)
g
#> <ZarrGroup> /
#>   Store type  : HttpStore
#>   Zarr format : 2
#>   Read-only   : TRUE
#>   No. members : 12

HTTP servers generally cannot be listed the way a directory can, so a remote store is only browsable if it publishes consolidated metadata — a .zmetadata key holding every array’s metadata in one document. gridMET does, and listdir() reports its contents:

store$listdir()
#>  [1] "crs"                                      
#>  [2] "lat"                                      
#>  [3] "lon"                                      
#>  [4] "max_air_temperature"                      
#>  [5] "max_relative_humidity"                    
#>  [6] "min_air_temperature"                      
#>  [7] "min_relative_humidity"                    
#>  [8] "precipitation_amount"                     
#>  [9] "specific_humidity"                        
#> [10] "surface_downwelling_shortwave_flux_in_air"
#> [11] "time"                                     
#> [12] "wind_speed"

Without consolidated metadata you have to know the array names in advance. This is why so many published Zarr stores are consolidated, and why the flag appears in the gridMET catalog entry alongside the connection settings.

gridMET’s arrays are Blosc-compressed, which the pure-R build handles only if the blosc package is installed. The chunks below need it:

lat <- g$get_item("lat")
lat
#> <ZarrArray> /lat
#>   Shape       : (585)
#>   Chunks      : (585)
#>   Data type   : <f8
#>   Fill value  : NaN
#>   Order       : C
#>   Read-only   : TRUE
#>   Compressor  : BloscCodec
#>   Store type  : HttpStore
#>   Zarr format : 2

lat$get_item(list(slice(1, 5)))$data
#> [1] 49.40000 49.35833 49.31667 49.27500 49.23333

Reaching the data variables is the same call. Their shape is worth a look before reading anything:

tmmx <- g$get_item("max_air_temperature")
tmmx
#> <ZarrArray> /max_air_temperature
#>   Shape       : (17369, 585, 1386)
#>   Chunks      : (2190, 150, 150)
#>   Data type   : <i2
#>   Fill value  : 
#>   Order       : C
#>   Read-only   : TRUE
#>   Compressor  : BloscCodec
#>   Store type  : HttpStore
#>   Zarr format : 2

That is 17369 days by 585 rows by 1386 columns, in chunks of 2190 by 150 by 150. A chunk is the smallest unit the store will return, so asking for a single value still transfers the roughly 98 MB block that contains it — compressed in flight, but decompressed in full. Slicing a remote array cheaply means slicing along chunk boundaries and keeping the request small in the dimensions that are chunked finely.

Reading over the S3 API

The s3:// route needs the endpoint, and nothing else for a public bucket:

Sys.setenv(AWS_ENDPOINT = "https://usgs.osn.mghpcc.org")

s3_url <- "s3://mdmf/gdp/gridMET.zarr"

zarrs_open_array_metadata(s3_url, "lat")

Reads go through zarrs_get_subset(), which takes zero-based, stop-exclusive ranges — one per dimension — and returns the data along with its shape:

zarrs_get_subset(s3_url, "lat", list(c(0L, 5L)), NULL)

Blosc decompression happens in Rust here, so this path does not need the blosc R package. When you are done with a store, drop its cached handle:

zarrs_close_store(s3_url)

S3Store and GcsStore exist to mark a URL as cloud-backed so that dispatch picks the right backend. They are not full stores — they carry a URL and nothing else, so calling get_item() or listdir() on one raises an error naming zarrs_get_subset(), HttpStore, and this vignette. The functions above are the working entry point for s3:// data.

Alternate endpoints

The endpoint variable is the whole trick for MinIO, Ceph, Wasabi, Cloudflare R2, and Open Storage Network pods. A self-hosted MinIO over plain HTTP with path-style addressing needs three settings:

Sys.setenv(
  AWS_ENDPOINT = "http://localhost:9000",
  AWS_ALLOW_HTTP = "true",
  AWS_VIRTUAL_HOSTED_STYLE_REQUEST = "false",
  AWS_ACCESS_KEY_ID = "minioadmin",
  AWS_SECRET_ACCESS_KEY = "minioadmin"
)

zarrs_get_subset("s3://my-bucket/data.zarr", "temperature",
                 list(c(0L, 10L)), NULL)

Most managed services need only AWS_ENDPOINT; region is often unnecessary outside AWS itself. Requests are signed as soon as any credential variable is present and unsigned when none are, which is what makes public buckets work with no setup. AWS_SKIP_SIGNATURE overrides that decision in either direction — set it to "true" to force anonymous access when stale credentials are lying around in your environment, or "false" to insist on signing.

Google Cloud Storage

GCS works the same way, with GOOGLE_ variables in place of AWS_ ones, and it needs the gcs feature compiled in. The one difference that matters: GCS does not infer anonymous access from the absence of credentials the way S3 does. A gs:// read against a world-readable bucket still tries to authenticate — via GOOGLE_APPLICATION_CREDENTIALS, GOOGLE_SERVICE_ACCOUNT, or the GCE metadata server — and fails if none are available. Ask for anonymous access explicitly:

Sys.setenv(GOOGLE_SKIP_SIGNATURE = "true")

gs_url <- "gs://pangeo-data/ECCO_basins.zarr"

zarrs_open_array_metadata(gs_url, "basin_mask")$shape

GOOGLE_BASE_URL overrides the endpoint, which is what you want for a local emulator such as fake-gcs-server.

The same data is also reachable over plain HTTPS at https://storage.googleapis.com/bucket/path, which needs no configuration and works on the CRAN tier as well:

z <- zarr_open(HttpStore$new(
  "https://storage.googleapis.com/pangeo-data/ECCO_basins.zarr"
))

What is not supported

Cloud stores are read-only. Writes to s3:// and gs:// fall back to the R-native path, which has no cloud implementation, so they fail — build locally and upload with another tool. Azure (az://) is not implemented at all. Requester-pays buckets have no way to signal the requester, and there is no per-store credential object: configuration is process-global, so two buckets needing different credentials in one session have to be read in sequence with zarrs_close_store() and an environment change between them.