Package {publicdataau}

publicdataau: Australian Government Open Data from 'publicdata.au'

Description

Search, filter, total and download the Australian government open data republished at https://publicdata.au/, where every version of a dataset is kept at a URL that never changes. Rows and totals come from the site's query API as typed, labelled 'tibble' data frames, and whole tables from its files in any of twelve formats. Map layers come as 'sf' objects, and rows that carry the code of an area join to the Australian Bureau of Statistics boundaries in one call. The 'DuckDB' file of any version can be attached over the web and queried in place with 'DBI' or 'dplyr'. Downloads can be kept in a cache, since a version never changes, the changes between versions are listed row by row, and the source file, fetch time and checksum of every version are on record. Every answer carries the version it came from and the attribution the publisher's licence requires, and citations come as 'BibTeX'. Any dataset the site serves works by its slug, so a new dataset needs no new release.

Details

Every function takes a dataset's slug, the part of its page's URL after '/d/', so a dataset added to the site works with no new release. Start with [pd_datasets()] to find one, [pd_fields()] to see what it holds, then [pd_rows()], [pd_aggregate()], [pd_read()] or [pd_tbl()] to use it.

Options

'publicdataau.cache'

'TRUE' keeps every downloaded file in [pd_cache_dir()] for reuse. Off by default.

'publicdataau.cache_dir'

Where the cache is kept.

'publicdataau.quiet'

'TRUE' turns off the licence notice and the download progress bar.

'publicdataau.site'

Another copy of the site to read, such as a preview. The 'PUBLICDATA_SITE' environment variable does the same.

Errors

Every error the package raises has the class 'publicdataau_error'. One that comes from failing to reach the site also has 'publicdataau_unreachable', and asking [pd_sf()] for a dataset without a map layer has 'publicdataau_no_layer', so either can be caught with 'tryCatch()'. An error answer from the site keeps the classes 'httr2' gives it, such as 'httr2_http_404'.

Author(s)

Maintainer: Cameron Young cameron@nationaldigital.com.au

Authors:

• Cameron Young cameron@nationaldigital.com.au

Other contributors:

• National Digital [copyright holder, funder]

See Also

Useful links:

• https://publicdata.au/

The download cache

Description

A version of a dataset never changes once published, so a file downloaded once can be kept and read again without another download. Nothing is kept unless you ask: pass 'cache = TRUE' to [pd_read()], [pd_download()], [pd_connect()], [pd_tbl()] or [pd_sf()], or set 'options(publicdataau.cache = TRUE)' for every call.

Usage

pd_cache_dir()

pd_cache_list(slug = NULL, version = NULL)

pd_cache_clear(slug = NULL, version = NULL)

Arguments

Details

'pd_cache_dir()' is where kept files go: 'tools::R_user_dir("publicdataau", "cache")' unless 'options(publicdataau.cache_dir = )' names another folder. 'pd_cache_list()' lists what is kept and 'pd_cache_clear()' deletes it.

Value

‘pd_cache_dir()' returns the folder’s path. 'pd_cache_list()' returns a tibble with one row per kept file: its dataset, version, file, size in bytes and when it was saved. 'pd_cache_clear()' returns the number of bytes freed, invisibly.

See Also

Other files: pd_download(), pd_read(), pd_url()

Examples

pd_cache_dir()
pd_cache_list()

Filters for pd_rows() and pd_aggregate()

Description

Each helper builds one condition in the query API's 'operator.value' form. A plain value passed to [pd_rows()] must match exactly, a vector of several values matches any of them, and 'NA' matches a blank or suppressed cell.

Usage

pd_eq(value)

pd_neq(value)

pd_gt(value)

pd_gte(value)

pd_lt(value)

pd_lte(value)

pd_like(pattern)

pd_ilike(pattern)

pd_in(...)

pd_is_null()

pd_not(filter)

Arguments

Value

A 'pd_filter' object.

See Also

Other query: pd_aggregate(), pd_connect(), pd_rows(), pd_tbl()

Examples

pd_gte(2020)
pd_in("QLD", "NSW")
pd_not(pd_is_null())

Totals of a dataset by group

Description

Totals of a dataset by group

Usage

pd_aggregate(slug, group = NULL, metric = "count", ..., .version = NULL)

Arguments

Value

A tibble with one row per group, typed and carrying the same '"publicdata"' attribute as [pd_rows()].

See Also

Other query: filters, pd_connect(), pd_rows(), pd_tbl()

Examples

pd_aggregate("au-road-deaths", group = "state", year = 2025)

The attribution an answer carries

Description

The attribution an answer carries

Usage

pd_attribution(x)

Arguments

Value

The attribution string the publisher's licence requires, or 'NULL'.

See Also

Other metadata: pd_changes(), pd_cite(), pd_fields(), pd_latest(), pd_provenance(), pd_schema(), pd_tables()

Examples

pd_attribution(pd_rows("au-road-deaths", .limit = 1))

Is publicdata.au reachable?

Description

Asks the site for its catalogue's headers, with a short timeout. Use it to skip work that needs the site when there is no connection.

Usage

pd_available()

Value

'TRUE' when the site answers, else 'FALSE'. It never fails.

See Also

Other find: pd_browse(), pd_catalogue(), pd_dataset(), pd_datasets(), pd_versions()

Examples

pd_available()

A boundary layer as an sf object

Description

A boundary layer as an sf object

Usage

pd_boundaries(layer, cache = NULL)

Arguments

Value

An 'sf' tibble with a row per area: its code, name and the other fields of the layer, in GDA2020 (EPSG:7844).

See Also

Other maps: pd_boundary_layers(), pd_join_boundaries(), pd_sf()

Examples

# A boundary file is tens of megabytes, so this runs only when asked.
electorates <- pd_boundaries("federal_electorate")

Boundary layers

Description

The ABS boundaries publicdata.au serves, each keyed by a code that rows of other datasets carry: council areas (LGA), SA2s, suburbs, postal areas and state and federal electorates.

Usage

pd_boundary_layers()

Value

A tibble with one row per layer: its 'key' (such as '"lga"'), 'slug', 'title', the 'code' and 'name' fields that identify an area, a 'noun' for one area, and the 'version' served.

See Also

Other maps: pd_boundaries(), pd_join_boundaries(), pd_sf()

Examples

pd_boundary_layers()

Open a dataset's page

Description

Opens the dataset's page, or one version's, in the browser, in an interactive session.

Usage

pd_browse(slug, version = NULL)

Arguments

Value

The page's URL, invisibly.

See Also

Other find: pd_available(), pd_catalogue(), pd_dataset(), pd_datasets(), pd_versions()

Examples

if (interactive()) pd_browse("au-road-deaths")

Search every dataset on the government portals

Description

publicdata.au reads the catalogues of the Australian government open data portals, well over a hundred thousand datasets, of which it serves a small part. This searches all of them, so a dataset that is not served yet can be found, and says which are served. Vote for one to be built on its page.

Usage

pd_catalogue(
  q = NULL,
  jurisdiction = NULL,
  status = NULL,
  limit = 20,
  offset = 0
)

Arguments

Value

A tibble with one row per record: its 'id', 'title', 'summary', 'publisher', 'jurisdiction', portal 'url', 'licence', 'formats', 'modified', 'status', the publicdata.au 'page' when served and the 'reason' when closed. 'attr(x, "total")' is the number of matches.

See Also

Other find: pd_available(), pd_browse(), pd_dataset(), pd_datasets(), pd_versions()

Examples

pd_catalogue("water quality", jurisdiction = "Queensland", limit = 5)

What changed between versions

Description

Each time a publisher releases again, the site compares the new version with the one before it, row by row on the dataset's key. 'pd_changes()' lists those comparisons; 'pd_diff()' reads one of them in full.

Usage

pd_changes(slug, from = NULL, to = NULL)

pd_diff(slug, version = NULL)

Arguments

Value

'pd_changes()' returns a tibble with one row per comparison, oldest first: the two versions, their rows, how many rows were added, removed, changed and unchanged, the fields added and removed, and the comparison's URL. A database is compared by each table's row count, so its added, removed and changed are 'NA' and its tables added and removed are listed as fields.

'pd_diff()' returns the comparison as a list of class 'pd_diff': the counts above, the keys of the rows added, removed and changed (up to 50,000 of each, with 'truncated' saying when there were more) and up to ten changed rows with each changed field's old and new value.

See Also

Other metadata: pd_attribution(), pd_cite(), pd_fields(), pd_latest(), pd_provenance(), pd_schema(), pd_tables()

Examples

pd_changes("rba-money-market-daily")


steps <- pd_changes("rba-money-market-daily")
if (nrow(steps)) pd_diff("rba-money-market-daily", steps$to[nrow(steps)])

How to cite a version

Description

A citation for one version of a dataset, naming the publisher, the version and the URL it is kept at, with the attribution the publisher's licence requires. Print it for text, or pass it to [utils::toBibtex()] for a reference manager.

Usage

pd_cite(slug, version = NULL)

Arguments

Value

A [utils::bibentry()] object.

See Also

Other metadata: pd_attribution(), pd_changes(), pd_fields(), pd_latest(), pd_provenance(), pd_schema(), pd_tables()

Examples

cit <- pd_cite("rba-cash-rate")
print(cit, style = "text")
utils::toBibtex(cit)

Connect to a version with DuckDB

Description

Attaches a version's DuckDB file read-only over HTTPS and makes it the current database, so every table and view can be queried by name with 'DBI' or 'dplyr'. Only the blocks a query touches are read, so a count over millions of rows runs without a download. For a database such as G-NAF the file holds every table, the keys between them and the publisher's views; for a single table it holds 'records'.

Usage

pd_connect(
  slug,
  version = NULL,
  name = NULL,
  shared_home = FALSE,
  cache = NULL,
  ...
)

Arguments

Value

A 'DBI' connection. 'attr(con, "publicdata")' names the dataset, the version, the file's URL and local path if cached, and carries the file's provenance: licence, attribution and citation. Disconnect with 'DBI::dbDisconnect(con)'.

See Also

Other query: filters, pd_aggregate(), pd_rows(), pd_tbl()

Examples

con <- pd_connect("au-road-deaths")
DBI::dbGetQuery(con, "SELECT state, count(*) AS n FROM records GROUP BY 1")
DBI::dbDisconnect(con)

A dataset's description and files

Description

A dataset's description and files

Usage

pd_dataset(slug)

Arguments

Value

The dataset's Frictionless data package as a list: title, licence, attribution, fields and every file of the newest version.

See Also

Other find: pd_available(), pd_browse(), pd_catalogue(), pd_datasets(), pd_versions()

Examples

pd_dataset("au-road-deaths")$title

Datasets served by publicdata.au

Description

Datasets served by publicdata.au

Usage

pd_datasets(q = NULL, publisher = NULL, topic = NULL, jurisdiction = NULL)

Arguments

Value

A tibble with one row per dataset: its slug, title, publisher, licence, page and newest Parquet file. Every condition given must match.

See Also

Other find: pd_available(), pd_browse(), pd_catalogue(), pd_dataset(), pd_versions()

Examples

pd_datasets("road crashes")
pd_datasets(publisher = "Transport", jurisdiction = "Queensland")

Download one version's file

Description

Download one version's file

Usage

pd_download(
  slug,
  format = "parquet",
  version = NULL,
  path = NULL,
  table = NULL,
  cache = NULL
)

Arguments

Value

The path of the saved file, invisibly, with the version it resolved to in 'attr(path, "version")'.

See Also

Other files: cache, pd_read(), pd_url()

Examples

f <- pd_download("qld-road-crash-factors", "csv")

A dataset's fields

Description

What each field holds: its name, type and the publisher's description, with the range of a number or date and the values of a field that holds few. Read it before filtering with [pd_rows()], since a condition must name a field exactly and match its values exactly.

Usage

pd_fields(slug, version = NULL)

Arguments

Value

A tibble with one row per field: 'name', 'type' ('"string"', '"integer"', '"number"', '"boolean"', '"date"' or '"datetime"'), 'description', 'min' and 'max' as text, and 'values', a list of the values a field holds when it holds few. For a database such as G-NAF there is a row per field of every table, with a 'table' column, and no ranges. Without ‘version' the fields are the newest version’s; a pinned version is described by its schema, without ranges or values.

See Also

Other metadata: pd_attribution(), pd_changes(), pd_cite(), pd_latest(), pd_provenance(), pd_schema(), pd_tables()

Examples

pd_fields("au-road-deaths")

Join rows to their boundaries

Description

Adds the boundary of the area each row names, by its ABS code, and returns an 'sf' object ready to map. Rows from [pd_rows()] or [pd_aggregate()] carry the codes of the areas their point falls in, such as 'lga_2025_code', and the layer is found from the column's name; any data frame with a column of codes works with 'layer' and 'by'. Aggregate first: a map of 500 council areas is quicker to draw than 80,000 rows each carrying its area's shape.

Usage

pd_join_boundaries(x, layer = NULL, by = NULL, cache = NULL)

Arguments

Value

An 'sf' tibble with the rows of ‘x' in their order, the area’s name when ‘x' lacks it, and the area’s geometry. A row whose code matches no area has an empty geometry, and a message says how many.

See Also

Other maps: pd_boundaries(), pd_boundary_layers(), pd_sf()

Examples

# Downloads the SA2 boundaries, tens of megabytes, so this runs only when asked.
by_sa2 <- pd_aggregate("act-road-crashes", group = "sa2_2021_code")
map <- pd_join_boundaries(by_sa2)
plot(map["count"])

The newest version of a dataset

Description

The newest version of a dataset

Usage

pd_latest(slug)

Arguments

Value

The newest version's date as text, such as '"2026-09-28"'.

See Also

Other metadata: pd_attribution(), pd_changes(), pd_cite(), pd_fields(), pd_provenance(), pd_schema(), pd_tables()

Examples

pd_latest("au-road-deaths")

Where a version came from

Description

The record kept with every version: the publisher's file it was read from, with its address, name, size and SHA-256, when it was fetched, the licence as the publisher stated it and when that was read, and the rows and fields it holds. It answers "where did this number come from" for a reviewer.

Usage

pd_provenance(slug, version = NULL)

Arguments

Value

A list of class ‘pd_provenance', the version’s 'manifest.json'.

See Also

Other metadata: pd_attribution(), pd_changes(), pd_cite(), pd_fields(), pd_latest(), pd_schema(), pd_tables()

Examples

pd_provenance("au-road-deaths")

Read a whole table

Description

Downloads one version's Parquet file, or one table of a database, and reads it with 'arrow'.

Usage

pd_read(slug, version = NULL, table = NULL, cache = NULL, columns = NULL)

Arguments

Value

A tibble, each column labelled with its field's description. 'attr(x, "publicdata")' is the provenance header the file itself carries: version, licence, attribution, citation and source.

See Also

Other files: cache, pd_download(), pd_url()

Examples

crashes <- pd_read("qld-road-crash-factors")

Rows of a dataset

Description

Reads rows from the query API. For a whole table, [pd_read()] or [pd_download()] is faster and has no rate limit.

Usage

pd_rows(
  slug,
  ...,
  .select = NULL,
  .order = NULL,
  .limit = NULL,
  .offset = NULL,
  .version = NULL,
  .all = FALSE
)

Arguments

Value

A tibble, with each column typed as its field says (dates as 'Date', booleans as logical) and labelled with the field's description. 'attr(x, "publicdata")' holds the version, licence and attribution the answer came with; see [pd_attribution()].

See Also

Other query: filters, pd_aggregate(), pd_connect(), pd_tbl()

Examples

pd_rows("au-road-deaths", state = "QLD", year = pd_gte(2024), .limit = 5)

A version's schema

Description

A version's schema

Usage

pd_schema(slug, version = NULL)

Arguments

Value

The version's 'schema.json' as a list: the fields of a table, or for a database every table with its fields, keys and references, and the views. [pd_fields()] gives the fields as a tibble.

See Also

Other metadata: pd_attribution(), pd_changes(), pd_cite(), pd_fields(), pd_latest(), pd_provenance(), pd_tables()

Examples

names(pd_schema("qld-road-crash-factors"))

Read a dataset's map layer as an sf object

Description

Downloads one version's GeoPackage and reads it with 'sf'. Datasets with a location, such as crash points, or a shape, such as ABS boundaries, have one; their coordinates are in the reference system the publisher used, which the GeoPackage names, usually GDA2020 (EPSG:7844).

Usage

pd_sf(slug, version = NULL, cache = NULL)

Arguments

Value

An 'sf' tibble with a row per feature. 'attr(x, "publicdata")' is the provenance the file carries: version, licence, attribution, citation and source.

See Also

Other maps: pd_boundaries(), pd_boundary_layers(), pd_join_boundaries()

Examples

storages <- pd_sf("au-water-storages")
sf::st_crs(storages)$epsg

The tables of a dataset

Description

A database such as G-NAF is served as one DuckDB file holding several tables, with one Parquet file per table beside it. A dataset that is one table has one entry, 'records'.

Usage

pd_tables(slug, version = NULL)

Arguments

Value

A tibble with one row per table: its name, description, rows, number of fields and key.

See Also

Other metadata: pd_attribution(), pd_changes(), pd_cite(), pd_fields(), pd_latest(), pd_provenance(), pd_schema()

Examples

pd_tables("qld-road-crash-factors")

A table to query with dplyr

Description

Attaches a version's DuckDB file with [pd_connect()] and returns one of its tables or views as a lazy 'dplyr' table. Verbs such as 'filter()', ‘group_by()' and 'summarise()' are translated to SQL and run by ’DuckDB' over HTTPS, reading only the blocks they need; 'collect()' brings the result into R. Calls for the same dataset and version share one connection.

Usage

pd_tbl(slug, table = NULL, version = NULL, cache = NULL)

Arguments

Value

A lazy table from 'dbplyr'. Its connection is 'dbplyr::remote_con(x)', which carries the same '"publicdata"' attribute as [pd_connect()].

See Also

Other query: filters, pd_aggregate(), pd_connect(), pd_rows()

Examples

library(dplyr)
pd_tbl("au-road-deaths") |>
  filter(year >= 2020) |>
  count(state) |>
  collect()

A file's URL

Description

The address of one version's file, for reading with another tool such as ‘read.csv()', ’arrow' or 'DuckDB', or for a 'targets' pipeline with 'format = "url"', which reruns when the address changes.

Usage

pd_url(slug, format = "parquet", version = NULL, table = NULL)

Arguments

Value

The URL as text. With a 'version' it names a file that never changes; without one it names the newest version's file, which redirects to a dated URL and moves when the publisher releases again.

See Also

Other files: cache, pd_download(), pd_read()

Examples

pd_url("au-road-deaths", "csv", "2026-08-07")

Every version of a dataset

Description

Every version of a dataset

Usage

pd_versions(slug)

Arguments

Value

A tibble with one row per version, newest first: its date, as-at date, rows, fields and the SHA-256 of the publisher's file.

See Also

Other find: pd_available(), pd_browse(), pd_catalogue(), pd_dataset(), pd_datasets()

Examples

pd_versions("au-road-deaths")