sfOpenData

sfOpenData provides a lightweight R interface to the San Francisco Open Data Portal.

The package allows users to search, filter, and download datasets from the San Francisco Open Data Portal directly into R without manually constructing API queries, handling JSON responses, or performing type conversion.

Designed for students, educators, researchers, journalists, civic technologists, and analysts, sfOpenData reduces the technical overhead required to begin working with municipal Open Data while preserving access to the underlying Socrata infrastructure.

How sfOpenData Works

The package provides a streamlined interface to the San Francisco Open Data Portal’s Socrata API.

Internally, sfOpenData:

• retrieves metadata from the live San Francisco Open Data catalog
• constructs parameterized HTTP requests
• downloads JSON responses from Socrata endpoints
• converts results into tidy tibble outputs
• optionally cleans column names
• optionally performs conservative type coercion

Most workflows begin with sf_list_datasets(), which retrieves a live catalog of datasets available through the San Francisco Open Data Portal.

Datasets can then be downloaded using either:

• a human-readable catalog key
• the official Socrata dataset UID, such as "abcd-1234"

The human-readable key is designed to improve readability and usability, while the UID is the stable identifier used by the Socrata platform.

Core Functions

The package provides three primary functions:

• sf_list_datasets() retrieves a live catalog of available San Francisco Open Data datasets, including human-readable keys, Socrata UIDs, names, and other available metadata.

• sf_pull_dataset() downloads cataloged datasets using either a human-readable key or Socrata UID, with support for filtering, ordering, date ranges, optional column-name cleaning, and optional type coercion.

• sf_any_dataset() downloads data directly from a valid Socrata JSON endpoint without requiring the dataset to appear in the package catalog.

Datasets retrieved through sf_pull_dataset() support arguments including:

• limit
• filters
• date
• from
• to
• date_field
• where
• order
• clean_names
• coerce_types

All functions return tibble outputs.

Advanced users may also provide raw SoQL conditions through the where argument.

SoQL, or Socrata Query Language, is the query syntax used by Socrata-powered Open Data portals. Additional information is available from the Socrata developer documentation.

Installation

Install from CRAN

install.packages("sfOpenData")

Install the development version from GitHub

# install.packages("pak")
pak::pak("gomes-sh/sfOpenData")

Alternatively:

# install.packages("remotes")
remotes::install_github("gomes-sh/sfOpenData")

Example

library(sfOpenData)
library(dplyr)

# Browse available datasets
catalog <- sf_list_datasets()

# Search for datasets containing a keyword
catalog |>
  filter(grepl("Health", name, ignore.case = TRUE)) |>
  select(key, dataset_id, dataset_name)

# Pull a dataset using its UID
example_data <- sf_pull_dataset(
  dataset = "pyih-qa8i",
  limit = 100
)

# Pull the same dataset using its catalog key
example_data_by_key <- sf_pull_dataset(
  dataset = "health_inspection_scores_2016_2019",
  limit = 100
)

# Pull filtered data

filtered_data <- sf_pull_dataset(
  dataset = "pyih-qa8i",
  limit = 100,
  filters = list(
    business_name = "Laurel Court"
  )
)

The filters argument accepts a named list and automatically constructs the corresponding SoQL filtering conditions.

Multiple values may be supplied for one field:

filtered_data <- sf_pull_dataset(
  dataset = "pyih-qa8i",
  limit = 100,
  filters = list(
    business_name = c("Laurel Court", "Ike's Kitchen")
  )
)

Multiple fields may also be combined:

filtered_data <- sf_pull_dataset(
  dataset = "pyih-qa8i",
  limit = 100,
  filters = list(
    business_name = "Laurel Court",
    inspection_score = 85
  )
)

Date filtering is available for datasets containing date or datetime fields:

date_filtered_data <- sf_pull_dataset(
  dataset = "pyih-qa8i",
  from = "2017-01-01",
  to = "2018-01-01",
  date_field = "inspection_date",
  limit = 100
)

Accessing Any Socrata Endpoint

When a dataset is not available through sf_list_datasets(), it can be downloaded directly using sf_any_dataset().

endpoint_data <- sf_any_dataset(
  json_link = "https://data.sfgov.org/resource/pyih-qa8i.json",
  limit = 100
)

Use sf_pull_dataset() for catalog-based workflows and sf_any_dataset() when working directly with a Socrata JSON endpoint.

Learn by Example

A complete introductory workflow is available in the package vignette:

vignette("getting-started", package = "sfOpenData")

The vignette demonstrates how to:

• browse the dataset catalog
• download data using a key or UID
• filter records
• work with date ranges
• access direct JSON endpoints
• perform a simple analysis

Package Website

Complete documentation is available on the package website:

https://github.com/nyc-open-data-lab/sfOpenData

The website includes:

• function reference pages
• installation instructions
• introductory articles
• vignettes
• release notes

Development

To run the package tests locally:

devtools::test()

To rebuild the documentation:

devtools::document()

To run a complete package check:

devtools::check()

To rebuild the pkgdown website:

pkgdown::build_site()

Contributing

Contributions are welcome.

To report a bug, request a feature, or suggest an improvement, open an issue on GitHub:

https://github.com/nyc-open-data-lab/sfOpenData/issues

Pull requests are also welcome. Before submitting a pull request, please ensure that:

• package documentation has been regenerated
• automated tests pass
• devtools::check() completes successfully
• new behavior is documented and tested

Author

Shelby Lyn Gomes

Email: gomessh@mailbox.org
GitHub: @gomes-sh

Maintenance

Because the package retrieves metadata dynamically from the live San Francisco Open Data catalog, newly published datasets may become available without requiring a package update.

Package updates may still be required when the portal changes its catalog structure, dataset metadata fields, or API behavior.

Disclaimer

sfOpenData is an independent project and is not affiliated with, endorsed by, or maintained by San Francisco or the organization responsible for the San Francisco Open Data Portal.