The Integrity of Public Access to Federal Data

Evaluating Disruptions to Open Government Data, 2025-2026

This repository contains the source code, raw data, and the complete text for a report evaluating the state of open government data and the impacts of administrative disruptions between 2025 and 2026. The report is not yet finalized and is currently in-revision and under-review.

Repository Structure

This project is organized into several directories:

/report: Contains the full text of the report, built using Jekyll and the Just the Docs theme. It includes chapters on executive summaries, agency-specific audits (e.g., OMB, USAID, VA), administrative actions, data tools, and a glossary. The report is configured to be served as a static website or compiled directly into a full-bleed PDF (once finalized).
/code: Houses the analytical and data-gathering scripts. It includes Python scripts for interacting with CKAN APIs (Data.gov), auditing data catalogs, and parsing Wayback Machine archives, as well as R scripts for statistical analysis and visualization of the data.
/data: Stores the raw and processed datasets used in the report. This includes JSON data dictionary snapshots from various federal agencies, Data.gov inventory counts, reginfo.gov data pulls, and more.
/images: A placeholder for the charts, screenshots, graphs, and workflow diagrams generated from the data that supplement information featured in the report.
/workflows: Contains the source files for mermaid.js workflows developed for the data auditing routines used in the report.

Data.gov Auditing Scripts

The primary tools for auditing Data.gov are provided in /code/datagov-audit.py. This script is a reproducible command-line utility for auditing federal data inventories. It leverages the Data.gov CKAN API and the Internet Archive's Wayback Machine (CDX API) to track modifications to datasets, validate DCAT-US data.json files, and ensure the long-term integrity and availability of data assets. This script is intended to be called from the command line (via Bash, PowerShell, etc) and the internal functionality have convenience functions that provide a bespoke command line interface (CLI).

Setup and Requirements

Environment: Requires Python 3.
Dependencies: Install the required requests package via pip install -r requirements.txt.
Authentication: To prevent 403 Forbidden errors, Data.gov requires an API key for its api.gsa.gov gateway.
- Configure this by setting the environment variable: export DATAGOV_API_KEY="your_api_key_here".
- If no key is configured, the script will default to using "DEMO_KEY".

Global Configuration Arguments

The following optional arguments can be placed before any subcommand to alter the script's global settings:

--api-key: Directly supply the API key, overriding any environment variables.
--ckan-base: Override the default CKAN API base URL (Defaults to https://api.gsa.gov/technology/datagov/v3/action).
--api-key-env: Specify a custom environment variable to check for the API key (Defaults to DATAGOV_API_KEY).
--user-agent: Define a custom User-Agent header string (Defaults to datagov-inventory-audit/1.2 (research; contact: none)).

Command Reference

1. Catalog & Harvest Discovery

`list-harvest`

Retrieves a complete list of all harvest sources currently available on Data.gov.

Usage: python datagov-audit.py list-harvest [--rows N]
Arguments: * --rows: Set the number of rows fetched per API pagination request (Default: 1000).
Output: Returns a JSON array detailing each harvest source's id, title, source_url, frequency, and source_type.

`find-datajson`

Applies a regular expression filter over a saved list-harvest JSON output to locate specific agencies, URLs, or metadata.

Usage: python datagov-audit.py find-datajson <filepath> <pattern> [--limit N]
Arguments:
- filepath: The local path to the JSON file generated by the list-harvest command.
- pattern: The regex pattern to execute against the harvest metadata.
- --limit: Cap the maximum number of matched results returned (Default: 10).

`fetch-datajson`

Downloads and performs basic validation on an agency's DCAT-US data.json inventory file.

Usage: python datagov-audit.py fetch-datajson <url> [--out FILE]
Arguments:
- url: The target URL pointing to the data.json file.
- --out: An optional filepath to save the retrieved JSON locally.

2. Snapshot Generation

`snapshot-org`

Captures a static point-in-time snapshot of all Data.gov packages tied to a designated CKAN organization.

Usage: python datagov-audit.py snapshot-org <org> --out <FILE> [--rows-per-page N]
Arguments:
- org: The identifier or string for the CKAN organization.
- --out: (Required) The destination filepath for the snapshot JSON.
- --rows-per-page: Adjust the pagination batch size (Default: 1000).

`snapshot-source`

Captures a static point-in-time snapshot of all Data.gov packages originating from a specified harvest source UUID.

Usage: python datagov-audit.py snapshot-source <source_id> --out <FILE> [--rows-per-page N]
Arguments:
- source_id: The UUID of the specific harvest source.
- --out: (Required) The destination filepath for the snapshot JSON.

3. Diffing & Auditing Tools

`diff`

Executes a comparison between two Data.gov CKAN snapshots (generated via snapshot-org or snapshot-source) to identify datasets that have been added, removed, or modified.

Usage: python datagov-audit.py diff <old_file> <new_file> [--key KEY] [--out FILE]
Arguments:
- old / new: Filepaths pointing to the older and newer snapshot JSON files.
- --key: The primary field utilized to track dataset identity across snapshots (Default: id).
- --out: Optional filepath to output the complete diff report.

`diff-inventory`

Compares two raw agency inventory files (e.g., historical vs. current data.json representations). It uses stable SHA-256 canonical hashing to accurately detect underlying changes.

Usage: python datagov-audit.py diff-inventory <old> <new> [--dataset-key KEY] [--id-field FIELD] [--ignore-key KEY] [--out FILE]
Arguments:
- old / new: Filepaths or live URLs for the old and new inventory files.
- --dataset-key: The JSON key holding the datasets array (Default: dataset).
- --id-field: The specific field to use for identifying datasets. If omitted, the tool automatically falls back to identifier, landingPage, or title.
- --ignore-key: (Repeatable flag) Specifies dataset keys to ignore during the diff comparison, such as continuously updating timestamps.
- --out: Filepath to output the complete JSON diff report.

`inspect-diff`

Generates a human-readable, side-by-side view of modified fields for designated datasets found within a diff report.

Batch Mode Usage: python datagov-audit.py inspect-diff --report <diff_report.json> [--out FILE].
Single ID Mode Usage: python datagov-audit.py inspect-diff <old_file> <new_file> <target_id>.

`cross-check`

Validates catalog synchronization by cross-referencing an agency's live data.json with a Data.gov CKAN snapshot. This highlights datasets that are either missing from Data.gov or lingering as orphaned records on Data.gov.

Usage: python datagov-audit.py cross-check <agency_file> <ckan_snapshot> [--dataset-key KEY] [--out FILE]
Output: Displays summary counts in the console. If --out is not provided, it prints a sample list of the discrepancies.

4. Wayback Machine Integrations

`wayback-cdx`

Builds and optionally executes an Internet Archive CDX API query to locate historical snapshots of a target URL.

Usage: python datagov-audit.py wayback-cdx <url> [--from-ts TS] [--limit N] [--fetch]
Arguments:
- url: The target URL to query.
- --from-ts: Optional timestamp parameter (e.g., YYYYMMDD) to bound the search results.
- --fetch: If provided, the script executes the query directly against the CDX API instead of just printing the constructed URL.

`compare-wayback`

Downloads a live URL, fetches the latest Wayback Machine capture of that same URL, and compares the archived raw content against the live content. It includes specialized, lightweight byte and hash comparisons for .csv and .json files.

Usage: python datagov-audit.py compare-wayback <url> [--from-ts TS] [--out FILE]

`clean-wayback`

When JSON files are retrieved directly via the Wayback Machine, the Internet Archive injects a proxy prefix (https://web.archive.org/web/.../) into embedded URLs. This command recursively sanitizes the downloaded JSON by stripping these injected Wayback tokens.

Usage: python datagov-audit.py clean-wayback <input_file_or_url> [--wayback-token TOKEN] [--wayback-url URL] [--out FILE]
Arguments:
- --wayback-token: The precise timestamp identifier string injected by the Wayback Machine (e.g., 20240807053438).
- --wayback-url: Alternately, provide a full replay URL and the script will automatically extract the token.

Additional Scripts for Data.gov

Additional sripts for auditing Data.gov are also provided in the repository (some for legacy purposes). Two important ones are the resume-catalog python scripts which create comprehensive inventories of all datasets either in collections or not listed in the Federal Data Catalog.

The scripts in this repository that access data from Data.gov's CKAN API were written to use the either the public, unauthenticated API hosted at https://catalog.data.gov/api/3/action/ or the api-key authenticated version which has a different baseurl: https://api.gsa.gov/technology/datagov/v3/action/package_search?api_key=DEMO_KEY. This was done intentionally to maximize public accessibility and ease-of-use.

For many use cases these scripts work just fine using the unauthenticated API. However, some endpoints (notably package_show, and sometimes high-volume calls to package_search can return HTTP 403 Forbidden error responses due to access controls or rate limiting.

Steps to use authentication

To make these scripts more robust, some modification will be needed to make them compatible with the official GSA Data.gov API gateway using an API key.

1) Obtain an API key

API keys are issued by GSA’s API platform (not directly by Data.gov):

Sign up at: https://api.data.gov/signup/
After registration, you will receive an API key via email that can be used with the Data.gov CKAN gateway.

2) Provide the API key securely via the terminal

Never hard-code API keys into your scripts. Instead, provide the key via an environment variable:

macOS / Linux (bash, zsh):

export DATAGOV_API_KEY="YOUR API KEY"

Windows PowerShell:

$env:DATAGOV_API_KEY="YOUR_API_KEY"

3) Change the base-urls in the scripts

In many cases, the scripts in this repository accept an api-key and will automatically switch between baseurls. In some cases, however, you will need to manually edit the scripts. In general, to use the authenticated gateway, the CKAN API paths are the same, but the host for the baseurls needs to be modified in the scripts:

Replace:

https://catalog.data.gov/api/3/action/package_search
https://catalog.data.gov/api/3/action/package_show

with:

https://api.gsa.gov/technology/datagov/v3/action/package_search
https://api.gsa.gov/technology/datagov/v3/action/package_show

If the specific script only uses package_search, you only need to switch that one.

4) Update the scripts with new headers and requests

Somewhere after import requests add a minimal header dictionary to the script. Something like this should work:

HEADERS = {
    "User-Agent": "MyScript/1.0",
    "x-api-key": os.environ.get("DATAGOV_API_KEY"),
}

replacing MyScript with the appropriate script name.

Notes on the headers:

User-Agent is strongly recommended for bulk API usage.
If DATAGOV_API_KEY is not set, the value becomes None (which is fine if using the unauthenticated gateway.
You can add additional headers here too, including your email if you want. These are documented in the full CKAN API guide.

Find every API call that starts with:

requests.get(...)

and pass the headers:

requests.get(..., headers=HEADERS, ...)

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
code		code
data		data
images		images
report		report
workflows		workflows
.gitattributes		.gitattributes
.gitignore		.gitignore
README.md		README.md
datapackage.json		datapackage.json

Navigation Menu

Repository files navigation

The Integrity of Public Access to Federal Data

Repository Structure

Data.gov Auditing Scripts

Setup and Requirements

Global Configuration Arguments

Command Reference

1. Catalog & Harvest Discovery

`list-harvest`

`find-datajson`

`fetch-datajson`

2. Snapshot Generation

`snapshot-org`

`snapshot-source`

3. Diffing & Auditing Tools

`diff`

`diff-inventory`

`inspect-diff`

`cross-check`

4. Wayback Machine Integrations

`wayback-cdx`

`compare-wayback`

`clean-wayback`

Additional Scripts for Data.gov

Steps to use authentication

1) Obtain an API key

2) Provide the API key securely via the terminal

3) Change the base-urls in the scripts

4) Update the scripts with new headers and requests

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

The Integrity of Public Access to Federal Data

Repository Structure

Data.gov Auditing Scripts

Setup and Requirements

Global Configuration Arguments

Command Reference

1. Catalog & Harvest Discovery

list-harvest

find-datajson

fetch-datajson

2. Snapshot Generation

snapshot-org

snapshot-source

3. Diffing & Auditing Tools

diff

diff-inventory

inspect-diff

cross-check

4. Wayback Machine Integrations

wayback-cdx

compare-wayback

clean-wayback

Additional Scripts for Data.gov

Steps to use authentication

1) Obtain an API key

2) Provide the API key securely via the terminal

3) Change the base-urls in the scripts

4) Update the scripts with new headers and requests

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

`list-harvest`

`find-datajson`

`fetch-datajson`

`snapshot-org`

`snapshot-source`

`diff`

`diff-inventory`

`inspect-diff`

`cross-check`

`wayback-cdx`

`compare-wayback`

`clean-wayback`