Federal Program Inventory (Pipeline)

Overview

The Federal Program Inventory (FPI) is a comprehensive tool with information about Federal programs.

This repository has been refactored to serve exclusively as the data pipeline for the FPI. It extracts data from the underlying SQLite database, transforms it, and exports it as static JSON files.

Architecture Evolution

This repository represents the next evolution of the FPI data pipeline. It preserves the robust ETL logic established by previous maintainers while decoupling it from the legacy Jekyll frontend.

Key Decisions

1. Preservation of Core Logic: The complex data transformation logic (previously in data_processing/load.py) has been preserved in pipeline/load.py. We rely on the same SQL queries and business logic to ensure data consistency.
2. Decoupled Data Output: Instead of generating Markdown files tightly coupled to Jekyll's frontmatter format, the pipeline now exports standard JSON.
   • Benefit: This makes the FPI data universally consumable. It can now power any frontend (React, Vue, mobile apps) or be ingested by other systems without parsing Markdown.
3. Simplified Maintenance: By removing the website and API code, this repository focuses entirely on its primary value: producing high-quality, clean Federal Program data.

Data Fidelity

The JSON output is a direct serialization of the program objects generated by the original pipeline. No data fields were lost in this transition; they are simply formatted for modern consumption.

Project Structure

• pipeline/: Contains the Python scripts for Extract, Transform, and Load (ETL).
  • load.py: Core logic for generating program data.
  • export_data.py: The main entry point that orchestrates the export.
• data/: The output directory where generated JSON files are stored.
• package.json: Manages build scripts.

How to Run

To regenerate the data:

npm run build:data

This command runs the Python pipeline and populates the data/ directory with:

• programs.json: A summary list of all programs.
• programs/{id}.json: Detailed data for each individual program.
• filters.json: Shared data for filtering (agencies, categories, etc.).
• metadata.json: Build timestamp and fiscal year information.