Methodology

Every number on NYC Dashboard is traceable to a public data source. This page documents our data sources, update cadences, caching strategy, and known limitations. We believe transparency about our methodology is as important as the data itself.

Data Sources

Dataset	Source	ID	Update Freq.
Expense Budget Adopted and modified budget by agency, budget code, and fiscal year.	NYC Open Data (Socrata)	mwzb-yiwb	1 day
Capital Budget Capital project authorizations, commitments, and expenditures.	NYC Open Data (Socrata)	46m8-77gv	1 day
Payroll Individual employee payroll records: base salary, overtime, total compensation.	NYC Open Data (Socrata)	k397-673e	7 days
Contracts Registered contracts: vendor, amount, type, start/end dates.	NYC Open Data (Socrata)	qyyg-4tf5	6 hours
Capital Projects Active capital projects: managing agency, budget type, phase, expenditures.	NYC Open Data (Socrata)	fb86-vt7u	1 day
311 Complaints 311 service requests: complaint type, agency, location, resolution time.	NYC Open Data (Socrata)	erm2-nwe9	30 min
Crime (Historic) NYPD complaint reports, 2006-2019.	NYC Open Data (Socrata)	qgea-i56i	1 day
Crime (YTD) NYPD complaint reports, current year-to-date.	NYC Open Data (Socrata)	5uac-w243	1 day
MMR Performance Mayor's Management Report: agency performance metrics with targets.	NYC Open Data (Socrata)	rbed-zzin	1 day
Math Test Results NYC public school math test scores by grade, district, and demographic.	NYC Open Data (Socrata)	74kb-55u9	1 day
Graduation Rates 4-year and 6-year graduation rates by school, borough, and cohort.	NYC Open Data (Socrata)	mjm3-8dw8	1 day
Property Valuation Individual property assessments: market value, assessed value, tax class.	NYC Open Data (Socrata)	8y4t-faws	7 days
Property Tax Rates Historical property tax rates by class, FY2000-present.	NYC Open Data (Socrata)	7zb8-7bpk	7 days
Tax Revenue Actuals Actual tax collections by category, FY2001-present.	NYC Open Data (Socrata)	j3uq-sh95	1 day
Revenue Budget Revenue projections by source across fiscal years.	NYC Open Data (Socrata)	ugzk-a6x4	1 day
Spending Transactions Actual spending transactions: vendor, agency, check amount.	Checkbook NYC	REST API	6 hours

Caching Strategy

Layer 1: In-Memory

Bounded Map cache (max 500 entries) with TTL-based expiry. Fastest access, cleared on server restart.

Layer 2: Database (Turso)

Persistent cache stored in Turso (libSQL). Survives server restarts. Metadata table tracks TTL per dataset.

Layer 3: Live API

Direct query to NYC Open Data or Checkbook NYC. Fallback when cache misses. Stale cache is served if the live API fails.

Known Limitations

1.Agency names are inconsistent across datasets (e.g., "POLICE DEPARTMENT" vs "NYPD" vs "056"). We maintain an alias map covering the top 22 agencies; smaller agencies may not resolve correctly.
2.SODA API returns all numeric fields as strings. We parse with fallback defaults, but edge cases (e.g., parenthetical negatives) may be mishandled.
3.The 311 dataset has 20M+ rows. We filter to the most recent 12 months to keep response times reasonable. Historical 311 data is available but not displayed.
4.Crime data merges two datasets (historic + YTD) using complaint date. There may be a brief overlap or gap at the cutover date.
5.Checkbook NYC uses a REST/XML API that is less reliable than Socrata. Spending transaction data may be stale if the API is temporarily unavailable.
6.Property valuation data is annual. Mid-year changes (e.g., reassessments, exemptions) are not reflected until the next annual update.
7.Federal and state aid figures are from the official NYC budget. Actual disbursements may differ from budgeted amounts.
8.Per-pupil spending is calculated as DOE adopted budget divided by total enrollment. This does not account for charter school funding or special education allocations.

Methodology Notes

Agency Name Resolution

Each dataset uses different identifiers for the same agency (budget codes, abbreviations, full names with varying capitalization). We maintain an alias map covering 22 agencies with 5-10 aliases each, enabling cross-dataset joins for budget, payroll, contracts, 311 complaints, crime, and MMR performance.

Data Freshness

Every API response includes a fromCache boolean and fetchedAt timestamp. Cache TTLs vary from 30 minutes (311 complaints) to 7 days (payroll, property data). If a live API call fails, stale cached data is served with a staleness indicator.