Skip to content

Commit 68b26e6

Browse files
shvngisinghshvngisingh
committed
Add new Commerce EDA USASpending import
1 parent 3371e36 commit 68b26e6

9 files changed

Lines changed: 616 additions & 0 deletions

File tree

‎statvar_imports/commerce_eda_usaspending/Investmentmetadata.csv‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
parameter,value
2+
url,https://www.usaspending.gov/
3+
description,EDA Investments by Applicant State and Program (sourced from USASpending.gov)
4+
#place_type,AdministrativeArea1
5+
#places_within,country/USA
6+
#start_date,2012
7+
#end_date,2026
8+
release_frequency,Yearly
9+
process,
10+
comments,
11+
output_columns,"observationAbout,observationDate,variableMeasured,value,unit,scalingFactor,measurementMethod"
12+
header_rows,1

‎statvar_imports/commerce_eda_usaspending/Investmentpvmap.csv‎

Lines changed: 72 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,72 @@
1+
key,prop,value,p1,v1,p2,v2,p3,v3,,
2+
Place,,,observationAbout,geoId/{Place},,,,,
3+
Year,observationDate,{Number},,,,,,,,
4+
Value,value,{Number},populationType,Investment,measuredProperty,amount,unit,USDollar
5+
Alabama,observationAbout,geoId/01
6+
Alaska,observationAbout,geoId/02
7+
Arizona,observationAbout,geoId/04
8+
Arkansas,observationAbout,geoId/05
9+
California,observationAbout,geoId/06
10+
Colorado,observationAbout,geoId/08
11+
Connecticut,observationAbout,geoId/09
12+
Delaware,observationAbout,geoId/10
13+
District of Columbia,observationAbout,geoId/11
14+
Florida,observationAbout,geoId/12
15+
Georgia,observationAbout,geoId/13
16+
Hawaii,observationAbout,geoId/15
17+
Idaho,observationAbout,geoId/16
18+
Illinois,observationAbout,geoId/17
19+
Indiana,observationAbout,geoId/18
20+
Iowa,observationAbout,geoId/19
21+
Kansas,observationAbout,geoId/20
22+
Kentucky,observationAbout,geoId/21
23+
Louisiana,observationAbout,geoId/22
24+
Maine,observationAbout,geoId/23
25+
Maryland,observationAbout,geoId/24
26+
Massachusetts,observationAbout,geoId/25
27+
Michigan,observationAbout,geoId/26
28+
Minnesota,observationAbout,geoId/27
29+
Mississippi,observationAbout,geoId/28
30+
Missouri,observationAbout,geoId/29
31+
Montana,observationAbout,geoId/30
32+
Nebraska,observationAbout,geoId/31
33+
Nevada,observationAbout,geoId/32
34+
New Hampshire,observationAbout,geoId/33
35+
New Jersey,observationAbout,geoId/34
36+
New Mexico,observationAbout,geoId/35
37+
New York,observationAbout,geoId/36
38+
North Carolina,observationAbout,geoId/37
39+
North Dakota,observationAbout,geoId/38
40+
Ohio,observationAbout,geoId/39
41+
Oklahoma,observationAbout,geoId/40
42+
Oregon,observationAbout,geoId/41
43+
Pennsylvania,observationAbout,geoId/42
44+
Rhode Island,observationAbout,geoId/44
45+
South Carolina,observationAbout,geoId/45
46+
South Dakota,observationAbout,geoId/46
47+
Tennessee,observationAbout,geoId/47
48+
Texas,observationAbout,geoId/48
49+
Utah,observationAbout,geoId/49
50+
Vermont,observationAbout,geoId/50
51+
Virginia,observationAbout,geoId/51
52+
Washington,observationAbout,geoId/53
53+
West Virginia,observationAbout,geoId/54
54+
Wisconsin,observationAbout,geoId/55
55+
Wyoming,observationAbout,geoId/56
56+
American Samoa,observationAbout,geoId/60
57+
Federated States of Micronesia,observationAbout,geoId/64
58+
Guam,observationAbout,geoId/66
59+
Northern Mariana Islands,observationAbout,geoId/69
60+
Palau,observationAbout,geoId/70
61+
Puerto Rico,observationAbout,geoId/72
62+
Marshall Islands,observationAbout,geoId/68
63+
U.S. Virgin Islands,observationAbout,geoId/78
64+
Economic Adjustment Assistance,investmentProgram,EconomicAdjustmentAssistance
65+
Planning,investmentProgram,ProgramPlanning
66+
Public Works,investmentProgram,PublicWorks
67+
Regional Innovation Strategies,investmentProgram,RegionalInnovationStrategies
68+
Technical Assistance,investmentProgram,TechnicalAssistance
69+
Research and National Technical Assistance,investmentProgram,ResearchNationalTechnicalAssistance
70+
Trade Adjustment Assistance for Firms,investmentProgram,TradeAdjustmentAssistance
71+
Research,investmentProgram,Research
72+
Total,investmentProgram,

‎statvar_imports/commerce_eda_usaspending/README.md‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Commerce_EDA (USASpending Source)
2+
3+
This importer fetches and processes U.S. Economic Development Administration (EDA) investment datasets dynamically from USASpending.gov.
4+
5+
- **Source**: [USASpending.gov API](https://api.usaspending.gov/)
6+
- **Place Type**: U.S. States and Territories (`AdministrativeArea1`)
7+
- **Time Coverage**: FY 2012 to Present (dynamically fetched up to current year + 1)
8+
- **Import Type**: Automated API-based import
9+
- **Release Frequency**: Continuous / On-demand
10+
11+
## Pipeline Steps
12+
13+
### 1. Preprocessing (`process.py`)
14+
Run the python script to dynamically download all awards from USAspending.gov and pivot/structure them into a clean tall format:
15+
```bash
16+
python3 process.py
17+
```
18+
This writes the formatted data to `output/Investment_cleaned.csv` with columns:
19+
`Place,State or Territory / EDA Program,Year,Value`
20+
21+
### 2. Statistical Variable Processing
22+
Run the Data Commons `stat_var_processor.py` to generate the final MCF and CSV files for import:
23+
```bash
24+
python3 ../../tools/statvar_importer/stat_var_processor.py \
25+
--input_data=output/Investment_cleaned.csv \
26+
--pv_map=Investmentpvmap.csv \
27+
--config_file=Investmentmetadata.csv \
28+
--output_path=output/Investment_output \
29+
--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf \
30+
--output_counters=counters/Investment_counters.csv
31+
```
32+
33+
## Generic Updates for Coming Years
34+
- The preprocessing script (`process.py`) dynamically queries the USASpending.gov API up to the current fiscal year + 1.
35+
- `Investmentpvmap.csv` is configured in tall format to resolve years and values generically using `{Number}` and `{Place}` wildcard templates.
36+
- Start and end dates are commented out in `Investmentmetadata.csv` to ensure the importer remains entirely year-agnostic and never needs manual updates as new year records become available.

‎statvar_imports/commerce_eda_usaspending/counters/Investment_counters.csv‎

Lines changed: 74 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,74 @@
1+
"key","value"
2+
"1:process_input_generated-svobs",2483
3+
"1:process_input_generated-svobs-Investment_cleaned.csv",2483
4+
"1:process_input_input-data-rows",2483
5+
"1:process_input_input-files-processed",1
6+
"1:process_input_input-header-rows",1
7+
"1:process_input_input-rows-processed",2484
8+
"1:process_input_input-sections",1
9+
"1:process_input_num-rows-output/Investment_cleaned.csv",2679
10+
"1:process_input_process-mem",2505457664
11+
"1:process_input_process-mem-rss",1249935360
12+
"1:process_input_process-time-sys-secs",2.84
13+
"1:process_input_process-time-user-secs",54.44
14+
"1:process_input_process_elapsed_time",2.30
15+
"1:process_input_process_remaining_time",2.66
16+
"1:process_input_processed",2484
17+
"1:process_input_processing-input-rows-rate",1082.50
18+
"1:process_input_processing-input-rows-rate_output/Investment_cleaned.csv",1083.17
19+
"1:process_input_processing-time-seconds",2.29
20+
"1:process_input_processing-time-seconds_output/Investment_cleaned.csv",2.29
21+
"1:process_input_processing_rate",1082.29
22+
"1:process_input_start_time",551908.59
23+
"1:process_input_total",5358
24+
"1:process_input_warning-unresolved-value-ref",2521
25+
"2:prepare_output_process-mem",2505457664
26+
"2:prepare_output_process-mem-rss",1251016704
27+
"2:prepare_output_process-time-sys-secs",2.88
28+
"2:prepare_output_process-time-user-secs",54.83
29+
"2:prepare_output_process_elapsed_time",6.17
30+
"2:prepare_output_processed",0
31+
"2:prepare_output_start_time",551910.88
32+
"3:write_statvar_mcf_process-mem",2505457664
33+
"3:write_statvar_mcf_process-mem-rss",1251090432
34+
"3:write_statvar_mcf_process-time-sys-secs",2.88
35+
"3:write_statvar_mcf_process-time-user-secs",54.85
36+
"3:write_statvar_mcf_process_elapsed_time",0.27
37+
"3:write_statvar_mcf_processed",0
38+
"3:write_statvar_mcf_start_time",551917.06
39+
"4:write_svobs_csv_process-mem",2505457664
40+
"4:write_svobs_csv_process-mem-rss",1251090432
41+
"4:write_svobs_csv_process-time-sys-secs",2.88
42+
"4:write_svobs_csv_process-time-user-secs",54.91
43+
"4:write_svobs_csv_process_elapsed_time",0.06
44+
"4:write_svobs_csv_processed",0
45+
"4:write_svobs_csv_start_time",551917.33
46+
"PVs-matched",41
47+
"dropped-output-statvars-mcf",6
48+
"error-input-ignore-node-diff",6
49+
"error-node-merge-conflict",6
50+
"existing-nodes-from-api",6
51+
"existing-statvars-from-mcf",2483
52+
"generated-statvars",2483
53+
"generated-unique-statvars",6
54+
"nodes-with-diff",6
55+
"output-svobs-csv-rows",2483
56+
"output-svobs-unique-observationAbout",57
57+
"output-svobs-unique-observationDate",15
58+
"output-svobs-unique-typeOf",1
59+
"output-svobs-unique-unit",1
60+
"output-svobs-unique-value",1868
61+
"output-svobs-unique-variableMeasured",6
62+
"process-mem",2505457664
63+
"process-mem-rss",1249566720
64+
"process-time-sys-secs",2.84
65+
"process-time-user-secs",52.15
66+
"process_elapsed_time",0.00
67+
"processed",0
68+
"pvs-added",6
69+
"pvs-added-definition",6
70+
"recon-lookup-schema",1969
71+
"recon-unmodified-nodes",2489
72+
"start_time",551908.59
73+
"svobs-added",2483
74+
"warning-missing-property-key",119496

‎statvar_imports/commerce_eda_usaspending/manifest.json‎

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,26 @@
1+
{
2+
"import_specifications": [
3+
{
4+
"import_name": "Commerce_EDA_USASpending",
5+
"curator_emails": [
6+
"support@datacommons.org"
7+
],
8+
"provenance_url": "https://api.usaspending.gov/",
9+
"provenance_description": "EDA Investments by Applicant State and Program, dynamically sourced from USASpending.gov.",
10+
"scripts": [
11+
"process.py",
12+
"../../tools/statvar_importer/stat_var_processor.py --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf --input_data=output/Investment_cleaned.csv --pv_map=Investmentpvmap.csv --config_file=Investmentmetadata.csv --output_path=output/Investment_output --output_counters=counters/Investment_counters.csv"
13+
],
14+
"import_inputs": [
15+
{
16+
"template_mcf": "output/Investment_output.tmcf",
17+
"cleaned_csv": "output/Investment_output.csv"
18+
}
19+
],
20+
"source_files": [
21+
"output/Investment_cleaned.csv"
22+
],
23+
"cron_schedule": "30 05 * * 1"
24+
}
25+
]
26+
}

‎statvar_imports/commerce_eda_usaspending/output/Investment_cleaned.csv‎

Lines changed: 101 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,101 @@
1+
Place,State or Territory / EDA Program,Year,Value
2+
Alabama,Total,2012,1587960
3+
Alabama,Total,2013,7018395
4+
Alabama,Total,2014,2471320
5+
Alabama,Total,2015,4095340
6+
Alabama,Total,2016,9226910
7+
Alabama,Total,2017,2130378
8+
Alabama,Total,2018,1974304
9+
Alabama,Total,2019,284500
10+
Alabama,Total,2020,5996739
11+
Alabama,Total,2021,1347486
12+
Alabama,Total,2022,24244719
13+
Alabama,Total,2023,4927240
14+
Alabama,Total,2024,3622744
15+
Alabama,Economic Adjustment Assistance,2013,6076100
16+
Alabama,Economic Adjustment Assistance,2014,1376320
17+
Alabama,Economic Adjustment Assistance,2015,3087340
18+
Alabama,Economic Adjustment Assistance,2016,5786817
19+
Alabama,Economic Adjustment Assistance,2020,4699739
20+
Alabama,Economic Adjustment Assistance,2022,21577743
21+
Alabama,Economic Adjustment Assistance,2023,1229560
22+
Alabama,Planning,2012,945000
23+
Alabama,Planning,2014,1045000
24+
Alabama,Planning,2015,1008000
25+
Alabama,Planning,2016,333000
26+
Alabama,Planning,2017,1050000
27+
Alabama,Planning,2018,1050000
28+
Alabama,Planning,2020,1297000
29+
Alabama,Planning,2021,1047486
30+
Alabama,Planning,2023,1260000
31+
Alabama,Planning,2024,980000
32+
Alabama,Public Works,2013,942295
33+
Alabama,Public Works,2016,2607093
34+
Alabama,Public Works,2022,1680500
35+
Alabama,Public Works,2023,2437680
36+
Alabama,Public Works,2024,2642744
37+
Alabama,Regional Innovation Strategies,2022,657622
38+
Alabama,Technical Assistance,2012,642960
39+
Alabama,Technical Assistance,2014,50000
40+
Alabama,Technical Assistance,2016,500000
41+
Alabama,Technical Assistance,2017,1080378
42+
Alabama,Technical Assistance,2018,924304
43+
Alabama,Technical Assistance,2019,284500
44+
Alabama,Technical Assistance,2021,300000
45+
Alabama,Technical Assistance,2022,328854
46+
Alaska,Total,2012,7300925
47+
Alaska,Total,2013,188155
48+
Alaska,Total,2014,2799384
49+
Alaska,Total,2015,1637587
50+
Alaska,Total,2016,782377
51+
Alaska,Total,2017,4947906
52+
Alaska,Total,2018,699418
53+
Alaska,Total,2019,2281777
54+
Alaska,Total,2020,3601347
55+
Alaska,Total,2021,4616907
56+
Alaska,Total,2022,39760024
57+
Alaska,Total,2023,2982734
58+
Alaska,Total,2024,12585199
59+
Alaska,Total,2025,1204252
60+
Alaska,Total,2026,499874
61+
Alaska,Economic Adjustment Assistance,2012,2133425
62+
Alaska,Economic Adjustment Assistance,2014,324000
63+
Alaska,Economic Adjustment Assistance,2016,326576
64+
Alaska,Economic Adjustment Assistance,2017,1109263
65+
Alaska,Economic Adjustment Assistance,2020,2799716
66+
Alaska,Economic Adjustment Assistance,2021,1000000
67+
Alaska,Economic Adjustment Assistance,2022,39057524
68+
Alaska,Economic Adjustment Assistance,2023,2682734
69+
Alaska,Economic Adjustment Assistance,2024,8384059
70+
Alaska,Economic Adjustment Assistance,2025,1059252
71+
Alaska,Economic Adjustment Assistance,2026,499874
72+
Alaska,Planning,2012,1742500
73+
Alaska,Planning,2013,140000
74+
Alaska,Planning,2014,159800
75+
Alaska,Planning,2015,1138600
76+
Alaska,Planning,2016,250000
77+
Alaska,Planning,2017,2942643
78+
Alaska,Planning,2018,180000
79+
Alaska,Planning,2019,60000
80+
Alaska,Planning,2020,210000
81+
Alaska,Planning,2021,989150
82+
Alaska,Planning,2022,297500
83+
Alaska,Planning,2023,210000
84+
Alaska,Planning,2024,1140000
85+
Alaska,Planning,2025,145000
86+
Alaska,Public Works,2012,2800000
87+
Alaska,Public Works,2014,2315584
88+
Alaska,Public Works,2018,519418
89+
Alaska,Public Works,2019,2144677
90+
Alaska,Public Works,2021,2028704
91+
Alaska,Public Works,2024,3061140
92+
Alaska,Regional Innovation Strategies,2020,591630
93+
Alaska,Technical Assistance,2012,625000
94+
Alaska,Technical Assistance,2013,48155
95+
Alaska,Technical Assistance,2015,498987
96+
Alaska,Technical Assistance,2016,205801
97+
Alaska,Technical Assistance,2017,896000
98+
Alaska,Technical Assistance,2019,77100
99+
Alaska,Technical Assistance,2021,599053
100+
Alaska,Technical Assistance,2022,405000
101+
Alaska,Technical Assistance,2023,90000

0 commit comments

Comments
 (0)