Skip to content

[IndiaNSS_HealthAilments] Made changes in the download script and pvmap , added goldens checks - #2170

Open
Krishnam24maheshwari wants to merge 4 commits into
datacommonsorg:masterfrom
Krishnam24maheshwari:indianss
Open

[IndiaNSS_HealthAilments] Made changes in the download script and pvmap , added goldens checks#2170
Krishnam24maheshwari wants to merge 4 commits into
datacommonsorg:masterfrom
Krishnam24maheshwari:indianss

Conversation

@Krishnam24maheshwari

@Krishnam24maheshwari Krishnam24maheshwari commented Aug 20, 2026

Copy link
Copy Markdown

This PR resolves multiple pipeline and data quality issues for the IndiaNSS_HealthAilments import :

  1. Restores missing "60+" senior demographic mapping in pvmap.csv that previously caused 992 lint errors, 51 dropped StatVars, and demographic attribution corruption.
  2. Cleans up download_script.py and fixes error handling by removing dead duplicate code, enforcing deterministic script-relative output paths, and handling retry exhaustion gracefully with explicit RuntimeError.
  3. Synchronizes and expands download_script_test.py to achieve full mock unit test coverage.
  4. Added goldens rules (golden_observations.csv) goldens_summary_report is missing because non of the statvars generated in this import is there in "gs://unresolved_mcf/import_validation/nl_statvars.csv".
  5. Aligns manifest.json and README.md with standard naming conventions and output counters.

Differ summary to verify data - https://pantheon.corp.google.com/storage/browser/_details/datcom-import-test/statvar_imports/india_ndap/india_nss_health_ailments/IndiaNSS_HealthAilments/2026_08_20T06_39_52_485565_07_00/input0/validation/differ_summary.json;tab=live_object?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&project=datcom-infosys-dev

Validation_output - https://pantheon.corp.google.com/storage/browser/_details/datcom-import-test/statvar_imports/india_ndap/india_nss_health_ailments/IndiaNSS_HealthAilments/2026_08_20T06_39_52_485565_07_00/input0/validation/validation_output.csv;tab=live_object?pageState=(%22StorageObjectListTable%22:(%22f%22:%22%255B%255D%22))&project=datcom-infosys-dev

@google-cla

google-cla Bot commented Aug 20, 2026

Copy link
Copy Markdown

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request cleans up redundant code in download_script.py, updates the property-value mapping for age groups in india_nss_health_ailments_pvmap.csv, and updates the output path in manifest.json to use PascalCase. Additionally, it introduces a validation configuration (validation_config.json) and a golden observations dataset (golden_observations.csv) to verify output data integrity. There are no review comments, so I have no feedback to provide.

@Krishnam24maheshwari Krishnam24maheshwari changed the title Made changes in the download script and pvmap , added goldens checks … [IndiaNSS_HealthAilments] Made changes in the download script and pvmap , added goldens checks Aug 20, 2026
@saanikaaa
saanikaaa self-requested a review August 20, 2026 12:15

@Krishnam24maheshwari Krishnam24maheshwari left a comment

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Made changes accordingly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants