Enhancement: Add Provenance Information to CKAN DataRegistration

[mosoriob]

Enhancement: Add Provenance Information to CKAN DataRegistration

Summary

Enhance CKAN DataRegistration to include comprehensive provenance information for registered outputs, such as region, inputs, parameters, and execution links. This requires defining a provenance schema for the CKAN instance.

Problem Statement

Currently, outputs registered in CKAN lack crucial provenance information that would help users understand:

• How the output was generated (execution workflow)
• What region the data covers
• Which input datasets were used
• What parameters were applied
• Link to the original execution for reproducibility

Additionally, the CKAN instance doesn't have a defined schema for provenance, making it necessary to establish standardized metadata fields before implementation.

This missing context makes it difficult for users to:

• Understand the data generation process
• Assess data quality and reliability
• Reproduce or build upon existing results
• Trace data lineage for scientific reproducibility

Proposed Enhancement

Phase 0: CKAN Schema Definition

CRITICAL: Define and implement a provenance schema in the CKAN instance before proceeding with integration.

CKAN Schema Tasks

• Define provenance metadata schema for CKAN instance
• Create custom CKAN fields for provenance information
• Implement schema validation in CKAN configuration
• Document schema specification for team reference
• Test schema deployment on CKAN instance

Proposed CKAN Schema Structure

{
    "provenance_fields": [
        {
            "field_name": "execution_id",
            "label": "Execution ID",
            "form_snippet": "text.html",
            "display_snippet": "text.html",
            "validators": "ignore_missing unicode_safe",
            "required": true
        },
        {
            "field_name": "execution_url",
            "label": "Execution URL",
            "form_snippet": "text.html",
            "display_snippet": "link.html",
            "validators": "ignore_missing unicode_safe url_validator"
        },
        {
            "field_name": "model_info",
            "label": "Model Information",
            "form_snippet": "textarea.html",
            "display_snippet": "text.html",
            "validators": "ignore_missing unicode_safe json_validator"
        },
        {
            "field_name": "region_info",
            "label": "Geographic Region",
            "form_snippet": "textarea.html",
            "display_snippet": "text.html",
            "validators": "ignore_missing unicode_safe json_validator"
        },
        {
            "field_name": "input_datasets",
            "label": "Input Datasets",
            "form_snippet": "textarea.html",
            "display_snippet": "text.html",
            "validators": "ignore_missing unicode_safe json_validator"
        },
        {
            "field_name": "parameters",
            "label": "Model Parameters",
            "form_snippet": "textarea.html",
            "display_snippet": "text.html",
            "validators": "ignore_missing unicode_safe json_validator"
        },
        {
            "field_name": "time_range",
            "label": "Temporal Coverage",
            "form_snippet": "textarea.html",
            "display_snippet": "text.html",
            "validators": "ignore_missing unicode_safe json_validator"
        }
    ]
}

Provenance Metadata Structure

interface DatasetProvenance {
    // Execution Information
    executionId: string;
    executionUrl: string;
    executionTimestamp: string;
    executionStatus: string;
    
    // Model Information
    modelId: string;
    modelName: string;
    modelVersion: string;
    
    // Geographic Information
    region: {
        name: string;
        boundingBox?: BoundingBox;
        coordinates?: GeoJSON;
    };
    
    // Input Datasets
    inputDatasets: Array<{
        datasetId: string;
        datasetName: string;
        datasetUrl: string;
        role: string; // 'forcing', 'calibration', 'validation', etc.
    }>;
    
    // Parameters
    parameters: Record<string, any>;
    
    // Temporal Information
    timeRange: {
        startDate: string;
        endDate: string;
        temporalResolution?: string;
    };
    
    // Workflow Information
    problemStatementId: string;
    taskId: string;
    subtaskId: string;
}

CKAN Dataset with Provenance

{
    "name": "output-dataset-name",
    "title": "Model Output Dataset",
    "notes": "Dataset description...",
    "execution_id": "exec-uuid-123",
    "execution_url": "https://ensemble-manager.mint.edu/executions/exec-uuid-123",
    "model_info": "{\"id\": \"model-uuid-456\", \"name\": \"PIHM\", \"version\": \"1.0\"}",
    "region_info": "{\"name\": \"South Sudan\", \"bbox\": [...]}",
    "input_datasets": "[{\"id\": \"input-1\", \"name\": \"Precipitation Data\", \"role\": \"forcing\"}]",
    "parameters": "{\"calibration_method\": \"auto\", \"time_step\": \"daily\"}",
    "time_range": "{\"start\": \"2020-01-01\", \"end\": \"2020-12-31\"}"
}

Implementation Plan

Phase 0: CKAN Schema Setup (NEW)

• Research CKAN schema extension mechanisms
• Design provenance field definitions
• Create CKAN schema configuration
• Deploy schema to CKAN instance
• Validate schema functionality
• Create schema documentation

Phase 1: Data Collection

• Identify all provenance data sources in execution workflow
• Map execution data to provenance metadata structure
• Design provenance data aggregation service
• Create provenance metadata builder

Phase 2: CKAN Integration

• Enhance CKAN DataRegistration service to include provenance
• Update dataset creation to use new provenance fields
• Add provenance validation and error handling
• Create provenance metadata transformation utilities

Phase 3: Documentation & Testing

• Document provenance metadata schema
• Create examples and best practices guide
• Add integration tests for provenance registration
• Performance testing for enhanced registration

Technical Implementation

CKAN Schema Configuration

# ckan_schema_extension.py
import ckan.plugins as plugins
import ckan.plugins.toolkit as toolkit

class ProvenanceSchemaPlugin(plugins.SingletonPlugin):
    plugins.implements(plugins.IDatasetForm)
    
    def create_package_schema(self):
        schema = super().create_package_schema()
        schema.update({
            'execution_id': [toolkit.get_validator('ignore_missing'),
                           toolkit.get_validator('unicode_safe')],
            'execution_url': [toolkit.get_validator('ignore_missing'),
                            toolkit.get_validator('unicode_safe'),
                            toolkit.get_validator('url_validator')],
            'model_info': [toolkit.get_validator('ignore_missing'),
                         toolkit.get_validator('unicode_safe'),
                         toolkit.get_validator('json_validator')],
            # ... other provenance fields
        })
        return schema

Service Layer Changes

// Enhance DataRegistrationService
class DataRegistrationService {
    async registerDatasetWithProvenance(
        datasetInfo: DatasetInfo,
        executionContext: ExecutionContext
    ): Promise<CKANDataset> {
        const provenance = await this.buildProvenanceMetadata(executionContext);
        const enhancedDataset = this.embedProvenance(datasetInfo, provenance);
        return await this.ckanClient.createDataset(enhancedDataset);
    }
    
    private async buildProvenanceMetadata(context: ExecutionContext): Promise<DatasetProvenance> {
        // Aggregate provenance from execution, model, inputs, parameters
    }
    
    private embedProvenance(dataset: DatasetInfo, provenance: DatasetProvenance): CKANDataset {
        // Convert provenance to CKAN schema fields
    }
}

Benefits

For Users

• Better Understanding: Clear context about how data was generated
• Reproducibility: Ability to trace and reproduce results
• Quality Assessment: Understanding of model inputs and parameters
• Discovery: Enhanced search and filtering capabilities

For Science

• Transparency: Open science principles with full provenance
• Validation: Ability to verify and validate results
• Collaboration: Better sharing of methodology and results
• Compliance: Meet data management plan requirements

Files to Modify

• CKAN instance schema configuration (NEW)
• src/classes/mint/DataRegistrationService.ts
• src/classes/graphql/queries/executions.ts
• src/api/api-v1/services/ (execution services)
• CKAN integration utilities

Dependencies

• CKAN schema extension capability
• CKAN instance administrative access
• Schema validation framework
• Existing execution and model data

Related Issues

• Model output registration workflow
• Execution result management
• CKAN dataset metadata standards
• Data lineage tracking

Acceptance Criteria

• CKAN instance has defined provenance schema (CRITICAL)
• Schema fields are properly validated and displayed
• Registered datasets include comprehensive provenance metadata
• Provenance information is accurately captured from executions
• UI displays provenance information clearly
• Users can navigate from datasets back to executions
• Provenance enables dataset discovery and filtering
• Performance impact is minimal on registration process