Generic Drought-map Hub

This repository contains the core components and automation scripts for a Drought-map Hub, a platform designed to provide streamlined, user-friendly, and validated Combined Drought Indicator (CDI) products. It enhances drought monitoring capabilities by automating data management and promoting public access to critical information.

1. Overview and Purpose

The Drought-map Hub is a collaborative platform focused on validating, publishing, and exploring Combined Drought Indicator (CDI) products. It aims to enhance a country's drought monitoring capabilities through efficient data management and public accessibility.

The platform is structured into two main areas:

• Admin Area: Dedicated to authorized users (administrators and reviewers) for reviewing, validating, discussing, and approving CDI outputs before public release. This ensures high-quality, reliable information is made available.
• Public Area: Provides open and intuitive access to comprehensive drought-related datasets, allowing users to explore trends and patterns for research, planning, and decision-making.

The Combined Drought Indicator (CDI) is a scientifically robust indicator that combines multiple drought-relevant datasets into a single map reflecting overall drought severity. These datasets typically include:

• Land Surface Temperature (LST): Measures heat stress and temperature anomalies.
• Normalized Difference Vegetation Index (NDVI): Captures vegetation health and greenness.
• Precipitation (e.g., Standardized Precipitation Index or CHIRPS): Assesses rainfall deficits.
• Soil Moisture (SM): Reflects water availability for crops and plants.

The CDI is computed using a weighted average system, which can be adjusted based on regional characteristics or data quality.

2. Key Features

The Drought-map Hub offers a range of functionalities for efficient drought monitoring:

• Automated CDI Generation: The platform automatically computes CDI monthly using external datasets and scripts from the National Drought Mitigation Center (NDMC).
• Structured Data Processing and Storage: CDI data, including GeoTIFF raster files and aggregated GeoJSON vector files for administrative regions, are processed and stored on a GeoNode platform.
• Web Application for Collaboration and Publication: A web interface allows authorized users to review, validate, and discuss CDI maps before their public release.
• Role-Based User Access Control: The platform supports three user types:
  • Visitors: Read-only access to published CDI maps and public features.
  • Reviewers: Access to review and provide feedback on computed CDI maps prior to publication.
  • Administrators: Full access to manage the platform, users, and data publication processes.
• Interactive Data Visualization: Users can interact with published CDI maps, zoom, pan, and view CDI values for different administrative regions.
• Historical Map Comparison: Allows visitors to compare two published maps from different time periods to observe changes in CDI values.
• Map Download Options: Published maps are available for download in formats like GeoJSON, shapefile, PNG, and SVG. Reviewers can also access non-aggregated CDI Raster Maps for context.
• Feedback Mechanism: A public feedback form enables users to submit comments or report bugs directly to administrators.

3. System Architecture and Interdependencies

The Drought-map Hub's high-level architecture integrates several components, external services, and software dependencies to automate and manage CDI products.

• Front-end (Next.js):

  • Role: User-facing interface with dynamic routing and server-side rendering (SSR) for performance and SEO.
  • Interdependencies: Communicates with the Back-end via RESTful APIs, exchanging JSON data. It consumes TopoJSON data from a config.js file generated by the backend. Styling is handled by Tailwind CSS and Ant Design, and interactive maps use Leaflet.

• Back-end (Django):

  • Role: Manages user authentication, data validation workflows, and publishing processes. It acts as an intermediary between the front-end and the GeoNode API for dataset management.
  • Interdependencies:
    • Interacts with the Database (PostgreSQL) using Django’s ORM.
    • Utilizes Django Rest Framework (DRF) for Web APIs and PyJWT for secure user authentication.
    • Integrates with Mailjet for email notifications (e.g., user invitations, password resets, review requests, feedback).
    • Uses Pandas and GeoPandas for data processing and analysis of geospatial data.
    • Communicates with Rundeck to trigger and manage CDI automation jobs.
    • Employs Worker/DjangoQ for background processes like monitoring automation and downloading maps.

• Database (PostgreSQL):

  • Role: Stores application data, including user details, administrative regions, publication statuses, reviews, and job logs. GeoNode maintains its own external database accessed via its API.

• CDI Automation with Rundeck:

  • Role: Orchestrates the automated monthly process for generating and updating drought data. This includes checking external datasets (CHIRPS/Precipitation, NDVI, LST, Soil Moisture), running NDMC scripts, and uploading outputs to GeoNode. It also manages the VM hosting the automation server (turning it on/off).
  • Interdependencies: Triggers an endpoint in the DroughtMap Hub to create a new Publication Process object upon success. Sends email notifications via Mailjet for job status (success or error). Requires NASA Earthdata credentials for some dataset downloads (LST, NDVI, SM).

• GeoNode (Spatial Data Infrastructure):

  • Role: Central repository for spatial data, storing CDI raster data and aggregation results (GeoJSON files for administrative regions). It manages categories (e.g., cdi-raster-map) for uploaded datasets.
  • Interdependencies: The DroughtMap Hub's back-end interacts with GeoNode via its API for data retrieval and upload.

• Container Orchestration (Docker Compose):

  • Role: Manages and orchestrates containers for consistent deployment across development and production environments.
  • Interdependencies: Defines how backend, frontend, and database services are deployed and interact within their containers.

• CI/CD Pipeline (GitHub Workflows):

  • Role: Automates continuous integration and deployment, ensuring code quality through automated tests and triggering builds and deployments to appropriate environments upon success.

• Jobs (DjangoQ):

  • Role: Manages job queues and asynchronous task execution within the Drought-map Hub, handling background job processing and schedule management.
  • Interdependencies: Schedules tasks like checking automation status and downloading maps, monitors their progress, and notifies users of outcomes.

4. CDI Automation Workflow

The CDI automation script (job.sh) is designed to run periodically (ideally monthly) to keep drought-related data up-to-date. This automated process, orchestrated by Rundeck, involves several key steps:

1. Check External Datasets: The system verifies the availability of the latest datasets from global sources, including CHIRPS/Precipitation, LST, NDVI, and Soil Moisture. Some datasets require NASA Earthdata login credentials.
2. Check Data is Up-to-Date: It compares the number of available external dataset entries with the number of downloaded input data. If discrepancies are found, missing data is identified and downloaded.
3. Run NDMC Scripts (CDI Scripts): Once all necessary datasets for the previous month are updated, the CDI script is executed to produce the CDI Map output, typically as GeoTIFF raster files.
4. Run Aggregation Script: The raw CDI raster output is processed by an aggregation script. This script filters out no-data pixels, creates percentile values, and generates a median percentile value and a drought category for each polygon/administrative boundary as defined in a GeoJSON boundary file.
5. Upload Outputs to GeoNode: All CDI outputs (both raster and GeoJSON) are uploaded to the GeoNode instance via its authenticated API. The system tracks the asynchronous upload progress and updates metadata to set specific categories (e.g., cdi-raster-map, spi-raster-map) for each output.
6. Send Success Message: Upon successful upload, a notification is sent to Drought-map Hub administrators, and an endpoint in the Drought-map Hub is triggered to create a new Publication Process object, initiating the review and validation workflow.

5. Starting a New Instance of the Drought-map Hub (DMH) for a New Client

To deploy a new instance of the DMH for a client, follow these steps:

1. Clone the Repository: Obtain the DMH repository and navigate into its directory.
2. Run Installation Script: Execute the install.sh script provided within the repository.
3. Configure Domain Names:
   • Set the GeoNode web domain by configuring WEBDOMAIN_GEONODE (default: http://localhost).
   • Set the DMH web domain by configuring WEBDOMAIN (default: http://localhost:3000).
4. Configure Client-Specifics:
   • Configure the DMH for the Technical Working Group (TWG) list.
   • Configure CDI by setting the Country Name and Boundaries.
5. Set Earthdata Credentials: Configure the NASA Earthdata credentials (username, password) required for downloading certain datasets. These variables are typically defined in the .env file for the CDI Automation repository.
6. Confirm Deployment Environment: Confirm the deployment environment (e.g., --mode=development and --service=all are defaults).

The install.sh script will automatically perform the following:

• Save all configuration data to their respective environment services.
• Start the GeoNode and DMH containers using Docker Compose.
• Seed default administrator users for both GeoNode and DMH.
• Generate a RUNDECK_TOKEN using the CLI.
• Create a new Rundeck project and import the Job template via the API.
• Seed the Rundeck integration into the DMH.
• Wait until all containers are ready and accessible.

Manual Configurations Post-Deployment

After the initial script execution, some manual configurations are required:

1. GeoNode Configuration:
   • Log in to the GeoNode Admin panel.
   • Enable "Unrestricted XML External Entity Resolution" in GeoServer.
   • Add four CDI categories: cdi-raster-map, spi-raster-map, ndvi-raster-map, and lst-raster-map.
   • Set GeoNode base URL, admin username, and password in the Droughtmap Hub project's environment variables.
2. DMH Configuration:
   • Log in to the DMH using the default administrator credentials.
   • Navigate to the settings page.
   • If Rundeck integration is successful, configure the email for CDI notifications.
   • If Rundeck integration fails, create a new Rundeck connection with the available projects and jobs, then configure email notifications.
3. Boundary Data Configuration:
   • The geospatial boundary file, typically named [country].topojson (e.g., eswatini.topojson), is used for map visualization and processing.
   • This file is stored in the backend/source folder of the Drought-map Hub repository.
   • To apply or update this file, convert the shapefile to TopoJSON format (e.g., using https://mapshaper.org) and ensure it contains necessary properties like name, region, and administration_id.
   • After placing the .topojson file, execute python manage.py generate_administrations_seeder and python manage.py generate_config commands within the backend container to update the system and generate the frontend config.js file.
4. Restart the Drought-map Hub to apply any updated environment variables.

The DMH configuration is now complete and ready for use.