TSFM Robustness Benchmark

English | 中文

The TSFM Robustness Benchmark is a systematic testing tool designed to evaluate the engineering robustness of Time Series Foundation Models (TSFMs) in edge cases (e.g., frequency mismatch, data contamination, covariate interference). This release includes a systematic evaluation of TimechoAI as the first targeted model. More models will be integrated in subsequent iterations.

1. Core Architecture - Layered Architecture

• This project is built on Python 3.12, with core dependencies on timecho-ai and pandas.

2. Directory and File Specifications

• config/: Global configuration management module
  • settings.py: Global environment variable configuration (e.g., TIMECHO_API_KEY).
  • constants.py: Global constants definition.
• core/: Business Core Layer (Encapsulates logic and state)
  • results.py: Test Result Manager (Batch buffering/Persistence).
  • resume.py: Strategy Controller (Rate limit handling/Checkpoint resume).
  • timecho.py: API Interaction Wrapper.
• features/: Business feature implementation layer, containing specific business scenario logic
• utils/: Utility Layer (Stateless pure functions)
  • client.py: Low-level Client Connection.
  • data_sanitizer.py: Data Sanitization & Type Safety Utils.
  • files.py: File Operation Utils.
  • metrics.py: Evaluation Metrics Calculator.
• run.py: Unified entry point; configures sys.path and dispatches execution by module name or file path.
• README.md: Project documentation, providing an overview, usage instructions, and notes.

3. Testing Process

1. Configuration initialization: Reads environment variable from config/settings.py.
2. Model initialization: Initializes the TimechoAI model using the provided API key.
3. Testing execution: Executes the specified testing process based on the provided command-line arguments.
4. Result output: Outputs the testing results to the console or specified file.

4. Commands and Installation

• Create virtual environment (universal):
  python -m venv .venv

• Activate the virtual environment (choose the command based on your OS):

  • macOS / Linux: source .venv/bin/activate
  • Windows (CMD): .venv\Scripts\activate.bat
  • Windows (PowerShell): .venv\Scripts\Activate.ps1

• Install dependencies (run this the first time after activating the venv):
  python -m pip install timecho-ai pandas

• Deactivate the virtual environment (universal):
  deactivate

Windows PowerShell users: If you see an error about script execution being disabled, open PowerShell as Administrator and run:
Set-ExecutionPolicy -ExecutionPolicy RemoteSigned -Scope CurrentUser

• Run:
  python run.py features.futureCovs.conceptDrift.concept_drift_test_v1 # Concept Drift Test (Simplified Edition)
  python run.py features.futureCovs.conceptDrift.concept_drift_test_v2 # Concept Drift Test (XYZ Scenario)
  python run.py features.futureCovs.covariant.cov_test # Covariate Effectiveness Test
  python run.py features.futureCovs.covariant.cov_test_models # Covariate Support Test (Iterate All Models)
  python run.py features.futureCovs.dirtyData.dirty_test # Dirty Data Robustness Test (No NaN Support)
  python run.py features.futureCovs.dirtyData.dirty_test_v2 # Dirty Data Robustness Test
  python run.py features.futureCovs.forecastHorizon.forecast_horizon_ablation # C2 Forecast Horizon Ablation Experiment
  python run.py features.futureCovs.freqMismatch.frequency_mismatch_test # C5 Frequency Mismatch Robustness Test
  python run.py features.futureCovs.inputLength.input_length_test # Input_length Ablation Test
  python run.py features.futureCovs.irregularSampling.irregular_sampling_test # Irregular Sampling Robustness Test

5. Testing Objectives

• Edge case exploration: Systematically verify the engineering robustness of the model against boundary conditions such as complex queries, replica inconsistencies, and out-of-order time-series writes.
• Defensive architecture verification: Apply strict engineering standards to test the model's degradation behavior and recovery capabilities under non-ideal inputs.

6. Scope of Testing Disclaimer

The test results of this framework are limited by the specific model version, data preprocessing strategy, and runtime environment. This tool aims to provide an objective reference perspective for the engineering defensive architecture design of time-series models, rather than an absolute assertion of the final performance of any commercial product.