Predicting Dementia Based on Speech Using GPT-3 Text Embeddings

Dementia is the loss of cognitive function, which affects how a person can use language and communicate. Early detection and intervention are crucial for improving the quality of life for affected individuals. This project aims to harness the power of natural language processing and deep learning to diagnose dementia at an early stage based on speech patterns. By utilizing GPT-3 text embeddings, we seek to develop an innovative tool that can assist in the early detection of dementia.
Explore the docs »

Report Bug · Request Feature

Table of Contents

1. About The Project
   • Built With
2. Getting Started
   • Prerequisites
   • Installation
3. Usage
   • Config
   • Transcription
   • Embedding
   • Classification
4. Results
   • Run Configuration
   • Performance
   • Selected Plots
5. License
6. Acknowledgments

About The Project

This project was created during my position as a research assistant at Karlsruhe Institute of Technology (KIT). It involves several steps:

• Transcription using Whisper
• Creation of GPT-3 text embeddings to leverage contextual depth from the transcribed text
• Training supervised machine learning classifiers using the extracted embedding features and diagnosis labels
• Prediction on test data

Built With

(back to top)

Getting Started

To get a local copy up and running follow these steps.

Prerequisites

To gain access to the data used in this project, you must first join as a DementiaBank member.

For simplicity, we used the dataset provided for the ADReSSo-challenge which has been balanced with respect to age and gender in order to eliminate potential confounding and bias.

In our case, the downloaded ADReSSo audio files had an incompatible format to transcribe them with Whisper. Therefore, we first had to format them with ffmpeg. Since we cannot reformat and replace the files at the same time, we have to save them temporarily and then replace the old files:

find . -name '*.wav' -exec sh -c 'mkdir -p fix && ffmpeg -i "$0" "fix/$(basename "$0")"' {} \;

Now replace the original audio files with the formatted files in the fix folder, which can then be deleted.

Installation

1. Get an OpenAI API Key at https://platform.openai.com/account/api-keys.
2. Set an environment variable OPENAI_API_KEY (replace ~/.zshrc with ~/.bashrc if you use Bash):

   echo "export OPENAI\_API\_KEY='your key'" | cat >> ~/.zshrc

3. Clone the repo:

   git clone https://github.com/probstlukas/gpt3-dementia-detection.git

4. Install required Python packages:

   pip install -r requirements.txt

(back to top)

Usage

Config

Before running the program, make sure that the default settings (logging level, embedding engine, directory paths, ...) in config.py suit you. Then start the program by running main.py.

Transcription

If the data has not yet been transcribed, confirm with yes, select your preferred Whisper model and wait until the transcription process is complete.

Embedding

The embeddings are created separately for training and test data. train_embeddings.csv also contains the MMSE score and the diagnosis label for each audio file.

The embedding step uses only the transcript text as input to the embedding model. This keeps the feature extraction step independent from the target labels and avoids label leakage: the embedding should represent what is present in the transcript, not the known diagnosis.

Diagnosis labels are carried forward with the training rows and used later as supervised classification targets. MMSE scores are kept as metadata, but they are not used to create embeddings or train the current classifiers.

Remark: It is not necessary to scale the embeddings before using them. They are already normalized and are in the vector space with a certain distribution.

Classification

In this step, machine learning models are trained and evaluated using the provided embeddings. This is a supervised classification step: embeddings are the input features, and the diagnosis labels from the training set are the targets.

The pipeline does not perform clustering, and it does not classify samples by direct cosine-similarity or nearest-neighbor comparison between embeddings. The test labels are only used after prediction to compute the final evaluation metrics. For comparison with other, more complex classifiers, we create a dummy classifier that makes predictions that ignore the input features. This gives us a baseline performance (like flipping a coin, i.e. about 50%).

We use three different models: Support Vector Machine, Logistic Regression, and Random Forest.

The classification process can be divided into two parts:

Model checking

• Perform nested stratified K-fold cross-validation on the training set.
• Tune hyperparameters on inner folds and report validation metrics from outer folds.
• Record model performance (accuracy, precision, recall, f1-score)
• Visualize results: one plot for each metric per model, resulting in a total of 12 plots.

Model building

• Train each model on the entire training set with the best hyperparameters.
• Predict labels on the test data using the trained model.
• Evaluate performance by comparing the results to real medical diagnoses.
• Record trained model sizes.

The classification step writes its outputs to results/embedding/, including:

• embedding_results.csv with train, validation, and test metrics.
• embedding_models_size.csv with serialized model sizes.
• task1_*.csv files with test predictions for each classifier.
• plots/*.png with fold-wise training vs. validation plots for each metric and model.

(back to top)

Results

Run Configuration

The following results and plots were obtained with this run configuration:

• Whisper model: base
• Embedding engine: text-embedding-ada-002
• Outer cross-validation: nested stratified K=10
• Inner cross-validation for hyperparameter search: K=5
• Evaluated classifiers: SVC, LogisticRegression, RandomForestClassifier

Performance

Model	Size
SVC	1625787 B
LR	13010 B
RF	175183 B
Total	1813980 B

Set	Model	Accuracy	Precision	Recall	F1
Train	SVC	0.918 (0.031)	0.923 (0.028)	0.918 (0.031)	0.918 (0.031)
Validation	SVC	0.771 (0.107)	0.785 (0.099)	0.771 (0.107)	0.766 (0.112)
Test	SVC	0.817	0.823	0.817	0.816
Train	LR	0.959 (0.034)	0.959 (0.034)	0.959 (0.034)	0.958 (0.034)
Validation	LR	0.802 (0.095)	0.809 (0.098)	0.802 (0.095)	0.8 (0.095)
Test	LR	0.845	0.845	0.845	0.845
Train	RF	0.985 (0.026)	0.985 (0.026)	0.985 (0.026)	0.985 (0.026)
Validation	RF	0.728 (0.074)	0.74 (0.078)	0.728 (0.074)	0.724 (0.073)
Test	RF	0.746	0.747	0.746	0.746
Train	Dummy	0.517 (0.011)	0.514 (0.011)	0.517 (0.011)	0.512 (0.011)
Validation	Dummy	0.417 (0.068)	0.417 (0.07)	0.417 (0.068)	0.415 (0.069)

The metrics show a consistent ranking across validation and held-out test data:

• Logistic Regression performs best overall with the strongest validation accuracy (0.802) and test accuracy (0.845).
• SVC is the second-best model and generalizes better than Random Forest on the held-out test set.
• Random Forest reaches the highest training scores but also shows the largest training-validation gap, which is visible in the plots and points to overfitting.

Selected Plots

The plots below highlight the clearest takeaways from the documented run:

• Logistic Regression accuracy: best overall validation and test performance.
• Logistic Regression F1: strongest balance between precision and recall across folds.
• Random Forest accuracy: clearest sign of overfitting, with training scores near 1.0 and weaker validation scores.

These results show that GPT-3 text embeddings can be used to distinguish individuals with Alzheimer's disease from healthy individuals in the control group based on their speech behavior, while model selection and validation strategy have a strong impact on the reported performance.

License

Distributed under the MIT License. See LICENSE.txt for more information.

(back to top)

Acknowledgments

• DementiaBank
• ADReSSo-Challenge
• "Predicting dementia from spontaneous speech using large language models" by Felix Agbavor and Hualou Liang from Drexel University

(back to top)