Add local llm option

README.md

@@ -2,51 +2,159 @@
 
 ## Overview
 
-The voice-activated assistant is a Python application that listens for specific audio cues and performs actions based on those cues.
-It uses a combination of audio recording, transcription, and action handling to provide a voice-activated interface for various tasks.
+This voice-activated assistant is a Python application designed to listen for specific audio cues and perform corresponding actions. It combines audio recording, transcription, and action execution to provide a voice-activated interface for various tasks. The assistant uses a combination of local speech-to-text processing and customizable actions, integrated with external services for enhanced functionality.
 
 ## Features
 
-- Wake phrase detection to start listening for commands.
-- Transcription of audio to text for command recognition.
-- Customizable actions that can be triggered by voice commands.
-- Integration with external services for advanced functionalities (e.g., LLM, transcription services).
+- **Wake Phrase Detection**: Activates listening mode upon hearing a specified phrase.
+- **Audio Transcription**: Converts spoken words into text using a speech-to-text model.
+- **Action Handling**: Executes customizable actions based on transcribed text.
+- **External Service Integration**: Can be integrated with APIs for additional functionalities such as text generation.
 
 ## Requirements
 
-- Python 3.12+ (Rye will install this for you)
-- Rye for python package management
+- Rye package manager for Python projects
+  - Python 3.12 will be installed by Rye when setting up the project
+- FFMPEG for audio processing
+- Access to text generation tool:
+  1. Ollama server, recommended for local LLM
+     - requires a server running on your local machine
+  2. HuggingFace's Transformers library
+     - works out of the box
+     - runs slower than Ollama
+  3. OpenAI API key, most powerful
+     - requires an internet connection
+     - incurs costs
+     - may have privacy implications
+- Microphone access on your device
+
+## Installation
+
+1. **Install [Rye](https://rye-up.com/guide/installation/)**:
+    ```bash
+    curl -sSf https://rye-up.com/get | bash
+    ```
+2. **Install [FFMPEG](https://ffmpeg.org/)**:
+
+    FFmpeg is required for processing audio files.
+    Install it using your system's package manager or download the binaries:
+    <details>
+
+      <summary>On Ubuntu/Debian:</summary>
+
+      ```bash
+      sudo apt update
+      sudo apt install ffmpeg
+      ```
+    </details>
+    <details>
+      <summary>On macOS:</summary>
+
+      ```bash
+      brew install ffmpeg
+      ```
+    </details>
+    <details>
+      <summary>On Windows:</summary>
+
+    - Download the binaries from [FFmpeg's official website](https://ffmpeg.org/download.html) and add the path to the executable to your system's PATH variable.
+    </details>
+3. **Set Up Project**:
+    Navigate to your project directory and sync the project using Rye:
+    ```bash
+    rye sync
+    ```
+4. **Create Your Configure Files**:
+
+    Copy the example configuration files to use them as a starting point.
+    ```bash
+    cp example_actions_config.yml actions_config.yml
+    cp example_settings_config.yml settings_config.yml
+    ```
+    You should look at the exaple tools provided. You can delete these and make your own as needed.
+
+5. **Set up OpenAI credentials or use a local LLM**
+    <details>
+      <summary>Use a local LLM (recommended):</summary>
+
+      You can use a local language model (LLM) for text generation instead of the OpenAI API.
+      This can be more cost-effective and provide better privacy.
+      We enable to ways to de this:
+      1. Use Ollama to host a local LLM server.
+      2. Use Hugging Face's Transformers library to directly load your local LLM at voice-assistant start-up.
+
+      ### Update the `settings_config.yml` file.
+      - `settings_config.yml` file to use a local LLM for text generation.
+      If you are using Ollama, set the `OLLAMA_SERVER_URL` environment variable to the URL of your Ollama server.
+      Otherwise, set the `HUGGINGFACE_MODEL_ID` environment variable to the model ID of the LLM you want to use.
+
+      ### 1. Set up Ollama
+      1. Go to [Ollama](https://ollama.com/) and download the server for your operating system.
+      2. Follow the instructions to set up the server and get the server URL.
+      - If the voice assistant is unable to connect to the server, you may need to adjust the Ollama CORS settings.
+      - In general setting `OLLAMA_ORIGINS "*"` will allow access from any origin and fixes the problem.
+        - This is probably fine locally but is not recommended for production environments.
+
+      ```yaml
+      llm_config:
+        llm_id: "llama3"
+        llm_type: "ollama"
+        server_url: "http://localhost:11434/v1"
+      ```
+
+      ### 2. Use HuggingFace's Transformers
+      You can use Hugging Face's Transformers library to load a local LLM. You can find the model ID for the LLM you want to use on the Hugging Face Model Hub.
+      You can also use the `transformers` library to load a local model from a file path.
+      ```yaml
+      llm_config:
+        llm_id: "unsloth/llama-3-8b-Instruct-bnb-4bit"
+        llm_type: "huggingface"
+      ```
+    </details>
+
+    <details open>
+      <summary>Use OpenAI for your LLM:</summary>
+
+      Sign up for api access at [OpenAI](https://openai.com/index/openai-api) and follow instructions to create an api key.
+      If for non-personal, make sure this repo adheres to all required policies, then get an API key from an Admin in your Organization (as applicable).
+
+      ### Update the `settings_config.yml` file.
+      - `settings_config.yml` file to use the OpenAI API for text generation.
+      Set the `OPENAI_API_KEY` environment variable to your OpenAI API key.
+      ```yaml
+      llm_config:
+        llm_id: "gpt-4-turbo"
+        llm_type: "openai"
+      ```
+
+      ### Set API key on macOS
+
+      1. Append your API key to your `.bashrc` or `.zshrc` file:
+      ```bash
+      echo 'export OPENAI_API_KEY="your_openai_api_key"' >> ~bashrc
+      ```
+      2. Source the file to update your environment variables:
+      ```bash
+      source ~/.bashrc
+      ```
+
+      ### Set API key on Windows
+
+      1. Open a new Command Prompt as Administrator.
+
+      2. Set your API key as an environment variable using the `setx` command. Replace `"your_openai_api_key"` with your actual OpenAI API key:
+      ```cmd
+      setx OPENAI_API_KEY "your_openai_api_key"
+      ```
+      3. Close the Command Prompt. The changes will take effect when you open a new Command Prompt.
+
+    </details>
 
-```bash
-curl -sSf https://rye-up.com/get | bash
-```
-
-## Setup
-
-1. [Install Rye](https://rye-up.com/guide/installation/)
-2. Initialize the project and install dependencies via `rye sync`
-
-```bash
-rye sync
-```
-
-3. Ensure you have the necessary credentials and settings for any external services used by the actions (e.g., LLM, transcription services).
-    - For now the app assumes you have a valid OpenAI API key set as an environment variable. You can add this to your .bashrc or .zshrc (or equivalent) to make it permanent.
-
-```bash
-export OPENAI_API_KEY="your_openai_api_key"
-```
-
-4. Copy the example_actions_config.yml and settings files. Add, remove, or modify actions as needed.
-
-```bash
-cp example_actions_config.yml actions_config.yml
-cp example_settings_config.yml settings_config.yml
-
-```
 
 ## Usage
 
+After setting up the configuration file and environment variables, you can run your application.
+
 To start the voice-activated assistant, run EITHER of the following command in your terminal:
 
 ```bash
@@ -83,15 +191,7 @@ Updates to the action config will create new instances of your action classes an
 
 To adjust the application's behavior, modify the `settings_config.yml` file. For example, you can change the model ID, enable or disable copying to the clipboard, adjust the maximum audio length, and more.
 
-## Logging
-
-The application uses `loguru` for logging.
-You can adjust the logging level by modifying the `logger_init` function call in the script.
-
-## Graceful Shutdown
-
-The application handles `SIGTERM` and `SIGINT` signals to ensure a graceful shutdown when the process is terminated.
-
 ## License
 
 Please ensure you comply with the licenses of all components and dependencies used within this application.
+

example_settings_config.yml

@@ -1,39 +1,27 @@
-# The ID of the model to use for generating responses.
-# In this case, it's set to "gpt-4-turbo", which is a powerful language model developed by OpenAI.
-MODEL_ID: "gpt-4-turbo"
-
-# Whether to automatically copy the generated responses to the clipboard.
-# If this is set to true, you can easily paste the responses into other applications.
-COPY_TO_CLIPBOARD: true
-
-# Whether to extract the first code block from the generated response to your clipboard.
-# If this is set to true, the system will only grab what's in triple backticks (```) and ignore everything else in the response.
-EXTRACT_CODE_BLOCKS: true
-
-# Whether to run the system locally.
-# If this is set to true, the system will run on your local machine. If it's false, it might run on a remote server or in the cloud.
-LOCAL: true
+# Configuration for clipboard operations
+clipboard_config:
+  # Whether to copy the generated responses to your clipboard.
+  copy_to_clipboard: true
+  # Whether to extract the first code block from the generated response to your clipboard.
+  # If this is set to true, the system will only grab what's in triple backticks (```) and ignore everything else in the response.
+  copy_from_code_blocks: true
+  # Whether to paste the generated responses at the cursor position.
+  # If this is set to true, the responses will be inserted at the current cursor position in the active application.
+  paste_at_cursor: true
+
+# Configuration for the language model
+llm_config:
+  # The ID of the language model to use for generating responses.
+  llm_id: "llama3"
+  # The type of the language model. This can be "ollama", "openai", or "huggingface".
+  llm_type: "ollama"
+  # The URL of the server where the language model is hosted for "ollama" models.
+  server_url: "http://localhost:11434/v1"
 
 # The maximum length of audio recordings, in seconds.
 # This setting is used when the system is recording audio input.
-MAX_AUDIO_LENGTH_SECONDS: 3600
-
-# Whether to use text-to-speech (TTS) to read out the generated responses.
-# If this is set to true, the system will use a TTS engine to convert the responses to speech.
-USE_TTS: false
-
-# The speed at which to play back the audio.
-# This is a multiplier, so 1.0 is normal speed, 2.0 is twice as fast, 0.5 is half as fast, etc.
-AUDIO_SPEED: 1.25
-
-# The directory where audio files are stored.
-# This setting is used when the system needs to save or load audio files.
-AUDIO_FILES_DIR: "src/audio_files"
-
-# The directory where prompts for the language model are stored.
-# These prompts are used to guide the language model's responses.
-LLM_ACTION_PROMPTS_DIR: "src/llm-action-prompts"
+max_audio_length_seconds: 3600
 
-# Whether to paste the generated responses at the cursor position.
-# If this is set to true, the responses will be inserted at the current cursor position in the active application.
-PASTE_AT_CURSOR: false
+# The ID of the model to use for speech-to-text conversion.
+# In this case, we're using OpenAI & HuggingFace trained "distil-whisper".
+whisper_id: "distil-whisper/distil-small.en"

pyproject.toml

@@ -4,6 +4,7 @@ version = "0.1.0"
 description = "Add your description here"
 dependencies = [
   "accelerate",
+  "bitsandbytes",
   "huggingface_hub",
   "keyboard",
   "loguru",
@@ -39,6 +40,10 @@ build-backend = "hatchling.build"
 managed = true
 dev-dependencies = ["ruff"]
 
+[[tool.rye.sources]]
+name = "pytorch"
+url = "https://download.pytorch.org/whl/cu121"
+
 [tool.hatch.metadata]
 allow-direct-references = true
 

scripts/print_files.py

@@ -1,5 +1,5 @@
-from typing import Optional
 import sys
+from typing import Optional
 
 
 def print_file_contents(filename):

scripts/speech-to-text-benchmark.py

@@ -12,7 +12,7 @@
 times = []
 for i in range(10):
     start_time = time()
-    stt.transcribe(os.path.join(config.AUDIO_FILES_DIR, "output.mp3"))
+    stt.transcribe(os.path.join(config.audio_dir, "output.mp3"))
     transcribed_time = time() - start_time
     logger.debug(f"transcribed in {transcribed_time:0.2f} seconds")
     times.append(transcribed_time)