|
| 1 | +## llama.cpp Deployment of MiniCPM (Ubuntu 24.04 + ROCm 7+) |
| 2 | + |
| 3 | +### Model Overview |
| 4 | + |
| 5 | +[MiniCPM](https://github.com/OpenBMB/MiniCPM) is an on-device large language model series developed by ModelBest and Tsinghua University NLP Lab (OpenBMB). MiniCPM5-1B is the latest text-only model in the series, with only 1B parameters and support for Think / No-think reasoning modes. |
| 6 | + |
| 7 | +- Model: [openbmb/MiniCPM5-1B](https://huggingface.co/openbmb/MiniCPM5-1B) |
| 8 | +- GGUF: [openbmb/MiniCPM5-1B-GGUF](https://huggingface.co/openbmb/MiniCPM5-1B-GGUF) |
| 9 | + |
| 10 | +This guide deploys **MiniCPM5-1B Q4_K_M (GGUF)** using **llama.cpp**, covering: |
| 11 | + |
| 12 | +- Prebuilt executables (recommended) |
| 13 | +- Docker + official ROCm image for building from source |
| 14 | + |
| 15 | +MiniCPM is text-only — a single GGUF file with standard `llama-cli` / `llama-server`. For multimodal (image + text), see the `minicpmv/` directory. |
| 16 | + |
| 17 | +> Prerequisite: ROCm 7+ system installation and verification is complete |
| 18 | +> (see `env-prepare-ubuntu24-rocm7.md`). Verified on **AMD Ryzen AI MAX+ 395 (Radeon 8060S, |
| 19 | +> gfx1151), ROCm 7.13**. |
| 20 | +
|
| 21 | +--- |
| 22 | + |
| 23 | +### Method 1 (Recommended): Prebuilt Executables |
| 24 | + |
| 25 | +#### 1. Download the Prebuilt Version |
| 26 | + |
| 27 | +Use the prebuilt llama.cpp provided by Lemonade, where: |
| 28 | + |
| 29 | +- **370** corresponds to the **gfx1150** architecture |
| 30 | +- **395** corresponds to the **gfx1151** architecture |
| 31 | + |
| 32 | +Related links: |
| 33 | + |
| 34 | +- https://github.com/lemonade-sdk/llamacpp-rocm |
| 35 | +- https://github.com/lemonade-sdk/llamacpp-rocm/releases |
| 36 | + |
| 37 | +```bash |
| 38 | +mkdir -p ~/minicpm-rocm && cd ~/minicpm-rocm |
| 39 | +# Pick the asset matching your architecture (gfx1151 shown here) |
| 40 | +curl -L -o llama-rocm-gfx1151.zip \ |
| 41 | + https://github.com/lemonade-sdk/llamacpp-rocm/releases/download/b1292/llama-b1292-ubuntu-rocm-gfx1151-x64.zip |
| 42 | +mkdir -p llama-bin && unzip -q llama-rocm-gfx1151.zip -d llama-bin |
| 43 | +``` |
| 44 | + |
| 45 | +--- |
| 46 | + |
| 47 | +#### 2. Verify ROCm 7+ Installation (Must Be System-level ROCm) |
| 48 | + |
| 49 | +```bash |
| 50 | +amd-smi |
| 51 | +``` |
| 52 | + |
| 53 | +You should see your GPU model, driver, and ROCm version, e.g.: |
| 54 | + |
| 55 | +``` |
| 56 | +MARKET_NAME: Radeon 8060S Graphics |
| 57 | +TARGET_GRAPHICS_VERSION: gfx1151 |
| 58 | +ROCm version: 7.13.0 |
| 59 | +``` |
| 60 | + |
| 61 | +Confirm llama.cpp can see the GPU: |
| 62 | + |
| 63 | +```bash |
| 64 | +cd ~/minicpm-rocm/llama-bin |
| 65 | +export LD_LIBRARY_PATH=$PWD:/opt/rocm/lib:$LD_LIBRARY_PATH |
| 66 | +./llama-cli --list-devices |
| 67 | +# Available devices: |
| 68 | +# ROCm0: Radeon 8060S Graphics (65536 MiB, ... free) |
| 69 | +``` |
| 70 | + |
| 71 | +--- |
| 72 | + |
| 73 | +#### 3. Set Permissions and Environment Variables |
| 74 | + |
| 75 | +```bash |
| 76 | +cd ~/minicpm-rocm/llama-bin |
| 77 | +chmod +x llama-cli llama-server |
| 78 | +export LD_LIBRARY_PATH=$PWD:/opt/rocm/lib:$LD_LIBRARY_PATH |
| 79 | +``` |
| 80 | + |
| 81 | +> The Lemonade build bundles its own ROCm runtime libraries next to the binaries, so add `$PWD` to `LD_LIBRARY_PATH` in addition to `/opt/rocm/lib`. |
| 82 | +
|
| 83 | +--- |
| 84 | + |
| 85 | +#### 4. Download the MiniCPM5-1B GGUF |
| 86 | + |
| 87 | +llama.cpp uses the **GGUF model format**. Available quantizations: |
| 88 | + |
| 89 | +| File | Size | Notes | |
| 90 | +| --- | --- | --- | |
| 91 | +| `MiniCPM5-1B-F16.gguf` | 2.1 GB | Full precision | |
| 92 | +| `MiniCPM5-1B-Q8_0.gguf` | 1.1 GB | Minimal quality loss | |
| 93 | +| `MiniCPM5-1B-Q4_K_M.gguf` | 657 MB | For limited VRAM | |
| 94 | + |
| 95 | +Using the Chinese Hugging Face mirror: |
| 96 | + |
| 97 | +```bash |
| 98 | +mkdir -p ~/models/MiniCPM5-1B-GGUF && cd ~/models/MiniCPM5-1B-GGUF |
| 99 | +export HF_ENDPOINT=https://hf-mirror.com |
| 100 | + |
| 101 | +curl -L --fail -o MiniCPM5-1B-Q4_K_M.gguf \ |
| 102 | + "https://hf-mirror.com/openbmb/MiniCPM5-1B-GGUF/resolve/main/MiniCPM5-1B-Q4_K_M.gguf" |
| 103 | +``` |
| 104 | + |
| 105 | +> You can also use `huggingface-cli download` or `hfd.sh` + `aria2` for resumable downloads. |
| 106 | +
|
| 107 | +--- |
| 108 | + |
| 109 | +#### 5. CLI Text Test (`llama-cli`) |
| 110 | + |
| 111 | +```bash |
| 112 | +cd ~/minicpm-rocm/llama-bin |
| 113 | +export LD_LIBRARY_PATH=$PWD:/opt/rocm/lib:$LD_LIBRARY_PATH |
| 114 | + |
| 115 | +./llama-cli \ |
| 116 | + -m ~/models/MiniCPM5-1B-GGUF/MiniCPM5-1B-Q4_K_M.gguf \ |
| 117 | + -ngl 99 -c 4096 --temp 0.7 --top-p 0.95 -n 2048 |
| 118 | +``` |
| 119 | + |
| 120 | +MiniCPM5-1B supports reasoning mode and may emit a `[Start thinking]` block before its final answer. |
| 121 | + |
| 122 | +--- |
| 123 | + |
| 124 | +#### 6. Start llama-server (OpenAI-compatible API) |
| 125 | + |
| 126 | +```bash |
| 127 | +export LD_LIBRARY_PATH=$PWD:/opt/rocm/lib:$LD_LIBRARY_PATH |
| 128 | +cd ~/minicpm-rocm/llama-bin |
| 129 | + |
| 130 | +./llama-server \ |
| 131 | + -m ~/models/MiniCPM5-1B-GGUF/MiniCPM5-1B-Q4_K_M.gguf \ |
| 132 | + -ngl 99 -c 8192 --jinja --host 127.0.0.1 --port 8080 |
| 133 | +``` |
| 134 | + |
| 135 | +> `--jinja` enables the model's bundled chat template (recommended for MiniCPM5-1B). |
| 136 | +
|
| 137 | +--- |
| 138 | + |
| 139 | +#### 7. Test the API |
| 140 | + |
| 141 | +```bash |
| 142 | +curl -s -X POST http://127.0.0.1:8080/v1/chat/completions \ |
| 143 | + -H "Content-Type: application/json" \ |
| 144 | + -d '{ |
| 145 | + "model": "MiniCPM5-1B", |
| 146 | + "messages": [{"role": "user", "content": "1+1=? Then explain in one short sentence."}], |
| 147 | + "temperature": 0.7, "top_p": 0.95, "max_tokens": 256 |
| 148 | +}' | jq -r ' |
| 149 | +.choices[0].message.content as $txt | |
| 150 | +(.usage.completion_tokens / (.timings.predicted_ms / 1000)) as $tps | |
| 151 | +"Answer:\n\($txt)\n\ntokens/s: \($tps|tostring)" |
| 152 | +' |
| 153 | +``` |
| 154 | + |
| 155 | +Reference: **~185 tokens/s** decode on Radeon 8060S (gfx1151), ROCm 7.13, ctx=8192. Actual speed depends on hardware. |
| 156 | + |
| 157 | +#### Generation Parameters |
| 158 | + |
| 159 | +| Mode | `--temp` | `--top-p` | When to use | |
| 160 | +| --- | --- | --- | --- | |
| 161 | +| Think | 0.9 | 0.95 | reasoning, math, code | |
| 162 | +| No-think | 0.7 | 0.95 | fast assistant, low latency | |
| 163 | + |
| 164 | +--- |
| 165 | + |
| 166 | +### Method 2: Docker (Official ROCm llama.cpp Image) |
| 167 | + |
| 168 | +> Docker requires `amdgpu-dkms`: |
| 169 | +> https://rocm.docs.amd.com/projects/install-on-linux/en/latest/how-to/docker.html |
| 170 | +
|
| 171 | +#### 1. Start the Container |
| 172 | + |
| 173 | +```bash |
| 174 | +export MODEL_PATH='~/models' |
| 175 | + |
| 176 | +sudo docker run -it \ |
| 177 | + --name=$(whoami)_llamacpp_minicpm \ |
| 178 | + --privileged --network=host \ |
| 179 | + --device=/dev/kfd --device=/dev/dri \ |
| 180 | + --group-add video --cap-add=SYS_PTRACE \ |
| 181 | + --security-opt seccomp=unconfined \ |
| 182 | + --ipc=host --shm-size 16G \ |
| 183 | + -v $MODEL_PATH:/data \ |
| 184 | + rocm/dev-ubuntu-24.04:7.0-complete |
| 185 | +``` |
| 186 | + |
| 187 | +--- |
| 188 | + |
| 189 | +#### 2. Prepare the Workspace |
| 190 | + |
| 191 | +```bash |
| 192 | +apt-get update && apt-get install -y nano libcurl4-openssl-dev cmake git |
| 193 | +mkdir -p /workspace && cd /workspace |
| 194 | +``` |
| 195 | + |
| 196 | +--- |
| 197 | + |
| 198 | +#### 3. Clone llama.cpp |
| 199 | + |
| 200 | +```bash |
| 201 | +git clone https://github.com/ROCm/llama.cpp |
| 202 | +cd llama.cpp |
| 203 | +``` |
| 204 | + |
| 205 | +--- |
| 206 | + |
| 207 | +#### 4. Set ROCm Architecture |
| 208 | + |
| 209 | +```bash |
| 210 | +# AI MAX 395 (gfx1151) example |
| 211 | +export LLAMACPP_ROCM_ARCH=gfx1151 |
| 212 | +``` |
| 213 | + |
| 214 | +--- |
| 215 | + |
| 216 | +#### 5. Build llama.cpp |
| 217 | + |
| 218 | +```bash |
| 219 | +HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \ |
| 220 | +cmake -S . -B build \ |
| 221 | + -DGGML_HIP=ON \ |
| 222 | + -DAMDGPU_TARGETS=$LLAMACPP_ROCM_ARCH \ |
| 223 | + -DCMAKE_BUILD_TYPE=Release \ |
| 224 | + -DLLAMA_CURL=ON && \ |
| 225 | +cmake --build build --config Release -j$(nproc) |
| 226 | +``` |
| 227 | + |
| 228 | +--- |
| 229 | + |
| 230 | +#### 6. Run the Test |
| 231 | + |
| 232 | +```bash |
| 233 | +./build/bin/llama-cli \ |
| 234 | + -m /data/MiniCPM5-1B-GGUF/MiniCPM5-1B-Q4_K_M.gguf \ |
| 235 | + -ngl 99 -c 4096 -p "Explain AMD ROCm in two sentences." |
| 236 | +``` |
| 237 | + |
| 238 | +#### Build a GGUF from Your Own Checkpoint |
| 239 | + |
| 240 | +If you fine-tuned your own MiniCPM5-1B variant: |
| 241 | + |
| 242 | +```bash |
| 243 | +python ./convert_hf_to_gguf.py /path/to/your-MiniCPM5-fp16-hf --outfile F16.gguf --outtype f16 |
| 244 | +./build/bin/llama-quantize F16.gguf MiniCPM5-1B-Q4_K_M.gguf Q4_K_M |
| 245 | +``` |
| 246 | + |
| 247 | +--- |
| 248 | + |
| 249 | +### Screenshot Example |
| 250 | + |
| 251 | +<div align='center'> |
| 252 | + <img src="../../../public/images/01-deploy/minicpm/minicpm5-example.jpg" alt="MiniCPM5-1B llama.cpp example" width="90%"> |
| 253 | +</div> |
0 commit comments