This is the C/CUDA implementation for DynaSpec (modify the FR-Spec)
vim setup.py # change arch="80" to other code for your platform, see https://developer.nvidia.com/cuda-gpus#compute
pip install .All scripts for evaluation are located in the scripts folder. Here we use Llama-3-8B-Instruct as an example:
# 1. Run evaluations
python3 evaluation/inference_eagle.py --model-path Meta-Llama-3-8B-Instruct --eagle-path LLaMA3-Instruct-8B-FR-Spec --cuda-graph --model-id llama-3-8b-instruct/eagle-dyna-50-epochs-ffn --memory-limit 0.80 --bench-name spec_bench --dtype "float16" --chat-template "llama-3" --V -1 --calc-stream-priority -5 --meta-stream-priority 0 --using-meta-clf --first-token-topk-meta-classes 1550 --topk-meta-classes 820 --num-meta-classes 4096 --token2meta-path ./LLaMA3-Instruct-8B-FR-Spec/token_id_to_cluster_id_kmeanspp_cosine_8b.pt --using-ffn-meta-clf --ffn-meta-hidden-size 1024
# 2. Evaluate speed
bash scripts/<benchmark>/llama3-8b-instruct/speed_up.sh
# 3. Check correctness (for human_eval)
bash scripts/<benchmark>/llama3-8b-instruct/check_correctness.shReplace <benchmark> with one of: spec_bench, human_eval