int4-quantization

From-scratch, heavily-annotated CUDA inference runtime for Qwen2.5-Coder-7B on H100 (sm_90). Custom INT4 packer, fused GEMV, paged KV, split-KV attention, CUDA graph decode — every hot path commented for the why. Educational, not a llama.cpp replacement.

g023's TurboXInf 🚀: 2x+ faster inference for Qwen3-1.77B or Qwen3.5-2B on RTX 3060! Custom Triton INT8 GEMV kernels halve memory traffic by fusing dequantization, paired with torch.compile. Hits 113 tok/s (vs 56.4 baseline) with no quality loss with INT8 even better results for INT4. MIT License.

vLLM serving poolside Laguna S 2.1 (118B MoE, 8B active, INT4) on AMD Strix Halo (Ryzen AI Max+ 395, gfx1151, 128GB unified) via TheRock ROCm nightly. OpenAI-compatible, 256K context, DFlash speculative decoding.

Implemented post-training quantisation (PTQ) on transformer-based reasoning models using 8-bit and 4-bit weight quantisation (INT8, INT4) with frameworks like PyTorch and Hugging Face Transformers. Leveraged libraries such as bitsandbytes to reduce model size and accelerate inference, while evaluating performance degradation on reasoning tasks. Com

Trajectory-aware mixed-precision quantization for Stable Diffusion XL, perceptually lossless inference under 8GB VRAM, no retraining.

Three hand-written Triton kernels for LLM inference (fused RMSNorm plus residual, online softmax, INT4 g128 GEMV) benchmarked on NVIDIA Blackwell against PyTorch eager and torch.compile, with every raw CUDA-event sample, measured device ceiling, and Nsight Compute report committed and CI-verified.