W4A16 GEMM on XDNA2: 5.94 TOPS via open stack (IRON 1.4.x + Peano) - #8
Merged
Conversation
…N whole-array design - mix_int4_ATB.cc / zero.cc vendored byte-identical from glassescrab/mlir-aie@8c3d2be (sha256 matches scripts/check-w4a16-compile.sh pins) - packing.py: exact 4352 B/tile byte layout (int4 data in 8x8 micro-tiles, bf16 per-column scales, duplicated int8 zero-points) + NumPy dequant and tile-faithful bf16-accumulation references; packer verified bit-exact against a scalar model of the kernel's read path - w4a16_gemm.py: stock whole_array dataflow with packed-byte B fifos and half-M-tile A consumption (consumer_obj_type) matching the kernel's DIV=2 weight-stationary L1 dequant cache Not yet hardware-verified; first NPU run follows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Numerics (signed A, random int4 AWQ-g128 weights, seed 1726250518): - 512^3: max err/|A||B| scale 1.07e-2, mean 2.1e-3 -> PASS - 2048^3: max err/|A||B| scale 8.95e-3, mean 1.3e-3 -> PASS (elementwise-relative stats vs f32 reference are cancellation-dominated on near-zero outputs; the |A||B| accumulation-scale bound is the honest bf16/bfp16 criterion and a real dataflow bug would sit ~100x above it) Performance (NPU time, 20 iters): - 2048^3: 2.89 ms = 5.94 TOPS vs 4.64 TFLOPS bf16-via-bfp16 baseline (+28%) with 3.76x less B DRAM traffic; i8 record on this array is 6.65 TOPS - 512^3: 158 us = 1.70 TOPS Three fixes en route: - -DAIE_API_EMULATE_BFLOAT16_MMUL_WITH_BFP16 on the kernel: without it the 8x8x8 bf16 aie::mmul lowers via 1/4-rate native bf16 (0.94 TOPS, 6.2x) - B packed in column-distribution order so each column's per-pass stream is one contiguous fill: a strided multi-row B tap needs an inexpressible 4D BD at 2048^3, and per-row fills exhaust the 16 BDs per shim tile - verifier bounds error against the |A| @ |Bdq| accumulation scale Gotcha found: changing ExternalFunction compile_flags inside a helper does not invalidate the @iron.jit cache (recipe hash covers only the jit'd function's bytecode and its own flags) - clear ~/.npu/cache or rename the object file when toggling kernel defines. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…14, changelog - examples/mlir-aie/w4a16_gemm/README.md: measured table (1.70/2.98/5.94/6.24 TOPS, all CPU-reference PASS), design delta vs stock whole_array, packed tile format, numerics methodology, shape constraints - MLIR-AIE.md: W4A16 section beside the GEMM baselines - XDNA2.md: 'where this goes next' item 3 upgraded from compile-only to hardware-verified; headline bullet added - GOTCHAS.md M12 (1/4-rate 8x8x8 bf16 mmul without the bfp16 define, 6.2x), M13 (ExternalFunction compile_flags invisible to the @iron.jit cache), M14 (shim-DMA BD dims/count/stride limits that only bite at larger shapes) - validate-repo.sh: exempt the two sha256-pinned vendored kernel files from whitespace normalization (byte-identity is the provenance guarantee) Translations (de/fr/ko/ja) deliberately not updated in this commit. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…ipped packer test - packing: pack_b returns a PackedB blob tagged with the n_aie_cols it was packed for; assert_packed_for turns a blob/design column mismatch (same size, same dtype, silently wrong C) into a loud error, checked in main() - design: enforce N <= 4096 (inclusive 2^20 C write-back stride range) and even M/(m*4) row-block count as clear asserts in _build_design, instead of cryptic aiecc/step_tiler failures minutes later - DIV is now derived from the vendored source's 'constexpr int DIV' (no #ifndef guard upstream, so -D cannot override it without breaking the byte pin); core_fn documents the exactly-DIV-calls-per-B-tile g_counter invariant and why desync is persistent and silent - ship test_packing.py: bit-exactness harness walking the kernel's exact read path over the packed blob, including a non-identity column ordering and a negative assert_packed_for check (CPU-only) - check-w4a16-compile.sh: pin -DAIE_API_EMULATE_BFLOAT16_MMUL_WITH_BFP16 (the M12 6.2x trap) and point its scope line at the verified example - validate-repo.sh: vendored files are now sha256-asserted against the pins instead of skipped — byte-identity subsumes whitespace/conflict scanning and catches formatter or merge damage (negative-tested) - --atol help now states it only floors the printed rel-err denominators and takes no part in the PASS/FAIL verdict Re-verified after the changes: test_packing.py PASS, 2048^3 NPU rerun PASS (max normalized err 8.95e-3), compile probe PASS, validate-repo.sh PASS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…mber consistency - retire every remaining 'W4A16 is compile-only' claim contradicting the hardware result: OPEN-NPU-LAB.md (asset table, project ladder, boundary bullet) + OPEN-NPU-LAB.ko.md, APPLICATIONS.md table row + ko/de/fr/ja, XDNA2 item-3 status paragraph in ko/de/fr/ja, plus two more found by sweep: LLM-ROADMAP.md step 1 (now struck through as done) and RESEARCH.md's TileFuse note — all now state CPU-reference PASS and 5.94 TOPS with the open problems (energy, llama.cpp, chess gap) explicit - 0.94-vs-X consistency: the controlled A/B pair is 0.94 vs 5.81 TOPS (6.2x, same session, 10 iters); 5.94 is the separate final tuned run. GOTCHAS M12, example README, and the kernel-flag comment now all say so - GOTCHAS M14 + README: BD stride limit corrected to inclusive <= 2^20 (the verified N=4096 C stride is exactly 2^20) - README documents the shipped test_packing.py harness and the PackedB column-count tag; CHANGELOG references the harness by name validate-repo.sh (link checks included): PASS. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TileFuse-style W4A16 GEMM (int4 AWQ-g128 weights, bf16 activations) running on the whole 4×8 Strix array under the fully open stack.
Results
Contents
examples/mlir-aie/w4a16_gemm/: vendored pinned kernel (byte-identical, now sha256-gated in validate-repo.sh), NumPy AWQ-g128 packer (bit-exactness test shipped:test_packing.py), IRON 1.4.x whole-array design with packed-byte B fifos (3.76× less weight DRAM traffic) and L1 weight-stationary dequantKnown limits
🤖 Generated with Claude Code