Knowledge note
SIMD/SIMT Hybrid Programming
A programming model that combines lane-level SIMD throughput with thread-level SIMT flexibility for regular and irregular vector work.
SIMD/SIMT Hybrid Programming
SIMD/SIMT Hybrid Programming combines single-instruction multiple-data lane execution with single-instruction multiple-thread execution so regular and irregular vector work can use different strengths of the same subsystem.
Why It Exists
Regular elementwise operations benefit from packed lanes and predictable memory access, while gathers, scatters, hash updates, and branch-heavy code are easier to express when individual threads can follow different addresses or paths. A single execution model makes one of those workloads awkward or inefficient.
How It Works
The Ascend 950 white paper describes SIMD as the main path, using dual-issue and out-of-order behavior for throughput, and SIMT as a complementary path for irregular access and control flow. A vector function can select either mode and switch between function types as the operator’s structure changes.
Tradeoffs
SIMT flexibility does not guarantee SIMD-level throughput, and SIMD loses its advantage on fragmented access or divergent control. The compiler and kernel author still have to choose a mode that matches the data pattern and tile size.
Common Confusions
- SIMD vs. SIMT: SIMD applies one instruction across packed data lanes; SIMT presents many threads executing a common program with more independent addressing and control.
- Hybrid programming vs. automatic fusion: The hybrid model provides execution choices; it does not automatically fuse every operator or remove synchronization costs.
Where It Appears
- Ascend 950 NPU Architecture: Compute, Memory, and Unified Bus — Uses SIMD-first Vector execution and SIMT for irregular Transformer-side operations.
Related Terms
- GEMM — The matrix workload handled by the Cube side of the AI Core.
- Memory Banking — On-chip access organization that can affect vector throughput.