1 SIMD

Fundamentals

Data-Level Parallelism (DLP) arises in applications that perform the same operation across large collections of data — scientific matrix computations, media processing, and machine learning. SIMD (Single Instruction Multiple Data) architectures exploit this by launching many data operations from a single fetched instruction, making them far more energy-efficient than MIMD where each operation requires its own fetch. The programmer thinks sequentially; the hardware achieves parallelism through concurrent data operations.

DLP is realized through three architectural approaches:

• Vector architectures: Operate on entire vectors using deep pipelines and large register files.
• Multimedia SIMD extensions: Partition wide ALUs to process multiple narrow elements per instruction.
• GPUs: Execute thousands of threads in lockstep across many simple cores.

All three are simpler to program than classical MIMD for data-parallel workloads. Vector architecture is the most general and shares the deepest structural similarities with GPUs, so it is the natural starting point.

Vector Architecture

Vector architectures grab sets of data elements scattered throughout memory, place them into large sequential register files, operate on them using deep pipelines, and disperse the results back to memory. A single vector instruction controls dozens of register-register operations on independent data elements. These large register files function as compiler-controlled buffers that hide memory latency and leverage memory bandwidth. By paying the long memory latency only once per vector load or store, vector architectures achieve high performance without the energy demands and design complexity of highly out-of-order superscalar processors.

Hardware Components

To execute these operations efficiently, vector architectures rely on specialized hardware components.

• Vector registers: Large sequential storage structures. The RV64V architecture implements 32 vector registers. The register file provides multiple read and write ports (e.g., 16 read and 8 write ports) connected to functional units via crossbar switches to allow simultaneous vector operations.
• Vector functional units: Fully pipelined execution units capable of starting a new operation on every clock cycle.
• Vector load/store unit: Moves data between vector registers and memory with high bandwidth, typically one word per clock cycle after an initial latency.
• Scalar registers: Standard general-purpose and floating-point registers that provide input data to vector functional units and compute addresses for the vector load/store unit.
• Maximum Vector Length (VLMAX): A hardware-specific parameter defining the maximum number of elements a vector operation can process, uncoupled from the instruction opcode.
• Dynamic Register Typing: Vector registers are configured dynamically prior to execution rather than specifying sizes in individual instruction opcodes.
  • A standard element width (SEW) field specifies the data size (e.g., 8, 16, 32, or 64 bits).
  • A length multiplier (LMUL) field allows grouping multiple registers to form longer vectors, optimizing register file usage.
  • Dynamic typing enables implicit type conversions during arithmetic operations.

These specialized components work together to process complete loops of data through deep pipelines.

Vector Execution

Vector execution dramatically reduces dynamic instruction bandwidth — a single vector instruction replaces an entire scalar loop body. The DAXPY loop (Y = a×X + Y) illustrates this: RV64G executes 258 instructions for 32 elements, while RV64V executes just 8. Loops are vectorizable only when they have no loop-carried dependences (where iteration $i$ reads a value written by iteration $i-1$); the compiler confirms this statically or flags the issue to the programmer.

• Pipeline stalls occur only once per vector instruction rather than once per element.
• Chaining: Results from one vector functional unit are forwarded directly to the input of another, allowing dependent vector instructions to overlap.
• Flexible chaining: A vector instruction can chain to any active vector instruction simultaneously, provided no structural hazard is introduced.
• Convoy: A set of vector instructions that can execute together without structural hazards. Chaining allows RAW-dependent instructions to share a convoy.
• Chime: The time to execute one convoy. A sequence of $m$ convoys over vectors of length $n$ takes approximately $m\times n$ clock cycles.

For DAXPY (vle, vfmul, vle, vfadd, vse): convoy 1 is vle+vfmul, convoy 2 is vle+vfadd, convoy 3 is vse alone (structural hazard on the load/store unit). Three convoys, two FP ops per result: 1.5 cycles per FLOP.

The chime model ignores startup latency — the cycles until the pipeline fills. RV64V pipeline depths: 6 cycles (FP add), 7 (FP multiply), 20 (FP divide), 12 (vector load). The model is accurate for long vectors but underestimates time for short ones.

Further Features

• Multiple Lanes: Each lane holds an interleaved slice of the vector register file and one pipeline per functional unit, processing multiple elements per clock cycle with no inter-lane communication. Doubling lanes halves the chime time.
• Vector-Length Register ($vl$): Controls the active length of any vector operation ($vl\le \text{VLMAX}$). When $n>\text{VLMAX}$, strip mining partitions the loop into full VLMAX blocks plus a remainder; vsetvl sets $vl$ automatically to $\min (\text{VLMAX},\text{remaining})$.
• Mask Registers: Conditional statements inside loops are handled via IF-conversion — the compiler generates a Boolean mask vector. Instructions update only elements where the mask bit is 1; masked-off elements still consume pipeline cycles.
• Memory Banks: Multiple independent banks sustain one word per clock cycle bandwidth. Startup penalties (often >100 cycles) are amortized across the vector block.
• Stride: Strided load/store instructions gather non-contiguous elements (e.g., a matrix column in row-major layout) by stepping a scalar stride between accesses. Non-unit strides cause bank conflicts when $\frac{\text{banks}}{\gcd (\text{stride},\text{banks})}<\text{bank busy time}$.
• Gather/Scatter: An index vector holds offsets into a sparse structure. Gather loads elements at base + offsets into a dense register; scatter stores them back. Slower than unit-stride due to per-element address generation and likely bank conflicts.

Programming Vector Architectures

Vector compilers auto-vectorize loops when they can prove no loop-carried dependences exist, reporting exactly why vectorization fails when it cannot. Programmer directives are needed for cases like gather-scatter, where the compiler cannot statically prove that index vectors contain distinct, dependency-free offsets.

SIMD Instruction Set Extensions

Core Mechanics

• Fundamental Principle: Media applications frequently operate on narrow data types, such as 8-bit values for color and transparency or 16-bit values for audio samples.
• Partitioned ALU Design: Processors divide internal carry chains within wide arithmetic-logical units (ALUs) to execute simultaneous operations on short data vectors.
  • A 256-bit adder can be partitioned to concurrently process thirty-two 8-bit, sixteen 16-bit, eight 32-bit, or four 64-bit operands.
• Instruction Semantics: A single SIMD instruction dictates an identical operation across all partitioned data elements, executing within relatively small register files.
• To understand the constraints of this partitioned approach, it is necessary to examine the specific architectural features it omits compared to dedicated vector processors.

Architectural Omissions

• Fixed Data Width in Opcodes: Unlike vector architectures, SIMD extensions lack a dedicated vector-length register. The exact number of data operands is strictly determined by the opcode, forcing the instruction set to expand significantly every time the hardware register width increases.
• Restricted Addressing Modes: Early SIMD extensions omitted advanced addressing modes like strided accesses or gather-scatter data transfers, initially requiring all memory accesses to be contiguous and aligned.
• Absence of Mask Registers: Conditional execution of individual vector elements was historically unsupported due to the lack of mask registers, severely complicating compiler auto-vectorization.
• Despite these strict architectural compromises, the ease and low cost of hardware integration sparked a continuous, generational evolution of SIMD capabilities.

Evolution of x86 SIMD Extensions

• MMX (1996): Repurposed standard 64-bit floating-point registers to perform parallel 8-bit or 16-bit integer operations, efficiently reusing existing data-transfer instructions.
• SSE (1999): Introduced sixteen dedicated 128-bit XMM registers, supporting parallel single-precision floating-point arithmetic and requiring new, separate data transfer instructions.
• AVX (2010): Doubled register width to 256 bits (YMM registers) to double operation throughput across all narrower data types. As width increases, permutation instructions become important — AVX includes shuffles across 32-, 64-, and 128-bit operands within a 256-bit register, and BROADCAST replicates a 64-bit operand to all four positions.

AVX instruction	Description
VADDPD	Add four packed double-precision operands
VSUBPD	Subtract four packed double-precision operands
VMULPD	Multiply four packed double-precision operands
VDIVPD	Divide four packed double-precision operands
VFMADDPD	Multiply and add four packed double-precision operands
VFMSUBPD	Multiply and subtract four packed double-precision operands
VCMPxx	Compare four packed double-precision operands for EQ, NEQ, LT, LE, GT, GE
VMOVAPD	Move aligned four packed double-precision operands
VBROADCASTSD	Broadcast one double-precision operand to four locations in a 256-bit register

• AVX2 (2013) & AVX-512 (2017): AVX2 introduced gather operations and vector shifts. AVX-512 subsequently doubled register width to 512 bits (ZMM registers), increased the register count to 32, and added scatter instructions alongside mask registers.
• AMX (2022): Advanced Matrix Extensions specifically targeted machine learning by introducing eight two-dimensional vector registers called tiles.
  • Each tile consists of 16 rows of 256-bit width.
  • The hardware performs direct matrix multiplication on 8-bit integers or 16-bit brain floating-point (BF16) formats.
• Evaluating the computational gains of these escalating hardware structures requires a standardized framework to analyze memory and arithmetic bottlenecks

Multimedia SIMD vs. Vector Architectures

• Instruction Set Bloat vs. Stability: Because SIMD opcodes hardcode the data width, achieving greater parallelism requires creating hundreds of new instructions, escalating the instruction set size (e.g., the x86 ISA grew from 80 to over 1400 instructions). Vector ISAs remain completely stable regardless of hardware scaling by relying on a dynamic vector-length register.
• Code Size and Execution Overhead:
  • SIMD code requires substantial bookkeeping and strip-mining logic to handle boundary conditions when array sizes are not exact multiples of the register width.
  • SIMD loops process far fewer elements per instruction (e.g., 2 or 4) compared to vector loops (e.g., 64), resulting in 10 to 20 times more dynamic instructions executed and significantly higher instruction-decoding energy.
• Integration Rationale: Despite clear architectural disadvantages, SIMD extensions persist due to lower implementation costs, minimal extra processor state (which aids fast context switches), and simplified virtual memory management, since aligned block memory accesses are guaranteed not to cross page boundaries.
• These foundational trade-offs firmly establish multimedia SIMD as a pragmatic hardware compromise, trading the elegance of scalable vector processing for low-cost integration into general-purpose scalar pipelines.

Programming SIMD Extensions

Early SIMD code was written using hand-tuned libraries or assembly. Modern compilers auto-vectorize scientific loops, emitting SIMD instructions directly, though programmers often rely on intrinsics for precise control. The key programmer responsibility is data alignment: structures and arrays must be aligned to the SIMD register width so that block memory accesses do not cross page boundaries.