5 Power and Cooling

WSC Power Distribution and Cooling

Warehouse-scale computers (WSCs) concentrate massive computational capacity, frequently exceeding 100 MW of critical power. Supplying this power and removing the resulting heat represent the majority of WSC infrastructure costs and fundamentally limit the scale of deployable IT equipment.

Power Utilization Effectiveness (PUE)

Efficiency in power distribution and cooling is quantified by Power Utilization Effectiveness (PUE), which measures the overhead required to support the actual computing hardware:

$PUE=\frac{\text{Total facility power}}{\text{IT equipment power}}$

• IT equipment power: The energy consumed by the compute, storage, and networking hardware housed within the racks.
• Total facility power: The IT load plus all power delivery losses (voltage conversions) and cooling loads (fans, air conditioning units, chillers).
• Ideal baseline: A theoretical perfect facility has a PUE of 1.0. Historically, conventional data centers operated at a PUE of 2.5 (1 W for IT requiring 1.5 W of overhead), whereas highly optimized modern WSCs maintain PUEs around 1.10.

Because high PUE drives up capital expenditures (CAPEX), operational expenditures (OPEX), and environmental impact, modern WSC design strictly optimizes the two primary sources of overhead: power delivery pathways and heat removal systems.

---

WSC Power Delivery Architecture

Power routing from the utility grid to the processor die requires a hierarchy of voltage step-downs, each introducing conversion losses.

• Standard AC Distribution Hierarchy:
  • Utility Grid: Supplies high-voltage lines at 115 KV AC.
  • Utility Substation: Steps down to 10–50 KV AC for intra-site distribution.
  • Unit Substation: Steps down to 400–500 V AC and routes through an Uninterruptible Power Supply (UPS).
  • Power Distribution Units (PDUs): Steps down to 110–220 V AC, allocating ~100–200 KW per unit with individual 6 KW circuit breakers.
  • Server Power Supply: Converts 110–220 V AC to 12 V DC.
  • Voltage Regulator Modules (VRMs): Steps 12 V DC down to the specific operational voltage of the chips (e.g., 1.1 V DC for a CPU).
• Uninterruptible Power Supply (UPS): Masks utility failures to maintain 99.99% availability.
  • Generators: Diesel-powered engines that require 10–15 seconds to start and synchronize.
  • Bridging Storage: Batteries, chemical arrays, or mechanical flywheels that supply instantaneous power until the generators assume the load.
  • Redundancy: Deployed in N+1 or N+2 configurations to allow maintenance and tolerate component failures.
• Conversion Inefficiencies:
  • Centralized UPS Losses: Traditional facility-level UPS systems require a double conversion (AC-to-DC for battery connection, then DC-to-AC for distribution), resulting in 3–6% power loss.
  • Server-Level Losses: Inefficiencies in the server power supply and VRMs introduce another ~10% loss. This specific intra-server overhead is tracked via a localized metric known as Server-Level PUE (sPUE).
• Power Delivery Optimizations:
  • Distributed UPS: Replaces centralized facility UPS arrays with localized battery modules inside every server rack. This yields 99% efficiency by removing the double-conversion penalty and increases fault tolerance by isolating battery failures.
  • Direct DC Distribution: Modifies unit substations to output 380 V DC, while rack-level power supplies step down directly to 12–48 V DC. This removes all AC-to-DC conversions at the rack level and naturally integrates with the DC nature of distributed battery arrays.

Just as minimizing voltage conversions recovers stranded energy in the power delivery path, eliminating mechanical refrigeration is the critical path to minimizing the remaining facility overhead: heat removal.

---

WSC Cooling Systems

All electrical energy consumed by a WSC is ultimately converted into heat, which must be systematically moved from the silicon dies to the external atmosphere.

• Traditional Computer Room Air-Conditioning (CRAC):
  • Raised-Floor Airflow: Cold air is forced into an underfloor plenum and released through perforated tiles into isolated “cold aisles”.
  • Heat Transfer: Rack-mounted servers draw from the cold aisle, pass the air over internal heatsinks, and exhaust the heated air into “hot aisles”.
  • Chilled Water Loops: The hot air rises to CRAC intakes, where it passes over coils filled with cold water. The warmed water is pumped to external chillers and cooling towers, which use mechanical refrigeration to cool the water and release the heat outdoors.
• Cooling Optimizations:
  • Thermal Setpoint Elevation: Raising the cold aisle ambient temperature from <20°C (68°F) to 27°C (80°F) significantly reduces the workload on external chillers.
  • Aisle Isolation: Strict physical containment of hot and cold aisles prevents thermal mixing, ensuring the CRAC units receive undiluted hot exhaust, which maximizes thermodynamic efficiency.
  • Holistic Airflow Management: Individual server fans are inherently inefficient (consuming 6–9 W each) and prone to mechanical failure. WSCs synchronize low-power server fans with massive facility-level air impellers. Fan speeds are governed by the pressure differential between hot and cold aisles rather than localized temperature readings.
  • Economization (Free Cooling): Bypasses mechanical refrigeration by utilizing the ambient environment.
    • Air-economization: Large fans pull in low-temperature outside air, filter it, and circulate it directly through the facility.
    • Water-economization: Replaces mechanical chillers with heat exchangers connected to naturally cold local water sources (e.g., lakes, rivers, or the sea).
  • Liquid and Immersion Cooling: As processor thermal design power (TDP) reaches 400 W for CPUs and 800 W for GPUs, forced-air cooling becomes physically insufficient. Advanced WSCs deploy closed-loop, direct-to-chip liquid cooling or submerge equipment entirely into dielectric coolants to drastically improve heat removal rates, enabling higher clock frequencies and denser chip geometries.

Cost and Efficiency

Warehouse-scale computers (WSCs) require massive capital investment and incur high recurring costs. Optimizing the Total Cost of Ownership (TCO) demands maximizing the hardware’s efficiency across power provisioning, utilization, performance, and energy consumption.

Cost of a WSC

The economic viability of a WSC relies on balancing Capital Expenditures (CAPEX) to build the facility with Operational Expenditures (OPEX) to run it.

• TCO Components and Amortization:
  • CAPEX: The upfront costs to construct the facility and purchase IT equipment. WSC facilities cost between $7and$13 per Watt to build. CAPEX is amortized over the effective life of the components: facilities over 10–20 years, networking equipment over 4 years, and servers over 3–4 years.
  • OPEX: The recurring monthly costs, primarily driven by power consumption, cooling infrastructure maintenance, and personnel.
• Cost Distribution:
  • Server purchasing dominates the amortized CAPEX, representing 60% to 65% of the total TCO.
  • Networking equipment represents roughly 8% of OPEX and 19% of server CAPEX, a ratio that is not decreasing as rapidly as server costs due to constant demands for higher bandwidth.
  • Facility costs account for roughly 10% of the TCO.
• Cost of Power:
  • Power and cooling represent more than a third of the OPEX.
  • The fully burdened cost of a watt per year—including the amortized infrastructure cost to deliver that watt—can be modeled as: $\text{Fully burdened cost}=\frac{\text{Monthly cost of infrastructure}+\text{Monthly cost of power}}{\text{Facility size in watts}}\times 12$
  • This formula yields approximately $2perwatt-year,establishingacriticalthreshold:hardwareoptimizationsshouldgenerallynotspendmorethan$2 per watt-year to save energy, barring long-term environmental objectives.

Because power infrastructure dictates a massive portion of WSC costs, extracting the maximum compute capacity from every provisioned watt is the foundation of WSC efficiency.

---

Efficiency Through Power Provisioning

A WSC facility has a strict maximum power capacity. Deploying the maximum number of servers without exceeding this capacity requires aggressive power provisioning strategies.

• Nameplate vs. Actual Peak Power:
  • Manufacturers provide a “nameplate” power rating, indicating the theoretical maximum power a server could draw.
  • In practice, actual peak power consumption is often 60% lower than the nameplate rating because real workloads rarely stress all server components (CPU, memory, disks) to their maximums simultaneously.
• Power Oversubscription:
  • Basing WSC capacity on actual peak power rather than nameplate power allows for significantly more servers to be deployed.
  • WSC operators safely oversubscribe server deployments by up to 40% beyond theoretical power limits, relying on the statistical improbability of all servers peaking simultaneously.
• Mitigating Correlated Power Spikes:
  • Oversubscription requires strict monitoring to prevent rack or facility-wide power failures during correlated load spikes.
  • Hardware and software controllers mitigate these spikes by:
    1. Reducing server clock frequencies.
    2. Drawing temporary power from rack-level backup batteries.
    3. Pausing or descheduling low-priority tasks.
    4. Migrating tasks across racks, PDUs, or other WSCs to balance the load.
• Domain-Specific Architectures (DSAs): Deploying DSAs for targeted workloads, like machine learning, delivers two orders of magnitude higher performance per watt than general-purpose CPUs. This drastically reduces the need to build new WSC facilities, easily offsetting the $50–$100 million custom ASIC development costs.

Deploying the maximum number of servers under a power budget is only economically effective if those servers are kept busy performing useful work.

---

Efficiency Through Higher Utilization

Because amortized server CAPEX heavily dominates TCO, running servers at high utilization is mandatory to recover the investment.

• Utilization Profiles:
  • Mixed Workloads: Clusters running a mix of online interactive services and batch applications typically average low utilization, ranging from 10% to 50%.
  • Batch Workloads: Clusters dedicated to large, continuous batch processing exhibit low demand variation and achieve 50% to 85% average utilization.
• Causes of Low Utilization:
  • Interactive workloads inherently vary by time of day.
  • Developers systematically oversize their resource requests to avoid performance interference from co-scheduled workloads and to mitigate hardware heterogeneity within the WSC.
• Techniques for Increasing Utilization:
  • Right-sizing and Autoscaling: Automatically adjusting the resources granted to workloads based on real-time needs.
  • Interference-aware Scheduling: Tracking workload conflicts and adjusting placements to minimize contention.
  • Resource Harvesting: Utilizing idle reserved capacity to run low-priority, preemptable batch analytics (e.g., spot instances).
  • Hardware Isolation: Employing cache and memory bandwidth partitioning, alongside networking and storage QoS, to guarantee performance isolation for co-scheduled tasks.

While consolidating workloads drives up utilization and lowers costs, it risks introducing latency spikes that directly degrade user-facing performance.

---

Efficiency Through Higher Performance

Performance in a WSC translates directly to revenue. High latency degrades user satisfaction, reducing productivity and interaction rates.

• The Impact of Latency:
  • User productivity is inversely proportional to interaction time, which consists of human entry time, system response time, and human think time.
  • Minor server delays (e.g., an additional 500 ms) significantly drop user query volumes and revenue per user.
• Tail-Tolerance:
  • WSC developers must prioritize the 99th percentile of latency (the “tail”) over the average response time. If 99% of requests are fast but the final 1% is slow, users experiencing the slow tail may abandon the service.
  • Unpredictable latency spikes are caused by shared resource contention, queuing delays, variable processor states (like DVFS/Turbo mode), and software garbage collection.
  • Instead of attempting to eliminate all hardware variability, WSCs utilize software techniques, such as fine-grained load balancing, to swiftly shift small tasks between servers and mask temporary latency spikes.
• Data Locality:
  • Performance heavily depends on data placement. DRAM bandwidth drops dramatically across the network hierarchy: intra-server bandwidth is 40 times greater than intra-rack bandwidth, which is 4 to 10 times greater than intra-row bandwidth.

Maintaining strict latency targets at high performance requires substantial power, but maintaining that same power draw when load drops ruins overall system efficiency.

---

Efficiency Through Energy Proportionality

An ideal WSC server exhibits energy proportionality: it consumes energy directly in proportion to the amount of work it performs, and consumes near-zero power when idle.

• Historical Inefficiency: In 2007, an idle computer consumed 60% of its peak power, and 70% of its peak power at only 20% load.
• Modern Improvements: Modern servers have improved significantly, consuming approximately 20% of peak power at idle and 50% of peak power at 20% load.
• Power Supply Losses: Server power supply units (PSUs) convert high AC/DC voltages to the lower voltages needed by chips. Historically, these PSUs were 60% to 80% efficient and were particularly inefficient at low loads (<25%). Voltage regulator modules (VRMs) on motherboards introduce further efficiency losses.
• Software-Driven Proportionality:
  • Traditional operating systems maximize resource usage (e.g., using all available memory for caches) regardless of energy cost.
  • Modern WSC software architects deploy dynamic controllers that enforce fine-grained power management, tuning servers to barely meet Service Level Objectives (SLOs).
  • This software-driven throttling yields 20% to 30% energy savings when servers operate in the 10% to 60% utilization range, synthetically improving the system’s energy proportionality.

Custom Silicon Strategic Objectives

• Cloud Infrastructure Scale and Topology: AWS operates a global infrastructure encompassing 30 geographic regions, 96 availability zones, and over 410 points of presence.
• Motivations for Custom Silicon Development:
  • Elimination of Virtualization Overheads: Removing hypervisor loads from host CPUs prevents performance jitter, reduces energy consumption, and provisions 100% of compute cycles to client applications.
  • Deployment Velocity: In-house hardware-software co-design compresses the deployment lifecycle of new technologies from months to weeks, enabling hitless upgrades with zero customer downtime.
  • Edge Expansion: Custom hardware allows core cloud services to be packaged into standalone units (e.g., Outpost racks) for secure, on-premise, or edge deployments.
• Economic and Engineering Advantages:
  • Vertical Specialization: Custom Application-Specific Integrated Circuits (ASICs) target exact warehouse-scale computer (WSC) bottlenecks, bypassing the design compromises inherent in commodity chips.
  • Capital Efficiency: Developing in-house silicon circumvents the 20% to 100% profit margins typically charged by commodity microprocessor vendors.
  • Holistic Security: Controlling the silicon design minimizes the attack surface through a custom hardware root of trust and formally verified firmware.

The operational imperative to bypass commodity silicon limitations directly motivated the architectural decoupling of hypervisor functions from the main compute processors.

The Nitro System Architecture

• Hypervisor Decomposition: Traditional hypervisors mediate all I/O operations, consuming critical host CPU cores and introducing significant latency variations under heavy network or storage loads.
• Nitro Hypervisor: A minimalist KVM-based software layer that configures CPU virtualization features (VT-x, VT-d) only during Virtual Machine (VM) launch or resize events.
  • It relies on posted interrupts to remain entirely outside the execution path during standard I/O operations, yielding performance mathematically identical to bare-metal servers.
• Nitro Cards: A family of dedicated, Arm-based PCIe accelerators that completely offload networking, storage, and management tasks from the host CPU.
  • Elastic Network Adapter (ENA): A PCIe Network Interface Card (NIC) utilizing Single-Root I/O Virtualization (SR-IOV) to manage Virtual Private Cloud (VPC) data planes, including encapsulation, rate limiting, and transparent encryption.
  • Elastic Fabric Adapter (EFA): An OS-bypass NIC engineered for HPC and machine learning, utilizing the AWS Scalable Reliable Datagram (SRD) protocol to enable zero-copy, multipath routing across the WSC fabric.
  • Elastic Block Store (EBS) Controller: Exposes NVMe virtual functions for remote block storage, handling network-based zero-copy transfers, snapshots, and storage volume encryption.
  • Instance Storage Controller: Manages directly attached local Flash and Hard Disk Drives (HDDs), applying transparent encryption that permanently destroys cryptographic keys upon VM termination.
  • Nitro Controller: Acts as the primary interface to the EC2 control plane, coordinating all local Nitro cards, allocating cores/memory, and managing the VM lifecycle.

Offloading virtualization to dedicated PCIe interfaces necessitates an isolated, hardware-enforced mechanism to manage physical device integrity and remote attestation.

Nitro Security and Hardware Root of Trust

• Hardware Interface Monitoring: The Nitro Security chip resides on the server motherboard, actively monitoring localized non-volatile storage buses (SPI and I2C) to detect anomalous behaviors.
• Firmware Write Protection: The security chip structurally prohibits host CPU software from modifying any system firmware; all updates route exclusively through the authenticated Nitro Controller.
• Secure Boot Sequencing:
  • During initialization, the Security chip holds the host CPUs and Baseboard Management Controller (BMC) in reset.
  • The Nitro Controller utilizes a tamper-resistant Trusted Platform Module (TPM) to cryptographically verify the integrity of its boot ROM against known-good signatures.
  • Only upon successful verification of the complete firmware chain are the host processors permitted to execute.
• Ubiquitous Encryption: The architecture enforces transparent encryption across all physical layer interconnects, VPC network traffic, local/remote storage volumes, and management APIs, ensuring that operator access to plaintext user data is physically impossible.

With I/O virtualization and security fully encapsulated by the Nitro subsystem, the primary CPU architecture could be strictly optimized for multi-tenant, cloud-native workload execution.

Graviton Processor Microarchitecture

• Design Philosophy: The Graviton Arm-based CPU family is engineered specifically for cloud cost-performance and high energy proportionality.
• Graviton3 Physical Implementation: Fabricated on a 5 nm process, the processor utilizes a 7-chiplet package containing 64 cores.
  • The topology includes a central compute mesh providing 2 TB/s of bisection bandwidth, flanked by specialized chiplets for memory controllers, PCIe, and I/O offloading.
• Cloud-Native Core Adjustments:
  • Simultaneous Multithreading (SMT) Elimination: The cores omit SMT to prevent cross-thread interference, dedicating entire L1 caches, branch predictors, and Translation Lookaside Buffers (TLBs) to a single thread.
  • Single-Socket Restriction: Hardware is artificially constrained to single-socket configurations, eradicating Non-Uniform Memory Access (NUMA) software complexities while significantly lowering thermal and power envelopes.
• System-Level Density: The server chassis amortizes base infrastructure costs by packaging three independent 1-socket Graviton3 servers that share a unified power supply, BMC, and Nitro card array.
• Memory Subsystem: Integrates DDR5-4800 memory to yield a 50% bandwidth improvement, which sustains load performance and allows for a proportionally smaller Last-Level Cache (LLC).

The structural simplification of the Graviton processor establishes the foundation for realizing maximum energy proportionality and cost-performance across the data center fleet.

Cloud Efficiency and Performance Scaling

• Load Percentile Optimization: Instead of designing for theoretical peak performance, WSC silicon is modeled to achieve peak energy proportionality at the 50th and 90th percentiles of operational load.
• Workload Heterogeneity: Providing architectural diversity (x86 vs. Arm) enables precise hardware matching; specific microarchitectures out-perform others by over 25% depending on whether the workload is integer-heavy or floating-point intensive.
• Economic and Power Yields: For scale-out workloads, the Graviton3 microarchitecture delivers between a $2\times$ to $3\times$ improvement in both performance-per-watt and performance-per-cost when benchmarked against contemporary, general-purpose x86 cloud instances.