Cost and Efficiency of Warehouse-Scale Computing
Warehouse-scale computers (WSCs) require massive capital investment and incur high recurring costs. Optimizing the Total Cost of Ownership (TCO) demands maximizing the hardware’s efficiency across power provisioning, utilization, performance, and energy consumption.
Cost of a WSC
The economic viability of a WSC relies on balancing Capital Expenditures (CAPEX) to build the facility with Operational Expenditures (OPEX) to run it.
- TCO Components and Amortization:
- CAPEX: The upfront costs to construct the facility and purchase IT equipment. WSC facilities cost between 13 per Watt to build. CAPEX is amortized over the effective life of the components: facilities over 10–20 years, networking equipment over 4 years, and servers over 3–4 years.
- OPEX: The recurring monthly costs, primarily driven by power consumption, cooling infrastructure maintenance, and personnel.
- Cost Distribution:
- Server purchasing dominates the amortized CAPEX, representing 60% to 65% of the total TCO.
- Networking equipment represents roughly 8% of OPEX and 19% of server CAPEX, a ratio that is not decreasing as rapidly as server costs due to constant demands for higher bandwidth.
- Facility costs account for roughly 10% of the TCO.
- Cost of Power:
- Power and cooling represent more than a third of the OPEX.
- The fully burdened cost of a watt per year—including the amortized infrastructure cost to deliver that watt—can be modeled as:
- This formula yields approximately 2 per watt-year to save energy, barring long-term environmental objectives.
Because power infrastructure dictates a massive portion of WSC costs, extracting the maximum compute capacity from every provisioned watt is the foundation of WSC efficiency.
Efficiency Through Power Provisioning
A WSC facility has a strict maximum power capacity. Deploying the maximum number of servers without exceeding this capacity requires aggressive power provisioning strategies.
- Nameplate vs. Actual Peak Power:
- Manufacturers provide a “nameplate” power rating, indicating the theoretical maximum power a server could draw.
- In practice, actual peak power consumption is often 60% lower than the nameplate rating because real workloads rarely stress all server components (CPU, memory, disks) to their maximums simultaneously.
- Power Oversubscription:
- Basing WSC capacity on actual peak power rather than nameplate power allows for significantly more servers to be deployed.
- WSC operators safely oversubscribe server deployments by up to 40% beyond theoretical power limits, relying on the statistical improbability of all servers peaking simultaneously.
- Mitigating Correlated Power Spikes:
- Oversubscription requires strict monitoring to prevent rack or facility-wide power failures during correlated load spikes.
- Hardware and software controllers mitigate these spikes by:
- Reducing server clock frequencies.
- Drawing temporary power from rack-level backup batteries.
- Pausing or descheduling low-priority tasks.
- Migrating tasks across racks, PDUs, or other WSCs to balance the load.
- Domain-Specific Architectures (DSAs): Deploying DSAs for targeted workloads, like machine learning, delivers two orders of magnitude higher performance per watt than general-purpose CPUs. This drastically reduces the need to build new WSC facilities, easily offsetting the 100 million custom ASIC development costs.
Deploying the maximum number of servers under a power budget is only economically effective if those servers are kept busy performing useful work.
Efficiency Through Higher Utilization
Because amortized server CAPEX heavily dominates TCO, running servers at high utilization is mandatory to recover the investment.
- Utilization Profiles:
- Mixed Workloads: Clusters running a mix of online interactive services and batch applications typically average low utilization, ranging from 10% to 50%.
- Batch Workloads: Clusters dedicated to large, continuous batch processing exhibit low demand variation and achieve 50% to 85% average utilization.
- Causes of Low Utilization:
- Interactive workloads inherently vary by time of day.
- Developers systematically oversize their resource requests to avoid performance interference from co-scheduled workloads and to mitigate hardware heterogeneity within the WSC.
- Techniques for Increasing Utilization:
- Right-sizing and Autoscaling: Automatically adjusting the resources granted to workloads based on real-time needs.
- Interference-aware Scheduling: Tracking workload conflicts and adjusting placements to minimize contention.
- Resource Harvesting: Utilizing idle reserved capacity to run low-priority, preemptable batch analytics (e.g., spot instances).
- Hardware Isolation: Employing cache and memory bandwidth partitioning, alongside networking and storage QoS, to guarantee performance isolation for co-scheduled tasks.
While consolidating workloads drives up utilization and lowers costs, it risks introducing latency spikes that directly degrade user-facing performance.
Efficiency Through Higher Performance
Performance in a WSC translates directly to revenue. High latency degrades user satisfaction, reducing productivity and interaction rates.
- The Impact of Latency:
- User productivity is inversely proportional to interaction time, which consists of human entry time, system response time, and human think time.
- Minor server delays (e.g., an additional 500 ms) significantly drop user query volumes and revenue per user.
- Tail-Tolerance:
- WSC developers must prioritize the 99th percentile of latency (the “tail”) over the average response time. If 99% of requests are fast but the final 1% is slow, users experiencing the slow tail may abandon the service.
- Unpredictable latency spikes are caused by shared resource contention, queuing delays, variable processor states (like DVFS/Turbo mode), and software garbage collection.
- Instead of attempting to eliminate all hardware variability, WSCs utilize software techniques, such as fine-grained load balancing, to swiftly shift small tasks between servers and mask temporary latency spikes.
- Data Locality:
- Performance heavily depends on data placement. DRAM bandwidth drops dramatically across the network hierarchy: intra-server bandwidth is 40 times greater than intra-rack bandwidth, which is 4 to 10 times greater than intra-row bandwidth.
Maintaining strict latency targets at high performance requires substantial power, but maintaining that same power draw when load drops ruins overall system efficiency.
Efficiency Through Energy Proportionality
An ideal WSC server exhibits energy proportionality: it consumes energy directly in proportion to the amount of work it performs, and consumes near-zero power when idle.
- Historical Inefficiency: In 2007, an idle computer consumed 60% of its peak power, and 70% of its peak power at only 20% load.
- Modern Improvements: Modern servers have improved significantly, consuming approximately 20% of peak power at idle and 50% of peak power at 20% load.
- Power Supply Losses: Server power supply units (PSUs) convert high AC/DC voltages to the lower voltages needed by chips. Historically, these PSUs were 60% to 80% efficient and were particularly inefficient at low loads (<25%). Voltage regulator modules (VRMs) on motherboards introduce further efficiency losses.
- Software-Driven Proportionality:
- Traditional operating systems maximize resource usage (e.g., using all available memory for caches) regardless of energy cost.
- Modern WSC software architects deploy dynamic controllers that enforce fine-grained power management, tuning servers to barely meet Service Level Objectives (SLOs).
- This software-driven throttling yields 20% to 30% energy savings when servers operate in the 10% to 60% utilization range, synthetically improving the system’s energy proportionality.