As an observer of the AI industry, I’ve been struck by how rapidly the landscape is evolving – not just in model capabilities, but in the hardware and infrastructure that underpin these advances. The last few months have seen record-breaking AI chip performance, new approaches to data center design (to handle unprecedented power and cooling demands), and a growing spotlight on the energy footprint of large-scale AI. Below, I’ve compiled a rundown of key developments from late 2024 through spring 2025.

AI Chip Performance: Faster and More Efficient Than Ever

The market for AI chips in data centers hit new highs in Q4 2024, with quarterly sales reaching $32.6 billion (a 22% jump over the previous quarter). Demand for AI-optimized compute is “unprecedented,” and GPUs continue to dominate – accounting for 87.6% of that revenue (about $28.5 B, +24.8% QoQ).

NVIDIA solidified its lead, capturing ~85% of data-center AI chip share (its Q4 data-center revenue was up 93% YoY) while AMD trailed at ~6%. This NVIDIA supremacy is underscored by the ubiquity of its high-end GPUs (like the H100) in new AI deployments. At the same time, custom silicon is on the rise: industry giants are investing in tailor-made AI chips to boost performance-per-watt and reduce dependency on NVIDIA.

Google’s TPUs, Amazon’s Trainium, and Microsoft’s rumored Athena/Maia accelerators are all examples of in-house designs now supplementing or replacing off-the-shelf GPUs.

New chips push the envelope

Recent months brought a wave of next-generation AI chips promising higher speed and efficiency. NVIDIA announced an intermediate upgrade to its flagship GPU – the “H200” – which uses faster HBM3e memory to boost bandwidth by ~43% (to 4.8 TB/s) and increase VRAM from 80GB to 141GB.

This mid-generation refresh is aimed at keeping GPUs fed with data for gargantuan models (the largest language models were already maxing out the 80GB H100 memory).

AMD, for its part, began shipping the Instinct MI300X GPU (128 GB HBM3) and previewed a MI325X with a whopping 288 GB of next-gen HBM3E memory and 6 TB/s bandwidth.

This “memory monster” approach lets AI accelerators handle larger models in-memory, improving efficiency by reducing data transfers. AMD also outlined an aggressive roadmap (new architectures in 2025 and 2026) to close the gap with NVIDIA – an ambitious plan given NVIDIA’s head start.

Meanwhile, Intel officially launched its Gaudi 3 AI accelerator, targeting cost-efficient training for large models. Gaudi 3 delivers 4× the FP16/BF16 compute and 1.5× the memory bandwidth of its predecessor.

Notably, Intel claims Gaudi 3 can train models ~50% faster than NVIDIA’s H100 on popular large-language benchmarks (Llama-2 7B/13B and GPT-3 175B) , and likewise achieve 50% higher throughput in inference – a bold claim aimed at breaking NVIDIA’s monopoly.

If real-world tests support these figures, it suggests a meaningful efficiency win (achieving more AI performance per dollar, or per watt) outside the NVIDIA ecosystem.

It’s astounding how quickly AI chip capabilities are leaping forward. In just 18 months, we’ve seen performance per accelerator roughly double or triple in many cases. Equally important is the focus on efficiency – nearly every vendor is talking about performance-per-watt and memory optimization, not just raw FLOPs. Google, for instance, unveiled its 7th-gen TPU “Ironwood” in April 2025, designed specifically for efficient large-scale inference. Ironwood delivers nearly 2× better performance-per-watt than Google’s prior TPUs and uses liquid cooling to sustain maximum throughput. In fact, Google says Ironwood is 30× more energy-efficient than their first-gen Cloud TPU from 2018 – a staggering improvement in five years.

All these developments underscore a broader trend: as AI models demand more compute, the industry is racing to innovate chips that are not just faster, but also more tailored and power-conscious than ever. It’s an exciting arms race to watch, with huge implications for what AI can do (and how accessible it becomes) in the coming years.

Data Center Infrastructure: Cooling, Density and Modular Designs

The surge in AI hardware capability comes with a literal surge in power density. Today’s top GPUs can consume 2–3× more power than those of a few years ago, which means 2–3× more heat to dissipate.

For example, where a 2020-era AI server might have drawn 3–5 kW per rack, some AI racks now easily demand 30+ kW, and upcoming designs could exceed 150 kW per rack.

This is stretching the limits of traditional air cooling. In response, data centers are widely adopting liquid cooling techniques to keep these hot-running chips cool. Industry reports note that in new high-density deployments, a hybrid cooling approach is common – roughly 70% liquid cooling (via cold plate loops or rear-door heat exchangers) with only 30% air.

Many new server racks now come with direct-to-chip liquid coolers by default, and even existing data centers are retrofitting liquid cooling to handle AI workloads. Immersion cooling – literally submerging servers in special dielectric fluid baths – is emerging as the next step once rack densities climb beyond ~100 kW. While still niche (under 10% of data centers use immersion today), we’re seeing pilot implementations in AI-centric facilities.

Operators report benefits like extremely efficient heat removal and lower overall PUE, though they also face challenges (fluid maintenance, tank weight requiring reinforced floors, etc.).

In short, cooling infrastructure is in a state of rapid evolution, driven by the physics of high-performance AI gear. What was once exotic (liquid cooling) is quickly becoming standard. As one real-estate advisory noted, “NVIDIA’s latest AI chips consume up to 300% more power than their predecessors“ – so data centers must pivot technologies just to keep up.

Advanced cooling isn’t just about efficiency; it’s about enabling performance. (Case in point: Google has said its liquid-cooled pods can run 2× the performance of air-cooled setups before hitting thermal limits).

Denser, smarter, more modular.

Alongside cooling upgrades, the industry is rethinking facility design to handle AI at scale. One trend is toward modular data centers – essentially prefabricated units (sometimes even in shipping container form) that can be rapidly deployed and scaled. This quarter saw, for example, Vertiv launch an all-in-one prefab system combining power distribution, liquid cooling piping, and heat management in a single modular kit.

What was the result? Data hall build-outs that can be done up to 85% faster than traditional construction, adding over 1 MW of IT load per day with a single crew. Such speed is a huge boon when cloud providers are racing to add capacity for AI clusters.

In an industry where a typical brick-and-mortar data center can take 12–18 months to build, cutting deployment time to a few months or weeks is game-changing.

These modular approaches also often integrate innovative cooling (i.e. built-in liquid loops) and can be placed nearer to power sources or users as needed – providing agility in location choice. Meanwhile, data center operators are beefing up power delivery and networking within facilities to accommodate massive “AI superclusters.”

At SC’23 in late 2024, NVIDIA detailed Project Jupiter, a supercomputer for Europe with 24,000+ GPU nodes and 18 MW power draw.

Building something of that scale means engineering for extreme rack density, robust interconnects (Jupiter uses advanced InfiniBand and NVLink networks), and power provisioning on par with a small town. Even on a smaller scale, many cloud data centers are now evaluating their electrical and cooling distribution, ensuring each rack can get tens of kilowatts, and that backup power and heat removal can scale accordingly.

It’s remarkable (and a bit alarming) how data center infrastructure is being pushed to its limits by AI. Five years ago, a 30 kW rack was rare; now we’re talking about designs for 5× that load. The upside is a wave of engineering innovation in cooling and power. I find it fascinating that technologies like immersion cooling, which were experimental not long ago, are now viewed as viable mainstream solutions – essentially, out of necessity.

There’s also a spirit of pragmatism: instead of huge bespoke buildings, the focus is on modularity and speed. If AI demand is skyrocketing, you can’t spend years building every data center. The industry’s answer is to standardize and pre-build components, whether that’s factory-assembled power skids or entire containerized server farms.

This agility is encouraging, but it also means a lot of uncharted territory: how to maintain and operate these new cooling systems at scale, how to train staff for them, and how to ensure reliability when pushing equipment so hard. In essence, the backbone of the AI revolution – the physical servers and facilities – is quietly undergoing its own revolution. It may not grab headlines like ChatGPT did, but without these behind-the-scenes leaps in data center design, the AI boom would quite literally overheat.

The Energy Impact: AI’s Growing Power Appetite

The rapid scaling of AI has a direct consequence: energy consumption is climbing steeply. Data centers worldwide consumed an estimated 415 TWh of electricity in 2024, about 1.5% of global usage.

AI is now a major driver of growth on top of that base load. The International Energy Agency (IEA) projects that by 2030, data center demand will more than double, reaching ~945–1050 TWh (which is roughly equal to Japan’s entire power consumption today).

A significant portion of this increase is attributed to AI workloads and the specialized hardware running them.

In fact, the IEA’s new “Energy and AI” report (April 2025) notes that electricity demand from AI-specific data centers could quadruple by 2030 if current trends continue.

To put it in perspective, a single large AI training cluster can consume as much power as 100,000 homes, and some planned hyper-scale AI data centers might use 20× that – putting them in the league of industrial plants like steel mills or aluminum smelters in terms of energy draw.

These comparisons, once startling, are increasingly reality. For example, Meta’s newest AI data center and Microsoft’s Azure AI super-computing clusters each are reported to plan for hundreds of megawatts of capacity. The local grid impacts are already being felt: regions with a high concentration of data centers (Northern Virginia in the US, Dublin in Ireland, etc.) have seen grid strains and rising power prices, prompting regulators to sometimes pause new facility approvals.

Regional Power Profiles

The energy footprint of AI is distributed unevenly around the world. The United States currently leads in data center power use – accounting for about 45% of global data center electricity in 2024.

China is second (~25%), and Europe third (~15%), with the rest of the world making up the balance.

This means the U.S. and China together consume well over half of all data center energy. In the U.S., the growth is so rapid that by 2030, running data centers (much of it for AI) is expected to draw more power than all heavy industries (like steel, cement, aluminum, chemical manufacturing) combined.

Advanced economies in general are seeing data centers become a top-tier electricity consumer – for instance, in Japan, data centers could account for over 50% of all new power demand growth through 2030.

By contrast, in China the overall grid is still dominated by industrial manufacturing; data centers there are projected to be <10% of new demand in the near term.

Even so, China’s AI push means its data center energy use could triple to ~600 TWh by 2030 (Goldman Sachs estimate) from ~200 TWh today.

Regional policy differences are emerging

Europe has taken a proactive stance on curbing data center energy and emissions. The EU is rolling out a sustainability rating system for data centers and, under its Energy Efficiency Directive, now requires large data centers (>500 kW) to report their energy use, efficiency, and waste-heat metrics annually.

The goal is to increase transparency and set minimum efficiency standards by late 2025.

Some European countries (and U.K.) are also incentivizing heat reuse projects – for example. capturing server waste heat to warm local buildings – to improve overall energy ROI.

In Asia, Singapore just lifted a moratorium on new data centers but is enforcing a strict PUE ≤1.3 efficiency requirement and mandating adoption of liquid cooling by 2025. China’s government has set targets for average PUE <1.5 by this year and is pushing large cloud operators to boost renewable energy usage by 10% annually.

In the U.S., there aren’t federal mandates yet – though discussions are underway. The U.S. Dept. of Energy and industry groups are watching Europe’s moves closely. For now, American tech companies are mostly self-regulating, often committing to carbon neutrality and buying renewable power, but without binding rules specific to data center efficiency.

Energy solutions and sustainability

The surging electricity appetite of AI is prompting creative solutions. One notable trend is interest in clean power procurement dedicated to data centers.

In 2024 we saw multiple long-term power purchase agreements (PPAs) signed with renewable farms, and even a few exploratory deals for nuclear energy.

Some hyperscalers are eyeing advanced nuclear reactors (including small modular reactors, SMRs) as a stable, zero-carbon source to feed energy-hungry AI facilities by the early 2030s.

While SMRs are still developmental, existing nuclear plants are already being tapped – for example, several U.S. data center operators agreed to offtake power from nuclear plants slated for 2028 restart.

On the efficiency side, there’s a continuous push to improve the power usage effectiveness (PUE) of AI centers despite higher densities. Techniques like AI-driven workload scheduling (to avoid idle power waste) and improved chip-level power management are being employed to mitigate the growth in consumption.

The server hardware itself is also getting more efficient generation by generation (as seen with Google’s 2× perf-per-watt TPU improvement and NVIDIA/AMD’s focus on performance per watt in new GPUs).

These gains are crucial – the IEA emphasizes that outcomes to 2030 vary widely depending on efficiency: in a high-efficiency scenario, data center demand might grow more slowly, whereas in a pessimistic case (no big efficiency improvements) it could nearly triple (IEA’s “Lift-Off” scenario ~1300 TWh by 2035).

The numbers we’re now seeing in terms of AI’s power consumption are eye-opening. It’s one thing to celebrate that we can train incredible AI models – but when I read that an AI data center can equal the power draw of a mid-sized city, it certainly gives pause.

There is a real balancing act in progress: on one hand, AI capabilities are racing ahead; on the other, sustainability concerns are mounting.

I find it encouraging that many stakeholders (from policymakers in the EU to engineers tweaking PUE) are treating this seriously. The flurry of new energy-efficiency mandates and green power projects shows the industry acknowledges its responsibility.

Yet, it’s a global challenge with disparate approaches. The U.S. and China are in an AI hardware arms race, which could mean huge energy growth, while Europe tries to moderate impact via regulation.

As someone who loves the promise of AI, I’m hopeful we’ll innovate our way to a happy medium – where we can have trillion-parameter models and a low-carbon grid. In any case, the next few years will be pivotal.

The “AI gold rush” is no longer just about algorithms and chips; it’s also about electrical engineering, climate strategy, and international cooperation. Watching how each region tackles the compute vs. consumption conundrum will be just as fascinating as the tech itself, and it will likely shape the narrative around AI’s role in society moving forward.