1.5 Million Threads: AMD-Powered Archer 2 Supercomputer With 12,000 EPYC Rome CPUs

What would you do with 1.5 million threads of computational horsepower? The UK Research and Innovation (UKRI) center has an idea.

The Archer supercomputer system based in Edinburgh was first introduced in 2013 based on the Cray XC30 design and has since been in use by researchers in need of serious compute power. It was once among the top 20 supercomputers in the world, but that’s no longer the case and the time has come to replace it.

As such, the center announced that it has contracted with Cray to build the Archer 2 supercomputer, which will be based on AMD EPYC Rome processors.

The center will build the system out of 5,848 Shasta compute nodes (deep dive here). Each node will contain two second-gen AMD EPYC Rome processors with 64 cores apiece, with each clocked at 2.2GHz (likely the base frequency).

This adds up to 11,969 CPUs, and a grand total of 748,544 CPU cores and about 1.5 million threads (we rounded that number up simply because we’ve always wanted to).

Depending on how it is measured, averaged out over a handful of benchmarks, the new supercomputer will perform an average of 11 times faster than its predecessor.

Full technical specifications as posted on HPC Wire below:

• Peak performance estimated at ~ 28 PFLOP/s
• System Design:
• 5,848 compute nodes, each with dual AMD Rome 64 core CPUs at 2.2GHz, for 748,544 cores in total and 1.57 PBytes of total system memory
• 23x Shasta Mountain direct liquid cooled cabinets
• 14.5 PBytes of Lustre work storage in 4 file systems
• 1.1 PByte all-flash Lustre BurstBuffer file system
• 1+1 PByte home file system in Disaster Recovery configuration using NetApp FAS8200
• Cray next-generation Slingshot 100Gbps network in a diameter-three dragonfly topology, consisting of 46 compute groups, 1 I/O group and 1 Service group
• Shasta River racks for management and post processing
• Test and Development System (TDS) platform, to be installed in advance
• Collaboration platform with 4 x compute nodes attached to 16 x Next Generation AMD GPUs
• Software stack:
• Cray Programming Environment including optimizing compilers and libraries for the AMD Rome CPU
• Cray Linux Environment optimized for the AMD CPU blade based on SLES 15
• Shasta Software Stack
• SLURM work load manager
• CrayPat as profiler
• GDB4HPC as debugger

The center will install the Archer 2 supercomputer in the same room as the supercomputer it is replacing, meaning there will be a period of downtime. Archer will be shut down on February 18, 2020, and Archer 2 will be booted up 78 days later on May 6, if all goes according to schedule.

However, after bootup, the system will first go through a one-month stress testing period before the service will be open to researchers. Some researchers might be allowed to access the system during this testing phase, but the UKRI did say that access would be ‘limited.’

The new win builds on an ever-expanding list of EPYC Rome supercomputer wins, including the recent announcement of the exascale-class Frontier supercomputer.