X

Up close with America's national supercomputer, Blue Waters (pictures)

Researchers in need of serious supercomputing power now have one of the most powerful computers in the world at their disposal. CNET Road Trip 2013 checked out Blue Waters.

New_CNET_profile_picture.jpg
New_CNET_profile_picture.jpg

Daniel Terdiman

See full bio
Inside_tape_storage_cabinet.jpg
1 of 20 Daniel Terdiman/CNET

Deep inside tape storage

CHAMPAIGN, Ill.--While those with high security clearances sometimes have access to powerful government supercomputers, academics, scientists, and engineers often have not. That's why the National Center for Supercomputing Applications launched Blue Waters, one of the world's most powerful supercomputers, in March. Funded with an initial $208 million grant and a five-year operations grant of $151 million by the National Science Foundation, Blue Waters was installed at the National Petascale Computing Facility at the University of Illinois at Champaign-Urbana.

With the capacity of completing in excess of a quadrillion calculations per second on a sustained basis -- and ten times that at peak performance -- Blue Waters was meant to be ideal for providing the scientific community with the processing speed, data storage, memory, and communication necessary to take on many of the most complex computing tasks.

The supercomputer comprises 22,640 Cray XE6 nodes and 3,072 Cray XK7 nodes with NVIDIA graphics processor acceleration. The XE6 nodes have 64 GB of memory per node, while the XK7s each have 32 GB of memory. Blue Waters also boasts a quick online storage capacity of 26 petabytes, and 380 petabytes of longer term tape data storage.

This is a look inside one of the Spectra T-Finity cabinets that make up the 380 petabytes of long term storage.

Blue_Waters_whole_system.jpg
2 of 20 Daniel Terdiman/CNET

20,000 square feet of space

The National Petascale Computing Facility was built with 20,000 of usable space for Blue Waters. The facility was built with biometric scanners for security -- so that it can compete for future applications with high-security operations elsewhere -- and can withstand an F3 tornado.

cables_above.jpg
3 of 20 National Center for Supercomputing Applications/University of Illinois

Masses of cables

It takes a huge amount of cables to power and connect all the Blue Waters nodes, as this look at the wiring infrastructure underneath the supercomputer reveals.

Blue_Waters_rear_open_close-up.jpg
4 of 20 Daniel Terdiman/CNET

Blue Waters wiring

Blue Waters consists of 12 rows of 24 Cray cabinets, including this one, seen from the back with its wiring on display.

Blue_Waters_blades.jpg
5 of 20 Daniel Terdiman/CNET

Blue Waters blades

Each of the Blue Waters cabinets has 24 blades arrayed in three rows of eight blades each.

Blue_Waters_down_a_row.jpg
6 of 20 Daniel Terdiman/CNET

Looking down a row

There are 12 rows of 24 Cray cabinets in the Blue Waters configuration.

Cray.jpg
7 of 20 Daniel Terdiman/CNET

Cray

Cray has a number of employees on-site at the National Petascale Computing Facility in order to service and maintain Blue Waters.

Tape_storage_1.jpg
8 of 20

Tape storage

For long-term storage needs, the National Petascale Computing Facility provides Blue Waters users with 380 petabytes of tape storage.

Blue_Waters_cooling_piping_silver.jpg
9 of 20 Daniel Terdiman/CNET

Cooling piping

Each of Blue Waters' rows has three cooling cabinets at the end, which are connected to the computing nodes via the silver piping on the top.

Cray_Sonexion_Cabinets_-_closeup.jpg
10 of 20 Daniel Terdiman/CNET

Sonexion cabinets

In addition to the 380 petabytes of long-term tape storage available to Blue Waters users, there is also 26 petabytes of short-term, online storage available via these Cray Sonexion machines.

Cabling_under_floor.jpg
11 of 20 Daniel Terdiman/CNET

Cabling under the floor

Opening up the floor in the Blue Waters room reveals massive amounts of cables and wires that connect the various nodes.

Pump_room_1.jpg
12 of 20 Daniel Terdiman/CNET

Pump room

In order to operate Blue Waters, the National Petascale Computing Facility maintains these massive cooling pipes, which altogether circulate 5,400 tons of chilled water with a capacity of 9,300 gallons per minute.

Pump_room_with_heat_exchangers.jpg
13 of 20 Daniel Terdiman/CNET

Heat exchangers

Water is pumped through giant sumps and then to heat exchangers (on the left side, on the floor) in order to provide Blue Waters with enough cooling water.

Transformer.jpg
14 of 20 Daniel Terdiman/CNET

Transformer

In order to provide Blue Waters with enough power to stay operational 24 hours a day, the National Petascale Computing Facility has 20 transformers -- of which 14 are dedicated to the supercomputer. The system has a capacity of 13,800 volts coming in, and it is approved for a total of 24 megawatts, although it is currently only using about 13.5 megawatts. That means the facility can almost double in size if needed.

Electrical_room.jpg
15 of 20 Daniel Terdiman/CNET

The electrical room

The NPCF has a total of 20 giant transformers, 14 for Blue Waters, two spares, two which serve non-Blue Waters systems, and two which serve the facility itself.

Sump.jpg
16 of 20 Daniel Terdiman/CNET

Sump

The NPCF cooling system includes several of these giant sumps.

Tape_storage_cabinet_info_display.jpg
17 of 20 Daniel Terdiman/CNET

Tape storage information display

This display provides technicians with a quick view about the status of the 380-petabyte tape storage system.

Cooling_cabinet_info_display.jpg
18 of 20 Daniel Terdiman/CNET

Cooling cabinet information display

This display provides information about the status of the cooling cabinets.

Cooling_piping_under_floor.jpg
19 of 20 Daniel Terdiman/CNET

Cooling piping under the floor

There is a six-foot deep well under the floor of the supercomputer room at the NPCF, inside of which is a network of these large cooling pipes.

Whole_room.jpg
20 of 20 Daniel Terdiman/CNET

Whole room

A look at the 20,000 square feet of space available for supercomputers at the National Petascale Computing Facility at the University of Illinois at Champaign-Urbana. Though Blue Waters is already one of the world's most powerful supercomputers, the facility has room to expand.

More Galleries

'Stranger Things' Stars Then and Now: Wow, They've Really Grown Up

More Galleries

'Stranger Things' Stars Then and Now: Wow, They've Really Grown Up

40 Photos
2022's Best TV Shows You Can't Miss on Netflix, HBO, Disney Plus and More

More Galleries

2022's Best TV Shows You Can't Miss on Netflix, HBO, Disney Plus and More

95 Photos
Movies Coming in 2022 From Marvel, Netflix, DC and More

More Galleries

Movies Coming in 2022 From Marvel, Netflix, DC and More

81 Photos
Astronomy Photo Contest Finalists Will Take Your Breath Away

More Galleries

Astronomy Photo Contest Finalists Will Take Your Breath Away

36 Photos
The 40 Absolute Best Games on Nintendo Switch

More Galleries

The 40 Absolute Best Games on Nintendo Switch

41 Photos
The Dodge Hornet SUV Will Be a Rebadged Alfa Romeo Tonale

More Galleries

The Dodge Hornet SUV Will Be a Rebadged Alfa Romeo Tonale

8 Photos
2022 Porsche 911 Carrera GTS Coupe: Just Right

More Galleries

2022 Porsche 911 Carrera GTS Coupe: Just Right

36 Photos