IceCube Neutrino Observatory’s Use of High Throughput Computing: Making Discoveries in Astrophysics Using Neutrinos

IceCube captures 233 million cosmic rays every day, opening the door to groundbreaking discoveries in high-energy physics.

Located in the depths of the South Pole almost a mile under the earth’s surface, the IceCube Neutrino Observatory spans one cubic kilometer of ice—roughly the equivalent of one million swimming pools. The Observatory searches for subatomic particles called neutrinos, an elementary particle created by various forms of radioactive decay, and studies some of the most cataclysmic events in the universe—such as exploding stars. The first gigaton neutrino detector ever built, IceCube was primarily designed to observe neutrinos from the most violent astrophysical sources in our universe, allowing scientists to unlock new insights into astrophysics, cosmic rays, dark matter, and more. To do so, the IceCube Observatory captures 233 million cosmic rays every day, opening the door to groundbreaking discoveries in high-energy physics. The immense scale of the work IceCube does is made possible through its collaboration with the Center for High Throughput Computing (CHTC) who provides computing resources, allowing IceCube to analyze the massive amounts of data it collects.

Notoriously difficult to observe due to their physical properties, the study of neutrinos from astrophysical phenomena is the primary focus of IceCube. Taking the innovative approach to study these particles, IceCube detects and studies the particles via light detectors buried in Antarctic ice. The optical clarity of the ice, when interacting with the neutrinos, produces electrically charged particles that can be then captured by IceCube sensors. By detecting neutrinos in this indirect manner, IceCube has the heavy computational workload of converting messages into light patterns that disclose the energy and direction of neutrinos.

This is where the Center for High Throughput Computing (CHTC) comes in. Utilizing CHTC’s HTCondor Software Suite (HTCSS) allows IceCube to run parallel computing jobs, which make processing the data possible. IceCube’s computing manager, Benedikt Ridel notes that “with the help of a public cloud, researchers can quickly get access to large computational resources when they need them.” The data is transported through the cloud and into the resources at CHTC, which run 24 hours a day. On a daily basis, IceCube generates an alert line of low throughput data that provides real-time monitoring of neutrino events. This continuous stream of data feeds directly into the CHTC systems, computing the thousands of daily tasks. In the past year alone, IceCube utilized over 50 million core hours and ran over 36 million jobs using HTCondor.

Each day, IceCube collects one terabyte of unfiltered data and about 100 gigabytes are sent over satellite for analysis. The volume of information makes it difficult for scientists to keep up with the constant stream of data while also staying ahead of time-sensitive discoveries. Riedel adds that, “IceCube has grown organically over time, leading to some challenges in managing this overwhelming amount of data.” Even with CHTC resources, IceCube still faces occasional data backups, which, if relying only on CHTC computing resources, could disrupt other projects. When a high energy event requiring more resources with immediate computational attention occurs, IceCube utilizes the Open Science Pool (OSPool), giving it access to high throughput computing resources from other institutions and universities across the United States.

In 2022, IceCube received the Readers’ Choice Award for Best Use of High-Performance Computing, recognizing its use of cloud technologies and its expansion into the OSPool. This expansion boosted IceCube’s throughput by approximately 40%. According to Francis Halzen, a principal investigator of IceCube, “Without CHTC and OSPool resources, we would simply be unable to make any of IceCube’s groundbreaking discoveries.”

By identifying high-energy neutrinos produced by cosmic phenomena such as supermassive black holes and gamma-ray bursts, IceCube made the breakthrough discovery of detecting neutrinos from outside our galaxy—just two years after the Observatory’s founding in 2013.

Since then, IceCube has provided insights into violent cosmic phenomena like supermassive black holes and gamma-ray bursts by detecting high energy events. One recent finding, using data from high energy neutrinos, was the discovery of the NGC 1068and TXS 0506+056 galaxies, two active galaxies. High data neutrino clumping also led to discovering supermassive black holes in these active galaxies. Additionally, IceCube scientists are mapping properties of Antarctic ice, contributing to the field of glaciology and offering insight on the relationships between climate science history and particle detection.

Before IceCube, there was Amanda, the Antarctic Muon and Neutrino Detector Array, a precursor experiment that laid the groundwork for this larger-scale investigation of neutrinos. IceCube expanded on Amanda’s finding that extremely clear Antarctic ice was suitable for detecting energetic neutrinos, and maintained this expansion through collaboration with CHTC for computing resources.

Despite the advancements that IceCube has made with the help of CHTC, challenges still remain. “We have been around for so long that we have grown organically, which has led to an issue of feeding servers in the most effective way,” says Riedel. Upon starting IceCube, the information was fed into smaller and less complex servers, but the rapid growth required larger servers with an optimized system. Ensuring that data is processed without delays, and addressing bottlenecks that can slow down computations, is crucial to neutrino detection.

The IceCube Neutrino Observatory’s work has already changed our understanding of the universe, offering new insights into phenomena like black holes, gamma-ray bursts, and even dark matter. Today, the IceCube collaboration involves 300 physicists from 57 institutions in 14 countries; and it continues to grow. By partnering with CHTC, IceCube has been able to process and analyze the data it collects at quicker speeds, ensuring up to date discoveries and fueling our understanding of the universe.

Read more about IceCube’s work here.