NASA Manages Deluge of “Big Data” for Global Benefit

The center of the Milky Way galaxy as imaged by NASA's Spitzer Space Telescope.

Washington — For NASA and its dozens of missions, data pour in every day like rushing rivers. Spacecraft monitor everything from our home planet to faraway galaxies, beaming images and information to Earth. All those digital records need to be stored, indexed and processed so engineers, scientists and people across the globe can use the data to understand Earth and the universe beyond.

Mission planners and software engineers are coming up with new strategies for managing the ever-increasing flow of such large and complex data streams, referred to in the information technology community as “big data.”

How big is big data? For NASA missions, hundreds of terabytes are gathered every hour. Just one terabyte is equivalent to the information printed on 50,000 trees worth of paper.

“Scientists use big data for everything from predicting weather on Earth to monitoring ice caps on Mars to searching for distant galaxies,” said Eric De Jong of NASA’s Jet Propulsion Laboratory in Pasadena, California, principal investigator for NASA’s Solar System Visualization project, which converts NASA mission science into visualization products that researchers can use. “We are the keepers of the data, and the users are the astronomers and scientists who need images, mosaics, maps and movies to find patterns and verify theories.”

De Jong explains that there are three aspects to wrangling data from space missions: storage, processing and access. The first task, to store or archive the data, is naturally more challenging for larger volumes of data. The Square Kilometer Array, a planned array of thousands of telescopes in South Africa and Australia, is expected to produce 700 terabytes of images every day, equivalent to all the data flowing on the Internet every two days. Rather than build more hardware, engineers are busy developing creative software tools to better store the information.

“We don’t need to reinvent the wheel,” said Chris Mattmann, a principal investigator for NASA’s big-data initiative. NASA has been increasingly bringing open-source software into its fold, creating improved data processing tools for space missions. The NASA tools then go out into the world for others to use for different applications.

De Jong and his team are developing new ways to visualize the information. Each image from one of the cameras on NASA’s Mars Reconnaissance Orbiter, for example, contains 120 megapixels. His team creates movies from data sets like these, in addition to computer graphics and animations that enable scientists and the public to get up close with the red planet.

“Data are not just getting bigger but more complex,” De Jong said. “We are constantly working on ways to automate the process of creating visualization products, so that scientists and engineers can easily use the data.”

Another big job in the field of big data is making it easy for users to grab what they need from the data archives.

“If you have a giant bookcase of books, you still have to know how to find the book you’re looking for,” said Steve Groom, manager of NASA’s Infrared Processing and Analysis Center. Sometimes users want to access all the data at once to look for global patterns, a benefit of big data archives. “Astronomers can also browse all the ‘books’ in our library simultaneously, something that can’t be done on their own computers,” Groom said.

Ultimately, NASA said, the tide of big data will continue to swell, and NASA will develop new strategies to manage the flow for the world.