Distributed computing and sophisticated new software will allow radio astronomers to study the cosmos with improved vision
High performance computing no longer requires having access to the latest model supercomputer. Instead, the combined brainpower of dozens, hundreds or even thousands of fast processors linked together can solve the most difficult problems. While in the past, astronomers had to rely on expensive supercomputers for big computational projects, today the large collection of smaller workstations connected together on the Internet offers a huge resource of unused computer cycles. With the proper software, big applications could automatically search the Internet for available computers, and assign computational tasks to those processors that are free-the entire network becomes the computer.
Radio astronomers at the Naval Research Laboratory and the National Radio Astronomy Observatory are working toward making this vision of distributed computing a reality in order to solve large scientific problems. Tomorrow, the same tools needed for scientific scalable computing may be used in other applications in as-yet-unimagined ways. Computer cycles could be bought and sold like electricity. Small users could run intensive applications, paying only for computing when they really need it. Owners of large computers with lots of available capacity would be compensated, thus improving the potential return on their hardware investment.
In addition to large computers, radio astronomers need large telescopes to map the radio universe. Complex calculations must be performed to transform the detected faint radio signals into beautiful maps like the wide-field image (large panoramic) shown on pages 18-19. One of the principal instruments used to map our radio universe is the Very Large Array of radio telescopes located on the Plains of San Augustin, west of Socorro, New Mexico.
Maintained by the National Radio Astronomy Observatory, which is operated by Associated Universities, Inc. under contract with the National Science Foundation, the Very Large Array consists of an array of twenty-seven 25-meter radio telescopes spread out over 35 kilometers of high desert. The Very Large Array's multiple antennas separated by large distances allow astronomers to view objects with very high resolution using large computers. The Very Large Array regularly makes detailed observations of exploding stars, molecular clouds, supernova remnants, pulsars, radio galaxies, active galactic nuclei and even cosmic remnants from the Big Bang. The Very Large Array is now nearly 100 times more efficient at observing radio signals than it was at the time of its dedication in 1980, primarily due to the advances in computing, both hardware and software. Still, to make use of the full capability of the Very Large Array in creating the most demanding wide-field images, more computing power is necessary.
High Performance Computing
Historically, increases in computing performance have allowed astronomers to incorporate more complex algorithms and work with larger data sets, which improves the fidelity and the scientific return of the final images. The continual increase in performance of microprocessors over the last few decades has meant that the cost of computing has decreased commensurately. Yet, for very large problems the speed of a single modern microprocessor is still not enough. Large scale parallel computers (or large networks of interconnected workstations) are needed to solve the most demanding applications, such as the wide-field imaging problem. The growth and application of distributed computing is likely to make huge strides forward due to government support for the High Performance Computing and Communication (HPCC) program.
The HPCC program is part of the National Information Infrastructure (NII) Initiative announced by President Clinton and Vice President Gore on February 22, 1993. This initiative is more popularly known as the "Information Superhighway." The HPCC component of the program follows a three-pronged approach to bring about greater utilization of high-end computer technologies. First, the program supports the development of large computing systems that take advantage of the commodity pricing of microprocessors by building scalable computers from inexpensive microprocessors. Second, the government supports an advanced high speed network for rapid communication that will be deployed and available on a general basis nationally. Third, the initiative makes a commitment to the development of scalable software that can operate seamlessly on different computers over the networks.
Together, these three components are designed to provide supercomputing capabilities to a variety of new users in order to enhance national security, education, healthcare and improve our global environment, among other things. Effectively, the current generation of large parallel computers will help stimulate the growth of distributed computing, but first we need to develop the applications.
The Software Crisis
The development of powerful software tools that can be used and shared across all the high performance platforms and over networks of distributed computers is the weak link in the HPCC triad. To date, the high performance computing programs have successfully focused on the hardware and networking aspects of the initiative, but software is the critical component in making optimal use of these assets. In fact, the software needed has two distinct components to it: development tools and end-user applications. HPCC funds supported a number of experimental tools for use in distributed computing environments. Unfortunately, less support has been available for end-use application development. Program officers and policy makers have felt that software development should be left to the commercial market place, or at least supported by the program sponsors that require the applications. However, program sponsors are reluctant to pay for software development on specialized platforms that may have a very short product lifetime. Furthermore, commercial software vendors have little interest in writing code for traditional supercomputers where the number of platforms on which they could install their product is small.
The short-term fulfillment of the promise of high performance computing remains cloudy, primarily due to the lack of good software. Each of the major high performance computer vendors has developed software solutions that take advantage of their proprietary hardware systems. The result is that different styles of programming languages and environments are supported on different high performance platforms. This makes the problem of porting codes between different systems difficult even on systems of similar architectural topologies. The often noted phenomenon that the half-life of a supercomputer company is shorter than the time needed to develop a typical supercomputer application only exacerbates the problem.
Developing New Software Tools for Radio Astronomy
The software problem is now resulting in an application crisis in the field of radio astronomy. Many imaging computations that astronomers want to tackle are now larger than can be accomplished on a single workstation. Applications are not being developed rapidly for use in distributed computing or high performance computing environments, because of a lack of good tools to support cross-platform code development. Research scientists at the National Research Laboratory with experience in high performance computing are working to overcome the applications software crisis. New codes are being developed for a variety of high performance systems that provide a seamless scaling from single processors to dozens, hundreds or even thousands.
The goal of the National Research Laboratory scalable software development effort is to create a truly distributed computing environment where the application uses the available resources on the network, whenever and wherever they are available. By coupling the applications development with the tool development, both efforts will better serve the entire high performance computing community. In the end, the same technologies that enable radio astronomers to improve our ability to image the radio universe with unprecedented resolution will help individuals all over the globe take advantage of the next generation of high performance computers-distributed computers running on the Internet.
The mission of the NRL Remote Sensing Division is research and development utilizing remotely sensed information or leading to remote sensing systems for terrestrial applications in the broadest sense, including ocean, atmosphere and space environments. See the home page at http://rsd-www.nrl.navy.mil/ for a sample of some of the Division's activities. Computing resources were provided by the Department of Defense High Performance Computing Modernization program.
Roger S. Foster is a radio astronomer in the Remote Sensing Division at the United States Naval Research Laboratory in Washington, DC. [email protected]