Scientific Applications:
New Obstacle Scattering Numerical Model Running on ENEA Grid
By Silvio Migliori (ENEA), Giovanni Bracco (ENEA) and Francesco Zirilli (UniversitÃ di Roma La Sapienza)
1. Introduction
As you are flying on an airplane approaching for landing in a busy
airport, you feel secure thanks to the radar system that governs the
air traffic. In an apparently very different situation, you might feel
secure about your health condition after hearing an expert opinion on
the output of medical tests (e.g., ecography, X-ray imaging) on your
body. The common scientific ground of the technologies mentioned above,
and of many others not mentioned, is wave propagation and scattering.
The waves used may be of different nature. For example acoustic waves
are used in ecography, electromagnetic waves are used in radars and
X-ray imaging, and elastic waves are used in geological prospection.
The scattering phenomena under scrutiny are those generated by the fact
that these waves propagating in a suitable medium find on their ways
"obstacles" (e.g., an organ of the human body in medical ecography, an
airplane in air traffic control or a discontinuity in the physical
properties of the underground in geological prospection). While most
people do not know or do not think about it, high performance computing
(HPC) is a fundamental tool in advancing our knowledge of wave
propagation and scattering phenomena.
The use of HPC is based on the existence of mathematical models that
describe wave propagation and scattering phenomena. The main
mathematical models of wave propagation go back to 19th century
mathematical physics, for example, the wave equation for acoustic
waves, Maxwell's equations for electromagnetic waves. The mathematical
modeling of scattering phenomena usually corresponds to choosing
coefficients or boundary conditions for the equations of the
propagation models.
In the 19th century, the quantitative solution of these models in the
concrete circumstances where they are used today was impossible. It is
only the advent of electronic computing that has made possible the
quantitative solution of these models in practical situations. HPC
gives us the possibility of further extending the complexity of the
models that can be solved.
2. Industrial Interest in Wave Propagation, Scattering and HPC
The number of companies both in the military and civil sectors working
in businesses related to wave propagation and scattering is huge and
continues to grow. These companies are both manufacturers (e.g.,
companies building radars or biomedical instruments) or service
providers (e.g., companies doing data analysis in geological
prospecting). To meet the challenges posed by the competition on their
own markets, these companies rely increasingly on HPC. A particularly
interesting form of HPC in this context is Grid computing.
In fact, the ubiquitous presence of computing hardware (i.e. PCs,
workstations, etc.) in modern companies makes possible for each company
to have its own computing Grid, investing very little extra money to
create and managing the Grid from the available computing hardware.
Moreover, when the internal Grid is not enough, it is easy to access to
state-of-the-art Grids thank to the existence of service providers that
sell computing time worldwide. Of course, supercomputing, that is MIMD
(Multiple Instructions Multiple Data) machines with hundred or
thousands of processors or computing on dedicated hardware, cannot be
available in house in a medium or small company due to costs.
We can conclude that the companies, especially the medium and small
ones working on technologies related to wave propagation and
scattering, are interested in developing application software able to
exploit the power of Grid computing. That is, given the mathematical
models of the phenomena considered, the way in which these models are
solved quantitatively through electronic computing is the factor that
determines the ability of the application software of exploiting the
power of Grid computing.
3. The ENEA Grid Project
In order to meet the needs of the medium and small Italian companies,
ENEA (Italian National Agency for New Technologies, Energy and the
Environment) has started a Grid project. The main goals of this project
are:
- providing ENEA for its internal use, and the eventual use by external users, with a computing Grid.
- stimulating the demand of Grid computing by medium and small
companies, providing examples of successful applications running on the
Grid, and helping the users in formulating their models in a way that
can exploit the power of the Grid.
3.1 ENEA Grid
ENEA research activity is performed in 12 sites around Italy, and the
ENEA Grid infrastructure is born with the mission to provide an
integrated environment covering most of ENEA's multi-platform
computational resources, connected in a wide area network. The key
elements in the ENEA Grid implementation are: a distributed file system
(AFS/OpenAFS), a wide area resource manager (Platform LSF) and a
unified user interface (based on Java and Citrix Technologies). These
components form the middleware that provides the functionalities
typical of a Grid, "unique authentication, authorization, resource
access and resource discovery," as stated by Ian Foster and Carl
Kesselman in their classical paper "The Anatomy of the Grid." The
choice of mature and multi-platform software components, either
proprietary as LSF and Citrix or open-source as OpenAFS, greatly
improves the reliability, ease of management and update of the system.
ENEA Grid implementation began in 1998 and the new resources have been
progressively introduced in what is now a production-quality Grid
infrastructure.
The services offered by ENEA Grid include the access to computation
resources, graphic and 3-D immersive facilities, and software
resources, including commercial codes (e.g., Fluent, Abacus, Catia) and
elaboration environments (e.g., Matlab, IDL, SAS). ENEA Grid
computational resources include about 300 processors with either MIMD
architecture -- as IBM SP, SGI Altix and Onyx, and Cray XD1 -- or in
cluster configuration -- with Linux 32/64 systems and Apple Mac OSX. A
128-node IBM SP4, [~1 Tflops] is the most important resource at the
moment. These resources are located in six computer centers in
northern, central and southern Italy and are connected over the WAN by
the Italian Academic and Research Network (GARR). ENEA Grid has, at
present, about 600 registered users and more that 2TB of stored data,
with 12TB available.
3.2 Example of a Scattering Application Running on the Grid
A highly parallelizable numerical method to solve time-dependent
acoustic obstacle scattering problems has been developed by Francesco
Zirilli's team. The method proposed is a generalization of the
"operator expansion method." Using a Fourier transform with respect to
time, the time-dependent scattering problem for the wave equation
considered is transformed in an exterior boundary value problem for the
Helmholtz equation depending on a parameter. The "operator expansion
method" is used to solve the exterior problems for the Helmholtz
equation and reduces, via a perturbative approach, the solution of each
exterior problem to the solution of a sequence of systems of first kind
integral equations defined on a suitable "reference surface."
The numerical solution of these systems of integral equations is
challenging when scattering problems involving realistic obstacles and
wavelengths small compared to the characteristic dimensions of the
obstacles are solved. Let RT be the ratio between the characteristics
dimensions of the obstacle and the wavelength considered. In the
numerical experiments done, the greater ratios RT considered range in
the interval [13-60]. This implies the use of high- dimensional vector
spaces to satisfactorily approximate the corresponding systems of
integral equations. That is, each system of integral equations in the
sequence mentioned above after being discretized to be solved
numerically becomes a large system of linear equations. Represented on
a generic base, the matrices that approximate the integral operators
after the discretization will be dense matrices.
Zirilli's team has found a new way of using the wavelet transform, and
has introduced new bases of wavelets and a version of the operator
expansion method that constructs directly element by element the
coefficient matrix of the sparse linear systems that approximate in the
wavelet basis considered the systems of integral equations. The
resulting numerical method, using ENEA Grid, is able to deal with
realistic acoustic scattering problems solving large (sparse) linear
systems (up to approximately 5Â·105 (real)) unknowns and equations.
These results are obtained running simultaneously clusters of 500 jobs
on ENEA's Grid. The computation of the elements of the coefficient
matrices of the linear systems considered and several other aspects of
the numerical algorithm proposed can be carried out independently one
from the other allowing the development of an efficient solver for
exterior boundary value problems for the Helmholtz equation,
particularly suited to be executed on a computing Grid. Several
numerical experiments involving realistic obstacles and "small"
wavelengths have been performed. Some animations and virtual reality
applications relative to these numerical experiments can be seen at the
Web site www.econ.univpm.it/recchioni/w12/.
About the Authors
Silvio Migliori received his nuclear engineer degree from UniversitÃ di
Roma La Sapienza. He has been with ENEA since 1983, where he has worked
in the Controlled Nuclear Fusion Department in designing and
implementing experimental devices. Since 1995, he has held a position
in the Computer and Networking Service, where he is now adjoint
director for scientific computation. His main interests are oriented to
advanced systems as computational Grids and virtual environments. He
can be reached at migliori@enea.it.
Giovanni Bracco received his physics degree at Pavia University, and
has worked in ENEA since 1981. He has worked in the controlled nuclear
fusion department with a particular interest in plasma diagnostics
systems and in the development of data analysis tools. Since 2004, he
has held a position in the computer and networking service, where he
works in Grid computing and distributed scientific computation.
Francesco Zirilli received a mathematics degree from UniversitÃ di Roma
La Sapienza. He is currently a Professor of Istituzioni di Matematiche
at UniversitÃ di Roma La Sapienza. His scientific interests are in
mathematical modeling and scientific computing in mathematical physics
and engineering. He is author of over 100 scientific publications in
refereed journals and books.