32.7.1 Checking Parallel Performance

The performance meter allows you to report the wall clock time elapsed during a computation, as well as message-passing statistics. Since the performance meter is always activated, you can access the statistics by printing them after the computation is completed. To view the current statistics, use the Parallel/Timer/Usage menu item.

Parallel $\rightarrow$ Timer $\rightarrow$ Usage

Performance statistics will be printed in the text window (console).

To clear the performance meter so that you can eliminate past statistics from the future report, use the Parallel/Timer/Reset menu item.

Parallel $\rightarrow$ Timer $\rightarrow$ Reset

The following example demonstrates how the current parallel statistics are displayed in the console window:

Performance Timer for 1 iterations on 4 compute nodes
 Average wall-clock time per iteration:              4.901 sec
 Global reductions per iteration:                      408 ops
 Global reductions time per iteration:               0.000 sec (0.0%)
 Message count per iteration:                          801 messages
 Data transfer per iteration:                        9.585 MB
 LE solves per iteration:                               12 solves
 LE wall-clock time per iteration:                   2.445 sec (49.9%)
 LE global solves per iteration:                        27 solves
 LE global wall-clock time per iteration:            0.246 sec (5.0%)
 AMG cycles per iteration:                              64 cycles
 Relaxation sweeps per iteration:                     4160 sweeps
 Relaxation exchanges per iteration:                   920 exchanges

 Total wall-clock time:                              4.901 sec
 Total CPU time:                                    17.030 sec

A description of the parallel statistics is as follows:

• Average wall-clock time per iteration describes the average real (wall clock) time per iteration.

• Global reductions per iteration describes the number of global reduction operations (such as variable summations over all processes). This requires communication among all processes.

  A global reduction is a collective operation over all processes for the given job that reduces a vector quantity (the length given by the number of processes or nodes) to a scalar quantity (e.g., taking the sum or maximum of a particular quantity). The number of global reductions cannot be calculated from any other readily known quantities. The number is generally dependent on the algorithm being used and the problem being solved.

• Global reductions time per iteration describes the time per iteration for the global reduction operations.

• Message count per iteration describes the number of messages sent between all processes per iteration. This is important with regard to communication latency, especially on high-latency interconnnects.

  A message is defined as a single point-to-point, send-and-receive operation between any two processes. (This excludes global, collective operations such as global reductions.) In terms of domain decomposition, a message is passed from the process governing one subdomain to a process governing another (usually adjacent) subdomain.

  The message count per iteration is usually dependent on the algorithm being used and the problem being solved. The message count and the number of messages that are reported are totals for all processors.

  The message count provides some insight into the impact of communication latency on parallel performance. A higher message count indicates that the parallel performance may be more adversely affected if a high-latency interconnect is being used. Ethernet has a higher latency than Myrinet or Infiniband. Thus, a high message count will more adversely affect performance with Ethernet than with Infiniband.

  To check the latency of the overall cluster interconnect, refer to Section  32.7.1.

• Data transfer per iteration describes the amount of data communicated between processors per iteration. This is important with respect to interconnect bandwidth.

  Data transfer per iteration is usually dependent on the algorithm being used and the problem being solved. This number generally increases with increases in problem size, number of partitions, and physics complexity.

  The data transfer per iteration may provide some insight into the impact of communication bandwidth (speed) on parallel performance. The precise impact is often difficult to quantify because it is dependent on many things including: ratio of data transfer to calculations, and ratio of communication bandwidth to CPU speed. The unit of data transfer is a byte.

  To check the bandwidth of the overall cluster interconnect, refer to
  Section  32.7.1.

• LE solves per iteration describes the number of linear systems being solved per iteration. This number is dependent on the physics (non-reacting versus reacting flow) and the algorithms (pressure-based versus density-based solver), but is independent of mesh size. For the pressure-based solver, this is usually the number of transport equations being solved (mass, momentum, energy, etc.).

• LE wall-clock time per iteration describes the time (wall-clock) spent doing linear equation solvers (i.e., multigrid).

• LE global solves per iteration describes the number of solutions on the coarse level of the AMG solver where the entire linear system has been pushed to a single processor ( n0). The system is pushed to a single processor to reduce the computation time during the solution on that level. Scaling generally is not adversely affected because the number of unknowns is small on the coarser levels.

• LE global wall-clock time per iteration describes the time (wall-clock) per iteration for the linear equation global solutions (see above).

• AMG cycles per iteration describes the average number of multigrid cycles (V, W, flexible, etc.) per iteration.

• Relaxation sweeps per iteration describes the number of relaxation sweeps (or iterative solutions) on all levels for all equations per iteration. A relaxation sweep is usually one iteration of Gauss-Siedel or ILU.

• Relaxation exchanges per iteration describes the number of solution communications between processors during the relaxation process in AMG. This number may be less than the number of sweeps because of shifting the linear system on coarser levels to a single node/process.

• Time-step updates per iteration describes the number of sub-iterations on the time step per iteration.

• Time-step wall-clock time per iteration describes the time per sub-iteration.

• Total wall-clock time describes the total wall-clock time.

• Total CPU time describes the total CPU time used by all processes. This does not include any wait time for load imbalances or for communications (other than packing and unpacking local buffers).

The most relevant quantity is the Total wall clock time. This quantity can be used to gauge the parallel performance (speedup and efficiency) by comparing this quantity to that from the serial analysis (the command line should contain -t1 in order to obtain the statistics from a serial analysis). In lieu of a serial analysis, an approximation of parallel speedup may be found in the ratio of Total CPU time to Total wall clock time.

Checking Latency and Bandwidth

You can check the latency and bandwidth of the overall cluster interconnect, to help identify any issues affecting ANSYS FLUENT scalability, by using the Parallel/Network/Show Latency... and Parallel/Network/Show Bandwidth... menu items.

Parallel $\rightarrow$ Network $\rightarrow$ Show Latency...

Depending on the number of machines and processors being used, a table containing information about the communication speed for each node will be printed to the console. The table will also summarize the minimum and maximum latency between two nodes.

Consider the following example when checking for latency:

Latency (usec) with 1000 samples [1.83128 sec]
------------------------------------------ 
ID        n0    n1    n2    n3    n4    n5 
------------------------------------------ 
n0         \  48.0  48.2  48.2  48.3   *50 
n1      48.0     \  48.2  48.3  48.3   *48 
n2      48.2  48.2     \  48.8  49.1   *53 
n3      48.2  48.3   *49     \  48.6  48.5 
n4      48.3  48.3  49.1  48.6     \   *50 
n5      49.7  48.5   *53  48.5  49.7     \ 

------------------------------------------ 
Min: 47.9956 [n0<-->n1] 
Max: 52.6836 [n5<-->n2] 
------------------------------------------

In the above table, (*) is the maximum value in that row. The smaller the latency, the better.

Six processors (n0 to n5) are spawned. The latency between n0 and n1 is 48.0 $\mu s$ . Similarly, the latency between n1 and n2 is 48.2 $\mu s$. The minimum latency occurs between n0 and n1 and the maximum latency occurs between n2 and n5, as noted in the table. Checking the latency is particularly useful when you are not seeing expected speedup on a cluster.

Parallel $\rightarrow$ Network $\rightarrow$ Show Bandwidth...

In addition to checking for latency, you can check your bandwidth. A table containing information about the amount of data communicated within one second between two nodes is printed to the console. The table will also summarize the minimum and maximum bandwidth between two nodes.

Consider the following example when checking for bandwidth:

Bandwidth (MB/s) with 5 messages of size 4MB [4.36388 sec] 
------------------------------------------  
ID        n0    n1    n2    n3    n4    n5 
------------------------------------------ 
n0         \ 111.8   *55 111.8  97.5 101.3 
n1     111.8     \  69.2  98.7 111.7   *51 
n2      54.7  69.2     \  72.9 104.8   *45 
n3     111.8  98.7  72.9     \  64.0   *45 
n4      97.6 111.7 104.8   *64     \  76.9 
n5     101.2  50.9  45.5   *45  76.9     \ 

------------------------------------------ 
Min: 45.1039 [n5<-->n3] 
Max: 111.847 [n0<-->n3] 
------------------------------------------

In the above table, (*) is the minimum value in that row. The larger the bandwidth, the better.

The bandwidth between n0 and n1 is 111.8 MB/s. Similarly, the bandwidth between n1 and n2 is 69.2 MB/s. The minimum amount of bandwidth occurs between n3 and n5 and the maximum occurs between n0 and n3, as noted in the table. Checking the bandwidth is particularly useful when you cannot see good scalability with relatively large cases.