32.5.6 Mesh Partitioning Methods

Partitioning the mesh for parallel processing has three major goals:

• Create partitions with equal numbers of cells.

• Minimize the number of partition interfaces--i.e., decrease partition boundary surface area.

• Minimize the number of partition neighbors.

Balancing the partitions (equalizing the number of cells) ensures that each processor has an equal load and that the partitions will be ready to communicate at about the same time. Since communication between partitions can be a relatively time-consuming process, minimizing the number of interfaces can reduce the time associated with this data interchange. Minimizing the number of partition neighbors reduces the chances for network and routing contentions. In addition, minimizing partition neighbors is important on machines where the cost of initiating message passing is expensive compared to the cost of sending longer messages. This is especially true for workstations connected in a network.

The partitioning schemes in ANSYS FLUENT use bisection or METIS algorithms to create the partitions, but unlike other schemes which require the number of partitions to be a factor of two, these schemes have no limitations on the number of partitions. For each available processor, you will create the same number of partitions (i.e., the total number of partitions will be an integral multiple of the number of processors).

Partition Methods

The mesh is partitioned using a bisection or METIS algorithm. The selected algorithm is applied to the parent domain, and then recursively applied to the child subdomains. For example, to divide the mesh into four partitions with a bisection method, the solver will bisect the entire (parent) domain into two child domains, and then repeat the bisection for each of the child domains, yielding four partitions in total. To divide the mesh into three partitions with a bisection method, the solver will "bisect'' the parent domain to create two partitions--one approximately twice as large as the other--and then bisect the larger child domain again to create three partitions in total.

The mesh can be partitioned using one of the algorithms listed below. The most efficient choice is problem-dependent, so you can try different methods until you find the one that is best for your problem. See Section  32.5.4 for recommended partitioning strategies.

Cartesian Axes   bisects the domain based on the Cartesian coordinates of the cells (see Figure  32.5.10). It bisects the parent domain and all subsequent child subdomains perpendicular to the coordinate direction with the longest extent of the active domain. It is often referred to as coordinate bisection.

Cartesian Strip   uses coordinate bisection but restricts all bisections to the Cartesian direction of longest extent of the parent domain (see Figure  32.5.11). You can often minimize the number of partition neighbors using this approach.

Cartesian X-, Y-, Z-Coordinate   bisects the domain based on the selected Cartesian coordinate. It bisects the parent domain and all subsequent child subdomains perpendicular to the specified coordinate direction. (See Figure  32.5.11.)

Cartesian R Axes   bisects the domain based on the shortest radial distance from the cell centers to that Cartesian axis ( $x$, $y$, or $z$) which produces the smallest interface size. This method is available only in 3D.

Cartesian RX-, RY-, RZ-Coordinate   bisects the domain based on the shortest radial distance from the cell centers to the selected Cartesian axis ( $x$, $y$, or $z$). These methods are available only in 3D.

Cylindrical Axes   bisects the domain based on the cylindrical coordinates of the cells. This method is available only in 3D.

Cylindrical R-, Theta-, Z-Coordinate   bisects the domain based on the selected cylindrical coordinate. These methods are available only in 3D.

Metis   uses the METIS software package for partitioning irregular graphs, developed by Karypis and Kumar at the University of Minnesota and the Army HPC Research Center. It uses a multilevel approach in which the vertices and edges on the fine graph are coalesced to form a coarse graph. The coarse graph is partitioned, and then uncoarsened back to the original graph. During coarsening and uncoarsening, algorithms are applied to permit high-quality partitions. Detailed information about METIS can be found in its manual [ 37].

Note that when using the socket version ( -pnet), the METIS partitioner is not available. In this case, METIS partitioning can be obtained using the partition filter, as described below.

If you create non-conformal interfaces, and generate virtual polygonal faces, your METIS partition can cross non-conformal interfaces by using the connectivity of the virtual polygonal faces. This improves load balancing for the parallel solver and minimizes communication by decreasing the number of partition interface cells.

Polar Axes   bisects the domain based on the polar coordinates of the cells (see Figure  32.5.14). This method is available only in 2D.

Polar R-Coordinate, Polar Theta-Coordinate   bisects the domain based on the selected polar coordinate (see Figure  32.5.14). These methods are available only in 2D.

Principal Axes   bisects the domain based on a coordinate frame aligned with the principal axes of the domain (see Figure  32.5.12). This reduces to Cartesian bisection when the principal axes are aligned with the Cartesian axes. The algorithm is also referred to as moment, inertial, or moment-of-inertia partitioning.

This is the default bisection method in ANSYS FLUENT.

Principal Strip   uses moment bisection but restricts all bisections to the principal axis of longest extent of the parent domain (see Figure  32.5.13). You can often minimize the number of partition neighbors using this approach.

Principal X-, Y-, Z-Coordinate   bisects the domain based on the selected principal coordinate (see Figure  32.5.13).

Spherical Axes   bisects the domain based on the spherical coordinates of the cells. This method is available only in 3D.

Spherical Rho-, Theta-, Phi-Coordinate   bisects the domain based on the selected spherical coordinate. These methods are available only in 3D.

Figure 32.5.10: Partitions Created with the Cartesian Axes Method

Figure 32.5.11: Partitions Created with the Cartesian Strip or Cartesian X-Coordinate Method

Figure 32.5.12: Partitions Created with the Principal Axes Method

Figure 32.5.13: Partitions Created with the Principal Strip or Principal X-Coordinate Method

Figure 32.5.14: Partitions Created with the Polar Axes or Polar Theta-Coordinate Method

Optimizations

Additional optimizations can be applied to improve the quality of the mesh partitions. The heuristic of bisecting perpendicular to the direction of longest domain extent is not always the best choice for creating the smallest interface boundary. A "pre-testing'' operation (see Section  32.5.6) can be applied to automatically choose the best direction before partitioning. In addition, the following iterative optimization schemes exist:

Smooth   attempts to minimize the number of partition interfaces by swapping cells between partitions. The scheme traverses the partition boundary and gives cells to the neighboring partition if the interface boundary surface area is decreased. (See Figure  32.5.15.)

Merge   attempts to eliminate orphan clusters from each partition. An orphan cluster is a group of cells with the common feature that each cell within the group has at least one face which coincides with an interface boundary. (See Figure  32.5.16.) Orphan clusters can degrade multigrid performance and lead to large communication costs.

Figure 32.5.15: The Smooth Optimization Scheme

Figure 32.5.16: The Merge Optimization Scheme

In general, the Smooth and Merge schemes are relatively inexpensive optimization tools.

Pretesting

If you choose the Principal Axes or Cartesian Axes method, you can improve the bisection by testing different directions before performing the actual bisection. If you choose not to use pretesting (the default), ANSYS FLUENT will perform the bisection perpendicular to the direction of longest domain extent.

If pretesting is enabled, it will occur automatically when you click the Partition button in the Partitioning and Load Balancing dialog box, or when you read in the mesh if you are using automatic partitioning. The bisection algorithm will test all coordinate directions and choose the one which yields the fewest partition interfaces for the final bisection.

Note that using pretesting will increase the time required for partitioning. For 2D problems partitioning will take 3 times longer than without pretesting, and for 3D problems it will take 4 times longer.

Using the Partition Filter

As noted above, you can use the METIS partitioning method through a filter in addition to within the Auto Partition Mesh and Partitioning and Load Balancing dialog boxes. To perform METIS partitioning on an unpartitioned mesh, use the File/Import/Partition/Metis... menu item.

File $\rightarrow$ Import $\rightarrow$ Partition $\rightarrow$ Metis...

ANSYS FLUENT will use the METIS partitioner to partition the mesh, and then read the partitioned mesh into the solver. The number of partitions will be equal to the number of processes. You can then proceed with the model definition and solution.

Direct import to the parallel solver through the partition filter requires that the host machine has enough memory to run the filter for the specified mesh. If not, you will need to run the filter on a machine that does have enough memory. You can either start the parallel solver on the machine with enough memory and repeat the process described above, or run the filter manually on the new machine and then read the partitioned mesh into the parallel solver on the host machine.

To manually partition a mesh using the partition filter, enter the following command:

utility partition input_filename partition_count output_filename

where input_filename is the filename for the mesh to be partitioned, partition_count is the number of partitions desired, and output_filename is the filename for the partitioned mesh. You can then read the partitioned mesh into the solver (using the standard File/Read/Case... menu item) and proceed with the model definition and solution.

When the File/Import/Partition/Metis... menu item is used to import an unpartitioned mesh into the parallel solver, the METIS partitioner partitions the entire mesh. You may also partition each cell zone individually, using the File/Import/Partition/Metis Zone... menu item.

File $\rightarrow$ Import $\rightarrow$ Partition $\rightarrow$ Metis Zone...

This method can be useful for balancing the work load. For example, if a case has a fluid zone and a solid zone, the computation in the fluid zone is more expensive than in the solid zone, so partitioning each zone individually will result in a more balanced work load.