32.5.5 Using the Partitioning and Load Balancing Dialog Box

Partitioning

In order to partition the mesh, you need to select the partition method for creating the mesh partitions, set the number of partitions, select the zones and/or registers, and choose the optimizations to be used. For some methods, you can also perform pretesting to ensure that the best possible partition is performed. Once you have set all the parameters in the Partitioning and Load Balancing dialog box to your satisfaction, click the Partition button to subdivide the mesh into the selected number of partitions using the prescribed method and optimization(s). See above for recommended partitioning strategies.

You can set the relevant inputs in the Partitioning and Load Balancing dialog box (Figure  32.5.3 in the parallel solver, or Figure  32.5.4 in the serial solver) in the following manner:

Parallel $\rightarrow$ Partitioning and Load Balancing...

Figure 32.5.3: The Partitioning and Load Balancing Dialog Box in the Parallel Solver

Figure 32.5.4: The Partitioning and Load Balancing Dialog Box in the Serial Solver

1.   Select the Method from the drop-down list. The choices are described in Section  32.5.6.

2.   In the Options tab

(a)   Set the desired number of mesh partitions in the Number of Partitions field. You can use the counter arrows to increase or decrease the value, instead of typing in the box. The number of mesh partitions must be an integer number which is divisible by the number of processors available for parallel computing.

(b)   Set the Reporting Verbosity. This allows you to control what is printed to the console. See Section  32.5.5 for more information.

(c)   You can choose to independently apply partitioning to each cell zone, or you can allow partitions to cross zone boundaries using the Across Zones check button. It is recommended that you not partition cell zones independently (by turning off the Across Zones check button) unless cells in different zones will require significantly different amounts of computation during the solution phase (e.g., if the domain contains both solid and fluid zones).

(d)   You can select Encapsulate Mesh Interfaces if you would like the cells surrounding all non-conformal mesh interfaces in your mesh to reside in a single partition at all times during the calculation. If your case file contains non-conformal interfaces on which you plan to perform adaption during the calculation, you will have to partition it in the serial solver, with the Encapsulate Mesh Interfaces and Encapsulate for Adaption options turned on.

(e)   If you have enabled the Encapsulate Mesh Interfaces option in the serial solver, the Encapsulate for Adaption option will also be available. When you select this option, additional layers of cells are encapsulated such that transfer of cells will be unnecessary during parallel adaption.

3.   In the Optimization tab

(a)   You can activate and control the desired optimization methods (described in Section  32.5.6). You can activate the Merge and Smooth schemes by enabling the check button next to each one. For each scheme, you can also set the number of Iterations. Each optimization scheme will be applied until appropriate criteria are met, or the maximum number of iterations has been executed. If the Iterations counter is set to 0, the optimization scheme will be applied until completion, with no limit on maximum number of iterations.

(b)   If you have chosen the Principal Axes or Cartesian Axes method, you can improve the partitioning by enabling the automatic testing of the different bisection directions before the actual partitioning occurs. To use pretesting, enable the Pre-Test option. Pretesting is described in Section  32.5.6.

4.   In the Zones and/or Registers lists, select the zone(s) and/or register(s) for which you want to partition. For most cases, you will select all Zones (the default) to partition the entire domain. See below for details.

5.   You can assign selected Zones and/or Registers to a specific partition ID by entering a value for the Set Selected Zones and Registers to Partition ID. For example, if the Number of partitions for your mesh is 2, then you can only use IDs of 0 or 1. If you have three partitions, then you can enter IDs of 0, 1, or 2. This can be useful in situations where the gradient at a region is known to be high. In such cases, you can mark the region or zone and set the marked cells to one of the partition IDs, thus preventing the partition from going through that region. This in turn will facilitate convergence. This is also useful in cases where mesh manipulation tools are not available in parallel. In this case, you can assign the related cells to a particular ID so that the mesh manipulation tools are now functional.

If you are running the parallel solver, and you have marked your region and assigned an ID to the selected Zones and/or Registers, click the Use Stored Partitions button to make the new partitions valid.

Refer to the example described later in this section for a demonstration of how selected registers are assigned to a partition (Section  32.5.5).

6.   In the Weighting tab (Figure  32.5.5), you can set the appropriate weights, prior to partitioning the mesh, to improve load balancing and overall performance. You can control weights for cells, solid zones, VOF, and DPM. You can either rely on ANSYS FLUENT timers to set the weight scaling, or you can specify the value by enabling User-Specified.

Figure 32.5.5: The Weighting Tab in the Partitioning and Load Balancing Dialog Box for the Parallel Solver

(a)   Enable Cell Type weighting to allow you to partition the mesh based on the number of faces in each cell. Cell type weights are applicable when the case has mixed or polyhedral cell zones. If a value is specified, then the cell weight equals that value in addition to the number of faces.

(b)   Enable Solid Zones weighting to allow the partitioning to take solid zones into consideration. By default, ANSYS FLUENT will scale the weight based on the computational time of the solid and fluid zones. If an iteration has not been run, then you may need to specify a value after enabling the user-specified check box. The value you enter is relative to the fluid weight. Typically, it should be less than 1 (e.g., 0.0001) so that the calculation will be quicker for the solid zone compared to the fluid zone. Entering a value greater than 1 for the Solid Zones means that the calculation will take longer and will be more computationally expensive for the solid zone compared to the fluid zone.

Setting the Solid Zones either too high or too low can cause a load imbalance, depending on which equations are being solved. If you are not sure what would be an appropriate value, have ANSYS FLUENT automatically set it for you by making sure that User-Specified is disabled.

(c)   Enable VOF weighting to allow the partitioning to consider the imbalance caused by the free surface reconstruction with the geo-reconstruct scheme. Therefore, it is only available when using the VOF model with geometric reconstruction. You may use the user-specified value before timers are collected, or if you want to specify a value other than timing statistics. The specified value is the VOF proportion of the total computational effort.

(d)   Enable DPM weighting to set the weight of DPM particles relative to the continuous phase. DPM weights are valid when you have particle tracking in your simulation, where the user-specified value is the DPM proportion of the total computational effort relative to the continuous phase. Note that this is only available when you have injections defined. See Chapter  23 for more information.

The DPM weight takes into account the distribution of the tracking effort over the partitions and it is available after at least one calculation step with particle tracking. Displaying Particle Tracks does not change the weights. The computational effort is determined by the number of DPM steps performed in each cell. This weight becomes more important when the time for the particle tracking of particles exceeds the time for solving the flow. Enabling this option in the Weighting tab activates the counting of the particle steps in the cells. These values are available for contour and vector plots when using the Discrete Phase Model and DPM Steps per Cell variable. Additionally, it is preferable to partition along the main path of the particles, which avoids crossing of partitions and communication overhead. The particular partitioning algorithm avoids the partition interfaces on the faces with high numbers of particle paths crossing. After repartitioning, the DPM weights are reset before the next particle tracking.

(e)   Enable ISAT weighting to balance the load during the ISAT table lookup for the stiff-chemistry Laminar, EDC or PDF Transport models. The ISAT algorithm builds an unstructured table in $N$ species dimensions for storage and retrieval of the chemistry mappings. Since chemistry is usually computationally expensive, this storage/retrieval can be very time-consuming (for information about ISAT, refer to this section in the separate Theory Guide). Each parallel node builds its own table, and there is no message passing to tables on other nodes. As some nodes may have more chemical reactions than others (for example one parallel node may contain just air at a constant temperature, in which case the ISAT table will contain only one entry and calculation will be rapid), there may be a load imbalance. The dynamic load balancing algorithm will migrate cells from high computational load nodes to low computational load nodes.

If you decide to specify a value, this user-specified value is the ISAT proportion of the total computational effort.

7.   When using the dynamic mesh model in your parallel simulations, the Partition dialog box includes an Auto Repartition option and a Repartition Interval setting. These parallel partitioning options are provided because ANSYS FLUENT migrates cells when local remeshing and smoothing are performed. Therefore, the partition interface becomes very wrinkled and the load balance may deteriorate. By default, the Auto Repartition option is selected, where a percentage of interface faces and loads are automatically traced. When this option is selected, ANSYS FLUENT automatically determines the most appropriate repartition interval based on various simulation parameters. Sometimes, using the Auto Repartition option provides insufficient results, therefore, the Repartition Interval setting can be used. The Repartition Interval setting lets you to specify the interval (in time steps or iterations respectively) when a repartition is enforced. When repartitioning is not desired, you can set the Repartition Interval to zero.

Note that when dynamic meshes and local remeshing is utilized, updated meshes may be slightly different in parallel ANSYS FLUENT (when compared to serial ANSYS FLUENT or when compared to a parallel solution created with a different number of compute nodes), resulting in very small differences in the solutions.

8.   Click the Partition button to partition the mesh.

9.   If you decide that the new partitions are better than the previous ones (if the mesh was already partitioned), click the Use Stored Partitions button to make the newly stored cell partitions the active cell partitions . The active cell partition is used for the current calculation, while the stored cell partition (the last partition performed) is used when you save a case file.

Example of Setting Selected Registers to Specified Partition IDs

1.   Start ANSYS FLUENT in parallel. The case in this example was partitioned across two nodes.

2.   Read in your case.

3.   Display the mesh with the Partitions option enabled in the Mesh Display dialog box (Figure  32.5.6).

Figure 32.5.6: The Partitioned Mesh

4.   Adapt your region and mark your cells (see Section  27.6.1). This creates a register.

5.   Open the Partitioning and Load Balancing dialog box.

6.   Set the Set Selected Zones and Registers to Partition ID to 0 and click the corresponding button. This prints the following output to the ANSYS FLUENT console window:

>> 2 Active Partitions:
   ----------------------------------------------------------------------
   Collective Partition Statistics:        Minimum   Maximum   Total     
   ----------------------------------------------------------------------
   Cell count                              459       459       918       
   Mean cell count deviation               0.0\%      0.0\%      
   Partition boundary cell count           11        11        22        
   Partition boundary cell count ratio     2.4\%      2.4\%      2.4\%      

   Face count                              764       1714      2461      
   Mean face count deviation               -38.3\%    38.3\%     
   Partition boundary face count           13        13        17        
   Partition boundary face count ratio     0.8\%      1.7\%      0.7\%      

   Partition neighbor count                1         1         
   ----------------------------------------------------------------------
   Partition Method                        Principal Axes                       
   Stored Partition Count                  2                                    
Done.

7.   Click the Use Stored Partitions button to make the new partitions valid. This migrates the partitions to the compute-nodes. The following output is then printed to the ANSYS FLUENT console window:

Migrating partitions to compute-nodes.
>> 2 Active Partitions:
       P   Cells I-Cells Cell Ratio   Faces I-Faces Face Ratio Neighbors
       0     672      24      0.036    2085      29      0.014         1
       1     246      24      0.098     425      29      0.068         1

   ----------------------------------------------------------------------
   Collective Partition Statistics:        Minimum   Maximum   Total     
   ----------------------------------------------------------------------
   Cell count                              246       672       918       
   Mean cell count deviation               -46.4\%    46.4\%     
   Partition boundary cell count           24        24        48        
   Partition boundary cell count ratio     3.6\%      9.8\%      5.2\%      

   Face count                              425       2085      2461      
   Mean face count deviation               -66.1\%    66.1\%     
   Partition boundary face count           29        29        49        
   Partition boundary face count ratio     1.4\%      6.8\%      2.0\%      

   Partition neighbor count                1         1         
   ----------------------------------------------------------------------
   Partition Method                        Principal Axes                       
   Stored Partition Count                  2                                    
Done.

8.   Display the mesh (Figure  32.5.7).

Figure 32.5.7: The Partitioned ID Set to Zero

9.   This time, set the Set Selected Zones and Registers to Partition ID to 1 and click the corresponding button. This prints a report to the ANSYS FLUENT console.

10.   Click the Use Stored Partitions button to make the new partitions valid and to migrate the partitions to the compute-nodes.

11.   Display the mesh (Figure  32.5.8). Notice now that the partition appears in a different location as specified by your partition ID.

Figure 32.5.8: The Partitioned ID Set to 1

Although this example demonstrates setting selected registers to specific partition IDs in parallel, it can be similarly applied in serial.

Partitioning Within Zones or Registers

The ability to restrict partitioning to cell zones or registers gives you the flexibility to apply different partitioning strategies to subregions of a domain. For example, if your geometry consists of a cylindrical plenum connected to a rectangular duct, you may want to partition the plenum using the Cylindrical Axes method, and the duct using the Cartesian Axes method.

If the plenum and the duct are contained in two different cell zones, you can select one at a time and perform the desired partitioning, as described in Section  32.5.5. If they are not in two different cell zones, you can create a cell register (basically a list of cells) for each region using the functions that are used to mark cells for adaption. These functions allow you to mark cells based on physical location, cell volume, gradient or isovalue of a particular variable, and other parameters. See Chapter  27 for information about marking cells for adaption. Section  27.11.1 provides information about manipulating different registers to create new ones. Once you have created a register, you can partition within it as described above.

Note that partitioning within zones or registers is not available when Metis is selected as the partition Method.

For dynamic mesh applications (see item 11 above), ANSYS FLUENT stores the partition method used to partition the respective zone. Therefore, if repartitioning is done, ANSYS FLUENT uses the same method that was used to partition the mesh.

Reporting During Partitioning

As the mesh is partitioned, information about the partitioning process will be printed in the text (console) window. By default, the solver will print the number of partitions created, the number of bisections performed, the time required for the partitioning, and the minimum and maximum cell, face, interface, and face-ratio variations. (See Section  32.5.7 for details.) If you increase the Reporting Verbosity to 2 from the default value of 1, the partition method used, the partition ID, number of cells, faces, and interfaces, and the ratio of interfaces to faces for each partition will also be printed in the console window. If you decrease the Reporting Verbosity to 0, only the number of partitions created and the time required for the partitioning will be reported.

You can request a portion of this report to be printed again after the partitioning is completed. When you click the Print Active Partitions or Print Stored Partitions button in the parallel solver, ANSYS FLUENT will print the partition ID, number of cells, faces, and interfaces, and the ratio of interfaces to faces for each active or stored partition in the console window. In addition, it will print the minimum and maximum cell, face, interface, and face-ratio variations. In the serial solver, you will obtain the same information about the stored partition when you click Print Partitions. See Section  32.5.7 for details.

Recall that to make the stored cell partitions the active cell partitions you must click the Use Stored Partitions button. The active cell partition is used for the current calculation, while the stored cell partition (the last partition performed) is used when you save a case file.

Resetting the Partition Parameters

If you change your mind about your partition parameter settings, you can easily return to the default settings assigned by ANSYS FLUENT by clicking on the Default button. When you click the Default button, it will become the Reset button. The Reset button allows you to return to the most recently saved settings (i.e., the values that were set before you clicked on Default). After execution, the Reset button will become the Default button again.

Load Balancing

A dynamic load balancing capability is available in ANSYS FLUENT. The principal reason for using parallel processing is to reduce the turnaround time of your simulation, which may be achieved by the following means:

• Faster machines, e.g., faster CPU, memory, cache, and, transportation speed between the CPU and memory

• Faster interconnects, e.g., smaller latency and larger bandwidth

• Better Load balancing, e.g., load is evenly distributed and CPUs are not idled during calculation

The first two are evolving with technology advancement, which is beyond the scope of this document. The third item is regarding optimization of available computation power. Here we are mainly talking about load balancing on dedicated homogeneous resources, which is often the case nowadays. If you are not using a dedicated homogeneous resource, you may need to account for differences in CPU speeds during partitioning by specifying a load distribution (see Section  32.5.8).

On a dedicated homogeneous system, the key for load balancing is how to evaluate the computational requirement of each cell. By default, ANSYS FLUENT assumes that each cell requires the same computational work, but this is often not the case. For example

• A hexahedral cell demands more CPU and memory than a tetrahedral cell.

• A cell with particle tracking will use more time than a cell without particle tracking.

• ISAT species model cells may have magnitude differences in time usage.

To balance these differences, ideally, the time used in each cell can be recorded and load balancing can be achieved based on these time statistics. But in reality, this is hardly possible, because the statistics itself may require more time than the real computation effort. In addition, accuracy of timers is questionable in highly granulized accounting. Instead, we identify features causing computation unevenness and try to record time usage for these features in a collective manner. For a more detailed description of this, please refer to Section  32.5.5 in the discussion of the Weighting tab. In addition, the imbalance may be in a dynamic manner, for example

• The mesh may be changed by adaption or mesh movement.

• In unsteady cases, particle tracking may move from one region to another region.

The current dynamic load balancing has been implemented for better scalability of cases with imbalanced physical or geometrical models, thus reducing the simulation time. The implementation is based on considering weights from these models scaled by CPU time usage. Load balancing for DPM, VOF, cell type (number of faces per cell), and solid zones can be performed. In addition, cell weight based load balancing and machine load distribution can also be specified (see Section  32.5.8). ANSYS FLUENT takes the weights from physical models and considers them for partitioning. The weights are assembled based on the time used by each physical model. For dynamic load balancing, the load is checked and balanced based on your specified imbalance threshold. To apply dynamic load balancing on the various models, click the Dynamic Load Balancing tab and select the required balancing as follows:

Figure 32.5.9: The Dynamic Load Balancing Tab

1.   Enable Physical Models load balancing during iterations so that the load will be evaluated for time usage and weight distribution, based on the Interval that you provide. If the imbalance exceeds the specified Threshold, then repartitioning will be performed by considering the selected weights. Physical Models load balancing will only be available when you have the specific physical models enabled in the case. You will be prompted to enable the weights for those models. When weights for the physical models are all disabled, you will be prompted to disable Physical Models load balancing.

2.   Enable Dynamic Mesh if there is any dynamic mesh movement. Load balancing, based on the number of cells, will be checked and balanced if the imbalance threshold is exceeded. These parallel partitioning options are provided because ANSYS FLUENT migrates cells when local remeshing and smoothing are performed. Therefore, the partition interface becomes very wrinkled and the load balance may deteriorate. By default, the Auto option is selected, where a percentage of interface faces and loads are automatically traced. When this option is selected, ANSYS FLUENT automatically determines the most appropriate repartition interval based on various simulation parameters. Sometimes, using the Auto option provides insufficient results; therefore, the Interval setting can be used. The Interval setting lets you specify the interval (in time steps or iterations, respectively) when load balancing is enforced. When load balancing is not desired, you may disable Dynamic Mesh load balancing. Dynamic Mesh load balancing is only available when you have dynamic models enabled in your case.

Note that when dynamic meshes and local remeshing are utilized, updated meshes may be slightly different in parallel ANSYS FLUENT (when compared to serial ANSYS FLUENT or when compared to a parallel solution created with a different number of compute nodes), resulting in very small differences in the solutions.

3.   Enable Mesh Adaption. Any time mesh adaption occurs, load balancing, based on the number of cells, will be checked and balanced if the imbalance threshold is exceeded. If problems arise in your computations due to adaption, you can disable the load balancing for Mesh Adaption.