OPENFOAM

By Francesco Iannone

Policy 

OpenFOAM is produced by OpenCFD Ltd, is freely available and open source, licensed under the GNU General Public Licence.

General 

The OpenFOAM (Open Field Operation and Manipulation) CFD Toolbox can simulate anything from complex fluid flows involving chemical reactions, turbulence and heat transfer, to solid dynamics, electromagnetics and the pricing of financial options.

Description 

The core technology of OpenFOAM is a flexible set of efficient C++ modules. These are used to build a wealth of: solvers, to simulate specific problems in engineering mechanics; utilities, to perform pre- and post-processing tasks ranging from simple data manipulations to visualisation and mesh processing; libraries, to create toolboxes that are accessible to the solvers/utilities, such as libraries of physical models.

OpenFOAM is supplied with numerous pre-configured solvers, utilities and libraries and so can be used like any typical simulation package. However, it is open, not only in terms of source code, but also in its structure and hierarchical design, so that its solvers, utilities and libraries are fully extensible.

OpenFOAM uses finite volume numerics to solve systems of partial differential equations ascribed on any 3D unstructured mesh of polyhedral cells. The fluid flow solvers are developed within a robust, implicit, pressure-velocity, iterative solution framework, although alternative techniques are applied to other continuum mechanics solvers. Domain decomposition parallelism is fundamental to the design of OpenFOAM and integrated at a low level so that solvers can generally be developed without the need for any parallel-specific coding.

Availability 

On CRESCO6 OpenFOAM is available as a modules.

Usage at CRESCO6 

Example, loading openfoam version 8

After loading the module, you also need to do:

Important notes

• You have to make sure that $HOME/OpenFOAM  are softlinks pointing to either $HOME/private/OpenFOAM or into your GPFS storage area $PFS_POR_STORE0 directory. If you do not know what this means, please look at data management on CRESCO portal.
• WM_PROJECT_USER_DIR=$HOME/$WM_PROJECT/$USER-$WM_PROJECT_VERSION.
• FOAM_RUN is set to $WM_PROJECT_USER_DIR/run
• It is best to make sure that the $FOAM_RUN directory is located under your /GPFS storage area tree, using softlinks, to be able to run batch jobs.
• CRESCO6 has compiled OpenFOAM with the Intel/IntelMPI 19  compilers with AVX-512 option.
• Important! You should NOT attempt to use the cases and applications in the tutorials for a different version of OpenFOAM! Some of them may work, but most will not. Make a fresh copy of the applications and cases you wish to modify, from the version of OpenFOAM you wish to run.
• paraFoam does not work on CRESCO6, so that part will have to be done on your own computer.
• Use the environment variables ($FOAM_USER_APPBIN and $FOAM_USER_LIBBIN) to specify destination directories when building your own extensions, this ensures that correct directories are specified and eases future upgrades.
• $FOAM_USER_APPBIN is included in the search path for binaries ($PATH).

Serial jobs

There are some examples in the tutorials directory of the OpenFOAM installation ($FOAM_TUTORIALS).

Here we will look at $FOAM_TUTORIALS/incompressible/icoFoam/cavity and we will run OpenFOAM 8. In order to run a serial job like the one in the directory above, you should do the following:

module unload mpi_flavour intel

module load module load intel/intel19  mpi_flavour/impi-intel19

module load libfftw/3.3.8-intel19

module load  openfoam/v8-impi19source $FOAM_BASH

Create your working directory (won't delete existing). This should be done in GPFS, if you want to run batch jobs:

mkdir -p $FOAM_RUN

Change to that directory:

cd $FOAM_RUN

Copy the tutorials there and change permissions:

cp -r $FOAM_TUTORIALS tutorials

chmod -R 755 tutorials

The following is correct for OpenFOAM 8.0.

 it is tutorials/incompressible/icoFoam/cavity

cd tutorials/incompressible/icoFoam/cavity/cavity

There will always be (at least) three subdirectories. Cases for OpenFOAM are setup by editing case files. A case being simulated involves data for mesh, fields, properties, control parameters, etc. The structure can be seen here: File Structure of OpenFOAM cases. The three subdirectories that are always present are:

-      0: containing the files p and U, with information about the boundary and initial conditions for the pressure and the velocity. More information can be found in the example here: Lid-driven cavity flow. There can be more than one 'time' directory.

-      constant: containing the directory polymesh and one or more files with the suffix ...Properties. polymesh has files for mesh generation, and the ...Properties are files for the physical properties. In the case of icoFoam, the only property that needs to be specified is the kinematic viscosity (in transportProperties)

-      system: this directory contains files for controlling the case (controlDict), discretisation schemes (fvSchemes), the specification of linear solvers and tolerances (fvSolution), and other things like setting the initial field (setFields), depending on the case. They can be found in the OpenFOAM User Guide.

In order to run the case, you must either be located in the directory or give the path to it.

The first you must do is run blockMesh to generate the mesh - several needed files.

$ blockMesh

It is sometimes a good idea to view the mesh to check for any errors before running. You can read more about how to do this here: Viewing the mesh (go down to section 1.1.2).

You then run the application by typing the name of the solver or utility (here the solver icoFoam) while standing in the case directory, or with the path (here icoFoam -case $path_to/tutorials/incompressible/icoFoam/cavity/cavity)

$ icoFoam

Parallel jobs

The example we will look at is tutorials/multiphase/interFoam/laminar/damBreak. The example will be for OpenFOAM 8,. Remember, if you want to submit.

You probably need to make some changes to the example. Note that this example assumes you copied the tutorials as shown under the serial example. First, make a copy to make the changes in:

cd $FOAM_RUN/tutorials/multiphase/interFoam/laminar

Create a new directory to play in

mkdir damBreakFine

Copy all the files to it (earlier versions have the files directly under the first "damBreak")

cp -r damBreak/damBreak/0 damBreakFine

cp -r damBreak/damBreak/system damBreakFine

cp -r damBreak/damBreak/constant damBreakFine

Enter the new case directory and change the blocks description in the blockMeshDict dictionary (in the directory /system) to

blocks

(        

    hex (0 1 5 4 12 13 17 16) (46 10 1) simpleGrading (1 1 1)

    hex (2 3 7 6 14 15 19 18) (40 10 1) simpleGrading (1 1 1)

    hex (4 5 9 8 16 17 21 20) (46 76 1) simpleGrading (1 2 1)

    hex (5 6 10 9 17 18 22 21) (4 76 1) simpleGrading (1 2 1)

    hex (6 7 11 10 18 19 23 22) (40 76 1) simpleGrading (1 2 1)

);

Run blockMesh (in the damBreakFine/ directory) to create the mesh etc.

$ blockMesh

As the mesh has now changed from the damBreak example, the user must re-initialise the phase field alpha1 in the 0 time directory since it contains a number of elements that is inconsistent with the new mesh. The best way to do this, is to rerun the setFields utility. There is a backup copy of the initial uniform α1 that the user should copy to 0/alpha1 before running setFields:

cd $FOAM_RUN/tutorials/multiphase/interFoam/laminar/damBreakFine

cp -r 0/alpha.water.orig 0/alpha.water

setFields

The method of parallel computing used by OpenFOAM is known as domain decomposition, in which the geometry and associated fields are broken into pieces and allocated to separate processors for solution. The first step required to run a parallel case is therefore to decompose the domain using the decomposePar utility. There is a dictionary associated with decomposePar named decomposeParDict which is located in the system directory of the tutorial case:

cd $FOAM_RUN/tutorials/multiphase/interFoam/laminar/damBreakFine/system

Open decomposeParDict in your favourite editor. The first entry is numberOfSubdomains which specifies the number of subdomains into which the case will be decomposed, usually corresponding to the number of processors available for the case.

In this example we are using 16 processors, so:

numberOfSubdomains 16;

We also need to adjust n = n_xn_yn_z in simpleCoeffs accordingly, so n_xn_y = numberOfSubdomains. There is information in section 3.1 of the User Guide (scroll down to 3.1.11) with further details of how to run a case in parallel.

After this parameter is sat, you should run (in damBreakFine)

$ decomposePar

in order to automatically construct subdirectories. These will have been created, one for each processor, in the case directory. The directories are named processorN, where N = 0,1, ...

To run this, you need to submit a batch job. Depending on which cluster you run on, you can have different amounts of cores. Abisko has 48 cores per node, while Kebnekaise has 28 cores per node (for the regular compute nodes - see the Kebnekaise hardware page for information about the other node types).

Make a job submit file like this (works for 16 nodes). Remember, LSF exports the environment (including modules), so you should do ml purge first to make sure that the submit file loads the expected module.

Prepare a submit script

In order to submit the job, we need to compose a shell script including module preparation. We will use the script cavity.sh:

cavity.sh

A brief reminder of LSF files structure:

16 nodes (768 cores), 1 MPI tasks per nodes)

-      Line 1: defines the job name.

-      Line 2: standard error from the job to a file.

-      Line 3: standard output from the job to a file

-      Line 4: queue name.

-      Line 5: number of cores

-      Line 6: assigned 48 cores per node.

-      Line 9: set n. of MPI processes

-      Line 10: set the hostfile

-      Line 11: run interFoma on 16 nodes with 1 task per node exclusively dedicated

Submit and monitor the job

In order to submit the job, we need to compose a shell script including module preparation and the Gaussian command. We will use the script cavity.sh:

To submit the job, we send the esculentin.slm script to bsub:

To check on the job status:

The output of this command looks like this:

·       JOBID indicates the number that identifies the job in the LSF system.

·       USER is the user who submitted the job.

·       STAT indicates job status, for example PEND (pending), RUN (running), etc.

·       NAME is the label we provided with BSUB -J in the script.

·       FROM_HOST ndicates the node from which was submitted.

·       EXEC_HOST indicates the node name where the job is executed

·       JOBNAME indicates the label we provided with BSUB -J in the script.

·       SUBMIT_TIME indicates the datetime of the job submission.

We can cancel a pending or running job: