7.5.8 Message Passing Macros

High-level communication macros of the form node_to_host... and host_to_node... that are described in Section  7.5.2 are typically used when you want to send data from the host to all of the compute nodes, or from node-0 to the host. You cannot, however, use these high-level macros when you need to pass data between compute nodes, or pass data from all of the compute nodes to compute node-0. In these cases, you can use special message passing macros described in this section.

Note that the higher-level communication macros expand to functions that perform a number of lower-level message passing operations which send sections of data as single arrays from one process to another process. These lower-level message passing macros can be easily identified in the macro name by the characters SEND and RECV. Macros that are used to send data to processes have the prefix PRF_CSEND, whereas macros that are used to receive data from processes have the prefix PRF_CRECV. Data that is to be sent or received can belong to the following data types: character ( CHAR), integer ( INT), REAL and logical ( BOOLEAN). BOOLEAN variables are TRUE or FALSE. REAL variables are assigned as float data types when running a single precision version of ANSYS FLUENTand double when running double precision. Message passing macros are defined in the prf.h header file and are listed below.

/*  message passing macros  */

PRF_CSEND_CHAR(to, buffer, nelem, tag)
PRF_CRECV_CHAR (from, buffer, nelem, tag)
PRF_CSEND_INT(to, buffer, nelem, tag)
PRF_CRECV_INT(from, buffer, nelem, tag)
PRF_CSEND_REAL(to, buffer, nelem, tag)
PRF_CRECV_REAL(from, buffer, nelem, tag)
PRF_CSEND_BOOLEAN(to, buffer, nelem, tag)
PRF_CRECV_BOOLEAN(from, buffer, nelem, tag)

There are four arguments to the message passing macros. For `send' messages, the argument to is the node ID of the process that data is being sent to. buffer is the name of an array of the appropriate type that will be sent. nelem is the number of elements in the array and tag is a user-defined message tag. The tag convention is to use myid when sending messages.

For `receive' messages, the argument from is the ID of the sending node. buffer is the name of an array of the appropriate type that will be received. nelem is the number of elements in the array. tag is the ID of the sending node, as the convention is to have the tag argument the same as the from argument (i.e., the first argument) for receive messages.

Note that if variables that are to be sent or received are defined in your function as real variables, then you can use the message passing macros with the _REAL suffix. The compiler will then substitute PRF_CSEND_DOUBLE or PRF_CRECV_DOUBLE if you are running double precision and PRF_CSEND_FLOAT or PRF_CRECV_FLOAT, for single precision.

Because message-passing macros are low-level macros, you will need to make sure that when a message is sent from a node process, a corresponding receiving macro appears in the receiving-node process. Note that your UDF cannot directly send messages from a compute node (other than 0) to the host using message-passing macros. They can send messages indirectly to the host through compute node-0. For example, if you want your parallel UDF to send data from all of the compute nodes to the host for postprocessing purposes, the data will first have to be passed from each compute node to compute node-0, and then from compute node-0 to the host. In the case where the compute node processes send a message to compute node-0, compute node-0 must have a loop to receive the N messages from the N nodes.

Below is an example of a compiled parallel UDF that utilizes message passing macros PRF_CSEND and PRF_CRECV. Refer to the comments (*/) in the code, for details about the function.

Example: Message Passing

#include "udf.h"
#define WALLID 3

DEFINE_ON_DEMAND(face_p_list)
{
#if !RP_HOST  /* Host will do nothing in this udf. Serial will */
  face_t f;
  Thread *tf;
  Domain *domain;
  real *p_array;
  real x[ND_ND], (*x_array)[ND_ND];
  int n_faces, i, j;

  domain=Get_Domain(1); /* Each Node will be able to access 
                                   its part of the domain */

  tf=Lookup_Thread(domain, WALLID); /* Get the thread from the domain */

  /* The number of faces of the thread on nodes 1,2... needs to be sent
     to compute node-0 so it knows the size of the arrays to receive
     from each */

  n_faces=THREAD_N_ELEMENTS_INT(tf);

  /* No need to check for Principal Faces as this UDF
     will be used for boundary zones only */

#if RP_NODE
  if(! I_AM_NODE_ZERO_P) /* Nodes 1,2... send the number of faces */
    {
      PRF_CSEND_INT(node_zero, &n_faces, 1, myid);
    }
#endif
/* Allocating memory for arrays on each node */
  p_array=(real * )malloc(n_faces*sizeof(real));
  x_array=(real (*)[ND_ND])malloc(ND_ND*n_faces*sizeof(real));

  begin_f_loop(f, tf)   
        /* Loop over interior faces in the thread, filling p_array
           with face pressure and x_array with centroid   */
    {
      p_array[f] = F_P(f, tf);
      F_CENTROID(x_array[f], f, tf);
    }
  end_f_loop(f, tf)

/* Send data from node 1,2, ... to node 0 */
Message0("\nstart\n");
#if RP_NODE
  if(! I_AM_NODE_ZERO_P) /* Only SEND data from nodes 1,2... */
    {
      PRF_CSEND_REAL(node_zero, p_array, n_faces, myid);
      PRF_CSEND_REAL(node_zero, x_array[0], ND_ND*n_faces, myid);
    }
  else
#endif
    {/* Node-0 and Serial processes have their own data,
          so list it out first */
      Message0("\n\nList of Pressures...\n"); 
              /* Same as Message() on SERIAL */

      for(j=0; j<n_faces; j++) 
          /* n_faces is currently node-0/serial value */
        {
# if RP_3D
         Message0("%12.4e %12.4e %12.4e %12.4e\n",
           x_array[j][0], x_array[j][1], x_array[j][2], p_array[j]);
# else /* 2D */
         Message0("%12.4e %12.4e %12.4e\n",
           x_array[j][0], x_array[j][1], p_array[j]);
# endif
        }
    }

/* Node-0 must now RECV data from the other nodes and list that too */
#if RP_NODE
  if(I_AM_NODE_ZERO_P)
    {
     compute_node_loop_not_zero(i) 
           /* See para.h for definition of this loop */
     {
      PRF_CRECV_INT(i, &n_faces, 1, i);
                    /* n_faces now value for node-i */
      /* Reallocate memory for arrays for node-i */
      p_array=(real *)realloc(p_array, n_faces*sizeof(real));
      x_array=(real(*)[ND_ND])realloc(x_array,ND_ND*n_faces*sizeof(real));

      /* Receive data */
      PRF_CRECV_REAL(i, p_array, n_faces, i);
      PRF_CRECV_REAL(i, x_array[0], ND_ND*n_faces, i);
      for(j=0; j<n_faces; j++)
           {
# if RP_3D
	    Message0("%12.4e %12.4e %12.4e %12.4e\n",
            x_array[j][0], x_array[j][1], x_array[j][2], p_array[j]);
# else /* 2D */
	    Message0("%12.4e %12.4e %12.4e\n",
             x_array[j][0], x_array[j][1], p_array[j]);
# endif
           }
        }
    }
#endif /* RP_NODE */

  free(p_array); /* Each array has to be freed before function exit */
  free(x_array);

#endif /* ! RP_HOST */
}