Parallelism

The MPI interface for C language is much more powerful than that for octave. But basically, it follows the same procedure: initializing MPI environment (world), communication with process id and message tag. I recommend the tutorial from [https://computing.llnl.gov/tutorials/mpi/] as a learning material. It provides many exercises for basic features of MPI in C.

Compile C source file using the command below, for more options, please check the document of the version you installed.

mpicc [YOUR_C_SOURCE_FILE]

• Point-to-point communication
  • blocked MPI_Send(buffer,count,type,dest,tag,comm) MPI_Recv(buffer,count,type,source,tag,comm, status)
    • buffer Buffer is the variable you want to send or you want to receive the data. In C, you have to pass the buffer by reference &var.
    • count Count is the number of data element to be sent since with the address/reference you send, the reciever can read an array.
    • type MPI defiend series of data types for each C type. These types are used for transfering between processes (on network for clusters).
    • dest/source The id (rank) of process you want to send or you want to receive from.
    • tag Tag is used to identify the data you received. Since MPI is an interface that can be used to transfer data through network, it is possible that the data you send later arrives earlier. If the processes of your MPI application have to communicate multiple times, tag is necessary to receive the data and execute on them in correct order.
    • comm The communicator or called MPI_COMM_WORLD, which is the MPI context created at the beginning of application.
    • status Em, it is status, but I have no idea what it exactly is…
  • non-blocked MPI_ISend(buffer,count,type,dest,tag,comm,request) MPI_IRecv(buffer,count,type,source,tag,comm,request) First, I have to clarify the blocking/non-blocking concept in this context. It is different to the block/non-block in multiple thread programming. Blocking means the application will not continue until the data in the application buffer is completely sent out (through internet protocol). Non-blocking means the application continue without waiting. Pay attention, it wait until the data is sent out, not received by other processes. And I think that is why tag is necessary in both cases, because if the application block until data received, the tag is not necessary and the application becomes sequential. Besides, you cannot put receive before send because it is blocked.

    What is the benefit of non-blocking? You don’t have to wait for the receiving of data. In the example code below, we can see that processes receive data before sending them.

    #include "mpi.h"
    #include <stdio.h>
    
    main(int argc, char *argv[])  {
    int numtasks, rank, next, prev, buf[2], tag1=1, tag2=2;
    MPI_Request reqs[4];
    MPI_Status stats[4];
    
    MPI_Init(&argc,&argv);
    MPI_Comm_size(MPI_COMM_WORLD, &numtasks);
    MPI_Comm_rank(MPI_COMM_WORLD, &rank);
    
    prev = rank-1;
    next = rank+1;
    if (rank == 0)  prev = numtasks - 1;
    if (rank == (numtasks - 1))  next = 0;
    
    MPI_Irecv(&buf[0], 1, MPI_INT, prev, tag1, MPI_COMM_WORLD, &reqs[0]);
    MPI_Irecv(&buf[1], 1, MPI_INT, next, tag2, MPI_COMM_WORLD, &reqs[1]);
    
    MPI_Isend(&rank, 1, MPI_INT, prev, tag2, MPI_COMM_WORLD, &reqs[2]);
    MPI_Isend(&rank, 1, MPI_INT, next, tag1, MPI_COMM_WORLD, &reqs[3]);
    
          {  do some work  }
    
    MPI_Waitall(4, reqs, stats);
    
    MPI_Finalize();
    }
    

• Collective communication Basically, all the collective communications can be implemented using the point-to-point communication methods.
  • MPI_Reduce Split data to processes, gather the result as reducing operation such as the sum of array.
  • MPI_Scatter Split data from one processes to all the processes.
  • MPI_Gather Gather data from all the processes to one process
  • MPI_Barrier For synchronizing.
• Derived data type Though the MPI provides series of primitive types, it is also useful to let useser to define their own data type. You can define a variable of type and use it as the MPI type.
  1. Contiguous TODO
  2. Vector TODO
  3. Indexed TODO
  4. Struct TODO
• Group and communicator management Group is an ordered set of processes, where the id(rank) of process go from zero to N-1. Communicator encompoasses a group of processes. Communicator that comprises all tasks is represented by MPI_COMM_WORLD. They are the same thing for programmer.

  Groups can be destroyed and created during execution. One process can be in multiple groups.

• Virtual topology routine Organize the processors in geometric way, like orginizing the processors in a two-dimensional array.