Collective Data Exchange

The functions introduced so far share a one-to-one relationship: that is, one process sends and one process receives. The link is established through a tag. This section introduces functions that are called with the same parameters in multiple processes but execute different operations. For one process the function might send data, for another process it might receive data. These functions are called collective operations.

#include <boost/mpi.hpp>
#include <boost/serialization/string.hpp>
#include <vector>
#include <string>
#include <iterator>
#include <algorithm>
#include <iostream>

int main(int argc, char *argv[])
{
  boost::mpi::environment env{argc, argv};
  boost::mpi::communicator world;
  if (world.rank() == 0)
  {
    std::vector<std::string> v;
    boost::mpi::gather<std::string>(world, "", v, 0);
    std::ostream_iterator<std::string> out{std::cout, "\n"};
    std::copy(v.begin(), v.end(), out);
  }
  else if (world.rank() == 1)
  {
    boost::mpi::gather(world, std::string{"Hello, world!"}, 0);
  }
  else if (world.rank() == 2)
  {
    boost::mpi::gather(world, std::string{"Hello, moon!"}, 0);
  }
}

Example 47.9 calls the function boost::mpi::gather() in multiple processes. Whether the function sends or receives depends on the parameters.

The processes with the ranks 1 and 2 use boost::mpi::gather() to send data. They pass, as parameters, the data being sent – the strings “Hello, world!” and “Hello, moon!” – and the rank of the process the data should be transmitted to. Since boost::mpi::gather() isn’t a member function, the communicator world also has to be passed.

The process with rank 0 calls boost::mpi::gather() to receive data. Since the data has to be stored somewhere, an object of type std::vector<std::string> is passed. Please note that you have to use this type with boost::mpi::gather(). No other containers or string types are supported.

The process with rank 0 has to pass the same parameters as the processes with rank 1 and 2. That’s why the process with rank 0 also passes world, a string to send, and 0 to boost::mpi::gather().

If you start Example 47.9 with three processes, Hello, world! and Hello, moon! are displayed. If you look at the output carefully, you’ll notice that an empty line is written first. The first line is the empty string the process with rank 0 passes to boost::mpi::gather(). There are three strings in v which were received from the processes with the ranks 0, 1 and 2. The indexes of the elements in the vector correspond to the ranks of the processes. If you run the example multiple times, you’ll always get an empty string as a first element in the vector, “Hello, world!” as the second element and “Hello, moon!” as the third one.

Please note that you must not run Example 47.9 with more than three processes. If you start mpiexec with, for example, -n 4, no data is displayed. The program will hang and will have to be aborted with CTRL+C.

Collective operations must be executed for all processes. If your program calls a function such as boost::mpi::gather(), you have to make sure that the function is called in all processes. Otherwise it’s a violation of the MPI standard. Because a function like boost::mpi::gather() has to be called by all processes, the call is usually not different per process, as in Example 47.9. Compare the previous example with Example 47.10, which does the same thing.

#include <boost/mpi.hpp>
#include <boost/serialization/string.hpp>
#include <vector>
#include <string>
#include <iterator>
#include <algorithm>
#include <iostream>

int main(int argc, char *argv[])
{
  boost::mpi::environment env{argc, argv};
  boost::mpi::communicator world;
  std::string s;
  if (world.rank() == 1)
    s = "Hello, world!";
  else if (world.rank() == 2)
    s = "Hello, moon!";
  std::vector<std::string> v;
  boost::mpi::gather(world, s, v, 0);
  std::ostream_iterator<std::string> out{std::cout, "\n"};
  std::copy(v.begin(), v.end(), out);
}

You call functions for collective operations in all processes. Usually the functions are defined in a way that it’s clear which operation has to be executed, even if all processes pass the same parameters.

Example 47.10 uses boost::mpi::gather(), which gathers data. The data is gathered in the process whose rank is passed as the last parameter to boost::mpi::gather(). This process gathers the data it receives from all processes. The vector to store data is used exclusively by the process that gathers data.

boost::mpi::gather() gathers data from all processes. This includes the process that gathers data. In Example 47.10, that is the process with rank 0. This process sends an empty string to itself in s. The empty string is stored in v. As you’ll see in the following examples, collective operations always include all processes.

You can run Example 47.10 with as many processes as you like because every process calls boost::mpi::gather(). If you run the example with three processes, the result will be similar to the previous example.

#include <boost/mpi.hpp>
#include <boost/serialization/string.hpp>
#include <vector>
#include <string>
#include <iostream>

int main(int argc, char *argv[])
{
  boost::mpi::environment env{argc, argv};
  boost::mpi::communicator world;
  std::vector<std::string> v{"Hello, world!", "Hello, moon!",
    "Hello, sun!"};
  std::string s;
  boost::mpi::scatter(world, v, s, 0);
  std::cout << world.rank() << ": " << s << '\n';
}

Example 47.11 introduces the function boost::mpi::scatter(). It does the opposite of boost::mpi::gather(). While boost::mpi::gather() gathers data from multiple processes in one process, boost::mpi::scatter() scatters data from one process across multiple processes.

Example 47.11 scatters the data in v from the process with rank 0 across all processes, including itself. The process with rank 0 receives the string “Hello, world!” in s, the process with rank 1 receives “Hello, moon!” in s, and the process with rank 2 receives “Hello, sun!” in s.

#include <boost/mpi.hpp>
#include <boost/serialization/string.hpp>
#include <string>
#include <iostream>

int main(int argc, char *argv[])
{
  boost::mpi::environment env{argc, argv};
  boost::mpi::communicator world;
  std::string s;
  if (world.rank() == 0)
    s = "Hello, world!";
  boost::mpi::broadcast(world, s, 0);
  std::cout << s << '\n';
}

boost::mpi::broadcast() sends data from a process to all processes. The difference between this function and boost::mpi::scatter() is that the same data is sent to all processes. In Example 47.12, all processes receive the string “Hello, world!” in s and write Hello, world! to the standard output stream.

#include <boost/mpi.hpp>
#include <boost/serialization/string.hpp>
#include <string>
#include <iostream>

std::string min(const std::string &lhs, const std::string &rhs)
{
  return lhs.size() < rhs.size() ? lhs : rhs;
}

int main(int argc, char *argv[])
{
  boost::mpi::environment env{argc, argv};
  boost::mpi::communicator world;
  std::string s;
  if (world.rank() == 0)
    s = "Hello, world!";
  else if (world.rank() == 1)
    s = "Hello, moon!";
  else if (world.rank() == 2)
    s = "Hello, sun!";
  std::string result;
  boost::mpi::reduce(world, s, result, min, 0);
  if (world.rank() == 0)
    std::cout << result << '\n';
}

boost::mpi::reduce() gathers data from multiple processes like boost::mpi::gather(). However, the data isn’t stored in a vector. boost::mpi::reduce() expects a function or function object, which it will use to analyze the data.

If you run Example 47.13 with three processes, the process with rank 0 receives the string “Hello, sun!” in result. The call to boost::mpi::reduce() gathers and analyzes the strings that all of the processes pass to it. They are analyzed using the function min(), which is passed as the fourth parameter to boost::mpi::reduce(). min() compares two strings and returns the shorter one.

If you run Example 47.13 with more than three processes, an empty string is displayed because all processes with a rank greater than 2 will pass an empty string to boost::mpi::reduce(). The empty string will be displayed because it is shorter than “Hello, sun!”

#include <boost/mpi.hpp>
#include <boost/serialization/string.hpp>
#include <string>
#include <iostream>

std::string min(const std::string &lhs, const std::string &rhs)
{
  return lhs.size() < rhs.size() ? lhs : rhs;
}

int main(int argc, char *argv[])
{
  boost::mpi::environment env{argc, argv};
  boost::mpi::communicator world;
  std::string s;
  if (world.rank() == 0)
    s = "Hello, world!";
  else if (world.rank() == 1)
    s = "Hello, moon!";
  else if (world.rank() == 2)
    s = "Hello, sun!";
  std::string result;
  boost::mpi::all_reduce(world, s, result, min);
  std::cout << world.rank() << ": " << result << '\n';
}

Example 47.14 uses the function boost::mpi::all_reduce(), which gathers and analyzes data like boost::mpi::reduce(). The difference between the two functions is that boost::mpi::all_reduce() sends the result of the analysis to all processes while boost::mpi::reduce() makes the result only available to the process whose rank is passed as the last parameter. Thus, no rank is passed to boost::mpi::all_reduce(). If you run Example 47.14 with three processes, every process writes Hello, sun! to the standard output stream.