Boost C++ Libraries

Pipes are a facility for communication between different threads, processes and in some cases machines, the operating system provides.

The typical feature of a pipe is, that it is one channel, to which two handles are given, one for reading (source), one for writing (sink). In that it is different than other facilities (like sockets) and provides another way to manage the connectivity: if one side of the pipe is closed (i.e. the pipe is broken), the other is notified.

Pipes are typically used for interprocess communication. The main reason is, that pipes can be directly assigned to the process stdio, i.e. stderr, stdin and stdout. Additionally, half of the pipe can be inherited to the child process and closed in the father process. This will cause the pipe to be broken when the child process exits.

Though please note, that if the same thread reads and writes to a pipe, it will only talk to itself.

A process is an independently executable entity, which is different from a thread, in that it has its own resources. Those include memory and hardware resources.

Every process is identified by a unique number^[27], called the process identification digit, pid.

The environment is a map of variables local to every process. The most significant one for this library is the PATH variable, which contains a list of paths, that ought to be searched for executables. A shell will do this automatically, while this library provides a function for that.

We want to start a process, so let's start with a simple process. We will invoke the gcc compiler to compile a simple program.

With the standard library this looks like this.

int result = std::system("g++ main.cpp");

Which we can write exactly like this in boost.process.

namespace bp = boost::process; //we will assume this for all further examples
int result = bp::system("g++ main.cpp");

If a single string is given (or the explicit form bp::cmd), it will be interpreted as a command line. That will cause the execution function to search the PATH variable to find the executable. The alternative is the exe-args style, where the first string will be interpreted as a filename (including the path), and the rest as arguments passed to said function.

So as a first step, we'll use the exe-args style.

int result = bp::system("/usr/bin/g++", "main.cpp");

With that syntax we still have "g++" hard-coded, so let's assume we get the string from an external source as boost::process::v1::filesystem::path, we can do this too.

boost::process::v1::filesystem::path p = "/usr/bin/g++"; //or get it from somewhere else.
int result = bp::system(p, "main.cpp");

Now we might want to find the g++ executable in the PATH-variable, as the cmd syntax would do. Boost.process provides a function to this end: bp::search_path.

boost::process::v1::filesystem::path p = bp::search_path("g++"); //or get it from somewhere else.
int result = bp::system(p, "main.cpp");

Given that our example used the system function, our program will wait until the child process is completed. This may be unwanted, especially since compiling can take a while.

In order to avoid that, boost.process provides several ways to launch a process. Besides the already mentioned system function and its asynchronous version async_system, we can also use the spawn function or the child class.

The spawn function launches a process and immediately detaches it, so no handle will be returned and the process will be ignored. This is not what we need for compiling, but maybe we want to entertain the user, while compiling:

bp::spawn(bp::search_path("chrome"), "www.boost.org");

Now for the more sensible approach for compiling: a non-blocking execution. To implement that, we directly call the constructor of child.

bp::child c(bp::search_path("g++"), "main.cpp");

while (c.running())
    do_some_stuff();

c.wait(); //wait for the process to exit   
int result = c.exit_code();

So we launch the process, by calling the child constructor. Then we check and do other things while the process is running and afterwards get the exit code. The call to wait is necessary, to obtain it and tell the operating system, that no one is waiting for the process anymore.

Until now, we have assumed that everything works out, but it is not impossible, that "g++" is not present. That will cause the launch of the process to fail. The default behaviour of all functions is to throw a std::system_error on failure. As with many other functions in this library, passing an std::error_code will change the behaviour, so that instead of throwing an exception, the error will be assigned to the error code.

std::error_code ec;
bp::system("g++ main.cpp", ec);

In the examples given above, we have only started a program, but did not consider the output. The default depends on the system, but usually this will just write it to the same output as the launching process. If this shall be guaranteed, the streams can be explicitly forwarded like this.

bp::system("g++ main.cpp", bp::std_out > stdout, bp::std_err > stderr, bp::std_in < stdin);

Now for the first example, we might want to just ignore the output, which can be done by redirecting it to the null-device. This can be achieved this way:

bp::system("g++ main.cpp", bp::std_out > bp::null);

Alternatively we can also easily redirect the output to a file:

bp::system("g++ main.cpp", bp::std_out > "gcc_out.log");

Now, let's take a more visual example for reading data. nm is a tool on posix, which reads the outline, i.e. a list of all entry points, of a binary. Every entry point will be put into a single line, and we will use a pipe to read it. At the end an empty line is appended, which we use as the indication to stop reading. Boost.process provides the pipestream (ipstream, opstream, pstream) to wrap around the pipe and provide an implementation of the std::istream, std::ostream and std::iostream interface.

std::vector<std::string> read_outline(std::string & file)
{
    bp::ipstream is; //reading pipe-stream
    bp::child c(bp::search_path("nm"), file, bp::std_out > is);

    std::vector<std::string> data;
    std::string line;

    while (c.running() && std::getline(is, line) && !line.empty())
        data.push_back(line);

    c.wait();

    return data;
}

What this does is redirect the stdout of the process into a pipe and we read this synchronously.

Now we get the name from nm and we might want to demangle it, so we use input and output. nm has a demangle option, but for the sake of the example, we'll use c++filt for this.

bp::opstream in;
bp::ipstream out;

bp::child c("c++filt", std_out > out, std_in < in);

in << "_ZN5boost7process8tutorialE" << endl;
std::string value;
out >> value;

c.terminate();

Now you might want to forward output from one process to another processes input.

std::vector<std::string> read_demangled_outline(const std::string & file)
{
    bp::pipe p;
    bp::ipstream is;

    std::vector<std::string> outline;

    //we just use the same pipe, so the output of nm is directly passed as input to c++filt
    bp::child nm(bp::search_path("nm"), file,  bp::std_out > p);
    bp::child filt(bp::search_path("c++filt"), bp::std_in < p, bp::std_out > is);

    std::string line;
    while (filt.running() && std::getline(is, line)) //when nm finished the pipe closes and c++filt exits
        outline.push_back(line);

    nm.wait();
    filt.wait();
}

This forwards the data from nm to c++filt without your process needing to do anything.

Boost.process allows the usage of boost.asio to implement asynchronous I/O. If you are familiar with boost.asio (which we highly recommend), you can use async_pipe which is implemented as an I/O-Object and can be used like pipe as shown above.

Now we get back to our compiling example. For nm we might analyze the output line by line, but the compiler output will just be put into one large buffer.

With boost.asio this is what it looks like.

boost::asio::io_service ios;
std::vector<char> buf(4096);

bp::async_pipe ap(ios);

bp::child c(bp::search_path("g++"), "main.cpp", bp::std_out > ap);

boost::asio::async_read(ap, boost::asio::buffer(buf),
                [](const boost::system::error_code &ec, std::size_t size){});

ios.run();
int result = c.exit_code();

To make it easier, boost.process provides a simpler interface for that, so that the buffer can be passed directly, provided we also pass a reference to an boost::asio::io_service.

boost::asio::io_service ios;
std::vector<char> buf(4096);

bp::child c(bp::search_path("g++"), "main.cpp", bp::std_out > boost::asio::buffer(buf), ios);

ios.run();
int result = c.exit_code();

To make it even easier, you can use std::future for asynchronous operations (you will still need to pass a reference to a boost::asio::io_service) to the launching function, unless you use bp::system or bp::async_system.

Now we will revisit our first example and read the compiler output asynchronously:

boost::asio::boost::asio::io_service ios;

std::future<std::string> data;

child c("g++", "main.cpp", //set the input
        bp::std_in.close(),
        bp::std_out > bp::null, //so it can be written without anything
        bp::std_err > data,
        ios);


ios.run(); //this will actually block until the compiler is finished

auto err =  data.get();

When launching several processes, they can be grouped together. This will also apply for a child process, that launches other processes, if they do not modify the group membership. E.g. if you call make which launches other processes and call terminate on it, it will not terminate all the child processes of the child unless you use a group.

The two main reasons to use groups are:

If we have a program like make, which does launch its own child processes, a call of terminate might not suffice. I.e. if we have a makefile launching gcc and use the following code, the gcc process will still run afterwards:

bp::child c("make");
if (!c.wait_for(std::chrono::seconds(10))) //give it 10 seconds
    c.terminate(); //then terminate

So in order to also terminate gcc we can use a group.

bp::group g;
bp::child c("make", g);
if (!g.wait_for(std::chrono::seconds(10)))
    g.terminate();

c.wait(); //to avoid a zombie process & get the exit code

Now given the example, we still call wait to avoid a zombie process. An easier solution for that might be to use spawn.

To put two processes into one group, the following code suffices. Spawn already launches a detached process (i.e. without a child-handle), but they can be grouped, to that in the case of a problem, RAII is still a given.

void f()
{
    bp::group g;
    bp::spawn("foo", g);
    bp::spawn("bar", g);

    do_something();

    g.wait();
};

In the example, it will wait for both processes at the end of the function unless an exception occurs. I.e. if an exception is thrown, the group will be terminated.

Please see the boost/process/group.hpp for more information.

This library provides access to the environment of the current process and allows setting it for the child process.

//get a handle to the current environment
auto env = boost::this_process::environment();
//add a variable to the current environment
env["VALUE_1"] = "foo";

//copy it into an environment separate to the one of this process
bp::environment env_ = env;
//append two values to a variable in the new env
env_["VALUE_2"] += {"bar1", "bar2"};

//launch a process with `env_`
bp::system("stuff", env_);

A more convenient way to modify the environment for the child is the env property, which can be used in the example as following:

bp::system("stuff", bp::env["VALUE_1"]="foo", bp::env["VALUE_2"]+={"bar1", "bar2"});

Please see the boost/process/environment.hpp for more information.

This library is meant to give a wrapper around the different OS-specific methods to launch processes. Its aim is to provide all functionality that is available on those systems and allow the user to do all related things, which require using the OS APIs.

This library does not try to provide a full library for everything process related. In many discussions the proposal was made to build boost.process into a DSEL ^[28] of some sort. This is not the goal, it rather provides the facilities to build such a DSEL-library on top of it. Therefore the library also does not force any particular use (such as only asynchronous communication) on its user. It rather could be integrated with such a library.

Boost.Process does use a very particular style when constructing a process. This is because a process holds many properties, which are not members of the actual child class. Those properties are in many cases not accessible by the father process, for example when using environments. Here the child process can modify its own environment, but there is no way for the father process to know. That means, that a child process has properties that cannot be accessed in C++.

This now leads to the two styles supported and mixed by this library. Overloading and properties. Consider that you may want to launch a process passing a number of arguments. This is supported in both styles, and would look like this:

system("gcc", "--version"); //overloading
system("gcc", args={"--version"}); //property style.

Both styles can also be mixed in some cases.

system("gcc", "-c", args+={"main.cpp"});

In the following section the available styles will be described. Note that the overload style is implemented via type traits, so the types will be listed.

When passing arguments to the process, two styles are provided, the cmd-style and the exe-/args-style.

The cmd style will interpret the string as a sequence of the exe and arguments and parse them as such, while the exe-/args-style will interpret each string as an argument.

When using the overloading variant, a single string will result in a cmd interpretation, several strings will yield a exe-args interpretation. Both versions can be set explicitly:

system("grep -c false /etc/passwd"); //cmd style
system("grep", "-c", "false", "/etc/passwd"); //exe-/args-

system(cmd="grep -c false /etc/passwd"); //cmd style
system(exe="grep", args={"-c", "false", "/etc/passwd"}); //exe-/args-

The simplest form to extend functionality is to provide another handler, which will be called on the respective events on process launching. The names are:

As an example:

child c("ls", on_setup([](){cout << "On Setup" << endl;}));

The executor calls different handlers of the initializers during the process launch. The basic structure consists of three functions, as given below:

Additionally posix provides three more handlers, listed below:

For more information see the reference of posix_executor.

The simplest extension just takes a single handler, which can be done in a functional style. So let's start with a simple hello-world example, while we use a C++14 generic lambda.

using namespace boost::process;
namespace ex = bp::extend;

child c("foo", ex::on_success=[](auto & exec) {std::cout << "hello world" << std::endl;});

Considering that lambdas can also capture values, data can easily be shared between handlers.

To see which members the executor has, refer to windows_executor and posix_executor.

Since the previous example is in a functional style, it is not very reusable. To solve that problem, the handler has an alias in the boost::process::v1::extend namespace, to be inherited. So let's implement the hello world example in a class.

struct hello_world : handler
{
    template<typename Executor>
    void ex::on_success(Executor & exec) const
    {
        std::cout << "hello world" << std::endl;
    }
};

//in our function
child c("foo", hello_world());

Every handler not implemented defaults to handler, where an empty handler is defined for each event.

Since boost.process provides an interface for boost.asio, this functionality is also available for extensions. If the class needs the boost::asio::io_context for some reason, the following code will do that.

struct async_foo : handler, ex::require_io_context
{
    template<typename Executor>
    void on_setup(Executor & exec)
    {
        boost::asio::io_context & ios = ex::get_io_context(exec.seq); //gives us a reference and a compiler error if not present.
        //do something with ios
    }
};

Additionally the handler can provide a function that is invoked when the child process exits. This is done through ex::async_handler.

struct async_bar : __handler, ex::async_handler
{
    template<typename Executor>
    std::function<void(int, const std::error_code&)> on_exit_handler(Executor & exec)
    {
        auto handler_ = this->handler;
        return [handler_](int exit_code, const std::error_code & ec)
               {
                   std::cout << "hello world, I exited with " << exit_code << std::endl;
               };

    }
};

If an error occurs in the initializers it shall be told to the executor and not handled directly. This is because the behaviour can be changed through arguments passed to the launching function. Hence the executor has the function set_error, which takes an std::error_code and a string. Depending on the configuration of the executor, this may either throw, set an internal error_code, or do nothing.

So let's take a simple example, where we set a randomly chosen error_code.

auto set_error = [](auto & exec)
        {
            std::error_code ec{42, std::system_category()};
            exec.set_error(ec, "a fake error");

        };
child c("foo", on_setup=set_error);

Since we do not specify the error-handling mode in this example, this will throw process_error.

Now that we have a custom initializer, let's consider how we can handle differences between different executors. The distinction is between posix and windows and char and wchar_t on windows. One solution is to use the BOOST_WINDOWS_API and BOOST_POSIX_API macros, which are automatically available as soon as any process-header is included.

Another variant are the type aliases ex::posix_executor and ex::windows_executor, where the executor, not on the current system is a forward-declaration. This works fine, because the function will never get invoked. So let's implement another example, which prints the executable name ex::on_success.

struct hello_exe : handler
{
    template<typename Sequence>
    void ex::on_success(ex::posix_executor<Sequence> & exec)
    {
        std::cout << "posix-exe: " << exec.exe << std::endl;
    }

    template<typename Sequence>
    void ex::on_success(ex::windows_executor<char, Sequence> & exec)
    {
        //note: exe might be a nullptr on windows.
        if (exec.exe != nullptr)
            std::cout << "windows-exe: " << exec.exe << std::endl;
        else
            std::cout << "windows didn't use exe" << std::endl;
    }

    template<typename Sequence>
    void ex::on_success(ex::windows_executor<wchar_t, Sequence> & exec)
    {
        //note: exe might be a nullptr on windows.
        if (exec.exe != nullptr)
            std::wcout << L"windows-exe: " << exec.exe << std::endl;
        else
            std::cout << "windows didn't use exe" << std::endl;
    }

};

So given our example, the definitions with the non-native executor are still a template so that they will not be evaluated if not used. Hence this provides a way to implement system-specific code without using the preprocessor.

.

Now let's revisit our c++filt example and we will put in an obvious mistake. This might however be not as obvious for more complex applications.

vector<string> demangle(vector<string> in)
{

    ipstream is;
    opstream os;
    child c("c++filt", std_out > is, std_in < os);

    vector<string> data;
    for (auto & elem : data)
    {
        string line;
        getline(is, line);
        os << elem;
    }
}

We switched the read and write operation up, so that's causing a dead-lock. This locks immediately. This is because c++filt expects input, before outputting any data. The launching process on the other hand waits for its output.

Now for another example, which might look correct, let's consider you want to use ls to read the current directory.

ipstream is;
child c("ls", std_out > is);

std::string file;
while (is >> file)
    cout << "File: " << file << endl;

This will also deadlock, because the pipe does not close when the subprocess exits. So the ipstream will still look for data even though the process has ended.

But, since pipes are buffered, you might get incomplete data if you do this:

ipstream is;
child c("ls", std_out > is);

std::string file;
while (c.running())
{
    is >> file;
    cout << "File: " << file << endl;
}

It is therefore highly recommended that you use the asynchronous API if you are not absolutely sure how the output will look.

Since windows does not use UTF-8 it is sometimes unavoidable to use the wchar_t version of the WinApi. To keep this library consistent it provides wchar_t support on posix also.

Since the posix api is purely char every wchar_t based type will be converted into char.

Windows on the other hand is more selective; the default is to use char, but if any parameter requires wchar_t, everything will be converted to wchar_t. This also includes boost::filesystem::path. Additionally, if the system does not provide the char api (as is the case with Windows CE) everything will also be converted.

namespace boost {
  namespace process {
    namespace v1 {
      class async_pipe;
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      template<typename ExitHandler, typename ... Args> 
        unspecified async_system(boost::asio::io_context &, ExitHandler &&, 
                                 Args &&...);
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      class child;

      typedef unspecified pid_t;  // Typedef for the type of an pid_t. 
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      template<typename Char> class basic_environment;
      template<typename Char> class basic_native_environment;

      typedef basic_native_environment< char > native_environment;  // Definition of the environment for the current process. 
      typedef basic_native_environment< wchar_t > wnative_environment;  // Definition of the environment for the current process. 
      typedef basic_environment< char > environment;  // Type definition to hold a seperate environment. 
      typedef basic_environment< wchar_t > wenvironment;  // Type definition to hold a seperate environment. 
    }
  }
  namespace this_process {

    // Get the process id of the current process. 
    int get_id();

    // Get the native handle of the current process. 
    native_handle_type native_handle();

    // Get the enviroment of the current process. 
    native_environment environment();

    // Get the enviroment of the current process. 
    wnative_environment wenvironment();

    // Get the path environment variable of the current process runs. 
    std::vector< boost::process::v1::filesystem::path > path();
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      struct process_error;
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      namespace extend {
        struct async_handler;
        struct handler;
        template<typename Sequence> struct posix_executor;
        struct require_io_context;
        template<typename Char, typename Sequence> struct windows_executor;

        unspecified on_setup;        // This handler is invoked before the process in launched, to setup parameters. The required signature is void(Exec &), where Exec is a template parameter. 
        unspecified on_error;        // This handler is invoked if an error occurred. The required signature is void(auto & exec, const std::error_code&), where Exec is a template parameter. 
        unspecified on_success;        // This handler is invoked if launching the process has succeeded. The required signature is void(auto & exec), where Exec is a template parameter. 
        unspecified on_fork_error;        // This handler is invoked if the fork failed. The required signature is void(auto & exec), where Exec is a template parameter. 
        unspecified on_exec_setup;        // This handler is invoked if the fork succeeded. The required signature is void(Exec &), where Exec is a template parameter. 
        unspecified on_exec_error;        // This handler is invoked if the exec call errored. The required signature is void(auto & exec), where Exec is a template parameter. 

        // Helper function to get the last error code system-independent. 
        std::error_code get_last_error();
        void throw_last_error(const std::string &);
        void throw_last_error();
        template<typename Sequence> 
          asio::io_context & get_io_context(const Sequence &);
      }
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      class group;
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      static unspecified limit_handles;
    }
  }
  namespace this_process {
    typedef unspecified native_handle_type;
    std::vector< native_handle_type > get_handles();
    std::vector< native_handle_type > get_handles(std::error_code & ec);
    bool is_stream_handle(native_handle_type);
    bool is_stream_handle(native_handle_type handle, std::error_code & ec);
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      typedef std::codecvt< wchar_t, char, std::mbstate_t > codecvt_type;  // The internally used type for code conversion. 

      // Internally used error cateory for code conversion. 
      const std::error_category & codecvt_category();

      // Get a reference to the currently used code converter. 
      const codecvt_type & codecvt();

      // Set the locale of the library. 
      std::locale imbue(const std::locale & loc);
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      template<typename CharT, typename Traits = std::char_traits<CharT> > 
        class basic_ipstream;
      template<typename CharT, typename Traits = std::char_traits<CharT> > 
        class basic_opstream;
      template<typename CharT, typename Traits = std::char_traits<CharT> > 
        class basic_pipe;

      template<typename CharT, typename Traits = std::char_traits<CharT> > 
        struct basic_pipebuf;

      template<typename CharT, typename Traits = std::char_traits<CharT> > 
        class basic_pstream;

      typedef basic_pipe< char > pipe;
      typedef basic_pipe< wchar_t > wpipe;
      typedef basic_pipebuf< char > pipebuf;
      typedef basic_pipebuf< wchar_t > wpipebuf;
      typedef basic_ipstream< char > ipstream;
      typedef basic_ipstream< wchar_t > wipstream;
      typedef basic_opstream< char > opstream;
      typedef basic_opstream< wchar_t > wopstream;
      typedef basic_pstream< char > pstream;
      typedef basic_pstream< wchar_t > wpstream;
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      boost::process::v1::filesystem::path 
      search_path(const boost::process::v1::filesystem::path &, 
                  const std::vector< boost::process::v1::filesystem::path > = ::boost::this_process::path());
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      template<typename ... Args> void spawn(Args &&...);
    }
  }
}

namespace boost {
  namespace process {
    namespace v1 {
      template<typename ... Args> int system(Args &&...);
    }
  }
}