POSIX Shared Memory in Linux

Shared Memory

Shared memory is an inter process communication (IPC) mechanism in Linux and other UNIX-like systems. Based on input parameters, the kernel provides a (shared) memory segment to the calling process. The calling process maps the shared memory segment to its address space. This way, the same shared memory segment can be mapped to the address space of multiple processes. As these processes can read and write to the shared memory segment, they can communicate with one another using it. The processes need to synchronize the usage of the shared memory segment by using semaphores so that multiple processes do not write at a location at the same time. Also, a process might need a signal to know that another process has written data in the shared memory so that it need not do a busy wait for data in the shared memory.

If we look at the other IPC mechanisms like message queues or the older mechanisms like pipes or fifos, the work required for passing message involves copying the message contents at least twice. The first process makes a send-like call and the data is copied from its address space to the kernel space. Then, the second process makes a receive-like call, and the message data is copied from the kernel to the address space of the second process. In the case of shared memory, the shared memory segment is mapped to the address space of both processes. As soon as the first process writes data in the shared memory segment, it becomes available to the second process. This makes shared memory faster than other mechanisms and is, in fact, the fastest way of passing data between two processes on the same host system.

In UNIX-like systems, shared memory, semaphores and message queues are designated as the three IPC mechanisms. For each mechanism, there are two sets of calls, the traditional System V calls and the newer POSIX calls. In this post, we will look at the POSIX shared memory calls. Example C programs for server and client processes that communicate via POSIX shared memory are given in the later sections of this post. System V shared memory is described here.

The POSIX shared memory calls seem to be based on the UNIX philosophy that if you do Input/Output operations on an object, that object has to be a file. So, since we do read and write to a POSIX shared memory object, the latter is to be treated as a file. A POSIX shared memory object is a memory-mapped file. POSIX shared memory files are provided from a tmpfs filesystem mounted at /dev/shm. The individual shared memory files are created using the shm_open system call under /dev/shm. There are just two specialized POSIX shared memory system calls, shm_open and shm_unlink, which are analogous to open and unlink system calls for files. Other operations on POSIX shared memory are done using the ftruncate, mmap and munmap system calls for files. A program using POSIX shared memory calls needs to be linked with real time extensions and pthread libraries. That is, the program needs to be linked with -lrt -lpthread.

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

int shm_open (const char *name, int oflag, mode_t mode);

shm_open is like the open system call for files. It opens a POSIX shared memory object and makes it available to the calling process via the returned file descriptor. The first parameter, name, is the name of the shared memory object and is of the form /somename, that is, it is a null-terminated string of a maximum of NAME_MAX characters, and, its first character is a slash and none of the other characters can be a slash. oflag is a bit mask constructed by OR-ing either O_RDONLY or O_RDWR with one or more of the following flags. O_CREAT creates the shared memory object if it does not exist. And, if it creates the object, the last nine bits of the third parameter, mode are taken for permissions except that the bits set in the file mode creation mask are cleared for the object. Also, if the shared memory object is created, the owner and group ids of the object are set the corresponding effective ids of the calling process. A newly created shared memory object is of size zero bytes. It can be made of the desired size by using the ftruncate system call. If O_EXCL flag is used together with the O_CREAT flag, and the shared memory object for the name already exists, shm_open fails and errno is set to EEXIST. There is another flag, O_TRUNC, which if specified, truncates the shared memory object, if it already exists, to size zero bytes. On success, shm_open returns the file descriptor for the shared memory object. If shm_open fails, it returns -1 and errno is set to the cause of the error.

#include <sys/mman.h>
#include <sys/stat.h>
#include <fcntl.h>

int shm_unlink (const char *name);

shm_unlink removes the previously created POSIX shared memory object. The name is the name of the shared memory object as described under shm_open, above.

#include <unistd.h>
#include <sys/types.h>

int ftruncate (int fd, off_t length);

The ftruncate system call makes the object referred to by the file descriptor, fd, of size length bytes. When a POSIX shared memory is created, it is of size zero bytes. Using ftruncate, we can make the POSIX shared memory object of size length bytes. ftruncate returns zero on success. In case of error, ftruncate returns -1 and errno is set to the cause of the error.

#include <sys/mman.h>

void *mmap (void *addr, size_t length, int prot, int flags,
            int fd, off_t offset);

With the mmap system call, we can map a POSIX shared memory object to the calling process’s virtual address space. addr specifies the address at which it should be mapped. In most cases, we do not care at what address mapping is done and a value of NULL for addr should suffice. length is the length of shared memory object that should be mapped. To keep things simple, we will map the whole object and length for us will be the length of the shared memory object. prot can have the values, PROT_EXEC, PROT_READ, PROT_WRITE and PROT_NONE. PROT_EXEC means that the mapped pages may be executed and PROT_NONE means that the mapped pages may not be accessed. These two values do not make sense for a shared memory object. So we will use PROT_READ | PROT_WRITE value for prot. There are many flags but the only one meaningful for shared memory is MAP_SHARED, which means that the updates to the mapped shared memory are visible to all other processes immediately. fd is, of course, the file descriptor for the shared memory received from an earlier shm_open call. offset is the location in the shared memory object at which the mapping starts; we will use the value zero for offset and map the shared memory object starting right from the beginning. On success, mmap returns the pointer to the location where the shared memory object has been mapped. In case of error, MAP_FAILED, which is, (void *) -1, is returned and errno is set to the cause of the error.

#include <sys/mman.h>

int munmap (void *addr, size_t length);

munmap unmapps the shared memory object at location pointed by addr and having size, length. On success, munmap returns 0. In case of error, munmap returns -1 and errno is set to the cause of the error.

Shared memory is normally used with semaphores. If two processes modify a part of the shared memory, it has to be protected with a semaphore for mutual exclusion. That is, only one process should be able to modify the shared memory at a time. Also, processes may need to be synchronized in the sense that a process gets to know when the other has put data in the shared memory and it can process the same.

As an example, consider a log server process, which writes messages to a log file. The log process creates a POSIX shared memory object and maps it to its address space. Clients also map the shared memory object to their address spaces. When a client wants to log a message, it creates a string in the format <client pid><timestamp><message> and writes the string in the shared memory object. The log process reads strings from the shared memory object, one by one, and writes them in a log file in the chronological order. The server and client programs are both in C language.

The server program is,

/*
 *
 *       logger.c: Write strings in POSIX shared memory to file
 *                 (Server process)
 */

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <unistd.h>
#include <semaphore.h>
#include <sys/mman.h>

// Buffer data structures
#define MAX_BUFFERS 10

#define LOGFILE "/tmp/example.log"

#define SEM_MUTEX_NAME "/sem-mutex"
#define SEM_BUFFER_COUNT_NAME "/sem-buffer-count"
#define SEM_SPOOL_SIGNAL_NAME "/sem-spool-signal"
#define SHARED_MEM_NAME "/posix-shared-mem-example"

struct shared_memory {
    char buf [MAX_BUFFERS] [256];
    int buffer_index;
    int buffer_print_index;
};

void error (char *msg);

int main (int argc, char **argv)
{
    struct shared_memory *shared_mem_ptr;
    sem_t *mutex_sem, *buffer_count_sem, *spool_signal_sem;
    int fd_shm, fd_log;
    char mybuf [256];
    
    // Open log file
    if ((fd_log = open (LOGFILE, O_CREAT | O_WRONLY | O_APPEND | O_SYNC, 0666)) == -1)
        error ("fopen");

    //  mutual exclusion semaphore, mutex_sem with an initial value 0.
    if ((mutex_sem = sem_open (SEM_MUTEX_NAME, O_CREAT, 0660, 0)) == SEM_FAILED)
        error ("sem_open");

    // Get shared memory 
    if ((fd_shm = shm_open (SHARED_MEM_NAME, O_RDWR | O_CREAT | O_EXCL, 0660)) == -1)
        error ("shm_open");

    if (ftruncate (fd_shm, sizeof (struct shared_memory)) == -1)
       error ("ftruncate");
    
    if ((shared_mem_ptr = mmap (NULL, sizeof (struct shared_memory), PROT_READ | PROT_WRITE, MAP_SHARED,
            fd_shm, 0)) == MAP_FAILED)
       error ("mmap");
    // Initialize the shared memory
    shared_mem_ptr -> buffer_index = shared_mem_ptr -> buffer_print_index = 0;

    // counting semaphore, indicating the number of available buffers. Initial value = MAX_BUFFERS
    if ((buffer_count_sem = sem_open (SEM_BUFFER_COUNT_NAME, O_CREAT | O_EXCL, 0660, MAX_BUFFERS)) == SEM_FAILED)
        error ("sem_open");

    // counting semaphore, indicating the number of strings to be printed. Initial value = 0
    if ((spool_signal_sem = sem_open (SEM_SPOOL_SIGNAL_NAME, O_CREAT | O_EXCL, 0660, 0)) == SEM_FAILED)
        error ("sem_open");

    // Initialization complete; now we can set mutex semaphore as 1 to 
    // indicate shared memory segment is available
    if (sem_post (mutex_sem) == -1)
        error ("sem_post: mutex_sem");

    while (1) {  // forever
        // Is there a string to print? P (spool_signal_sem);
        if (sem_wait (spool_signal_sem) == -1)
            error ("sem_wait: spool_signal_sem");
    
        strcpy (mybuf, shared_mem_ptr -> buf [shared_mem_ptr -> buffer_print_index]);

        /* Since there is only one process (the logger) using the 
           buffer_print_index, mutex semaphore is not necessary */
        (shared_mem_ptr -> buffer_print_index)++;
        if (shared_mem_ptr -> buffer_print_index == MAX_BUFFERS)
           shared_mem_ptr -> buffer_print_index = 0;

        /* Contents of one buffer has been printed.
           One more buffer is available for use by producers.
           Release buffer: V (buffer_count_sem);  */
        if (sem_post (buffer_count_sem) == -1)
            error ("sem_post: buffer_count_sem");
        
        // write the string to file
        if (write (fd_log, mybuf, strlen (mybuf)) != strlen (mybuf))
            error ("write: logfile");
    }
}

// Print system error and exit
void error (char *msg)
{
    perror (msg);
    exit (1);
}

mutex_sem semaphore is for mutual exclusion. It is for ensuring that only the server or one of the clients access the shared memory at any time.

buffer_count_sem is a counting semaphore. Its value indicates how many buffers are available for writing strings at any time. A client does a P operation on buffer_count_sem semaphore to acquire a buffer for writing. After the server writes the contents of a buffer to the log file, it does a V operation on buffer_count_sem semaphore to release the buffer.

spool_signal_sem is also a counting semaphore. A client writes a string on a buffer and does a V operation on spool_signal_sem semaphore to signal to the server that a string is available for writing to the log file. The server does a P operation on the spool_signal_sem semaphore to acquire a buffer containing a string for writing to the log file.

And, the client code is,

/*
 *
 *       client.c: Write strings for printing in POSIX shared memory object
 *                 
 */

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <string.h>
#include <time.h>
#include <unistd.h>
#include <semaphore.h>
#include <sys/mman.h>

// Buffer data structures
#define MAX_BUFFERS 10

#define LOGFILE "/tmp/example.log"

#define SEM_MUTEX_NAME "/sem-mutex"
#define SEM_BUFFER_COUNT_NAME "/sem-buffer-count"
#define SEM_SPOOL_SIGNAL_NAME "/sem-spool-signal"
#define SHARED_MEM_NAME "/posix-shared-mem-example"

struct shared_memory {
    char buf [MAX_BUFFERS] [256];
    int buffer_index;
    int buffer_print_index;
};

void error (char *msg);

int main (int argc, char **argv)
{
    struct shared_memory *shared_mem_ptr;
    sem_t *mutex_sem, *buffer_count_sem, *spool_signal_sem;
    int fd_shm;
    char mybuf [256];
    
    //  mutual exclusion semaphore, mutex_sem 
    if ((mutex_sem = sem_open (SEM_MUTEX_NAME, 0, 0, 0)) == SEM_FAILED)
        error ("sem_open");
    
    // Get shared memory 
    if ((fd_shm = shm_open (SHARED_MEM_NAME, O_RDWR, 0)) == -1)
        error ("shm_open");

    if ((shared_mem_ptr = mmap (NULL, sizeof (struct shared_memory), PROT_READ | PROT_WRITE, MAP_SHARED,
            fd_shm, 0)) == MAP_FAILED)
       error ("mmap");

    // counting semaphore, indicating the number of available buffers.
    if ((buffer_count_sem = sem_open (SEM_BUFFER_COUNT_NAME, 0, 0, 0)) == SEM_FAILED)
        error ("sem_open");

    // counting semaphore, indicating the number of strings to be printed. Initial value = 0
    if ((spool_signal_sem = sem_open (SEM_SPOOL_SIGNAL_NAME, 0, 0, 0)) == SEM_FAILED)
        error ("sem_open");

    char buf [200], *cp;

    printf ("Please type a message: ");

    while (fgets (buf, 198, stdin)) {
        // remove newline from string
        int length = strlen (buf);
        if (buf [length - 1] == '\n')
           buf [length - 1] = '\0';

        // get a buffer: P (buffer_count_sem);
        if (sem_wait (buffer_count_sem) == -1)
            error ("sem_wait: buffer_count_sem");
    
        /* There might be multiple producers. We must ensure that 
            only one producer uses buffer_index at a time.  */
        // P (mutex_sem);
        if (sem_wait (mutex_sem) == -1)
            error ("sem_wait: mutex_sem");

	    // Critical section
            time_t now = time (NULL);
            cp = ctime (&now);
            int len = strlen (cp);
            if (*(cp + len -1) == '\n')
                *(cp + len -1) = '\0';
            sprintf (shared_mem_ptr -> buf [shared_mem_ptr -> buffer_index], "%d: %s %s\n", getpid (), 
                     cp, buf);
            (shared_mem_ptr -> buffer_index)++;
            if (shared_mem_ptr -> buffer_index == MAX_BUFFERS)
                shared_mem_ptr -> buffer_index = 0;

        // Release mutex sem: V (mutex_sem)
        if (sem_post (mutex_sem) == -1)
            error ("sem_post: mutex_sem");
    
	// Tell spooler that there is a string to print: V (spool_signal_sem);
        if (sem_post (spool_signal_sem) == -1)
            error ("sem_post: (spool_signal_sem");

        printf ("Please type a message: ");
    }
 
    if (munmap (shared_mem_ptr, sizeof (struct shared_memory)) == -1)
        error ("munmap");
    exit (0);
}

// Print system error and exit
void error (char *msg)
{
    perror (msg);
    exit (1);
}

We can compile and run the server and a client as below.

$ gcc logger.c -o logger -lrt -lpthread
$ gcc client.c -o client -lrt -lpthread
$ ./logger &
[1] 3561
$ ps
  PID TTY          TIME CMD
 2924 pts/1    00:00:00 bash
 3561 pts/1    00:00:00 logger
 3562 pts/1    00:00:00 ps
$ ./client
Please type a message: Hello Logger
Please type a message: She sells seashells by the seashore.
...

The Logger writes all messages to the /tmp/example.log file. We can see the contents of the log file with the
tail -f command.

$ tail -f /tmp/example.log 
4535: Mon Jan 15 08:25:10 2018 Hello Logger
4582: Mon Jan 15 08:25:49 2018 Hello Logger
4535: Mon Jan 15 08:26:29 2018 It is a beautiful day!
4598: Mon Jan 15 08:26:57 2018 Hello Logger
4623: Mon Jan 15 08:27:17 2018 Hello Logger
4623: Mon Jan 15 08:27:47 2018 So foul and fair a day I have not seen. (William Shakespeare: Macbeth)
4535: Mon Jan 15 08:28:07 2018 He that lives upon hope will die fasting. (Benjamin Franklin)
4582: Mon Jan 15 08:28:32 2018 She sells seashells by the seashore.
4598: Mon Jan 15 08:29:09 2018 Not so happy, yet much happier. (William Shakespeare: Macbeth)

Since the Logger runs in an infinite loop and is terminated with a Control-C or the kill command, the semaphore files are left lying in the /dev/shm directory. These need to be removed between successive runs of Logger. That is,

$ cd /dev/shm
abc@node1:/dev/shm$ ls -ls
total 16
4 -rw-r----- 1 abc abc 2568 Jun 13 08:40 posix-shared-mem-example
4 -rw-r----- 1 abc abc   32 Jun 13 08:40 sem.sem-buffer-count
4 -rw-r----- 1 abc abc   32 Jun 13 08:40 sem.sem-mutex
4 -rw-r----- 1 abc abc   32 Jun 13 08:40 sem.sem-spool-signal
abc@node1:/dev/shm$ rm posix-shared-mem-example sem.sem-buffer-count sem.sem-mutex sem.sem-spool-signal 

1. POSIX Semaphores
2. Interprocess communication using System V Shared Memory in Linux