I'm working on a Nix package for MusicGrabber, when I found an interesting line in its docker-compose.yml:
services:
music-grabber:
# Browser fallbacks need shared memory: Spotify playlists and Monochrome authentication can fail without this.
shm_size: '2gb'
I've been consuming Compose files for almost 6 years by now, and have written many. I've never seen this parameter! I was curious what it could mean, and knew it had to have something to do with Chromium, and ended up learning something really interesting about how inter-process communication works on Unix systems.
To give some context, Chromium (and many other browsers) work as multi-process applications; for example, each tab is its own process, which is obvious if you've ever peeked into a process monitor while Chrome or another browser is running. These processes need to talk with each other - for example, the renderer process needs to send and receive data from the tab processes. The normal method of IPC is by sending messages directly from one process to another. However, this is insufficient for all applications; in the example above, a renderer process may need to send image data over to a tab process, which is infeasible to send over a normal 1:1 IPC connection. This is where shared memory comes in; memory is usually broken down into virtual "pages", and each page is mapped to a specific process. Shared memory allows two processes to map the same page of memory, therefore allowing them to share data in RAM rather than on disk.
shm_size
We'll start with the meat of the matter - shm_size determines the bounds of /dev/shm in the container, which is what older versions of Chromium use for shared memory. Named memory pages are created as objects under /dev/shm, which is a real tmpfs filesystem mounted on a Linux machine. These objects can be referred to by multiple processes on the same machine for IPC, though my agent would like me to note that more direct forms of IPC are still needed due to no native conflict mitigation existing - processes must still set locks when accessing the same region of memory.
The default for Docker containers, which are essentially tiny Linux filesystems, is to assign /dev/shm 64MiB of memory. When running Chromium in a container, this needs to be increased; the failure modes if you don't are apparently opaque, with random crashes instead of OOM (out of memory) errors (though I cannot attest to this personally, having never encountered it).
So, that's it right? WRONG! Astute readers may have noticed I mentioned "older versions of Chromium" using /dev/shm for shared memory; this is where we talk about how POSIX shared memory can be used for Linux applications today, rather than the older /dev/shm way.
memfd_create
memfd_create, contributed by David Rheinsburg to the Linux kernel, is a syscall (kernel level function) released in kernel version 3.17 that allows a process to create an anonymous memory entry, with some benefits over using /dev/shm like allowing "sealing" (setting permissions for modifications to the size and access to the entry). Later versions of Chromium already use this with /dev/shm as a fallback, but the pinned version in MusicGrabber still uses it, ergo shm_size.
What about MacOS?
MacOS works slightly differently, though maintaining the same shm_open POSIX API. The XNU kernel used by it utilises a BSD (read: POSIX) layer on top of the Mach virtual memory (VM) system, which provides primitives for managing shared memory.
What about BSD?
I don't know.
See you tomorrow!