Explaining OverlayFS – What it Does and How it Works

Posted by: Thom Denholm

Union file systems are a creative solution to allow a virtual merge of multiple folders, while keeping their actual contents separate. The Overlay file system (OverlayFS) is one example of these, though it is more of a mounting mechanism than a file system.

Brought into the Linux kernel mainline with version 3.18, OverlayFS allows you to overlay the contents (both files and directories) of one directory onto another. The source directories can be on different volumes and can even be different file systems, which creates an interesting mechanism for allowing temporary modification of read-only files and folders.

The simplest case (image below) involves two directories, each containing files and folders. We can think of them as “upper” and “lower,” with the rest of Linux and applications positioned above that. The “lower” directory is read-only. File access through the OverlayFS retrieves data from the “upper” directory first, and then defaults to the “lower” directory if a file doesn’t exist.

Linux OverlayFS allows you to overlay the contents of one directory onto another

Note that the two original directories (“upper” and “lower”) are still directly accessible to the Linux kernel, but this access could be limited by the application.

Modifications to files in the “upper” directory will take place as usual. Any modification to a file from the “lower” folder will create a copy in the upper directory, and that file will be the one modified. This leaves the base files untouched and available through direct access to the “lower” folder.

Interestingly, a second task could copy modified files from the “upper” folder to the “lower” when modifications are complete. In this way, an OverlayFS setup could simulate some of the functionality of transaction points within the Reliance Nitro file system.

A file removed from the OverlayFS directory would directly remove a file from the “upper” directory, and simulate that removal from the “lower” directory by creating what is called a “whiteout” file. This file exists only within the OverlayFS directory, without physically appearing in either the “upper” or “lower” directories. When the OverlayFS is dismounted, this state information will be lost, so care should be taken to reflect any necessary changes to the “lower” directory.

A subdirectory can also be deleted from the “lower” directory, which creates what is known as an “opaque directory” in the OverlayFS directory. Behind the scenes, OverlayFS uses the “trusted” extended attribute class or namespace to record “whiteouts” and “opaque directories.” Linux file systems that support the “trusted” namespace can be used for either, and Reliance Nitro is among that set.

Have you used OverlayFS in your projects? Drop me a line or comment below.

Discover Reliance Nitro

Comments (6)

  1. Olga Levy:
    May 21, 2018 at 02:33 AM

    How can I implement a fitering mechanism so that certain files will be written in lower dir?

  2. Thom Denholm:
    May 22, 2018 at 10:00 AM

    Offline changes are allowed to either the upper or lower file systems, but the "lower" directory is meant to be read-only while the OverlayFS is mounted. Per the documentation, if the underlying filesystem is changed, the behavior of the overlay is undefined.

    See https://www.kernel.org/doc/Documentation/filesystems/overlayfs.txt

  3. ranran:
    Oct 10, 2018 at 02:59 PM

    Hello Thom,

    Thank you for the tutorial.
    Maybe you can help with a problem I have ?
    I have an issue related to overlay filesystem, and I don't understand how to overcome it. :cry:
    I hope someone can help...

    I have vncserver (tigervnc), and in startup I see that vncserver service has started for all users (root+user) have vncserver (tigervnc).

    Recently we added overlayfs using init script as done here: https://blockdev.io/read-only-rpi/

    As you can see in above link, the overlay is above rootfs (rootfs is readonly).

    At first, it seemed to me that all works the same (as without overlay).

    But today I noticed a strange thing: With overlay rootfs, only the root's vncserver service has started, not the user's vncserver. I have no idea why.

    I actually assume that EVERYTHING should be the same whether I use overlay or not, including all the started services.

    What can be the reason for this change in behavior of started services per user with overlay ?

    Thanks for any idea,

  4. stuartiannaylor:
    Apr 18, 2019 at 04:15 AM

    I use an OverlayFS with https://github.com/StuartIanNaylor/zram-config for zramdrives & zramlog

    There is a complete lack of official tools for OverlayFS which just seems to be a little weird.
    https://github.com/kmxz/overlayfs-tools is the only tool but has a great offline merge down to single lower method.
    I am trying to get kmxz to kindly update it as redirect_dir has been added since they original repo was done.

  5. Thom Denholm:
    Apr 18, 2019 at 11:12 AM

    The lack of tools is disappointing. The folks at Suse who championed this have been quiet.

    Our team at Datalight was excited about OverlayFS because it effectively acts as a copy-on-write file system. This is our focus due to the requirements of NAND flash media. Maybe we should expand into the tool department.

  6. stuartiannaylor:
    Apr 18, 2019 at 11:41 AM

    Maybe if we start making noise on over at linux-unionfs@vger.kernel.org but the C behind https://github.com/kmxz/overlayfs-tools is beyound me and exactly what effect redirect_dir will have and does have isn't clear to me.
    I could use Aufs if it didn't mean a custom kernel compile for so many distro's now and know there must be strong rationales why Aufs never got mainline, but providing OverlayFS minus any tools seems an oversight.
    I would really like to use volatile non wear zram as the uppper but have some means to merge down to just a single lower persistent.
    https://github.com/kmxz/overlayfs-tools allows that but apparently was written before the redirect_dir of overlayFS was written so presuming on directory renames there could be problems, but to be honest not sure.
    OverlayFS is really interesting as Snapshots are also another great tool but again rotate down would be amazingly useful but as a ready toolset there is nothing and from what I gather nothing planned.

Add a Comment

Allowed tags: <b><i><br>Add a new comment: