Neither one is as accessible imo, fedora has long history of distro making and it shows

Same thing for OpenWrt I think. I believe it works by using a squashfs and tmpfs and using overlayfs to overlay the tmpfs on top of the r/o filesystem. But I'm not sure that fits the definition used here for immutable OS:

> We could say that a Linux LIVE-CD is immutable, because every time you boot it, you get the exact same programs running, and you can't change anything as the disk media is read only. But while the LIVE-CD is running, you can make changes to it, you can create files and directories, install packages, it's not stuck in an immutable state.

Last I looked at Distrobox, DNS didn't work on Ubuntu LTS or Debian Bookworm. It was literally useless.

How can anyone adopt an unpolished hobby project that only seems to be tested on the dev's Arch box is beyond me.

Distrobox and toolbox are developer tools, they explicitly are there as a convenience wrapper for podman/docker for development.

If you want hard separation you'd need a VM. It'd be awesome if you could just --firecracker on distrobox create and get that. :D

Oh, CoreOS Layering looks really useful! I'm using openSUSE MicroOS today on some Raspberry Pi's and a x86_64 server. One reason I picked MicroOS was because it was quite simple to install on Raspberry Pi.

How hard is it to install CoreOS on a Raspberry Pis? Some installation guides on the Internet look quite complex...?

Thanks for making this tool and showing how to get started with the layering!

Do you have any thoughts you’d like to share on flatcar as the other project with CoreOS lineage?

As for me my main difficulties have been figuring out what to do with these projects in a bare metal environment. Building VM images is cool, but much of the time I want to do things like install to an existing drive or even onto a ZFS pool underneath.

> stacking of custom overlay filesystems as a first class feature

At that point why not just use a mutable system? This reminds me of how a decade ago everybody rushed to move to nosql and then immediately reinvented schemae in their projects.

You can add plugins to applications in flatpak; package these plugins as extensions.

How it is done can be seen with OBS: flathub has several OBS plugins available (com.obsproject.Studio.Plugin.*).

> add the decsync plugin to the evolution flatpak

This sort of thing is very common with GUI software on Linux. Outside of this bubble, most software comes with batteries included. Eg Solidworks never makes me download optional dependencies, but FreeCAD made me do it literally every 15 minutes as I move to the next step in a CAD/CAM/sim/render workflow.

Also see https://www.joelonsoftware.com/2001/03/23/strategy-letter-iv...

> it’s never the same 20%. Everybody uses a different set of features. In the last 10 years I have probably heard of dozens of companies who, determined not to learn from each other, tried to release “lite” word processors that only implement 20% of the features. This story is as old as the PC. Most of the time, what happens is that they give their program to a journalist to review, and the journalist reviews it by writing their review using the new word processor, and then the journalist tries to find the “word count” feature which they need because most journalists have precise word count requirements, and it’s not there, because it’s in the “80% that nobody uses,” and the journalist ends up writing a story that attempts to claim simultaneously that lite programs are good, bloat is bad, and I can’t use this damn thing ’cause it won’t count my words. If I had a dollar for every time this has happened I would be very happy.

NixOS gives you precisely this kind of control- through a few different mechanisms. Some packages in nixpkgs, and most NixOS and home manager modules, expose a lot of configuration options where you can configure various plugins, add extra packages, etc. Nix also lets you provide overlays and overrides to add custom packages (new packages, or customized versions of existing packages) and gives you the option of changing parts of a package. If none of that is sufficient, you can even supply your own patches to the code, or build the package from your own fork of the upstream repository.

In practice this is one of my favorite things about nix. I find that I contribute much more to open source in general because it’s so easy to say “replace this dependency with my version when you build this package”.

If you did it before and after every system alteration, I'd count that. That's tedious and error-prone to do by hand, though, so you'd want a set of tools to do it automatically. At which point you've invented a new immutable distro.

I'd say yes, to an extent, although you don't get all of the benefits of immutable Linux this way. A good immutable Linux distro does some work to make sure that

  - upgrades/changes are atomic
  - it's very clear whose responsibility it is to track what state, i.e., what parts of the filesystem are snapshotted and when
  - it's easy to revert to any snapshot at boot time
  - you're unlikely to have 'gaps' where you wish there was a snapshot but there isn't one
  - snapshotting applies to configuration changes as well as package (un)installation

You can get most of those benefits of the box on openSUSE or other distros where the default filesystem is copy-on-write and the package manager is configured to take snapshots for you. You can also get something like that for configuration management via etckeeper, which gives you version control for system-wide config files. A distro which only takes these steps may not always be considered 'immutable Linux' as immutable distros typically go further, either in limiting what changes are possible outside the blessed methods (which generate new snapshots) or how what kinds of configuration/persistence it manages. But at some point it's just a question of degrees.

As it happens, snapshotting with automatic reversion is an approach some take with NixOS to better ensure that its configuration management is comprehensive. You can force all of your persistence requirements to be explicit by reverting to an old snapshot on boot.

Blog post which (afaik) first presented this idea to the community: https://grahamc.com/blog/erase-your-darlings/

Common implementation for NixOS: https://github.com/nix-community/impermanence

Presentation on that implementation from NixCon 2023 (just a couple weeks ago!): https://www.youtube.com/watch?v=QtBouFMyrWg

With this, you have a structured, transactional way of writing to whatever system files you wanted, with rollbacks and all. On macOS, only Apple does.

Don't represent everyone.

> ostree is what everyone should be using

I currently use Ubuntu for my server, and I see there's ostree in my repos, I haven't gotten around to trying it yet but I'd like to just start versioning my system as is with it. If that's too difficult/not possible to do, then I intend to switch my server over to Silverblue at some point because I really like the idea behind ostree.

Funny enaugh, once an update rendered my Silverblue install unbootable. I had to boot to a live image to fix GRUB misconfiguration. Now I know it's not invincible, but it does work well otherwise.

For what reasons?

I like the idea of ostree but having glanced at it, as a casual/intermediate user, it didn't seem as user-friendly as, say, Docker was (which a home user can learn in an afternoon). It wasn't obvious how to get to "Debian distro deployed as an ostree snapshot".

Is this one of those things designed for career sysadmins/system builders only?

Have you tried Nix? If so, how does it compare? I haven't tried Silverblue, but Nix also feels like it's the future.

I'm one year in and hell yes. I wonder if Red Hat realize what they have here. Not just in silverblue but in Fedora.

I read a quora answer that estimated the Windows OS development budget at around 18 billion dollars, based on salaries. Imagine if Red Hat invested 2bn into Fedora to make it the Firefox of the desktop OS world. Just a 10% share is very significant against Microsoft.

They've come so far with so little, on the back of thousands of open source packages. That money could be used to keep those projects alive, and to sponsor them while they're being developed. Red Hat employees are already involved in a lot of them.

The day Debian will make this immutable stuff mandatory will be the day I will ditch Linux from my computer.

I found Silverblue to not be flexible enough for my person computers. Maybe I use Linux in a hacky way, not having write access to /usr or /bin or other folders drove me crazy about once every two weeks.

For example, I was using a script written by an ubuntu user that was looking for a library with the name/location ubuntu puts it in. Fedora uses a different name for the library. My instinct here is to create a symbolic link with the Ubuntu name that points to the Fedora rpm managed library. Instead, to get it to work I forked the script, got it to build locally, made it look for either library name, ran local test cases, submitted the code as a PR upstream, etc etc etc. It took something that would normally cost me 1 line of shell to fix to something that took 90 minutes.

Did you finish the article? The author brought up that point, with the caveat that they couldn’t think of a better term.

This is called "reproduceability", meaning the same config always results in the same system state.

How deep do you want that property to apply?

Have a look at 'uniquely represented datastructures' and 'History Independent Data Structures'.

You'd need to take special care to make sure that your block device (eg SSD) allocates blocks independent of history.

I like that. Maybe you could say the package collection forms a lattice, where there is only one state for any subset of packages no matter how you got there.

As far as I can tell, her point is this is effectively impossible, at least for a user-facing system. Do you want all the files you wrote in your word processor program or text editor gone when you uninstall those? All the files you downloaded from your browser gone if you uninstall it? If not, there is no reliable way to tell what files are created by a program automatically and which are created by a human using that program. You can easily enough remove everything that was created during the installation process, but not all future changes.

Beyond that, consider other changes, like say you decide to change DNS implementations and then later decide to change your default DNS server. If you uninstall the provider to change back to previous one, do you also want to change back to the old server you were using or do you want to retain that change? Personally, I'd want to only change the provider but keep the new server.

Then consider difficulties with directories shared across multiple machines. Let's say you have /home/${USER} set up as an NFS or Samba mount so you can keep the same files across multiple workstations. If a program respects XDG config dirs and stores stuff in there and you uninstall on one workstation, should it remove the files from all of them? Do you want all your devices to be identical or do you only want the home directory to be identical? There is no possible way for a package manager on a single system to know this.

"Immutable" is a strange term for this:

* system upgrades aren't done on the live system

* packages changes are applied on the next boot

* you can roll back a change

That's a atomic transaction, like a database. Although having to shut down the system to do a commit is a bit much.

Microsoft put atomic transactions into their file system years ago, but file system transactions were never used much. You'd like to have an install system where all changes commit all at once, and if anything goes wrong during install, nothing commits and you roll back to the previous state. In theory a transactional file system could do that. In practice, there's probably too much other non file systems state involved.

I am using this for my home lab. Extremely useful.

I did for a while on a raspberry pi. It was my home server for some months but i ended up not using it.

Alpine is awesome, but as the article says it is terribly badly documented. On my todo list is contributing some better RPi install docs, and a more sensible A/B boot partition process.

But you still need to reboot for the new kernel to go live right? Or is the kernel switched via kexec or something?

in the last two weeks I've been using Fedora Workstation, I haven't use Linux in 20 years and i have to say this is an incredibly improved experience.

I haven't yet needed to boot back into windows! If it stays like that for the next 6 months I'll cut over to Linux permanently and wipe the windows partition.

I agree.

Just like Wayland, they're pushing it for other reasons (breaking existing stuff, locking you in to the GNOME desktop, etc), and using spurious reasons to promote it.

Depends on what kind of machine you're running.

I purchased a 7900XTX on release day, with the Linux drivers being in a fairly rough state and new fixes being added daily. So, for the next 3-4 months, I was running Fedora Rawhide with the Koji repo added - about as bleeding edge as it gets, short of building locally from source. Rolling back definely came in handy once or twice.

Once things stabilized, I rebased back to non-Rawhide Fedora 37 and stayed there. Then, a few weeks ago, I found out that AMD had been working on ROCM for the 7xxx series, so I've rebased to Rawhide again to play around with AI tools on the 6.6 kernel.

I've also occasionally encountered non-critical bugs, suspected it might have been fixed upstream, and temporarily rebased to Rawhide just to check out if that was the case before reporting it. Pretty nice.

Tangential, but I had my mind blown in about 2009 by a big hypervisor running Windows remote desktop hosts. I believe it was Citrix.

The VMs booted from images. The image and the mutable differencing disks were entirely in RAM (although user profiles were on spinning rust). A desktop host for 25 users would boot to accepting remote logins in about 4 seconds.

Least painful Windows system to patch.

I see that UBlue uses Github actions to rebuild the images regularly to roll in package updates. Who foots the bill? Are the images also served from Github? Does Github charge for egress? What happens when a lot of users want to download the same image?

I'm confused, isn't Fedora Silverblue also image-based? I thought the default installation doesn't use layering, and that layering only comes when you want to install extra RPM packages.

Have you tried the nix package manager on silverblue? https://julianhofer.eu/blog/01-silverblue-nix/

This helps keep the (I agree, awful) configuration portion of nix minimal.

> anything else being wiped on reboot

Have you tried Impermanence?

https://nixos.wiki/wiki/Impermanence

VSCode with the devcontainer and docker is great on Silverblue.

I've been prototyping some developer workflows with friends here: https://universal-blue.org/images/bluefin/developer-experien...

So far the major patterns are vscode with distrobox, vscode with devcontainers, vscode with devpod, jetbrains toolbox thing (which just runs everything out of the home directory, the OS doesn't care).

And then devbox/nix and homebrew in ~ is also an option if you're into that.

This sounds painful for no reason. Especially the debugging part. Why would you want that? Is it really such a strain to install the tools you need for development on your computer?

I can understand wanting an immutable system for a server, as it will likely cut down on maintenance, but for personal use... that just sends shivers down my spine... As someone having to support other (especially not very savvy) programmers when it comes to tool usage and their environment I hate to imagine having to deal with someone who'd want that kind of setup.

> MicroOS Desktop turned into Kalpa (KDE) and Aeon (GNOME), but the latter has all the momentum.

Honestly tying the OS to the desktop environment is the only reason I haven't installed microos. Making a derivative with my preferred DE is on my "maybe someday" list, but that's a long list...

This could be used as an argument in favour of immutability. If you effectively spin up a "new" computer every time you restart a box, then it forcibly surfaces any issues with your deployment or first-run.

It's not like you don't have all the same issues with deploying your long-lived mutable systems -- you're just feeling the pain less frequently, and postponing all the work to debug them until you're in e.g. a disaster recovery scenario.

They work perfectly fine on NixOS at least, can't speak for the others.

The more extreme form of that is to have the OS run from read-only memory. Some embedded systems work that way. Reset, and you're back to the cold start state. QNX can be built to run that way, for systems with no disk.

Ideally, Kali is meant to be used this way, so you can use it to perform forensics on a potentially compromised system without inadvertently changing anything and destroying evidence. You can really trivially do this with any Live CD by putting it on write-once read many media. You typically need at least /var and /tmp to be writeable, but that can be accomplished via tmpfs so they only write to memory and not disk. You don't really need to do anything to enable this at the distro level, other than maybe make the mounts default, but in practice, live CDs tend to mount the root filesystem as SquashFS, which is also read-only at the filesystem level, and then use OverlayFS for partitions that need to be writeable for software to work, not retaining the writeable layer on shutdown.

If you mean how would you do it yourself, you can use the tooling used by real distros. I'm not sure what tools they all provide, but Archiso (https://wiki.archlinux.org/title/Archiso) is probably the simplest to understand and modify because it's purely shell scripts.

Yeah TinyCore is always missing from these intros to immutable Linux… it’s been around a while and has a great design (though the implementation leaves some to be desired, and probably has no corporate backing to explain people not knowing about it…)

It’s solid and simple, unlike these other immutable Linux distros

If it’s a systemd service (which it usually is) you can simply run systemctl cat <service in question> which pretty much always will have WorkingDirectory property set or RuntimeDirectory or similar (RuntimeDirectory you’ll have to prefix with /var/run which you just sort of have to know but that’s not NixOS specific)

I’ve found that’s usually in /var or ~. Can’t think of any instance where it wasn’t one of those.

Unfortunately, that's not possible. However, as NixOS modules usually are Systemd services, the StateDirectory= of the service is a good starting point (systemctl cat <service>).

It has not been possible/practical to use immutable system images until recent advances like ostree/flatpak/systemd-homed.

We didn't "just discover" some secret only known to the embedded community, lots of people have been working toward this exact goal for a long time, because we already knew for a long time that it has a lot of advantages.