"Just patch" is advice for a windows administrator, where updates break everything so you have to sit and baby them and apply them manually.

On Linux, there are ways to enable automatic security updates, including automatic reboots, so you can safely receive the mitigations your distro provides. That way, you don't have to worry about forgetting to patch (until the distro release becomes unmaintained, at least).

Now, dirty frag was a zero day, meaning that it was released and probably in the wild before a mitigation was pushed out to handle it. So you did need to apply an actual configuration patch... unless you had some form of kernel based isolation, which I mention as #2 of my other comment in this thread: https://programming.dev/post/52129409/24414213

Excellent writeup, and I appreciate the transparency. I have some suggestions on how to mitigate something like this from happening in the future.

1. Use a separate DBMS (that is, a separate postgres/mariasql/etc container) for each service. Give each one service unique passwords, which you can define in the docker compose.

This is simpler than trying to control postgres permissions granularity. Even if one application that connects to a database gets owned, it doesn't have access to other postgres databases, preventing data leaks/exfiltration.

2. Use a virtual machine or application container based runtime for your containers.

Kata containers is a container runtime, that is virtual machine.

There is also Gvisor and Syd Box, which are application kernels. Application kernels are reimplimentations of the parts of the Linux kernel needed to run apps, and in this case both Gvisor (Go) and Syd Box (Rust) are in memory safe langauges.

Kata containers are faster, but you will need nested virtualization in order to use them. Application kernels are slower, but you can install them anywhere, including hosts where virtualization is disabled (like a VPS that doesn't let you enable nested virtualization.

Both take a tiny bit more resources intensive due to no longer being able to share the host kernel, but for most part, it is worth it. They don't bring an entire kernel along, just what is needed to run apps.

Both offer similar levels of isolation, and preventing applications running inside them from touching the host kernel directly. They effectively manage to prevent issues like copy fail, dirty frag, and so on, from owning your host.

They are fairly easy to install, docker has some docs here: https://docs.docker.com/engine/daemon/alternative-runtimes/. But if you are using podman or kubernetes, you can also install them there.

3. Enable automatic security updates (and reboots) on stable distros.

A large part of the draw of stable Linux like Debian or Red Hat, is that they only do security updates. They don't do feature updates, or even bug fixes (except for critical ones). In doing so, there is essentially a guarantee of reliability, where it is impossible for updates to break anything.

This makes it possible to enable automatic security updates, and you can even configure it to automatically reboot in order to load a new kernel that includes mitigations against issues like dirty frag. Make sure your docker containers are configured to automatically restart and everything will be smooth.

"Just patch" is a good but it is never enough, and I am frustrated hearing it so frequently. The way I view it is, any time I have to patch, what I really need to do is to improve my security architecture so I never have to "patch" this specific issues again. Patches are the exact kind of security toil that I complain about in this comment.

Somewhat reminds me of:

https://holdtherobot.com/blog/linux-on-android-with-ar-glasses/

Woops, a duplicate?

Do you have an alternative solution to the technical challenges I have mentioned above?

This is one of the most popular methods to handle apps that are using a storage backend directly over the internet for increased bandwidth, and separation of deployment and state.

If I'm hosting something like Nextcloud at a massive scale, it simply isn't feasible to use internal networks because they don't have enough bandwidth, and overlay/vpn network solutions have too much overhead. A common solution is to just run the services and directly connect to them over the internet. So I point my 10,000 node Nextcloud instance at S3, either my own cluster or somebody else's and S3 handles encryption while remaining reasonably performant. And scalable.

Garage uses their software internally for something similar: https://garagehq.deuxfleurs.fr/documentation/design/goals/. On this page, they describe using it as their matrix image/file storage cache. Same thing there, they probably have a large distributed matrix cluster that needs storage that scales with it, while also being secure over different networks.

the most simple answer is that you selected s3 as your sole storage backend for whatever services/apps you have deployed. But now, you want to integrate something that does not support s3 into your stack. So you'll look at stuff like this.

Now, why would you self host S3 in the first place:

The big reason is security. NFS is a security nightmare that believes connections when they claim to be root. It has no encryption and basically no authentication by default [1]

NBD is a little better, but both NBD and NFS have the problem in that services run in the kernel, exposing frightening attack surface to privileged parts of linux systems. Also, NBD is block storage and a lot less supported as a storage backend for various services.

S3 on the other hand, is actually designed to be exposed over the internet. http based, you can use https for encryption, and http basic auth or other methods for authentication trivially. So if you ever end up running something distributed over the internet, NFS is basically not an option, and S3 is a much better solution.

[1] Was posted to lemmy: https://programming.dev/post/34520407

Late to the party but:

https://wiki.archlinux.org/title/Visual_Studio_Code#Unable_to_debug_C#

Programs/orgs like Conda are like the #1 reason projects like Guix exist.

Conda's default repos are only technically free for personal use, and you have to pay an exorbitant amount if you want to use them in a company. But what happens is devs install Conda anyways, not realizing this, the software phones home, and all of a sudden you have a bunch of lawyers on your case, demanding 10 gorbillion dollars.

And because programs like Conda, or Oracle Java, or so on are technically not malware (even though they literally act like ransomware in some ways), they aren't, and will not ever be caught by antivirus software.

So the solution people come up to not have to deal with those, with, is to restrict all installation of software entirely, via things like AppLocker on Windows. This makes it so that only approved software can be installed. Software can be manually vetted, confirmed to actually be free for the business, or paid for, before being explicitly allowed.

But the problem with this, is that users like being able to autonomously install the tools they need in order to solve problems. So now they just get frustrated that they can't do that at all.

Guix, and other projects which only ship open source software, present a middle ground. They distribute a large repo of software, that is essentially confirmed safe for a business to use, and for their users to install autonomously. If I gave someone Guix, I could feel confident that they could install various tools they needed without risking totally-not-ransomware from getting onto the systems.

Anyway. There is nonguix and other additional guix package channels if you want, say CUDA so it's an option. I'm just trying to explain why some people insists on this model, and why someone would see that as a benefit.