Error

One of the really nice side-effects of it running rootless is that you get all the benefits of it running as an actual Unix user.

For instance, you can set up wireguard with IP route to send all traffic from a given UID through the VPN.

Using that, I set up one user as the single user for running all the stuff I want to have VPN'd for outgoing connections, like *arr services, with absolutely no extra work. I don't need to configure a specific container, I don't need to change a docker-compose etc.

In rootful docker, I had to use a specific IP subnet to achieve the same, which was way more clunky.

Could you explain or show how to do that?

Yeah sure.

I'm going to assume you're starting from the point of having a second linux user also set up to use rootless podman. That's just following the same steps for setting up rootless podman as any other user, so there shouldn't be too many problems there.

If you have wireguard set up and running already - i.e. with Mullvad VPN or your own VPN to a VPS - you should be able to run ip link to see a wireguard network interface. Mine is called wg. I don't use wg-quick, which means I don't have all my traffic routing through it by default. Instead, I use a systemd unit to bring up the WG interface and set up routing.

I'll also assume the UID you want to forward is 1001, because that's what I'm using. I'll also use enp3s0 as the default network link, because that's what mine is, but if yours is eth0, you should use that. Finally, I'll assume that 192.168.0.0 is your standard network subnet - it's useful to avoid routing local traffic through wireguard.

#YOUR_STATIC_EXTERNAL_IP# should be whatever you get by calling curl ifconfig.me if you have a static IP - again, useful to avoid routing local traffic through wireguard. If you don't have a static IP you can drop this line.

[Unit]
Description=Create wireguard interface
After=network-online.target

[Service]
RemainAfterExit=yes
ExecStart=/usr/bin/bash -c " \
        /usr/sbin/ip link add dev wg type wireguard || true; \
        /usr/bin/wg setconf wg /etc/wireguard/wg.conf || true; \
        /usr/bin/resolvectl dns wg #PREFERRED_DNS#; \
        /usr/sbin/ip -4 address add #WG_IPV4_ADDRESS#/32 dev wg || true; \
        /usr/sbin/ip -6 address add #WG_IPV6_ADDRESS#/128 dev wg || true; \
        /usr/sbin/ip link set mtu 1420 up dev wg || true; \
        /usr/sbin/ip rule add uidrange 1001-1001 table 200 || true; \
        /usr/sbin/ip route add #VPN_ENDPOINT# via #ROUTER_IP# dev enp3s0 table 200 || true; \
        /usr/sbin/ip route add 192.168.0.0/24 via 192.168.0.1 dev enp3s0 table 200 || true; \
        /usr/sbin/ip route add #YOUR_STATIC_EXTERNAL_IP#/32 via #ROUTER_IP# dev enp3s0 table 200 || true; \
        /usr/sbin/ip route add default via #WG_IPV4_ADDRESS# dev wg table 200 || true; \
"

ExecStop=/usr/bin/bash -c " \
        /usr/sbin/ip rule del uidrange 1001-1001 table 200 || true; \
        /usr/sbin/ip route flush table 200 || true; \
        /usr/bin/wg set wg peer '#PEER_PUBLIC_KEY#' remove || true; \
        /usr/sbin/ip link del dev wg || true; \
"

[Install]
WantedBy=multi-user.target

There's a bit to go through here, so I'll take you through why it works. Most of it is just setting up WG to receive/send traffic. The bits that are relevant are:

        /usr/sbin/ip rule add uidrange 1001-1001 table 200 || true; \
        /usr/sbin/ip route add #VPN_ENDPOINT# via #ROUTER_IP# dev enp3s0 table 200 || true; \
        /usr/sbin/ip route add 192.168.0.0/24 via 192.168.0.1 dev enp3s0 table 200 || true; \
        /usr/sbin/ip route add #YOUR_STATIC_EXTERNAL_IP#/32 via #ROUTER_IP# dev enp3s0 table 200 || true; \
        /usr/sbin/ip route add default via #WG_IPV4_ADDRESS# dev wg table 200 || true; \

ip rule add uidrange 1001-1001 table 200 adds a new rule where requests from UID 1001 go through table 200. A table is a subset of ip routing rules that are only relevant to certain traffic.

ip route add #VPN_ENDPOINT# ... ensures that traffic already going through the VPN - i.e. wireguard traffic - does. This is relevant for handshakes.

ip route add 192.168.0.0/24 via 192.168.0.1 ... is just excluding local traffic, as is ip route add #YOUR_STATIC_EXTERNAL_IP

Finally, we add ip route add default via #WG_IPV4_ADDRESS# ... which routes all traffic that didn't match any of the above rules (local traffic, wireguard) to go to the wireguard interface. From there, WG handles all the rest, and passes returning traffic back.

There's going to be some individual tweaking here, but the long and short of it is, UID 1001 will have all their external traffic routed through WG. Any internal traffic between docker containers in a docker-compose should already be handled by podman pods and never reach the routing rules. Any traffic aimed at other services in the network - i.e. sonarr calling sabnzbd or transmission - will happen with a relevant local IP of the machine it's hosted on, and so will also be skipped. Localhost is already handled by existing ip route rules, so you shouldn't have to worry about that either.

Hopefully that helps - sorry if it's a bit confusing. I learned to set up my own IP routing to avoid wg-quick so that I could have greater control over the traffic flow, so this is quite a lot of my learning that I'm attempting to distill into one place.

Saving this for later, thank you very much for the detailed writeup. I might look into this for my main machine to partition the vpn tasks from the non-vpn tasks

You're welcome. It has some really nice side-effects - i.e. if I want to quickly grab a file without it being from my normal IP, I can just SSH to the right user on my server and it just works - no configuration, no needing to interrupt other traffic.

Thank you so much, i was just looking for a way to do this the other day!

Thank you very much!

I see it as a feature that Podman containers are run via systemd. This makes their management consistent with the other systemd-managed services. Also, Docker does it own things with logs, while with systemd, the logs are managed in a consistent way as well.

Maybe you missed podman generate systemd? Podman will generate the systemd unit files for you.

For me, the two big benefits of podman are being able to run containers via systemd and improved security by being able to run them rootless.

Ok. I was already very familiar with systemd when I started using podman and didn’t have any trouble at that step. In my case, I created the systems unit files by hand.

The easiest is actually using Quadlet (integrated in Podman 4.x or later): https://www.redhat.com/sysadmin/quadlet-podmanand write simple .container files.

Works very well and does auto-start, service dependencies and even container auto-updates.

Also running containers in Pods is a really nice way to handle virtual networking and allows you to manage all the containers in one go similar to docker-compose.

I actually find this a huge problem. Not all distros are built around LSB, XDG, or FreeDesktop.org nor should they be since not everyone is running Linux as a workstation/PC replacement. While yes for the most part podman can be ran on the likes of Gentoo, Alpine, Arch and etc. It becomes a pain in the arse to decouple the tooling for podman away from freedesktop.org standards. Even more a pain in the arse for clustering options (e.g. podman-remote expects freedesktop.org norms, kubernetes expects docker containerd or freedesktop.org with podman, and nomad stack is just bulky vaporware).

The really sad part of this is that podman isn't adding much of anything new that LXC or linux namespaces outside of not needing a daemon, allowing rootless execution (again because it doesn't need a daemon) and giving ACLs around which OCI repos could be pulled from unlike docker's wildcard by default. It shouldn't be hard to do linux containerization without being tied to anything other than the linux kernel.

I guess this: "you run a “root” container in your completely unprivileged Unix user and everything just works" sounds like chroot. Also managing your container starts with systemd sounds pretty good to me because this is what systemd is designed for, dependencies between services, etc.

Fyi, you can create a pod, create containers in that pod and generate the systemd file for that. Now you have one service to start/stop everything.

There's also podman-compose, which I've been using. It's not quite feature complete, but it's pretty close.

I work somewhere that doesn't have licensing with Docker Inc. And I work on a Mac. With Docker desktop out of the picture, I got some experience with the alternatives. I know this post is about the native implementation and not the VM one, but I just wanted to add my 2 cents:

Alternatives run by me: Podman, Rancher Desktop, Finch

Results:

  • Podman uses a lot more energy on idle than Finch and Rancher. On AVG 4 more Wats on an M1. (Normal idle is about 5W, so 9 almost doubles it cutting greatly in my battery life)
  • Podman and Finch are not compatible with some tools that expect a full docker sock. In my case the AWS CDK and SAM CLI have issues. (Which is fun as Finch is also made by AWS)
  • Finch does not offer a sock at all
  • Finch requires you to recreate the full VM when updated.
  • If you really want to have a drop-in replacement for Docker Desktop, use Rancher Desktop. Rancher lacks in UI and the extension feature. But I never had issues with the sock, as I can run it with containerd.
  • Finch has no UI
  • Podman's VM has clock drift if you put your machine in sleep. Only solution I found is to reboot the podman VM.
  • Podman allows you to log in the VM with a command. I haven't found a way on the others.

The clock drift issue has been resolved recently: https://github.com/containers/podman/issues/11541

That is awesome. I prefer podman, despite what my list might suggest.

Wow, 4 watts? That's a lot. Any insight on what is taking up all this extra energy? I thought that podman would be thinner than docker honestly.

I did some shallow digging, and my guess is the virtual machine that is started for each.

I see that the podman vm is a whole ass fedora image, at least back in 2021 when this article was written.

Rancher seems to use alpine if I understand the configuration correctly

Finch also uses fedora... I think. Their config is seemingly simple to the point it looks deceptive.

Oh, are you using Podman on windows? Yeah, it needs a virtual machine because it has to load the linux kernel. I would definitely believe that the windows version (or mac, I guess) of podman is way heavier than the alternatives on those platforms, but on linux it just ends up using the host kernel.

If you are doing this on linux, and still need to load a vm to use podman, that would be interesting. I haven't run across that, but I haven't been able to use podman too much.

You forgot the third option: A Mac ;)

How are mac's with VMs? Windows has a bunch of Hypervisor stuff that makes them work pretty well, I don't use macs, so I wouldn't know how well they run VMs.

I won't say it is the same. Mac doesn't have subsystems, for example. But it allows for hypervisors to run. For a while apple had their own on intel CPUs, bootcamp. But they stopped updating it for the Mx line.

Nevertheless, Mac might not be Linux, it is still UNIX. So it can run qemu and all the tools built for qemu. Wine is also a tool to run windows specifically.

Didn't know about Podman yet, I'll probably give it a spin on some personal projects to get a bit more used to tools other than docker. Thanks for the great intro!

Sure thing, ranting is therapeutic and I wouldn't miss the chance~

I've switched over my own server last week, using ansible to generate the systemd files, and it worked great. It's just a dozen containers or so.

The only problems I had were with container interdependencies (network-mode=container:x). That didn't work so well with systemd, restarting and updating, but when I used a pod instead these problems all went away.

So I can't say I regret my experience so far. Now I'll be starting to use it at work too, where the user-namespace problem rears its head, but only because we have this very specific, very dumb big lamp dev container that houses apache, sql, redis, and more under one supervisord. That's why we have more than one user in it and frankly that's our own damn fault! When you make proper containers they shouldn't have more than one user in it and then userns=keep-id should work just fine.

So far, I fully recommend podman.

I understand very well wanting to stay with the declarative nature of docker-compose. Someone should really build a better podman-compose. (or sooner or later I'll do it myself >_<)

The other big annoying thing about Podman is that because there’s no Big Bad Daemon managing everything, there are certain things you give up. Like containers actually starting on boot. [...] until you realize that means Podman wants you to manage your containers entirely with systemd. So… running each container with a systemd service, using those services to stop/start/manage your containers, etc.

Surprisingly, they have a solution for that that doesn't involve using systemd for everything. They put an --all option to podman start, and a systemd service to run it at boot with the correct --filter (yeah. because unix philosophy). Debian seems to enable it by default AFAICT.

No idea how well it works rootless though.

Edit: Oh and for rootless networking, Podman 4.4.0 seems to ship pasta which seems to be the solution to slirp4netns's existence. Unfortunately I have no idea if it works at all because I run Debian stable which is still on 4.3.1

I've tried to switch in the past, but tripped over the differences in Podman vs Docker networking. IIRC Docker is better for creating an isolated network.

I have noticed that Docker doesn't do the best job at graceful shutdowns (say for automatic installation of updates). I suspect Podman with systemd integration could do much much beter.

At least podman does not circumvent my firewall (ufw) like docker did. Had to use a workaround to get it to work with docker.

Yep, turns out it doesn't insert its own iptables rules like docker does, so the special rules from ufw-docker weren't necessary anymore.

...and of course I learn this after switching to firewalld.

At least firewalld feels relatively painless compared to rest of the redhat-container-verse.

Lurker. Never self hosted. Nothing useful to add here. Fedora Silverblue has a lot of integration with podman. It is the basis of toolbx, which has its issues, namely that the distro it spins up can't be upgraded in each toolbx container. The tools are there to mess with containers stuff.

IMO it is the integration with toolbx that is interesting to me, but I struggle with these things.

Honestly, I had to use podman at work due to... issues.

Its close enough to docker, that most docker commands will work just fine. You can even alias docker as podman, and things will for the most part, just work.

HOWEVER, there are some big changes and difference. First- podmon creates a systemctl for your containers, for starting them. This- is different, and if you forget to tell it to create the service- then your containers won't start.

My personal opinion after using it for a few years? I strongly prefer docker. It gives me very few issues. I have spent too much time troubleshooting odd things podman does.

Don't forget the multiple flavors of kubernetes too.

Kubernetes just runs docker containers, (and lots more).

Imagine a solution to manage containers on multiple hosts, with tons of redundancy. With network and storage automation built in.

That's Kubernetes in a nutshell.

But, has a steep learning curve

had a knowledge sharing meeting at work recently on container security. the guy was using podman like a docker cli, but it said "this is only an emulation of docker" - is there any downsides to running podman like this? im very familiar with docker on the command line

I think calling it an emulation downplays podman. Docker and podman are both container runtimes. Docker came first and is known synonymously with containers, whereas podman is newer and attempts to fix docker's problems.

One outcome of this is podman chose to match docker's cli very closely so nobody needs to learn a new cli. You can even put podman on the docker socket so "docker [command]" runs with podman.

Its not too bad, you can define the UID that your podmab process runs on since it can run as its own process and not be reliant on a daemon. It makes you do a little more admin work, but honestly you have to start doing that anyway in a lot of environments.

I have had good experimental experiences with podman, but my biggest barrier to adopting it is how old the packaged version with debian is. I'm hoping to see if I can do some damage with the release of bookworm.

Docker and k8s aren't the same thing though? Docker is a container runtime while k8s is an orchestration platform.

I've been running Docker with Portainer for last few years. Been working great. Tried making the switch over to podman but couldn't get it to work with Portainer (which a lot of people say it can) and had some issues with NFS shares. One day I'll have to give it a try again.

give cockpit a shot if you only need a gui for general management. works great together with podman

One thing I personally love with buildah/podman is that you can run it INSIDE docker very easily. Perfect for build scripting or creating OCIs inside CD/CI systems managed by docker. Remounting the socket within the container always felt like a hack to me, and for my own setup I've never done it out of principal.

Awesome summary of how podman works.

I still haven't figured out some issues with rootless podman where I pass the PUID and PGID of "myuser" 1000 as environment variable following the linuxserver.io examples... but then get files and folders owned by 100999:100999, if I chown files to "myuser" the service gets permission denied, I give up and chown everything to 100999 as workaround it works but is a bit annoying... Maybe someone here knows what's going on?

This is a consequence of user namespaces, which tripped me up until I read this article from Red Hat about running rootless containers as a non-root user. At that point I got that (the default options) map UID 0 in the container to my UID (i.e. 1000), but the other mappings were confusing.

The short version of the useful part (for me) of that article was podman unshare (man podman-unshare), which launches a shell in a user namespace, like when you start a container. You can run the following command to see how the UIDs are mapped inside of the namespace:

$ podman unshare cat /proc/self/uid_map
         0       1000          1
         1     100000      65536

This is read (for this purpose, see man user_namespaces for a more detailed explanation of this) as "inside this namespace, the UIDs in column 1 map to the UID in column 2 on the caller process, for (column 3) IDs". There is also gid_map which works the same way, but for groups.

The snippet above is from my machine, so in a podman container, UID 0 maps to UID 1000 on the "host", which is me, and this is "good" for only 1 user. Then, starting with UID 1, the container maps to UID 100000 in the container, and is good for 65536 UIDs. This is why when you set the PUID and GUID environment variables, on your filesystem you see the files are owned by 100999:100999 - you can use the mapping to figure the math out: 100000+1000-1=100999.

Since podman unshare puts you in a shell that has the same (? terminology might not be totally right here) user namesapce as your containers, you can use it for lots of stuff -- like in your comment you mentioned using chown to change the permissions to 100999:100999. Instead, you could have used podman unshare chown 1000:1000 which have correctly set the permissions for your volume mount, and on your filesystem outside the container, the permissions would be 100999:100999.

I still don't know what "unshare" means.

Thank you! I will look into this.

as far as i remember, processes run inside container as root (0:0) end up under your own UID.

everything else gets mapped to those weird UIDs.

Not a podman user, so please take this with a whole bag of salt. That seems to me a namespace issue. Does podman by default uses user namespaces? Because if that's the case, it's normal that UIDs are remapped inside the container namespace, and 1000 inside it corresponds to something else (maybe 100999?) outside.

One way to check could be cat /proc/PROC_INSIDE_CONTAINER/status | grep uid or cat /proc/PROC_INSIDE_CONTAINER/uid_map.

This https://stackoverflow.com/questions/70770437/mapping-of-user-idsseems also to be somewhat relevant to your scenario?

I've used podman on an RHEL server at work because it works nicely with selinux. I had a hell of a time with rootless containers and network throughput when using an nginx reverse proxy. Made the site painfully slow. Turned out it was due to the slirp4netns rootless networking and MTU size. Just decided to say screw the rootless thing and went rootfull. Next time honestly would just use docker since it's more common

Not necessarily. For networking, I wrote a bash script with just a few lines that creates and assigns a private networking namespace to a pod and sets up the default routes. That script is run by a systemd user instance and has the suid flag set. One could argue that it's not rootless because of that but that's just the moment when it's starting. No performance impact and very robust. A lot better than the docker network bridges imho.

Even the root user has to use syscalls at the userspace/kernel boundary so there's really no difference here. The slow part would be the userspace networking implementation (slirp4netns) which can be avoided. Btw, docker also supports rootless containers now. Personally I haven't tried them yet but maybe that's also an option if you prefer docker over podman.

Do you think podman could replace Docker Desktop on my Dev machine? Its become so bloated...

For instance, there’s no way to apply a change in-place without totally stopping and starting a container with two separate commands. What is this, 2003?

Wait, doesn't the same apply to Docker? There's no "change settings" feature of already running containers, you have to stop container (maybe commit image before) and start with new settings.

This makes me anxious... How do you cope with all these different technologies... I mean everything is evolving so fast and everyone wants to have his OWN way of doing things... This is messed up ! Right now IT seems a big maze of technologies and nobody seems to be in sync with each other... specially in devOP and Networking...

I don't know about Podman, but it's baffling how much you need to know and understand in IT... And If every 3 years you have to relearn everything, it's a never ending chase of dying and abandoned technologies and a wast of time :/

Just my 2cent, nothing special !

@deepdive @witten I think the more you dig the more you find you could learn, probably like every other topic with enough people on it. If you want to keep it simple you mostly still have the chance to just use a little linux machine and put everything there the "old" way. For example: I spend some 3-4 months building a kubernetes stack for my homelab, getting everything to run perfectly, then scrapped everything to rewrite it again with a bit of ansible and a single machine because it justworks

I think the more you dig the more you find you could learn

True, but it's really frustrating to spend time to learn something that's maybe going to be useless ? Just look at networking in linux distros between networkd, NetworkManager, netplan, nmtui, nmcli, networkctl, ifupdown... all working in different locations and all having their own way of doing things... This is is fucked up :/

Imagine learning docker's all subtilities and next year it's deprecated in favor of another technology with his own flavors and commands... :/

@deepdive Yes this can get frustrating if you let it get to you. I‘m 25 years into this and all i learned is how to look stuff up and forgot the rest. I don‘t learn technologies, i try to reduce them to some basic knowledge so i can handle them well enough. Things change all the time and i‘m too lazy to keep track of all that stuff, docker is dead. Its especially true in my actual playground at work where we are using kubernetes. Some of the most complex and fast paced stuff i ever worked with.

This really bumps me down ! I begin to feel somehow confident with docker/docker-compose and reading it's an already "dead" technology in corporate situation... Uhhg, this makes me rethink if IT is the right direction for me :/

The proprietary world is no better, to be honest. Look at a single company like Google, who can’t even manage to keep a single messaging app operational for more than a couple years before up and replacing it with something shiny and new.

Yeah, but GAFAM's motivation are different. That's more or like marketing stuff and because changing keep the user base busy and they have to keep up with new technologies, also its more appealing to have somehow a fresh redesign ... But I get your point !

Doesn't make it you sick to always relearn something new? I mean, that's some useless braincells going from years of mastering something, to nothing !

I like the saying: "IT is 1inch deep but miles large" it's like impossible to have to full picture of IT.

Sorry if my not clear or somehow mixing things together, but right now i'm rethinking my IT pathway :/

And this is why the trick is learning and focusing the technologies that stick at a "lower level" of the stack, and that have been battle-tested by years or even decades so it's understood that they won't just "go away". Like eg.: learning C or Fortran instead of learning ${niche_language_of_year_20xx}. For the docker bracket for example the near equivalent would be hmmm I'd say (s)chroot.

Then again from here to around 5 years docker will the the schroot of its tech bracket.