Replacing Proxmox
I have been using Proxmox for about 5 years. In Feburary, I decided to stop using Proxmox. If you're expecting some compelling reason, you're not going to find one. I use my homelab for many different purposes. One of those is enjoyment, frankly I was bored with Proxmox and I wanted a change. That being said, are some of the things that I didn't like.
It is based on Debian. I don't have an issue with Debian or any traditional Linux distribution, I just think that using one is a poor choice for a hypervisor. Every manual change has the possibility of living forever. That config file you edited because a particular version of a package had a bug and you needed to work around it, did you remember to revert it when you updated the packages? Over time, the system drifts off of the image that Proxmox provides. Of course, there are ways to control this but, this is my homelab. I don't want to apply that level of rigor. I know that one of my hypervisors had a ceph service continuously restarting since I last used Ceph (a subject for another time).
As a result of the aforementioned accumulation of changes, I always found upgrades to be perilous. I had very few issues during upgrades, I could probably count the number of times that an upgrade caused an issue on one hand. It is more that each upgrade was incredibly stressful. Rolling back is not a simple task and spending hours troubleshooting an issue after an upgrade is just not something I'm particularly interested in doing.
This is highly subjective but, the UI always bothered me. It certainly works and a great deal of functionality is exposed but, I didn't enjoy it. The placement of various settings always felt a little strange to me and some operations would require me to go to many different pages to make the changes. I think the main problem, for me, is that the way the settings are grouped is not how I think about them. I did learn where everything was at but, it always felt suboptimal. Finally, I just feel like the interface is dated. Sure, a hypervisor doesn't need to be shiny but, this is also my homelab. I want to enjoy using it.
There has been great work done on a proxmox provider for Terraform but it is hampered by Proxmox's perplexing decisions around cloud-init. It makes some sense that the Proxmox console only exposes some commonly used cloud-init parameters. The API allows you to use a custom file and set whatever options you want and the terraform provider allows you to utilize this. The issue is that it needs to be a file. There isn't an api to upload the file. You have to place the file in one of Proxmox's stores and then set a reference to it in the vm configuration.
Another one of these Terraform limitations is related to cloning. You can clone a vm to any node that has access to the same storage. So, if you have a vm on shared storage, you can clone it to any member of your cluster. If it is using local storage, you can only clone it to the same node. Once it is cloned, you can migrate it to another node but that is two separate operations. This makes utilizing Terraform a bit difficult if you're trying to spin up multiple vms over multiple nodes. You either have to make a duplicate template on each member of your cluster or utilize shared storage. I preferred to utilize local storage for these things as the primary reason for utilizing Terraform was experimentation with different technologies. Being able to rapidly remake the things being managed was the draw. Duplicating the template over all of the members of the cluster was an annoyance. Proxmox could alleviate this by having a compound api action that first does the clone and then migrates to the targeted node.
It is incredibly likely that this isn't exhaustive. I probably should have written closer to the time that I started getting off of it but, the processes of getting off of it wasn't the smoothest and I wanted to have my replacement in place before writing this. If anything, the struggles in setting up my replacement and the subjectivity of my complaints is a ringing endorsement for Proxmox. I would not hesitate to recommend Proxmox to someone looking to start their homelab. It worked very well for me over the years.
So what did I end up with? The first thing was xcp-ng. Before I moved everything onto it, I thought it was great. Then, I ran into some problems and figured out that it does not work in the way that I want it to but, that is, again, a subject for another time. What I did settle on was k3s on top of nixos utilizing KubeVirt. This gets me no UI (at least built-in), a clunky Terraform experience, no built-in way to move local storage between nodes and a fairly convoluted networking setup. I fixed none of the issues and yet I like the change. A wholly irrational choice and sometimes, the homelab is about making irrational choices.