Some big changes in 2016 and 2017
I can't believe its been over two years since my last post! Where has the time gone?
Well for one thing, I have gotten engaged near the end of 2016, so lots of 2017 have been taken up with planning the wedding!
But that's not to say that I haven't been working on lots of Linux and open source related tech in the last couple of years too!
Farewell OpenVZ. Hello LXC
Probably the open source tech that was newest to me and has had the most impact this year is my switch from OpenVZ to LXC Linux containers.
I have been a big fan of OpenVZ Linux containers for the past 10 years, right back when using it with CentOS 5, and containers were called "virtual environments" or VEs. I found both OpenVZ and the CentOS distribution to be very stable and easy to use, and it was fantastic for separating physical servers into several logical virtual roles.
One of the things I liked most about OpenVZ containers was that it supported SAN-free migration of containers across different machines, using the rsync command under the hood. It allowed maintenance to be performed on the physical servers by easily migrating the services running on it to another machine.
The other great feature was called "simfs", and it allowed you to store all of the container file systems in one big file system directory on the physical machine, and have soft quotas and file system separation implemented in OpenVZ rather than having to mess around with disk images (like my colleagues who were using 'proper' virtualisation had to).
I continue to use OpenVZ to this day at work, using CentOS 6. However CentOS 6 is starting to show its age (like really badly), and I was keen to switch to CentOS 7 for several years. However OpenVZ depends on a custom Linux kernel to work, and the OpenVZ team did not support CentOS 7 properly until mid 2016.
Sadly, this year I have spent some time trying out OpenVZ 7 running on CentOS 7, and alas my disappointment with it was what got me looking for alternative Linux container technologies and led me to LXC.
OpenVZ underwent an ownership change in 2016 too, and I believe this is what has caused some of the issues with OpenVZ 7.
Here are a list of issues with OpenVZ 7 that I have found:
Doesn't use vanilla CentOS 7
It requires its own variant Linux distribution called "vzlinux", which is based on CentOS 7, but you cannot install OpenVZ 7 directly on a pre-installed CentOS 7 server. Instead the official way to install it is via an ISO image.
There is an unofficial script that will convert a CentOS 7 to a vzlinux server and then install OpenVZ 7, but that seems to be risky to depend on that for a solution.
Still requires a custom kernel
OK this isn't a major issue, as its been this way for the last 10 years, but it still unfortunate that you can't use OpenVZ containers without a custom kernel.
Custom kernel updates are infrequent
This is a big problem. The open source version of OpenVZ 7 kernel does not get regular kernel security and bug updates, instead they plan to release infrequent updates, and only make regular updates available to their commercial ReadyKernel subscribers or you have to use the automatically generated nightly versions. At the time of writing, the latest open source kernel was vzkernel-3.10.0-514.26.1.vz7.33.22 from July 2017 whereas their latest ReadyKernel patch was from today. There is also the nightly build version in their "factory" repo, that was at version vzkernel-3.10.0-693.1.1.vz7.37.19.
Now clearly, they need to make money as a company, but as an open source project, to withhold security updates to only paying customers is in my view irresponsible. Here is a more thorough post on their kernel release process and here too.
There does appear to be some potential for movement on this issue though, as more recently there was a post on the mailing list suggesting that paid for ReadyKernel updates could be available for OpenVZ without having to buy a Virtuozzo license.
SIMFS is removed from OpenVZ 7
One of the best features (in my opinion) from OpenVz 6 is removed from OpenVZ 7, and that is SIMFS, which gave the ability to do per-container 'soft' quotas using a single directory of container file systems on the host. This was great as it made backing up the containers easy.
In its place we now have to use disk based images called "ploop", and the back up process is not clear and straight forward.
These issues started to lead me toward alternative solutions. Initially I explored using OpenVZ with LVM thin volumes, however the tooling did not support it well, and it felt rather hacky.
Enter LXC
So I started looking for another "system" container solution and remembered I had heard about LXC a couple of years earlier, but had never seriously looked into it. It appears that I was not alone in starting to look for alternatives and finding LXC.
It would be worth pointing out at this stage, that I had discounted Docker for the time being because I wanted something that replaced OpenVZ without needing to re-structure all my current services into "process" or "application" containers, which Docker is more suited for. Whereas OpenVZ and LXC containers provide more of a full system like virtualisation environment.
I will continue my LXC journey in my next post, but suffice to say ultimately it lead me away from both OpenVZ and CentOS, and I have now switched over to Debian 9 and LXC. Which is a massive change for someone who has been exclusively using CentOS for 10 years!
