Nix process management framework
In the first few months of the year, I have dedicated quite a bit of time on the development of the experimental Nix process framework that I started in 2019.
As explained in my blog reflection over 2020, I have reached all of my original objectives. However, while developing these features and exploring their underlying concepts, I discovered that there were still a number side issues that I needed to address to make the framework usable.
s6-rc backend
The first thing I did was developing a s6-rc backend. Last year, I did not know anything about s6 or s6-rc , but it was provided to me as a feature suggestion by people from the Nix community. Aside from the fact that it is a nice experiment to evaluate how portable the framework is, I also learned a great deal about s6, its related tools, and its ancestor: daemontools from which many of s6's ideas are inspired.
Mutable multi-process containers
I also worked on a mutable multi-process container deployment approach. Last year, I have developed a Docker backend for the Nix process management framework making it possible to expose each running process instance as a container. Furthermore, I also made it possible to conveniently construct multi-process containers in which any capable process manager that the Nix process management framework supports can be used as a root process.
Unfortunately, multi-process containers have a big drawback: they are immutable, and when any of the processes need to be changed or upgraded, the container as a whole needs to be discarded and redeployed from a new image, causing all processes to be terminated.
To cope with this limitation, I have developed an extension that makes it possible to deploy mutable multi-process containers, in which most of the software in containers can be upgraded by the Nix package manager.
As long as the root process is not affected, a container does not have to be discarded when a process is upgraded. This extension also makes it possible to run Hydra: the Nix-based continuous integration service in a Docker container.
Using the Nix process management framework as an infrastructure deployment solution
I was also able to use the Nix process management framework to solve the bootstrap problem for Disnix on non-NixOS systems -- in order to use Disnix, every target machine needs to run the Disnix service and a number of container provider services.
For a NixOS machine this process is automated, but on non-NixOS systems a manual installation is still required, which is quite cumbersome. The Nix process management framework can automatically deploy Disnix and all required container provider services on any system capable of running the Nix package manager and the Nix process management framework.
Test framework
Finally, I have developed a test framework for the Nix process management framework. As I have already explained, the framework makes it possible to use multiple process managers, multiple operating systems, multiple instances of all kinds of services, and run services as an unprivileged user, if desired.
Although the framework facilitates all these features, it cannot guarantee that a service will work with under all possible conditions. The framework makes it possible to conveniently reproduce all these conditions so that a service can be validated.
With the completion of the test framework, I consider the Nix process management framework to be quite practical. I have managed to automate the deployments of all services that I frequently use (e.g. web servers, databases, application services etc.) and they seem to work quite well. Even commonly used Nix projects are packaged, such as the Nix daemon for multi-user installations and Hydra: the Nix-based continuous integration server.
Future work
There are still some open framework issues that I intend to address at some point in the future. We still cannot test any services on non-Linux systems such as FreeBSD, which requires a more generalized test driver.
I also still need to start writing an RFC that identifies the concepts of the framework so that these ideas can be integrated into Nixpkgs. The Nix process management framework is basically a prototype to explore ideas, and it has always been my intention to push the good parts upstream.
Home computing
After the completion of the test framework, I have shifted my priorities and worked on improving my home computing experience.
For many years, I have been using a custom script that uses rsync to exchange files, but implements a Git-like workflow, to make backups of my personal files and exchange files between multiple machines, such as my desktop machine and laptop. I have decided to polish the script, release it, and write a blog post that explains how it came about.
Last summer, I visited the Home Computer Museum, that gave me inspiration to check if both of my vintage computers: the Commodore 128 and Commodore Amiga 500 still work. I have not touched the Amiga since 2011 (the year that I wrote a blog post about it) and it was lying dormant in a box on the attic every since.
Unfortunately, a few peripherals were broken or in a bad condition (such as the hard drive). I have decided to bring it to the museum for repairs and order replacement peripherals. It turns out that it was quite a challenge to have it all figured out, in particular the installation process of the operating system.
Because not all information that I needed is available on the Internet, I have decided to write a blog post about my experiences.
I am still in the process of figuring out all the details for my Commodore 128 and I hope I can publish about it soon.
Revising the NPM package deployment infrastructure for Nix
Aside from doing a nice "home computing detour", I have also been shifting my priorities to a new major development area: improving the NPM deployment infrastructure for Nix. Although node2nix is doing its job pretty well in most cases, its design is heavily dated, and giving me more and more problems in correctly supporting the new features of NPM.
As a first step, I have revised what I consider the second most complicated part of node2nix: the process that populates the node_modules/ folder and makes all necessary modifications so that npm install will not attempt to download source artifacts from their original locations.
This is an important requirement -- the fact that NPM and Nix do not play well together is because dependency management is conflicting -- Nix's purity principles are more strict. As a result, NPM's dependency management needs to be bypassed.
The result is a companion tool that I call: placebo-npm that will replace most of the complicated shell code in the node-env.nix module.
I am still working on revising many other parts of node2nix. This should eventually lead to a new and more modular design, that will support NPM's newer features and should be much easier to maintain.
Next year, I hope to report more about my progress.
Some thoughts
As with 2020, I consider 2021 an exceptional year for the record books.
Development
Compared to last year, I am much less productive from a blogging perspective. Partially, this is caused by the fact that there are still many things I have worked on that I could not properly finish.
I have also noticed that there was a considerable drop in my energy level after I completed the test framework for the Nix process management framework. I think this can be attributed to the fact that the process management framework has basically been my only spare time project for over two years.
For a small part, this kind of work is about exploring ideas, but is even more about the execution of those ideas -- unfortunately, being in execution mode for such a long time (while ignoring the exploration of ideas you come up in other areas) gradually made it more difficult to keep enjoying the work.
Despite struggling with my energy levels, I remained motivated to complete all of it, because I know that I am also a very bad multi-tasker. Switching to something else makes it even more difficult to complete it in a reasonable time span.
After I reached all my goals, for a while, it became extremely difficult to get myself focused on any technical challenge.
Next year, I have another big project that I am planning to take on (node2nix), but at the same I will try to schedule proper "breaks" in between to keep myself in balance.
The COVID-19 pandemic
In my annual reflection from last year, I have also elaborated about the COVID-19 pandemic that reached my home country (The Netherlands) in March 2020. Many things have happened that year, and at the time writing my reflection blog post over 2020, we were in our second lockdown.
The second lockdown felt much worse than the first, but I was still mildly optimistic because of the availability of the first vaccine: Pfizer/BioNTech that looked like our "way out". Furthermore, I was also quite fascinated by the mRNA technique making it possible to produce a vaccine so quickly.
Last year, I was hoping that next year I could report that the problem was under control or at least partially solved, but sadly that does not seem to be the case.
Many things have happened: the variant that appeared in England eventually became dominant (Alpha variant) and was considerably more contagious than the original variant, causing the second lockdown to last another three months.
In addition, two more contagaious mutations appeared at the same time in South Africa and Brazil (Beta and Gamma), of which the Alpha variant became the dominant.
Several months later, there was another huge outbreak in India introducing the Delta variant, that was even more contagious than the Alpha variant. Eventually, that variant became the most dominant in the world. Fortunately, the mRNA vaccines that were developed were still effective enough, but the Delta variant was so contagious that it was considered impossible to build up herd immunity to eradicate the virus.
In the summer, the problem seemed to be mostly under control because many people have been vaccinated in my home country. Nonetheless, we have learned in a painful way that relaxing restrictions too much could still lead to very big rapid outbreaks.
We have also learned in a painful way that the virus spreads more easily in the winter. As a result, we also observed that, despite the fact that over 85% of all adults are fully vaccinated, there are still enough clusters of people that have not built up any resistance against the virus (either by vaccination or contracting the virus), again leading to significant problems in the hospitals and ICs.
Furthermore, the effectiveness of vaccines also drops over time, causing vaccinated people with health problems to still end up in hospitals.
As a result of all these hospitalizations and low IC capacity, we are now in yet another lockdown.
What has always worried me the most is the fact that in so many areas in the world, people hardly have any access to vaccines and medicines causing the virus to easily spread on a massive scale and mutate. I knew because of these four variants, it would only be a matter of time before a new dominant mutation will appear.
And that fear eventually became reality -- in November, in Botswana and South Africa, a new and even more contagious mutation appeared: the Omicron variant with a spike-protein that is much different than the delta variant, reducing the effectiveness of our vaccines.
At the moment, we are not sure the implications are. In the Netherlands, as well as many other countries in the world, the Omicron variant has now become the most dominant virus mutation.
The only thing that may be good news is that the Omicron variant could also potentially cause less severe sickness, but so far we have no clarity on that yet.
I think next year we still have much to learn -- to me it has become very clear that this problem will not go away any time soon and that we have to find more innovative/smarter ways to cope with it.
Furthermore, the mutations have also demonstrated that we should probably do more about inequality in the world. As a consequence, the virus could still significantly spread and mutate becoming a problem to everybody in the world.
Blog posts
Last year I forgot about it, but every year I also typically reflect over my top 10 of most frequently read blog posts:
- Managing private Nix packages outside the Nixpkgs tree. As with previous years, this blog post remains the most popular because it is very practical and unanswered in official Nix documentation.
- On Nix and GNU Guix. This blog post used to be my most popular blog post for a long time, and still remains the second most popular. I believe this can be attributed to the fact that this comparison is still very relevant.
- An evaluation and comparison of Snappy Ubuntu. Also a very popular blog post since 2015. It seems that the comparison with Snappy and Flatpak (a tool with similar objectives) remains relevant.
- Disnix 0.5 release announcement and some reflection. This is a blog post that I wrote in 2016 and suddenly appeared in the overall top 10 this year. I am not sure why this has become so relevant all of a sudden.
- On using Nix and Docker as deployment solutions: similarities and differences. This is a blog post that I wrote last year to compare Nix and Docker and explain in what ways they are similar and different. It seems to be very popular despite the fact that it was not posted on discussion sites such as Reddit and Hackernews.
- Yet another blog post about Object Oriented Programming and JavaScript. This explanation blog post is pretty old but seems to stay relevant, despite the fact that modern JavaScript has a class construct.
- Setting up a multi-user Nix installation on non-NixOS systems. Setting up multi-user Nix installations on non-NixOS machines used to be very cumbersome, but fortunately that has been improved in the recent versions. Still, the discussion seems to remain relevant.
- An alternative explanation of the Nix package manager. An alternative explanation that I consider to be better of two that I wrote. It seems to remain popular because I refer to it a lot.
- On NixOps, Disnix, service deployment and infrastructure deployment. A very popular blog post, that has dropped somewhat in popularity. It still seems that the tools and the discussion is relevant.
- Composing FHS-compatible chroot environments with Nix (or deploying steam in NixOS). An old blog post, but it remains relevant because it addresses a very important compatibility concern with binary-only software and a common Nix-criticism that it is not FHS-compatible.
Conclusion
As with 2020, 2021 has been quite a year. I hope everybody stays safe and healthy.
The remaining thing I'd like to say is: HAPPY NEW YEAR!!!
