tag:blogger.com,1999:blog-13971152496316822282024-03-19T12:52:59.545+01:00Sander van der Burg's blogSander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.comBlogger160125tag:blogger.com,1999:blog-1397115249631682228.post-54270541292271997342023-12-30T18:22:00.000+01:002023-12-30T18:22:45.483+01:0013th annual blog reflectionToday, it is the 13th anniversary of my blog. As usual, this is a nice opportunity to reflect over last year's writings.<br />
<br />
Similar to 2022, 2023 was not a very productive year compared to the years before -- I am still a bit in a state of recovery, because of the pressure that I was exposed to two years ago. Another reason is that a substantial amount of my spare time is spend on voluntary work for the musical society that I am a member of.<br />
<br />
<h2>Web framework</h2>
<br />
I have been maintaining the website for the musical society for quite a few years and it is one of last applications that still use my custom web framework. Thanks to my voluntary work, I made a new feature addition to the layout framework: <a href="https://sandervanderburg.blogspot.com/2023/07/using-site-map-for-generating-dynamic.html">generating dynamic menus by using an HTML representation of a site map as a basis</a>, such as a mobile navigation/hamburger menus and dropdown menus.<br />
<br />
I never liked these kinds of menus very much, but they are quite commonly used. In particular, on mobile devices, a web application feels weird if it does not provide a mobile navigation menu.<br />
<br />
The nice thing about using a site map as a basis is that web pages are still able to degrade gracefully -- when using a text-oriented browser or when JavaScript is disabled (JavaScript is mandatory to create an optimal mobile navigation menu), a web site remains usable.<br />
<br />
<h2>Nix development</h2>
<br />
Last year, I explained that I had put my Nix development work on hold. This year, I still did not write any Nix-related blog posts, but I have picked up Nix development work again and I am much more active in the community.<br />
<br />
I have visited <a href="https://2023.nixcon.org">NixCon 2023</a> and I had a great time. While I was at NixCon, I have decided to pick up my work for the experimental process management framework from where I left it behind -- I started writing <a href="https://github.com/NixOS/rfcs/pull/163">RFC 163</a> that explains its features so that they can be integrated into the main Nixpkgs/NixOS distribution.<br />
<br />
Writing an RFC was already on my TODO list for two years, and I always had the intention integrate the good ideas on this framework into Nixpkgs/NixOS so that the community can benefit from it.<br />
<br />
The RFC is still being discussed and we are investigating some of the raised questions and concerns.<br />
<br />
<h2>Research papers</h2>
<br />
I have also been sorting files on my hard drive, something that I commonly do at the end of the year. The interesting thing is that I also ran into research papers that I collected in the last sixteen years.<br />
<br />
Since reading papers and maintaining your knowledge is quite important for researchers and not something that is easy to do, <a href="https://sandervanderburg.blogspot.com/2023/12/on-reading-research-papers-and.html">I wrote a blog post about my experiences</a>.<br />
<br />
<h2>Retro computing</h2>
<br />
Another area that I worked on is retro computing. I finally found the time to get all my old 8-bit Commodore machines (a Commodore 64 and 128) working in the way they should. The necessary repairs were made and I have ordered new and replacement peripherals. <a href="https://sandervanderburg.blogspot.com/2023/12/using-my-commodore-64-and-128-in-2023.html">I wrote a blog post that shows how I have been using these 8-bit Commodore machines</a>.<br />
<br />
<h2>Conclusion</h2>
<br />
Next year, I intend to focus myself more on Nix development. I already have enough ideas the I am working on, so stay tuned!<br />
<br />
The last thing I would like to say is:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfJ6_b6zJeBVbfKyfmB4r_gL_aa_BeNPKkK5aVtwLv4HJzsJtdAk_VJ-V3zfnPPrfJX3iG_WWp0m0HjSTppZhDIW4uw5nK_PIelE795rgU2GkA1UEE1C9V6Clzi7nouCIjqT141WgpCNYippPbIbl_AYo2izhApUMiCIMMfPzqRjR59p_hKWIGWTR-DnKz/s1024/White_bright_fireworks.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="768" data-original-width="1024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfJ6_b6zJeBVbfKyfmB4r_gL_aa_BeNPKkK5aVtwLv4HJzsJtdAk_VJ-V3zfnPPrfJX3iG_WWp0m0HjSTppZhDIW4uw5nK_PIelE795rgU2GkA1UEE1C9V6Clzi7nouCIjqT141WgpCNYippPbIbl_AYo2izhApUMiCIMMfPzqRjR59p_hKWIGWTR-DnKz/s600/White_bright_fireworks.jpg"/></a></div>
<br />
HAPPY NEW YEAR!!!<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-10120884141883453252023-12-28T23:43:00.005+01:002024-02-27T09:35:41.774+01:00Using my Commodore 64 and 128 in 2023<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmvo8t7fV1F0kznA_naTVVHRancfE_ekey9jHlrL-n4y45H0iixaXtf7_nk0i1JoLcUegUoNxQQ1qIzS934fH1eyWG4Slu3GkeRM_j7mG0o6QzJEqDhD9j_HmRNYPgnqLBABQRzLFARzaBvqPB5Gip5Z0_mUzb_YkxQvvn2BxBz1uQHPWKfWOA5DzHQdXB/s4032/20210905_123050.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgmvo8t7fV1F0kznA_naTVVHRancfE_ekey9jHlrL-n4y45H0iixaXtf7_nk0i1JoLcUegUoNxQQ1qIzS934fH1eyWG4Slu3GkeRM_j7mG0o6QzJEqDhD9j_HmRNYPgnqLBABQRzLFARzaBvqPB5Gip5Z0_mUzb_YkxQvvn2BxBz1uQHPWKfWOA5DzHQdXB/s600/20210905_123050.jpg"/></a></div>
<br />
Two years ago, I wrote <a href="https://sandervanderburg.blogspot.com/2021/10/using-my-commodore-amiga-500-in-2021.html">a blog post about using my Commodore Amiga 500 in 2021</a> after not having touched it in ten years. Although the computer was still mostly functional, some peripherals were broken.<br />
<br />
To fix my problems, I brought it to the <a href="https://www.homecomputermuseum.nl">Home Computer Museum in Helmond</a> for repairs.<br />
<br />
Furthermore, I have ordered replacement peripherals so that the machine can be more conveniently used, such as a GoTek floppy emulator. The GoTek floppy emulator makes it possible to conveniently use disk images stored on an USB memory stick as a replacement for physical floppy disks.<br />
<br />
I also briefly mentioned that I have been using <a href="https://sandervanderburg.blogspot.com/2011/03/first-computer.html">my first computer</a>: a Commodore 128 for a while. Moreover, I also have a functional Commodore 64, that used to be my third computer.<br />
<br />
Although I have already been these 8-bit machines on a more regular basis since 2022, I was not satisfied enough yet to write about them, because there were still some open issues, such as a broken joystick cable and the unavailability of the <a href="https://ultimate64.com/CartridgesWithTape">1541 Ultimate II cartridge</a>. The delivery took a while because it had to be redesigned/reproduced due to chip shortages.<br />
<br />
A couple of weeks ago, the cartridge was finally delivered. In my Christmas holiday, I finally found the time to do some more experiments and write about these old 8-bit Commodore machines.<br />
<br />
<h2>My personal history</h2>
<br />
The Commodore 128, that is still in my possession, originally belonged to my parents and was the first computer I was exposed to. Already as a six year old, I knew the essential BASIC commands to control it, such as requesting a disk's contents (e.g. <i>LOAD"$",8: LIST</i>), loading programs from tape and disk (e.g. <i>LOAD</i>, <i>LOAD"*",8,1</i>) and running programs (<i>RUN</i>).<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGRyqLnT7xhwRTlc86nH5wVdFW1AwjSVa5neDcg9ZRBewVYnI2PA8Jfd97qoXL935m12sjwLspBJ6x1138LA5gru6FgPwoP4pUAdqnwKKhQr-7OAijrsLImzdf-yi668pNBL23HrGTpLusKogRlQ8tzhKePXH9I9O1Y4_J5lYtKX8A53y2k4Hxqi1BxwZN/s1600/tmnt.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="272" data-original-width="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGRyqLnT7xhwRTlc86nH5wVdFW1AwjSVa5neDcg9ZRBewVYnI2PA8Jfd97qoXL935m12sjwLspBJ6x1138LA5gru6FgPwoP4pUAdqnwKKhQr-7OAijrsLImzdf-yi668pNBL23HrGTpLusKogRlQ8tzhKePXH9I9O1Y4_J5lYtKX8A53y2k4Hxqi1BxwZN/s1600/tmnt.png"/></a></div>
<br />
One of my favorite games was the <a href="https://www.lemon64.com/game/teenage-mutant-ninja-turtles">Commodore 64 version of Teenage Mutant Ninja Turtles developed by Ultra Software</a>, as can be seen in the above screenshot.<br />
<br />
I liked the game very much because I was a fan of the TV show, but it was also quite buggy and notoriously difficult. Some parts of the game were as good as impossible to finish. As a result, I was never able to complete the game, despite having played it for many hours.<br />
<br />
Many relatives of mine used to have an 8-bit Commodore machine. A cousin and uncle used to own a Commodore 64C, and another uncle owned a Commodore 128. We used to exchange ideas and software quite a lot.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3qrhxflvdK2paBnlXjQIig2vRcBVCRXT9fP_ce5LiUJQ5xSIQc0Y0SE9I9W_VBCFBMyCvXzuH4S7dDdDjEJ-ekg16uliHj7b7QZw6HGb0S_lrBxYi9vNYhjRZHuoXpEVVjjP4O0P-bNQZDKT60OLrXXkN5-cTvwbFQhbb0JRB7GPYm1VlB_FLYx2meFjs/s4032/20210905_122720.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3qrhxflvdK2paBnlXjQIig2vRcBVCRXT9fP_ce5LiUJQ5xSIQc0Y0SE9I9W_VBCFBMyCvXzuH4S7dDdDjEJ-ekg16uliHj7b7QZw6HGb0S_lrBxYi9vNYhjRZHuoXpEVVjjP4O0P-bNQZDKT60OLrXXkN5-cTvwbFQhbb0JRB7GPYm1VlB_FLYx2meFjs/s600/20210905_122720.jpg"/></a></div>
<br />
At first, I did not know that a Commodore 128 was a more capable machine than an ordinary Commodore 64. My parents used to call it a Commodore 64, and for quite some time I did not know any better.<br />
<br />
The main reason behind the confusion is that a Commodore 128 is nearly 100% backwards compatible with a Commodore 64 -- it contains the same kinds of chips and it offers a so-called Commodore 64 mode.<br />
<br />
You can switch to Commodore 64 mode by holding the Commodore logo key on bootup or by typing: <i>GO64</i> on the command prompt. When a utility cartridge is inserted, the machine always boots in Commodore 64 mode. The picture above shows my Commodore 128 running in Commodore 64 mode.<br />
<br />
At the time, we had a utility cartridge inserted into the cartridge socket that offered fast loading, preventing us from seeing the Commodore 128 mode altogether. Moreover, with the exception of the standard software that was bundled with the machine, we only had Commodore 64 software at our disposal.<br />
<br />
In 1992, I wrote my first BASIC program. The program was very simple -- it changes the colors of the text, screen and screen border, asks somebody to provide his name and then outputs a greeting.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7zAJKWh6qkkga6O1Orm-u7bMpJWVrBvL4M_-w0PnRTRobhySu1oom3OSoXh4bP3UBBlDQdzDHIiGnek5mOo3DJKK2Blj3tLhIUrSxMDsnTRyEap-wRCmbGGtUlJD-JhOziJnkhPMyH0luh54T0F0lwXk2X7O9WM_ioTBy6kYHlDnSHdjTd4CTgncHvE-8/s4032/20210905_122601.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7zAJKWh6qkkga6O1Orm-u7bMpJWVrBvL4M_-w0PnRTRobhySu1oom3OSoXh4bP3UBBlDQdzDHIiGnek5mOo3DJKK2Blj3tLhIUrSxMDsnTRyEap-wRCmbGGtUlJD-JhOziJnkhPMyH0luh54T0F0lwXk2X7O9WM_ioTBy6kYHlDnSHdjTd4CTgncHvE-8/s600/20210905_122601.jpg"/></a></div>
<br />
At some point, by accident, the utility cartridge was removed and I discovered the Commodore 128 mode, as can be seen in the picture above. I learned that the Commodore 128 ROM had a more advanced BASIC version that, for example, also allows you to play music with the <i>PLAY</i> command.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfNEerTyDjEUCdn8vgNBjBi9sq9P56zgjVAcp-s1r8XQKwNfPmQRBP7RfmgNxzc4Y4QxZj33W-bKqL_Z6WwSNufy6CCIzdQHXvHmqqUlorYwVK5xpBzK9mUTM-zqCWjGOGVmU-HE6Q4d5MGPKToIMD18wE725TY8VYdmFEo_d4-E1ipYGDu62dHubiIbNJ/s4032/20210905_123217.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgfNEerTyDjEUCdn8vgNBjBi9sq9P56zgjVAcp-s1r8XQKwNfPmQRBP7RfmgNxzc4Y4QxZj33W-bKqL_Z6WwSNufy6CCIzdQHXvHmqqUlorYwVK5xpBzK9mUTM-zqCWjGOGVmU-HE6Q4d5MGPKToIMD18wE725TY8VYdmFEo_d4-E1ipYGDu62dHubiIbNJ/s600/20210905_123217.jpg"/></a></div>
<br />
I also discovered the CP/M disk that was included with the machine and tried it a few times. It looked interesting (as can be seen in the picture above) but I had no applications for it, so I had no idea what to do with it. :)<br />
<br />
I liked the Commodore 128 features very much, but not long after my discovery, my parents bought a Commodore Amiga 500 and gave the Commodore 128 to another uncle. All my relatives that used to have an 8-bit Commodore machine already made the switch to the Amiga, and we were the last to make the transition.<br />
<br />
Although switching to a next generation machine may sound exciting, I felt disappointed. In the last year that the Commodore 128 was still our main machine, I learned so much, and I did not like it that I was no longer be able to use the machine and learn more about my discoveries. Fortunately, I could still play with the old Commodore 128 once in a while when we visited that uncle that we gave the machine to.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheQdUgHuPcXttmijAVHhpk-iw1eZBxWgWSdRtu7m1rsdtcutzGptM0ffsIS_MUukGf-Ga-JnX1WF4q5R0FDn7G0b6tfpgMzCuLgeNOym3sCj-QKh1rCjslvOYDnkVJpQCW-1EHNrbUUINHBCvxOn8evV_lUhPo1FoW1Z29770uMov6_224hEJ3gLBzMuau/s4032/20231228_142649.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEheQdUgHuPcXttmijAVHhpk-iw1eZBxWgWSdRtu7m1rsdtcutzGptM0ffsIS_MUukGf-Ga-JnX1WF4q5R0FDn7G0b6tfpgMzCuLgeNOym3sCj-QKh1rCjslvOYDnkVJpQCW-1EHNrbUUINHBCvxOn8evV_lUhPo1FoW1Z29770uMov6_224hEJ3gLBzMuau/s600/20231228_142649.jpg"/></a></div>
<br />
Some time later, in late 1993, my parents gave me a Commodore 64 (the old fashioned breadbin model) that they found at a garage sale (as shown in the picture above). This was the third computer model that I was exposed to and the first computer that was truly mine, because I did not have to share it with my parents and brother. This machine gave me my second 8-bit Commodore experience, and I have been using this old machine for quite some time, until mid-1997.<br />
<br />
Originally, the Commodore 64 did not come with any additional peripherals. It was just the computer with a cassette drive and no utility cartridges for fast loading. I had a cassette with some games and a fast loading program that was the first program on the tape. Nothing more. :)<br />
<br />
I was given a few books and I picked up Commodore 64 programming again. In the following years, I learned much more about programming and the capabilities of the Commodore 64, such as for-loops, how to do I/O, how to render sprites and re-program characters.<br />
<br />
I have also been playing around with audio, but my sound and music skills were far too limited to do anything that makes sense. Moreover, I did quite a few interesting attempts to create games, but nothing truly usable came out of it. :)<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0vIzzPIsftbwHArdSv9rzWjajS6l7ZKdKYgOUXy7mTAC3bZYnraACyLkj2aSbRO_8GqALkfM0KRwhTjEZx7DLnZlF1Eg4q1sN9yG31IUNJsnnDq-W_EOOhtcpUNBaisLZ3NqTFtYyri8QUQ5oMUuxCNMihBJuX1uDTcrq0SYI2_1_latLatqHzDv6VAS1/s4032/20231228_143209.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="500" data-original-height="4032" data-original-width="3024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj0vIzzPIsftbwHArdSv9rzWjajS6l7ZKdKYgOUXy7mTAC3bZYnraACyLkj2aSbRO_8GqALkfM0KRwhTjEZx7DLnZlF1Eg4q1sN9yG31IUNJsnnDq-W_EOOhtcpUNBaisLZ3NqTFtYyri8QUQ5oMUuxCNMihBJuX1uDTcrq0SYI2_1_latLatqHzDv6VAS1/s600/20231228_143209.jpg"/></a></div>
<br />
In 1994, I bought a 1541 disk drive and several utility cartridges at a garage sale, such as the Final Cartridge (shown above). The Final Cartridge provides all kinds of useful features, such as fast loading and the ability to interrupt the machine and inspect its memory with a monitor.<br />
<br />
Owning a disk drive also allowed me to make copies of the games that I used to play on my parents' Commodore 128.<br />
<br />
Eventually, in 1998 I switched to the Commodore Amiga 500 as my personal machine, but I kept my Commodore 64. In 1998, Commodore was already out of business for four years and completely lost its relevance. My parents bought a PC in 1996. After using the Amiga for a while on the attic, the Amiga's display broke rendering it unusable. In 1998, I discovered how to attach the Amiga to a TV.<br />
<br/>
In late 1999, I was finally able to buy my own PC. I kept the Amiga 500, because I still considered it a cool machine.<br />
<br />
Several years later, the Commodore 128 was returned to me. My uncle no longer had any use for it and was considering to throw it away. Because I still remembered its unique features (compared to a regular Commodore 64), I have decided to take it back.<br />
<br />
<h2>Some facts</h2>
<br />
Why is the Commodore 64 such an interesting machine? Besides the fact that it was the first machine that I truly owned, it has also been <a href="https://content.time.com/time/specials/2007/article/0,28804,1638782_1638778_1638764,00.html">listed in the Guinness World Records as the highest-selling single computer model of all time</a>.<br />
<br />
Moreover, it also has interesting capabilities, such as:<br />
<br />
<ul>
<li>64 KiB of RAM. This may not sound very impressive for nowadays' standards (for example, my current desktop PC has 64 GiB of RAM, a million times as much :) ), but in 1982 this used to be a huge amount.</li>
<li>A 6510 CPU, which is a modified 6502 CPU that has an 8-bit I/O port added. On machines with a PAL display, it runs at a clock speed slightly under 1 MHz.<br />
Compared to modern CPUs, this may not sound impressive (a single core of my current CPU runs at 3.7 GHz :) ), but in the 80s the CPU was quite good -- it was cheap and very efficient.<br />
<br />
Despite the fact that there were competing CPUs at the time that ran at higher clock speeds, most 6502 instructions only take a few cycles and fetch the next instruction from memory while the previous instruction is still in execution. As a result, it was still a contender to most of its competitors that ran at higher clock speeds.</li>
<li>A nice video chip: the VIC chip. It supports 16 preconfigured colors and various screen modes, including high resolution screen modes that can be used for addressing pixels. It also supports eight hardware managed <strong>sprites</strong> -- movable objects directly controlled by the video chip.</li>
<li>A nice sound chip: the SID chip. It offers three audio channels, four kinds of waveforms (triangle, sawtooth, squarewave and white noise) and analog mixing. This may not sound impressive, but at the time, the fact that the three audio channels can be used to mix waveforms arbitrarily was a very powerful capability.</li>
<li>An operating system using a BASIC programming language interpreter (Commodore BASIC v2) as a shell. In the 70s and 80s, BASIC was a very popular programming language due to its simplicity.</li>
</ul>
<br />
Other interesting capabilities of the Commodore 64 were:<br />
<br />
<ul>
<li>The RAM is shared between the CPU and other chips, such as the VIC and SID. As a result, the CPU is offloaded to do calculation work only.</li>
<li>The CPU's clock speed is aligned with the video beam. The screen updates 50 times per second and the screen is rendered from top to bottom, left to right. Each screen block takes two cpu cycles to render.<br />
<br />
These properties make it possible to change the screen while it is rendered (a technique called <strong>racing the beam</strong>). For example, while the screen is drawn, it is possible to adjust the colors in color RAM, multiplexing sprites (by default you can only configure eight sprites), changing screen modes (e.g. from text to high res etc).<br />
<br />
For example, the following screenshot of a computer game called: <a href="https://www.lemon64.com/game/mayhem-in-monsterland">Mayhem in Monsterland</a> demonstrates what is possible by "racing the beam". In the intro screen (that uses <a href="https://www.c64-wiki.com/wiki/Multicolor_Bitmap_Mode">multi-color bitmap mode</a>), we can clearly see more colors per scanline than three unique colors and a background color per 8x8 block, that is normally only possible in this screen mode:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHV5gY8DQNk_RiVW9WGeTiDvSnWoPE8dlu-MEU70rJ_Yr-dtMrjW-qcHICopZSNXZayYHn-xW3ZRhwZfhMXijL4YeMtmAgd6ypJDZRuVW7i13x4pJxeFhuwJDA2FB-Pzrr9ZxnPMWwLXI_8ZqIi2zIMfL7P_AbOhpieRZvtrXW7VrjfWGvO9bDnwVhVtQH/s384/mayhem.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="272" data-original-width="384" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjHV5gY8DQNk_RiVW9WGeTiDvSnWoPE8dlu-MEU70rJ_Yr-dtMrjW-qcHICopZSNXZayYHn-xW3ZRhwZfhMXijL4YeMtmAgd6ypJDZRuVW7i13x4pJxeFhuwJDA2FB-Pzrr9ZxnPMWwLXI_8ZqIi2zIMfL7P_AbOhpieRZvtrXW7VrjfWGvO9bDnwVhVtQH/s600/mayhem.png"/></a></div>
</li>
<li>And of course, the Commodore 64 has <a href="http://www.lemon64.com">a huge collection games</a> and applications.</li>
</ul>
<br />
The Commodore 128 has similar kinds of chips (as a result, it is nearly 100% compatible with the Commodore 64).<br />
<br />
It has the following changes and additions:<br />
<br />
<ul>
<li>Double the amount of RAM: 128 KiB</li>
<li>A second video chip: the VDC chip, that can render 80-column text and higher resolution graphics, but no sprites. To render 80-column output, you need to connect an RGBI cable to a monitor that is capable of displaying 80-column graphics. The VDC chip does not work with a TV screen.</li>
<li>A CPU that is twice as fast: the 8502, but entirely backwards compatible with the 6510. However, if you use the VIC chip for rendering graphics (40 column mode) the chip still runs at half its speed, which is the same as the ordinary 6510. In 80-column mode or when the screen is disabled, it runs at twice the speed of a 6510.</li>
<li>A second CPU: the Zilog Z80. This CPU is used after booting the machine in CP/M mode from the CP/M boot disk.</li>
<li>An improved BASIC interpreter that supports many more features: Commodore BASIC v7.0</li>
</ul>
<br />
<h2>Using the Commodore machines</h2>
<br />
To conveniently use the Commodore machines in 2023, I have made a couple of repairs and I have ordered new peripherals.<br />
<br />
<h3>Power supplies</h3>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiebI8eIkCqMKSV2UV_Ovhd_qCbkmyKSGJL6_vrOTMqj6eZMcE6wse_znK8mJ_P6g4oYgkidl4Pr30prbpjPpjr1F0fgkmxvNt9Lq0E1UOOeTraM9L1ZrsVGtNkCX8XIVYeIatw9y3_TXDYj5kQroxHLpU-lM0H3tKXlviy-LWqXmJSiPYEO1xUhkDMy5q_/s4032/powersupplies.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiebI8eIkCqMKSV2UV_Ovhd_qCbkmyKSGJL6_vrOTMqj6eZMcE6wse_znK8mJ_P6g4oYgkidl4Pr30prbpjPpjr1F0fgkmxvNt9Lq0E1UOOeTraM9L1ZrsVGtNkCX8XIVYeIatw9y3_TXDYj5kQroxHLpU-lM0H3tKXlviy-LWqXmJSiPYEO1xUhkDMy5q_/s600/powersupplies.jpg"/></a></div>
<br />
I bought new power supplies. I learned that <a href="https://retrogamestart.com/answers/replace-c64-power-supply-voltage-failure-will-kill-your-c64">it is not safe to use the original Commodore 64 power supply</a> as it gets older -- it may damage your motherboard and chips:<br />
<br />
<blockquote>
In a nutshell, the voltage regulator on the 5 volt DC output tends to fail in such a way that it lets a voltage spike go straight to your C64 motherboard, frying the precious chips.
</blockquote>
<br />
Fortunately, modern replacement power supplies exist. I bought one from: <a href="http://www.c64lover.com">c64lover.com</a> that seems to work just fine.<br />
<br />
I also bought <a href="https://www.keelog.com/power-supply/">a replacement power supply for the Commodore 128 from Keelog</a>. The original Commodore 128 power supply is more robust than the Commodore 64 power supply, but I still want some extra safety.<br />
<br />
<h3>Cassette drive</h3>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8NZlSPEb1ZUM_4VV5Uv7qJkqJTgvHPrCbWAw8QuLkjv5m0bj0yicWK28eHMj_aQV66Q5BwmTozqYutUtfD_CDGkev-mvCC4a2HmABocvxg1V2jWJYCdsApn4ZHTAevyMryn1uWuPZl8SzXgZhkFpMHucuYUWtAoDCfcjuG0PEuB5R1zyLwWQ1sRUcPJH3/s4032/20231228_141758.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8NZlSPEb1ZUM_4VV5Uv7qJkqJTgvHPrCbWAw8QuLkjv5m0bj0yicWK28eHMj_aQV66Q5BwmTozqYutUtfD_CDGkev-mvCC4a2HmABocvxg1V2jWJYCdsApn4ZHTAevyMryn1uWuPZl8SzXgZhkFpMHucuYUWtAoDCfcjuG0PEuB5R1zyLwWQ1sRUcPJH3/s600/20231228_141758.jpg"/></a></div>
<br />
As I have already explained, my Commodore 64 breadbin model only included a cassette drive. I still have that cassette drive (as shown in the picture above), but after obtaining a 1541 disk drive, I never used it again.<br />
<br />
Two years ago, I ordered an SD2IEC that included a bonus cassette with a game: <a href="https://misfit.itch.io/hi-score">Hi-Score</a>. I wanted to try the game, but it turned out there was a problem with the cassette drive -- it seems to spin irregularly.<br />
<br />
After a brief investigation I learned that the drive belt was in a bad condition. I have ordered a replacement belt from Ebay. Installing it was easy, and the game works great:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPfUuFoMAPZkD4oVAqAznd-HYioSlL0AJra4lcVSmh9wI808Xy3wuSaEu1xp86BIaADYeoiHq6_6TZjpqNDkB4WlUfC3IjZMDBW1rgnsVMIBIGgJu7hzp86iW6gyd5VagNWnha41zE8qDW5Vt2y2nkh2wtqnM0YHep067ZE2ST0wJz9j0Kmi7ege9hAdln/s4032/20231228_142851.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjPfUuFoMAPZkD4oVAqAznd-HYioSlL0AJra4lcVSmh9wI808Xy3wuSaEu1xp86BIaADYeoiHq6_6TZjpqNDkB4WlUfC3IjZMDBW1rgnsVMIBIGgJu7hzp86iW6gyd5VagNWnha41zE8qDW5Vt2y2nkh2wtqnM0YHep067ZE2ST0wJz9j0Kmi7ege9hAdln/s600/20231228_142851.jpg"/></a></div>
<br />
<h3>Disk drives</h3>
<br />
<div class="separator" style="float: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFOCl0J_lVDxrCzR7hh_dKde3OT3QV6YvBhs3SRPS_xE-UOJzp3SGQJuyC1fDlpmrTmveIT62EATVK-g909knjOe99P3Xl6-rFwMJ8qop7QqnWUKrZccL7QW3WrFYE4-k2ashQrtHdJ8Tn6J-eQK86J2uJTpJLHSQNelviqwa2MTBJkuu57pmu4KbzYgwj/s4032/1541.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="300" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgFOCl0J_lVDxrCzR7hh_dKde3OT3QV6YvBhs3SRPS_xE-UOJzp3SGQJuyC1fDlpmrTmveIT62EATVK-g909knjOe99P3Xl6-rFwMJ8qop7QqnWUKrZccL7QW3WrFYE4-k2ashQrtHdJ8Tn6J-eQK86J2uJTpJLHSQNelviqwa2MTBJkuu57pmu4KbzYgwj/s600/1541.jpg"/></a></div>
<div class="separator" style="float: right;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2m_GJOwaHd7IRGqLuLQTmbgL7POUdpCcwQ5a95q5zOkKB0TLtZvfhR1bGMsuMhNtL2MWGbh2CcL0iCErdjtoU4BjpegwscxC7pZxrUcZnr1bJr7vjl5QYTC96HtOSFOTSWanYjvdmK3KBix3QriX4wwhJgFH17QAMt2kofBYt3EQCcBYWvSSVRklIpnmw/s4032/20220205_145551.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="250" data-original-height="4032" data-original-width="3024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj2m_GJOwaHd7IRGqLuLQTmbgL7POUdpCcwQ5a95q5zOkKB0TLtZvfhR1bGMsuMhNtL2MWGbh2CcL0iCErdjtoU4BjpegwscxC7pZxrUcZnr1bJr7vjl5QYTC96HtOSFOTSWanYjvdmK3KBix3QriX4wwhJgFH17QAMt2kofBYt3EQCcBYWvSSVRklIpnmw/s600/20220205_145551.jpg"/></a></div>
<div style="clear: both;"></div>
<br />
I have two disk drives. As I have already explained, I have a 1541 drive that I bought from a garage sale for my Commodore 64. The pictures above show the exterior and interior of the disk drive.<br />
<br />
The disk drive still works, but I had a few subtle problems with running modern demos that concurrently load data while playing the demo. Such demos would sometimes fail (crash or the sound starts to run out of sync with the rest of the demo), because of timing problems.<br />
<br />
I have <a href="https://retro64.altervista.org/blog/commodore-1541-disk-drive-maintenance-part-1/">cleaned the disk drive head with some alcohol</a> and that seemed to improve the situation.<br />
<br />
<div class="separator" style="float: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6LZ2v1dT8xWhyr8YbFeMPrqf4uDw-G0BRWrR0Un7Ijd3sDEe4f3Wj34xVGyAQGPwrA2LxlFuvEs_gODmRmFcJUgZy65w3Kqfnh073hs3y1WFR1w-FZLx5g94EGdAEcTninzKHkwUK9c5LQtvp1zfal3z9E7dfQTKTKVgu_rWWvhIVNVMp0RpXYjtwFHyw/s4032/1571.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="300" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh6LZ2v1dT8xWhyr8YbFeMPrqf4uDw-G0BRWrR0Un7Ijd3sDEe4f3Wj34xVGyAQGPwrA2LxlFuvEs_gODmRmFcJUgZy65w3Kqfnh073hs3y1WFR1w-FZLx5g94EGdAEcTninzKHkwUK9c5LQtvp1zfal3z9E7dfQTKTKVgu_rWWvhIVNVMp0RpXYjtwFHyw/s600/1571.jpg"/></a></div>
<div class="separator" style="float: right;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9Rjdsk4y98ZaCFDfovJq82pftKrYtjvbcLBVPyfH2KVgq6RZtRz5GgvATL5eFKbL7CoCI75po1OeK_tfgFatbvbuzYn6u55UnTVUFMmGztjGSQRjVOqqnRtoQGvPLg-CqqOJEwZubR76UldrKDgGvkk3PcRQ2TZFbHG863Qh1QgGnBHwa84-ReKcqPKiz/s4032/20220206_152658.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="250" data-original-height="4032" data-original-width="3024" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9Rjdsk4y98ZaCFDfovJq82pftKrYtjvbcLBVPyfH2KVgq6RZtRz5GgvATL5eFKbL7CoCI75po1OeK_tfgFatbvbuzYn6u55UnTVUFMmGztjGSQRjVOqqnRtoQGvPLg-CqqOJEwZubR76UldrKDgGvkk3PcRQ2TZFbHG863Qh1QgGnBHwa84-ReKcqPKiz/s600/20220206_152658.jpg"/></a></div>
<div style="clear: both;"></div>
<br />
I also have a 1571 disk drive that came with the Commodore 128. The 1571 disk drive is a more advanced disk drive, that is largely backwards compatible with the 1541. The pictures above show the exterior and interior of the drive.<br />
<br />
In addition to ordinary Commodore 64 disks, it can also read both sides of a floppy disk at the same time and use disks that are formatted to be as such. It can also run in MPM mode to read CP/M floppy disks.<br />
<br />
My 1571 disk drive still seems to work fine. The main reason why I did not have to clean it is because I have not used it as much as the 1541 disk drive.<br />
<br />
The 1541 and 1571 disk drives are interesting devices. They are computer systems themselves -- each of them contains two embedded machines each having their own 6502 CPUs. One sub system is responsible for managing filesystem operations and the communication with the main computer, while the other sub system is used for controlling the drive.<br />
<br />
The 1541 disk drive contains 2 KiB of RAM and runs its own software from a ROM chip that provides the Commodore Disk Operating System.<br />
<br />
Technically, using a disk drive on an 8-bit Commodore machine is the same as two computer systems communicating with each other over Commodore's proprietary serial interface (the IEC).<br />
<br />
<h3>Monitor</h3>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxPZTG6oFBaWfLcsoLXPC9DM-PYt3veH37itlTotYIBoW7vIJaGsWwZRdf7Z4al_6vpNZmlcmH9PjRxzaUc9-JUbE9tZ2MaTtQvtxsuQU0j0iJ3t0jm2GxkOK1PaC8-zExqpPeCwSxsbyTXUcWCRTOXtkypag7fWbyXEj6U6AzxBYfps1mcbROzRuQJ8iw/s4032/20210905_122646.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgxPZTG6oFBaWfLcsoLXPC9DM-PYt3veH37itlTotYIBoW7vIJaGsWwZRdf7Z4al_6vpNZmlcmH9PjRxzaUc9-JUbE9tZ2MaTtQvtxsuQU0j0iJ3t0jm2GxkOK1PaC8-zExqpPeCwSxsbyTXUcWCRTOXtkypag7fWbyXEj6U6AzxBYfps1mcbROzRuQJ8iw/s600/20210905_122646.jpg"/></a></div>
<br />
I also have a monitor for the Commodore 128 machine: the Commodore 1901 that is capable of displaying graphics in 40 and 80 column modes. It has an RGBI socket for 80-column graphics and RCA sockets for 40-column graphics. I need to use a switch on the front panel to switch between the 40 and 80 column graphics modes. In the picture shown above, I have switched the monitor to 80-column mode.<br />
<br />
The monitor still works fine, but in 2019 a capacitor burned out, causing smoke to come out of the monitor, which was a scary experience.<br />
<br />
Fortunately, the monitor was not irreparably damaged and the home computer museum managed to replace the broken capacitors. After it was returned to me, it seems to work just fine again.<br />
<br />
In 1992, besides "The Very First": a programming tutorial and CP/M, I did not have any software for the Commodore 128. In the recent years I have downloaded a few interesting applications for the Commodore 128, such as a Tetris game that works in 80-column mode:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimqaJSrcARvYxdPmgmfnIAjfvkdLZFqsw49Glq85JKAKcw_XI-9aid7vYefAYeRGAc6mDmLFfReoHKNhkUjA9-oCVc360lkaOf5FPoqKIU1e08uJiQHORmJ27hSx72Vn7r6Xol_e1AEvLDBRuymC9WtncrC6j7PFOynWOnSTnV9fl4FhNp4Sd1-8itYedC/s4032/20231229_144742.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimqaJSrcARvYxdPmgmfnIAjfvkdLZFqsw49Glq85JKAKcw_XI-9aid7vYefAYeRGAc6mDmLFfReoHKNhkUjA9-oCVc360lkaOf5FPoqKIU1e08uJiQHORmJ27hSx72Vn7r6Xol_e1AEvLDBRuymC9WtncrC6j7PFOynWOnSTnV9fl4FhNp4Sd1-8itYedC/s600/20231229_144742.jpg"/></a></div>
<br />
<h3>Joysticks</h3>
<br />
As already shown in earlier pictures, I am using <a href="https://en.wikipedia.org/wiki/The_Arcade_(joystick)">The Arcade joysticks</a> produced by Suzo International. I have four of them.<br />
<br />
Unfortunately, three of them did not work properly anymore because their cables were damaged. I have managed to have them repaired by using <a href="https://www.amiga-shop.net/en/Amiga-Hardware/Amiga-cables-adapters/Joystick-joypad-replacement-cable-set-DB9::897.html">this cable from the Amiga shop</a> as a replacement.<br />
<br />
<h3>Mouse</h3>
<br />
The Commodore 128 also came with a 1351 mouse (a.k.a. tank mouse), but it was lost. I never used a mouse much, except for <a href="https://www.c64-wiki.com/wiki/GEOS">GEOS</a>: a graphical operating system.<br />
<br />
To get that GEOS experience back, I first bought <a href="https://www.pcbway.com/project/shareproject/PS_2_Mouse_adapter_for_Commodore_64__1351_mouse_hardware_emulation_.html">an adapter device that allows you to use a PS/2 mouse as a 1351 mouse</a>. Later, I found the same 1351 mouse model on Ebay:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxB6AoFI_jFhl6YoHU2UcLlTCZVV9BO7VEicdsycOHf_JTQvv6yvjSDmlank9PuCeIV6GkdBaYT8DEGaty47EcocxptzCQmqDiBtS3RaoArMz5PQbmotoruhXPmMhsX6I03OlN9LnGInLeGEPbyVOBku543HYdzn5bFnQg1DUjtKfza3QKedtbVMmw0thE/s4032/20231228_143738.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjxB6AoFI_jFhl6YoHU2UcLlTCZVV9BO7VEicdsycOHf_JTQvv6yvjSDmlank9PuCeIV6GkdBaYT8DEGaty47EcocxptzCQmqDiBtS3RaoArMz5PQbmotoruhXPmMhsX6I03OlN9LnGInLeGEPbyVOBku543HYdzn5bFnQg1DUjtKfza3QKedtbVMmw0thE/s600/20231228_143738.jpg"/></a></div>
<br />
<h3>SD2IEC</h3>
<br />
I have also been looking into more convenient ways to use software that I have downloaded from the Internet. Transferring downloaded disk images from and to physical floppy disks is possible, but quite inconvenient.<br />
<br />
The <a href="https://www.c64-wiki.com/wiki/SD2IEC">SD2IEC</a> is a nice modern replacement for a 1541 disk drive -- it is cheap, it can be attached to the IEC port and all high-level disk commands seem to be compatible. Physically, it also looks quite nice:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTTcYfQ1nlDd9csu32qY6NBtZ-7nYu5RC_7YLeZkj4MrCapVb1w26IIyVug4OxokEyn1Y4O-ESsZsM8qtx-rXeq9wSKqEpXiaXDKyG89WR6kX1rvQWAazaUD7KhdDmBXu2ggcG0UuWf1brWG7LAMULomWePoN9lwwTEfVCYGkLm6guL51BcWd28UxHw1de/s4032/20211103_111916.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTTcYfQ1nlDd9csu32qY6NBtZ-7nYu5RC_7YLeZkj4MrCapVb1w26IIyVug4OxokEyn1Y4O-ESsZsM8qtx-rXeq9wSKqEpXiaXDKyG89WR6kX1rvQWAazaUD7KhdDmBXu2ggcG0UuWf1brWG7LAMULomWePoN9lwwTEfVCYGkLm6guL51BcWd28UxHw1de/s600/20211103_111916.jpg"/></a></div>
<br />
As can be seen it the picture above, it looks very similar to an ordinary 1541 disk drive.<br />
<br />
Unfortunately, low-level disk-drive operations are not supported -- some applications need to carry out low-level operations for fast loading, such as demos.<br />
<br />
Nonetheless, the SD2IEC is still a great solution, because there are plenty of applications and games that do not require any fast loading capabilities.<br />
<br />
<h3>1541 Ultimate II cartridge</h3>
<br />
After happily using the SD2IEC for a while, I wanted better compatibility with the 1541 disk drive. For example, many modern demos are not compatible, and I do not have enough physical floppy disks to store these demos on.<br />
<br />
As I have already explained, this year, I have ordered the 1541 Ultimate II cartridge, but it took a while before it could be delivered.<br />
<br />
The 1541 Ultimate II cartridge is an impressive device -- it can be attached to the IEC port and the tape port (by using a tape adapter) and provides many interesting features, such as:<br />
<br />
<ul>
<li>It can cycle-exact emulate two 1541 disk drives</li>
<li>It offers two emulated SID chips</li>
<li>It can load cartridge images</li>
<li>It simulates disk drive sounds</li>
<li>You can attach up to two USB storage devices. You can load disk images and tape images, but you can also address files from the file system directly.</li>
<li>It has an ethernet adapter.</li>
</ul>
<br />
<div class="separator" style="float: left;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9TLRrbgyKs9_DjS557ftC38KhTffr2vpTJVQjUXqFRlMrTzz7azxTdGBdAe2r-_fjpXt_i_1-o-KVgPPFgUxZuucmuzGmrz28-ZShXfHzQD5mAYy73AZ1JtYQFpAkO6vTygereaOVrb9FmGtevdsFiDMT06cFnz0WGhi7yJ3gf8cavA_r_VslRcD-XxUS/s4032/20231228_143358.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="250" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh9TLRrbgyKs9_DjS557ftC38KhTffr2vpTJVQjUXqFRlMrTzz7azxTdGBdAe2r-_fjpXt_i_1-o-KVgPPFgUxZuucmuzGmrz28-ZShXfHzQD5mAYy73AZ1JtYQFpAkO6vTygereaOVrb9FmGtevdsFiDMT06cFnz0WGhi7yJ3gf8cavA_r_VslRcD-XxUS/s600/20231228_143358.jpg"/></a></div>
<div class="separator" style="float: right;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQS_gGK3Cf0ri3ycxExgWO6DaHqQc-Pyjm0Q9wrThrbik4HnFoaYJZvxsKfyqxvxaC4CWLDhgEXmg8rmz2yGKe_1GYhSbMctMOkbNTc6l0AdzL2scgq5_jp4QUL6jOa5fnQZbzLuU4LpL0JZgXsEBmFSkLlvTq5zPg7zixDO2fWIev3YLRawT6JPoteV6j/s4032/20231228_143406.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="250" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjQS_gGK3Cf0ri3ycxExgWO6DaHqQc-Pyjm0Q9wrThrbik4HnFoaYJZvxsKfyqxvxaC4CWLDhgEXmg8rmz2yGKe_1GYhSbMctMOkbNTc6l0AdzL2scgq5_jp4QUL6jOa5fnQZbzLuU4LpL0JZgXsEBmFSkLlvTq5zPg7zixDO2fWIev3YLRawT6JPoteV6j/s600/20231228_143406.jpg"/></a></div>
<div style="clear: both;"></div>
<br />
The above two pictures demonstrate how the cartridge works and how it is attached to the Commodore 64.<br />
<br />
I am very happy that I was able to run many modern demos developed by the Demoscene community, such as: <a href="https://www.pouet.net/prod.php?which=64283">Comaland by Oxyron</a> and <a href="https://www.pouet.net/prod.php?which=94504">Next level by Performers</a> etc. on a real Commodore 64 without using physical floppy disks:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihySp_BpDZk8tCXScOaVZnZSCer1k_rhI-iVjJGU31sH4NgECnIIjG_3SFDeRU-cnQ79pnzyZvvUlUJbSgdorxaTyQdpztlbm7yeNIgaALzrr1TqkqZFzn8gR7O0ELYqE2GMqXRJNAqcnANqdK-bNqRIukxEkcEqA4osWs7xpit84XFdhxp1_GDfvU3AYq/s4032/20231228_145443.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEihySp_BpDZk8tCXScOaVZnZSCer1k_rhI-iVjJGU31sH4NgECnIIjG_3SFDeRU-cnQ79pnzyZvvUlUJbSgdorxaTyQdpztlbm7yeNIgaALzrr1TqkqZFzn8gR7O0ELYqE2GMqXRJNAqcnANqdK-bNqRIukxEkcEqA4osWs7xpit84XFdhxp1_GDfvU3AYq/s600/20231228_145443.jpg"/></a></div>
<br />
<h3>ZoomFloppy</h3>
<br />
Sometimes I also want to transfer data from my PC to physical floppy disks and vice versa.<br />
<br />
For example, a couple of years ago, I wanted to make a backup of the programs that I wrote when I was young.<br />
<br />
In 2014, I have ordered a <a href="https://www.go4retro.com/products/zoomfloppy">ZoomFloppy</a> device to make this possible.<br />
<br />
ZoomFloppy is a device that offers an IEC socket to which a Commodore disk drive can be connected. As I have already explained, Commodore disk drives are self-contained computers. As a result, linking to the disk drive directly suffices.<br />
<br />
The ZoomFloppy device can be connected to a PC through the USB port:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5QyTlAs4W9mZcwYDPZcClwW_ne1wSMYXEpXdZCnAgFwJOcDFbahMS892ftZGkOGVDd4LQs_WGW1HCNtoYDYVpyoza9Gapx9ft29ophuIdeWdAdSad0A_cL2qyLuAjaobAGyzn4tK3LkNnkl6L6xEAozRt4eG5zsXzZx_lFPvCZ_Gi-xoRDIz7NMWYDTto/s4032/20231228_151223.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="3024" data-original-width="4032" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg5QyTlAs4W9mZcwYDPZcClwW_ne1wSMYXEpXdZCnAgFwJOcDFbahMS892ftZGkOGVDd4LQs_WGW1HCNtoYDYVpyoza9Gapx9ft29ophuIdeWdAdSad0A_cL2qyLuAjaobAGyzn4tK3LkNnkl6L6xEAozRt4eG5zsXzZx_lFPvCZ_Gi-xoRDIz7NMWYDTto/s600/20231228_151223.jpg"/></a></div>
<br />
The above picture shows how my 1541 disk drived is connected to my ThinkPad laptop by using the ZoomFloppy device. If you carefully look at the screen, I have requested an overview of the content of the disk that is currently in the drive.<br />
<br />
I use <a href="https://github.com/OpenCBM/OpenCBM">OpenCBM</a> on my Linux machine to carry out disk operations. Although graphical shells exist (for example, the OpenCBM project provides a graphical shell for Windows called: <i>cbm4wingui</i>), I have been using command-line instructions. They may look scary, but I learned them quite quickly.<br />
<br />
Here are some command-line instructions that I use frequently:<br />
<br />
Request the status of the disk drive:<br />
<br />
<pre>
$ cbmctrl status 8
73,speeddos 2.7 1541,00,00
</pre>
<br />
Format a floppy disk (with label: <i>mydisk</i> and id: <i>aa</i>):<br />
<br />
<pre>
$ cbmformat 8 "mydisk,aa"
</pre>
<br />
Transfer a disk image's contents (<i>mydisk.d64</i>) to the floppy disk in the drive:<br />
<br />
<pre>
$ d64copy mydisk.d64 8
</pre>
<br />
Make a D64 disk image (<i>mydisk.d64</i>) from the disk in the drive:<br />
<br />
<pre>
$ d64copy 8 mydisk.d64
</pre>
<br />
Request and display the directory contents of a floppy:<br />
<br />
<pre>
$ cbmctrl dir 8
</pre>
<br />
Transfer a file (<i>myfile</i>) from floppy disk to PC:<br />
<br />
<pre>
$ cbmcopy --read 8 myfile
</pre>
<br />
Transfer a file (<i>myfile</i>) from PC to floppy disk:<br />
<br />
<pre>
$ cbmcopy --write 8 myfile
</pre>
<br />
<h2>Conclusion</h2>
<br />
In this blog post, I have explained how I have been using my old 8-bit Commodore 64 and 128 machines in 2023. I made some repairs and I have ordered some replacement peripherals. With these new peripherals I can conveniently run software that I have downloaded from the Internet.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-49078883734439397072023-12-21T22:22:00.007+01:002024-01-22T11:28:14.030+01:00On reading research papers and maintaining knowledgeTen years ago <a href="https://sandervanderburg.blogspot.com/2013/06/dr-sander.html">I have obtained my PhD degree</a> and made <a href="https://sandervanderburg.blogspot.com/2012/10/my-post-phd-carreer-aka-leaving-academia.html">my (somewhat gradual) transition from academia to industry</a>. Despite the fact that I made this transition a long time ago, I still often get questions from people who are considering doing a PhD.<br />
<br />
Most of the discussions that I typically have with such people are about <strong>writing</strong> -- I have already explained plenty about writing in the past, including <a href="https://sandervanderburg.blogspot.com/2020/12/blog-reflection-over-last-decade.html">a recommendation to start a blog so that writing becomes a habit</a>. Having a blog allows you to break up your work into manageable pieces and build up an audience for your work.<br />
<br />
Recently, I have been elaborately reorganizing files on my hard-drive, a tedious task that I often do at the end of the year. This year, I have also been restructuring my private collection of research papers.<br />
<br />
Reading research papers became a habit while working on my master's thesis and doing my PhD. Although I have left academia a long time ago, I have retained the habit, although the amount of papers and articles that I read today is much lower than my PhD days. I no longer need to study research works much, but I have retained the habit to absorb existing knowledge and put things into context whenever I intend to do something new, for example for my blog posts or software projects.<br />
<br />
In 2020, during the first year of the COVID pandemic, I have increased my interest in research papers somewhat, because <a href="https://sandervanderburg.blogspot.com/2020/10/transforming-disnix-models-to-graphs.html">I had to revise some of the implementations of algorithms in the Dynamic Disnix framework</a> that were based on work done by other researchers. Fortunately, the ACM temporarily opened their entire <a href="http://dl.acm.org">digital library</a> to the public for free so that I could get access to quite an amount of interesting papers, without requiring to pay.<br />
<br />
In addition to writing, reading in academic research is also very important, for the following reasons:<br />
<br />
<ul>
<li>To <strong>expand</strong> your <strong>knowledge</strong> about your research domain.</li>
<li>To put your own work into <strong>context</strong>. If you want to publish a paper about your work, having a cool idea is not enough -- you have to explain what your research contributes: what the innovation is. As a result, you need to relate to earlier work and (optionally) to studies that motivate the relevance of your work. Furthermore, you can also not just take credit for work that has already been done by others. As a consequence, you need to very carefully investigate what is out there.</li>
<li>You may have to <strong>peer review</strong> papers for acceptance in conference proceedings and journals.</li>
</ul>
<br />
Reading papers is not an easy job -- it often takes me quite a bit of time and dedication to fully grasp a paper.<br />
<br />
Moreover, when studying a research paper, you may also have to dive into related work (by examining a paper's references, and the references of these) to fully get an understanding. You may have to dive several levels deep to gain enough understanding, which is not a straightforward job.<br />
<br />
In this blog post, I want to share my personal experiences with reading papers and maintaining knowledge.<br />
<br />
<h2>My personal history with reading</h2>
<br />
I have an interesting relationship with reading. Already at young age, I used to study programming books to expand my programming knowledge.<br />
<br />
With only limited education and knowledge, I was very practically minded -- I would relentlessly study books and magazines to figure out how to get things done, but as soon as I figured out enough to get something done I stopped reading, something that I consider a huge drawback of the younger version of me.<br />
<br />
For example, I still vividly remember how I used to program 2D side scroller games for the <a href="https://sandervanderburg.blogspot.com/2011/07/second-computer.html">Commodore Amiga 500 using the AMOS BASIC programming language</a>. I figured out many basic concepts by reading books, magazines and the help pages, such as how to load IFF/ILBM pictures as backgrounds, load ProTracker modules as background music, using blitter objects for moving actors, using a double buffer to smoothly draw graphics, side scrolling, responding to joystick input etc.<br />
<br />
Although I have managed to make somewhat playable games by figuring out these concepts, the games were often plagued by bugs and very slow performance. A big reason that contributed to these failures is because I stopped reading after mastering the basics.<br />
<br />
For example, to improve performance, I should have disabled the autoback feature that automatically swaps the physical and logical screens on every drawing command and do the screen swap manually after all drawing instructions were completed. I knew that using a double screen buffer would take graphics glitches away, but I never bothered to study the concepts behind it.<br />
<br />
As I grew older and entered middle school, I became more critical of myself. For example, I learned that it is essential to properly cite where you get your knowledge from rather than "pretending" that you are pulling something out of your own hat. :)<br />
<br />
Fast forwarding to my studies at the university: reading papers from the academic literature became something that I had to commonly do. For example, I still remember the real-time systems and software architecture courses.<br />
<br />
The end goal of the former course was to write your own research paper about a subject in the real-time systems domain. In this course, I learned, in addition to real-time system concepts, how academic research works: writing a research paper is not just about writing down a cool idea (with references that you used as an inspiration), but you also need to put your work into <strong>context</strong> -- a research paper is typically based on work already done by others, and your paper typically serves as an ingredient that can be picked up by other researchers.<br />
<br />
In the latter course, I had to read quite a few papers in the software architecture domain, write summaries and discuss my findings with other students. Here, I learned that reading papers is all but a trivial job:<br />
<br />
<ul>
<li>Papers are often <strong>densely written</strong>. As a result, I get overwhelmed with information and it requires quite a bit of energy from my side to consume all of it.</li>
<li>The <strong>formatting</strong> of many papers is not always helpful. Papers are typically written for print, not for reading from a screen. Also, the formatting of papers are not always good for displaying code fragments or diagrams.</li>
<li>There is often quite a bit of <strong>unexplained jargon</strong> in a paper. To get a better understanding you need to dive deeper into the literature, such as also studying the references of the papers or books that are related to the subject.</li>
<li>Sometimes authors frequently use <strong>multi-syllable words</strong>.</li>
<li>It is also not uncommon for authors to use <strong>logic and formulas</strong> to formalize concepts and mathematically prove their contributions. Although formalization helps to do this, reading formulas is often a tough job for me -- there is typically a huge load of information and Greek symbols. These symbols IMO are not always very helpful to relate to what concepts they represent.</li>
<li>Authors often tend to elaborately stress out the <strong>caveats</strong> of their contributions, making things hard to read.</li>
</ul>
<br />
Despite having read many papers in the last 16 years and I got better at it, reading still remains a tough job because of the above reasons.<br />
<br />
In the final year of my master's, I had to do a literature survey before starting the work on my master's thesis. The first time I heard about this, I felt scared, because of my past experiences with reading papers.<br />
<br />
Fortunately, my former supervisor: Eelco Visser, was very practically minded about the process -- he wanted us to first work on practical aspects of their research projects, such as <a href="https://sandervanderburg.blogspot.com/2011/05/deployment-abstractions-for-webdsl.html">WebDSL</a>: a domain-specific language for developing web applications with a rich data model and related tools, such as <a href="https://strategoxt.org/">Stratego/XT</a> and the <a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix package manager</a>.<br />
<br />
After mastering the practical concepts of these projects, doing a literature survey felt much easier -- instinctively, while using these tools in practice, I became more interested in learning about the concepts behind them. Many of their underlying concepts were described in research papers published my my colleagues in the same research department. While studying these papers, I also got more motivated/interested into diving deeper in the academic literature by studying the papers' references and searching for related subjects in the digital libraries of the ACM, IEEE, USENIX, Springer, Elsevier etc.<br />
<br />
During my PhD reading research papers became even more important. In the first six months of my PhD, I had a very good start. I published a paper about an important aspect of my master's thesis: atomic upgrading of the static parts of a distributed system, and a paper about the overall objective of the research project that I was in. I have to admit that, despite having these papers instantly accepted, I still had the wrong mindset -- I was basically just "selling my cool ideas" and finding support in the academic literature, rather than critically studying what is out there.<br />
<br />
For my third paper, that covers a new implementation of <a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a> (<a href="https://sandervanderburg.blogspot.com/2016/01/disnix-05-release-announcement-and-some.html">the third major revision to be precise</a>), I learned an important/hard lesson. The first version of the paper got badly rejected by the program committee, because of my "advertising cool ideas mindset" that I always used to have -- I failed to study the academic literature well enough to explain what the innovation of my paper is in comparison to other deployment solutions. As a consequence, I got some very hard criticisms from the reviewers.<br />
<br />
Fortunately, they gave me good feedback. For example, I had to study papers from the Working Conference on Component Deployment. I have addressed their criticisms and the revised paper got accepted. I learned what I had to do in the future -- it is a requirement to also study the academic literature well enough to explain what your contribution is and demonstrate its relevance.<br />
<br />
This rejection also changed my attitude how I deal with research papers. Previously, after my work for a paper was done, I would typically discard the artifacts that I no longer needed, including the papers that I used as a reference. After this rejection, I learned that I need to build my own personal knowledge base so that for future work, I could always relate to the things that I have read previously.<br />
<br />
<h2>Reading research papers</h2>
<br />
I have already explained that for various reasons, reading research papers is all but an easy job. For some papers, in particular the ones in my former research domain: <a href="https://sandervanderburg.blogspot.com/2011/10/software-deployment-complexity.html">software deployment</a>, I got better as I grew more familiar to the research domain.<br />
<br />
Nonetheless, I still sometimes find reading papers challenging. For example, studying algorithmic papers is extremely hard IMO. In 2021, I had to revise my implementations of approximation solutions for the multi-way cut, and graph coloring problems in the Dynamic Disnix framework. I had to re-read the corresponding papers again. Because they were so hard to grasp, <a href="https://sandervanderburg.blogspot.com/2020/10/transforming-disnix-models-to-graphs.html">I wrote a blog post that explains how I practically applied them</a>.<br />
<br />
To fully grasp a paper, reading it a single time is often not enough. In particular the algorithmic papers that I mentioned earlier, I had to read them many times.<br />
<br />
Interestingly enough, I learned that reading papers is also a subject of study. A couple of years ago I discovered a paper titled: "<a href="http://ccr.sigcomm.org/online/files/p83-keshavA.pdf">How to Read a Paper</a>" that explains a strategy for reading research papers using a three-pass approach:<br />
<br />
<ul>
<li>First pass: bird's eye view. Study the title, abstract, introduction, headings, conclusions. A single pass is often already enough to decide whether a paper is relevant to read or not.</li>
<li>Second pass: study in greater detail, but ignore the big details, such as mathematical proofs.</li>
<li>Third pass: read everything in detail by attempting to virtually re-implement the paper.</li>
</ul>
<br />
After discovering this paper, I have also been using the three pass approach. I have studied most of my papers in my collection in two passes, and some of them in detail in three passes.<br />
<br />
Another thing that I discovered by accident is that to extensively study literature, a <strong>continuous approach</strong> works better for me (e.g. reserving certain timeslots in a week) than just reserving longer periods of time that consist of only reading papers.<br />
<br />
Also, regularly discussing papers with your colleagues helps. During my PhD days, I did not do it that often (we had no formal "process" for it) but there were several good sessions, such as a program committee simulation organized by Arie van Deursen, head of our research group.<br />
<br />
In this simulation, we organized a program committee meeting of the ICSE conference in which the members of the department represented program committee members. We have discussed submitted papers and voted for acceptance or rejection. Moreover, we also had to leave the room if there was a conflict of interest.<br />
<br />
I also learned that Edsger Dijkstra, a famous Dutch computer scientist, organized the ETAC (Eindhoven Tuesday Afternoon Club) and <a href="https://www.cs.utexas.edu/~EWD/ewd09xx/EWD927.PDF">ATAC (Austin Tuesday Afternoon Club)</a> in which amongst other activities, reading and discussing research papers was a recurring activity.<br />
<br />
<h2>Building up your personal knowledge base</h2>
<br />
As I have explained earlier, I used to throw away my downloaded papers when the work for a paper was done, but I changed that habit after that hard paper rejection.<br />
<br />
There are many good reasons to keep and organize the papers that you have read, even if they do not seem to be directly relevant to your work:<br />
<br />
<ul>
<li>As I have already explained, in addition to reading a single paper and writing your own research papers, you need to maintain your knowledge base so that you can put them into context.</li>
<li>It is not always easy to obtain papers. Many of them are behind a paywall. Without a subscription you cannot access them, so once you have obtained them it is better to think twice before you discard them. Fortunately, open access becomes more common but it still remains a challenge. Arie van Deursen has written a variety of blog posts about <a href="https://avandeursen.com/tag/open-access/">open access</a>.</li>
<li>Although many papers are challenging to read, I also started to appreciate certain research papers.</li>
</ul>
<br />
My own personal paper collection has evolved in an interesting way. In the beginning, I just used to put any paper that I have obtained into a single folder called: <i>papers</i> until it grew large enough that I had to start classifying them.<br />
<br />
Initially, there was a one-level folder structure, consisting of categories such as: deployment, operating systems, programming languages, DSL engineering etc. At some point, the content of some of these folders grew large enough and I introduced a second level directory structure.<br />
<br />
For example, the sub folder for my former research domain: software deployment (the process that consists of all activities to make a software system available for use) contains the largest amount of papers. Currently, I have collected 168 deployment papers that I have divided over the following sub categories:<br />
<br />
<ul>
<li><strong>Deployment models</strong>. Papers whose main contribution is a means to model various deployment aspects of a system, such as the structure of a system and deployment activities.</li>
<li><strong>Deployment planning</strong>. Papers whose main contribution are algorithms that decide a suitable/optimal deployment architecture based on functional and non-functional requirements of a system.</li>
<li><strong>Empirical studies</strong>. Papers containing empirical studies about deployment in practice.</li>
<li><strong>Execution</strong>. Papers in which the main contribution is executing deployment activities. I have also sub categorized this folder into technology-specific solutions (e.g. a solution is specific to a programming language, such as Java or component technology, such as CORBA) and generic solutions.</li>
<li><strong>Practice reports</strong>. Papers that report on the use of deployment technologies in practice.</li>
<li><strong>Surveys</strong>. Papers that analyse the literature and draw conclusions from them.</li>
</ul>
<br />
A hierarchical directory structure is not perfect for organizing papers -- for many papers there is an overlap between multiple sub domains in the software engineering domain. For example, deployment may also be related to a certain component technology, in service of optimizing the architecture of a system, related to other configuration management activities (versioning, status accounting, monitoring etc.) or an ingredient in integration testing. If there is an overlap, I typically look at the strongest kind of contribution that the paper makes.<br />
<br />
For example, in the deployment domain, Eelco Dolstra wrote a paper about <a href="https://research.tudelft.nl/en/publications/maximal-laziness-an-efficient-interpretation-technique-for-purely">maximal laziness</a>, an important implementation aspect of the Nix expression language. The Nix package manager is a deployment solution, but the contribution of the paper is not deployment, but making the implementation of a purely functional DSL efficient. As a result, I have categorized the paper under DSL engineering rather than deployment.<br />
<br />
The organization of my paper collection is always in motion. Sometimes I gain new insights causing me to adjust the classifications, or when a collection of papers for a sub domain grows, I may introduce a second-level classification.<br />
<br />
<h2>Some practical tips to get familiar with a certain research subject</h2>
<br />
So what is my recommended way to get familiar with a certain research subject in the software engineering domain?<br />
<br />
I would start by doing something practical first. <a href="https://sandervanderburg.blogspot.com/2012/01/engineering-versus-science.html">In software engineering research domain, often the goal is to develop or examine tools</a>. Start by using these tools first and see if you can contribute to them from a practical point of view -- for example, by improving features, fixing bugs etc.<br />
<br />
As soon as I have mastered the practical aspects, I may typically already get motivated to dive into their underlying concepts by studying the papers that cover them. Then I will apply the three pass reading strategy and eventually study the references of the papers to get a better understanding.<br />
<br />
After my orientation phase has finished, the next thing I would typically look at is the conferences/venues that are directly related to the subject. For software deployment, for example, there used to be only one subject-related conference: the Working Conference On Component Deployment (that unfortunately was no longer organized after 2005). It is typically a good thing to have examined all the papers of the related conferences/venues, by at least using a first-pass approach.<br />
<br />
Then a potential next step is to search for "early defining papers" in that research area. In my experience, many research papers are improving on concepts pioneered by these papers, so it is IMO a good thing to know where it all started.<br />
<br />
For example, in the software deployment domain the paper: "<a href="https://ics.uci.edu/~andre/papers/T3.pdf">A Characterization Framework for Software Deployment Technologies</a>" is such an early defining paper, covering a deployment solution called "The Software Dock". The paper comes with a definition for the term: "software deployment" that is considered the canonical definition in academic research.<br />
<br />
Alternatively, the paper: "<a href="https://dl.acm.org/doi/10.1109/FOSE.2007.20">Software Deployment, Past, Present and Future</a>" is a more recent yet defining paper covering newer deployment technologies and also offers its own definition of the term software deployment.<br />
<br />
For unknown reasons, I always seem to like early defining papers in various software engineering domains. These are some of my recommendations of early defining papers in other software engineering domains:<br />
<br />
<ul>
<li>General purpose programming languages: <a href="https://dl.acm.org/doi/10.1145/320764.320766">The IBM 701 Speedcoding System</a>, <a href="https://dl.acm.org/doi/10.1145/1455567.1455599">The FORTRAN Automatic Coding System</a>, <a href="https://dl.acm.org/doi/10.1145/367177.367199">Recursive Functions of Symbolic Expressions and Their Computation by Machine, Part I</a>, <a href="https://academic.oup.com/comjnl/article/19/4/364/326695">Modified Report on the Algorithmic Language ALGOL 60</a>.</li>
<li>Domain-specific languages: <a href="https://dl.acm.org/doi/10.1145/6424.315691">Programming Pearls: Little Languages</a>, <a href="https://dl.acm.org/doi/10.1145/352029.352035">Domain-Specific Languages: An Annotated Bibliography</a></li>
<li>Programming: <a href="https://www.cs.utexas.edu/~EWD/ewd02xx/EWD215.PDF">A case against the GO TO statement</a>, <a href="https://www.cs.utexas.edu/~EWD/ewd02xx/EWD249.PDF">Notes on Structured Programming</a></li>
<li>Operating systems: <a href="https://dl.acm.org/doi/10.1145/357980.357999">The Structure of the "THE"-multiprogramming System</a>, <a href="https://dl.acm.org/doi/10.1145/800212.806497">The Multics Input/Output System</a>, <a href="https://dl.acm.org/doi/10.1145/1463891.1463915">A General-Purpose File System for Secondary Storage</a>, <a href="https://dl.acm.org/doi/10.1145/357980.358014">The UNIX Time-Sharing System</a></li>
<li>Software engineering in general: <a href="https://dl.acm.org/doi/10.5555/800253.807694">Research paradigms in computer science</a></li>
</ul>
<br />
After studying all these kinds of papers, your knowledge level should already be decent enough to find your way to study the remaining papers that are out there.<br />
<br />
<h2>Literature surveys</h2>
<br />
In addition to research papers that need to put themselves into context, extensive literature surveys can also be quite valuable to the research community. During my PhD, I learned that it is also possible to publish a paper about a literature survey.<br />
<br />
For example, some of my former colleagues did an <a href="https://repository.tudelft.nl/islandora/object/uuid%3Afd04993a-2d0c-4eaa-920d-e135e6a3645d?collection=research">extensive and systematic literature survey in the dynamic analysis domain</a>. In addition to the results, the authors also explain their methodology, that consists of searching on keywords, looking for appropriate conferences and journals and following the papers' references. From these results they have derived an attribute framework and classified all the papers into this attribute framework.<br />
<br />
I have kept the paper as a reference for myself, because I like the methodology. I am not so interested in dynamic analysis or program comprehension from a research perspective.<br />
<br />
Literature surveys also exist in my former research domain, such as a <a href="https://oatao.univ-toulouse.fr/14816/1/Arcangeli_14816.pdf">survey of deployment solutions for distributed systems</a>.<br />
<br />
<h2>Conclusions</h2>
<br />
In this blog post, I have shared my experiences with reading papers and maintaining knowledge. In research, it is quite important and you need to take it seriously.<br />
<br />
Fortunately, during my PhD I have learned a lot. In summary, my recommendations are:<br />
<br />
<ul>
<li>Archive your papers and build up a personal knowledge base</li>
<li>Start with something practical</li>
<li>Follow paper references</li>
<li>Study early defining papers</li>
<li>Find people to discuss with</li>
<li>Study continuously in small steps</li>
</ul>
<br />
Although I never did an extensive literature survey in the software deployment domain (it is not needed for submitting papers that contribute new techniques) I can probably even write a paper about software deployment literature myself. The only problem is that I am not quite up to date with work that has been published in the last few years, because I no longer have access to these digital libraries.<br />
<br />
Moreover, I also need to find the time and energy to do it, if I really want to :)<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-61723485039039473082023-07-04T22:09:00.000+02:002023-07-04T22:09:57.291+02:00Using a site map for generating dynamic menus in web applicationsIn the last few weeks, I have been playing around with a couple of old and obsolete web applications that I have developed in the past with <a href="https://sandervanderburg.blogspot.com/2017/07/some-reflections-on-my-experiences-with.html">my own web framework</a>. Much of the functionality that these custom web applications offer are facilitated by my framework, but sometimes these web applications also contain significant chunks custom code.<br />
<br />
One of the more interesting features provided by custom code is <strong>folding menus</strong> (also known as dropdown and dropright menus etc.), that provide a similar experience to the <a href="https://en.wikipedia.org/wiki/Start_menu">Windows start menu</a>. My guess is that because the start menu experience is so familiar to many users, it remains a frequently used feature by many web applications as of today.<br />
<br />
When I still used to actively develop my web framework and many custom web applications (over ten years ago), implementing such a feature heavily relied on JavaScript code. For example, I used the <i>onmouseover</i> attribute on a hyperlink to invoke a JavaScript function that unfolds a panel and the <i>onmouseout</i> attribute to fold a panel again. The <i>onmouseover</i> event handler injects a menu section into the DOM using CSS absolute positioning to put it in the right position on the screen.<br />
<br />
I could not use the standard menu rendering functionality of my <a href="https://sandervanderburg.blogspot.com/2014/03/implementing-consistent-layouts-for.html">layout framework</a>, because it deliberately does not rely on the usage of JavaScript. As a consequence, I had to write a custom menu renderer for a web application that requires dynamic menu functionality.<br />
<br />
Despite the fact that folding menus are popular and I have implemented them as custom code, I never made it a feature of my layout framework for the following two reasons:<br />
<br />
<ul>
<li>I want web applications built around my framework to be as <a href="https://sandervanderburg.blogspot.com/2016/03/the-nixos-project-and-deploying-systems.html"><strong>declarative</strong></a> as possible -- this means that I want to concisely express as much as possible <strong>what</strong> I want to render (a paragraph, an image, a button etc. -- this is something HTML mostly does), rather than specifying in detail <strong>how</strong> to do it (in JavaScript code). As a result, the usage of JavaScript code in my framework is minimized and non-essential.<br />
<br />
All functionality of the web applications that I developed with my framework must be accessible without JavaScript as much possible.</li>
<li>Another property that I appreciate of web technology is the ability to <strong>degrade gracefully</strong>: the most basic and primary purpose of web applications is to provide information as <strong>text</strong>.<br />
<br />
Because this property is so important, many non-textual elements, such as an image (<i>img</i> element), provide fallbacks (such as an <i>alt</i> attribute) that simply renders alternative text when graphics capabilities are absent. As a result, it is possible to use more primitive browsers (such as text-oriented browsers) or alternative applications to consume information, such as a text-to-speech system.<br />
<br />
When essential functionality is only exposed as JavaScript code (which more primitive browsers cannot interpret), this property is lost.</li>
</ul>
<br />
Recently, I have discovered that there is a way to implement folding menus that does not rely on the usage of JavaScript.<br />
<br />
Moreover, there is also another kind of dynamic menu that has become universally accepted -- the <strong>mobile navigation</strong> menu (or <a href="https://en.wikipedia.org/wiki/Hamburger_button">hamburger menu</a>) making navigation convenient on smaller screens, such as mobile devices.<br />
<br />
Because these two types of dynamic menus have become so common, I want to facilitate the implementation of such dynamic menus in my layout framework.<br />
<br />
I have found an interesting way to make such use cases possible while retaining the ability to render text and degrade gracefully -- we can use an HTML representation of a <strong>site map</strong> consisting of a root hyperlink and a nested unordered list as a basis ingredient.<br />
<br />
In this blog post, I will explain how implementing these use cases are possible.<br />
<br />
<h2>The site map feature</h2>
<br />
As already explained, the basis for implementing these dynamic menus is a textual representation of a site map. Generating site maps is a feature that is already supported by the layout framework:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNZKAFc_QGgTEKlB3BQIi1FdUDihIsAgad636lhdGTrBYDqtRn51oMWtU21RYZ_c28zDqO7PBOscVBlRaYEBxD8fKHh86m18n6f0XKLoeQD7r9JO5BbLX_hKsmlo-IjWiX2EbkalKd8mB0CzawwMQqRmbWgi-ToRn4_vuF1Y4PH-Y_yBIaEDoXlWUCgWHp/s1578/sitemap-feature.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1256" data-original-width="1578" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjNZKAFc_QGgTEKlB3BQIi1FdUDihIsAgad636lhdGTrBYDqtRn51oMWtU21RYZ_c28zDqO7PBOscVBlRaYEBxD8fKHh86m18n6f0XKLoeQD7r9JO5BbLX_hKsmlo-IjWiX2EbkalKd8mB0CzawwMQqRmbWgi-ToRn4_vuF1Y4PH-Y_yBIaEDoXlWUCgWHp/s600/sitemap-feature.png"/></a></div>
<br />
The above screenshot shows an example page that renders a site map of the entire example web application. In HTML, the site map portion has the following structure:<br />
<br />
<pre style="font-size: 90%; overflow-x: auto;">
<a href="/examples/simple/index.php">Home</a>
<ul>
<li>
<a href="/examples/simple/index.php/home">Home</a>
</li>
<li>
<a href="/examples/simple/index.php/page1">Page 1</a>
<ul>
<li>
<a href="/examples/simple/index.php/page1/page11">Subpage 1.1</a>
</li>
<li>
<a href="/examples/simple/index.php/page1/page12">Subpage 1.2</a>
</li>
</ul>
</li>
<li>
<a href="/examples/simple/index.php/page2">Page 2</a>
...
</li>
...
</ul>
</pre>
<br />
The site map, shown in the screenshot and code fragment above, consists of three kinds of links:<br />
<br />
<ul>
<li>On top, the <strong>root link</strong> is displayed that brings the user to the entry page of the web application.</li>
<li>The <strong>unordered list</strong> displays links to all the pages visible in the main menu section that are reachable from the entry page.</li>
<li>The <strong>nested unordered list</strong> displays links to all the pages visible in the sub menu section that are reachable from the selected sub page in the main menu.</li>
</ul>
<br />
With a few simple modifications to my layout framework, I can use a site map as an alternative type of menu section:<br />
<br />
<ul>
<li>I have extended the site map generator with the ability to mark selected sub pages and as <strong>active</strong>, similar to links in menu sections. By adding the <i>active</i> CSS class as an attribute to a hyperlink, a link gets marked as active.</li>
<li>I have introduced a <i>SiteMapSection</i> to the layout framework that can be used as a replacement for a <i>MenuSection</i>. A <i>MenuSection</i> displays reachable pages as hyperlinks from a selected page on one level in the page hierarchy, whereas a <i>SiteMap</i> section renders the selected page as a root link and all its visible sub pages and transitive sub pages.</li>
</ul>
<br />
With the following model of a layout:<br />
<br />
<pre>
$application = new Application(
/* Title */
"Site map menu website",
/* CSS stylesheets */
array("default.css"),
/* Sections */
array(
"header" => new StaticSection("header.php"),
"menu" => new SiteMapSection(0),
"contents" => new ContentsSection(true)
),
...
);
</pre>
<br />
We may render an application with pages that have the following look:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQFTXCTfxjBXfiDTT0Nlol8WYSRN8l3eEpPEvAHVNWwuVR2TZAM_M5EmI-sPp3Bby3BoEuCPzDkrWPOZUNnQ8adlJzQkQr-OQdljIcnHuKAXCnONIuNxE6m5K7jxrM0zgNfzHWCaJvi3hvPB4lXnuxWfu1ttwaRL2vQ2kbuH5N9wpj8idLIyWVXzF3YNZu/s1578/rawsitemapsection.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1257" data-original-width="1578" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQFTXCTfxjBXfiDTT0Nlol8WYSRN8l3eEpPEvAHVNWwuVR2TZAM_M5EmI-sPp3Bby3BoEuCPzDkrWPOZUNnQ8adlJzQkQr-OQdljIcnHuKAXCnONIuNxE6m5K7jxrM0zgNfzHWCaJvi3hvPB4lXnuxWfu1ttwaRL2vQ2kbuH5N9wpj8idLIyWVXzF3YNZu/s600/rawsitemapsection.png"/></a></div>
<br />
As can be seen in the above screenshot and code fragment, the application layout defines three kinds of sections: a <strong>header</strong> (a static section displaying a logo), a <strong>menu</strong> (displaying links to sub pages) and a <strong>contents</strong> section that displays the content based on the sub page that was selected by the user (in the menu or by opening a URL).<br />
<br />
The menu section is displayed as a site map. This site map will be used as the basis for the implementation of the dynamic menus that I have described earlier in this blog post.<br />
<br />
<h2>Implementing a folding menu</h2>
<br />
Turning a site map into a folding menu, by using only HTML and CSS, is a relatively straight forward process. To explain the concepts, I can use the following trivial HTML page as a template:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix75daVI1_Tboxq5VvO5GStv-SPGDnUPwkQGPifLDxT0wEmHh2CE0UU8QBcnsUSi41OL_HopZfRkwoIgEXRUhjO9RPaVYPuRkftw1ZdCJBRfl1nDSDs6_wLLc8C5IY16ZLzqQxplbhXZ7qEBwk7tUTSm9TK1XglyXtDcFt22YVrsJTPmbs3sSE5PwYDldD/s715/foldingmenu-step1.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="695" data-original-width="715" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEix75daVI1_Tboxq5VvO5GStv-SPGDnUPwkQGPifLDxT0wEmHh2CE0UU8QBcnsUSi41OL_HopZfRkwoIgEXRUhjO9RPaVYPuRkftw1ZdCJBRfl1nDSDs6_wLLc8C5IY16ZLzqQxplbhXZ7qEBwk7tUTSm9TK1XglyXtDcFt22YVrsJTPmbs3sSE5PwYDldD/s600/foldingmenu-step1.png"/></a></div>
<br />
The above page only contains a root link and nested unordered list representing a site map.<br />
<br />
In CSS, we can hide the root link and the nested unordered lists by default with the following rules:<br />
<br />
<pre>
/* This rule hides the root link */
body > a
{
display: none;
}
/* This rule hides nested unordered lists */
ul li ul
{
display: none;
}
</pre>
<br />
resulting in the following page:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6uhCgFYtJ6D_jtgvsHsCLcrK6q-Egz29xMgGToeRkjZADzQZZAW9VBDzVqrxz49hVvtKaJVyWMFul1JfSOmu_7GUCnWhR2R6bTAzo9Y0pczxCj4t-GlEDS4_zArOKwvHns8dlfJnkYubXfsa-sznwGq9PpnMze3OU-1kcIwTuUp7oyIpzce91pNy1lG2H/s715/foldingmenu-step2.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="695" data-original-width="715" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi6uhCgFYtJ6D_jtgvsHsCLcrK6q-Egz29xMgGToeRkjZADzQZZAW9VBDzVqrxz49hVvtKaJVyWMFul1JfSOmu_7GUCnWhR2R6bTAzo9Y0pczxCj4t-GlEDS4_zArOKwvHns8dlfJnkYubXfsa-sznwGq9PpnMze3OU-1kcIwTuUp7oyIpzce91pNy1lG2H/s600/foldingmenu-step2.png"/></a></div>
<br />
With the following rule, we can make a nested unordered list visible when a user hovers over the surrounding list item:<br />
<br />
<pre>
ul li:hover ul
{
display: block;
}
</pre>
<br />
Resulting in a web page that behaves as follows:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEWaXVGMysc9UGs7NZLLO4TejNqYP1uxfiAMuVWm9cSvP1EnbRH6MTNBa4mZozxA-fEyHR5klrW0BqFRV4oXxy6Of0EKB2umkWvJEs0XDfVuztVh6OQ-OGRzJGhmWsKkX-2c-mbVYfRAKTHZR7pbTE69_2mJz3GnPXXblCr6b1758_fRQUnQ20Uap5QO0W/s715/foldingmenu-step3.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="695" data-original-width="715" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiEWaXVGMysc9UGs7NZLLO4TejNqYP1uxfiAMuVWm9cSvP1EnbRH6MTNBa4mZozxA-fEyHR5klrW0BqFRV4oXxy6Of0EKB2umkWvJEs0XDfVuztVh6OQ-OGRzJGhmWsKkX-2c-mbVYfRAKTHZR7pbTE69_2mJz3GnPXXblCr6b1758_fRQUnQ20Uap5QO0W/s600/foldingmenu-step3.png"/></a></div>
<br />
As can be seen, the unordered list that is placed under the <i>Page 2</i> link became visible because the user hovers over the surrounding list item.<br />
<br />
I can make the menu a bit more fancy if I want to. For example, I can remove the bullet points with the following CSS rule:<br />
<br />
<pre>
ul
{
list-style-type: none;
margin: 0;
padding: 0;
}
</pre>
<br />
I can add borders around the list items to make them appear as buttons:<br />
<br />
<pre>
ul li
{
border-style: solid;
border-width: 1px;
padding: 0.5em;
}
</pre>
<br />
I can horizontally align the buttons by adopting a <a href="https://css-tricks.com/snippets/css/a-guide-to-flexbox">flexbox layout</a> using the row direction property:<br />
<br />
<pre>
ul
{
display: flex;
flex-direction: row;
}
</pre>
<br />
I can position the sub menus right under the buttons of the main menu by using a combination of relative and absolute positioning:<br />
<br />
<pre>
ul li
{
position: relative;
}
ul li ul
{
position: absolute;
top: 2.5em;
left: 0;
}
</pre>
<br />
Resulting in a menu with the following behaviour:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2kP0V6rhixj1j_cWNaiJ5PRUx9-u0X6PYqY-_aovTyCBljiw6GcBnUkSgI8vh-oHeGVki5u-6LQvhOv9DWr0cSu0u0alUxlVYmxqlU5Op2oIhashM6-uZe33OOxvnkCOh1wnfaiV_B-zXL6YZkXS-WmCCmXKd6_16FQjuWemJ0KHTaa_DinH8RvVVFHWm/s715/foldingmenu-step4.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="695" data-original-width="715" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi2kP0V6rhixj1j_cWNaiJ5PRUx9-u0X6PYqY-_aovTyCBljiw6GcBnUkSgI8vh-oHeGVki5u-6LQvhOv9DWr0cSu0u0alUxlVYmxqlU5Op2oIhashM6-uZe33OOxvnkCOh1wnfaiV_B-zXL6YZkXS-WmCCmXKd6_16FQjuWemJ0KHTaa_DinH8RvVVFHWm/s600/foldingmenu-step4.png"/></a></div>
<br />
As can be seen, the trivial example application provides a usable folding menu thanks to the CSS rules that I have described.<br />
<br />
In my example application bundled with the layout framework, I have applied all the rules shown above and combined them with the already existing CSS rules, resulting in a web application that behaves as follows:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwDE0ipameFT8X7LqagQfvX0gV6IFaRyG51B8ZLJGLfEFWUzZawfKRJo5pf3RGl5A1Yq99DmpgLxHP5muRgutikrhWSmvM6bD8gDKU1KBti5NSD1Ouaw3YpiW4N47y0oWR15cE9OQwxw6wSgixNk65fc-wKcdYkvioSn5R1RT4zgoW_r00BxgJfdmljdWM/s1578/sitemapapp-unfold.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1257" data-original-width="1578" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhwDE0ipameFT8X7LqagQfvX0gV6IFaRyG51B8ZLJGLfEFWUzZawfKRJo5pf3RGl5A1Yq99DmpgLxHP5muRgutikrhWSmvM6bD8gDKU1KBti5NSD1Ouaw3YpiW4N47y0oWR15cE9OQwxw6wSgixNk65fc-wKcdYkvioSn5R1RT4zgoW_r00BxgJfdmljdWM/s600/sitemapapp-unfold.png"/></a></div>
<br />
<h2>Displaying a mobile navigation menu</h2>
<br />
As explained in the introduction, another type of dynamic menu that has been universally accepted is the mobile navigation menu (also known as a hamburger menu). Implementing such a menu, despite its popularity, is challenging IMHO.<br />
<br />
Although there seem to be ways to implement such a menu without JavaScript (such as this <a href="https://blog.logrocket.com/create-responsive-mobile-menu-with-css-no-javascript/">example using a checkbox</a>) the only proper way to do it IMO is still to use JavaScript. Some browsers have trouble accepting such HTML+CSS-only implementations and it requires the use of an HTML element (an <i>input</i> element) that is not designed for that purpose.<br />
<br />
In my example web application, I have implemented a custom JavaScript module, that dynamically transforms a site map (that may have already been displayed as a folding menu) into a mobile navigation menu by performing the following steps:<br />
<br />
<ul>
<li>We query the root link of the site map and transform it into a mobile navigation menu button by replacing the text of the root link by an icon image. Clicking on the menu button makes the navigation menu visible or invisible.</li>
<li>The first level sub menu becomes visible by adding the CSS class: <i>navmenu_active</i> to the unordered list.</li>
<li>The menu button becomes active by adding the CSS class: <i>navmenu_icon_active</i> to the image of the root link.</li>
<li>Nested menus can be unfolded or folded. The JavaScript code adds fold icons to each list item of the unordered lists that embed a nested unordered list.</li>
<li>Clicking on the fold icon makes the nested unordered list visible or invisible.</li>
<li>A nested unordered list becomes visible by adding the CSS class: <i>navsubmenu_active</i> to the unordered list</li>
<li>A fold button becomes active by adding the CSS class: <i>navmenu_unfold_active</i> to the fold icon image</li>
</ul>
<br />
It was quite a challenge to implement this JavaScript module, but it does the trick. Moreover, the basis remains a simple HTML-rendered site map that can still be used in text-oriented browsers.<br />
<br />
The result of using this JavaScript module is the following navigation menu that has unfoldable sub menus:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3tKiqK2tnVJO9c7aEieIr1IL9P2Or1yahjHQILTDm1UJI_72BgBQpy2O3dzk7jj9HEOOWyOIq1PNk4PekRjPKOHgR0h7cU11Eta_HLE7YgigQhDGO56f-UnBYXD3ip3eOx3FQQZPIOKqiXRRyiyEvnTd2PqhqyQRDaME-URICiXRNkR-F-1Yuci2ETEVi/s1097/sitemapapp-navmenu.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="600" data-original-height="1097" data-original-width="715" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3tKiqK2tnVJO9c7aEieIr1IL9P2Or1yahjHQILTDm1UJI_72BgBQpy2O3dzk7jj9HEOOWyOIq1PNk4PekRjPKOHgR0h7cU11Eta_HLE7YgigQhDGO56f-UnBYXD3ip3eOx3FQQZPIOKqiXRRyiyEvnTd2PqhqyQRDaME-URICiXRNkR-F-1Yuci2ETEVi/s600/sitemapapp-navmenu.png"/></a></div>
<br />
<h2>Concluding remarks</h2>
<br />
In this blog post, I have explained a new feature addition to my layout framework: the <i>SiteMapSection</i> that can be used to render menu sections as site maps. Site maps can be used as a basis to implement dynamic menus, such as folding menus and mobile navigation menus.<br />
<br />
The benefit of using a site map as a basis ingredient is that a web page still remains useful in its most primitive form: text. As a result, I retain two important requirements of my web framework: declarativity (because a nested unordered list describes concisely what I want) and the ability to degrade gracefully (because it stays useful when it is rendered as text).<br />
<br />
Developing folding/navigation menus in the way I described is not something new. There are plenty of examples on the web that show how such features can be developed, such as these W3Schools <a href="https://www.w3schools.com/howto/howto_css_dropdown.asp">dropdown menu</a> and <a href="https://www.w3schools.com/howto/howto_js_mobile_navbar.asp">mobile navigation menu</a> examples.<br />
<br />
Compared to many existing solutions, my approach is somewhat puristic -- I do not abuse HTML elements (such as a check box), I do not rely on using helper elements (such as <i>div</i>s and <i>span</i>s) or helper CSS classes/ids. The only exception is to support dynamic features that are not part of HTML, such as "active links" and the folding/unfolding buttons of the mobile navigation menu.<br />
<br />
Although it has become possible to use my framework to implement mobile navigation menus, I still find it sad that I have to rely on JavaScript code to do it properly.<br />
<br />
Folding menus, despite their popularity, are nice but the basic one-level menus (that only display a collection of links/buttons of sub pages) are in my opinion fine too and much simpler -- the same implementation is usable on desktops, mobile devices and text-oriented browsers.<br />
<br />
With folding menus, I have to test multiple resolutions and devices to check whether they provide the right user experience. Folding menus are useless on mobile devices --- you cannot separately trigger a hover event without generating a click event, making it impossible to unfold a sub menu and peek what is inside.<br />
<br />
When it is also desired to provide an optimal mobile device experience, you also need to implement an alternative menu. This requirement makes the implementation of a web application significantly more complex.<br />
<br />
<h2>Availability</h2>
<br />
The <i>SiteMapSection</i> has become a new feature of the Java, PHP and JavaScript implementations of my layout framework and can be obtained from <a href="https://github.com/svanderburg">my GitHub page</a>.<br />
<br />
In addition, I have added a <i>sitemapmenu</i> example web application that displays a site map section in multiple ways:<br />
<br />
<ul>
<li>In text mode, it is just displayed as a (textual) site map</li>
<li>In graphics mode, when the screen width is 1024 pixels or greater, it displays a horizontal folding menu.</li>
<li>In graphics mode, when the screen width is smaller than 1024 pixels and JavaScript is disabled, it displays a vertical folding menu.</li>
<li>In graphics mode, when the screen width is smaller than 1024 pixels and JavaScript is enabled, it displays a mobile navigation menu.</li>
</ul>
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-91141083014531663742022-12-30T20:37:00.000+01:002022-12-30T20:37:37.824+01:00Blog reflection over 2022Today, it is my blog's anniversary. As usual, this is a nice opportunity to reflect over the last year.<br />
<br />
<h2>Eelco Visser</h2>
<br />
The most shocking event of this year is the unfortunate passing of my former PhD supervisor: <a href="https://eelcovisser.org">Eelco Visser</a>. I still find it hard to believe that he is gone.<br />
<br />
Although I left the university for quite some time now, the things I learned while I was employed at the university (such as having all these nice technical discussions with him) still have a profound impact on me today. Moreover, without his suggestion this blog would probably not exist.<br />
<br />
Because the original purpose of my blog was to augment my research with extra details and practical information, <a href="https://sandervanderburg.blogspot.com/2022/04/in-memoriam-eelco-visser-1966-2022.html">I wrote a blog post with some personal anecdotes about him</a>.<br />
<br />
<h2>COVID-19 pandemic</h2>
<br />
In <a href="https://sandervanderburg.blogspot.com/2021/12/11th-annual-blog-reflection.html">my previous blog reflection</a>, I have explained that we were in the <a href="https://www.rivm.nl/nieuws/omikron-subvariant-BA.2-dominant-daling-ziekenhuisopnames">third-wave of the COVID pandemic caused by the even more contagious Omicron variant of the COVID-19 virus</a>. Fortunately, it turned out that, despite being more contagious, this variant is less hostile than the previous Delta variant.<br />
<br />
Several weeks later, the situation got under control and things were opened up again. The situation remained pretty stable afterwards. This year, it was possible for me to travel again and to go to physical concerts, which feels a bit weird after staying home for two whole years.<br />
<br />
The COVID-19 virus is not gone, but the situation is under control in Western Europe and the United States. There have not been any lockdowns or serious capacity problems in the hospitals.<br />
<br />
When the COVID-19 pandemic started, my employer: Mendix adopted a work-from-home-first culture. By default, people work from home and if they need to go to the office (for example, to collaborate) they need to make a desk reservation.<br />
<br />
As of today, I am still working from home most of my time. I typically visit the office only once a week, and I use that time to collaborate with people. In the remaining days, I focus myself on development work as much as possible.<br />
<br />
I have to admit that I like the quietness at home -- not everything can be done at home, but for programming tasks I need to think, and for thinking I need silence. Before the COVID-19 pandemic started, the office was typically very noisy making it sometimes difficult for me to focus.<br />
<br />
<h2>Learning modern JavaScript features</h2>
<br />
I used to intensively work with JavaScript at my previous employer: Conference Compass, but since I joined Mendix I am mostly using different kinds of technologies. During my CC days, I was still mostly writing old fashioned (ES5) JavaScript code, and I still wanted to familiarise myself with modern ES6 features.<br />
<br />
One of the challenging aspects of using JavaScript is asynchronous programming -- making sure that the main thread of your JavaScript application never blocks too long (so that it can handle multiple connections or input events) and keeping your code structured.<br />
<br />
With old fashioned ES5 JavaScript code, I had to rely on software abstractions to keep my code structured, but with the addition of Promises/A+ and the async/await concepts to the core of the JavaScript language, this can be done in a much cleaner way without using any custom software abstractions.<br />
<br />
In 2014, I wrote a blog post about the problematic synchronous programming concepts in JavaScript and their equivalent asynchronous function abstractions. This year, <a href="https://sandervanderburg.blogspot.com/2022/01/structured-asynchronous-programming.html">I wrote a follow-up blog post about the ES6 concepts that I should use</a> (rather than software abstractions).<br />
<br />
To motivate myself learning about ES6 concepts, I needed a practical use case -- <a href="https://sandervanderburg.blogspot.com/2022/02/a-layout-framework-experiment-in.html">I have ported the layout component of my web framework</a> (for which a Java and PHP version already exist) to JavaScript using modern ES6 features, such as <i>async/await</i>, classes and modules.<br />
<br />
An interesting property of the JavaScript version is that it can be used both on the server-side (as a Node.js application) and client-side (directly in the browser by dynamically updating the DOM). The Java and PHP versions only work server-side.<br />
<br />
<h2>Fun projects</h2>
<br />
In earlier blog reflections I have also decided to spend more time on useless fun projects.<br />
<br />
In the summer of 2021, when I decided not to do any traveling, I had lots of time left to tinker with all kinds of weird things. One of my hobby projects was to play around with my custom maps for Duke3D and Shadow Warrior that I created while I was still a teenager.<br />
<br />
While playing with these maps, I noticed a number of interesting commonalities and differences between Duke3D and Shadow Warrior.<br />
<br />
Although both games use the same game engine: the BUILD-engine, their game mechanics are completely different. As an exercise, <a href="https://sandervanderburg.blogspot.com/2022/08/porting-duke3d-map-to-shadow-warrior.html">I have ported one of my Duke3D maps to Shadow Warrior and wrote a blog post about the process</a>, including a description of some of their different game mechanics.<br />
<br />
Although I did the majority of the work already back in 2021, I have found some remaining free time in 2022 to finally finish the project.<br />
<br />
<h2>Web framework improvements</h2>
<br />
This year, I have also intensively worked on improving several aspects of my own web framework. My custom web framework is an old project that I started in 2004 and many parts of it have been rewritten several times.<br />
<br />
I am not actively working on it anymore, but once in a while I still do some development work, because it is still in use by a couple of web sites, including the web site of my musical society.<br />
<br />
One of my goals is to improve the user experience of the musical society web site on mobile devices, such as phones and tablets. This particular area was already problematic for years. Despite making the promise to all kinds of people to fix this, it took me several years to actually take that step. :-).<br />
<br />
To improve the user experience for mobile devices, I wanted to convert the layout to a flexbox layout, for which I needed to extend my layout framework because it does not generate nested <i>div</i>s.<br />
<br />
I have managed to improve my layout framework to support flexbox layouts. In addition, I have also made many additional improvements. I wrote a blog post with <a href="https://sandervanderburg.blogspot.com/2022/12/a-summary-of-my-layout-framework.html">a summary of all my feature changes</a>.<br />
<br />
<h2>Nix-related work</h2>
<br />
In 2022, I also did Nix-related work, but I have not written any Nix-related blog posts this year. Moreover, 2022 is also the first time since the end of the pandemic that a physical <a href="http://2022.nixcon.org">NixCon</a> was held -- unfortunately, I have decided not to attend it.<br />
<br />
The fact that I did not write any Nix-related blog posts is quite exceptional. Since 2010, the majority of my blog posts are Nix-related and about software deployment challenges in general. So far, it has never happened that there has been an entire year without any Nix-related blog posts. I think I need to explain a thing or two about what has happened.<br />
<br />
This year, it was very difficult for me to find the energy to undertake any major Nix developments. Several things have contributed to that, but the biggest take-away is that I have to find the right balance.<br />
<br />
The reason why I got so extremely out of balance is that I do most of my Nix-related work in my spare time. Moreover, my primary motivation to do Nix-related work is because of idealistic reasons -- I still genuinely believe that we can automate the deployment of complex systems in a much better way than the conventional tools that people currently use.<br />
<br />
Some of the work for Nix and NixOS is relatively straight forward -- sometimes, we need to package new software, sometimes a package or NixOS service needs to be updated, or sometimes broken features need to be fixed or improved. This process is often challenging, but still relatively straight forward.<br />
<br />
There are also quite a few major challenges in the Nix project, for which no trivial solutions exist. These are problem areas that cannot be solved with quick fixes and require fundamental redesigns. Solving these fundamental problems is quite challenging and typically require me to dedicate a significant amount of my free time.<br />
<br />
Unfortunately, due to the fact that most of my work is done in my spare time, and I cannot multi-task, I can only work on one major problem area at the time.<br />
<br />
For example, I am quite happy with my last major development project: the <a href="https://github.com/svanderburg/nix-processmgmt">Nix process management framework</a>. It has all features implemented that I want/need to consistently eat my own dogfood. It is IMHO a pretty decent solution for use cases where most conventional developers would normally use Docker/docker-compose for.<br />
<br />
Unfortunately, to reach all my objectives I had to pay a huge price -- I have published the first implementation of the process management framework already in 2019, and all my major objectives were reached in the middle of 2021. As a consequence, I have spend nearly two years of my spare time only working on the implementation of this framework, without having the option to switch to something else. For the first six months, I remained motivated, but slowly I ran into motivational problems.<br />
<br />
In this two-year time period, there were lots of problems appearing in other projects I used to be involved in. I could not get these projects fixed, because these projects also ran into fundamental problems requiring major redesigns/revisions. This resulted in a number of problems with members in the Nix community.<br />
<br />
As a result, I got the feeling the I lost control. Moreover, doing anything Nix-related work also gave (and in some extent still gives) me a lot of negative energy.<br />
<br />
Next year, I intend to return and I will look into addressing my issues. I am thinking about the following steps:<br />
<br />
<ul>
<li><strong>Leaving the solution of some major problem areas to others</strong>. One of such areas is NPM package deployments with Nix. node2nix's was probably a great tool in combination with older versions of NPM, but its design has reached the boundaries of what is possible already years ago.<br />
<br />
As a result, node2nix does not support the new features of NPM and does not solve the package scalability issues in Nixpkgs. It is also not possible to properly support these use cases by implementing "quick fixes". To cope with these major challenges and keep the solution maintainable, a new design is needed.<br />
<br />
I have already explained my ideas on the Discourse mailing list and outlined what such a new design could look like. Fortunately, there are already some good initiatives started to address these challenges.</li>
<li><strong>Building prototypes and integrate the ideas into Nixpkgs</strong> rather than starting an independent project/tool that attracts a sub community.<br />
<br />
I have implemented the Nix process management framework as a prototype with the idea to show how certain concepts work, rather than advertising the project as a new solution.<br />
<br />
My goal is to write an RFC to make sure that these ideas get integrated into the upstream Nixpkgs, so that it can be maintained by the community and everybody can benefit from it.<br />
<br />
The only thing I still need to do is write that RFC. This should probably be one of my top priorities next year.</li>
<li><strong>Move certain things out of Nixpkgs</strong>. The Nixpkgs project is a huge project with several thousands of packages and services, making it quite a challenge to maintain and implement fundamental changes.<br />
<br />
One of the side effects of its scale is that the Nixpkgs issue tracker is a good as useless. There are thousands of open issues and it is impossible to properly track the status of individual aspects in the Nixpkgs repository.<br />
<br />
Thanks to <a href="https://nixos.wiki/wiki/Flakes">Nix flakes</a>, which unfortunately is still an experimental feature, we should be able to move certain non-essential things out of Nixpkgs and conveniently deploy them from external repositories. I have some things that I could move out of the Nixpkgs repository when flakes have become a mainstream feature.</li>
<li><strong>Better communication about the context in which something is developed</strong>. When I was younger, I always used to advertise a new project as the next great thing that everybody should use -- these days, I am more conservative about the state of my projects and I typically try to warn people upfront that something is just a prototype and not yet ready for production use.</li>
</ul>
<br />
<h2>Blog posts</h2>
<br />
In my previous reflection blog posts, I always used to reflect over my overall top 10 of most popular blog posts. There are no serious changes compared to last year, so I will not elaborate about them. The fact that I have not been so active on my blog this year has probably contributed that.<br />
<br />
<h2>Concluding remarks</h2>
<br />
Next year I will look into addressing my issues with Nix development. I hope to return to my software deployment/Nix-related work next year!<br />
<br />
The final thing I would like to say is:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf_7mqysBYTZlQrPdVjn8dUqwQxcOUJ5CPRjtr81UHM3mhBn-Hr9wl9BbsMPGwvvkXy7QhgLn7f1okrIxo1dlps-U_WekueJJPcBTmKmuedwl82xwiSSJqp9Rr6KkiMvvpeh_OTv4XvH584xHpa8MVdz4hth9DISGQRDLaSkFDFHNNLG06_ibhBtRObQ/s640/p0b977q7.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="360" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhf_7mqysBYTZlQrPdVjn8dUqwQxcOUJ5CPRjtr81UHM3mhBn-Hr9wl9BbsMPGwvvkXy7QhgLn7f1okrIxo1dlps-U_WekueJJPcBTmKmuedwl82xwiSSJqp9Rr6KkiMvvpeh_OTv4XvH584xHpa8MVdz4hth9DISGQRDLaSkFDFHNNLG06_ibhBtRObQ/s600/p0b977q7.jpg"/></a></div>
<br />
HAPPY NEW YEAR!!!<br />
<br />Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-92026966379542323162022-12-30T20:17:00.003+01:002023-07-03T18:37:30.421+02:00A summary of my layout framework improvementsIt has been quiet for a while on my blog. In the last couple of months, I have been improving <a href="https://sandervanderburg.blogspot.com/2017/07/some-reflections-on-my-experiences-with.html">my personal web application framework</a>, after several years of inactivity.<br />
<br />
The reason why I became motivated to work on it again, is because I wanted to improve the website of the musical society that I am a member of. This website is still one of the few consumers of my personal web framework.<br />
<br />
One of the areas for improvement is the user experience on mobile devices, such as phones and tablets.<br />
<br />
To make these improvements possible, I wanted to get rid of complex legacy functionality, such as the "<a href="https://css-tricks.com/fluid-width-equal-height-columns/">One True Layout</a>" method, that heavily relies on all kinds of interesting hacks that are no longer required in modern browsers. Instead, I wanted to use a <a href="https://css-tricks.com/snippets/css/a-guide-to-flexbox">flexbox layout</a> that is much more suitable for implementing the layout aspects that I need.<br />
<br />
As I have already explained in previous blog posts, my web application framework is not monolithic -- it consists of multiple components each addressing a specific concern. These components can be used and deployed independently.<br />
<br />
The most well-explored component is the <a href="https://sandervanderburg.blogspot.com/2014/03/implementing-consistent-layouts-for.html">layout framework</a> that addresses the layout concern. It generates pages from a high-level <strong>application model</strong> that defines common layout aspects of an application and the pages of which an application consists including their unique content parts.<br />
<br />
I have created multiple implementations of this framework in three different programming languages: Java, PHP, and JavaScript.<br />
<br />
In this blog post, I will give a summary of all the recent improvements that I made to the layout framework.<br />
<br />
<h2>Background</h2>
<br />
As I have already explained in previous blog posts, the layout framework is very straight forward to use. As a developer, you need to specify a high-level <strong>application model</strong> and invoke a <strong>view function</strong> to render a sub page belonging to the application. The layout framework uses the path components in a URL to determine which sub page has been selected.<br />
<br />
The following code fragment shows an application model for a trivial test web application:<br />
<br />
<pre style="overflow: auto;">
use SBLayout\Model\Application;
use SBLayout\Model\Page\StaticContentPage;
use SBLayout\Model\Page\Content\Contents;
use SBLayout\Model\Section\ContentsSection;
use SBLayout\Model\Section\MenuSection;
use SBLayout\Model\Section\StaticSection;
$application = new Application(
/* Title */
"Simple test website",
/* CSS stylesheets */
array("default.css"),
/* Sections */
array(
"header" => new StaticSection("header.php"),
"menu" => new MenuSection(0),
"contents" => new ContentsSection(true),
),
/* Pages */
new StaticContentPage("Home", new Contents("home.php"), array(
"page1" => new StaticContentPage("Page 1", new Contents("page1.php")),
"page2" => new StaticContentPage("Page 2", new Contents("page2.php")),
"page3" => new StaticContentPage("Page 3", new Contents("page3.php"))
))
);
</pre>
<br />
The above application model captures the following application layout properties:<br />
<br />
<ul>
<li>The <strong>title</strong> of the web application is: "Simple test website" and displayed as part of the title of any sub page.</li>
<li>Every page references the same external <strong>CSS stylesheet</strong> file: <i>default.css</i> that is responsible for styling all pages.</li>
<li>Every page in the web application consists of the same kinds of <strong>sections</strong>:
<ul>
<li>The <i>header</i> element refers to a static header section whose purpose is to display a logo. This section is the same for every sub page.</li>
<li>The <i>menu</i> element refers to a <i>MenuSection</i> whose purpose is to display menu links to sub pages that can be reached from the entry page.</li>
<li>The <i>contents</i> element refers to a <i>ContentsSection</i> whose purpose is to display contents (text, images, tables, itemized lists etc.). The content is different for each selected page.</li>
</ul>
</li>
<li>The application consists of a number of <strong>pages</strong>:
<ul>
<li>The entry page is a page called: 'Home' and can be reached by opening the root URL of the web application: <i>http://localhost</i></li>
<li>The entry page refers to three sub pages: <i>page1</i>, <i>page2</i> and <i>page3</i> that can be reached from the entry page.<br />
<br />
The array keys refer to the path component in the URL that can be used as a selector to open the sub page. For example, <i>http://localhost/page1</i> will open the <i>page1</i> sub page and <i>http://localhost/page2</i> will open the <i>page2</i> sub page.</li>
</ul>
</li>
</ul>
<br />
The currently selected page can be rendered with the following function invocation:<br />
<br />
<pre>
\SBLayout\View\HTML\displayRequestedPage($application);
</pre>
<br />
By default, the above function generates a simple HTML page in which each section gets translated to an HTML <i>div</i> element:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5rlL_ZJo9ArBQF9r5EmCz-Gs9vDwXuK-gOo410Ht32FDNHPG01hR1vo1eyIb4QtPVfcv7_j2R0S8E2Uz5pm1r5flqhzyEO24dxHT5teXkWERzAZscwcAhQTJ6FbApII9EcskWmbL8bWduqekyTBD5gOUABZTaFuTnLdDhqfeFtbSRcBxexXzMWmkH-A/s1636/simplelayout.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1077" data-original-width="1636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj5rlL_ZJo9ArBQF9r5EmCz-Gs9vDwXuK-gOo410Ht32FDNHPG01hR1vo1eyIb4QtPVfcv7_j2R0S8E2Uz5pm1r5flqhzyEO24dxHT5teXkWERzAZscwcAhQTJ6FbApII9EcskWmbL8bWduqekyTBD5gOUABZTaFuTnLdDhqfeFtbSRcBxexXzMWmkH-A/s600/simplelayout.png"/></a></div>
<br />
The above screenshot shows what a page in the application could look like. The grey panel on top is the <i>header</i> that displays the logo, the blue bar is <i>menu</i> section (that displays links to sub pages that are reachable from the entry page), and the black area is the <i>content</i> section that displays the selected content.<br />
<br />
One link in the menu section is marked as <strong>active</strong> to show the user which page in the page hierarchy (<i>page1</i>) has been selected.<br />
<br />
<h2>Compound sections</h2>
<br />
Although the framework's functionality works quite well for most of my old use cases, I learned that in order to support flexbox layouts, I need to nest <i>div</i>s, which is something the default HTML code generator: <i>displayRequestedPage()</i> cannot do (as a sidenote: it is possible to create nestings by developing a custom generator).<br />
<br />
For example, I may want to introduce another level of pages and add a <i>submenu</i> section to the layout, that is displayed on the left side of the screen.<br />
<br />
To make it possible to position the menu bar on the left, I need to horizontally position the <i>submenu</i> and <i>contents</i> sections, while the remaining sections: <i>header</i> and <i>menu</i> must be vertically positioned. To make this possible with flexbox layouts, I need to nest the <i>submenu</i> and <i>contents</i> in a <i>container</i> div.<br />
<br />
Since flexbox layouts have become so common nowadays, I have introduced a <i>CompoundSection</i> object, that acts as a generic container element.<br />
<br />
With a <i>CompoundSection</i>, I can nest <i>div</i>s:<br />
<br />
<pre>
/* Sections */
array(
"header" => new StaticSection("header.php"),
"menu" => new MenuSection(0),
"container" => new CompoundSection(array(
"submenu" => new MenuSection(1),
"contents" => new ContentsSection(true)
))
),
</pre>
<br />
In the above code fragment, the <i>container</i> section will be rendered as a container <i>div</i> element containing two sub div elements: <i>submenu</i> and <i>contents</i>. I can use the nested divs structure to vertically and horizontally position the sections in the way that I described earlier.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhK-dmIgeUtn83ECi41QR9WeSnFmBesNnhAax0xSamEGsvQSBoHcRH1ywIdA-Qtc9_eqjhLEeO6TXaiy1GV_jIK3z0P5Zk2X2KrkmD3kBW_swPphRUOC1lgfR_Wc9ZGBsG7wR20B0kPfXF07BJfUTw5p-_k-KLWXk7FOsPaSXmapOwt1-DPFXibYyUiYQ/s1636/nesteddivs.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1077" data-original-width="1636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhK-dmIgeUtn83ECi41QR9WeSnFmBesNnhAax0xSamEGsvQSBoHcRH1ywIdA-Qtc9_eqjhLEeO6TXaiy1GV_jIK3z0P5Zk2X2KrkmD3kBW_swPphRUOC1lgfR_Wc9ZGBsG7wR20B0kPfXF07BJfUTw5p-_k-KLWXk7FOsPaSXmapOwt1-DPFXibYyUiYQ/s600/nesteddivs.png"/></a></div>
<br />
The above screenshot shows the result of introducing a secondary page hierarchy and a <i>submenu</i> section (that has a red background).<br />
<br />
By introducing a <i>container</i> element (through a <i>CompoundSection</i>) it has become possible to horizontally position the <i>submenu</i> next to the <i>contents</i> section.<br />
<br />
<h3>Easier error handling</h3>
<br />
Another recurring issue is that most of my applications have to validate user input. When user input is incorrect, a page needs to be shown that displays an error message.<br />
<br />
Previously, error handling and error page redirection was entirely the responsibility of the programmer -- it had to be implemented in every controller, which is quite a repetitive process.<br />
<br />
In one of my test applications of the layout framework, I have created a page with a form that asks for the user's first and last name:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTq7Qk5ZrrgLXjB_xg2ZFlx2pDNSi8gF8pTfRCzMC3rpr-KCd_z7si5PdcAZXGn7GP0gDy_coHX-Lkty6ow3XjswMocSy8TyU5dhSEOyJSDlzPro0mIzZ7VlyxdIXqg_EmCbN6cxU4FEmyqCcSnzgW-r-ofurXfqTGlCDGUlNdoEd6sr83UjVCSF4YIA/s1636/form.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1077" data-original-width="1636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTq7Qk5ZrrgLXjB_xg2ZFlx2pDNSi8gF8pTfRCzMC3rpr-KCd_z7si5PdcAZXGn7GP0gDy_coHX-Lkty6ow3XjswMocSy8TyU5dhSEOyJSDlzPro0mIzZ7VlyxdIXqg_EmCbN6cxU4FEmyqCcSnzgW-r-ofurXfqTGlCDGUlNdoEd6sr83UjVCSF4YIA/s600/form.png"/></a></div>
<br />
I wanted to change the example application to return an error message when any of these mandatory attributes were not provided.<br />
<br />
To ease that burden, I have made framework's error handling mechanism more generic. Previously, the layout manager only took care of two kinds of errors: when an invalid sub page is requested, a <i>PageNotFoundException</i> is thrown redirecting the user to the 404 error page. When the accessibility criteria have not been met (e.g. a user is not authenticated) a <i>PageForbiddenException</i> is thrown directing the user to the 403 error page.<br />
<br />
In the revised version of the layout framework, the <i>PageNotFoundException</i> and <i>PageForbiddenException</i> classes have become sub classes of the generic <i>PageException</i> class. This generic error class makes it possible for the error handler to redirect users to error pages for any HTTP status code.<br />
<br />
Error pages should be added as sub pages to the entry page. The numeric keys should match the corresponding HTTP status codes:<br />
<br />
<pre style="overflow: auto;">
/* Pages */
new StaticContentPage("Home", new Contents("home.php"), array(
"400" => new HiddenStaticContentPage("Bad request", new Contents("error/400.php")),
"403" => new HiddenStaticContentPage("Forbidden", new Contents("error/403.php")),
"404" => new HiddenStaticContentPage("Page not found", new Contents("error/404.php"))
...
))
</pre>
I have also introduced a <i>BadRequestException</i> class (that is also a sub class of <i>PageException</i>) that can be used for handling input validation errors.<br />
<br />
<i>PageException</i>s can be thrown from controllers with a custom error message as a parameter. I can use the following controller implementation to check whether the first and last names were provided:<br />
<br />
<pre style="overflow: auto;">
use SBLayout\Model\BadRequestException;
if($_SERVER["REQUEST_METHOD"] == "POST") // This is a POST request
{
if(array_key_exists("firstname", $_POST) && $_POST["firstname"] != ""
&& array_key_exists("lastname", $_POST) && $_POST["lastname"] != "")
$GLOBALS["fullname"] = $_POST["firstname"]." ".$_POST["lastname"];
else
throw new BadRequestException("This page requires a firstname and lastname parameter!");
}
</pre>
<br />
The side effect is that if the user forgets to specify any of these mandatory attributes, he gets automatically redirected to the bad request error page:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMOklBkhRbq81GjQsvqPbeBzxj1pNR_DTyeAS20-k1ZPHbrJ341lEJulBVDt8BOOwjB-a9ZUOn4rIT6G18C8V9-Hn1444UTtltepQLWwhexQGOlxS7VEh0GW2eoWfe35PMvjpDjENA8BbMBp8t10YAap8QI2YxYWCKUci1ndF0B0XdBBfJ_ZjTVPSKvg/s1636/badrequest.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1077" data-original-width="1636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhMOklBkhRbq81GjQsvqPbeBzxj1pNR_DTyeAS20-k1ZPHbrJ341lEJulBVDt8BOOwjB-a9ZUOn4rIT6G18C8V9-Hn1444UTtltepQLWwhexQGOlxS7VEh0GW2eoWfe35PMvjpDjENA8BbMBp8t10YAap8QI2YxYWCKUci1ndF0B0XdBBfJ_ZjTVPSKvg/s600/badrequest.png"/></a></div>
<br />
This improved error handling mechanism significantly reduces the amount of boilerplate code that I need to write in applications that use my layout framework.<br />
<br />
<h3>Using the iterator protocol for sub pages</h3>
<br />
As can be seen in the application model examples, some pages in the example applications have sub pages, such as the entry page.<br />
<br />
In the layout framework, there are three kinds of pages that may provide sub pages:<br />
<br />
<ul>
<li>A <i>StaticContentPage</i> object is a page that may refer to a fixed/static number of sub pages (as an array object).</li>
<li>A <i>PageAlias</i> object, that redirects the user to another sub page in the application, also offers the ability to refer users to a fixed/static number of sub pages (as an array object).</li>
<li>There is also a <i>DynamicContentPage</i> object in which a sub page can interpret the path component as a dynamic value. That dynamic value can, for example, be used as a parameter for a query that retrieves a record from a database.</li>
</ul>
<br />
In the old implementation of my framework, the code that renders the menu sections always has to treat these objects in a special way to render links to their available sub pages. As a result, I had to use the <i>instanceof</i> operator a lot, which is in a bad <a href="https://eprints.lancs.ac.uk/id/eprint/127419/1/Tosem_code_smells.pdf">code smell</a>.<br />
<br />
I have changed the framework to use a different mechanism for stepping over sub pages: <a href="https://en.wikipedia.org/wiki/Iterator">iterators or iterables</a> (depending on the implementation language).<br />
<br />
The generic <i>Page</i> class (that is the parent class of all page objects) provides a method called: <i>subPageIterator()</i> that returns an iterator/iterable that yields no elements. The <i>StaticContentPage</i> and <i>PageAlias</i> classes override this method to return an interator/iterable that steps over the elements in the array of sub pages.<br />
<br />
Using iterators/iterables has a number of nice consequences -- I have eliminated two special cases and a bad code smell (the intensive use of <i>instanceof</i>), significantly improving the quality and readability of my code.<br />
<Br />
Another nice property is that it is also possible to override this method with a custom iterator, that for example, fetches sub page configurations from a database.<br />
<br />
The pagemanager framework (another component in my web framework) offers a content management system giving end-users the ability to change the page structure and page contents. The configuration of the pages is stored in a database.<br />
<br />
Although the pagemanager framework uses the layout framework for the construction of pages, it used to rely on custom code to render the menu sections.<br />
<br />
By using the iterator protocol, it has become possible to re-use the menu section functionality from the layout framework eliminating the need for custom code. Moreover, it has also become much easier to integrate the pagemanager framework into an application because no additional configuration work is required.<br />
<br />
I have also created a gallery application that makes it possible to expose the albums as items in the menu sections. Rendering the menu sections also used to rely on custom code, but thanks to using the iterator protocol that custom code was completely eliminated.<br />
<br />
<h2>Flexible presentation of menu items</h2>
<br />
As I have already explained, an application layout can be divided into three kinds of sections. A <i>StaticSection</i> remains the same for any requested sub page, and a <i>ContentSection</i> is filled with content that is unique for the selected page.<br />
<br />
In most of my use-cases, it is only required to have a single dynamic content section.<br />
<br />
However, the framework is flexible enough to support multiple content sections as well. For example, the following screenshot shows the advanced example application (included with the web framework) in which both the header and the content sections change for each sub page:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTXnPT7Q4ZNOtH1_7ARSZKq6WPJGDXLpEkwO6FQOOVfxDKVTYThzzr4_AKvD45KQ_ykYFTwAg6G_pBXMnvchlyDuNUnXo-tv3p2citL3tDQUo-ptLBLXx9dBkah_9DvCHX0VV3Esgaj7imp19zkVSbbrh0b3nfIM7r8EAaNSQfRfV-Yprh0OrHGJFg5A/s1636/advancedlayout.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1077" data-original-width="1636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiTXnPT7Q4ZNOtH1_7ARSZKq6WPJGDXLpEkwO6FQOOVfxDKVTYThzzr4_AKvD45KQ_ykYFTwAg6G_pBXMnvchlyDuNUnXo-tv3p2citL3tDQUo-ptLBLXx9dBkah_9DvCHX0VV3Esgaj7imp19zkVSbbrh0b3nfIM7r8EAaNSQfRfV-Yprh0OrHGJFg5A/s600/advancedlayout.png"/></a></div>
<br />
The presentation of the third kind of section: <i>MenuSection</i> still used to remain pretty static -- they are rendered as <i>div</i> elements containing hyperlinks. The page that is currently selected is marked as active by using the <i>active</i> class property.<br />
<br />
For most of my use-cases, just rendering hyperlinks suffices -- with CSS you can still present them in all kinds of interesting ways, e.g. by changing their colors, adding borders, and changing some its aspects when the user hovers with the mouse cursor over it.<br />
<br />
In some rare cases, it may also be desired to present links to sub pages in a completely different way. For example, you may want to display an icon or add extra styling properties to an individual button.<br />
<br />
To allow custom presentations of hyperlinks, I have added a new parameter: <i>menuItem</i> to the constructors of page objects. The <i>menuItem</i> parameter refers to a code snippet that decides how to render the link in a menu section:<br />
<br />
<pre>
new StaticContentPage("Icon", new Contents("icon.php"), "icon.php")
</pre>
<br />
In the above example, the last parameter to the constructor, refers to an external file: <i>menuitem/icon.php</i>:<br />
<br />
<pre style="font-size: 90%; overflow: auto;">
<span>
<?php
if($active)
{
?>
<a class="active" href="<?= $url ?>">
<img src="<?= $GLOBALS["baseURL"] ?>/image/menu/go-home.png" alt="Home icon">
<strong><?= $subPage->title ?></strong>
</a>
<?php
}
else
{
?>
<a href="<?= $url ?>">
<img src="<?= $GLOBALS["baseURL"] ?>/image/menu/go-home.png" alt="Home icon">
<?= $subPage->title ?>
</a>
<?php
}
?>
</span>
</pre>
<br />
The above code fragment specifies how a link in the menu section should be displayed when the page is active or not active. We use the custom rendering code to display a home icon before showing the hyperlink.<br />
<br />
In the advanced test application, I have added an example page in which every sub menu item is rendered in a custom way:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjlUu07UJLULy0oVxL1uxOPJ70KBleu1YitOb2zqSAYiYH4Jt1uX0P1MWKyljjD_libObXr-EYWe1SHbex8P-c5puFJ9hlwFX3Zkg4uexzkUOIdl133YxJAc7IR49R3ZCAKRz9hvlvZP23i7PJzS_EXt_yROw6S2NbzN1j42TCsxbP0edVs0Egcc0USA/s1636/custommenuitems.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1077" data-original-width="1636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgjlUu07UJLULy0oVxL1uxOPJ70KBleu1YitOb2zqSAYiYH4Jt1uX0P1MWKyljjD_libObXr-EYWe1SHbex8P-c5puFJ9hlwFX3Zkg4uexzkUOIdl133YxJAc7IR49R3ZCAKRz9hvlvZP23i7PJzS_EXt_yROw6S2NbzN1j42TCsxbP0edVs0Egcc0USA/s600/custommenuitems.png"/></a></div>
<br />
In the above screenshot, we should see two custom presented menu items in the <i>submenu</i> section on the left. The first has the home icon added and the second uses a custom style that deviates from the normal page style.<br />
<br />
If no <i>menuItem</i> parameter was provided, the framework just renders a menu item as a normal hyperlink.<br />
<br />
<h2>Other functionality</h2>
<br />
In addition to the new functionality explained earlier, I also made a number of nice small feature additions:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLQEiEbPu_GENGTQA-ZL6p2JgazuATpQxmj8WWzMghQX9rl9WdLPZGfBAHXKvLFDhpT7bK4WGt0wCB6Gm4meEJoSazR2E_y26rd3p5wKV5u_ubVRzWmTaB9BUjp2ih4Fl0qwv13wkoiRH9JDYSSqkPzzTT9iYGXOmKh1GJsPLoe7dYMVG7PUA42nGW6A/s1636/sitemap.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1077" data-original-width="1636" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjLQEiEbPu_GENGTQA-ZL6p2JgazuATpQxmj8WWzMghQX9rl9WdLPZGfBAHXKvLFDhpT7bK4WGt0wCB6Gm4meEJoSazR2E_y26rd3p5wKV5u_ubVRzWmTaB9BUjp2ih4Fl0qwv13wkoiRH9JDYSSqkPzzTT9iYGXOmKh1GJsPLoe7dYMVG7PUA42nGW6A/s600/sitemap.png"/></a></div>
<br />
<ul>
<li>A function that displays <strong>bread crumbs</strong> (the route from the entry page to the currently opened page). The route is derived automatically from the requested URL and application model.</li>
<li>A function that displays a <strong>site map</strong> that shows the hierarchy of pages.</li>
<li>A function that makes it possible to embed a <strong>menu section</strong> in arbitrary sections of a page.</li>
</ul>
<br />
<h2>Conclusion</h2>
<br />
I am quite happy with the recent feature changes that I made to the layout framework. Although I have not done any web front-end development for quite some time, I had quite a bit of fun doing it.<br />
<br />
In addition to the fact that useful new features were added, I have also simplified the codebase and improved its quality.<br />
<br />
<h2>Availability</h2>
<br />
The <a href="https://github.com/svanderburg/java-sblayout">Java</a>, <a href="https://github.com/svanderburg/php-sblayout">PHP</a> and <a href="https://github.com/svanderburg/js-sblayout">JavaScript</a> implementations of my layout framework can be obtained from <a href="https://github.com/svanderburg">my GitHub page</a>. Use them at your own risk!<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-19415182811978571522022-08-20T00:49:00.001+02:002022-08-20T00:49:49.114+02:00Porting a Duke3D map to Shadow Warrior<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHtJUK6oq6szB-UxOkSIFKpjw5odolJPPfPY7EWb8nS6lkrsh21XeK2b0x14E7m6SKXitcGHLIVMTt2N3bZuec5Woyf7_PqV2zWpH_On4BL9kCfLJqHJMBFrJpEveV4EKZMS7LJXlqGU0eeC-uuAx6fNl4J5PJpqyZMSFabp7EKB5Gc-Ps0LGzuNxo9A/s640/originalmap.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhHtJUK6oq6szB-UxOkSIFKpjw5odolJPPfPY7EWb8nS6lkrsh21XeK2b0x14E7m6SKXitcGHLIVMTt2N3bZuec5Woyf7_PqV2zWpH_On4BL9kCfLJqHJMBFrJpEveV4EKZMS7LJXlqGU0eeC-uuAx6fNl4J5PJpqyZMSFabp7EKB5Gc-Ps0LGzuNxo9A/s600/originalmap.png"/></a></div>
<br />
Almost six years ago, I wrote <a href="https://sandervanderburg.blogspot.com/2016/12/creating-total-conversion-for-duke3d.html">a blog post about Duke Nukem 3D, the underlying BUILD engine and my own total conversion</a> that consists of 22 maps and a variety of interesting customizations.<br />
<br />
Between 1997 and 2000, while I was still in middle school, I have spent a considerable amount of time developing my own maps and customizations, such as modified monsters. In the process, I learned a great deal about the technical details of the <a href="http://advsys.net/ken/build.htm">BUILD engine</a>.<br />
<br />
In addition to <a href="https://en.wikipedia.org/wiki/Duke_Nukem_3D">Duke Nukem 3D</a>, the BUILD engine is also used as a basis for many additional games, such as Tekwar, Witchaven, Blood, and <a href="https://en.wikipedia.org/wiki/Shadow_Warrior_(1997_video_game)">Shadow Warrior</a>.<br />
<br />
In my earlier blog post, I also briefly mentioned that in addition to the 22 maps that I created for Duke Nukem 3D, I have also developed one map for Shadow Warrior.<br />
<br />
Last year, in my summer holiday, that was still mostly about improvising my spare time because of the COVID-19 pandemic, I did many interesting retro-computing things, such as fixing my old computers. I also played a bit with some of my old BUILD engine game experiments, after many years of inactivity.<br />
<br />
I discovered an interesting <a href="https://www.moddb.com/games/shadow-warrior/addons/the-wang-light-district-v10">Shadow Warrior map that attempts to convert the E1L2 map from Duke Nukem 3D</a>. Since both games use the BUILD engine with mostly the same features (Shadow Warrior uses a slightly more advanced version of the BUILD engine), this map inspired me to also port one of my own Duke Nukem 3D maps, as an interesting deep dive to compare both game's internal concepts.<br />
<br />
Although most of the BUILD engine and editor concepts are the same in both games, their game mechanics are totally different. As a consequence, the porting process turned out to be very challenging.<br />
<br />
Another reason that it took me a while to complete the project is because I had to put it on hold in several occasions due to all kinds obligations. Fortunately, I have managed to finally finish it.<br />
<br />
In this blog post, I will describe some of the things that both games have in common and the differences that I had to overcome in the porting process.<br />
<br />
<h2>BUILD engine concepts</h2>
<br />
As explained in my previous blog post, the BUILD-engine is considered a 2.5D engine, not a true 3D engine due to the fact that it had to cope with all kinds of technical limitations of home computers commonly used at that time.<br />
<br />
In fact, most of the BUILD-engine concepts are two-dimensional -- maps are made out two-dimensional surfaces called <strong>sectors</strong>:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgkeopaz5bHFbz55tyLBd-vr4YxwVFaja9Q94nQ_xnTIx8pZo3dU4lr1_dWNDYZSWC1FSQ-ZXB1JQRoVmZA0jlVf18b3_aYMC93B-yAj_kiV1Jvlx2ajTaZniHVyEWG43KQjXb2ViIHAlaEZ_-LJ4Dy5M2z8ONeW07RfrqpZ7XPzOgx07LJTMo7W5aRA/s640/2dview.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhgkeopaz5bHFbz55tyLBd-vr4YxwVFaja9Q94nQ_xnTIx8pZo3dU4lr1_dWNDYZSWC1FSQ-ZXB1JQRoVmZA0jlVf18b3_aYMC93B-yAj_kiV1Jvlx2ajTaZniHVyEWG43KQjXb2ViIHAlaEZ_-LJ4Dy5M2z8ONeW07RfrqpZ7XPzOgx07LJTMo7W5aRA/s600/2dview.png"/></a></div>
<br />
The above picture shows a 2-dimensional top-level view of my ported Shadow Warrior map. Sectors are two dimensional areas surrounded by <strong>walls</strong> -- the white lines denote solid walls and red lines the walls between adjacent sectors. Red walls are invisible in 3D mode.<br />
<br />
The purple and cyan colored objects are <strong>sprites</strong> (objects that typically provide some form of interactivity with the player, such as monsters, weapons, items or switches). The "sticks" that are attached to the sprites indicate in which direction the sprite is facing. When a sprite is purple, it will block the player. Cyan colored sprites allow a player to move through it.<br />
<br />
You can switch between 2D and 3D mode in the editor by pressing the Enter-key on the numeric key pad.<br />
<br />
In 3D mode, each sector's ceiling and floor can be given its own height, and we can configure textures for the walls, floors and ceilings (by pointing to any of these objects and pressing the 'V' key) giving the player the illusion to walk around in a 3D world:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt2MIo1WWKvjcsS4nemxAVOKo60s-N85qN_xtj1_98VIyZYKLXGD7ns6DanYpSAyMddQ6uGZudmthgPypAu-Do2n4EzaJUHu-ZsXDgazyFy9oclXiq4JS7YoSVnOxI1ZaawllXWw8mg77e9dfxUqUKpkXs9KiuUh0crERAHEkhjYXsg9NnzjHtPI-9bA/s640/sectors.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjt2MIo1WWKvjcsS4nemxAVOKo60s-N85qN_xtj1_98VIyZYKLXGD7ns6DanYpSAyMddQ6uGZudmthgPypAu-Do2n4EzaJUHu-ZsXDgazyFy9oclXiq4JS7YoSVnOxI1ZaawllXWw8mg77e9dfxUqUKpkXs9KiuUh0crERAHEkhjYXsg9NnzjHtPI-9bA/s600/sectors.png"/></a></div>
<br />
In the above screenshot, we can see the corresponding 3D view of the 2D grid shown earlier. It consists of an outdoor area, grass, a lane, and the interior of the building. Each of these areas are separate 2D sectors with their own custom floor and ceiling heights, and their own textures.<br />
<br />
The BUILD engine has all kinds of limitations. Although a world may appear to be (somewhat) 3-dimensional, it is not possible to stack multiple sectors on top of each other and simultaneously see them in 3D mode, although there are some tricks to cope with that limitation.<br />
<br />
(As a sidenote: Shadow Warrior has a hacky feature that makes it possible for a player to observe multiple rooms stacked on top of each other, by using specialized wall/ceiling textures, special purpose sprites and a certain positioning of the sectors themselves. Sectors in the map are still separated, but thanks to the hack they can be visualized in such a way that they appear to be stacked on top of each other).<br />
<br />
Moreover, the BUILD engine can also not change the perspective when a player looks up or down, although there is the possibility to give a player that illusion by stretching the walls. (As a sidenote: modern source ports of the BUILD engine have been adjusted to use Polymost, an OpenGL rendering extension, which actually makes it possible to provide a true 3D look).<br />
<br />
Monsters, weapons, items, and most breakable/movable objects are sprites. Sprites are not really "true 3D" objects. Normally, sprites will always face the player from the same side, regardless of the position or the perspective of the player:
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHuK5AOUWVQAAytdP7bZrKmqM5FvcJ0UbGWnsz9rYPTPlDs1NffOKRzzPvCrLpZ5WYrf53Lcku4MNrWrwkrfN5lWhtItP1Ix3bNhfsLctWMkG76aqck8n0Gyz9pdeIqH3gSOrzmvUEr2E2FAoDOEdv2Vy_fE2NQWWxUdELdMJM_QdYCOgdLajn3UkgdA/s640/spritefront1.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiHuK5AOUWVQAAytdP7bZrKmqM5FvcJ0UbGWnsz9rYPTPlDs1NffOKRzzPvCrLpZ5WYrf53Lcku4MNrWrwkrfN5lWhtItP1Ix3bNhfsLctWMkG76aqck8n0Gyz9pdeIqH3gSOrzmvUEr2E2FAoDOEdv2Vy_fE2NQWWxUdELdMJM_QdYCOgdLajn3UkgdA/s320/spritefront1.png"/></a></div>
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8y4crLG1gLY8vfCb7RD32e9fb85J4oAh41jfLt5puxPJxKYzDiVumC_6nlRpjNWca5L4C0I7LQthOyXh6l2pX69Z36lDWoWApM4LwkVHivQnOOXdc3cySLgIXGFLUUguVex--xvH2ERGV-MdKHaGFd30FFwS0CN7jreHyGlDD8PGkkVjtXYzt02UDg/s640/spritefront2.png" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx8y4crLG1gLY8vfCb7RD32e9fb85J4oAh41jfLt5puxPJxKYzDiVumC_6nlRpjNWca5L4C0I7LQthOyXh6l2pX69Z36lDWoWApM4LwkVHivQnOOXdc3cySLgIXGFLUUguVex--xvH2ERGV-MdKHaGFd30FFwS0CN7jreHyGlDD8PGkkVjtXYzt02UDg/s320/spritefront2.png"/></a></div>
<div style="clear: both;"></div>
<br />
As can be seen, the guardian sprite always faces the player from the front, regardless of the angle of the camera.<br />
<br />
Sprites can also be flattened and rotated, if desired. Then they will appear as a flat surface to the player:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrugtybEQcgAJZabMxM_QiT6YPzAwprkRW9ESpJ4ZOudUJjvzBFjLxYvI7pjThtwMemLesQ3SdZfeqru1-94Hy3TDWmZPG3dUHNJ2mYEZK2rT4X_TGE08sKGg8SmVqFpmBmtYeZOEbQQg3IHAtLf1Zf7UsUfg8yFe94tYMBK0J8EXmE8KZvugkQcPHJA/s640/poster.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjrugtybEQcgAJZabMxM_QiT6YPzAwprkRW9ESpJ4ZOudUJjvzBFjLxYvI7pjThtwMemLesQ3SdZfeqru1-94Hy3TDWmZPG3dUHNJ2mYEZK2rT4X_TGE08sKGg8SmVqFpmBmtYeZOEbQQg3IHAtLf1Zf7UsUfg8yFe94tYMBK0J8EXmE8KZvugkQcPHJA/s600/poster.png"/></a></div>
<br />
For example, the wall posters in the screenshot above are flattened and rotated sprites.<br />
<br />
Shadow warrior uses a slightly upgraded BUILD engine that can provide a true 3D experience for certain objects (such as weapons, items, buttons and switches) by displaying them as voxels (3D pixels):<br />
<br /><div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmXQCcxVcKK4NAjE64qKjbirizO6aHQUf6V32wRyWOeB80OdAF_uI2hMsFB6xn-uwyiftUzoBV2Bdh5WXugqWyQlREHzVnkRLa3PprjXqjrhXbNVGILuYU8BcMVxzW97hy1ZVnwgJ_-9dB_kPrPTTJQ8lu0phQb51ANSqCerjVGvpZVFQFTrZ8mwi6Gg/s640/voxels.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhmXQCcxVcKK4NAjE64qKjbirizO6aHQUf6V32wRyWOeB80OdAF_uI2hMsFB6xn-uwyiftUzoBV2Bdh5WXugqWyQlREHzVnkRLa3PprjXqjrhXbNVGILuYU8BcMVxzW97hy1ZVnwgJ_-9dB_kPrPTTJQ8lu0phQb51ANSqCerjVGvpZVFQFTrZ8mwi6Gg/s600/voxels.png"/></a></div>
<br />
The BUILD engine that comes with Duke Nukem 3D lacks the ability to display voxels.<br />
<br />
<h2>Porting my Duke Nukem 3D map to Shadow Warrior</h2>
<br />
The map format that Duke Nukem 3D and Shadow Warrior use are exactly the same. To be precise: they both use version 7 of the map format.<br />
<br />
At first, it seemed to look relatively straight forward to port a map from one game to another.<br />
<br />
The first step in my porting process was to simply make a copy of the Duke Nukem 3D map and open it in the Shadow Warrior BUILD editor. What I immediately noticed is that all the textures and sprites look weird. The textures still have the same indexes and refer to textures in the Shadow Warrior catalog that are completely different:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnu2hxFeAoGLxOIKTVFYN7mnB4u_tw5UyKkjnU4MtX-g8bqkVUcMua081uV_7qroLPxwA2hA3KZozJA_Kn1d46cgWVDQ0hiUiPT7gcsHNDHUVhYTDUFeQd-ff3nHYeDhmmlkt_XBC793jR3Ug5_tDufMAl86nZBY0yfx36IHqydi-VHkQzLOFkHcFv1A/s1600/unported.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="200" data-original-width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgnu2hxFeAoGLxOIKTVFYN7mnB4u_tw5UyKkjnU4MtX-g8bqkVUcMua081uV_7qroLPxwA2hA3KZozJA_Kn1d46cgWVDQ0hiUiPT7gcsHNDHUVhYTDUFeQd-ff3nHYeDhmmlkt_XBC793jR3Ug5_tDufMAl86nZBY0yfx36IHqydi-VHkQzLOFkHcFv1A/s1600/unported.png"/></a></div>
<br />
Quite a bit of my time was spent on fixing all textures and sprites by looking for suitable replacements. I ended up replacing textures for the rocks, sky, buildings, water, etc. I also had to replace the monsters, weapons, items and other dynamic objects, and overcome some limitations for the player in the map, such as the absence of a jet pack. The process was a labourious, but straight forward.<br />
<br />
For example, this is how I have fixed the beach area:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirKIwgJMfOSzQrsJNj00-ArJ8V7dvseIc0OKatDYP4ZaVZSsR_8KIrgUD8jJXx08mnXan7KC9G4u8bKZjIxQXLIgmXZvqA2RcsdVKIig_WTefE9xr__ICnlWhrPETmarwQd_vYK69N9s5CU0aG-lpeXvjZBe-sPXz6fzissCuVYTPQeW1-4VYFeHsY8Q/s640/d3d_beach.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEirKIwgJMfOSzQrsJNj00-ArJ8V7dvseIc0OKatDYP4ZaVZSsR_8KIrgUD8jJXx08mnXan7KC9G4u8bKZjIxQXLIgmXZvqA2RcsdVKIig_WTefE9xr__ICnlWhrPETmarwQd_vYK69N9s5CU0aG-lpeXvjZBe-sPXz6fzissCuVYTPQeW1-4VYFeHsY8Q/s320/d3d_beach.png"/></a></div>
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBAK9iha3I4XKfBZSZOiKW-2Fy9P58P5SEdKbRoTor8qFTOsK6mkHFqYKLTeVeSLY6OY8yvqVWet837LVKEOWd6ceu7uGZUaE1lhAOLPswkuMrwdObDqzs8LHVjHfkJdKitdonj7Lk6s9bTGX3ni26bvI1itHAH6CS6AdAm2KpmPCm3Hzzjjelr7QqZA/s640/sw_beach.png" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiBAK9iha3I4XKfBZSZOiKW-2Fy9P58P5SEdKbRoTor8qFTOsK6mkHFqYKLTeVeSLY6OY8yvqVWet837LVKEOWd6ceu7uGZUaE1lhAOLPswkuMrwdObDqzs8LHVjHfkJdKitdonj7Lk6s9bTGX3ni26bvI1itHAH6CS6AdAm2KpmPCm3Hzzjjelr7QqZA/s320/sw_beach.png"/></a></div>
<div class="clear: both;"></div>
<br />
I have changed the interior of the office building as follows:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnKOd-7DxIfJmp4OWIjN894xduIb8JMU0s8rJv8UIkNcHi1BB89m5eEVi95qFb_pWmFMWcV2b1ZhiCmEGeIoIoA24IsgfOykFspBQySevhfFQNXn6ckiWd2NmpKdSTY3bST0LqIZUF29ALDm43v0saMtkk_C0t_qM34Rq4_zqcVSq1NShjK_P2jFKhKA/s640/d3d_office.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjnKOd-7DxIfJmp4OWIjN894xduIb8JMU0s8rJv8UIkNcHi1BB89m5eEVi95qFb_pWmFMWcV2b1ZhiCmEGeIoIoA24IsgfOykFspBQySevhfFQNXn6ckiWd2NmpKdSTY3bST0LqIZUF29ALDm43v0saMtkk_C0t_qM34Rq4_zqcVSq1NShjK_P2jFKhKA/s320/d3d_office.png"/></a></div>
<div class="separator" style=""><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzZmi6wphlgZgP2-G4ra_mEKWZK-d6-mzqtQohHmGvmA74gjjIH6tlof5tCriX5PjJpMWmj5e02WUBVm-jMvoVFOMsfIqMIzOAFpEmFvCBwNcSlkm8wwxWjz9vUr_D7VksmEOBKXjOe94FsGJ_E8n9iahMPG_nxWm86GL0ptV4Xdj7OEQtvkBhi2bDog/s640/sw_office.png" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjzZmi6wphlgZgP2-G4ra_mEKWZK-d6-mzqtQohHmGvmA74gjjIH6tlof5tCriX5PjJpMWmj5e02WUBVm-jMvoVFOMsfIqMIzOAFpEmFvCBwNcSlkm8wwxWjz9vUr_D7VksmEOBKXjOe94FsGJ_E8n9iahMPG_nxWm86GL0ptV4Xdj7OEQtvkBhi2bDog/s320/sw_office.png"/></a></div>
<div style="clear: both;"></div>
<br />
And the back garden as follows:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEE2nl5VYNwlEDkyroJ_OAg2pM-QQIeXlcSKdV8mNJMQPOkh42bvAJ1PEWa_DoKO20cAqAdqDjB6Vdc0t_YapDgkGStdHSeNX5bXOxymXrolasb0MnZ4OrMOL8kUcB181T2tQn5gLbS4O8R5DUwNEc-CCl_ICjHO-GndNR58hM4HrLHsa8gQYp6TXLCQ/s640/d3d_garden.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEE2nl5VYNwlEDkyroJ_OAg2pM-QQIeXlcSKdV8mNJMQPOkh42bvAJ1PEWa_DoKO20cAqAdqDjB6Vdc0t_YapDgkGStdHSeNX5bXOxymXrolasb0MnZ4OrMOL8kUcB181T2tQn5gLbS4O8R5DUwNEc-CCl_ICjHO-GndNR58hM4HrLHsa8gQYp6TXLCQ/s320/d3d_garden.png"/></a></div>
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiha_XMoz81_dJjuoC-Y1ULl1yXcjFDe4fXIrTGAYwQZXzYjlQaozLKEP-_6CmbOKh_d-FJPifi7LqCsqHYW03Ho5yvF8x0TRlshKL2umk_wxW0VCbHtyZain2uOKW25R2m0hUATYgqkSNxf_RkpvyUmRXh4qxbyrRGHrOw91D4sCeJ1X042jTzkR05xA/s640/sw_garden.png" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiha_XMoz81_dJjuoC-Y1ULl1yXcjFDe4fXIrTGAYwQZXzYjlQaozLKEP-_6CmbOKh_d-FJPifi7LqCsqHYW03Ho5yvF8x0TRlshKL2umk_wxW0VCbHtyZain2uOKW25R2m0hUATYgqkSNxf_RkpvyUmRXh4qxbyrRGHrOw91D4sCeJ1X042jTzkR05xA/s320/sw_garden.png"/></a></div>
<div style="clear: both;"></div>
<br />
The nice thing about the garden area is that Shadow Warrior has a more diverse set of vegetation sprites. Duke Nukem 3D only has palm trees.<br />
<br />
<h2>Game engine differences</h2>
<br />
The biggest challenge for me was porting the interactive parts of the game. As explained earlier, game mechanics are not implemented by the engine or the editor. BUILD-engine games are separated into an engine and game part in which only the former component is generalized.<br />
<br />
This diagram (that I borrowed from a <a href="https://fabiensanglard.net/duke3d/index.php">Duke Nukem 3D code review article written by Fabien Sanglard</a>) describes the high-level architecture of Duke Nukem 3D:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO9rb-TlPqT31TT8HFmPsASp6P8cX1mEBZ50XGOY5xvuBK6We5wY0mgnE0krjnQP_ea5ETeokVmIIy-wc3WD38WElntLo2CHlrjoNj0AhomJ8QyePxmmjySzwVYW7rOr0Otj_gCS0n7S0tadEDKkOovGlRxiIEDj_95yfydwGeL4ZI-A-4umTOJUwhEg/s629/duke_nukem_dev_team.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="349" data-original-width="629" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiO9rb-TlPqT31TT8HFmPsASp6P8cX1mEBZ50XGOY5xvuBK6We5wY0mgnE0krjnQP_ea5ETeokVmIIy-wc3WD38WElntLo2CHlrjoNj0AhomJ8QyePxmmjySzwVYW7rOr0Otj_gCS0n7S0tadEDKkOovGlRxiIEDj_95yfydwGeL4ZI-A-4umTOJUwhEg/s600/duke_nukem_dev_team.png"/></a></div>
<br />
In the above diagram, the BUILD engine (on the right) is general purpose component developed by <a href="http://advsys.net/ken/">Ken Silverman</a> (the author of the BUILD engine and editor) and shipped as a header and object code file to 3D Realms. 3D Realms combines the engine with the game artifacts on the left to construct a game executable (<i>DUKE3D.EXE</i>).<br />
<br />
To configure game effects in the BUILD editor, you need to annotate objects (walls, sprites and sectors) with tags and add special purpose sprites to the map. To the editor these objects are just meta-data, but the game engine treats them as parameters to create special effects.<br />
<br />
Every object in a map can be annotated with meta data properties called <strong>Lotags</strong> and <strong>Hitags</strong> storing a 16-bit numeric value (by using the Alt+T and Alt+H key combinations in 2D mode).<br />
<br />
In Shadow Warrior, the tag system was extended even further -- in addition to Lotags and Hitags, objects can potentially have 15 numerical tags (TAG1 corresponds to the Hitag, and TAG2 to the Lotag) and 11 boolean tags (BOOL1-BOOL11). In 2D mode, these can be configured with the ' and ; keys in combination with a numeric key (0-9).<br />
<br />
We can also use special purpose sprites that are visible in the editor, but hidden in the game:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkCZsvNjkrnrq1tLvdfO14O2jQHt5k2kA-6mmM_EvfxVQ81qOZ3yVFEYzeCg-ET5W8ziR-yLXGHLpFVRQJxKwvf5lNKKaug-MI7fF53ax92AmK440SRH68xfy68VxiMHOSr_Am8xvDs1w_GBs5bt3ANdmCnLrQMWxVnNBq9uJgGT0fkNiEGYPHo9Mwfw/s640/st1.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhkCZsvNjkrnrq1tLvdfO14O2jQHt5k2kA-6mmM_EvfxVQ81qOZ3yVFEYzeCg-ET5W8ziR-yLXGHLpFVRQJxKwvf5lNKKaug-MI7fF53ax92AmK440SRH68xfy68VxiMHOSr_Am8xvDs1w_GBs5bt3ANdmCnLrQMWxVnNBq9uJgGT0fkNiEGYPHo9Mwfw/s600/st1.png"/></a></div>
<br />
In the above screenshot of my Shadow Warrior map, there are multiple special purpose sprites visible: the ST1 sprites (that can be used to control all kinds of effects, such as moving a door). ST1 sprites are visible in the editor, but not in the game.<br />
<br />
Although both games use the same principles for configuring game effects, their game mechanics are completely different.<br />
<br />
In the next sections, I will show all the relevant game effects in my Duke Nukem 3D map and explain how I translated them to Shadow Warrior.<br />
<br />
<h3>Differences in conventions</h3>
<br />
As explained earlier, both games frequently use Lotags and Hitags to create effects.<br />
<br />
In Duke Nukem 3D, a Lotag value typically determines the kind of effect, while an Hitag value is used as a <strong>match tag</strong> to group certain events together. For example, multiple doors can be triggered by the same switch by using the same match tag.<br />
<br />
Shadow Warrior uses the opposite convention -- a Hitag value typically determines the effect, while a Lotag value is often used as a match tag.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgScvRZjTBoyQgW40LSN175bu8pE9xqDvoCXI9J7DFY6fQHyHrJKH0t2kGtEXIBBXkyLpB0irKDBnkUWVJFiXLgofAYiNKvYaIodOWxuIDTSfDg4seEt98RJ-DSjsXnYU9YbdS7S4zkgq639FHDYdk40KdjDvWZ97Aubra10FUjI-avMAE7ym94po0mTw/s680/specialpurposesprites.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="520" data-original-width="680" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgScvRZjTBoyQgW40LSN175bu8pE9xqDvoCXI9J7DFY6fQHyHrJKH0t2kGtEXIBBXkyLpB0irKDBnkUWVJFiXLgofAYiNKvYaIodOWxuIDTSfDg4seEt98RJ-DSjsXnYU9YbdS7S4zkgq639FHDYdk40KdjDvWZ97Aubra10FUjI-avMAE7ym94po0mTw/s600/specialpurposesprites.png"/></a></div>
<br />
Furthermore, in Duke Nukem 3D there are many kinds of special purpose sprites, as shown in the screenshot above. The S-symbol sprite is called a Sector Effector that determines the kind of effect that a sector has, the M-symbol is a MUSIC&SFX sprite used to configure a sound for a certain event, and a GPSPEED sprite determines the speed of an effect.<br />
<br />
Shadow Warrior has fewer special purpose sprites. In almost all cases, we end up using the ST1 sprite (with index 2307) for the configuration of an effect.<br />
<br />
ST1 sprites typically combine multiple interactivity properties. For example, to make a sector a door, that opens slowly, produces a sound effect and that closes automatically, we need to use three Sector Effector sprites and one GPSPEED sprite in Duke Nukem 3D. In Shadow Warrior, the same is accomplished by only using two ST1 sprites.<br />
<br />
The fact that the upgraded BUILD engine in Shadow Warrior makes it possible to change more than two numerical tags (and boolean values), makes it possible to combine several kinds of functionality into one sprite.<br />
<br />
<h3>Co-op respawn points</h3>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBPOHFoShM82QiLtFIrRn4M5IlNfE765FEUx8__THu40CvchgPWpJTeurEJqcZoC0Qv47ACInR2v2KnfjetS87YMIhBSxfrNrAgcEkduROoVdhfGKlb_7_DJdezsi86SNvZuGASyun85_AkBwv9WMu0-MX8ppzalE19ngP9G5BbXxmXbjVax28DNf58A/s640/coop.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBPOHFoShM82QiLtFIrRn4M5IlNfE765FEUx8__THu40CvchgPWpJTeurEJqcZoC0Qv47ACInR2v2KnfjetS87YMIhBSxfrNrAgcEkduROoVdhfGKlb_7_DJdezsi86SNvZuGASyun85_AkBwv9WMu0-MX8ppzalE19ngP9G5BbXxmXbjVax28DNf58A/s600/coop.png"/></a></div>
<br />
To make it possible to play a multiplayer cooperative game, you need to add co-op respawn points to your map. In Duke Nukem 3D, this can be done by adding seven sprites with texture 1405 and setting the Lotag value of the sprites to 1. Furthermore, the player's respawn point is also automatically a co-op respawn point.<br />
<Br />
In Shadow Warrior, co-op respawn points can be configured by adding ST1 sprites with Hitag 48. You need eight of them, because the player's starting point is not a co-op start point. Each respawn point requires a unique Lotag value (a value between 0 and 7).<br />
<br />
<h3>Duke match/Wang Bang respawn points</h3>
<br />
For the other multiplayer game mode: Duke match/Wang Bang, we also need re-spawn points. In both games the process is similar to their co-op counterparts -- in Duke Nukem 3D, you need to add seven sprites with texture 1405, and set the Lotag value to 0. Moreover, the player's respawn point is also a Duke Match respawn point.<br />
<br />
In Shadow Warrior, we need to use ST1 sprites with a Hitag value of 42. You need eight of them and give each of them a unique Lotag value between 0-7 -- the player's respawn point is not a Wang Bang respawn point.<br />
<br />
<h3>Underwater areas</h3>
<br />
As explained earlier, the BUILD engine makes it possible to have overlapping sectors, but they cannot be observed simultaneously in 3D mode -- as such, it is not possible to natively provide a room over room experience, although there are some tricks to cope with that limitation.<br />
<br />
In both games it is possible to dive into the water and swim in underwater areas, giving the player some form of a room over room experience. The trick is that the BUILD engine does not render both sectors. When you dive into the water or surface again, you get teleported from one sector to another sector in the map.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib2fSzqFvCygDKWBFXY5w_aWMGVJpzSvo15jaMgpKWU-c8JmaG5k3caDimMNEALIc5xOlVzJ5UY5BBUBi_HPD_3ktjqMPJvlSNkHhp0EKsNxFQ25DPN0w11kmFZjE6z5Mec4Fn2igpyvZ3uGycF4_yvzgMkikWirKyNpyz5NiSeZaZNJd7QSuCOGeCLA/s640/d3d_upperarea.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEib2fSzqFvCygDKWBFXY5w_aWMGVJpzSvo15jaMgpKWU-c8JmaG5k3caDimMNEALIc5xOlVzJ5UY5BBUBi_HPD_3ktjqMPJvlSNkHhp0EKsNxFQ25DPN0w11kmFZjE6z5Mec4Fn2igpyvZ3uGycF4_yvzgMkikWirKyNpyz5NiSeZaZNJd7QSuCOGeCLA/s400/d3d_upperarea.png"/></a></div>
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk70qf1y8OZejddiujCIrBkhuBn8stSMXO-TGu6VmpVavwjJnt8DWVgCrEcSSCtLmSHxAs4DRo0LPCW_uGESLWmArEJV0Z4cCtPCSEjEy1wliQn6VGPVy4bOx3vqvrfjHjwaohR4_jX3ZQWtXMhefXkepDQS6ZsEV4vWYfQ5IohwNBo6tbDssgnF5dew/s640/d3d_lowerarea.png" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhk70qf1y8OZejddiujCIrBkhuBn8stSMXO-TGu6VmpVavwjJnt8DWVgCrEcSSCtLmSHxAs4DRo0LPCW_uGESLWmArEJV0Z4cCtPCSEjEy1wliQn6VGPVy4bOx3vqvrfjHjwaohR4_jX3ZQWtXMhefXkepDQS6ZsEV4vWYfQ5IohwNBo6tbDssgnF5dew/s400/d3d_lowerarea.png"/></a></div>
<div style="clear: both;"></div>
<br />
Although both games use a similar kind of teleportation concept for underwater areas, they are configured in a slightly different way.<br />
<br />
In both games, you need the ability to sink into the water in the upper area. In Duke Nukem 3D, the player automatically sinks by giving the sector a Lotag value of 1. In Shadow Warrior, you need to add a ST1 sprite with a Hitag value of 0, and a Lotag value that determines how much the player will sink. 40 is typically a good value for water areas.<br />
<br />
The underwater sector in Duke Nukem 3D needs a Lotag value of 2. In the game, the player will automatically swim when it enters the sector and the colors will be turned blue-ish.<br />
<br />
We also need to determine from what position in a sector a player will teleport. Both the upper and lower sector should have the same 2 dimensional shape. In Duke Nukem 3D, teleportation can be specified by two Sector Effector sprites having a Lotag 7. These sprites need to be exactly in the same position in the upper and lower sectors. The Hitag value (match tag) needs to be the same:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhixUIACDQrU_izYdVkcRI2B9pcfRJjih3UVzMJix7fo5pLKAIRdclnBxSrrMKrGGzEciLJcaFh3PRI5e79vs6_5WgeQzbTzSiReTviJtH3ULtfzCxsQGJLMkoFs4c6WndSt1Vk6PMEdCfZoxL5bIaL86hI6e20HxWYEPvj7kw0UrWsNCzQWLv2R8WLQw/s680/watersurface.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="520" data-original-width="680" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhixUIACDQrU_izYdVkcRI2B9pcfRJjih3UVzMJix7fo5pLKAIRdclnBxSrrMKrGGzEciLJcaFh3PRI5e79vs6_5WgeQzbTzSiReTviJtH3ULtfzCxsQGJLMkoFs4c6WndSt1Vk6PMEdCfZoxL5bIaL86hI6e20HxWYEPvj7kw0UrWsNCzQWLv2R8WLQw/s600/watersurface.png"/></a></div>
<br />
In the screenshot above, we should see a 2D grid with two Sector Effector sprites having a Lotag of 7 (teleporter) and unique match tags (110 and 111). Both the upper and underwater sectors have exactly the same 2-dimensional shape.<br />
<br />
In Shadow Warrior, teleportation is also controlled by sprites that should be in exactly the same position in the upper and lower sectors.<br />
<br />
In the upper area, we need an ST1 sprite with a Hitag value of 7 and a unique Lotag value. In the underwater area, we need an ST1 sprite with an Hitag value of 8 and the same match Lotag. The latter ST1 sprite (with Hitag 8) automatically lets the player swim. If the player is an under water area where he can not surface, the match Lotag value should be 0.<br />
<br />
In Duke Nukem 3D the landscape will automatically look blue-ish in an underwater area. To make the landscape look blue-ish in Shadow Warrior, we need to adjust the palette of the walls, floors and ceilings from 0 to 9.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikbrmpj94XRkH8RMMXh0wuc5ebPWEyFf3TBpxQ15v_PeIz0dbVX9cj_OCx-PyrlQns5mJwn869mOgm-WguzCvFlLJM_igI9IDLHOgtuifjGXnEYftPmeWHPTGzNx-YEKpuWIvdsn0W4Pv1DXvmpZ6w50XeSBfouA7r57DcP1KuSYglTXeHQnNtgmH9rQ/s640/sw_lowerarea.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEikbrmpj94XRkH8RMMXh0wuc5ebPWEyFf3TBpxQ15v_PeIz0dbVX9cj_OCx-PyrlQns5mJwn869mOgm-WguzCvFlLJM_igI9IDLHOgtuifjGXnEYftPmeWHPTGzNx-YEKpuWIvdsn0W4Pv1DXvmpZ6w50XeSBfouA7r57DcP1KuSYglTXeHQnNtgmH9rQ/s600/sw_lowerarea.png"/></a></div>
<br />
<h3>Garage doors</h3>
<br />
In my map, I commonly use garage/DOOM-style doors that move up when you touch them.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEpLnmjf-NdOEPsgzUElgbLCfvgDHhajgT4DXX8oeWjPDwDVrtqOmCR_iuXFJbFuMorjMKwoPlQInrgTKr4yioI2a66N8yyJLetjp7DGoUYirTBW_u3PUd4yDoOcoHTwkxnWjTTunGZUAr1xQMgAsxP13praW76W9YSkhQG4hAw7gfNnlIYM16M1uxeA/s640/d3d_garagedoor.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhEpLnmjf-NdOEPsgzUElgbLCfvgDHhajgT4DXX8oeWjPDwDVrtqOmCR_iuXFJbFuMorjMKwoPlQInrgTKr4yioI2a66N8yyJLetjp7DGoUYirTBW_u3PUd4yDoOcoHTwkxnWjTTunGZUAr1xQMgAsxP13praW76W9YSkhQG4hAw7gfNnlIYM16M1uxeA/s600/d3d_garagedoor.png"/></a></div>
<br />
In Duke Nukem 3D, we can turn a sector into a garage door by giving it a Lotag value of 20 and lowering the ceiling in such a way that it touches the floor. By default, opening a door does not produce any sound. Moreover, a door will not close automatically.<br />
<br />
We can adjust that behaviour by placing two special purpose sprites in the door sector:<br />
<br />
<ul>
<li>By adding a MUSIC&SFX sprite we can play a sound. The Lotag value indicates the sound number. 166 is typically a good sound.</li>
<li>To automatically close the door after a certain time interval, we need to add a Sector Effector sprite with Lotag 10. The Hitag indicates the time interval. For many doors, 100 is a good value.</li>
</ul>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCcngm2SmiWa8KCx9_p2nOgszlwKH05uyPhr7rjxS2dt_Q2GZnVOzH80-mTOWKd33YVKizV0S_RCizpOYEvdgigjOxMZt3DiyzK3siCs_G5toJ0Fv70huR18kEzC7IXvWD4Gs7dLxuuvjN8AHLM1RKYYqq7eYcLEGnoEpeDnwMHDqm8KERdJNylujRmQ/s640/d3d_garagedoor_open.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiCcngm2SmiWa8KCx9_p2nOgszlwKH05uyPhr7rjxS2dt_Q2GZnVOzH80-mTOWKd33YVKizV0S_RCizpOYEvdgigjOxMZt3DiyzK3siCs_G5toJ0Fv70huR18kEzC7IXvWD4Gs7dLxuuvjN8AHLM1RKYYqq7eYcLEGnoEpeDnwMHDqm8KERdJNylujRmQ/s600/d3d_garagedoor_open.png"/></a></div>
<br />
In the above screenshot, we can see what the garage door looks like if I slightly move the ceiling up (normally the ceiling should touch the floor). There is both a MUSIC&SFX (to give it a sound effect) as well as a Sector Effector sprite (to ensure that the door gets closed automatically) in the door sector.<br />
<br />
In Shadow Warrior, we can accomplish the same thing by adding an ST1 sprite to the door sector with Hitag 92 (Vator). A vator is a multifunctional concept that can be used to move sectors up and down in all kinds of interesting ways.<br />
<br />
An auto closing garage door can be configured by giving the ST1 sprite the following tag and boolean values:<br />
<br />
<ul>
<li>TAG2 (Lotag). Is a match that that should refer to a unique numeric value</li>
<li>TAG3 specifies the type of vator. 0 indicates that it is operated manually or by a switch/trigger</li>
<li>TAG4 (angle) specifies the speed of the vator. 350 is a reasonable value.</li>
<li>TAG9 specifies the auto return time. 35 is a reasonable value.</li>
<li>BOOL1 specifies whether the door should be opened by default. Setting it to 1 (true) allows us to keep the door open in the editor, rather than moving the ceiling down so that it touches the floor.</li>
<li>BOOL3 specifies whether the door could crush the player. We set it to 1 to prevent this from happening.</li>
</ul>
<br />
By default, a vator moves a sector down on first use. To make the door move up, we must rotate the ST1 sprite twice in 3D mode (by pressing the F key twice).<br />
<br />
We can configure a sound effect by placing another ST1 sprite near the door sector with a Hitag value of 134. We can use TAG4 (angle) to specify the sound number. 473 is a good value for many doors.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbubuz7QbPeO8M0Q0Ng2ex-aHULkUyj2wsnaeZBzk3008i_VMe-x_x8QIKG3wFNZrbNEg4HUjyrryDlnVKtSFeq3XaEFcIrBeXsS22jl-FhD-2tV5nZHdmTkanFD6unm0XDJz7f335ZhzGtkumG5vJL85DUBFs_smRpH2mcfND7s75d6e-kHWEAwhXgQ/s640/sw_garagedoor.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjbubuz7QbPeO8M0Q0Ng2ex-aHULkUyj2wsnaeZBzk3008i_VMe-x_x8QIKG3wFNZrbNEg4HUjyrryDlnVKtSFeq3XaEFcIrBeXsS22jl-FhD-2tV5nZHdmTkanFD6unm0XDJz7f335ZhzGtkumG5vJL85DUBFs_smRpH2mcfND7s75d6e-kHWEAwhXgQ/s600/sw_garagedoor.png"/></a></div>
<br />
In the above screenshot, we should see what a garage door looks like in Shadow Warrior. The rotated ST1 sprite defines the Vator whereas the regular ST1 provides the sound effect.<br />
<br />
<h3>Lifts</h3>
<br />
Another prominent feature of my Duke Nukem 3D map are lifts that allow the player to reach the top or roofs of the buildings.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRHrEn1agTaHDQpuE9Fmd0nEsG41qDRXY7ffsHxTQ1yaHywN8xiQmExaDpz7TL5hQfEnnGZ8t0yw3sHVB1iKRldf1RmMQOTPfeaXNDIqYJr2hteBbs8lKCszBr3e1Iynh4uHtbdG5YicYbLw0rwmLZ3H6_3ybLZ82P-zp4Eb-JPna5_F9MooJGtu34XA/s640/d3d_lift.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjRHrEn1agTaHDQpuE9Fmd0nEsG41qDRXY7ffsHxTQ1yaHywN8xiQmExaDpz7TL5hQfEnnGZ8t0yw3sHVB1iKRldf1RmMQOTPfeaXNDIqYJr2hteBbs8lKCszBr3e1Iynh4uHtbdG5YicYbLw0rwmLZ3H6_3ybLZ82P-zp4Eb-JPna5_F9MooJGtu34XA/s600/d3d_lift.png"/></a></div>
<br />
In Duke Nukem 3D, lift mechanics are a fairly concept -- we should give a sector a Lotag value of 17 and the sector will automatically move up or down when the player presses the use key while standing in the sector. The Hitag of a MUSIC&SFX sprite determines the stop sound and a Lotag value the start sound.<br />
<br />
In Shadow Warrior, there is no direct equivalent of the same lift concept, but we can create a switch-operated lift by using the Vator concept (the same ST1 sprite with Hitag 92 used for garage doors) with the following properties:<br />
<br />
<ul>
<li>TAG2 (Lotag) should refer to a unique match tag value. The switches should use the exact same value.</li>
<li>TAG3 determines the type of vator. 1 is used to indicate that it can only be operated by switches.</li>
<li>TAG4 (Angle) determines the speed of the vator. 325 is a reasonable value.</li>
</ul>
<br />
We have to move the ST1 sprite to the same height where the lift should arrive after it was moved up.<br />
<br />
Since it is not possible to respond to the use key while the player is standing in the sector, we have to add switches to control the lift. A possible switch is sprite number 575. The Hitag should match the Lotag value of the ST1 sprite. The switch sprite should have a Lotag value of 206 to indicate that it controls a Vator.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQa_8gi0ClJJPQDxEZ8iPgm7A8g3CpEEUtlo_Sr20GAt9kglsWQ1hzPeRCfTZeSxZvB16kB3BF7LuE2JoRUnkornylk-KSX9nbeXa14HmI_lMnae1Y7aBTosAHpUDACFPBeQfuuwmVkygECk978XgLJICtKpb7I64_qc7CENPR4u0fcjYo_mGjCk9cSg/s640/sw_lift.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgQa_8gi0ClJJPQDxEZ8iPgm7A8g3CpEEUtlo_Sr20GAt9kglsWQ1hzPeRCfTZeSxZvB16kB3BF7LuE2JoRUnkornylk-KSX9nbeXa14HmI_lMnae1Y7aBTosAHpUDACFPBeQfuuwmVkygECk978XgLJICtKpb7I64_qc7CENPR4u0fcjYo_mGjCk9cSg/s600/sw_lift.png"/></a></div>
<br />
The above screenshot shows the result of my porting effort -- switches have been added, the MUSIC&SFX sprite was replaced by an equivalent ST1 sprite. The ST1 sprite that controls the movement is not visible because it was moved up to the same height as the adjacent upper floor.<br />
<br />
<h3>Swinging doors</h3>
<br />
In addition to garage doors, my level also contains a number of swinging doors.<br />
<br />
In Duke Nukem 3D, a sector can be turned into a swinging door by giving it a Lotag of 23 and moving the floor up a bit. We also need to add a Sector Effector with Lotag 11 and a unique Hitag value that acts as the door's pivot.<br />
<br />
As with garage doors, they will not produce any sound effects or close automatically by default, unless we add a MUSIC&SFX and a Sector Effector sprite (with Lotag 10) to the door sector.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm8gpGcVPyrW09-Th6JQo45QBxobm380MAR-EWa0F9gWT0MU8BD7MLAy1W6TbN6t_9e7Yf8-ihUhatNAzgcovVaRJ8M_BEyxnolmm3LAdMWMxPrZWX79Q0vpmyR5ZUK4RQyzSH_tX5ODEws-lmlFUoTfuWi1jfNuexIFttOzkXZ9AngbaNvVK0oFLWIQ/s640/d3d_swingdoor.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjm8gpGcVPyrW09-Th6JQo45QBxobm380MAR-EWa0F9gWT0MU8BD7MLAy1W6TbN6t_9e7Yf8-ihUhatNAzgcovVaRJ8M_BEyxnolmm3LAdMWMxPrZWX79Q0vpmyR5ZUK4RQyzSH_tX5ODEws-lmlFUoTfuWi1jfNuexIFttOzkXZ9AngbaNvVK0oFLWIQ/s600/d3d_swingdoor.png"/></a></div>
<br />
In Shadow Warrior, the rotating door concept is almost the same. We need to add an ST1 sprite with Hitag 144 and a unique Lotag value to the sector that acts as the door's pivot.<br />
<br />
In addition, we need to add an ST1 sprite to the sector that configures a rotator:<br />
<br />
<ul>
<li>TAG2/Lotag determines a unique match tag value that should be identical to the door's pivot ST1 sprite.</li>
<li>TAG3 determines the type of rotator. 0 indicates that it can be manually triggered or by a switch.</li>
<li>TAG5 determines the angle move amount. 512 specifies that it should move 90 degrees to the right. -512 is moving the door 90 degrees to the left.</li>
<li>TAG7 specifies the angle increment. 50 is a good value.</li>
<li>TAG9 specifies the auto return time. 35 is a good value.</li>
</ul>
<br />
As with garage doors, we also need to add an ST1 sprite (with Hitag 134) to produce a sound. TAG4 (the angle) can be used to specify the sound number. 170 is a good value for rotating doors.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuN3APhO2GhJbGnjqIvi2dkWVU81NXWM3ltWkrJmdAoARbddZdnHzkWB1iD89IM7k9QD8QgMXWYgBjyU7RQukqLIKdKxEK4aa9pjry5Lb3g9ENGebAtPUIY5Mo3h3AZ2g58YG0kjeFNleqSvbZ2nE-AFG8Pzz7PEsEVffjsiGf_ZMbA1YFe183P9DQVw/s640/sw_swingdoor.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhuN3APhO2GhJbGnjqIvi2dkWVU81NXWM3ltWkrJmdAoARbddZdnHzkWB1iD89IM7k9QD8QgMXWYgBjyU7RQukqLIKdKxEK4aa9pjry5Lb3g9ENGebAtPUIY5Mo3h3AZ2g58YG0kjeFNleqSvbZ2nE-AFG8Pzz7PEsEVffjsiGf_ZMbA1YFe183P9DQVw/s600/sw_swingdoor.png"/></a></div>
<br />
<h3>Secret places</h3>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRHjbV1FPh8Rk30hHeMsB9ldcJJ143ozkfnjc0z6oldUPbFWt1Af0P3R8jG51C6vD_g7x-N3LNkwp2swybXGrI0FoG23YIp3-OwzB7uvlusc8HZywuZ16-UUinBPq1X_n_RaWV23EuBsIyPF0N42_C7xrDSObPIZDNr-NJo6fXc_8gfMtFlshXotra3A/s640/secretplace.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiRHjbV1FPh8Rk30hHeMsB9ldcJJ143ozkfnjc0z6oldUPbFWt1Af0P3R8jG51C6vD_g7x-N3LNkwp2swybXGrI0FoG23YIp3-OwzB7uvlusc8HZywuZ16-UUinBPq1X_n_RaWV23EuBsIyPF0N42_C7xrDSObPIZDNr-NJo6fXc_8gfMtFlshXotra3A/s600/secretplace.png"/></a></div>
<br />
My map also has a number of secret places (please do not tell anyone :-) ). In Duke Nukem 3D, any sector that has a Lotag value of <i>32767</i> is considered a secret place. In Shadow Warrior the idea is the same -- any sector with a Lotag of 217 is considered a secret place.<br />
<br />
<h3>Puzzle switches</h3>
<br />
Some Duke Nukem 3D maps also have so-called puzzle switches requiring the player to find the correct on-and-off combination to unlock something. In my map they are scattered all over the level to unlock the final key. The E2L1 map in Duke Nukem 3D shows a better example:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYCGBToCI0LMiuuy5v47VJNwGY2ma5VVXOo40GyUqcZPdPEltuGtflcnq3DivV6B6X12h-SMHK51nWQ_YdAg3f7xIVVIVQH1yA4STBhZB6NrEjj2OKYY9IegRw5aZHuY82hSrAKcDz6R6g1d0Z_2G8QIFq4iPVmfQPHy1Ex0ZFvvmdkh2BBg5jdt_dxQ/s640/puzzle.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiYCGBToCI0LMiuuy5v47VJNwGY2ma5VVXOo40GyUqcZPdPEltuGtflcnq3DivV6B6X12h-SMHK51nWQ_YdAg3f7xIVVIVQH1yA4STBhZB6NrEjj2OKYY9IegRw5aZHuY82hSrAKcDz6R6g1d0Z_2G8QIFq4iPVmfQPHy1Ex0ZFvvmdkh2BBg5jdt_dxQ/s600/puzzle.png"/></a></div>
<br />
We can use the Hitag value to determine whether the switch needs to be switched off (0) or on (1). We can use the Lotag as a match tag to group multiple switches.<br />
<br />
In Shadow Warrior, each switch uses a Hitag as a match tag and a Lotag value to configure the switch type. Giving a switch a Lotag value of 213 makes it a combo switch. TAG3 can be used set to 0 to indicate that it needs to be turned off and 1 that it needs to be turned on.<br />
<br />
<h3>Skill settings</h3>
<br />
Both games have four skill levels. The idea is that the higher the skill level is, the more monsters you will have to face.<br />
<br />
In Duke Nukem 3D you can specify the minimum skill level of a monster by giving the sprite a Lotag value that corresponds to the minimum skill level. For example, giving a monster a Lotag value of 2 means that it will only show up when the skill level is two or higher (Skill level 2 corresponds to: Let's rock). 0 (the default value) means that it will show up in any skill level:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimTINDybsPAAZ4AKZC3iJesrLQzecUykOwhTl95whcbWTrBa7MWxa4aBKR5TX3q2srpboX_tcOqoyZ0ashF3OkFkVqecGS-at3rVvREGbJo1bw0o101fgm2d1qWFDz0g48rJwqkZWVHQU0e2E87ANUEU_Tt9nSkHrIBl2mUZy1n41EMY25s2BlhMUkDQ/s680/d3d_skills.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="520" data-original-width="680" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEimTINDybsPAAZ4AKZC3iJesrLQzecUykOwhTl95whcbWTrBa7MWxa4aBKR5TX3q2srpboX_tcOqoyZ0ashF3OkFkVqecGS-at3rVvREGbJo1bw0o101fgm2d1qWFDz0g48rJwqkZWVHQU0e2E87ANUEU_Tt9nSkHrIBl2mUZy1n41EMY25s2BlhMUkDQ/s600/d3d_skills.png"/></a></div>
<br />
In Shadow Warrior, each sprite has its own dedicated skill attribute that can be set by using the key combination ' + K. The skill level is displayed as one of the sprite's attributes.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzuLqSlUDbzeXojtoQ_qV6DvCogHh1EtULCyI7z1kgnV8O1SvVV9g6N-xxTZMEkHWd9j1oiQTpF1CFeO0YqPsKZ4ODgTRGVzP_HM0P9XOVFt1W2oQb2GnjE4p3dp2AsBkutmTJ1R-xY0yC9eOisSzQwJN10UhQvDUefKL0ZrjKRZb-K6bsuesHVokGSg/s640/sw_skills.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgzuLqSlUDbzeXojtoQ_qV6DvCogHh1EtULCyI7z1kgnV8O1SvVV9g6N-xxTZMEkHWd9j1oiQTpF1CFeO0YqPsKZ4ODgTRGVzP_HM0P9XOVFt1W2oQb2GnjE4p3dp2AsBkutmTJ1R-xY0yC9eOisSzQwJN10UhQvDUefKL0ZrjKRZb-K6bsuesHVokGSg/s600/sw_skills.png"/></a></div>
<br />
In the above screenshot, the sprite on the left has a <i>S:0</i> prefix meaning that it will be visible in skill level 0 or higher. The sprite on the right (with a prefix: <i>S:2</i>) appears from skill level 2 or higher.<br />
<br />
<h3>End switch</h3>
<br />
In both games, you typically complete a level by touching a so-called end switch. In Duke Nukem 3D an ending switch can be created by using sprite 142 and giving it a Lotag of 65535. In Shadow Warrior the idea is the same -- we can create an end switch by using sprite 2470 and giving it a Lotag of 116.<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8mKCzKB0Z0l9WGc91V2IyOfJmlULAimSJgj-2cNg1qv-7_kL5HobNJ7kWI7f5jeDwRMOin3cSYO6AkZEM_SadcXF0eKGuFtDYELEOYFMcxq0xOwSA1RCm0ezElyowECZTo4Obo_3y1UBq_MSLaIP-jAxqsGd3326swTbdBtvoEzVnt-FoUZn85nfkaA/s640/d3d_end.png" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg8mKCzKB0Z0l9WGc91V2IyOfJmlULAimSJgj-2cNg1qv-7_kL5HobNJ7kWI7f5jeDwRMOin3cSYO6AkZEM_SadcXF0eKGuFtDYELEOYFMcxq0xOwSA1RCm0ezElyowECZTo4Obo_3y1UBq_MSLaIP-jAxqsGd3326swTbdBtvoEzVnt-FoUZn85nfkaA/s400/d3d_end.png"/></a></div>
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghQvlOTm9PYOiU7yGKmZ4oxYizil_moaXY0yl9giDqPbJhhBpUC9dmJYTh9Bxi2ivo2VR3uZfHLrrmEI4z6IKrdnLUx2KmNrb1k_NAwtdiLNM-PLpWva2sqCNU0Dd7LQMPScPDu8nb0lPyjl0xHM8yvP_foQr8rQh4qSuc--ZLUj2XRj56SBJvuOuTjw/s640/sw_end.png" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" width="250" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEghQvlOTm9PYOiU7yGKmZ4oxYizil_moaXY0yl9giDqPbJhhBpUC9dmJYTh9Bxi2ivo2VR3uZfHLrrmEI4z6IKrdnLUx2KmNrb1k_NAwtdiLNM-PLpWva2sqCNU0Dd7LQMPScPDu8nb0lPyjl0xHM8yvP_foQr8rQh4qSuc--ZLUj2XRj56SBJvuOuTjw/s400/sw_end.png"/></a></div>
<div style="clear: both;"></div>
<br />
<h2>Conclusion</h2>
<br />
In this blog post, I have described the porting process of a Duke Nukem 3D map to Shadow Warrior and explained some of the properties that are common and different in both games.<br />
<br />
Although this project was a pretty useless project (the game is quite old, from the late 90s), I had a lot of fun doing it after not having touched this kind of technology for over a decade. I am quite happy with the result:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb-mK-bG_h79kJF1C6mIUDfoG0N08UiMseHL5E8DX54dRdQ269oKfMuIZoNe6cp0fG-gyQ_1y3FpQb7qxewQ0LNL-YtY_hW1hZ_XGXTrHAc4wJRy7HNI1qJuZ5Vlb1UI8NVcqUgGSeZnymEvFDhEQrZx9snco6wssGNIwCy5uviA61zlPiLEmD2xg0Kw/s640/portedmap.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="640" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhb-mK-bG_h79kJF1C6mIUDfoG0N08UiMseHL5E8DX54dRdQ269oKfMuIZoNe6cp0fG-gyQ_1y3FpQb7qxewQ0LNL-YtY_hW1hZ_XGXTrHAc4wJRy7HNI1qJuZ5Vlb1UI8NVcqUgGSeZnymEvFDhEQrZx9snco6wssGNIwCy5uviA61zlPiLEmD2xg0Kw/s600/portedmap.png"/></a></div>
<br />
Despite the fact that this technology is old, I am still quite surprised to see how many maps and customizations are still being developed for <a href="https://www.moddb.com/games/duke-nukem-3d">these</a> ancient <a href="https://www.moddb.com/games/shadow-warrior/">games</a>. I think this can be attributed to the fact that these engines and game mechanics are highly customizable and still relatively simple to use due to the technical limitations at the time they were developed.<br />
<br />
Since I did most of my mapping/customization work many years before I started this blog, I thought that sharing my current experiences can be useful for others who intend to look at these games and creating their own customizations.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-88435317210058564242022-04-20T12:10:00.000+02:002022-04-20T12:10:18.643+02:00In memoriam: Eelco Visser (1966-2022)On Tuesday 5 April 2022 I received the unfortunate news that my former master's and PhD thesis supervisor: <a href="https://eelcovisser.org">Eelco Visser</a> has unexpectedly passed away.<br />
<br />
Although I made <a href="https://sandervanderburg.blogspot.com/2012/10/my-post-phd-carreer-aka-leaving-academia.html">my transition from academia to industry almost 10 years ago</a> and I have not been actively doing academic research anymore (I published my last paper in 2014, almost two years after completing my PhD), I have always remained (somewhat) connected to the research world and the work carried out by my former supervisor, who started his own <a href="https://pl.ewi.tudelft.nl">programming languages research group</a> in 2013.<br />
<br />
He was very instrumental in the domain-specific programming languages research domain, but also in the software deployment research domain, a very essential part in almost any software development process.<br />
<br />
Without him and his ideas, his former PhD student: Eelco Dolstra would probably never have started the work that resulted in the <a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix package manager</a>. As a consequence, my research on software deployment (resulting in <a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a> and many other Nix-related tools and articles) and this blog would also not exist.<br />
<br />
In this blog post, I will share some memories of my time working with Eelco Visser.<br />
<br />
<h2>How it all started</h2>
<br />
The first time I met Eelco was in 2007 when I was still a MSc student. I just completed the first year of the TU Delft master's programme and I was looking for an assignment for my master's thesis.<br />
<br />
Earlier that year, I was introduced to a concept called <strong>model-driven development</strong> (also ambiguously called model-driven engineering/architecture, the right terminology is open to interpretation) in a guest lecture by Jos Warmer in the software architecture course.<br />
<br />
Modeling software systems and automatically generating code (as much as possible), was one of the aspects that really fascinated me. Back then, I was already convinced that working from a higher abstraction level, with more "accessible" building blocks could be quite useful to hide complexity, reduce the chances on errors and make developers more productive.<br />
<br />
In my first conversation with Eelco, he asked me why I was looking for a model-driven development assignment and he asked me various questions about my past experience.<br />
<br >
I told him about my experiences with Jos Warmer's lecture. Although he seemed to understand my enthusiasm, he also explained me that his work was mostly about creating textual languages, not visual languages such as UML profiles, that are commonly used in MDA development processes.<br />
<br />
He also specifically asked me about the compiler construction course (also part of the master's programme), that is required for essential basic knowledge about textual languages.<br />
<br />
The compiler construction course (as it was taught in 2007) was considered to be very complex by many students, in particular the practical assignment. As a practical assignment, you had to rewrite the a parser from using <a href="https://www.gnu.org/software/bison">GNU Bison</a> (a LARL(1) parser) to <a href="https://os.ghalkes.nl/LLnextgen">LLnextgen</a> (a LL(1) parser) and extend the reference compiler with additional object-oriented programming features. Moreover, the compiler was implemented in C, and relied on advanced concepts, such as function pointers and proper alignment of members in a struct.<br />
<br />
I explained Eelco that despite the negative image of the course because of its complexity, I actually liked it very much. Already at a young age I had the idea to develop my own programming language, but I had no idea how to do it, but when I was exposed to all these tools and concepts I finally learned about all the missing bits and pieces.<br />
<br />
I was also trying to convince him that I am always motivated to deep dive into technical details. As an example, I explained him that one of my personal projects is creating customized Linux distributions by following the <a href="https://linuxfromscratch.org">Linux from Scratch</a> book. Manually following all the instructions in the book is time consuming and difficult to repeat. To make deploying a customized Linux distribution doable, I developed my own automated solution.<br />
<br />
After elaborating about my (somewhat crazy) personal project, he told me that there is an ongoing research project that I will probably like very much. A former PhD student of his: Eelco Dolstra developed the Nix package manager and this package manager was used as the foundation for an entire Linux distribution: <a href="https://sandervanderburg.blogspot.com/2011/01/nixos-purely-functional-linux.html">NixOS</a>.<br />
<br />
He gave me a printed copy of Eelco Dolstra's thesis and convinced me that I should give NixOS a try.<br />
<br />
<h2>Research assignment</h2>
<br />
After reading Eelco Dolstra's PhD thesis and trying out NixOS (that was much more primitive in terms of features compared to today's version), Eelco Visser gave me my first research assignment.<br />
<br />
When he joined Delft University of Technology in 2006 (a year before I met him) as an associate professor, he started working on a new project called: <a href="https://webdsl.org">WebDSL</a>. Previously, most of his work was focused on the development of various kinds meta-tools for creating domain specific languages, such as:<br />
<br />
<ul>
<li><strong>SDF2</strong> is a formalism used to write a lexical and context free syntax. It has many interesting features, such as a module system and scannerless parsing, making it possible to embed a guest language in an host language (that may share the same keywords). SDF2 was originally developed by Eelco Visser for his PhD thesis for the <a href="https://github.com/cwi-swat/meta-environment">ASF+SDF Meta Environment</a>.</li>
<li><strong>ATerm library</strong>. A library that implements the annotated terms format to exchange structured data between tools. SDF2 uses it to encode parse/abstract syntax trees.</li>
<li><strong><a href="http://strategoxt.org">Stratego/XT</a></strong>. A language and toolset for program transformation.</li>
</ul>
<br />
WebDSL was a new step for him, because it is an <strong>application language</strong> (built with the above tools) rather than a meta language.<br />
<br />
With WebDSL, in addition to just building an application language, he also had all kinds of interesting ideas about web application development and how to improve it, such as:<br />
<br />
<ul>
<li><strong>Reducing/eliminating boilerplate code</strong>. Originally WebDSL was implemented with <a href="https://jboss.org">JBoss</a> and the <a href="https://www.seamframework.org">Seam</a> framework using Java as an implementation language, requiring you to write a lot of boilerplate code, such as getters/setters, deployment descriptors etc.<br />
<br />
WebDSL is <a href="https://sandervanderburg.blogspot.com/2016/03/the-nixos-project-and-deploying-systems.html"><strong>declarative</strong></a> in the sense that you could more concisely describe <strong>what</strong> you want in a rich web application: a data model, and pages that should render content and data.</li>
<li><strong>Improving static consistency checking</strong>. Java (the implementation language used for the web applications) is statically typed, but not every concern of a web application can be statically checked. For example, for interacting with a database, embedded SQL queries (in strings) are often not checked. In JSF templates, page references are not checked.<br />
<br />
With WebDSL all these concerns are checked before the deployment of an web application.</li>
</ul>
<br />
By the time I joined, he already assembled several PhD and master's students to work on a variety of aspects of WebDSL and the underlying tooling, such as Stratego/XT.<br />
<br />
Obviously, in a development process of a WebDSL application, like any application, you will also eventually face a <strong>deployment problem</strong> -- you need to perform activities to make the application available for use.<br />
<br />
For solving deployment problems in our department, Nix was already quite intensively used. For example, we had a Nix-based continuous integration service called the Nix buildfarm (several years later, its implementation was re-branded into <a href="https://sandervanderburg.blogspot.com/2013/04/setting-up-hydra-build-cluster-for.html">Hydra</a>), that built all bleeding edge versions of WebDSL, Stratego/XT and all other relevant packages. The Nix package manager was used by all kinds of people in the department to install bleeding edge versions of these tools.<br />
<br />
My research project was to automate the deployment of WebDSL applications using tooling from the Nix project. In my first few months, I have packaged all the infrastructure components that a WebDSL application requires in NixOS (JBoss, MySQL and later Apache Tomcat). I changed WebDSL to use GNU Autotools as build infrastructure (which was a common practice for all Stratego/XT related projects at that time) and made subtle modifications to prevent unnecessary recompilations of WebDSL applications (such as making the root folder dynamically configurable) and wrote an abstraction function to automatically build WAR files.<br />
<br />
Thanks to Eelco I ended up in a really friendly and collaborative atmosphere. I came in touch with his fellow PhD and master's students and we frequently had very good discussions and collaborations.<br />
<br />
Eelco was also quite helpful in the early stages of my research. For example, whenever I was stuck with a challenge he was always quite helpful in discussing the underlying problem and bringing me in touch with people that could help me.<br />
<br />
<h2>My master's thesis project</h2>
<br />
After completing my initial version of the <a href="https://sandervanderburg.blogspot.com/2011/05/deployment-abstractions-for-webdsl.html">WebDSL deployment tool</a>, that got me familiarised with the basics of Nix and NixOS, I started working on my master's thesis which was a collaboration project between Delft University of Technology and Philips Research.<br />
<br />
Thanks to Eelco I came in contact with a former master's thesis student and postdoc of his: Merijn de Jonge who was employed by Philips Research. He was an early contributor to the Nix project and collaborated on the first two research papers about Nix.<br />
<br />
While working on my master's thesis I developed the first prototype version of Disnix.<br />
<br />
During my master's thesis project, Eelco Dolstra, who was formerly a postdoc at Utrecht University joined our research group in Delft. Eelco Visser made sure that I got all the help from Eelco Dolstra about all technical questions about Nix.<br />
<br />
<h2>Becoming a PhD student</h2>
<br />
My master's thesis project was a pilot for a bigger research project. Eelco Visser, Eelco Dolstra and Merijn de Jonge (I was already working quite intensively with them for my master's thesis) were working on a research project proposal. When the proposal got accepted by <a href="https://www.nwo.nl/en/researchprogrammes/jacquard">NWO/Jacquard</a> for funding, Eelco Visser was the first to inform me about the project to ask me what I thought about it.<br />
<br />
At that moment, I was quite surprised to even consider doing a PhD. A year before, I attended somebody's else's PhD defence (someone who I really considered smart and talented) and thought that doing such a thing myself was way out of my grasp.<br />
<br />
I also felt a bit like an impostor because I had interesting ideas about deployment, but I was still in the process of finishing up/proving some of my points.<br />
<br />
Fortunately, thanks to Eelco my attitude completely changed in that year -- during my master's thesis project he convinced me that the work I was doing is relevant. What I also liked is the attitude in our group to actively build tools, have the time and space to explore things, and eat our own dogfood with it to solve relevant practical problems. Moreover, much of the work we did was also publicly available as <a href="https://sandervanderburg.blogspot.com/2011/02/free-and-open-source-software.html">free and open source software</a>.<br />
<br />
As a result, I easily grew accustomed to the research process and the group's atmosphere and it did not take long to make the decision to do a PhD.<br />
<br />
<h2>My PhD</h2>
<br />
Although Eelco Visser only co-authored one of my published papers, he was heavily involved in many aspects of my PhD. There are way too many things to talk about, but there are some nice anecdotes that I really find worth sharing.<br />
<br />
<h3>OOPSLA 2008</h3>
<br />
I still remember the first research conference that I attended: <a href="https://sandervanderburg.blogspot.com/2012/07/a-review-of-conferences-in-2008-2009.html">OOPSLA 2008</a>. I had a very quick publication start, with a paper covering an important aspect of master's thesis: the upgrade aspect for distributed systems. I had to present my work at HotSWUp, an event co-located with OOPSLA 2008.<br />
<br />
(As a sidenote: because we had to put all our efforts in making the deadline, I had to postpone the completion of my master's thesis a bit, so it started overlap with my PhD).<br />
<br />
It was quite an interesting experience, because in addition to the fact that it was my first conference, it was also my first time to travel to the United States and to step into an airplane.<br />
<br />
The trip was basically a group outing -- I was joined by Eelco and many of his PhD students. In addition to my HotSWUp 2008 paper, we also had an OOPSLA paper (about the Dryad compiler), a WebDSL poster, and another paper about the implementation of WebDSL (the paper titled: "When frameworks let you down") to present.<br />
<br />
I was surprised to see how many people Eelco knew at the conference. He was also actively encouraging us to meet up with people and bringing us in touch with people that he know that could be relevant.<br />
<br />
We were having a good time together, but I also remember him saying that it is actually much better to visit a conference alone, rather than in a group. Being alone makes it much easier and more encouraging to meet new people. That lesson stuck and in many future events, I took the advantage of being alone as an opportunity to meet up.<br />
<br />
<h3>Working on practical things</h3>
<br />
Once in a while I had casual discussions with him about ongoing things in my daily work. For my second paper, I had to travel to ICSE 2009 in Vancouver, Canada all by myself (there were some colleagues traveling to co-located events, but took different flights).<br />
<br />
Despite the fact that I was doing research on Nix-related things, NixOS at that time was not my main operating system yet on my laptop because it was missing features that I consider a must-have in a Linux distribution.<br />
<br />
In the weeks before the planned travel date, I was intensively working on getting all the software packaged that I consider important. One major packaging effort was getting KDE 4.2 to work, because I was dissatisfied with only having the KDE 3.5 base package available in NixOS. VirtualBox was another package that I consider critical, so that I could still run a conventional Linux distribution and Microsoft Windows.<br />
<br />
Nothing about this work is considered scientific "research" that may result in a paper that we can publish. Nonetheless, Eelco recognized the value of making NixOS more usable and encouraged me to get all that software packaged. He even asked me: "Are you sure that you have packaged enough software in NixOS so that you can survive that week?"<br />
<br />
<h3>Starting my blog</h3>
<br />
Another particularly helpful advice that he gave me is that I should start a blog. Although I had a very good start of my PhD, having a paper accepted in my first month and another several months later, I slowly ran into numerous paper rejections, with reviews that were not helpful at all.<br />
<br />
I talked to him about my frustrations and explained that software deployment research is generally a neglected subject. There is no research-related conference that is specifically about software deployment (there used to be a <a href="https://dblp.org/db/conf/cd/index.html">working conference on component deployment</a>, but by the time I became a PhD student it was no longer organized), so we always had to "package" our ideas into subjects for different kinds of conferences.<br />
<br />
He gave me the advice to start a blog to increase my interaction with the research community. As a matter of fact, many people in our research group, <a href="https://eelcovisser.org/blog">including Eelco</a>, had their own blogs.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2010/12/first-blog-post.html">It took me some time to take that step</a>. First, I had to "catch up" on my blog with relevant background materials. Eventually, it paid off -- I wrote a blog post titled: <a href="https://sandervanderburg.blogspot.com/2011/10/software-deployment-complexity.html">Software deployment complexity</a> to emphasize software deployment as an important research subject, and thanks to Eelco's Twitter network I came in touch with all kinds of people.<br />
<br />
<h3>Lifecycle management</h3>
<br />
For most of my publication work, I intensively worked with Eelco Dolstra. Eelco Visser left most of the practical supervision to him. The only published paper that we co-authored was: "Software Deployment in a Dynamic Cloud: From Device to Service Orientation in a Hospital Environment".<br />
<br />
There was also a WebDSL-related subject that we intensively worked on for a while, that unfortunately never fully materialized.<br />
<br />
Although I had already had the static aspects of a WebDSL application deployment automated -- the infrastructure components (Apache Tomcat, MySQL) as well as a function to compile a Java Web application Archive (WAR) with the WebDSL compiler, we also had to cope with the data that a WebDSL application stores -- WebDSL data models can evolve, and when this happens, the data needs to be migrated from an old to a new table structure.<br />
<br />
Sander Vermolen, a colleague of mine, worked on a solution to make automated data migrations of WebDSL possible.<br />
<br />
At some point, we came up with the idea to make this all work together -- deployment automation and data migration from a high-level point of view hiding unimportant implementation details. Due to the lack of a better name we called this solution: "lifecycle management".<br />
<br />
Although the project seemed to look straight forward to me in the beginning, I (and probably all of us) heavily underestimated how complex it was to bring Nix's functional deployment properties to data management.<br />
<br />
For example, Nix makes it possible to store multiple variants of the same packages (e.g. old and new versions) simultaneously on a machine without conflicts and makes it possible to cheaply switch between versions. Databases, on the other hand, make imperative modifications. We could manage multiple versions of a database by making snapshots, but doing this atomically and in an portable way is very expensive, in particular when databases are big.<br />
<br />
Fortunately, the project was not a complete failure. I have managed to publish <a href="https://sandervanderburg.blogspot.com/2015/07/deploying-state-with-disnix.html">a paper about a sub set of the problem</a> (automatic data migrations when databases move from one machine to another and a snapshotting plugin system), but the entire solution was never fully implemented.<br />
<br />
During my PhD defence he asked me a couple of questions about this subject, from which (of course!) I understood that it was a bummer that we never fully realized the vision that we initially came up with.<br />
<br />
Retrospectively, we should have divided the problem into a smaller chunks and solve each problem one by one, rather than working on the entire integration right from the start. The integrated solution would probably still consist of many trade-offs, but it still would have been interesting to have come up with at least a solution.<br />
<br />
<h3>PhD thesis</h3>
<br />
When I was about to write my PhD thesis, I was making the bold decision to not compose the chapters directly out of papers, but to write a coherent story using my papers as ingredients, similar to Eelco Dolstra's thesis. Although there are plenty of reasons to think of to not do such a thing (e.g. it takes much more time for a reading committee to review such a thesis), he was actually quite supportive in doing that.<br />
<br />
On the other hand, I was not completely surprised by it, considering the fact that his PhD thesis was several orders of magnitude bigger than mine (over 380 pages!).<br />
<br />
<h2>Spoofax</h2>
<br />
After I completed my PhD, and made my transition to industry, he and his research group relentlessly kept working on the solution ecosystem that I just described.<br />
<br />
Already during my PhD, many improvements and additions were developed that resulted in the <a href="https://www.spoofax.dev">Spoofax language workbench</a>, an Eclipse plugin in which all these technologies come together to make the construction of Domain Specific Languages as convenient as possible. For a (somewhat :-) ) <a href="https://eelcovisser.org/blog/2021/02/08/spoofax-mip/">brief history of the Spoofax language workbench</a> I recommend you to read this blog post written by him.<br />
<br />
Moreover, he also kept dogfooding his own practical problems. During my PhD, three serious applications were created with WebDSL: <a href="https://researchr.org">researchr</a> (a social network for researchers sharing publications), <a href="https://yellowgrass.org">Yellowgrass</a> (an issue tracker) and <a href="https://weblab.tudelft.nl">Weblab</a> (a system to facilitate programming exams). These applications are still maintained and used by the university as of today.<br />
<br />
A couple of months after <a href="https://sandervanderburg.blogspot.com/2013/06/dr-sander.html">my PhD defence</a> in 2013 (I had to wait for several months to get feedback and a date for my defence), he was awarded the prestigious Vici grant and became a full professor, starting his own programming language research group.<br />
<br />
In 2014, when I was already in industry for two years, I was invited for his inauguration ceremony and was given another demonstration of what Spoofax has become. I was really impressed by all the new meta languages that were developed and what Spoofax looked like. For example, SDF2 evolved into SDF3, a new meta-language for developing Name Address bindings (NaBL) was developed etc.<br />
<br />
Moreover, I liked his inauguration speech very much, in which he briefly demonstrated the complexities of computers and programming, and what value domain specific languages can provide.<br />
<br />
<h2>Concluding remarks</h2>
<br />
In this blog post, I have written down some of my memories working with Eelco Visser. I did this in the spirit of my blog, whose original purpose was to augment my research papers with practical information and other research aspects that you normally never read about.<br />
<br />
I am grateful for the five years that we worked together, that he gave me the opportunity to do a PhD with him, for all the support, the things he learned me, and the people who he brought me in touch with. People that I still consider friends as of today.<br />
<br />
My thoughts are with his family, friends, the research community and the entire programming languages group (students, PhD students, Postdocs, and other staff).<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-10284971178179474232022-02-14T22:23:00.003+01:002022-02-14T22:27:22.600+01:00A layout framework experiment in JavaScriptIt has been a while since I wrote a blog post about front-end web technology. The main reason is that I am not extensively doing front-end development anymore, but once in a while I still tinker with it.<br />
<br />
In my Christmas break, I wanted to expand my knowledge about modern JavaScript programming practices. To make the learning process more motivating, I have been digging up <a href="https://sandervanderburg.blogspot.com/2014/03/implementing-consistent-layouts-for.html">my old web layout framework project</a> and ported it to JavaScript.<br />
<br />
In this blog post, I will explain the rationale of the framework and describe the features of the JavaScript version.<br />
<br />
<h2>Background</h2>
<br />
Several years ago, I have elaborated about some of the challenges that I faced while creating layouts for web applications. Although front-end web technology (<a href="https://www.w3.org/html/">HTML</a> and <a href="https://www.w3.org/Style/CSS/Overview.en.html">CSS</a>) were originally created for pages (not graphical user interfaces), most web applications nowadays are <strong>complex information systems</strong> that typically have to present collections of data to end-users in a consistent manner.<br />
<br />
Although some concepts of web technology are powerful and straight forward, a native way to isolate layout from a page's content and style is still virtually non-existent (with the exception of <a href="https://www.w3.org/TR/html4/present/frames.html">frames</a> that have been deprecated a long time ago). As a consequence, it has become quite common to rely on custom <strong>abstractions</strong> and <strong>frameworks</strong> to organize layouts.<br />
<br />
Many years ago, I also found myself repeating the same patterns to implement consistent layouts. To make my life easier, I have developed my own layout framework that allows you to define a <strong>model</strong> of your application layout, that captures common layout properties and all available sub pages and their dynamic content.<br />
<br />
A <strong>view</strong> function can render a requested sub page, using the path in the provided URL as a selector.<br />
<br />
I have created two implementations of the framework: one in <a href="https://openjdk.java.net">Java</a> and another in <a href="https://php.net">PHP</a>. The Java version was the original implementation but I ended up using the PHP version the most, because nearly all of the web applications I developed were hosted at shared web hosting providers only offering PHP as a scripting language.<br />
<br />
Something that I consider both an advantage and disadvantage of my framework is that it has to <strong>generate</strong> pages on the server-side. The advantage of this approach is that pages rendered by the framework will work in many browsers, even primitive text-oriented browsers that lack JavaScript support.<br />
<br />
A disadvantage is that server-side scripting requires a more complex server installation. Although PHP is relatively simple to set up, a Java Servlet container install (such as <a href="https://tomcat.apache.org">Apache Tomcat</a>) is typically more complex. For example, you typically want to put it behind a reverse proxy that serves static content more efficiently.<br />
<br />
Furthermore, executing server-side code for each request is also significantly more expensive (in terms of processing power) than serving static files.<br />
<br />
The interesting aspect of using JavaScript as an implementation language is that we can use the framework both on the client-side (in the browser) as well as on the server-side (with <a href="http://nodejs.org">Node.js</a>). The former aspect makes it possible to host applications on a web servers that only serve static content, making web hosting considerably easier and cheaper.<br />
<br />
<h2>Writing an application model</h2>
<br />
As explained earlier, my layout framework separates the model from a view. An application layout model can be implemented in JavaScript as follows:<br />
<br />
<pre style="font-size: 80%; overflow: auto;">
import { Application } from "js-sblayout/model/Application.mjs";
import { StaticSection } from "js-sblayout/model/section/StaticSection.mjs";
import { MenuSection } from "js-sblayout/model/section/MenuSection.mjs";
import { ContentsSection } from "js-sblayout/model/section/ContentsSection.mjs";
import { StaticContentPage } from "js-sblayout/model/page/StaticContentPage.mjs";
import { HiddenStaticContentPage } from "js-sblayout/model/page/HiddenStaticContentPage.mjs";
import { PageAlias } from "js-sblayout/model/page/PageAlias.mjs";
import { Contents } from "js-sblayout/model/page/content/Contents.mjs";
/* Create an application model */
export const application = new Application(
/* Title */
"My application",
/* Styles */
[ "default.css" ],
/* Sections */
{
header: new StaticSection("header.html"),
menu: new MenuSection(0),
submenu: new MenuSection(1),
contents: new ContentsSection(true)
},
/* Pages */
new StaticContentPage("Home", new Contents("home.html"), {
"404": new HiddenStaticContentPage("Page not found", new Contents("error/404.html")),
home: new PageAlias("Home", ""),
page1: new StaticContentPage("Page 1", new Contents("page1.html"), {
page11: new StaticContentPage("Subpage 1.1", new Contents("page1/subpage11.html")),
page12: new StaticContentPage("Subpage 1.2", new Contents("page1/subpage12.html")),
page13: new StaticContentPage("Subpage 1.3", new Contents("page1/subpage13.html"))
}),
page2: new StaticContentPage("Page 2", new Contents("page2.html"), {
page21: new StaticContentPage("Subpage 2.1", new Contents("page2/subpage21.html")),
page22: new StaticContentPage("Subpage 2.2", new Contents("page2/subpage22.html")),
page23: new StaticContentPage("Subpage 2.3", new Contents("page2/subpage23.html"))
}),
}),
/* Favorite icon */
"favicon.ico"
);
</pre>
<br />
The above source code file (<i>appmodel.mjs</i>) defines an <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Guide/Modules">ECMAScript module</a> exporting an <i>application</i> object. The <i>application</i> object defines the layout of a web application with the following properties:<br />
<br />
<ul>
<li>The <strong>title</strong> of the web application is: "My application".</li>
<li>All pages use: <i>default.css</i> as a <strong>common stylesheet</strong>.</li>
<li>Every page consists of a number of <strong>sections</strong> that have a specific purpose:<br />
<ul>
<li>A static section (<i>header</i>) provides content that is the same for every page.</li>
<li>A menu section (<i>menu</i>, <i>submenu</i>) display links to sub pages part of the web application.</li>
<li>A content section (<i>contents</i>) displays variable content, such as text and images.</li>
</ul>
</li>
<li>An application consists of multiple <strong>pages</strong> that display the same sections. Every page object refers to a file with static HTML code providing the content that needs to be displayed in the content section.</li>
<li>The last parameter refers to a <strong>favorite icon</strong> that is the same for every page.</li>
</ul>
<br />
Pages in the application model are organized in a tree-like data structure. The application constructor only accepts a single page parameter that refers to the <strong>entry page</strong> of the web application. The entry page can be reached by opening the web application from the root URL or by clicking on the logo displayed in the <i>header</i> section.<br />
<br />
The entry page refers to two sub pages: <i>page1</i>, <i>page2</i>. The <i>menu</i> section displays links to the sub pages that are reachable from the entry page.<br />
<br />
Every sub page can also refer to their own sub pages. The <i>submenu</i> section will display links to the sub pages that are reachable from a selected the sub page. For example, when <i>page1</i> is selected the <i>submenu</i> section will display links to: <i>page11</i>, <i>page12</i>.<br />
<br />
In addition to pages that are reachable from the menu sections, the application model also has hidden error pages and a <i>home</i> link that is an alias for the entry page. In many web applications, it is a common habit that in addition to clicking on the logo, a home button can also be used to redirect a user to the entry page.<br />
<br />
Besides using the links in the menu sections, any sub page in the web application can be reached by using the URL as a selector. A common convention is to use the path components in the URL to determine which page and sub page need to be displayed.<br />
<br />
For example, by opening the following URL in a web browser:<br />
<br />
<pre>
http://localhost/page1/page12
</pre>
<br />
Brings the user to the second sub page of the first sub page.<br />
<br />
When providing an invalid selector in the URL, such as <i>http://localhost/page4</i>, the framework automatically redirects the user to the <i>404</i> error page, because the page cannot be found.<br />
<br />
<h2>Displaying sub pages in the application model</h2>
<br />
As explained earlier, to display any of the sub pages that the application model defines, we must invoke a view function.<br />
<br />
A reasonable strategy (that should suit most needs) is to generate an HTML page, with a title tag composed the application and page's title, globally include the application and page-level stylesheets, and translate every section to a <i>div</i> using the section identifier as its <i>id</i>. The framework provides a view function that automatically performs this translation.<br />
<br />
As a sidenote: for pages that require a more complex structure (for example, to construct a layout with more advanced visualizations), it is also possible to develop a custom view function.<br />
<br />
We can create a custom style sheet: <i>default.css</i> to position the <i>divs</i> and give each section a unique color. By using such a stylesheet, the application model shown earlier may be presented as follows:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEjwJN25GzKUqFFP2XaNIOky2fRkkPOHP67vKycRy-zdRf086oZNAJCHbXkSW9xyumeSo3blAiq4hkpgWZ1EukjZh84CXHR48Zzw8teEaToGjDGSAq_LneOd7RXjtmAJ4HODuy3n2TnbSjzweXw7L5c8mWRK-RuMkUMfD8HmRLvXe1jWXv4Y-aWJCdIWAA=s1070" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="680" data-original-width="1070" src="https://blogger.googleusercontent.com/img/a/AVvXsEjwJN25GzKUqFFP2XaNIOky2fRkkPOHP67vKycRy-zdRf086oZNAJCHbXkSW9xyumeSo3blAiq4hkpgWZ1EukjZh84CXHR48Zzw8teEaToGjDGSAq_LneOd7RXjtmAJ4HODuy3n2TnbSjzweXw7L5c8mWRK-RuMkUMfD8HmRLvXe1jWXv4Y-aWJCdIWAA=s600"/></a></div>
<br />
As can be seen in the screenshot above, the <i>header</i> section has a gray color and displays a logo, the <i>menu</i> section is blue, the <i>submenu</i> is red and the <i>contents</i> section is black.<br />
<br />
The second sub page from the first sub page was selected (as can be seen in the URL as well as the selected buttons in the menu sections). The view functions that generate the menu sections automatically mark the selected sub pages as active.<br />
<br />
With the Java and PHP versions (described in my previous blog post), it is a common practice to generate all requested pages server-side. With the JavaScript port, we can also use it on the client-side in addition to server-side.<br />
<br />
<h2>Constructing an application that generates pages server-side</h2>
<br />
For creating web applications with Node.js, it is a common practice to create an application that runs its own web server.<br />
<br />
(As a sidenote: for production environments it is typically recommended to put a more mature HTTP reverse proxy in front of the Node.js application, such as <a href="https://nginx.com">nginx</a>. A reverse proxy is often more efficient for serving static content and has more features with regards to security etc.).<br />
<br />
We can construct an application that runs a simple embedded HTTP server:<br />
<br />
<pre style="overflow: auto;">
import { application } from "./appmodel.mjs";
import { displayRequestedPage } from "js-sblayout/view/server/index.mjs";
import { createTestServer } from "js-sblayout/testhttpserver.mjs";
const server = createTestServer(function(req, res, url) {
displayRequestedPage(req, res, application, url.pathname);
});
server.listen(process.env.PORT || 8080);
</pre>
<br />
The above Node.js application (<i>app.mjs</i>) performs the following steps:<br />
<br />
<ul>
<li>It <strong>includes</strong> the <strong>application model</strong> shown in the code fragment in the previous section.</li>
<li>It constructs a simple test <strong>HTTP server</strong> that serves well-known static files by looking at common file extensions (e.g. images, stylesheets, JavaScript source files) and treats any other URL pattern as a dynamic request.</li>
<li>The embedded HTTP server listens to port 8080 unless a <i>PORT</i> environment variable with a different value was provided.</li>
<li>Dynamic URLs are handled by a callback function (last parameter). The callback invokes a <strong>view</strong> function from the framework that generates an HTML page with all properties and sections declared in the application layout model.</li>
</ul>
<br />
We can start the application as follows:<br />
<br />
<pre>
$ node app.mjs
</pre>
<br />
and then use the web browser to open the root page:<br />
<br />
<pre>
http://localhost:8080
</pre>
<br />
or any sub page of the application, such as the second sub page of the first sub page:<br />
<br />
<pre>
http://localhost:8080/page1/page12
</pre>
<br />
Although Node.js includes a library and JavaScript interface to run an embedded HTTP server, it is very low-level. Its only purpose is to map HTTP requests (e.g. <i>GET</i>, <i>POST</i>, <i>PUT</i>, <i>DELETE</i> requests) to callback functions.<br />
<br />
My framework contains an abstraction to construct a test HTTP server with reasonable set of features for testing web applications built with the framework, including serving commonly used static files (such as images, stylesheets and JavaScript files).<br />
<br />
For production deployments, there is much more to consider, which is beyond the scope of my HTTP server abstraction.<br />
<br />
It is also possible to use the de-facto web server framework for Node.js: <a href="https://expressjs.com">express</a> in combination with the layout framework:<br />
<br />
<pre style="overflow: auto;">
import { application } from "./appmodel.mjs";
import { displayRequestedPage } from "js-sblayout/view/server/index.mjs";
import express from "express";
const app = express();
const port = process.env.PORT || 8080;
// Configure static file directories
app.use("/styles", express.static("styles"));
app.use("/image", express.static("image"));
// Make it possible to parse form data
app.use(express.json());
app.use(express.urlencoded({ extended: true }));
// Map all URLs to the SB layout manager
app.get('*', (req, res) => {
displayRequestedPage(req, res, application, req.url);
});
app.post('*', (req, res) => {
displayRequestedPage(req, res, application, req.url);
});
// Configure listening port
app.listen(port, () => {
console.log("Application listening on port " + port);
});
</pre>
<br />
The above application invokes <i>express</i> to construct an HTTP web server that listens to port 8080 by default.<br />
<br />
In addition, express has been configured to serve static files from the <i>styles</i> and <i>image</i> folders, and maps all dynamic GET and POST requests to the <i>displayRequestedPage</i> view function of the layout framework.<br />
<br />
<h2>Using the model client-side and dynamically updating the DOM</h2>
<br />
As already explained, using JavaScript as an implementation language also makes it possible to directly consume the application model in the browser and dynamically generate pages from it.<br />
<br />
To make this possible, we only have to write a very minimal static HTML page:<br />
<br />
<pre style="overflow: auto;">
<!DOCTYPE html>
<html>
<head>
<title>My page</title>
<script type="module">
import { application } from "./appmodel.mjs";
import { initRequestedPage, updateRequestedPage } from "js-sblayout/view/client/index.mjs";
document.body.onload = function() {
initRequestedPage(application);
};
document.body.onpopstate = function() {
updateRequestedPage(application);
};
</script>
</head>
<body>
</body>
</html>
</pre>
<br />
The above HTML page has the following properties:<br />
<br />
<ul>
<li>It contains the <strong>bare minimum</strong> of HTML code to construct a page that is still valid HTML5.</li>
<li>We include the <strong>application model</strong> (shown earlier) that is identical to the application model that we have been using to generate pages server-side.</li>
<li>We configure two <strong>event handlers</strong>. When the page is loaded (<i>onload</i>) we initially render all required page elements in the DOM (including the sections that translate to <i>div</i>s). Whenever the user clicks on a link (<i>onpopstate</i>), we update the affected sections in the DOM.</li>
</ul>
<br />
To make the links in the menu sections work, we have to compose them in a slightly different way -- rather than using the path to derive the selected sub page, we have to use hashes instead.<br />
<br />
For example, the second sub page of the first page can be reached by opening the following URL:<br />
<br />
<pre>
http://localhost/index.html#/page1/page21
</pre>
<br />
The <i>popstate</i> event <a href="https://stackoverflow.com/questions/25806608/how-to-detect-browser-back-button-event-cross-browser">triggers whenever the browser's history changes, and makes it possible for the user to use the back and forward navigation buttons.</a><br />
<br />
<h2>Generating dynamic content</h2>
<br />
In the example application model shown earlier, all sections are made out of static HTML code fragments. Sometimes it may also be desired to generate the sections' content dynamically, for example, to respond to user input.<br />
<br />
In addition to providing a string with a static HTML code as a parameter, it is also possible to provide a function that generates the content of the section dynamically.<br />
<br />
<pre style="font-size: 90%; overflow: auto;">
new StaticContentPage("Home", new Contents("home.html"), {
...
hello: new StaticContentPage("Hello 10 times", new Contents(displayHello10Times))
})
</pre>
<br />
In the above code fragment, we have added a new sub page the to entry page that refers to the function: <i>displayHello10Times</i> to dynamically generate content. The purpose of this function is to display the string: "Hello" 10 times:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEg-UegerlOVi63kfqqW158mx2l6PUScJGZwLLoj8Yy1LdezpKs6BSx0RXkB_bo-OpERPNfBAY3H_Zj4-NtZJRSAVNRcV8DcFe5aLb0ee2-AepS7TMQghfYtioM8ZFSnymRZWPJZ7qV6nSYZk426595IElDzj0HGnJawRh09a6uVGBS4ZYaCJGu_L4Eg3Q=s1070" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="680" data-original-width="1070" src="https://blogger.googleusercontent.com/img/a/AVvXsEg-UegerlOVi63kfqqW158mx2l6PUScJGZwLLoj8Yy1LdezpKs6BSx0RXkB_bo-OpERPNfBAY3H_Zj4-NtZJRSAVNRcV8DcFe5aLb0ee2-AepS7TMQghfYtioM8ZFSnymRZWPJZ7qV6nSYZk426595IElDzj0HGnJawRh09a6uVGBS4ZYaCJGu_L4Eg3Q=s600"/></a></div>
<br />
When writing an application that generates pages server-side, we could implement this function as follows:<br />
<br />
<pre>
function displayHello10Times(req, res) {
for(let i = 0; i < 10; i++) {
res.write("<p>Hello!</p>\n");
}
}
</pre>
<br />
The above function follows a convention that is commonly used by applications using Node.js internal HTTP server:<br />
<br />
<ul>
<li>The <i>req</i> parameter refers to the Node.js internal HTTP server's <i>http.IncomingMessage</i> object and can be used to retrieve HTTP headers and other request parameters.</li>
<li>The <i>req.sbLayout</i> parameter provides parameters that are related to the layout framework.</li>
<li>The <i>res</i> parameter refers to the Node.js internal HTTP server's <i>http.ServerResponse</i> object and can be used to generate a response message.</li>
</ul>
<br />
It is also allowed to declare the function above <i>async</i> or let it return a <i>Promise</i> so that asynchronous APIs can be used.<br />
<br />
When developing a client-side application (that dynamically updates the browser DOM), this function should have a different signature:<br />
<br />
<pre>
function displayHello10Times(div, params) {
let response = "";
for(let i = 0; i < 10; i++) {
response += "<p>Hello!</p>\n";
}
div.innerHTML = response;
}
</pre>
<br />
In the browser, a dynamic content generation function accepts two parameters:<br />
<br />
<ul>
<li><i>div</i> refers to an <i>HTMLDivElement</i> in the DOM that contains the content of the section.</li>
<li><i>params</i> provides layout framework specific properties (identical to <i>req.sbLayout</i> in the server-side example).</li>
</ul>
<br />
<h2>Using a templating engine</h2>
<br />
Providing functions that generate dynamic content (by embedding HTML code in strings) may not always be the most intuitive way to generate dynamic content. It is also possible to configure <strong>template handlers</strong>: the framework can invoke a template handler function for files with a certain extension.<br />
<br />
In the following server-side example, we define a template handler for files with an <i>.ejs</i> extension to use the <a href="https://ejs.co">EJS</a> templating engine:<br />
<br />
<pre style="overflow: auto;">
import { application } from "./appmodel.mjs";
import { displayRequestedPage } from "js-sblayout/view/server/index.mjs";
import { createTestServer } from "js-sblayout/testhttpserver.mjs";
import * as ejs from "ejs";
function renderEJSTemplate(req, res, sectionFile) {
return new Promise((resolve, reject) => {
ejs.renderFile(sectionFile, { req: req, res: res }, {}, function(err, str) {
if(err) {
reject(err);
} else {
res.write(str);
resolve();
}
});
});
}
const server = createTestServer(function(req, res, url) {
displayRequestedPage(req, res, application, url.pathname, {
ejs: renderEJSTemplate
});
});
server.listen(process.env.PORT || 8080);
</pre>
<br />
In the above code fragment, the <i>renderEJSTemplate</i> function is used to open an <i>.ejs</i> template file and uses <i>ejs.renderFile</i> function to render the template. The resulting string is propagated as a response to the user.<br />
<br />
To use the template handlers, we invoke the <i>displayRequestedPage</i> with an additional parameter that maps the <i>ejs</i> file extension to the template handler function.<br />
<br />
In a client-side/browser application, we can define a template handler as follows:<br />
<br />
<pre style="overflow: auto;">
<!DOCTYPE html>
<html>
<head>
<title>My page</title>
<script type="text/javascript" src="ejs.js"></script>
<script type="module">
import { application } from "./appmodel.mjs";
import { initRequestedPage, updateRequestedPage } from "js-sblayout/view/client/index.mjs";
const templateHandlers = {
ejs: function(div, response) {
return ejs.render(response, {});
}
}
document.body.onload = function() {
initRequestedPage(application, templateHandlers);
};
document.body.onpopstate = function() {
updateRequestedPage(application, templateHandlers);
};
</script>
</head>
<body>
</body>
</html>
</pre>
<br />
In the above code fragment, we define a <i>templateHandlers</i> object that gets propagated to the view function that initially renders the page (<i>initRequestedPage</i>) and dynamically updates the page (<i>updateRequestedPage</i>).<br />
<br />
By adding the following sub page to the entry page, we can use an <i>ejs</i> template file to dynamically generate a page rather than a static HTML file or function:<br />
<br />
<pre>
new StaticContentPage("Home", new Contents("home.html"), {
...
stats: new StaticContentPage("Stats", new Contents("stats.ejs"))
})
</pre>
<br />
In a server-side application, we can use <i>stats.ejs</i> to display request variables:<br />
<br />
<pre>
<h2>Request parameters</h2>
<table>
<tr>
<th>HTTP version</th>
<td><%= req.httpVersion %></td>
</tr>
<tr>
<th>Method</th>
<td><%= req.method %></td>
</tr>
<tr>
<th>URL</th>
<td><%= req.url %></td>
</tr>
</table>
</pre>
<br />
resulting in a page that may have the following look:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEgSmQBQLxPf3QcNm0JyPqnDxnbEcNom0fS_4cCDjj_P-3mDibF4qQ-d3vQz7kJ-XlQjetDFF4jabz4tTSsHp6SjkOG9qAcy7LicVjgVd5EC8UwlJljF6uDduyhiOIb9MM5preF0xNfJqZGzJudt_7wXsBSTt7C97s3XloVHJ1GWSjoDszKYoQR5RbcViA=s1070" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="680" data-original-width="1070" src="https://blogger.googleusercontent.com/img/a/AVvXsEgSmQBQLxPf3QcNm0JyPqnDxnbEcNom0fS_4cCDjj_P-3mDibF4qQ-d3vQz7kJ-XlQjetDFF4jabz4tTSsHp6SjkOG9qAcy7LicVjgVd5EC8UwlJljF6uDduyhiOIb9MM5preF0xNfJqZGzJudt_7wXsBSTt7C97s3XloVHJ1GWSjoDszKYoQR5RbcViA=s600"/></a></div>
<br />
In a client-side application, we can use <i>stats.ejs</i> to display browser variables:<br />
<br />
<pre>
<h2>Some parameters</h2>
<table>
<tr>
<th>Location URL</th>
<td><%= window.location.href %></td>
</tr>
<tr>
<th>Browser languages</th>
<td>
<%
navigator.languages.forEach(language => {
%>
<%= language %><br>
<%
});
%>
</td>
</tr>
<tr>
<th>Browser code name</th>
<td><%= navigator.appCodeName %></td>
</tr>
</table>
</pre>
<br />
displaying the following page:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhzJEa0vJuiIWyhVB58PkGFIfI5xjU9kizQF-97vDNLUAJz9m4eQB1_727o0uwW7Sms2lyTXqslknTPcpP8xY6kWVXZeO7YORQKQhB9Y1WQfs3LQ0K32ZaWVYKjNOf5s7uIEHAGRlZLNbKDYZPP6D61LOIbk28n7shYbcezWy-InetfCeg_sgZ-u53LdQ=s1070" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="680" data-original-width="1070" src="https://blogger.googleusercontent.com/img/a/AVvXsEhzJEa0vJuiIWyhVB58PkGFIfI5xjU9kizQF-97vDNLUAJz9m4eQB1_727o0uwW7Sms2lyTXqslknTPcpP8xY6kWVXZeO7YORQKQhB9Y1WQfs3LQ0K32ZaWVYKjNOf5s7uIEHAGRlZLNbKDYZPP6D61LOIbk28n7shYbcezWy-InetfCeg_sgZ-u53LdQ=s600"/></a></div>
<br />
<h2>Strict section and page key ordering</h2>
<br />
In all the examples shown previously, we have used an <i>Object</i> to define sections and sub pages. In JavaScript, the order of keys in an object is somewhat deterministic but not entirely -- for example, numeric keys will typically appear before keys that are arbitrary strings, regardless of the insertion order.<br />
<br />
As a consequence, the order of the pages and sections may not be the same as the order in which the keys are declared.<br />
<br />
When the object key ordering is a problem, it is also possible to use iterable objects, such as a nested array, to ensure strict key ordering:<br />
<br />
<pre style="font-size: 80%; overflow: auto;">
import { Application } from "js-sblayout/model/Application.mjs";
import { StaticSection } from "js-sblayout/model/section/StaticSection.mjs";
import { MenuSection } from "js-sblayout/model/section/MenuSection.mjs";
import { ContentsSection } from "js-sblayout/model/section/ContentsSection.mjs";
import { StaticContentPage } from "js-sblayout/model/page/StaticContentPage.mjs";
import { HiddenStaticContentPage } from "js-sblayout/model/page/HiddenStaticContentPage.mjs";
import { PageAlias } from "js-sblayout/model/page/PageAlias.mjs";
import { Contents } from "js-sblayout/model/page/content/Contents.mjs";
/* Create an application model */
export const application = new Application(
/* Title */
"My application",
/* Styles */
[ "default.css" ],
/* Sections */
[
[ "header", new StaticSection("header.html") ],
[ "menu", new MenuSection(0) ],
[ "submenu", new MenuSection(1) ],
[ "contents", new ContentsSection(true) ],
[ 1, new StaticSection("footer.html") ]
],
/* Pages */
new StaticContentPage("Home", new Contents("home.html"), [
[ 404, new HiddenStaticContentPage("Page not found", new Contents("error/404.html")) ],
[ "home", new PageAlias("Home", "") ],
[ "page1", new StaticContentPage("Page 1", new Contents("page1.html"), [
[ "page11", new StaticContentPage("Subpage 1.1", new Contents("page1/subpage11.html")) ],
[ "page12", new StaticContentPage("Subpage 1.2", new Contents("page1/subpage12.html")) ],
[ "page13", new StaticContentPage("Subpage 1.3", new Contents("page1/subpage13.html")) ]
])],
[ "page2", new StaticContentPage("Page 2", new Contents("page2.html"), [
[ "page21", new StaticContentPage("Subpage 2.1", new Contents("page2/subpage21.html")) ],
[ "page22", new StaticContentPage("Subpage 2.2", new Contents("page2/subpage22.html")) ],
[ "page23", new StaticContentPage("Subpage 2.3", new Contents("page2/subpage23.html")) ]
])],
[ 0, new StaticContentPage("Last page", new Contents("lastpage.html")) ]
]),
/* Favorite icon */
"favicon.ico"
);
</pre>
<br />
In the above example, we have rewritten the application model example to use strict key ordering. We have added a section with numeric key: <i>1</i> and a sub page with key: <i>0</i>. Because we have defined a nested array (instead of an object), these section and page will come last (if we would have used an object, then they will appear first, which is undesired).<br />
<br />
Internally, the <i>Application</i> and <i>Page</i> objects use a <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Map">Map</a> to ensure strict ordering.<br />
<br />
<h2>More features</h2>
<br />
The framework has full feature parity with the PHP and Java implementations of the layout framework. In addition to the features described in the previous sections, it can also do the following:<br />
<br />
<ul>
<li><strong>Work with multiple content sections</strong>. In our examples, we only have one content section that changes when picking a menu item, but it is also possible to have multiple content sections.</li>
<li><strong>Page specific stylesheets and JavaScript includes</strong>. Besides including CSS stylesheets and JavaScript files globally it can also be done on page level.</li>
<li><strong>Using path components as parameters</strong>. Instead of selecting a sub page, it is also possible to treat a path component as a parameter and dynamically generate a response.</li>
<li><strong>Internationalized pages</strong>. Each sub page uses a ISO localization code and the framework will pick the most suitable language in which the page should be displayed by default.</li>
<li><strong>Security handlers</strong>. Every page can implements its own method that checks whether it should be accessible or not according to a custom security policy.</li>
<li><strong>Controllers</strong>. It is also possible to process GET or POST parameters before the page gets rendered to decide what to do with them, such as validation.</li>
</ul>
<br />
<h2>Conclusion</h2>
<br />
In this blog post, I have described the features of the JavaScript port of my layout framework. In addition to rendering pages server-side, it can also be directly used in the web browser to dynamically update the DOM. For the latter aspect, it is not required to run any server-side scripting language making application deployments considerably easier.<br />
<br />
One of the things I liked about this experiment is that the layout model is sufficiently high-level so that it can be used in a variety of application domains. To make client-side rendering possible, I only had to develop another view function. The implementation of the model aspect is exactly the same for server-side and client-side rendering.<br />
<br />
Moreover, the newer features of the JavaScript language (most notably ECMAScript modules) make it much easier to reuse code between Node.js and web browsers. Before ECMAScript modules were adopted by browser vendors, there was no module system in the browser at all (Node.js has CommonJS) forcing me to implement all kinds of tricks to make a reusable implementation between Node.js and browsers possible.<br />
<br />
As explained in the introduction of this blog post, web front-end technologies do not have a separated layout concern. A possible solution to cope with this limitation is to generate pages server-side. With the JavaScript implementation this is no longer required, because it can also be directly done in the browser.<br />
<br />
However, this does still not fully solve my layout frustrations. For example, dynamically generated pages are poorly visible to search engines. Moreover, a dynamically rendered web application is useless to users that have JavaScript disabled, or a web browser that does not support JavaScript, such as text browsers.<br />
<br />
Using JavaScript also breaks the declarative nature of web applications -- HTML and CSS allow you to write what the structure and style of a page without specifying how to render it. This has all kinds of advantages, such as the ability to degrade gracefully when certain features cannot be used, such as graphics. With JavaScript some of these properties are lost.<br />
<br />
Still, this project was a nice distraction -- I already had the idea to explore this for several years. During the COVID-19 pandemic, I have read quite a few technical books, such as <a href="https://www.oreilly.com/library/view/javascript-the-definitive/9781491952016">JavaScript: The Definitive Guide</a> and learned that with the introduction of new language JavaScript features, such as ECMAScript modules, it would be possible to exactly the same implementation of the model both server-side and client-side.<br />
<br />
As explained in <a href="https://sandervanderburg.blogspot.com/2021/12/11th-annual-blog-reflection.html">my blog reflection over 2021</a>, I have been overly focused on a single goal for almost two years and it started to negatively affect my energy level. This project was a nice short distraction.<br />
<br />
<h2>Future work</h2>
<br />
I have also been investigating whether I could use my framework to create offline web applications with a consistent layout. Unfortunately, it does not seem to be very straight forward to do that.<br />
<br />
It seems that it is not allowed to do any module imports from local files for security reasons. In theory, this restriction can be bypassed by packing up all the modules into a single JavaScript include with <a href="https://webpack.js.org">webpack</a>.<br />
<br />
However, it turns out that there is another problem -- it is also not possible to open any files from the local drive for security reasons. There is a <a href="https://wicg.github.io/file-system-access/">file system access API</a> in development, that is still not finished or mature yet.<br />
<br />
Some day, when these APIs have become more mature, I may revisit this problem and revise my framework to also make offline web applications possible.<br />
<br />
<h2>Availability</h2>
<br />
The JavaScript port of my layout framework can be obtained from <a href="https://github.com/svanderburg/js-sblayout">my GitHub page</a>. To use this framework client-side, a modern web browser is required, such as Mozilla Firefox or Google Chrome.<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-86371707677843415812022-01-11T19:14:00.000+01:002022-01-11T19:14:15.499+01:00Structured asynchronous programming revisited (Asynchronous programming with JavaScript part 5)It has been a while since I wrote a JavaScript related blog post. In my previous job, I was using it on a daily basis, but in the last few years I have been using it much less frequently.<br />
<br />
One of the reasons that I wrote so many JavaScript-related blog posts is because the language used to have many catches, such as:<br />
<br />
<ul>
<li><strong>Scoping</strong>. Contrary to many other mainstream programming languages, JavaScript uses function-level scoping as opposed to block-level scoping. Syntactically, function-level scoping looks very similar to block-level scoping.<br />
<br />
Function-level scoping has a number of implications that could have severe consequences. For example, you may unintentionally re-assign values.</li>
<li><a href="https://sandervanderburg.blogspot.com/2013/02/yet-another-blog-post-about-object.html"><strong>Simulating class-based inheritance</strong></a>. JavaScript supports Object Oriented programming with prototypes rather than classes (that most mainstream Object Oriented programming languages use). It is possible to use prototypes to simulate classes and class-based inheritance.<br />
<br />
Although I consider prototypes to be conceptually simple, using them in JavaScript used to be quite confusing. As a consequence, simulating class inheritance also used to be quite difficult.</li>
<li><strong>Asynchronous programming</strong>. JavaScript is originally designed for use in web browsers, but has also become quite popular outside the browser, such as <a href="http://nodejs.org">Node.js</a>, to write server/command-line applications. For both browser usage as well as server applications, it is often desired to do multiple tasks concurrently.<br />
<br />
Unfortunately, most of JavaScript's language constructs are synchronous, making such tasks quite difficult without using any software abstractions.</li>
</ul>
<br />
In particular about the last topic: asynchronous programming, I wrote many blog posts. I have elaborated about <a href="https://sandervanderburg.blogspot.com/2013/07/asynchronous-programming-with-javascript.html">callbacks in Node.js and abstraction libraries to do coordination</a> and another popular abstraction: <a href="https://sandervanderburg.blogspot.com/2013/12/asynchronous-programming-with.html">promises</a>.<br />
<br />
I have also argued that most of JavaScript's language constructs, that implement structured programming concepts, are synchronous and cannot be used with asynchronous functions that return (almost) immediately, and resume their executions with callbacks.<br />
<br />
I have built my own library: <a href="https://github.com/svanderburg/slasp">slasp</a>, that implements <a href="https://sandervanderburg.blogspot.com/2014/03/structured-asynchronous-programming.html">asynchronous equivalents for the synchronous language constructs that you should not use</a>.<br />
<br />
Fortunately, much has happened since I wrote that blog post. The JavaScript language has many new features (part of the <a href="http://es6-features.org">ECMAScript 6 standard</a>) that have become a standard practice nowadays, such as:<br />
<br />
<ul>
<li><strong>Block level scoping</strong>. Block scoped immutable values can be declared with: <i>const</i> and mutable values with: <i>let</i>.</li>
<li>An object with a custom <strong>prototype</strong> can now be directly created with: <i>Object.create</i>.</li>
<li>A <strong><i>class</i> construct</strong> was added that makes it possible to define classes (that are simulated with prototypes).</li>
</ul>
<br />
Moreover, modern JavaScript also has new constructs and APIs for asynchronous programming, making most of the software abstractions that I have elaborated about obsolete for the most part.<br />
<br />
Recently, I have been using these new constructs quite intensively and learned that my previous blog posts (that I wrote several years ago) are still mostly about old practices.<br/>
<br />
In this blog post, I will revisit the structured programming topic and explain how modern language constructs can be used to implement these concepts.<br />
<br />
<h2>Asynchronous programming in JavaScript</h2>
<br />
As explained in my previous blog posts, asynchronous programming in JavaScript is important for a variety of reasons. Some examples are:<br />
<br />
<ul>
<li>Animating objects and providing other visual effects in a web browser, while keeping the browser responsive so that it can respond to user events, such as mouse clicks.</li>
<li>The ability to serve multiple clients concurrently in a Node.js server application.</li>
</ul>
<br />
Multi-tasking in JavaScript is (mainly) cooperative. The idea is that JavaScript code runs in a hidden main loop that responds to events in a timely manner, such as user input (e.g. mouse clicks) or incoming connections.<br />
<br />
To keep your application responsive, it is required that the execution of a code block does <strong>not take long</strong> (to allow the application to respond to other events), and that an <strong>event</strong> is generated to allow the execution to be resumed at a later point in time.<br />
<br />
Not meeting this requirement may cause the web browser or your server application to block, which is often undesirable.<br />
<br />
Because writing non-blocking code is so important, many functions in the Node.js API are <strong>asynchronous</strong> by default: they return (almost) immediately and use a callback function parameter that gets invoked when the work is done.<br />
<br />
For example, reading a file from disk while keeping the application responsive can be done as follows:<br />
<br />
<pre>
const fs = require('fs');
fs.readFile("hello.txt", function(err, contents) {
if(err) {
console.error(err);
process.exit(1);
} else {
console.log(contents);
}
});
</pre>
<br />
Note that in the above code fragment, instead of relying on the return value of the <i>fs.readFile</i> call, we provide a callback function as a parameter that gets invoked when the operation has finished. The callback is responsible for displaying the file's contents or the resulting error message.<br />
<br />
While the file is being read (that happens in the background), the event loop is still able to process other events making it possible for the application to work on other tasks concurrently.<br />
<br />
To ensure that an application is responsive and scalable, I/O related functionality in the Node.js API is asynchronous by default. For some functions there are also synchronous equivalents for convenience, but as a rule of thumb they should be avoided as much as possible.<br />
<br />
Asynchronous I/O is an important ingredient in making Node.js applications scalable -- because I/O operations are typically several orders of magnitude slower than CPU operations, the application should remain responsive as long as no callback takes long to complete. Furthermore, because there is no thread per connection model, there is no context-switching and memory overhead for each concurrent task.<br />
<br />
However, asynchronous I/O operations and a callback-convention does not guarantee that the main loop never gets blocked.<br />
<br />
When implementing tasks that are heavily CPU-bound (such as <a href="https://www.semitwist.com/mirror/node-js-is-cancer.html">recursively computing a Fibonacci number</a>), the programmer has to make sure that the execution does not block the main loop too long (for example, by dividing it into smaller tasks that generate events, or using threads).<br />
<br />
<h2>Code structuring issues</h2>
<br />
Another challenge that comes with asynchronous functions is that it becomes much harder to keep your code structured and maintainable.<br />
<br />
For example, if we want to create a directory, then write a text file to it, and then read the text file, and only use non-blocking functions to keep the application responsive, we may end up writing:<br />
<br />
<pre>
const fs = require('fs');
fs.mkdir("test", function(err) {
if(err) {
console.error(err);
process.exit(1);
} else {
fs.writeFile("hello.txt", "Hello world!", function(err) {
if(err) {
console.error(err);
process.exit(1);
} else {
fs.readFile("hello.txt", function(err, contents) {
if(err) {
console.error(err);
process.exit(1);
} else {
// Displays: "Hello world!"
console.log(contents);
}
});
}
});
}
});
</pre>
<br />
As may be observed, for each function call, we define a callback responsible for checking the status of the call and executing the next step. For each step, we have to nest another callback function, resulting in pyramid code.<br />
<br />
The code above is difficult to read and maintain. If we want to add another step in the middle, we are forced to refactor the callback nesting structure, which is labourious and tedious.<br />
<br />
Because code structuring issues are so common, all kinds of software abstractions have been developed to coordinate the execution of tasks. For example, we can use the <a href="https://github.com/caolan/async">async library</a> to rewrite the above code fragment as follows:<br />
<br />
<pre>
const async = require('async');
async.waterfall([
function(callback) {
fs.mkdir("test", callback);
},
function(callback) {
fs.writeFile("hello.txt", "Hello world!", callback);
},
function(callback) {
fs.readFile("hello.txt", callback);
},
function(contents, callback) {
// Displays: "Hello world!"
console.log(contents);
callback();
}
], function(err) {
if(err) {
console.error(err);
process.exit(1);
}
});
</pre>
<br />
The <i>async.waterfall</i> abstraction flattens the code, allows us to conveniently add additional asynchronous steps and change the order, if desired.<br />
<br />
<h2>Promises</h2>
<br />
In addition to Node.js-style callback functions and abstraction libraries for coordination, a more powerful software abstraction was developed: <strong>promises</strong> (to be precise: there are several kinds of promise abstractions developed, but I am referring to the <a href="https://promisesaplus.com">Promises/A+ specification</a>).<br />
<br />
Promises have become very popular, in particular for APIs that are used in the browser. As a result, they have been accepted into the core JavaScript API.<br />
<br />
With promises the idea is that every asynchronous function quickly returns a promise object that can be used as a reference to a value that will be delivered at some point in the future.<br />
<br />
For example, we can wrap the function invocation that reads a text file into a function that returns promise:<br />
<br />
<pre>
const fs = require('fs');
function readHelloFile() {
return new Promise((resolve, reject) => {
fs.readFile("hello.txt", function(err, contents) {
if(err) {
reject(err);
} else {
resolve(contents);
}
});
});
}
</pre>
<br />
The above function: <i>readHelloFile</i> invokes <i>fs.readFile</i> from the Node.js API to read the <i>hello.txt</i> file and returns a promise. In case the file was successfully read, the promise is <strong>resolved</strong> and the file's contents is propagated as a result. In case of an error, the promise is <strong>rejected</strong> with the resulting error message.<br />
<br />
To retrieve and display the result, we can invoke the above function as follows:<br />
<br />
<pre>
readHelloFile().then(function(contents) {
console.log(contents);
}, function(err) {
console.error(err);
process.exit(1);
});
</pre>
<br />
Invoking the <i>then</i> method causes the main event loop to invoke either the resolve (first parameter) or reject callback function (second parameter) when the result is available.<br />
<br />
Because promises have become part of the ECMAScript standard, Node.js has introduced alternative APIs that are promise-based, instead of callback based (such as for filesystem operations: <i>fs.promises</i>).<br />
<br />
By using the promises-based API for filesystem operations, we can simplify the previous example to:<br />
<br />
<pre>
const fs = require('fs').promises;
fs.readFile("hello.txt").then(function(contents) {
console.log(contents);
}, function(err) {
console.error(err);
process.exit(1);
});
</pre>
<br />
As described in my old blog post about promises -- they are considered more powerful than callbacks. A promise provides you a reference to a value that is in the process of being delivered. Callbacks can only give you insights in the status of a background task as soon as it completes.<br />
<br />
Although promises have an advantage over callbacks, both approaches still share the same drawback -- we are forced to avoid most JavaScript language constructs and use alternative function abstractions.<br />
<br />
In modern JavaScript, it is also no longer necessary to always explicitly create promises. Instead, we can also declare a function as <i>async</i>. Simply returning a value or throwing exception in such a function automatically ensures that a promise is returned:<br />
<br />
<pre>
async function computeSum(a, b) {
return a + b;
}
</pre>
<br />
The function above returns the sum of the provided input parameters. The JavaScript runtime automatically wraps its execution into a promise that can be used to retrieve the return value at some point in the future.<br />
<br />
(As a sidenote: the function above returns a promise, but is still blocking. It does not generate an event that can be picked up by the event loop and a callback that can resume its execution at a later point in time.)<br />
<br />
The result of executing the following code:<br />
<br />
<pre>
const result = computeSum(1, 1);
console.log("The result is: " + result);
</pre>
<br />
is a promise object, not a numeric value:<br />
<br />
<pre>
Result is: [object Promise]
</pre>
<br />
When a function returns a promise, we also no longer have to invoke <i>.then()</i> and provide callbacks as parameters to retrieve the result or any thrown errors. The <i>await</i> keyword can be used to automatically wait for a promise to yield its return value or an exception, and then move to the next statement:<br />
<br />
<pre>
(async function() {
const result = await computeSum(1, 1);
console.log("The result is: " + result); // The result is: 2
})();
</pre>
<br />
The only catch is that you can only use <i>await</i> in the scope of an asynchronous function. The default scope of a Node.js program is synchronous. As a result, we have to wrap the code into an asynchronous function block.<br />
<br />
By using the promise-based <i>fs</i> API and the new language features, we can rewrite our earlier callback-based example (that creates a directory, writes and reads a file) as follows:<br />
<br />
<pre>
const fs = require('fs').promises;
(async function() {
try {
await fs.mkdir("test");
await fs.writeFile("hello.txt", "Hello world!");
const contents = await fs.readFile("hello.txt");
} catch(err) {
console.error(err);
process.exit(1);
}
})();
</pre>
<br />
As may be observed, the code is much simpler than manually orchestrating promises.<br />
<br />
<h2>Structured programming concepts</h2>
<br />
In my previous blog post, I have argued that most of JavaScript's language constructs (that implement structured programming concepts) cannot be directly used in combination with non-blocking functions (that return almost immediately and require callback functions as parameters).<br />
<br />
As a personal exercise, I have created function abstractions that are direct asynchronous equivalents for all these structured programming concepts that should be avoided and added them to my own library: <i>slasp</i>.<br />
<br />
By combining promises, <i>async</i> functions and <i>await</i> statements, these function abstractions have mostly become obsolete.<br />
<br />
In this section, I will go over the structured programming concepts I covered in my old blog post and show their direct asynchronous programming equivalents using modern JavaScript language constructs.<br />
<br />
<h3>Function definitions</h3>
<br />
As I have already explained in my previous blog post, the most basic thing one can do in JavaScript is executing statements, such as variable assignments or function invocations. This used to be already much different when moving from a synchronous programming to an asynchronous programming world.<br />
<br />
As a trivial example, I used a synchronous function whose only purpose is to print text on the console:<br />
<br />
<pre>
function printOnConsole(value) {
console.log(value);
}
</pre>
<br />
The above example is probably too trivial, but it is still possible to make it non-blocking -- we can generate a tick event so that the function returns immediately and use a callback parameter so that the task will be resumed at a later point in time:<br />
<br />
<pre>
function printOnConsole(value) {
return new Promise((resolve, reject) => {
process.nextTick(function() {
console.log(value);
resolve();
});
});
}
</pre>
<br />
To follow modern JavaScript practices, the above function is wrapped into a constructor that immediately returns a promise that can be used as a reference to determine when the task was completed.<br />
<br />
(As a sidenote: we compose a regular function that returns a promise. We cannot define an <i>async function</i>, because the <i>process.nextTick</i> is an asynchronous function that requires a callback function parameter. The callback is responsible for propagating the end result. Using a <i>return</i> only causes the callback function to return and not the enclosing function.)<br />
<br />
I have also shown that for functions that return a value, the same principle can be applied. As an example, I have used a function that translates a numeric digit into a word:<br />
<br />
<pre>
function generateWord(digit) {
const words = [ "zero", "one", "two", "three", "four",
"five", "six", "seven", "eight", "nine" ];
return words[digit];
}
</pre>
<br />
We can also make this function non-blocking by generating a tick event and wrapping it into a promise:<br />
<br />
<pre style="font-size: 90%;">
function generateWord(digit) {
return new Promise((resolve, reject) => {
process.nextTick(function() {
const words = [ "zero", "one", "two", "three", "four", "five",
"six", "seven", "eight", "nine" ];
resolve(words[digit]);
});
});
}
</pre>
<br />
<h3>Sequential decomposition</h3>
<br />
The first structured programming concept I elaborated about was <strong>sequential decomposition</strong> in which a number of statements are executed in sequential order.<br />
<br />
I have shown a trivial example that adds 1 to a number, then converts the resulting digit into a word, and finally prints the word on the console:<br />
<br />
<pre>
const a = 1;
const b = a + 1;
const number = generateWord(b);
printOnConsole(number); // two
</pre>
<br />
With the introduction of the <i>await</i> keyword, converting the above code to use the asynchronous implementations of all required functions has become straight forward:<br />
<br />
<pre>
(async function() {
const a = 1;
const b = a + 1;
const number = await generateWord(b);
await printOnConsole(number); // two
})();
</pre>
<br />
The above example is a one-on-one port of its synchronous counterpart -- we just have to use the <i>await</i> keyword in combination with our asynchronous function invocations (that return promises).<br />
<br />
The only unconventional aspect is that we need to wrap the code inside an asynchronous function block to allow the <i>await</i> keyword to be used.<br />
<br />
<h3>Alteration</h3>
<br />
The second programming concept that I covered is <strong>alteration</strong> that is used to specify conditional statements.<br />
<br />
I gave a simple example that checks whether a given name matches my own name:<br />
<br />
<pre>
function checkMe(name) {
return (name == "Sander");
}
const name = "Sander";
if(checkMe(name)) {
printOnConsole("It's me!");
printOnConsole("Isn't it awesome?");
} else {
printOnConsole("It's someone else!");
}
</pre>
<br />
It is also possible to make the <i>checkMe</i> function non-blocking by generating a tick event and wrapping it into a promise:<br />
<br />
<pre>
function checkMe(name) {
return new Promise((resolve, reject) => {
process.nextTick(function() {
resolve(name == "Sander");
});
});
}
</pre>
<br />
To invoke the asynchronous function shown above inside the if-statement, we only have to write:<br />
<br />
<pre>
(async function() {
const name = "Sander";
if(await checkMe(name)) {
await printOnConsole("It's me!");
await printOnConsole("Isn't it awesome?");
} else {
await printOnConsole("It's someone else!");
}
})();
</pre>
<br />
In my previous blog post, I was forced to abolish the regular if-statement and use an abstraction (<i>slasp.when</i>) that invokes the non-blocking function first, then uses the callback to retrieve the result for use inside an if-statement. In the above example, the only subtle change I need to make is to use <i>await</i> inside the if-statement.<br />
<br />
I can also do the same thing for the other alteration construct: the <i>switch</i> -- just using <i>await</i> in the conditional expression and the body should suffice.<br />
<br />
<h3>Repetition</h3>
<br />
For the <strong>repetition</strong> concept, I have shown an example program that implements the <a href="http://en.wikipedia.org/wiki/Leibniz_formula_for_%CF%80">Gregory-Leibniz formula</a> to approximate PI up to 6 digits:<br />
<br />
<pre>
function checkTreshold(approx) {
return (approx.toString().substring(0, 7) != "3.14159");
}
let approx = 0;
let denominator = 1;
let sign = 1;
while(checkTreshold(approx)) {
approx += 4 * sign / denominator;
printOnConsole("Current approximation is: "+approx);
denominator += 2;
sign *= -1;
}
</pre>
<br />
As with the previous example, we can also make the <i>checkTreshold</i> function non-blocking:<br />
<br />
<pre>
function checkTreshold(approx) {
return new Promise((resolve, reject) => {
process.nextTick(function() {
resolve(approx.toString().substring(0, 7) != "3.14159");
});
});
}
</pre>
<br />
In my previous blog post, I have explained that the <i>while</i> statement is unfit for executing non-blocking functions in sequential order, because they return immediately and have to resume their execution at a later point in time.<br />
<br />
As with the alteration language constructs, I have developed a function abstraction that is equivalent to the <i>while</i> statement (<i>slasp.whilst</i>), making it possible to have a non-blocking conditional check and body.<br />
<br />
With the introduction of the <i>await</i> statement, this abstraction function also has become obsolete. We can rewrite the code as follows:<br />
<br />
<pre>
(async function() {
let approx = 0;
let denominator = 1;
let sign = 1;
while(await checkTreshold(approx)) {
approx += 4 * sign / denominator;
await printOnConsole("Current approximation is: "+approx);
denominator += 2;
sign *= -1;
}
})();
</pre>
<br />
As can be seen, the above code is a one-on-one port of its synchronous counterpart.<br />
<br />
The function abstractions for the other repetition concepts: <i>doWhile</i>, <i>for</i>, <i>for-in</i> have also become obsolete by using <i>await</i> for evaluating the non-blocking conditional expressions and bodies.<br />
<br />
Implementing non-blocking recursive algorithms still remains tricky, such as the following (somewhat inefficient) recursive algorithm to compute a Fibonacci number:<br />
<br />
<pre>
function fibonacci(n) {
if (n < 2) {
return 1;
} else {
return fibonacci(n - 2) + fibonacci(n - 1);
}
}
const result = fibonacci(20);
printOnConsole("20th element in the fibonacci series is: "+result);
</pre>
<br />
The above algorithm is mostly CPU-bound and takes some time to complete. As long as it is computing, the event loop remains blocked causing the entire application to become unresponsive.<br />
<br />
To make sure that the execution does not block for too long by using cooperative multi-tasking principles, we should regularly generate events, suspend its execution (so that the event loop can process other events) and use callbacks to allow it to resume at a later point in time:<br />
<br />
<pre style="font-size: 90%;">
function fibonacci(n) {
return new Promise((resolve, reject) => {
if (n < 2) {
setImmediate(function() {
resolve(1);
});
} else {
let first;
let second;
fibonacci(n - 2)
.then(function(result) {
first = result;
return fibonacci(n - 1);
})
.then(function(result) {
second = result;
resolve(first + second);
});
}
});
}
(async function() {
const result = await fibonacci(20);
await printOnConsole("20th element in the fibonacci series is: "+result);
})();
</pre>
<br />
In the above example, I made the algorithm non-blocking by generating a macro-event with <i>setImmediate</i> for the base step. Because the function returns a promise, and cannot be wrapped into an <i>async</i> function, I have to use the promises' <i>then</i> methods to retrieve the return values of the computations in the induction step.<br />
<br />
<h2>Extensions</h2>
<br />
In my previous blog post, I have also covered the extensions to structured programming that JavaScript provides.<br />
<br />
<h3>Exceptions</h3>
<br />
I have also explained that with asynchronous functions, we cannot use JavaScript's <i>throw</i>, and <i>try-catch-finally</i> language constructs, because <strong>exceptions</strong> are typically not thrown instantly but at a later point in time.<br />
<br />
With <i>await</i>, using these constructs is also no longer a problem.<br />
<br />
For example, I can modify our <i>generateWord</i> example to throw an exception when the provided number is not between 0 and 9:<br />
<br />
<pre>
function generateWord(num) {
if(num < 0 || num > 9) {
throw "Cannot convert "+num+" into a word";
} else {
const words = [ "zero", "one", "two", "three", "four", "five",
"six", "seven", "eight", "nine" ];
return words[num];
}
}
try {
let word = generateWord(1);
printOnConsole("We have a: "+word);
word = generateWord(10);
printOnConsole("We have a: "+word);
} catch(err) {
printOnConsole("Some exception occurred: "+err);
} finally {
printOnConsole("Bye bye!");
}
</pre>
<br />
We can make <i>generateWord</i> an asynchronous function by converting it in the usual way:<br />
<br />
<pre style="font-size: 90%;">
function generateWord(num) {
return new Promise((resolve, reject) => {
process.nextTick(function() {
if(num < 0 || num > 9) {
reject("Cannot convert "+num+" into a word");
} else {
const words = [ "zero", "one", "two", "three", "four", "five",
"six", "seven", "eight", "nine" ];
resolve(words[num]);
}
});
});
}
(async function() {
try {
let word = await generateWord(1);
await printOnConsole("We have a: "+word);
word = await generateWord(10);
await printOnConsole("We have a: "+word);
} catch(err) {
await printOnConsole("Some exception occurred: "+err);
} finally {
await printOnConsole("Bye bye!");
}
})();
</pre>
<br />
As can be seen in the example above, thanks to the <i>await</i> construct, we have not lost our ability to use <i>try/catch/finally</i>.<br />
<br />
<h3>Objects</h3>
<br />
Another major extension is <strong>object-oriented</strong> programming. As explained in an old blog post about object oriented programming in JavaScript, JavaScript uses prototypes rather than classes, but prototypes can still be used to simulate classes and class-based inheritance.<br />
<br />
Because simulating classes is such a common use-case, a <i>class</i> construct was added to the language that uses prototypes to simulate it.<br />
<br />
The following example defines a <i>Rectangle</i> class with a method that can compute a rectangle's area:<br />
<br />
<pre>
class Rectangle {
constructor(width, height) {
this.width = width;
this.height = height;
}
calculateArea() {
return this.width * this.height;
}
}
const r = new Rectangle(2, 2);
printOnConsole("Area is: "+r.calculateArea());
</pre>
<br />
In theory, it is also possible that the construction of an object takes a long time and should be made non-blocking.<br />
<br />
Although JavaScript does not have a language concept to do asynchronous object construction, we can still do it by making a couple of small changes:<br />
<br />
<pre>
class Rectangle {
asyncConstructor(width, height) {
return new Promise((resolve, reject) => {
process.nextTick(() => {
this.width = width;
this.height = height;
resolve();
});
});
}
calculateArea() {
return this.width * this.height;
}
}
(async function() {
const r = new Rectangle();
await r.asyncConstructor(2, 2);
await printOnConsole("Area is: "+r.calculateArea());
})();
</pre>
<br />
As can be seen in the above example, the <i>constructor</i> function has been replaced with an <i>asyncConstructor</i> method that implements the usual strategy to make it non-blocking.<br />
<br />
To asynchronously construct the rectangle, we first construct an empty object using the <i>Rectangle</i> class object as its prototype. Then we invoke the asynchronous constructor to initialize the object in a non-blocking way.<br />
<br />
In my previous blog post, I have developed an abstraction function that could be used as an asynchronous replacement for JavaScript's <i>new</i> operator (<i>slasp.novel</i>) that performs the initialization of an empty object and then invokes the asynchronous constructor.<br />
<br />
Due to the fact that JavaScript introduced a <i>class</i> construct (that replaces all the obscure instructions that I had to perform to simulate an empty object instance with the correct class object as prototype) my abstraction function has mostly lost its value.<br />
<br />
<h2>Summary of concepts</h2>
<br />
In my previous blog post, I have given an overview of all covered synchronous programming language concepts and corresponding replacement function abstractions that should be used with non-blocking asynchronous functions.<br />
<br />
In this blog post, I will do the same with the concepts covered:<br />
<br />
<div style="font-size: 90%;">
<table style="border-style: solid; border-width: 1px;">
<tr>
<th style="border-style: solid; border-width: 1px;">Concept</th>
<th style="border-style: solid; border-width: 1px;">Synchronous</th>
<th style="border-style: solid; border-width: 1px;">Asynchronous</th>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">Function interface</td>
<td style="border-style: solid; border-width: 1px;"><pre>function f(a) { ... }</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>
async function f(a) { ... }
function f(a) {
return new Promise(() => {...});
}
</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">Return statement</td>
<td style="border-style: solid; border-width: 1px;"><pre>return val;</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>
return val;
resolve(val);
</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">Sequence</td>
<td style="border-style: solid; border-width: 1px;"><pre>a(); b(); ...</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>await a(); await b(); ...</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">if-then-else</td>
<td style="border-style: solid; border-width: 1px;"><pre>if(condFun()) {
thenFun();
} else {
elseFun();
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>
if(await condFun()) {
await thenFun();
} else {
await elseFun();
}
</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">switch</td>
<td style="border-style: solid; border-width: 1px;"><pre>switch(condFun()) {
case "a":
funA();
break;
case "b":
funB();
break;
...
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>switch(await condFun()) {
case "a":
await funA();
break;
case "b":
await funB();
break;
...
}</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">Recursion</td>
<td style="border-style: solid; border-width: 1px;"><pre>function fun() {
fun();
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>function fun(callback) {
return new Promise((res, rej) => {
setImmediate(function() {
return fun();
});
});
}</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">while</td>
<td style="border-style: solid; border-width: 1px;"><pre>while(condFun()) {
stmtFun();
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>while(await condFun()) {
await stmtFun();
}</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">doWhile</td>
<td style="border-style: solid; border-width: 1px;"><pre>do {
stmtFun();
} while(condFun());</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>do {
await stmtFun();
} while(await condFun());</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">for</td>
<td style="border-style: solid; border-width: 1px;"><pre>for(startFun();
condFun();
stepFun()
) {
stmtFun();
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>for(await startFun();
await condFun();
await stepFun()
) {
await stmtFun();
}</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">for-in</td>
<td style="border-style: solid; border-width: 1px;"><pre>for(const a in arrFun()) {
stmtFun();
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>for(const a in (await arrFun())) {
await stmtFun();
}</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">throw</td>
<td style="border-style: solid; border-width: 1px;"><pre>throw err;</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>throw err;
reject(err);</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">try-catch-finally</td>
<td style="border-style: solid; border-width: 1px;"><pre>try {
funA();
} catch(err) {
funErr();
} finally {
funFinally();
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>try {
await funA();
} catch(err) {
await funErr();
} finally {
await funFinally();
}</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">constructor</td>
<td style="border-style: solid; border-width: 1px;"><pre>
class C {
constructor(a) {
this.a = a;
}
}</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>class C {
asyncConstructor(a) {
return new Promise((res, rej) => {
this.a = a;
res();
}
}
}</pre></td>
</tr>
<tr>
<td style="border-style: solid; border-width: 1px;">new</td>
<td style="border-style: solid; border-width: 1px;"><pre>const obj = new C(a);</pre></td>
<td style="border-style: solid; border-width: 1px;"><pre>const obj = new C();
await obj.asyncConstructor(a);</pre></td>
</tr>
</table>
</div>
<br />
The left column in the table shows all language constructs that are synchronous by default and the right column shows their equivalent asynchronous implementations.<br />
<br />
Note that compared to the overview given in my previous blog post, the JavaScript language constructs are not avoided, but used.<br />
<br />
With the exceptions of wrapping callback-based function invocations in promises and implementing recursive algorithms, using <i>await</i> usually suffices to retrieve the results of all required sub expressions.<br />
<br />
<h2>Discussion</h2>
<br />
In all my previous blog posts that I wrote about asynchronous programming, I was always struck by the fact that most of JavaScript's language constructs were unfit for asynchronous programming. The abstractions that were developed to cope with this problem (e.g. callbacks, coordination libraries, promises etc.) make it possible to get the job done in a reasonable manner, but IMO they always remain somewhat tedious to use and do not prevent you from making many common mistakes.<br />
<Br />
Using these abstractions remained a common habit for years. With the introduction of the <i>async</i> and <i>await</i> concepts, we finally have a solution that is decent IMO.<br />
<br />
Not all problems have been solved with the introduction of these new language features. Callback-based APIs have been a common practice for a very long time, as can be seen in some of my examples. Not all APIs have been converted to promise-based solutions and there are still many APIs and third-party libraries that keep following old practices. Most likely, not all old-fashioned APIs will ever go away.<br />
<br />
As a result, we sometimes still have to manually compose promises and do the appropriate conversions from callback APIs. There are also <a href="https://sandervanderburg.blogspot.com/2016/01/integrating-callback-and-promise-based.html">nice facilities that make it possible to conveniently convert callback invocations into promises</a>, but it still remains a responsibility of the programmer.<br />
<br />
Another problem (that I often see with programmers that are new to JavaScript), is that they believe that using the <i>async</i> keyword automatically makes their functions non-blocking.<br />
<br />
Ensuring that a function does not block still remains the responsibility of the programmer. For example, by making sure that only non-blocking I/O functions are called, or CPU-bound instructions are broken up into smaller pieces.<br />
<br />
The <i>async</i> keyword is only an interface solution -- making sure that a function returns a promise and that the <i>await</i> keyword can be used so that a function can stop (by returning) and be resumed at a later point in time.<br />
<br />
The JavaScript language does not natively support threads or processes, but APIs have been added to Node.js (<a href="https://nodejs.org/api/worker_threads.html">worker threads</a>) and web browsers (<a href="https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Using_web_workers">web workers</a>) to allow code to be executed in a thread. Using these facilities somewhat relieve programmers of the burden to divide long running CPU-bound operations into smaller tasks. Moreover, context-switching is typically much more efficient than cooperative multi-tasking (by generating events and invoking callbacks).<br />
<br />
Another problem that remains is calling asynchronous functions from synchronous functions -- there is still no facility in JavaScript that makes it possible to wait for the completion of an asynchronous function in a synchronous function context.<br />
<br />
I also want to make a general remark about structured programming -- although the patterns shown in this blog post can be used to prevent that a main loop blocks, structured programming is centered around the idea that you need to execute a step after the completion of another. For long running tasks that do not have a dependency on each other this may not always be the most efficient way of doing things. You may end up waiting for an unnecessary amount of time.<br />
<br />
The fact that a promise gives you a reference to a value that will be delivered in the future, also gives you many new interesting abilities. For example, in an application that consists of a separate model and view, you could already start composing a view and provide the promises for the model objects as parameters to the views. Then it is no longer necessary to wait for all model objects to be available before the views can be constructed -- instead, the views can already be rendered and the data can be updated dynamically.<br />
<br />
Structured programming patterns are also limiting the ability to efficiently process collections of data -- the repetition patterns in this blog post expect that all data is retrieved before we can iterate over the resulting data collection. It may be more efficient to work with <a href="https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Statements/for-await...of">asynchronous iterators</a> that can retrieve data on an element-by-element basis, in which the element that comes first is processed first.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-57011066431562857482021-12-30T21:19:00.000+01:002021-12-30T21:19:26.881+01:0011th annual blog reflectionToday it is my blog's 11th anniversary. As with previous years, this is a nice opportunity to reflect over last year's writings.<br />
<br />
<h2>Nix process management framework</h2>
<br />
In the first few months of the year, I have dedicated quite a bit of time on the development of the experimental Nix process framework that I started in 2019.<br />
<br />
As explained in my <a href="https://sandervanderburg.blogspot.com/2020/12/annual-blog-reflection-over-2020.html">blog reflection over 2020</a>, I have reached all of my original objectives. However, while developing these features and exploring their underlying concepts, I discovered that there were still a number side issues that I needed to address to make the framework usable.<br />
<br />
<h3>s6-rc backend</h3>
<br />
The first thing I did was <a href="https://sandervanderburg.blogspot.com/2021/02/developing-s6-rc-backend-for-nix.html">developing a s6-rc backend</a>. Last year, I did not know anything about <a href="https://skarnet.org/software/s6/">s6</a> or <a href="https://skarnet.org/software/s6-rc/">s6-rc</a> , but it was provided to me as a feature suggestion by people from the Nix community. Aside from the fact that it is a nice experiment to evaluate how portable the framework is, I also learned a great deal about s6, its related tools, and its ancestor: <a href="https://cr.yp.to/daemontools.html">daemontools</a> from which many of s6's ideas are inspired.<br />
<br />
<h3>Mutable multi-process containers</h3>
<br />
I also worked on a <a href="https://sandervanderburg.blogspot.com/2021/02/deploying-mutable-multi-process-docker.html">mutable multi-process container deployment</a> approach. Last year, I have developed a Docker backend for the Nix process management framework making it possible to expose each running process instance as a container. Furthermore, I also made it possible to conveniently construct multi-process containers in which any capable process manager that the Nix process management framework supports can be used as a root process.<br />
<br />
Unfortunately, multi-process containers have a big drawback: they are immutable, and when any of the processes need to be changed or upgraded, the container as a whole needs to be discarded and redeployed from a new image, causing all processes to be terminated.<br />
<br />
To cope with this limitation, I have developed an extension that makes it possible to deploy mutable multi-process containers, in which most of the software in containers can be upgraded by the Nix package manager.<br />
<br />
As long as the root process is not affected, a container does not have to be discarded when a process is upgraded. This extension also makes it possible to run Hydra: the Nix-based continuous integration service in a Docker container.<br />
<br />
<h3>Using the Nix process management framework as an infrastructure deployment solution</h3>
<br />
I was also able to use the Nix process management framework to solve the bootstrap problem for Disnix on non-NixOS systems -- in order to use Disnix, every target machine needs to run the Disnix service and a number of container provider services.<br />
<br />
For a NixOS machine this process is automated, but on non-NixOS systems a manual installation is still required, which is quite cumbersome. <a href="https://sandervanderburg.blogspot.com/2021/03/using-nix-process-management-framework.html">The Nix process management framework can automatically deploy Disnix and all required container provider services</a> on any system capable of running the Nix package manager and the Nix process management framework.<br />
<br />
<h3>Test framework</h3>
<br />
Finally, I have developed <a href="https://sandervanderburg.blogspot.com/2021/04/a-test-framework-for-nix-process.html">a test framework for the Nix process management framework</a>. As I have already explained, the framework makes it possible to use multiple process managers, multiple operating systems, multiple instances of all kinds of services, and run services as an unprivileged user, if desired.<br />
<br />
Although the framework facilitates all these features, it cannot guarantee that a service will work with under all possible conditions. The framework makes it possible to conveniently reproduce all these conditions so that a service can be validated.<br />
<br />
With the completion of the test framework, I consider the Nix process management framework to be quite practical. I have managed to <a href="https://github.com/svanderburg/nix-processmgmt-services">automate the deployments of all services that I frequently use</a> (e.g. web servers, databases, application services etc.) and they seem to work quite well. Even commonly used Nix projects are packaged, such as the Nix daemon for multi-user installations and Hydra: the Nix-based continuous integration server.<br />
<br />
<h3>Future work</h3>
<br />
There are still some open framework issues that I intend to address at some point in the future. We still cannot test any services on non-Linux systems such as FreeBSD, which requires a more generalized test driver.<br />
<br />
I also still need to start writing an RFC that identifies the concepts of the framework so that these ideas can be integrated into Nixpkgs. The Nix process management framework is basically a prototype to explore ideas, and it has always been my intention to push the good parts upstream.<br />
<br />
<h2>Home computing</h2>
<br />
After the completion of the test framework, I have shifted my priorities and worked on improving my home computing experience.<br />
<br />
For many years, I have been using a custom script that uses rsync to exchange files, but implements a Git-like workflow, to make backups of my personal files and exchange files between multiple machines, such as my desktop machine and laptop. I have decided to polish the script, release it, and write <a href="https://sandervanderburg.blogspot.com/2021/06/an-unconventional-method-for-creating.html">a blog post that explains how it came about</a>.<br />
<br />
Last summer, I visited the <a href="https://www.homecomputermuseum.nl">Home Computer Museum</a>, that gave me inspiration to check if both of my vintage computers: the Commodore 128 and Commodore Amiga 500 still work. I have not touched the Amiga since 2011 (the year that I wrote a blog post about it) and it was lying dormant in a box on the attic every since.<br />
<br />
Unfortunately, a few peripherals were broken or in a bad condition (such as the hard drive). I have decided to bring it to the museum for repairs and order replacement peripherals. It turns out that it was quite a challenge to have it all figured out, in particular the installation process of the operating system.<br />
<br />
Because not all information that I needed is available on the Internet, I have decided to write <a href="https://sandervanderburg.blogspot.com/2021/10/using-my-commodore-amiga-500-in-2021.html">a blog post about my experiences</a>.<br />
<br />
I am still in the process of figuring out all the details for my Commodore 128 and I hope I can publish about it soon.<br />
<br />
<h2>Revising the NPM package deployment infrastructure for Nix</h2>
<br />
Aside from doing a nice "home computing detour", I have also been shifting my priorities to a new major development area: improving the NPM deployment infrastructure for Nix. Although node2nix is doing its job pretty well in most cases, its design is heavily dated, and giving me more and more problems in correctly supporting the new features of NPM.<br />
<br />
As a first step, I have revised what I consider the second most complicated part of node2nix: the process that populates the <i>node_modules/</i> folder and makes all necessary modifications so that <i>npm install</i> will not attempt to download source artifacts from their original locations.<br />
<br />
This is an important requirement -- the fact that NPM and Nix do not play well together is because dependency management is conflicting -- Nix's purity principles are more strict. As a result, NPM's dependency management needs to be bypassed.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2021/08/a-more-elaborate-approach-for-bypassing.html">The result is a companion tool that I call: placebo-npm</a> that will replace most of the complicated shell code in the <i>node-env.nix</i> module.<br />
<br />
I am still working on revising many other parts of node2nix. This should eventually lead to a new and more modular design, that will support NPM's newer features and should be much easier to maintain.<br />
<br />
Next year, I hope to report more about my progress.<br />
<br />
<h2>Some thoughts</h2>
<br />
As with 2020, I consider 2021 an exceptional year for the record books.<br />
<br />
<h3>Development</h3>
<br />
Compared to last year, I am much less productive from a blogging perspective. Partially, this is caused by the fact that there are still many things I have worked on that I could not properly finish.<br />
<br />
I have also noticed that there was a considerable drop in my energy level after I completed the test framework for the Nix process management framework. I think this can be attributed to the fact that the process management framework has basically been my only spare time project for over two years.<br />
<br />
For a small part, this kind of work is about exploring ideas, but is even more about the <strong>execution</strong> of those ideas -- unfortunately, being in execution mode for such a long time (while ignoring the exploration of ideas you come up in other areas) gradually made it more difficult to keep enjoying the work.<br />
<br />
Despite struggling with my energy levels, I remained motivated to complete all of it, because I know that I am also a very bad multi-tasker. Switching to something else makes it even more difficult to complete it in a reasonable time span.<br />
<br />
After I reached all my goals, for a while, it became extremely difficult to get myself focused on any technical challenge.<br />
<br />
Next year, I have another big project that I am planning to take on (node2nix), but at the same I will try to schedule proper "breaks" in between to keep myself in balance.<br />
<br />
<h3>The COVID-19 pandemic</h3>
<br />
In my annual reflection from last year, I have also elaborated about the COVID-19 pandemic that reached my home country (The Netherlands) in March 2020. Many things have happened that year, and at the time writing my reflection blog post over 2020, we were in our second lockdown.<br />
<br />
The second lockdown felt much worse than the first, but I was still mildly optimistic because of the availability of the first vaccine: <a href="https://www.ema.europa.eu/en/medicines/human/EPAR/comirnaty">Pfizer/BioNTech</a> that looked like our "way out". Furthermore, I was also quite fascinated by the <a href="https://www.cdc.gov/coronavirus/2019-ncov/vaccines/different-vaccines/mrna.html">mRNA technique</a> making it possible to produce a vaccine so quickly.<br />
<br />
Last year, I was hoping that next year I could report that the problem was under control or at least partially solved, but sadly that does not seem to be the case.<br />
<br />
Many things have happened: <a href="https://www.bbc.com/news/health-55388846">the variant that appeared in England</a> eventually became dominant (Alpha variant) and was considerably more contagious than the original variant, causing the second lockdown to last another three months.<br />
<br />
In addition, two more contagaious mutations appeared at the same time in <a href="https://en.wikipedia.org/wiki/SARS-CoV-2_Beta_variant">South Africa</a> and <a href="https://www.nature.com/articles/d41586-021-01480-3">Brazil</a> (Beta and Gamma), of which the Alpha variant became the dominant.<br />
<br />
Several months later, <a href="https://www.nytimes.com/2021/06/22/health/delta-variant-covid.html">there was another huge outbreak in India introducing the Delta variant</a>, that was even more contagious than the Alpha variant. Eventually, that variant became the most dominant in the world. Fortunately, the mRNA vaccines that were developed were still effective enough, but the Delta variant was so contagious that it was considered impossible to build up herd immunity to eradicate the virus.<br />
<br />
In the summer, the problem seemed to be mostly under control because many people have been vaccinated in my home country. Nonetheless, we have learned in a painful way that <a href="https://www.rtlnieuws.nl/nieuws/nederland/artikel/5241201/veel-besmettingen-groningen-toename-besmettingen">relaxing restrictions too much could still lead to very big rapid outbreaks</a>.<br />
<br />
We have also learned in a painful way that <a href="https://www.nature.com/articles/s41598-021-91798-9">the virus spreads more easily in the winter</a>. As a result, we also observed that, despite the fact that <a href="https://coronadashboard.rijksoverheid.nl/landelijk/vaccinaties">over 85% of all adults are fully vaccinated</a>, there are still enough clusters of people that have not built up any resistance against the virus (either by vaccination or contracting the virus), again leading to <a href="https://www.rivm.nl/nieuws/ruim-30-procent-meer-covid-19-ziekenhuisopnames">significant problems in the hospitals and ICs</a>.<br />
<br />
Furthermore, the effectiveness of vaccines also drops over time, causing vaccinated people with health problems to still end up in hospitals.<br />
<br />
As a result of all these hospitalizations and low IC capacity, we are now in yet another lockdown.<br />
<br />
What has always worried me the most is the fact that in so many areas in the world, people hardly have any access to vaccines and medicines causing the virus to easily spread on a massive scale and mutate. I knew because of these four variants, it would only be a matter of time before a new dominant mutation will appear.<br />
<br />
And that fear eventually became reality -- in November, in Botswana and South Africa, a new and even more contagious mutation appeared: <a href="https://www.aljazeera.com/podcasts/2021/12/8/how-south-africa-discovered-covid-variant-omicron">the Omicron variant</a> with a spike-protein that is much different than the delta variant, reducing the effectiveness of our vaccines.<br />
<br />
At the moment, we are not sure the implications are. In the Netherlands, as well as many other countries in the world, <a href="https://www.theguardian.com/world/2021/nov/26/omicron-covid-variant-spreads-europe">the Omicron variant has now become the most dominant virus mutation</a>.<br />
<br />
The only thing that may be good news is that the Omicron variant could also potentially cause less severe sickness, but so far we have no clarity on that yet.<br />
<br />
I think next year we still have much to learn -- to me it has become very clear that this problem will not go away any time soon and that we have to find more innovative/smarter ways to cope with it.<br />
<br />
Furthermore, the mutations have also demonstrated that we should probably do more about inequality in the world. As a consequence, the virus could still significantly spread and mutate becoming a problem to everybody in the world.<br />
<br />
<h2>Blog posts</h2>
<br />
Last year I forgot about it, but every year I also typically reflect over my top 10 of most frequently read blog posts:<br />
<br />
<ol>
<li><a href="https://sandervanderburg.blogspot.com/2014/07/managing-private-nix-packages-outside.html">Managing private Nix packages outside the Nixpkgs tree</a>. As with previous years, this blog post remains the most popular because it is very practical and unanswered in official Nix documentation.</li>
<li><a href="https://sandervanderburg.blogspot.com/2012/11/on-nix-and-gnu-guix.html">On Nix and GNU Guix</a>. This blog post used to be my most popular blog post for a long time, and still remains the second most popular. I believe this can be attributed to the fact that this comparison is still very relevant.</li>
<li><a href="https://sandervanderburg.blogspot.com/2015/04/an-evaluation-and-comparison-of-snappy.html">An evaluation and comparison of Snappy Ubuntu</a>. Also a very popular blog post since 2015. It seems that the comparison with Snappy and Flatpak (a tool with similar objectives) remains relevant.</li>
<li><a href="https://sandervanderburg.blogspot.com/2016/01/disnix-05-release-announcement-and-some.html">Disnix 0.5 release announcement and some reflection</a>. This is a blog post that I wrote in 2016 and suddenly appeared in the overall top 10 this year. I am not sure why this has become so relevant all of a sudden.</li>
<li><a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">On using Nix and Docker as deployment solutions: similarities and differences</a>. This is a blog post that I wrote last year to compare Nix and Docker and explain in what ways they are similar and different. It seems to be very popular despite the fact that it was not posted on discussion sites such as Reddit and Hackernews.</li>
<li><a href="https://sandervanderburg.blogspot.com/2013/02/yet-another-blog-post-about-object.html">Yet another blog post about Object Oriented Programming and JavaScript</a>. This explanation blog post is pretty old but seems to stay relevant, despite the fact that modern JavaScript has a <i>class</i> construct.</li>
<li><a href="https://sandervanderburg.blogspot.com/2013/06/setting-up-multi-user-nix-installation.html">Setting up a multi-user Nix installation on non-NixOS systems</a>. Setting up multi-user Nix installations on non-NixOS machines used to be very cumbersome, but fortunately that has been improved in the recent versions. Still, the discussion seems to remain relevant.</li>
<li><a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">An alternative explanation of the Nix package manager</a>. An alternative explanation that I consider to be better of two that I wrote. It seems to remain popular because I refer to it a lot.</li>
<li><a href="https://sandervanderburg.blogspot.com/2015/03/on-nixops-disnix-service-deployment-and.html">On NixOps, Disnix, service deployment and infrastructure deployment</a>. A very popular blog post, that has dropped somewhat in popularity. It still seems that the tools and the discussion is relevant.</li>
<li><a href="https://sandervanderburg.blogspot.com/2013/09/composing-fhs-compatible-chroot.html">Composing FHS-compatible chroot environments with Nix (or deploying steam in NixOS)</a>. An old blog post, but it remains relevant because it addresses a very important compatibility concern with binary-only software and a common Nix-criticism that it is not FHS-compatible.</li>
</ol>
<br />
<h2>Conclusion</h2>
<br />
As with 2020, 2021 has been quite a year. I hope everybody stays safe and healthy.<br />
<br />
The remaining thing I'd like to say is: HAPPY NEW YEAR!!!<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/a/AVvXsEhF42Djk2fkKuAO5cgshh468ykDbbKTOgekIu0fU0Ph3Cxo-hX_lx_doqiphrkQt4pCPqhz5ed1ypNxRYKcKi8me4k35QARdlyA3N5dYsWnXUIc__YRX3UvZWaPscIhl8v8kud805J5nKprrhHxD0je5nYBxRf0_jEJWKUy3CUG6jD07-Lv0k8mg_Nqdg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="253" data-original-width="506" src="https://blogger.googleusercontent.com/img/a/AVvXsEhF42Djk2fkKuAO5cgshh468ykDbbKTOgekIu0fU0Ph3Cxo-hX_lx_doqiphrkQt4pCPqhz5ed1ypNxRYKcKi8me4k35QARdlyA3N5dYsWnXUIc__YRX3UvZWaPscIhl8v8kud805J5nKprrhHxD0je5nYBxRf0_jEJWKUy3CUG6jD07-Lv0k8mg_Nqdg"/></a></div>
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-51678605473524951092021-10-19T22:41:00.000+02:002021-10-19T22:41:25.587+02:00Using my Commodore Amiga 500 in 2021<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6sc0H2vzMI-f6rXSUptI0haWmOqjsT96yF-q0eNvoBI5Nu-31SJnLQJ_foSzF-WP6YunjrJvXHnKRYmHUm6H6B87CSkfxY6cJuZRYGmOMNTWI4i8r8VyKQXc0iDKljervl-s4qfZUiDhC/s2048/playgame.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg6sc0H2vzMI-f6rXSUptI0haWmOqjsT96yF-q0eNvoBI5Nu-31SJnLQJ_foSzF-WP6YunjrJvXHnKRYmHUm6H6B87CSkfxY6cJuZRYGmOMNTWI4i8r8VyKQXc0iDKljervl-s4qfZUiDhC/s600/playgame.jpg"/></a></div>
<br />
Due to the <a href="https://www.nu.nl/coronavirus/6146047/reproductiegetal-coronavirus-nadert-3-hoogste-niveau-ooit.html">high number of new COVID-19 infections in my home country last summer</a>, I had to "improvise" yet another summer holiday. As a result, I finally found the time to tinker with my old computers again after a very long time of inactivity.<br />
<br />
As I have explained in two blog posts that I wrote over ten years ago, the <a href="https://sandervanderburg.blogspot.com/2011/03/first-computer.html">first computer</a> (a Commodore 128 bought by my parents in 1985) and <a href="https://sandervanderburg.blogspot.com/2011/07/second-computer.html">second computer</a> (Commodore Amiga 500 bought by my parents in 1992) that I ever used, are still in my possession.<br />
<br />
In the last few years, I have used the Commodore 128 a couple of times, but I have not touched the Commodore Amiga 500 since I wrote my blog post about it ten years ago.<br />
<br />
It turns out that the Commodore Amiga 500 still works, but I ran into a number of problems:<br />
<br />
<ul>
<li><strong>A black and white display</strong>. I used to have a monitor, but it broke down in 1997. Since then, I have been using <a href="https://en.wikipedia.org/wiki/Genlock">Genlock</a> device to attach the Amiga to a TV screen. Unfortunately, in 2021 the Genlock device no longer seems to work.<br />
<br />
The only display option I had left is to attach the Amiga to a TV with an RCA to SCART cable by using the monochrome video output. The downside is that it is only capable of displaying a black and white screen.</li>
<li><strong>No secondary disk drive</strong>. I used to have two 3.5-inch <a href="https://en.wikipedia.org/wiki/Floppy_disk_format">double density disk drives</a>: an internal disk drive (inside the case) and an external disk drive that you can attach to the disk drive port.<br />
<br />
The external disk drive still seems to respond when I insert a floppy disk (the led blinks), but it no longer seems to be capable of reading any disks.</li>
<li><strong>Bad hard drive and expansion slot problems</strong>. The expansion board (that contains the hard drive) seems to give me all kinds of problems.<br />
<br />
Sometimes the Amiga completely fails to detect it. In other occasions, I ran into crashes causing the filesystem to return me write errors. Attempting to repair them typically results in new errors.<br />
<br />
After thoroughly examining the disk with <a href="https://aminet.net/package/disk/salv/DiskSalv">DiskSalv</a>, I learned that the drive has physical damage and needs to be replaced.</li>
</ul>
<br />
I also ran into an interesting problem from a user point of view -- exchanging data to and from my Amiga (such as software downloaded from the Internet and programs that I used to write) is quite a challenge. In late 1996, when my parents switched to the PC, I used floppy disks to exchange data.<br />
<Br />
In 2021, floppy drives have completely disappeared from all modern computers. In the rare occasion that I still need to read a floppy disk, I have an external USB floppy drive at my disposal, but it is only capable of reading high density 3.5-inch floppy disks. A Commodore Amiga's standard floppy drive (with the exception of the Amiga 4000) is only capable of reading double density disks.<br />
<br />
Fortunately, I have discovered that there are still many things possible with old machines. I brought both my Commodore 128 and Commodore 500 to the <a href="https://www.homecomputermuseum.nl">Home Computer Museum in Helmond</a> for repairs. Furthermore, I have ordered all kinds of replacement peripherals.<br />
<br />
Getting it all to work, turned out to be quite a challenge. Eventually, I have managed to overcome all my problems and the machine works like a charm again.<br />
<br />
In this blog post, I will describe what problems I faced and how I solved them.<br />
<br />
<h2>Some interesting properties of the Amiga</h2>
<br />
I often receive many questions from all kinds of people who want to know why it is so interesting to use such an old machine. Aside from nostalgic reasons, I think the machine is an interesting piece of computer history. At the time the first model was launched: the Amiga 1000 in 1985, the machine was far ahead of its time and provided unique multimedia capabilities.<br />
<br />
Back in the late 80s, system resources were very limited (such as CPU, RAM and storage) compared to modern machines, but there were all kinds of interesting facilities to overcome their design limitations.<br />
<br />
For example, the original Amiga 500 model only had 512 KiB of RAM and 32 configurable color registers. Colors can be picked out of a range of 4096 possible colors.<br />
<br />
Despite only having the ability to configure a maximum 32 distinct colors, it could still display photo-realistic images:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTZbeM2of4l52tOzW0YjfCE6W2G-gc3oPNy-66U5lqlzVapLz2nC5haDdgw7bMZnmKiSQHZQNA9Gr7ZqoUDwhkew8ftIUWu5W9Nmg4yvAw0TbZS4WYpdUma5qwoPIMGpED0fqfJrM9cszw/s752/photorealistic.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="460" data-original-height="572" data-original-width="752" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjTZbeM2of4l52tOzW0YjfCE6W2G-gc3oPNy-66U5lqlzVapLz2nC5haDdgw7bMZnmKiSQHZQNA9Gr7ZqoUDwhkew8ftIUWu5W9Nmg4yvAw0TbZS4WYpdUma5qwoPIMGpED0fqfJrM9cszw/s400/photorealistic.png"/></a></div>
<br />
As can be seen, the screen shot above clearly has more than 32 distinct colors. This is made possible by using a special screen mode called Hold-and-Modify (HAM).<br />
<br />
In HAM mode, a pixel's color can be picked from a palette of 16 base colors, or a color component (red, green or blue) of the adjacent pixel can be changed. The HAM screen mode makes it possible to use all possible 4096 colors, albeit with some restrictions on the adjacent color values.<br />
<br />
Another unique selling point of the Amiga were its sound capabilities. It could mix 4 audio channels in hardware, and easily combined with graphics, animations and games. The Amiga has all kinds of interesting music productivity software, such as <a href="https://en.wikipedia.org/wiki/ProTracker">ProTracker</a>, that I used a lot.<br />
<br />
To make all these multimedia features possible, the Amiga has its own unique hardware architecture:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIy2Y-beRLFRD4p3Wr2nsLKLMdB0f16dD_IsYTqFvnq7CMJ72Owu5yOuyRnh-KCAf76fNM_dJLRPfs4l6IPnUgJ3iciCjkMgo5S55Ku06WBIDOwBdPJXjM9b8G9kaYETrl_aA1oDMJb07W/s427/chips.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="207" data-original-width="427" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgIy2Y-beRLFRD4p3Wr2nsLKLMdB0f16dD_IsYTqFvnq7CMJ72Owu5yOuyRnh-KCAf76fNM_dJLRPfs4l6IPnUgJ3iciCjkMgo5S55Ku06WBIDOwBdPJXjM9b8G9kaYETrl_aA1oDMJb07W/s600/chips.png"/></a></div>
<br />
The above diagram provides a simplified view of the most important chips in the Amiga 500 and how they are connected:<br />
<br />
<ul>
<li>On the left, the CPU is shown: a <strong>Motorola 68000</strong> that runs at approximately 7 MHz (the actual clock speeds differ somewhat on a PAL and NTSC display). The CPU is responsible for doing calculations and executing programs.</li>
<li>On the right, the unique Amiga chips are shown. Each of them has a specific purpose:<br />
<ul>
<li><a href="http://theamigamuseum.com/the-hardware/the-ocs-chipset/denise/"><strong>Denise</strong></a> (Display ENabler) is responsible for producing the RGB signal for the display, provides bitplane registers for storing graphics data, and is responsible for displaying sprites.</li>
<li><a href="http://theamigamuseum.com/the-hardware/the-ocs-chipset/agnus/"><strong>Agnus</strong></a> (Address GeNerator UnitS) provides a <strong>blitter</strong> (that is responsible for quick transfers of data in chip memory, typically graphics data), and a <strong>copper</strong>: a programmable co-processor that is aligned with the video beam.<br />
<br />
The copper makes all kinds of interesting graphical features possible, while keeping the CPU free for work. For example, the following screenshot of the game <a href="https://www.lemonamiga.com/games/details.php?id=2266">Trolls</a>:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3HXtBIg8f13fZus2Jp9ZGh97wXFDK49ygzkPt3WnUgSCBTqYa5wofedPKuWjaC7e6T4f5pdNSu290vED4AhF3AmwKzFB_FO2jLqtvfufrJCrTcdeL3AO9CnR3rBOHH9xkimpJ_20tGCSe/s960/trolls.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="460" data-original-height="540" data-original-width="960" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3HXtBIg8f13fZus2Jp9ZGh97wXFDK49ygzkPt3WnUgSCBTqYa5wofedPKuWjaC7e6T4f5pdNSu290vED4AhF3AmwKzFB_FO2jLqtvfufrJCrTcdeL3AO9CnR3rBOHH9xkimpJ_20tGCSe/s600/trolls.png"/></a></div>
<br />
clearly contains more than 32 distinct colors. For example, the rainbow-like background provides a unique color on each scanline. The copper is used in such a way that the value of the background color register is changed on each scanline, while the screen is drawn.<br />
<br />
The copper also makes it possible to switch between screen modes (low resolution, high resolution) on the same physical display, such as in the Workbench:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDdQwMQIQC6J_c29ZzT4uryxVZ6BTXh5Ii0JBt97P1HUUWateFwAFtHKEfCZeR0OGkjvNkxA7ZUHcPNluulqCc9RSX7Bii0UoSrp-Du1p19afK4NcOJfvCwD8SJVA-Ri_BxJmA1hEw_FDQ/s752/dpaint.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="460" data-original-height="572" data-original-width="752" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiDdQwMQIQC6J_c29ZzT4uryxVZ6BTXh5Ii0JBt97P1HUUWateFwAFtHKEfCZeR0OGkjvNkxA7ZUHcPNluulqCc9RSX7Bii0UoSrp-Du1p19afK4NcOJfvCwD8SJVA-Ri_BxJmA1hEw_FDQ/s600/dpaint.png"/></a></div>
<br />
As can be seen in the above screenshot, the upper part of the screen shows Deluxe Paint in low-res mode with its own unique set of colors, while the lower part shows the workbench in high resolution mode (with a different color palette). The copper can change the display properties while the screen is rendered, while keeping the CPU free to do work.<br />
</li>
<li>
<a href="http://theamigamuseum.com/the-hardware/the-ocs-chipset/paula/"><strong>Paula</strong></a> is a multi-functional chip that provides sound support, such as processing sample data from memory and mixing 4 audio channels. Because it does mixing in hardware, the CPU is still free to do work.<br />
<br />
It also controls the disk drive, serial port, mouse and joysticks.
</li>
</ul>
</li>
<li>All the chips in the above diagram require access to memory. <a href="http://theamigamuseum.com/the-hardware/chip-and-fast-ram/"><strong>Chip RAM</strong></a> is memory that is shared between all chips. As a consequence, they share the same memory bus.<br />
<br />
A shared bus imposes speed restrictions -- on even clock cycles the CPU can access chip memory, while on the uneven cycles the chips have memory access.<br />
<br />
Many Amiga programs are optimized in such a way that all CPU's memory access operations are at even clock cycles as much as possible. When the CPU needs to access memory on uneven clock cycles, it is forced to wait, losing execution speed.</li>
<li>An Amiga can also be extended with <strong>Fast RAM</strong> that does not suffer from any speed limitations. Fast RAM is on a different memory bus that can only be accessed by the CPU and not by any of the chips.<br />
<br />
(As a sidenote: there is also Slow RAM that is not shown in the diagram. It falls in between chip and fast RAM. Slow RAM is memory that is exclusive to the CPU, but cannot be used on uneven clock cycles).</li>
</ul>
<br />
Compared to other computer architectures used at the same time, such as the PC, 7 MHz of CPU clock speed does not sound all that impressive, but the combination of all these autonomous chips working together is what makes many incredible multimedia properties possible.<br />
<br />
<h2>My Amiga 500 specs</h2>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh078kZ_mDNMz9EJ7R1USFLZ1aQ_X7zYciUGKLpWhW1l0zArMRUqDsWJ8ai8XrMAVSdyUZ7EEhI_T0WIK44h5n5_TQmV2UK_IUoiSXBSUsLN2QWMAVj8PngaL-NeCBUlCdXbxJpOHC0J5fF/s1290/amigainside.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="968" data-original-width="1290" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh078kZ_mDNMz9EJ7R1USFLZ1aQ_X7zYciUGKLpWhW1l0zArMRUqDsWJ8ai8XrMAVSdyUZ7EEhI_T0WIK44h5n5_TQmV2UK_IUoiSXBSUsLN2QWMAVj8PngaL-NeCBUlCdXbxJpOHC0J5fF/s600/amigainside.jpg"/></a></div>
<br />
When my parents bought my Commodore Amiga 500 machine in 1992, it still had the original chipset and 512 KiB of Chip RAM. The only peripherals were an external 3.5-inch floppy drive and a kickstart switcher allowing me switch between Kickstart 1.3 and 2.0. (The kickstart are portions of the Amiga operating system residing in the ROM).<br />
<br />
Some time later, the Agnus and Denise chips were upgraded (we moved from the Original Chipset to the Enchanced Chipset), extending the amount of chip RAM to 1 MiB and making it possible to use super high resolution screen modes.<br />
<br />
At some point, we bought a <a href="http://amiga.resource.cx/exp/powerpc">KCS PowerPC board</a> making it possible to emulate a PC and run MS-DOS applications. Although the product calls itself an emulator, it is also provides a board that extends the hardware with a number of interesting features:<br />
<br />
<ul>
<li>A 10 MHz <a href="https://en.wikipedia.org/wiki/NEC_V20#V30">NEC V30 CPU</a> that is pin and instruction-compatible with an Intel 8086/8088 CPU. Moreover, it implements some 80186 instructions, some of its own instructions, and is between 10-30% faster.</li>
<li>1 MiB of RAM that can be used by the NEC V30 CPU for <a href="https://en.wikipedia.org/wiki/Conventional_memory">conventional</a> and <a href="https://en.wikipedia.org/wiki/Upper_memory_area">upper memory</a>. In addition, the board's memory can also be used by the Amiga as additional chip RAM, fast RAM and as a RAM disk.</li>
<li>A clock (powered by a battery) so that you do not have reconfigure the date and time on startup. This PC clock can also be used in Amiga mode.</li>
</ul>
<br />
Eventually, we also obtained a hard drive. The Amiga 500 does not include any hard drive, nor has it an internal hard drive connector.<br />
<br />
Nonetheless, it can be extended through the Zorro expansion slot with an extension board. We obtained this extension board: <a href="http://amiga.resource.cx/exp/evolution500">MacroSystem evolution</a> providing a SCSI connector, a whopping 8 MiB of fast RAM and an additional floppy drive connector. To the SCSI connector, a 120 MiB Maxtor 7120SR hard-drive was attached.<br />
<br />
<h2>Installing new and replacement peripherals</h2>
<br />
In this section, I will describe my replacement peripherals and what I did to make them work.<br />
<br />
<h2>RCB to SCART cable</h2>
<br />
As explained in the introduction, I no longer have a monitor and the Genlock device is broken, only making it possible to have a black and white display.<br />
<br />
Fortunately, all kinds of replacement options seem to be available to connect an Amiga to a more modern display.<br />
<br />
I have ordered an <a href="https://amigastore.eu/en/208-scart-cable-amiga-rgb-to-scart-tv-sound-modified.html">RGB to SCART cable</a>. It can be attached to the RGB and audio output of the Amiga and to the SCART input on my LCD TV.<br />
<br />
<h2>GoTek floppy emulator</h2>
<br />
Another problem is that the secondary floppy drive is broken and could not be repaired.<br />
<br />
Even if I could find a suitable replacement drive, floppy disks are very difficult media to use for data exchange these days.<br />
<br />
Even with an old PC that still has an internal floppy drive (capable of reading both high and double density floppy disks), exchanging information remains difficult -- due to limitations of a PC floppy controller, a PC is incapable of reading Amiga disks, but an Amiga can read and write to PC floppy disks. A PC formatted floppy disk has less storage capacity than an Amiga formatted disk.<br />
<br />
There is also an interesting alternative to a real floppy drive: <a href="http://www.gotekemulator.com">the GoTek floppy emulator</a>.<br />
<br />
The GoTek floppy emulator works with disk image files stored on a USB memory stick. The numeric digit on the display indicates which disk image is currently inserted into the drive. With the rotating switch you can switch between disk images. It operates at the same speed as a real disk drive and produces similar sounds.<br />
<br />
Booting from floppy disk 0 starts a program that allows you to configure disk images for the remaining numeric entries:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPSIebAKfRHlZfbb6zxwNEoxCFr8cpXkWnvIYtWOIE_5khMRsXsRecruXX0ETCc3BKFlpXjWhBUz_I1IxjC50E_7wdUv9o1-FHkDuA6oCY8zVx_AAGB4iDnfDGKkDMHGUOpLso5Fj6joFr/s2048/imageselect.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPSIebAKfRHlZfbb6zxwNEoxCFr8cpXkWnvIYtWOIE_5khMRsXsRecruXX0ETCc3BKFlpXjWhBUz_I1IxjC50E_7wdUv9o1-FHkDuA6oCY8zVx_AAGB4iDnfDGKkDMHGUOpLso5Fj6joFr/s600/imageselect.jpg"/></a></div>
<br />
The GoTek floppy emulator can act both as a replacement for the internal floppy drive as well as an external floppy drive and uses the same connectors.<br />
<br />
I have decided to buy <a href="https://amigastore.eu/en/323-usb-floppy-emulator-gotek.html">an external model</a>, because the internal floppy drive still works and I want to keep the machine as close to the original as possible. I can turn the GoTek floppy drive into the primary disk drive, by using the DF0 switch on the right side of the Amiga case.<br />
<br />
Because all disk images are stored on a FAT filesystem-formatted USB stick, makes exchanging information with a PC much easier. I can transfer the same disk files that I can use in the Amiga emulator to the USB memory stick on my PC and then natively use them on a real Amiga.<br />
<br />
<h2>SCSI2SD</h2>
<br />
As explained earlier, the 29-year old SCSI hard drive connected to the expansion board is showing all kinds of age-related problems. Although I could search for a compatible second-hand hard drive that was built in the same era, it is probably not going to last very long either.<br />
<br />
Fortunately, for retro-computing purposes, an interesting replacement device was developed: the <a href="http://www.codesrc.com/mediawiki/index.php/SCSI2SD">SCSI2SD</a>, that can be used as drop-in replacement for a SCSI hard drive and other kinds of SCSI devices.<br />
<br />
This device can be attached to the same SCSI and power connector cables that the old hard drive uses. As the name implies, its major difference is that is uses a (modern) SD-card for storage.<br />
<br />
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjjx8bbMVKTltOQcU4Br_rd1AIpLAnUHkvv7prT4t_OLjaftYc9dlG3PHN8zuLPtKNmNhemqDTfxGzus6-g7PXGIdunGhmHyUUDT4tM71RNvQ0V7nnqdtvdQ4R82mYkbs8VprDXGV2Yf_O/s2048/oldhd.jpg" style="display: block; padding: 1em 0; text-align: center; clear: left; float: left;"><img alt="" border="0" height="320" data-original-height="2048" data-original-width="1536" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjjjx8bbMVKTltOQcU4Br_rd1AIpLAnUHkvv7prT4t_OLjaftYc9dlG3PHN8zuLPtKNmNhemqDTfxGzus6-g7PXGIdunGhmHyUUDT4tM71RNvQ0V7nnqdtvdQ4R82mYkbs8VprDXGV2Yf_O/s320/oldhd.jpg"/></a></div>
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjktCtEzcpb7Tyf7lfxEjTjs3DNbFbOmt_-ouZwT4yOwwayGeMfMsh7qgrA3ogxQnm7kSo_QYkOz3jmOy-6SvjS5A2IXrV5dWeS_fXJ_qxTk3LeEVUNQad82GagFj9AdYOwltVfMuF4-i2g/s2048/scsi2sd.jpg" style="display: block; padding: 1em 0; text-align: center; clear: right; float: right;"><img alt="" border="0" height="320" data-original-height="2048" data-original-width="1536" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjktCtEzcpb7Tyf7lfxEjTjs3DNbFbOmt_-ouZwT4yOwwayGeMfMsh7qgrA3ogxQnm7kSo_QYkOz3jmOy-6SvjS5A2IXrV5dWeS_fXJ_qxTk3LeEVUNQad82GagFj9AdYOwltVfMuF4-i2g/s320/scsi2sd.jpg"/></a></div>
<div style="clear: both;"></div>
<br />
The left picture (shown above) shows the interior of the MacroSystem evolution board's case with the original Maxtor hard drive attached. On the right, I have replaced the hard drive with a SCSI2SD board (that uses a 16 GiB SD-card for storage).<br />
<br />
Another nice property of the SCSI2SD is that an SD card offers much more storage capacity. The smallest SD card that I could buy offers 16 GiB of storage, which is a substantially more than the 120 MiB that the old Maxtor hard drive from 1992 used to offer.<br />
<br />
Unfortunately, the designers of the original Amiga operating system did not forsee that people would use devices with so much storage capacity. From a technical point of view, AmigaOS versions 3.1 and older are incapable of addressing more than 4 GiB of storage per device.<br />
<br />
In addition to the operating system's storage addressing limit, I discovered that there is another limit -- the SCSI controller on the MacroSystem evolution extension board is unable to address more than 1 GiB of storage space per SCSI device. Trying to format a partition beyond this 1 GiB boundary results in a "DOS disk not found" error. This limit does not seem to be documented anywhere in the MacroSystem evolution manual.<br />
<br />
To cope with these limitations, the SCSI2SD device can be configured in such a way that it stays within the boundaries of the operating system. To do this, it needs to be connected to a PC with a micro USB cable and configured with the <i>scsi2sd-util</i> tool.<br />
<br />
After many rounds of trial and error, I ended up using the following settings:<br />
<br />
<ul>
<li>Enable SCSI terminator (V5.1 only): on</li>
<li>SCSI Host Speed: Normal</li>
<li>Startup Delay (seconds): 0</li>
<li>SCSI Selection Delay: 255</li>
<li>Enable Parity: on</li>
<li>Enable Unit Attention: off</li>
<li>Enable SCSI2 Mode: on</li>
<li>Disable glitch filter: off</li>
<li>Enable disk cache (experimental): off</li>
<li>Enable SCSI Disconnect: off</li>
<li>Respond to short SCSI selection pulses: on</li>
<li>Map LUNS to SCSI IDs: off</li>
</ul>
<br />
Furthermore, the SCSI2SD allows you to configure multiple SCSI devices and put restrictions on how much storage from the SD card can be used per device.<br />
<br />
I have configured one SCSI device (representing a 1 GiB hard drive) with the following settings:<br />
<br />
<ul>
<li>Enable SCSI Target: on</li>
<li>SCSI ID: 0</li>
<li>Device Type: Hard Drive</li>
<li>Quirks Mode: None</li>
<li>SD card start sector: 0</li>
<li>Sector size (bytes): 512</li>
<li>Sector count: leave it alone</li>
<li>Device size: 1 GB</li>
</ul>
<br />
I left the <i>Vendor</i>, <i>ProductID</i>, <i>Revision</i> and <i>Serial Number</i> values untouched. The <i>Sector count</i> is derived automatically from the start sector and device size.<br />
<br />
Before using the SD card, I recommend to erase it first. Strictly speaking, this is not required, but I have learned in a very painful way that <i>DiskSalv</i>, a tool that is frequently used to fix corrupted Amiga file systems, may get confused if there are traces of a previous filesystem left behind. As a result, it may incorrectly treat files as invalid file references causing further corruption.<br />
<br />
On Linux, I can clear the memory of the SD card with the following command (<i>/dev/sdb</i> refers to the device file of my SD-card reader):<br />
<br />
<pre>
$ dd if=/dev/zero of=/dev/sdb bs=1M status=progress
</pre>
<br />
After clearing the SD card, I can insert it into the SCSI2SD device, do the partitioning and perform the installation of the Workbench. This process turns out to be more tricky than I thought -- the MacroSystem evolution board seems to only include a manual that is in German, requiring me to brush up my German reading skills.<br />
<br />
The first step is to use the HDToolBox tool (included with the Amiga Workbench 2.1 installation disk) to detect the hard disk.<br />
<br />
(As a sidenote: check if the SCSI cable is properly attached to both the SCSI2SD device, as well as the board. In my first attempt, the firmware was able to detect that there was a SCSI device with LUN 0, but it could not detect that it was a hard drive. After many rounds of trial and error, I discovered that the SCSI cable was not properly attached to the extension board!).<br />
<br />
By default, HDToolBox works with the standard SCSI driver bundled with the Amiga operating system (<i>scsi.device</i>) which is not compatible with the SCSI controller on the MacroSystem Evolution board.<br />
<br />
To use the correct driver, I had to configure HDToolBox to use a different driver, by opening a shell session and running the following command-line instructions:<br />
<br />
<pre>
Install2.1:HDTools
HDToolBox evolution.device
</pre>
<br />
In the above code fragment, I pass the driver name: <i>evolution.device</i> as a command-line parameter to <i>HDToolBox</i>.<br />
<br />
With the above configuration setting, the SCSI2SD device gets detected by HDToolBox:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiihHP8DpzYFpy4sQuq5YjbFb7sTsm2FtGMYhIZ1ecP2Bz7lenSvc6Jznpwu_ASP8GlhSQn4YNG35ikRBRrFKZsT0COkEjUg_nX8KSLg7Ij_QSXeDLhAmjxdg8RvZgcyL2NyMtvXYdZenV5/s2048/hdtoolbox.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiihHP8DpzYFpy4sQuq5YjbFb7sTsm2FtGMYhIZ1ecP2Bz7lenSvc6Jznpwu_ASP8GlhSQn4YNG35ikRBRrFKZsT0COkEjUg_nX8KSLg7Ij_QSXeDLhAmjxdg8RvZgcyL2NyMtvXYdZenV5/s600/hdtoolbox.jpg"/></a></div>
<br />
I did the partitioning of my SD-card hard drive as follows:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBqWlP2sAluvlWKZxxTQv_4eib6CM-dj5UbOlCIO85TNO5DWOQSDIK4BMNrSuEVAuBIB2OUZdDnUMblw6NggjFHeQK0D3uoK77yBBHqhtFkSiD957amb-yF6bndyGZZzwGy9yTZdfRMdZ4/s2048/partitions.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgBqWlP2sAluvlWKZxxTQv_4eib6CM-dj5UbOlCIO85TNO5DWOQSDIK4BMNrSuEVAuBIB2OUZdDnUMblw6NggjFHeQK0D3uoK77yBBHqhtFkSiD957amb-yF6bndyGZZzwGy9yTZdfRMdZ4/s600/partitions.jpg"/></a></div>
<br />
<table>
<tr>
<th>Partition Device Name</th>
<th>Capacity</th>
<th>Bootable</th>
</tr>
<tr>
<td>DH0</td>
<td>100 MiB</td>
<td>yes</td>
</tr>
<tr>
<td>KCS</td>
<td>100 MiB</td>
<td>no</td>
</tr>
<tr>
<td>DH1</td>
<td>400 MiB</td>
<td>no</td>
</tr>
<tr>
<td>DH2</td>
<td>400 MiB</td>
<td>no</td>
</tr>
</table>
<br />
I did not change any advanced file system settings. I have configured all partitions to use mask: <i>0xfffffe</i> and max transfer: <i>0xffffff</i>.<br />
<br />
Beyond creating partitions, there was another tricky configuration aspect I had to take into account -- I had to reserve the second partition (the <i>KCS</i> partition) as a hard drive for the KCS PowerPC emulator.<br />
<br />
In my first partitioning attempt, I configured the KCS partition as the last partition, but that seems to cause problems when I start the KCS PowerPC emulator, typically resulting in a very slow startup followed by a system crash.<br />
<br />
It appears that this problem is a caused by a memory addressing problem. Putting the KCS partition under the 200 MiB limit seems to fix the problem. Since most addressing boundaries are power of 2 values, my guess is that the KCS PowerPC emulator expects a hard drive partition to reside below the 256 MiB limit.<br />
<br />
After creating the partitions and rebooting the machine, I can format them. For some unknown reason, a regular format does not seem to work, so I ended up doing a quick format instead.<br />
<br />
Finally, I can install the workbench on the <i>DH0:</i> partition by running the Workbench installer (that resides in the: <i>Install2.1</i> folder on the installation disk):<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhectxFku_Yh_q4WIha_PnZ36oL25V-eboAag09SfU9QeENRxwHhW8S8rj6lUsyzmorEoIdFJV1Xa5IQO8VwrlmEy1YmWk-7AxvQpWIe2fWgb0-sQUYlUSq1ZseLoVP46u-I6fpCV7Hfd1F/s2048/installation.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhectxFku_Yh_q4WIha_PnZ36oL25V-eboAag09SfU9QeENRxwHhW8S8rj6lUsyzmorEoIdFJV1Xa5IQO8VwrlmEy1YmWk-7AxvQpWIe2fWgb0-sQUYlUSq1ZseLoVP46u-I6fpCV7Hfd1F/s600/installation.jpg"/></a></div>
<br />
<h2>Null modem cable</h2>
<br />
The GoTek floppy drive and SCSI2SD already make it much easier to exchange data with my Amiga, but they are still somewhat impractical for exchanging small files, such as Protracker modules or software packages (in LhA format) downloaded from <a href="https://aminet.net">Aminet</a>.<br />
<br />
I have also bought a good old-fashioned <a href="https://amigastore.eu/en/275-cable-null-modem-amiga.html">null modem cable</a> that can be used to link two computers through their serial ports. Modern computers no longer have a <a href="https://en.wikipedia.org/wiki/RS-232">RS-232</a> serial port, but you can still use an USB to RS-232 converter that indirectly makes it possible to link up with a USB connection.<br />
<br />
To link up, the serial port settings on both ends need to be the same and the baud rate should not be to high. I have configured <a href="http://darioportfolio.com/tutorial/amiga_adf.html">the following settings on my Amiga</a> (configured with the <i>SYS:Prefs/Serial</i> preferences program):<br />
<br />
<ul>
<li>Baud rate: 19,200</li>
<li>Input buffer size: 512</li>
<li>Handshaking: RTS/CTS</li>
<li>Parity: None</li>
<li>Bits/Char: 8</li>
<li>Stop Bits: 1</li>
</ul>
<br />
With a terminal client, such as <a href="https://www.amigawiki.org/doku.php?id=en:software:communication:ncomm">NComm</a>, I can make a terminal connection to my Linux machine. By installing <a href="https://www.ohse.de/uwe/software/lrzsz.html">lrzsz</a> on my Linux machine, I can exchange files by using the <a href="https://en.wikipedia.org/wiki/ZMODEM">Zmodem</a> protocol.<br />
<br />
There are <a href="https://www.wicksall.net/Amiga/TransferFilesBetweenLinuxAndAmiga">a variety of ways to link my Amiga with a Linux PC</a>. A quick and easy way to exchange files, is by starting <a href="https://github.com/npat-efault/picocom">picocom</a> on the Linux machine with the following parameters:<br />
<br />
<pre>
$ picocom --baud 19200 \
--flow h \
--parity n \
--databits 8 \
--stopbits 1 \
/dev/ttyUSB0
</pre>
<br />
After starting Picocom, I can download files from my Linux PC by selecting: <i>Transfer -> Download</i> in the NComm menu. This action opens a file dialog on my Linux machine that allows me to pick the files that I want to download.<br />
<br />
Similarly, I can upload files to my Linux machine by selecting <i>Transfer -> Upload</i>. On my Linux machine, a file dialog appears that allows me to pick the target directory where the uploaded files need to be stored.<br />
<br />
In addition to simple file exchange, I can also expose a Linux terminal over a serial port and use my Amiga to remotely provide command-line instructions:<br />
<br />
<pre>
$ agetty --flow-control ttyUSB0 19200
</pre>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGRi9_TkmLGabKc3jMouY2tLido7CsaxhdmhB2jHI29wDAeS0dVG6ZgrBg0ZEBgdRuxHOO6Fc0V9pE25xfx5PeFpo21leoeLX1vtm4wfAv9s8BgufvE4Dzke90ay1e-eNqcrRAFsyArP__/s2048/ncomm2.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1182" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGRi9_TkmLGabKc3jMouY2tLido7CsaxhdmhB2jHI29wDAeS0dVG6ZgrBg0ZEBgdRuxHOO6Fc0V9pE25xfx5PeFpo21leoeLX1vtm4wfAv9s8BgufvE4Dzke90ay1e-eNqcrRAFsyArP__/s600/ncomm2.jpg"/></a></div>
<br />
To keep the terminal screen formatted nicely (e.g. a fixed number of rows and columns) I should run the following command in the terminal session:<br />
<br />
<pre>
stty rows 48 cols 80
</pre>
<br />
By using NComm's upload function, I can transfer files to the current working directory.<br />
<br />
Downloading a file from my Linux PC can be done by running the <i>sz</i> command:<br />
<br />
<pre>
$ sz mod.cool
</pre>
<br />
The above command allows me to download the ProTracker module file: <i>mod.cool</i> from the current working directory.<br />
<br />
It is also possible to remotely administer an Amiga machine from my Linux machine. Running the following command starts a shell session exposed over the serial port:<br />
<br />
<pre>
> NewShell AUX:
</pre>
<br />
With a terminal client on my Linux machine, such as <a href="https://salsa.debian.org/minicom-team/minicom">Minicom</a>, I can run Amiga shell instructions remotely:<br />
<br />
<pre>
$ minicom -b 19200 -D /dev/ttyUSB0
</pre>
<br />
showing me the following output:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvTQkwftWw0POSWfH7PQMWZvQFbes0sFUkJBCHM3TidBidwu57quizkKfCK4lL6lkQgfsEm5HbuS8Z4iLZ9jDsUnHTAbqi0sbkj8hiAHNosZK7cJCPSxOqxSUorPjSgKTmzjAELyrCZRz8/s848/minicom-amiga.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="614" data-original-width="848" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhvTQkwftWw0POSWfH7PQMWZvQFbes0sFUkJBCHM3TidBidwu57quizkKfCK4lL6lkQgfsEm5HbuS8Z4iLZ9jDsUnHTAbqi0sbkj8hiAHNosZK7cJCPSxOqxSUorPjSgKTmzjAELyrCZRz8/s600/minicom-amiga.png"/></a></div>
<br />
<h2>Usage</h2>
<br />
All these new hardware peripherals open up all kinds of new interesting possibilities.<br />
<br />
<h3>Using the SD card in FS-UAE</h3>
<br />
For example, I can detach the SD card from the SCSI2SD device, put it in my PC, and then use the hard drive in the emulator (both <a href="https://fs-uae.net/">FS-UAE</a> and <a href="https://www.winuae.net/">WinUAE</a> seem to work).<br />
<br />
By giving the card reader's device file public permissions:<br />
<br />
<pre>
$ chmod 666 /dev/sdb
</pre>
<br />
FS-UAE, that runs as an ordinary user, should be able to access it. By configuring a hard drive that refers to the device file:<br />
<br />
<pre>
hard_drive_0 = /dev/sdb
</pre>
<br />
we have configured FS-UAE to use the SD card as a virtual hard drive (allowing me to use the exact same installation):<br />
<br />
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBJskgESxuVxl5ft5tC5BU9aK8U6xsFpToFC41bNEGfRNjKrk1xzC-DBK0j7DUYouzeh5unzN7JeDJerfwKLiaC14stJWhRokVIzFfsWaFa_yxiJlboPH6OkUe0mHloC6JsP_rXxaZJGJs/s2048/workbench.jpg" style="display: block; padding: 1em 0; text-align: center; float: left; clear: left;"><img alt="" border="0" width="240" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhBJskgESxuVxl5ft5tC5BU9aK8U6xsFpToFC41bNEGfRNjKrk1xzC-DBK0j7DUYouzeh5unzN7JeDJerfwKLiaC14stJWhRokVIzFfsWaFa_yxiJlboPH6OkUe0mHloC6JsP_rXxaZJGJs/s320/workbench.jpg"/></a></div>
<div class="separator"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS841esPeHwUQZGIz2tcqLt2yDDra8t28qcTtGUyn9lzP-I-hJPksYftWMkfou3kyhhooGKVaOwabtrcvmudZ6MDE4pqP-vLYkovksw8kYu1j1eYaD0qtAovt-Q_fuPj30chB-p3n5bZgp/s1000/hd-in-fsuae.png" style="display: block; padding: 1em 0; text-align: center; float: right; clear: right;"><img alt="" border="0" width="240" data-original-height="609" data-original-width="1000" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgS841esPeHwUQZGIz2tcqLt2yDDra8t28qcTtGUyn9lzP-I-hJPksYftWMkfou3kyhhooGKVaOwabtrcvmudZ6MDE4pqP-vLYkovksw8kYu1j1eYaD0qtAovt-Q_fuPj30chB-p3n5bZgp/s600/hd-in-fsuae.png"/></a></div>
<div style="clear: both;"></div>
<br />
An advantage of using the SD card in the emulator is that we can perform installations of software packages much faster. I can temporarily boost the emulator's execution and disk drive speed, saving me quite a bit of installation time.<br />
<br />
I can also more conveniently transfer large files from my host system to the SD card. For example, I can create a <i>temp</i> folder and expose it in FS-UAE as a secondary virtual hard drive:<br />
<br />
<pre>
hard_drive_1 = /home/sander/temp
hard_drive_1_label = temp
</pre>
<br />
and then copy all files from the <i>temp:</i> drive to the SD card:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVxGybzd6q9OL5a-LZCQX0phkSOQaYGuPY2KrLnL3wRVzxOIGlIzYeZWxH0PImTcX8WBSB733Fm3FeSztDJTRmXKcJRLrOVUrHC-nxf39GsTnU795N4qsDZvrQt3_7LXVxA5wUQW2Um1xD/s960/copytosdcard.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="540" data-original-width="960" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhVxGybzd6q9OL5a-LZCQX0phkSOQaYGuPY2KrLnL3wRVzxOIGlIzYeZWxH0PImTcX8WBSB733Fm3FeSztDJTRmXKcJRLrOVUrHC-nxf39GsTnU795N4qsDZvrQt3_7LXVxA5wUQW2Um1xD/s600/copytosdcard.png"/></a></div>
<br />
<h3>Using the KCS PowerPC board with the new peripherals</h3>
<br />
The GoTek floppy emulator and the SCSI2SD device can also be used in the KCS PowerPC board emulator.<br />
<br />
In addition to Amiga floppy disks, the GoTek floppy emulator can also be used for emulating double density PC disks. The only inconvenience is that it is impossible to format an empty disk on the Amiga for a PC with CrossDOS.<br />
<br />
However, on my Linux machine, it is possible to create an empty 720 KiB disk image, format it as a DOS disk, and put the image file on the USB stick:<br />
<br />
<pre>
$ dd if=/dev/zero of=./mypcdisk.img bs=1k count=720
$ mkdosfs -n mydisk ./mypcdisk.img
</pre>
<br />
The KCS PowerPC emulator also makes it possible to use Amiga's serial and parallel ports. As a result, I can also transfer files from my Linux PC by using a PC terminal client, such as <a href="https://en.wikipedia.org/wiki/Telix">Telix</a>:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7KjE3O0YLlO6L4fsZpyfnp9P8LHIlf-5hpMCkJhbEpVyZSckw9nSZOP6f9f_F5Or8ipuVZy5d6m0j17QwMTC0U6VYwYGFs2sFjVrNH6sG_dj-DpEljjCHUfARsiwyv-aQF-P_18bADFZ9/s2048/telix2.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1234" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi7KjE3O0YLlO6L4fsZpyfnp9P8LHIlf-5hpMCkJhbEpVyZSckw9nSZOP6f9f_F5Or8ipuVZy5d6m0j17QwMTC0U6VYwYGFs2sFjVrNH6sG_dj-DpEljjCHUfARsiwyv-aQF-P_18bADFZ9/s600/telix2.jpg"/></a></div>
<br />
To connect to my Linux PC, I am using almost the same serial port settings as in the Workbench preferences. The only limitation is that I need to lower my baud rate -- it seems that Telix no longer works reliably for baud rates higher than 9600 bits per second.<br />
<br />
The KCS PowerPC board is a very capable PC emulator. Some PC aspects are handled by real hardware, so that there is no speed loss -- the board provides a real 8086/8088 compatible CPU and 1 MiB of memory.<br />
<br />
It also provides its own implementation of a system BIOS and VGA BIOS. As a result, text-mode DOS applications work as well as their native XT-PC counterparts, sometimes even slightly better.<br />
<br />
One particular aspect that is fully emulated in software is CGA/EGA/VGA graphics. As I have explained in a blog written several years ago, <a href="https://sandervanderburg.blogspot.com/2013/11/emulating-amiga-display-modes.html">the Amiga uses bitplane encoding for graphics whereas PC hardware uses chunky graphics</a>. To allow graphics to be displayed, the data needs to be translated into planar graphics format, making graphics rendering very slow.<br />
<br />
For example, it is possible to run Microsoft Windows 3.0 (in <a href="https://en.wikipedia.org/wiki/Real_mode">real mode</a>) in the emulator, but the graphics are rendered very very slowly:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGzsZ1qdoEjzUa8zI9qa8Z4XEb1yVPZLo0NToZGOsk6KOn-qOiH7t-T4VrmyDzpTxUHc_d1h4GB8QMzMvS7UaFYbMc0L_GmW9wNBeyq4trh0mcTs8LCJbNrSo-wzRHmq3sRC67fRPOFy7G/s2048/windows30.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiGzsZ1qdoEjzUa8zI9qa8Z4XEb1yVPZLo0NToZGOsk6KOn-qOiH7t-T4VrmyDzpTxUHc_d1h4GB8QMzMvS7UaFYbMc0L_GmW9wNBeyq4trh0mcTs8LCJbNrSo-wzRHmq3sRC67fRPOFy7G/s600/windows30.jpg"/></a></div>
<br />
Interestingly enough, the game: <a href="http://legacy.3drealms.com/keenhistory/keenhistory4.html">Commander Keen</a> seems to work at an acceptable speed:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwH8wUEPXFJH89-M5idZfS6zOk5mbX2Sr1L9lr_9IQ9mVBm_IVuQI2wVZUf8_DT_6xw_SPZMIhSCQxjfaYsW0lJEhCDQhh0rKm5sB9RpKdmgQhSSe1CS_KAPoLxhpym1unXk8zyo8HrC1A/s2048/keen.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjwH8wUEPXFJH89-M5idZfS6zOk5mbX2Sr1L9lr_9IQ9mVBm_IVuQI2wVZUf8_DT_6xw_SPZMIhSCQxjfaYsW0lJEhCDQhh0rKm5sB9RpKdmgQhSSe1CS_KAPoLxhpym1unXk8zyo8HrC1A/s600/keen.jpg"/></a></div>
<br />
I think Commander Keen runs so fast in the emulator (despite its slow graphics emulation), because of the <a href="https://en.wikipedia.org/wiki/Adaptive_tile_refresh">adaptive tile refresh</a> technique (updating the screen by only redrawing the necessary parts).<br />
<br />
<h2>File reading problems and crashes</h2>
<br />
Although all these replacement peripherals are nice, such as the SCSI2SD, I was also running into a very annoying recurring problem.<br />
<br />
I have noticed that after using the SCSI2SD for a while, sometimes a file may get incorrectly read.<br />
<br />
Incorrectly read files lead to all kinds of interesting problems. For example, unpacking an LhA or Zip archive from the hard drive may sometimes result in one or more CRC errors. I have also noticed subtle screen and audio glitches while playing games stored on the SD card.<br />
<br />
A really annoying problem is when an executable is incorrectly read -- this typically results in program failure crashes with error codes 8000 0003 or 8000 0004. The former error is caused by executing a wrong CPU instruction.<br />
<br />
These read errors do not seem to happen all the time. For example, reading a previously incorrectly read file may actually open it successfully, so it appears that files are correctly written to disk.<br />
<br />
After some investigation and comparing my SD card configuration with the old SCSI hard drive, I have noticed that the read speeds were a bit poor. <a href="https://sysinfo.d0.se/">SysInfo</a> shows me a read speed of roughly 698 KiB per second:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiE40DZUHX_rkBti6ZZe76lohnCYpZ1VisDlP9ylB0-hR1Pv6IZTufZZ1bG29V3Rlhj1f5WuKfP12h3N-aKgNx8Pdrsx_GNmonPmIymQ0TvCiu4w8P8eFSlfHB7cskTxMKXJBSWFum1Fa84/s2048/drivespeed-slow.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiE40DZUHX_rkBti6ZZe76lohnCYpZ1VisDlP9ylB0-hR1Pv6IZTufZZ1bG29V3Rlhj1f5WuKfP12h3N-aKgNx8Pdrsx_GNmonPmIymQ0TvCiu4w8P8eFSlfHB7cskTxMKXJBSWFum1Fa84/s600/drivespeed-slow.jpg"/></a></div>
<br />
By studying the MacroSystem Evolution manual (in German) and comparing the configuration with the Workbench installation on the old hard drive, I discovered that there is a burst mode option that can boost read performance.<br />
<br />
To enable burst mode, I need to copy the Evolution utilities from the MacroSystem evolution driver disk to my hard drive (e.g. by copying <i>DF0:Evolution3</i> to <i>DH0:Programs/Evolution3</i>). and add the following command-line instruction to <i>S:User-Startup</i>:<br />
<br />
<pre style="font-size: 90%;">
DH0:Programs/Evolution3/Utilities/HDParms 0 NOCHANGE NOFORMAT NOCACHE BURST
</pre>
<br />
Resulting in read speeds that are roughly 30% faster:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEpDdMrqKAM3twTDexO8zBv8cDSzwTjl6NOLT4DD2vHOrkMCfzES3ll-vbYk68QChFtuu8WJHKUl9yJqvvuXrkxjojUzoMTDq0iPICxleFfawvwgblXWUOtq_JC4dt8y-27-fc_SjdZGpy/s2048/drivespeed-fast.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjEpDdMrqKAM3twTDexO8zBv8cDSzwTjl6NOLT4DD2vHOrkMCfzES3ll-vbYk68QChFtuu8WJHKUl9yJqvvuXrkxjojUzoMTDq0iPICxleFfawvwgblXWUOtq_JC4dt8y-27-fc_SjdZGpy/s600/drivespeed-fast.jpg"/></a></div>
<br />
Unfortunately, faster read speeds also seem to dramatically increase the likelyhood on read errors making my system quite unreliable.<br />
<br />
I am still not completely sure what is causing these incorrect reads, but from my experiments I know that read speeds definitely have something to do with it. Restoring the configuration to no longer use burst mode (and slower reads), seems to make my system much more stable.<br />
<br />
I also learned that these read problems are very similar to problems reported about a wrong <i>MaxTransfer</i> value. <a href="http://eab.abime.net/showthread.php?t=68426">According to this page</a>, setting it to <i>0x1fe00</i> should be a safe value. I tried adjusting the <i>MaxTransfer</i> value, but it does not seem to change anything.<br />
<br />
Although my system seems to be stable enough after making these modifications, I would still like to expand my knowledge about this subject so that I can fully explain what is going on.<br />
<br />
<h2>Conclusion</h2>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx0a6ErzlEM2RzdWfxgMtvdNmrogyqe-gD5oMJkAqkbws8NDAtAzQ_JkOvFMRvpsD53H-fO-Y8SK_S2UCTM4wrWaV4XtdhGAflAYhioWaCow0yLinBxsGV-Vv0Bl4iAxrnhDi24kR3mUbx/s2048/protracker.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="1536" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjx0a6ErzlEM2RzdWfxgMtvdNmrogyqe-gD5oMJkAqkbws8NDAtAzQ_JkOvFMRvpsD53H-fO-Y8SK_S2UCTM4wrWaV4XtdhGAflAYhioWaCow0yLinBxsGV-Vv0Bl4iAxrnhDi24kR3mUbx/s600/protracker.jpg"/></a></div>
<br />
It took me several months to figure out all these details, but with my replacement peripherals, my Commodore Amiga 500 works great again. The machine is more than 29 years old and I can still run all applications and games that I used to work with in the mid 1990s and more. Furthermore, data exchange with my Linux PC has become much easier.<br />
<br />
Back in the early 90s, I did not have the luxury to download software and information from Internet.<br />
<br />
I also learned many new things about terminal connections. It seems that Linux (because of its UNIX heritage) has all kinds of nice facilities to expose itself as a terminal server.<br />
<br />
After visiting the home computer museum, I became more motivated to preserve my Amiga 500 in the best possible way. It seems that as of today, there are still replacement parts for sale and many things can be repaired.<br />
<br />
My recommendation is that if you still own a classic machine, do not just throw it away. You may regret it later.<br />
<br />
<h2>Future work</h2>
<br />
Aside from finding a proper explanation for the file reading problems, I am still searching for a real replacement floppy drive. Moreover, I still need to investigate whether the Genlock device can be repaired.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-44654118512353624692021-08-31T20:47:00.000+02:002021-08-31T20:47:33.189+02:00A more elaborate approach for bypassing NPM's dependency management features in Nix builds<a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix</a> is a <strong>general purpose</strong> package manager that can be used to automate the deployments of a variety of systems -- it can deploy components written in a variety of programming languages (e.g. C, C++, Java, Go, Rust, Perl, Python, JavaScript) using various kinds of technologies and frameworks, such as Django, Android, and Node.js.<br />
<br />
Another unique selling point of Nix is that it provides <strong>strong reproducibility guarantees</strong>. If a build succeeds on one machine, then performing the same build on another should result in a build that is (nearly) bit-identical.<br />
<br />
Nix improves build reproducibility by complementing build processes with features, such as:<br />
<br />
<ul>
<li>Storing all artifacts in isolation in a so-called <strong>Nix store</strong>: <i>/nix/store</i> (e.g. packages, configuration files), in which every path is unique by prefixing it with an SHA256 hash code derived from all build inputs (e.g. dependencies, build scripts etc.). Isolated paths make it possible for multiple variants and versions of the same packages to safely co-exist.</li>
<li><strong>Clearing environment variables</strong> or setting them to dummy values. In combination with unique and isolated Nix store paths, search environment variables must configured in such a way that the build script can find its dependencies in the Nix store, or it will fail.<br />
<br />
Having to specify all search environment variables may sound inconvenient, but prevents undeclared dependencies to accidentally make a build succeed -- deployment of such a package is very likely to fail on machine that misses an unknown dependency.</li>
<li>Running builds as an <strong>unprivileged user</strong> that does not have any rights to make modifications to the host system -- a build can only write in its designated temp folder or output paths.</li>
<li>Optionally running builds in a <strong>chroot environment</strong>, so that a build cannot possibly find any undeclared host system dependencies through hard-coded absolute paths.</li>
<li><strong>Restricting network access</strong> to prevent a build from obtaining unknown dependencies that may influence the build outcome.</li>
</ul>
<br />
For many build tools, the Nixpkgs repository provides abstraction functions that allow you to easily construct a package from source code (e.g. GNU Make, GNU Autotools, Apache Ant, Perl's MakeMaker, SCons etc.).<br />
<br />
However, certain tools are difficult to use in combination with Nix -- for example, <a href="http://npmjs.com">NPM</a> that is used to deploy <a href="http://nodejs.org">Node.js</a> projects.<br />
<br />
NPM is both a dependency and build manager and the former aspect conflicts with Nix -- builds in Nix are typically prevented from downloading files from remote network locations, with the exception of so-called fixed-output derivations in which the output hash is known in advance.<br />
<br />
If network connections would be allowed in regular builds, then Nix can no longer ensure that a build is reproducible (i.e. that the hash code in the Nix store path reflects the same build output derived from all inputs).<br />
<br />
To cope with the conflicting dependency management feature of NPM, various kinds of integrations have been developed. <a href="https://github.com/NixOS/npm2nix"><i>npm2nix</i></a> was the first, and <a href="https://sandervanderburg.blogspot.com/2014/10/deploying-npm-packages-with-nix-package.html">several years ago I have started <i>node2nix</i></a> to provide a solution that aims for accuracy.<br />
<br />
Basically, the build process of an NPM package in Nix boils down to performing the following steps in a Nix derivation:<br />
<br />
<pre>
# populate the node_modules/ folder
npm install --offline
</pre>
<br />
We must first obtain the required dependencies of a project through the Nix package manager and install them in the correct locations in the <i>node_modules/</i> directory tree.<br />
<br />
Finally, we should run NPM in offline mode forcing it not to re-obtain or re-install any dependencies, but still perform build management tasks, such as running build scripts.<br />
<br />
From a high-level point of view, this principle may look simple, but in practice it is not:<br />
<br />
<ul>
<li>With earlier versions of NPM, we were forced to imitate its dependency resolution algorithm. At first sight, it looked simple, but getting it right (such as coping with circular dependencies and <a href="https://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html">dependency de-duplication</a>) is much more difficult than expected.</li>
<li>NPM 5.x introduced lock files. For NPM development projects, they provide exact version specifiers of all dependencies and transitive dependencies, making it much easier to know which dependencies need to be installed.<br />
<br />
Unfortunately, NPM also introduced an offline cache, that prevents us from simply copying packages into the <i>node_modules/</i> tree. As a result, <a href="https://sandervanderburg.blogspot.com/2017/12/bypassing-npms-content-addressable.html">we need to make additional complex modifications to the <i>package.json</i></a> configuration files of all dependencies.<br />
<br />
Furthermore, end user package installations do not work with lock files, requiring us to still keep our custom implementation of the dependency resolution algorithm.</li>
<li>NPM's behaviour with dependencies on directories on the local file system has changed. In old versions of NPM, such dependencies were copied, but in newer versions, they are symlinked. Furthermore, each directory dependency maintains its own <i>node_modules/</i> directory for transitive dependencies.</li>
</ul>
<br />
Because we need to take many kinds of installation scenarios into account and work around the directory dependency challenges, the implementation of the build environment: <i>node-env.nix</i> in <i>node2nix</i> has become very complicated.<br />
<br />
It has become so complicated that I consider it a major impediment in making any significant changes to the build environment.<br />
<br />
In the last few weeks, I have been working on a companion tool named: <i>placebo-npm</i> that should simplify the installation process. Moreover, it should also fix a number of frequently reported issues.<br />
<br />
In this blog post, I will explain how the tool works.<br />
<br />
<h2>Lock-driven deployments</h2>
<br />
In NPM 5.x, <i>package-lock.json</i> files were introduced. The fact that they capture the exact versions of all dependencies and make all transitive dependencies known, makes certain aspects of an NPM deployment in a Nix build environment easier.<br />
<br />
For lock-driven projects, we no longer have to run our own implementation of the dependency resolution algorithm to figure out what the exact versions of all dependencies and transitive dependencies are.<br />
<br />
For example, a project with the following <i>package.json</i>:<br />
<br />
<pre>
{
"name": "simpleproject",
"version": "0.0.1",
"dependencies": {
"underscore": "*",
"prom2cb": "github:svanderburg/prom2cb",
"async": "https://mylocalserver/async-3.2.1.tgz"
}
}
</pre>
<br />
may have the following <i>package-lock.json</i> file:<br />
<br />
<pre style="overflow: auto;">
{
"name": "simpleproject",
"version": "0.0.1",
"lockfileVersion": 1,
"requires": true,
"dependencies": {
"async": {
"version": "https://mylocalserver/async-3.2.1.tgz",
"integrity": "sha512-XdD5lRO/87udXCMC9meWdYiR+Nq6ZjUfXidViUZGu2F1MO4T3XwZ1et0hb2++BgLfhyJwy44BGB/yx80ABx8hg=="
},
"prom2cb": {
"version": "github:svanderburg/prom2cb#fab277adce1af3bc685f06fa1e43d889362a0e34",
"from": "github:svanderburg/prom2cb"
},
"underscore": {
"version": "1.13.1",
"resolved": "https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz",
"integrity": "sha512-hzSoAVtJF+3ZtiFX0VgfFPHEDRm7Y/QPjGyNo4TVdnDTdft3tr8hEkD25a1jC+TjTuE7tkHGKkhwCgs9dgBB2g=="
}
}
}
</pre>
<br />
As you may notice, the <i>package.json</i> file declares three dependencies:<br />
<br />
<ul>
<li>The first dependency is <i>underscore</i> that refers to the latest version in the NPM registry. In the <i>package-lock.json</i> file, the dependency is frozen to version 1.13.1. The <i>resolved</i> property provides the URL where the tarball should be obtained from. Its <i>integrity</i> can be verified with the given SHA512 hash.</li>
<li>The second dependency: <i>prom2cb</i> refers to the latest revision of the <i>main</i> branch of the <i>prom2cb</i> Git repository on GitHub. In the <i>package-lock.json</i> file, it is pinpointed to the <i>fab277...</i> revision.</li>
<li>The third dependency: <i>async</i> refers to a tarball that is downloaded from an arbitrary HTTP URL. The <i>package-lock.json</i> records its SHA512 integrity hash to make sure that we can only deploy with the version that we have used previously.</li>
</ul>
<br />
As explained earlier, to ensure purity, in a Nix build environment, we cannot allow NPM to obtain the required dependencies of a project. Instead, we must let Nix obtain all the dependencies.<br />
<br />
When all dependencies have been obtained, we should populate the <i>node_modules/</i> folder of the project. In the above example, it is just simply a matter of unpacking the tarballs or copying the Git clones into the <i>node_modules/</i> folder of the project. No transitive dependencies need to be deployed.<br />
<br />
For projects that do not rely on build scripts (that perform tasks, such as linting, compiling code, such as TypeScript etc.) this typically suffices to make a project work.<br />
<br />
However, when we also need build management, we need to run the full installation process:<br />
<br />
<pre style="overflow: auto;">
$ npm install --offline
npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/async/-/async-3.2.1.tgz failed: cache mode is 'only-if-cached' but no cached response available.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/sander/.npm/_logs/2021-08-29T12_56_13_978Z-debug.log
</pre>
<br />
Unfortunately, NPM still tries to obtain the dependencies despite the fact that they have already been copied into the right locations into <i>node_modules</i> folder.<br />
<br />
<h2>Bypassing the offline cache</h2>
<br />
To cope with the problem that manually obtained dependencies cannot be detected, my initial idea was to use the NPM offline cache in a specific way.<br />
<br />
The offline cache claims to be <a href="https://en.wikipedia.org/wiki/Content-addressable_storage">content-addressable</a>, meaning that every item can be looked up by using a hash code that represents its contents, regardless of its origins. Unfortunately, it turns out that this property cannot be fully exploited.<br />
<br />
For example, when we obtain the <i>underscore</i> tarball (with the exact same contents) from a different URL:<br />
<br />
<pre>
$ npm cache add http://mylocalcache/underscore-1.13.1.tgz
</pre>
<br />
and run the installation in offline mode:<br />
<br />
<pre style="overflow: auto;">
$ npm install --offline
npm ERR! code ENOTCACHED
npm ERR! request to https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz failed: cache mode is 'only-if-cached' but no cached response available.
npm ERR! A complete log of this run can be found in:
npm ERR! /home/sander/.npm/_logs/2021-08-26T13_50_15_137Z-debug.log
</pre>
<br />
The installation still fails, despite the fact that we already have a tarball (with the exact same SHA512 hash) in our cache.<br />
<br />
However, downloading <i>underscore</i> from its original location (the NPM registry):<br />
<br />
<pre>
$ npm cache add underscore@1.13.1
</pre>
<br />
makes the installation succeed.<br />
<br />
The reason why downloading the same tarball from an arbitrary HTTP URL does not work is because NPM will only compute a SHA1 hash. Obtaining a tarball from the NPM registry causes NPM to compute a SHA512 hash. Because it was downloaded from a different source, it fails to recognize the SHA512 hash in the <i>package-lock.json</i> file.<br />
<br />
We also run into similar issues when we obtain an old package from the NPM registry that only has an SHA1 hash. Importing the same file from a local file path causes NPM to compute a SHA512 hash. As a result, <i>npm install</i> tries to re-obtain the same tarball from the remote location, because the hash was not recognized.<br />
<br />
To cope with these problems, <i>placebo-npm</i> will completely bypass the cache. After all dependencies have been copied to the <i>node_modules</i> folder, it modifies their <i>package.json</i> configuration files with hidden metadata properties to trick NPM that they came from their original locations.<br />
<br />
For example, to make the <i>underscore</i> dependency work (that is normally obtained from the NPM registry), we must add the following properties to the <i>package.json</i> file:<br />
<br />
<pre style="overflow: auto;">
{
...
_from: "underscore@https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz",
_integrity: "sha512-XdD5lRO/87udXCMC9meWdYiR+Nq6ZjUfXidViUZGu2F1MO4T3XwZ1et0hb2++BgLfhyJwy44BGB/yx80ABx8hg==",
_resolved: "https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz"
}
</pre>
<br />
For <i>prom2cb</i> (that is a Git dependency), we should add:<br />
<br />
<pre style="overflow: auto;">
{
...
_from = "github:svanderburg/prom2cb",
_integrity = "",
_resolved = "github:svanderburg/prom2cb#fab277adce1af3bc685f06fa1e43d889362a0e34"
}
</pre>
<br />
and for HTTP/HTTPS dependencies and local files we should do something similar (adding <i>_from</i> and <i>_integrity</i> fields).<br />
<br />
With these modifications, NPM will no longer attempt to consult the local cache, making the dependency installation step succeed.<br />
<br />
<h2>Handling directory dependencies</h2>
<br />
Another challenge is dependencies on local directories, that are frequently used for local development projects:<br />
<br />
<pre>
{
"name": "simpleproject",
"version": "0.0.1",
"dependencies": {
"underscore": "*",
"prom2cb": "github:svanderburg/prom2cb",
"async": "https://mylocalserver/async-3.2.1.tgz",
"mydep": "../../mydep",
}
}
</pre>
<br />
In the <i>package.json</i> file shown above, a new dependency has been added: <i>mydep</i> that refers to a relative local directory dependency: <i>../../mydep</i>.<br />
<br />
If we run <i>npm install</i>, then NPM creates a symlink to the folder in the project's <i>node_modules/</i> folder and installs the transitive dependencies in the <i>node_modules/</i> folder of the target dependency.<br />
<br />
If we want to deploy the same project to a different machine, then it is required to put <i>mydep</i> in the exact same relative location, or the deployment will fail.<br />
<br />
Deploying such an NPM project with Nix introduces a new problem -- all packages deployed by Nix are stored in the Nix store (typically <i>/nix/store</i>). After deploying the project, the relative path to the project (from the Nix store) will no longer be correct. Moreover, we also want Nix to automatically deploy the directory dependency as part of the deployment of the entire project.<br />
<br />
To cope with these inconveniences, we are required to implement a tricky solution -- we must rewrite directory dependencies in such a way that can refer to a folder that is automatically deployed by Nix. Furthermore, the dependency should still end up being symlink to satisfy NPM -- copying directory dependencies in the <i>node_modules/</i> folder is not accepted by NPM.<br />
<br />
<h2>Usage</h2>
<br />
To conveniently install NPM dependencies from a local source (and satisfying <i>npm</i> in such a way that it believes the dependencies came from their original locations), I have created a tool called: <i>placebo-npm</i>.<br />
<br />
We can, for example, obtain all required dependencies ourselves and put them in a local cache folder:<br />
<br />
<pre>
$ mkdir /home/sander/mycache
$ wget https://mylocalserver/async-3.2.1.tgz
$ wget https://registry.npmjs.org/underscore/-/underscore-1.13.1.tgz
$ git clone https://github.com/svanderburg/prom2cb
</pre>
<br />
The deployment process that <i>placebo-npm</i> executes is driven by a <i>package-placebo.json</i> configuration file that has the following structure:<br />
<br />
<pre style="overflow: auto;">
{
"integrityHashToFile": {
"sha512-hzSoAVtJF+3ZtiFX0VgfFPHEDRm7Y/QPjGyNo4TVdnDTdft3tr8hEkD25a1jC+TjTuE7tkHGKkhwCgs9dgBB2g==": "/home/sander/mycache/underscore-1.13.1.tgz",
"sha512-XdD5lRO/87udXCMC9meWdYiR+Nq6ZjUfXidViUZGu2F1MO4T3XwZ1et0hb2++BgLfhyJwy44BGB/yx80ABx8hg==": "/home/sander/mycache/async-3.2.1.tgz"
},
"versionToFile": {
github:svanderburg/prom2cb#fab277adce1af3bc685f06fa1e43d889362a0e34": "/home/sander/mycache/prom2cb"
},
"versionToDirectoryCopyLink": {
"file:../dep": "/home/sander/alternatedir/dep"
}
}
</pre>
<br />
The placebo config maps dependencies in a <i>package-lock.json</i> file to local file references:<br />
<br />
<ul>
<li><i>integrityHashToFile</i> maps dependencies with an <i>integrity</i> hash to local files, which is useful for HTTP/HTTPS dependencies, registry dependencies, and local file dependencies.</li>
<li><i>versionToFile</i>: maps dependencies with a <i>version</i> property to local directories. This is useful for Git dependencies.</li>
<li><i>versionToDirectoryCopyLink</i>: specifies directories that need to be copied into a shadow directory named: <i>placebo_node_dirs</i> and creates symlinks to the shadow directories in the <i>node_modules/</i> folder. This is useful for installing directory dependencies from arbitrary locations.</li>
</ul>
<br />
With the following command, we can install all required dependencies from the local cache directory and make all necessary modifications to let NPM accept the dependencies:<br />
<br />
<pre>
$ placebo-npm package-placebo.json
</pre>
<br />
Finally, we can run:<br />
<br />
<pre>
$ npm install --offline
</pre>
<br />
The above command does not attempt to re-obtain or re-install the dependencies, but still performs all required build management tasks.<br />
<br />
<h2>Integration with Nix</h2>
<br />
All the functionality that <i>placebo-npm</i> provides has already been implemented in the <i>node-env.nix</i> module, but over the years it has evolved into a very complex beast -- it is implemented as a series of Nix functions that generates shell code.<br />
<br />
As a consequence, it suffers from recursion problems and makes it extremely difficult to tweak/adjust build processes, such as modifying environment variables or injecting arbitrary build steps to work around Nix integration problems.<br />
<br />
With <i>placebo-npm</i> we can reduce the Nix expression that builds projects (<i>buildNPMProject</i>) to an implementation that roughly has the following structure:<br />
<br />
<pre style="overflow: auto;">
{stdenv, placebo-npm}:
{packagePlacebo}:
stdenv.mkDerivation ({
pname = builtins.replaceStrings [ "@" "/" ] [ "_at_" "_slash_" ] pname; # Escape characters that aren't allowed in a store path
placeboJSON = builtins.toJSON packagePlacebo;
passAsFile = [ "placeboJSON" ];
buildInputs = [ nodejs placebo-npm ] ++ buildInputs;
buildPhase = ''
runHook preBuild
true
runHook postBuild
'';
installPhase = ''
runHook preInstall
mkdir -p $out/lib/node_modules/${pname}
mv * $out/lib/node_modules/${pname}
cd $out/lib/node_modules/${pname}
placebo-npm --placebo $placeboJSONPath
npm install --offline
runHook postInstall
'';
} // extraArgs)
</pre>
<br />
As may be observed, the implementation is much more compact and fits easily on one screen. The function accepts a <i>packagePlacebo</i> attribute set as a parameter (that gets translated into a JSON file by the Nix package manager).<br />
<br />
Aside from some simple house keeping work, most of the complex work has been delegated to executing <i>placebo-npm</i> inside the build environment, before we run <i>npm install</i>.<br />
<br />
The function above is also tweakable -- it is possible to inject arbitrary environment variables and adjust the build process through build hooks (e.g. <i>preInstall</i> and <i>postInstall</i>).<br />
<br />
Another bonus feature of delegating all dependency installation functionality to the <i>placebo-npm</i> tool is that we can also use this tool as a build input for other kinds projects -- we can use it the construction process of systems that are built from monolithic repositories, in which NPM is invoked from the build process of the encapsulating project.<br />
<br />
The only requirement is to run <i>placebo-npm</i> before <i>npm install</i> is invoked.<br />
<br />
<h2>Other use cases</h2>
<br />
In addition to using <i>placebo-npm</i> as a companion tool for <i>node2nix</i> and setting up a simple local cache, it can also be useful to facilitate offline installations from external media, such as USB flash drives.<br />
<br />
<h2>Discussion</h2>
<br />
With <i>placebo-npm</i> we can considerably simplify the implementation of <i>node-env.nix</i> (part of <i>node2nix</i>) making it much easier to maintain. I consider the <i>node-env.nix</i> module the second most complicated aspect of <i>node2nix</i>.<br />
<br />
As a side effect, it has also become quite easy to provide tweakable build environments -- this should solve a large number of reported issues. Many reported issues are caused by the fact that it is difficult or sometimes impossible to make changes to a project so that it will cleanly deploy.<br />
<br />
Moreover, <i>placebo-npm</i> can also be used as a build input for projects built from monolithic repositories, in which a sub set needs to be deployed by NPM.<br />
<br />
The integration of the new <i>node-env.nix</i> implementation into <i>node2nix</i> is not completely done yet. I have reworked it, but the part that generates the <i>package-placebo.json</i> file and lets Nix obtain all required dependencies is still a work-in-progress.<br />
<br />
I am experimenting with two implementations: a static approach that generates Nix expressions and dynamic implementation that directly consumes a <i>package-lock.json</i> file in the Nix expression language. Both approaches have pros and cons. As a result, <i>node2nix</i> needs to combine both of them into a hybrid approach.<br />
<br />
In a next blog post, I will explain more about them.<br />
<br />
<h2>Availability</h2>
<br />
The initial version of <i>placebo-npm</i> can be obtained from <a href="https://github.com/svanderburg/placebo-npm">my GitHub page</a>.<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com1tag:blogger.com,1999:blog-1397115249631682228.post-19316116288508085982021-06-01T22:07:00.000+02:002021-06-01T22:07:44.874+02:00An unconventional method for creating backups and exchanging files<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhernn4pnX-77VLIdRZOcJlapEJ_d9LT59SLjdHsyBEDwejWYXPZ12B1xS4ZHiJfc9eOr874pTL9iXDGJsNo3rakPEJlNkHdyrKUqnK4AAUkrKKVNpvFSnr2-Q8uS9fEPwW08braHso4P6H/s600/floppydisk.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="480" data-original-width="600" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhernn4pnX-77VLIdRZOcJlapEJ_d9LT59SLjdHsyBEDwejWYXPZ12B1xS4ZHiJfc9eOr874pTL9iXDGJsNo3rakPEJlNkHdyrKUqnK4AAUkrKKVNpvFSnr2-Q8uS9fEPwW08braHso4P6H/s600/floppydisk.jpg"/></a></div>
<br />
I have written many blog posts about <a href="https://sandervanderburg.blogspot.com/2011/10/software-deployment-complexity.html">software deployment</a> and configuration management. For example, a couple of years ago, I have discussed <a href="https://sandervanderburg.blogspot.com/2015/10/setting-up-basic-software-configuration.html">a very basic configuration management process for small organizations</a>, in which I explained that one of the worst things that could happen is that a machine breaks down and everything that it provides gets lost.<br />
<br />
Fortunately, good configuration management practices and deployment tools (such as <a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix</a>) can help you to restore a machine's configuration with relative ease.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2012/03/deployment-of-mutable-components.html">Another problem is managing a machine's <strong>data</strong></a>, which in many ways is even more important and complicated -- software packages can be typically obtained from a variety of sources, but data is typically <strong>unique</strong> (and therefore more valuable).<br />
<br />
Even if a machine stays operational, the data that it stores can still be at risk -- it may get deleted by accident, or corrupted (for example, by the user, or a hardware problem).<br />
<br />
It also does not matter whether a machine is used for business (for example, storing data for information systems) or personal use (for example, documents, pictures, and audio files). In both cases, data is valuable, and as a result, needs to be protected from loss and corruption.<br />
<br />
In addition to recovery, the <strong>availability</strong> of data is often also very important -- many users (including me) typically own multiple devices (e.g. a desktop PC, laptop and phone) and typically want access to the same data from multiple places.<br />
<br />
Because of the importance of data, I sometimes get questions from non-technical users that want to know how I manage my personal data (such as documents, images and audio files) and what tools I would recommend.<br />
<br />
Similar to most computer users, I too have faced my own share of reliability problems -- of all the desktop computers I owned, I ended up with a completely broken hard drive three times, and a completely broken laptop once. Furthermore, I have also worked with all kinds of external media (e.g. floppy disks, CD-ROMs etc.) each having their own share of reliability problems.<br />
<br />
To cope with data availability and loss, I came up with a custom script that I have been conveniently using to create backups and synchronize my data between the machines that I use.<br />
<br />
In this blog post, I will explain how this script works.<br />
<br />
<h2>About storage media</h2>
<br />
To cope with the potential loss of data, I have always made it a habit to transfer data to external media. I have worked with a variety of them, each having their advantages and disadvantages:<br />
<br />
<ul>
<li>In the old days, I used <strong>floppy disks</strong>. Most people who are (at the time reading this blog post) in their early twenties or younger, may probably have no clue what I am talking about (for those people perhaps the 'Save icon' used in many desktop applications looks familiar).<br />
<br />
Roughly 25 years ago, floppy disks were a common means to exchange data between computers.<br />
<br />
Although they were common, they had many drawbacks. Probably the biggest drawback was their limited storage capacity -- I used to own 5.25 inch disks that (on PCs) were capable of storing ~360 KiB (if both sides are used), and the more sturdy 3.5 inch disks providing double density (720 KiB) and high density capacity (1.44 MiB).<br />
<br />
Furthermore, floppy disks were also quite slow and could be easily damaged, for example, by toughing the magnetic surface.</li>
<li>When I switched from the <a href="https://sandervanderburg.blogspot.com/2011/07/second-computer.html">Commodore Amiga</a> to the PC, I also used <strong>tapes</strong> for a while in addition to floppy disks. They provided a substantial amount of storage capacity (~500 MiB in 1996). As of 2019 (and this probably still applies to today), <a href="https://searchdatabackup.techtarget.com/news/252459656/How-is-tape-backup-doing-in-2019">tapes are still considered very cheap and reliable media for archival of data</a>.<br />
<br />
What I found impractical about tapes is that they are difficult to use as random access memory -- data on a tape is stored sequentially. As a consequence, it is typically very slow to find files or to "update" existing files. Typically, a backup tool needs to scan the tape from the beginning to the end or maintain a database with known storage locations.<br />
<br />
Many of my personal files (such as documents) are regularly updated and older versions do not have to be retained. Instead, they should be removed to clear up storage space. With tapes this is very difficult to do.</li>
<li>
When <strong>writable CD/DVDs</strong> became affordable, I used them as a backup media for a while. Similar to tapes, they also have substantial storage capacity. Furthermore, they are very fast and convenient to read.<br />
<br />
A similar disadvantage is that they are not a very convenient medium for updating files. Although it is possible to write multi-sessions discs, in which files can be added, overwritten, or made invisible (essentially a "soft delete"), it remained inconvenient because you can not clear up the storage space that a deleted file used to occupy.<br />
<br />
I also learned the hard way that writable discs (and in particular rewritable discs) are not very reliable for long term storage -- I have discarded many old writable discs (10 years or older) that can no longer be read.</li>
</ul>
<br />
Nowadays, I use a variety of USB storage devices (such as memory sticks, hard drives) as backup media. They are relatively cheap, fast, have more than enough storage capacity, and I can use them as random access memory -- it is no problem at all to update and delete data existing data.<br />
<br />
To cope with the potential breakage of USB storage media, I always make sure that I have at least two copies of my important data.<br />
<br />
<h2>About data availability</h2>
<br />
As already explained in the introduction, I have multiple devices for which I want the same data to be available. For example, on both my desktop PC and company laptop, I want to have access to my music and research papers collection.<br />
<br />
A possible solution is to use a <strong>shared storage</strong> medium, such as a network drive. The advantage of this approach is that there is a single source of truth and I only need to maintain a single data collection -- when I add a new document it will immediately be available to both devices.<br />
<br />
Although a network drive may be a possible solution, it is not a good fit for my use cases -- I typically use laptops for traveling. When I am not at home, I can no longer access my data stored on the network drive.<br />
<br />
Another solution is to <strong>transfer</strong> all required files to the hard drive on my laptop. Doing a bulk transfer for the first time is typically not a big problem (in particular, if you use <a href="https://sandervanderburg.blogspot.com/2018/01/syntax-highlighting-nix-expressions-in.html">orthodox file managers</a>), but keeping collections of files up-to-date between machines is in my experience quite tedious to do by hand.<br />
<br />
<h2>Automating data synchronization</h2>
<br />
For both backing up and synchronizing files to other machines I need to regularly compare and update files in directories. In the former case, I need to sync data between local directories, and for the latter I need to sync data between directories on remote machines.<br />
<br />
Each time I want make updates to my files, I want to inspect what has changed, and see which files require updating before actually doing it, so that I do not end up wasting time or risk modifying the wrong files.<br />
<br />
Initially, I started to investigate how to implement a synchronization tool myself, but quite quickly I realized that there is already a tool available that is quite suitable for the job: <a href="https://rsync.samba.org/"><i>rsync</i></a>.<br />
<br />
rsync is designed to efficiently transfer and synchronize files between drivers and machines across networks by comparing the modification times and sizes of files.<br />
<br />
The only thing that I consider a drawback is that it is not fully optimized to conveniently automate my personal workflow -- to accomplish what I want, I need to memorize all the relevant <i>rsync</i> command-line options and run multiple command-line instructions.<br />
<br />
To alleviate this problem, I have created a custom script, that evolved into a tool that I have named: <i>gitlike-rsync</i>.<br />
<br />
<h2>Usage</h2>
<br />
<i>gitlike-rsync</i> is a tool that facilitates synchronisation of file collections between directories on local or remote machines using <i>rsync</i> and a workflow that is similar to managing <a href="https://git-scm.com/">Git</a> projects.<br />
<br />
<h3>Making backups</h3>
<br />
For example, if we have a data directory that we want to back up to another partition (for example, that refers to an external USB drive), we can open the directory:<br />
<br />
<pre>
$ cd /home/sander/Documents
</pre>
<br />
and configure a destination directory, such as a directory on a backup drive (<i>/media/MyBackupDrive/Documents</i>):<br />
<br />
<pre>
$ gitlike-rsync destination-add /media/MyBackupDrive/Documents
</pre>
<br />
By running the following command-line instruction, we can create a backup of the <i>Documents</i> folder:<br />
<br />
<pre>
$ gitlike-rsync push
sending incremental file list
.d..tp..... ./
>f+++++++++ bye.txt
>f+++++++++ hello.txt
sent 112 bytes received 25 bytes 274.00 bytes/sec
total size is 10 speedup is 0.07 (DRY RUN)
Do you want to proceed (y/N)? y
sending incremental file list
.d..tp..... ./
>f+++++++++ bye.txt
4 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/3)
>f+++++++++ hello.txt
6 100% 5.86kB/s 0:00:00 (xfr#2, to-chk=0/3)
sent 202 bytes received 57 bytes 518.00 bytes/sec
total size is 10 speedup is 0.04
</pre>
<br />
The output above shows me the following:<br />
<br />
<ul>
<li>When no additional command-line parameters have been provided, the script will first do a dry run and show the user what it intends to do. In the above example, it shows me that it wants to transfer the contents of the <i>Documents</i> folder that consists of only two files: <i>hello.txt</i> and <i>bye.txt</i>.</li>
<li>After providing my confirmation, the files in the destination directory will be updated -- the backup drive that is mounted on <i>/media/MyBackupDrive</i>.</li>
</ul>
<br />
I can conveniently make updates in my documents folder and update my backups.<br />
<br />
For example, I can add a new file to the <i>Documents</i> folder named: <i>greeting.txt</i>, and run the <i>push</i> command again:<br />
<br />
<pre>
$ gitlike-rsync push
sending incremental file list
.d..t...... ./
>f+++++++++ greeting.txt
sent 129 bytes received 22 bytes 302.00 bytes/sec
total size is 19 speedup is 0.13 (DRY RUN)
Do you want to proceed (y/N)? y
sending incremental file list
.d..t...... ./
>f+++++++++ greeting.txt
9 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=1/4)
sent 182 bytes received 38 bytes 440.00 bytes/sec
total size is 19 speedup is 0.09
</pre>
<br />
In the above output, only the <i>greeting.txt</i> file is transferred to backup partition, leaving the other files untouched, because they have not changed.<br />
<br />
<h3>Restoring files from a backup</h3>
<br />
In addition to the <i>push</i> command, <i>gitlike-rsync</i> also supports <i>pull</i> that can be used to sync data from the configured destination folders. The <i>pull</i> command can be used as a means to restore data from a backup partition.<br />
<br />
For example, if I accidentally delete a file from the <i>Documents</i> folder:<br />
<br />
<pre>
$ rm hello.txt
</pre>
<br />
and run the <i>pull</i> command:<br />
<br />
<pre>
$ gitlike-rsync pull
sending incremental file list
.d..t...... ./
>f+++++++++ hello.txt
sent 137 bytes received 22 bytes 318.00 bytes/sec
total size is 19 speedup is 0.12 (DRY RUN)
Do you want to proceed (y/N)? y
sending incremental file list
.d..t...... ./
>f+++++++++ hello.txt
6 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=0/4)
sent 183 bytes received 38 bytes 442.00 bytes/sec
total size is 19 speedup is 0.09
</pre>
<br />
the script is able to detect that <i>hello.txt</i> was removed and restore it from the backup partition.<br />
<br />
<h2>Synchronizing files between machines in a network</h2>
<br />
In addition to local directories, that are useful for back ups, the <i>gitlike-rsync</i> script can also be used in a similar way to exchange files between machines, such as my desktop PC and office laptop.<br />
<br />
With the following command-line instruction, I can automatically clone the <i>Documents</i> folder from my desktop PC to the <i>Documents</i> folder on my office laptop:<br />
<br />
<pre>
$ gitlike-rsync clone sander@desktop-pc:/home/sander/Documents
</pre>
<br />
The above command connects to my desktop PC over SSH and retrieves the content of the <i>Documents/</i> folder. It will also automatically configure the destination directory to synchronize with the <i>Documents</i> folder on the desktop PC.<br />
<br />
When new documents have been added on the desktop PC, I just have to run the following command on my office laptop to update it:<br />
<br />
<pre>
$ gitlike-rsync pull
</pre>
<br />
I can also modify the contents of the <i>Documents</i> folder on my office laptop and synchronize the changed files to my desktop PC with a <i>push</i>:<br />
<br />
<pre>
$ gitlike-rsync push
</pre>
<br />
<h2>About versioning</h2>
<br />
As explained in the beginning of this blog post, in addition to the recovery of failing machines and equipment, another important reason to create backups is to protect yourself against accidental modifications.<br />
<br />
Although <i>gitlike-rsync</i> can detect and display file changes, it does <strong>not</strong> do any <strong>versioning</strong> of any kind. This feature is deliberately left unimplemented, for very good reasons.<br />
<br />
For most of my personal files (e.g. images, audio, video) I do not need any versioning. As soon as they are organized, they are not supposed to be changed.<br />
<br />
However, for certain kinds of files I do need versioning, such as software development projects. Whenever I need versioning, my answer is very simple: I use the "ordinary" Git, even for projects that are private and not supposed to be shared on a public hosting service, such as <a href="https://github.com">GitHub</a>.<br />
<br />
As seasoned Git users may probably already know, you can turn any local directory into a Git repository, by running:<br />
<br />
<pre>
$ git init
</pre>
<br />
The above command creates a local <i>.git</i> folder that tracks and stores changes locally.<br />
<br />
When using a public hosting service, such as GitHub, and cloning a repository from GitHub, a remote: <i>origin</i> has been automatically configured to automatically push and pull changes to and from GitHub.<br />
<br />
It is also possible to synchronize Git changes between arbitrary computers using a private SSH connection. I can, for example, configure a remote for a private repository, as follows:<br />
<br />
<pre style="font-size: 90%; overflow: auto;">
$ git remote add origin sander@desktop-pc:/home/sander/Development/private-project
</pre>
<br />
the above command configures the Git project that is stored in the <i>/home/sander/Development/private-project</i> directory on my desktop PC as a remote.<br />
<br />
I can pull changes from the remote repository, by running:<br />
<br />
<pre>
$ git pull origin
</pre>
<br />
and push locally stored changes, by running:<br />
<br />
<pre>
$ git push origin
</pre>
<br />
As you may probably have already noticed, the above workflow is very similar to exchanging documents, shown earlier in this blog post.<br />
<br />
What about backing up private Git repositories? To do this, I typically create tarballs of the Git project directories and sync them to my backup media with <i>gitlike-rsync</i>. <a href="https://superuser.com/questions/219668/how-do-i-backup-a-git-repo">The presence of the <i>.git</i> folder suffices to retain a project's history</a>.<br />
<br />
<h2>Conclusion</h2>
<br />
In this blog post, I have described <i>gitlike-rsync</i>, a simple opinionated wrapper script for exchanging files between local directories (for backups) and remote directories (for data exchange between machines).<br />
<br />
As its name implies, it heavily builds on top of <i>rsync</i> for efficient data exchange, and the concepts of <i>git</i> as an inspiration for the workflow.<br />
<br />
I have been conveniently using this script for over ten years, and it works extremely well for my own use cases and a variety of operating systems (Linux, Windows, macOS and FreeBSD).<br />
<br />
My solution is obviously not rocket science -- my contribution is only the workflow automation. The "true credits" should go the developers of rsync and Git.<br />
<br />
I also have to thank the COVID-19 crisis that allowed me to finally find the time to polish the script, document it and give it a name. In the Netherlands, as of today, there are still many restrictions, but the situation is slowly getting better.<br />
<br />
<h2>Availability</h2>
<br />
I have added the <i>gitlike-rsync</i> script described in this blog post to my <a href="https://github.com/svanderburg/custom-scripts">custom-scripts repository</a> that can be obtained from my GitHub page.<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-27355059361632784182021-04-26T21:32:00.000+02:002021-04-26T21:32:43.300+02:00A test framework for the Nix process management frameworkAs already explained in many previous blog posts, the <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html">Nix process management framework</a> adds new ideas to earlier service management concepts explored in Nixpkgs and <a href="https://sandervanderburg.blogspot.com/2011/01/nixos-purely-functional-linux.html">NixOS</a>:<br />
<br />
<ul>
<li>It makes it possible to deploy services on <strong>any operating system</strong> that can work with the Nix package manager, including conventional Linux distributions, macOS and FreeBSD. It also works on NixOS, but NixOS is not a requirement.</li>
<li>It allows you to construct <strong>multiple instances</strong> of the same service, by using constructor functions that identify conflicting configuration parameters. These constructor functions can be invoked in such a way that these configuration properties no longer conflict.</li>
<li>We can target <strong>multiple process managers</strong> from the same high-level deployment specifications. These high-level specifications are automatically translated to parameters for a target-specific configuration function for a specific process manager.<br />
<br />
It is also possible to override or augment the generated parameters, to work with configuration properties that are not universally supported.</li>
<li>There is a configuration option that conveniently allows you to disable user changes making it possible to deploy services as an <strong>unprivileged user</strong>.</li>
</ul>
<br />
Although the above features are interesting, one particular challenge is that the framework <strong>cannot guarantee</strong> that all possible variations will work after writing a high-level process configuration. The framework facilitates code reuse, but it is not a write once, run anywhere approach.<br />
<br />
To make it possible to validate multiple service variants, I have developed a test framework that is built on top of the <a href="https://sandervanderburg.blogspot.com/2011/02/using-nixos-for-declarative-deployment.html">NixOS test driver</a> that makes it possible to deploy and test a network of NixOS QEMU virtual machines with very minimal storage and RAM overhead.<br />
<br />
In this blog post, I will describe how the test framework can be used.<br />
<br />
<h2>Automating tests</h2>
<br />
Before developing the test framework, I was mostly testing all my packaged services manually. Because a manual test process is tedious and time consuming, I did not have any test coverage for anything but the most trivial example services. As a result, I frequently ran into many configuration breakages.<br />
<br />
Typically, when I want to test a process instance, or a system that is composed of multiple collaborative processes, I perform the following steps:<br />
<br />
<ul>
<li>First, I need to <strong>deploy</strong> the system for a specific process manager and configuration profile, e.g. for a privileged or unprivileged user, in an isolated environment, such as a virtual machine or container.</li>
<li>Then I need to <strong>wait</strong> for all process instances to become <strong>available</strong>. Readiness checks are critical and typically more complicated than expected -- for most services, there is a time window between a successful invocation of a process and its availability to carry out its primary task, such as accepting network connections. Executing tests before a service is ready, typically results in errors.<br />
<br />
Although there are process managers that can generally deal with this problem (e.g. <a href="https://www.freedesktop.org/wiki/Software/systemd">systemd</a> has the <a href="https://www.freedesktop.org/software/systemd/man/sd_notify.html"><i>sd_notify</i></a> protocol and <a href="https://skarnet.org/software/s6">s6</a> its <a href="https://skarnet.org/software/s6/notifywhenup.html">own protocol</a> and a <a href="https://skarnet.org/software/misc/sdnotify-wrapper.c"><i>sd_notify</i> wrapper</a>), the lack of a standardized protocol and its adoption still requires me to manually implement readiness checks.<br />
<br />
(As a sidenote: the only readiness check protocol that is standardized is for <a href="https://www.freedesktop.org/software/systemd/man/daemon.html">traditional System V services that daemonize on their own</a>. The calling parent process should almost terminate immediately, but still wait until the spawned daemon child process notifies it to be ready.<br />
<br />
As described in <a href="https://sandervanderburg.blogspot.com/2020/01/writing-well-behaving-daemon-in-c.html">an earlier blog post</a>, this notification aspect is more complicated to implement than I thought. Moreover, not all traditional System V daemons follow this protocol.)</li>
<li>When all process instances are ready, I can <strong>check</strong> whether they properly carry out their tasks, and whether the integration of these processes work as expected.</li>
</ul>
<br />
<h3>An example</h3>
<br />
I have developed a Nix function: <i>testService</i> that automates the above process using the NixOS test driver -- I can use this function to create a test suite for systems that are made out of running processes, such as the webapps example described in my previous blog posts about the Nix process management framework.<br />
<br />
The example system consists of a number of <i>webapp</i> processes with an embedded HTTP server returning HTML pages displaying their identities. Nginx reverse proxies forward incoming connections to the appropriate <i>webapp</i> processes by using their corresponding virtual host header values:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, libDir ? "${stateDir}/lib"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
sharedConstructors = import ../../../examples/services-agnostic/constructors/constructors.nix {
inherit pkgs stateDir runtimeDir logDir cacheDir libDir tmpDir forceDisableUserChange processManager;
};
constructors = import ../../../examples/webapps-agnostic/constructors/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager;
webappMode = null;
};
in
rec {
webapp1 = rec {
port = 5000;
dnsName = "webapp1.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "1";
};
};
webapp2 = rec {
port = 5001;
dnsName = "webapp2.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "2";
};
};
webapp3 = rec {
port = 5002;
dnsName = "webapp3.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "3";
};
};
webapp4 = rec {
port = 5003;
dnsName = "webapp4.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "4";
};
};
nginx = rec {
port = if forceDisableUserChange then 8080 else 80;
webapps = [ webapp1 webapp2 webapp3 webapp4 ];
pkg = sharedConstructors.nginxReverseProxyHostBased {
inherit port webapps;
} {};
};
webapp5 = rec {
port = 5004;
dnsName = "webapp5.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "5";
};
};
webapp6 = rec {
port = 5005;
dnsName = "webapp6.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "6";
};
};
nginx2 = rec {
port = if forceDisableUserChange then 8081 else 81;
webapps = [ webapp5 webapp6 ];
pkg = sharedConstructors.nginxReverseProxyHostBased {
inherit port webapps;
instanceSuffix = "2";
} {};
};
}
</pre>
<br />
The processes model shown above (<i>processes-advanced.nix</i>) defines the following process instances:<br />
<br />
<ul>
<li>There are six <i>webapp</i> process instances, each running an embedded HTTP service, returning HTML pages with their identities. The <i>dnsName</i> property specifies the DNS domain name value that should be used as a virtual host header to make the forwarding from the reverse proxies work.</li>
<li>There are two <i>nginx</i> reverse proxy instances. The former: <i>nginx</i> forwards incoming connections to the first four <i>webapp</i> instances. The latter: <i>nginx2</i> forwards incoming connections to <i>webapp5</i> and <i>webapp6</i>.</li>
</ul>
<br />
With the following command, I can connect to <i>webapp2</i> through the first <i>nginx</i> reverse proxy:<br />
<br />
<pre>
$ curl -H 'Host: webapp2.local' http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5001
</body>
</html>
</pre>
<br />
<h3>Creating a test suite</h3>
<br />
I can create a test suite for the web application system as follows:<br />
<br />
<pre style="overflow: auto;">
{ pkgs, testService, processManagers, profiles }:
testService {
exprFile = ./processes.nix;
readiness = {instanceName, instance, ...}:
''
machine.wait_for_open_port(${toString instance.port})
'';
tests = {instanceName, instance, ...}:
pkgs.lib.optionalString (instanceName == "nginx" || instanceName == "nginx2")
(pkgs.lib.concatMapStrings (webapp: ''
machine.succeed(
"curl --fail -H 'Host: ${webapp.dnsName}' http://localhost:${toString instance.port} | grep ': ${toString webapp.port}'"
)
'') instance.webapps);
inherit processManagers profiles;
}
</pre>
<br />
The Nix expression above invokes <i>testService</i> with the following parameters:<br />
<br />
<ul>
<li><i>processManagers</i> refers to a list of names of all the process managers that should be tested.</li>
<li><i>profiles</i> refers to a list of configuration profiles that should be tested. Currently, it supports <i>privileged</i> for privileged deployments, and <i>unprivileged</i> for unprivileged deployments in an unprivileged user's home directory, without changing user permissions.</li>
<li>The <i>exprFile</i> parameter refers to the processes model of the system: <i>processes-advanced.nix</i> shown earlier.</li>
<li>The <i>readiness</i> parameter refers to a function that does a readiness check for each process instance. In the above example, it checks whether each service is actually listening on the required TCP port.</li>
<li>The <i>tests</i> parameter refers to a function that executes tests for each process instance. In the above example, it ignores all but the <i>nginx</i> instances, because explicitly testing a <i>webapp</i> instance is a redundant operation.<br />
<br />
For each <i>nginx</i> instance, it checks whether all <i>webapp</i> instances can be reached from it, by running the <i>curl</i> command.</li>
</ul>
<br />
The <i>readiness</i> and <i>tests</i> functions take the following parameters: <i>instanceName</i> identifies the process instance in the processes model, and <i>instance</i> refers to the attribute set containing its configuration.<br />
<br />
Furthermore, they can refer to global process model configuration parameters:<br />
<br />
<ul>
<li><i>stateDir</i>: The directory in which state files are stored (typically <i>/var</i> for privileged deployments)</li>
<li><i>runtimeDir</i>: The directory in which runtime files are stored (typically <i>/var/run</i> for privileged deployments).</li>
<li><i>forceDisableUserChange</i>: Indicates whether to disable user changes (for unprivileged deployments) or not.</li>
</ul>
<br />
In addition to writing tests that work on instance level, it is also possible to write tests on system level, with the following parameters (not shown in the example):<br />
<br />
<ul>
<li><i>initialTests</i>: instructions that run right after deploying the system, but before the readiness checks, and instance-level tests.</li>
<li><i>postTests</i>: instructions that run after the instance-level tests.</li>
</ul>
<br />
The above functions also accept the same global configuration parameters, and <i>processes</i> that refers to the entire processes model.<br />
<br />
We can also configure other properties useful for testing:<br />
<br />
<ul>
<li><i>systemPackages</i>: installs additional packages into the system profile of the test virtual machine.</li>
<li><i>nixosConfig</i> defines a NixOS module with configuration properties that will be added to the NixOS configuration of the test machine.</li>
<li><i>extraParams</i> propagates additional parameters to the processes model.</li>
</ul>
<br />
<h3>Composing test functions</h3>
<br />
The Nix expression above is not self-contained. It is a function definition that needs to be invoked with all required parameters including all the process managers and profiles that we want to test for.<br />
<br />
We can compose tests in the following Nix expression:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, processManagers ? [ "supervisord" "sysvinit" "systemd" "disnix" "s6-rc" ]
, profiles ? [ "privileged" "unprivileged" ]
}:
let
testService = import ../../nixproc/test-driver/universal.nix {
inherit system;
};
in
{
nginx-reverse-proxy-hostbased = import ./nginx-reverse-proxy-hostbased {
inherit pkgs processManagers profiles testService;
};
docker = import ./docker {
inherit pkgs processManagers profiles testService;
};
...
}
</pre>
<br />
The above partial Nix expression (<i>default.nix</i>) invokes the function defined in the previous Nix expression that resides in the <i>nginx-reverse-proxy-hostbased</i> directory and propagates all required parameters. It also composes other test cases, such as <i>docker</i>.<br />
<br />
The parameters of the composition expression allow you to globally configure all the desired service variants:<br />
<br />
<ul>
<li><i>processManagers</i> allows you to select the process managers you want to test for.</li>
<li><i>profiles</i> allows you to select the configuration profiles.</li>
</ul>
<br />
With the following command, we can test our system as a privileged user, using <i>systemd</i> as a process manager:<br />
<br />
<pre>
$ nix-build -A nginx-reverse-proxy-hostbased.privileged.systemd
</pre>
<br />
we can also run the same test, but then as an unprivileged user:<br />
<br />
<pre>
$ nix-build -A nginx-reverse-proxy-hostbased.unprivileged.systemd
</pre>
<br />
In addition to <i>systemd</i>, any configured process manager can be used that works in NixOS. The following command runs a privileged test of the same service for <i>sysvinit</i>:<br />
<br />
<pre>
$ nix-build -A nginx-reverse-proxy-hostbased.privileged.sysvinit
</pre>
<br />
<h2>Results</h2>
<br />
With the test driver in place, I have managed to expand my repository of example services, provided test coverage for them and fixed quite a few bugs in the framework caused by regressions.<br />
<br />
Below is a screenshot of <a href="https://sandervanderburg.blogspot.com/2013/04/setting-up-hydra-build-cluster-for.html">Hydra</a>: the Nix-based continuous integration service showing an overview of test results for all kinds of variants of a service:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWq12hv377-anK1ZrM9qK31ZKqVpIhkEZ3XQP59gixkd31DTTky7rqXiG5rtlDrxKoJusWY27lthfMhrxo2q4AsvuOfmrQhV0_5Pw6eGaftwpsUUgtPr1Z-jFAH53Zm4DEXTrUeuiJHVlG/s1444/variantsonhydra.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" height="600" data-original-height="1444" data-original-width="1251" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWq12hv377-anK1ZrM9qK31ZKqVpIhkEZ3XQP59gixkd31DTTky7rqXiG5rtlDrxKoJusWY27lthfMhrxo2q4AsvuOfmrQhV0_5Pw6eGaftwpsUUgtPr1Z-jFAH53Zm4DEXTrUeuiJHVlG/s600/variantsonhydra.png"/></a></div>
<br />
So far, the following services work multi-instance, with multiple process managers, and (optionally) as an unprivileged user:<br />
<br />
<ul>
<li><strong>Apache HTTP server</strong>. In the services repository, there are multiple constructors for deploying an Apache HTTP server: to deploy static web applications or dynamic web applications with PHP, and to use it as a reverse proxy (via HTTP and AJP) with HTTP basic authentication optionally enabled.</li>
<li><strong>Apache Tomcat</strong>.</li>
<li><strong>Nginx</strong>. For Nginx we also have multiple constructors. One to deploy a configuration for serving static web apps, and two for setting up reverse proxies using paths or virtual hosts to forward incoming requests to the appropriate services.<br />
<br />
The reverse proxy constructors can also generate configurations that will cache the responses of incoming requests.</li>
<li><strong>MySQL/MariaDB</strong>.</li>
<li><strong>PostgreSQL</strong>.</li>
<li><strong>InfluxDB</strong>.</li>
<li><strong>MongoDB</strong>.</li>
<li><strong>OpenSSH</strong>.</li>
<li><strong>svnserve</strong>.</li>
<li><strong>xinetd</strong>.</li>
<li><strong>fcron</strong>. By default, the <i>fcron</i> user and group are hardwired into the executable. To facilitate unprivileged user deployments, we automatically create a package build override to propagate the <i>--with-run-non-privileged</i> configuration flag so that it can run as unprivileged user. Similarly, for multiple instances we create an override to use a different user and group that does not conflict with the primary instance.</li>
<li><strong>supervisord</strong></li>
<li><strong>s6-svscan</strong></li>
</ul>
<br />
The following service also works with multiple instances and multiple process managers, but not as an unprivileged user:<br />
<br />
<ul>
<li><strong>Docker</strong>. In theory, <a href="https://docs.docker.com/engine/security/rootless">Docker supports rootless deployments</a>, but it is still very highly experimental and I find it very cumbersome to set up.</li>
</ul>
<br />
The following services work with multiple process managers, but not multi-instance or as an unprivileged user:<br />
<br />
<ul>
<li><strong>D-Bus</strong></li>
<li><strong>Disnix</strong></li>
<li><strong>nix-daemon</strong></li>
<li><strong>Hydra</strong></li>
</ul>
<br />
In theory, the above services could be adjusted to work as an unprivileged user, but doing so is not very useful -- for example, the <i>nix-daemon</i>'s purpose is to facilitate multi-user package deployments. As an unprivileged user, you only want to facilitate package deployments for yourself.<br />
<br />
Moreover, the multi-instance aspect is IMO also not very useful to explore for these services. For example, I can not think of a useful scenario to have two Hydra instances running next to each other.<br />
<br />
<h2>Discussion</h2>
<br />
The test framework described in this blog post is an important feature addition to the Nix process management framework -- it allowed me to package more services and fix quite a few bugs caused by regressions.<br />
<br />
I can now finally show that it is doable to package services and make them work under nearly all possible conditions that the framework supports (e.g. multiple instances, multiple process managers, and unprivileged user installations).<br />
<br />
The only limitation of the test framework is that it is <strong>not operating system agnostic</strong> -- the NixOS test driver (that serves as its foundation), only works (as its name implies) with NixOS, which itself is a Linux distribution. As a result, we can not automatically test <i>bsdrc</i> scripts, <i>launchd</i> daemons, and <i>cygrunsrv</i> services.<br />
<br />
In theory, it is also possible to make a more generalized test driver that works with multiple operating systems. The NixOS test driver is a combination of ideas (e.g. a shared Nix store between the host and guest system, an API to control QEMU, and an API to manage services). We could also dissect these ideas and run them on conventional QEMU VMs running different operating systems (with the Nix package manager).<br />
<br />
Although making a more generalized test driver is interesting, it is beyond the scope of the Nix process management framework (which is about managing process instances, not entire systems).<br />
<br />
Another drawback is that while it is possible to test all possible service variants on Linux, it may be very expensive to do so.<br />
<br />
However, full process manager coverage is often not required to get a reasonable level of confidence. For many services, it typically suffices to implement the following strategy:<br />
<br />
<ul>
<li>Pick <strong>two process managers</strong>: one that prefers foreground processes (e.g. <i>supervisord</i>) and one that prefers daemons (e.g. <i>sysvinit</i>). This is the most significant difference (from a configuration perspective) between all these different process managers.</li>
<li>If a service supports multiple configuration variants, and multiple instances, then create a processes model that <strong>concurrently deploys all these variants</strong>.</li>
</ul>
<br />
Implementing the above strategy only requires you to test four variants, providing a high degree of certainty that it will work with all other process managers as well.<br />
<br />
<h2>Future work</h2>
<br />
Most of the interesting functionality required to work with the Nix process management framework is now implemented. I still need to implement more changes to make it more robust and "dog food" more of my own problems as much as possible.<br />
<br />
Moreover, the <i>docker</i> backend still requires a bit more work to make it more usable.<br />
<br />
Eventually, I will be thinking of an RFC that will upstream the interesting bits of the framework into Nixpkgs.<br />
<br />
<h2>Availability</h2>
<br />
The <a href="https://github.com/svanderburg/nix-processmgmt">Nix process management framework</a> repository as well as the <a href="https://github.com/svanderburg/nix-processmgmt-services">example services repository</a> can be obtained from my GitHub page.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-14766929827603467432021-03-12T23:28:00.000+01:002021-03-12T23:28:18.884+01:00Using the Nix process management framework as an infrastructure deployment solution for DisnixAs explained in many previous blog posts, I have developed <a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a> as a solution for automating the deployment of service-oriented systems -- it deploys <strong>heterogeneous systems</strong>, that consist of many different kinds of components (such as web applications, web services, databases and processes) to <strong>networks</strong> of machines.<br />
<br />
The deployment models for Disnix are typically not fully self-contained. Foremost, a precondition that must be met before a service-oriented system can be deployed, is that all target machines in the network require the presence of <a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix package manager</a>, Disnix, and a remote connectivity service (e.g. SSH).<br />
<br />
For multi-user Disnix installations, in which the user does not have super-user privileges, the Disnix service is required to carry out deployment operations on behalf of a user.<br />
<br />
Moreover, the services in the services model typically need to be managed by other services, called <strong>containers</strong> in Disnix terminology (not to be confused with <a href="https://en.wikipedia.org/wiki/List_of_Linux_containers">Linux containers</a>).<br />
<br />
Examples of container services are:<br />
<br />
<ul>
<li>The MySQL DBMS container can manage multiple databases deployed by Disnix.</li>
<li>The Apache Tomcat servlet container can manage multiple Java web applications deployed by Disnix.</li>
<li>systemd can act as a container that manages multiple systemd units deployed by Disnix.</li>
</ul>
<br />
Managing the life-cycles of services in containers (such as activating or deactivating them) is done by a companion tool called <a href="https://sandervanderburg.blogspot.com/2015/07/deploying-state-with-disnix.html">Dysnomia</a>.<br />
<br />
In addition to Disnix, these container services also typically need to be deployed in advance to the target machines in the network.<br />
<br />
The problem domain that Disnix works in is called <strong>service deployment</strong>, whereas the deployment of machines (bare metal or virtual machines) and the container services is called <strong>infrastructure deployment</strong>.<br />
<br />
Disnix can be complemented with a variety of infrastructure deployment solutions:<br />
<br />
<ul>
<li><a href="https://sandervanderburg.blogspot.com/2015/03/on-nixops-disnix-service-deployment-and.html">NixOps</a> can deploy networks of <a href="https://sandervanderburg.blogspot.com/2011/01/nixos-purely-functional-linux.html">NixOS</a> machines, both physical and virtual machines (in the cloud), such as <a href="http://aws.amazon.com/ec2">Amazon EC2</a>.<br />
<br />
As part of a NixOS configuration, the Disnix service can be deployed that facilitates multi-user installations. The Dysnomia NixOS module can expose all relevant container services installed by NixOS as container deployment targets.</li>
<li><i>disnixos-deploy-network</i> is a tool that is included with the DisnixOS extension toolset. Since services in Disnix can be any kind of deployment unit, it is also possible to deploy an entire NixOS configuration as a service. This tool is mostly developed for demonstration purposes.<br />
<br />
A limitation of this tool is that it cannot instantiate virtual machines and bootstrap Disnix.</li>
<li><a href="https://sandervanderburg.blogspot.com/2016/06/deploying-containers-with-disnix-as.html">Disnix itself</a>. The above solutions are all NixOS-based, a software distribution that is Linux-based and fully managed by the Nix package manager.<br />
<br />
Although NixOS is very powerful, it has two drawbacks for Disnix:<br />
<br />
<ul>
<li>NixOS uses the NixOS module system for configuring system aspects. It is very powerful but you can only deploy one instance of a system service -- Disnix can also work with multiple container instances of the same type on a machine.</li>
<li>Services in NixOS cannot be deployed to other kinds software distributions: conventional Linux distributions, and other operating systems, such as macOS and FreeBSD.</li>
</ul>
<br />
To overcome these limitations, Disnix can also be used as a container deployment solution on any operating system that is capable of running Nix and Disnix. Services deployed by Disnix can automatically be exposed as container providers.<br />
<br />
Similar to <i>disnix-deploy-network</i>, a limitation of this approach is that it cannot be used to bootstrap Disnix.</li>
</ul>
<br />
Last year, I have also added a new major feature to Disnix making it possible to <a href="https://sandervanderburg.blogspot.com/2020/04/deploying-container-and-application.html">deploy both application and container services in the same Disnix deployment models</a>, minimizing the infrastructure deployment problem -- the only requirement is to have machines with Nix, Disnix, and a remote connectivity service (such as SSH) pre-installed on them.<br />
<br />
Although this integrated feature is quite convenient, in particular for test setups, a separated infrastructure deployment process (that includes container services) still makes sense in many scenarios:<br />
<br />
<ul>
<li>The infrastructure parts and service parts can be managed by different people with <strong>different specializations</strong>. For example, configuring and tuning an application server is a different responsibility than developing a Java web application.</li>
<li>The service parts typically change more frequently than the infrastructure parts. As a result, they typically have <strong>different</strong> kinds of <strong>update cycles</strong>.</li>
<li>The infrastructure components can typically be <strong>reused</strong> between projects (e.g. many systems use a database backend such as PostgreSQL or MySQL), whereas the service components are typically very project specific.</li>
</ul>
<br />
I also realized that my other project: <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html">the Nix process management framework</a> can serve as a partial infrastructure deployment solution -- it can be used to bootstrap Disnix and deploy container services.<br />
<br />
Moreover, it can also deploy multiple instances of container services and used on any operating system that the Nix process management framework supports, including conventional Linux distributions and other operating systems, such as macOS and FreeBSD.<br />
<br />
<h2>Deploying and exposing the Disnix service with the Nix process management framework</h2>
<br />
As explained earlier, to allow Disnix to deploy services to a remote machine, a machine needs to have Disnix installed (and run the Disnix service for a multi-user installation), and be remotely connectible, e.g. through SSH.<br />
<br />
I have packaged all required services as constructor functions for the Nix process management framework.<br />
<br />
The following process model captures the configuration of a basic multi-user Disnix installation:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, spoolDir ? "${stateDir}/spool"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
ids = if builtins.pathExists ./ids-bare.nix then (import ./ids-bare.nix).ids else {};
constructors = import ../../services-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir spoolDir forceDisableUserChange processManager ids;
};
in
rec {
sshd = {
pkg = constructors.sshd {
extraSSHDConfig = ''
UsePAM yes
'';
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
dbus-daemon = {
pkg = constructors.dbus-daemon {
services = [ disnix-service ];
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
disnix-service = {
pkg = constructors.disnix-service {
inherit dbus-daemon;
};
requiresUniqueIdsFor = [ "gids" ];
};
}
</pre>
<br />
The above processes model (<i>processes.nix</i>) captures three process instances:<br />
<br />
<ul>
<li><i>sshd</i> is the OpenSSH server that makes it possible to remotely connect to the machine by using the SSH protocol.</li>
<li><i>dbus-daemon</i> runs a D-Bus system daemon, that is a requirement for the Disnix service. The <i>disnix-service</i> is propagated as a parameter, so that its service directory gets added to the D-Bus system daemon configuration.</li>
<li><i>disnix-service</i> is a service that executes deployment operations on behalf of an authorized unprivileged user. The <i>disnix-service</i> has a dependency on the <i>dbus-service</i> making sure that the latter gets activated first.</li>
</ul>
<br />
We can deploy the above configuration on a machine that has the Nix process management framework already installed.<br />
<br />
For example, to deploy the configuration on a machine that uses supervisord, we can run:<br />
<br />
<pre>
$ nixproc-supervisord-switch processes.nix
</pre>
<br />
Resulting in a system that consists of the following running processes:<br />
<br />
<pre>
$ supervisorctl
dbus-daemon RUNNING pid 2374, uptime 0:00:34
disnix-service RUNNING pid 2397, uptime 0:00:33
sshd RUNNING pid 2375, uptime 0:00:34
</pre>
<br />
As may be noticed, the above supervised services correspond to the processes in the processes model.<br />
<br />
On the coordinator machine, we can write a <strong>bootstrap</strong> infrastructure model (<i>infra-bootstrap.nix</i>) that only contains connectivity settings:<br />
<br />
<pre>
{
test1.properties.hostname = "192.168.2.1";
}
</pre>
<br />
and use the bootstrap model to capture the full infrastructure model of the system:<br />
<br />
<pre>
$ disnix-capture-infra infra-bootstrap.nix
</pre>
<br />
resulting in the following configuration:<br />
<br />
<pre>
{
"test1" = {
properties = {
"hostname" = "192.168.2.1";
"system" = "x86_64-linux";
};
containers = {
echo = {
};
fileset = {
};
process = {
};
supervisord-program = {
"supervisordTargetDir" = "/etc/supervisor/conf.d";
};
wrapper = {
};
};
"system" = "x86_64-linux";
};
}
</pre>
<br />
Despite the fact that we have not configured any containers explicitly, the above configuration (<i>infrastructure.nix</i>) already exposes a number of container services:<br />
<br />
<ul>
<li>The <i>echo</i>, <i>fileset</i> and <i>process</i> container services are built-in container providers that any Dysnomia installation includes.<br />
<br />
The <i>process</i> container can be used to automatically deploy services that daemonize. Services that daemonize themselves do not require the presence of any external service.</li>
<li>The <i>supervisord-program</i> container refers to the process supervisor that manages the services deployed by the Nix process management framework. It can also be used as a container for processes deployed by Disnix.<br />
</ul>
<br />
With the above infrastructure model, we can deploy any system that depends on the above container services, such as the trivial <a href="https://github.com/svanderburg/disnix-proxy-example">Disnix proxy example</a>:<br />
<br />
<pre style="overflow: auto;">
{ system, distribution, invDistribution, pkgs
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "supervisord"
, nix-processmgmt ? ../../../nix-processmgmt
}:
let
customPkgs = import ../top-level/all-packages.nix {
inherit system pkgs stateDir logDir runtimeDir tmpDir forceDisableUserChange processManager nix-processmgmt;
};
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
processType = import "${nix-processmgmt}/nixproc/derive-dysnomia-process-type.nix" {
inherit processManager;
};
in
rec {
hello_world_server = rec {
name = "hello_world_server";
port = ids.ports.hello_world_server or 0;
pkg = customPkgs.hello_world_server { inherit port; };
type = processType;
requiresUniqueIdsFor = [ "ports" ];
};
hello_world_client = {
name = "hello_world_client";
pkg = customPkgs.hello_world_client;
dependsOn = {
inherit hello_world_server;
};
type = "package";
};
}
</pre>
<br />
The services model shown above (<i>services.nix</i>) captures two services:<br />
<br />
<ul>
<li>The <i>hello_world_server</i> service is a simple service that listens on a TCP port for a "hello" message and responds with a "Hello world!" message.</li>
<li>The <i>hello_world_client</i> service is a package providing a client executable that automatically connects to the <i>hello_world_server</i>.</li>
</ul>
<br />
With the following distribution model (<i>distribution.nix</i>), we can map all the services to our deployment machine (that runs the Disnix service managed by the Nix process management framework):<br />
<br />
<pre>
{infrastructure}:
{
hello_world_client = [ infrastructure.test1 ];
hello_world_server = [ infrastructure.test1 ];
}
</pre>
<br />
and deploy the system by running the following command:<br />
<br />
<pre style="overflow: auto;">
$ disnix-env -s services-without-proxy.nix \
-i infrastructure.nix \
-d distribution.nix \
--extra-params '{ processManager = "supervisord"; }'
</pre>
<br />
The last parameter: <i>--extra-params</i> configures the services model (that indirectly invokes the <i>createManagedProcess</i> abstraction function from the Nix process management framework) in such a way that supervisord configuration files are generated.<br />
<br />
(As a sidenote: without the <i>--extra-params</i> parameter, the process instances will be built for the <a href="https://sandervanderburg.blogspot.com/2020/06/using-disnix-as-simple-and-minimalistic.html"><i>disnix</i> process manager</a> generating configuration files that can be deployed to the process container, expecting programs to daemonize on their own and leave a PID file behind with the daemon's process ID. Although this approach is convenient for experiments, because no external service is required, it is not as reliable as managing supervised processes).<br />
<br />
The result of the above deployment operation is that the <i>hello-world-service</i> service is deployed as a service that is also managed by supervisord:<br />
<br />
<pre>
$ supervisorctl
dbus-daemon RUNNING pid 2374, uptime 0:09:39
disnix-service RUNNING pid 2397, uptime 0:09:38
hello-world-server RUNNING pid 2574, uptime 0:00:06
sshd RUNNING pid 2375, uptime 0:09:39
</pre>
<br />
and we can use the <i>hello-world-client</i> executable on the target machine to connect to the service:<br />
<br />
<pre>
$ /nix/var/nix/profiles/disnix/default/bin/hello-world-client
Trying 192.168.2.1...
Connected to 192.168.2.1.
Escape character is '^]'.
hello
Hello world!
</pre>
<br />
<h2>Deploying container providers and exposing them</h2>
<br />
With Disnix, it is also possible to deploy systems that are composed of different kinds of components, such as web services and databases.<br />
<br />
For example, the <a href="https://github.com/svanderburg/disnix-stafftracker-java-example">Java variant of the ridiculous Staff Tracker example</a> consists of the following services:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbXv9Dy4IDF9spWF5ism7utetQ8s48_xSnSnYVY7EQF140iQhyphenhyphenEsQL46NnOEN007FDjvgxzWSafoy49awoRZEME2jITyGsl6Zx9T9trQrRUgnPm72vlrPby4ohyphenhyphen-hC9QnnmFV2hmz0a-rB/s1480/services.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="322" data-original-width="1480" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgbXv9Dy4IDF9spWF5ism7utetQ8s48_xSnSnYVY7EQF140iQhyphenhyphenEsQL46NnOEN007FDjvgxzWSafoy49awoRZEME2jITyGsl6Zx9T9trQrRUgnPm72vlrPby4ohyphenhyphen-hC9QnnmFV2hmz0a-rB/s600/services.png"/></a></div>
<br />
The services in the diagram above have the following purpose:<br />
<br />
<ul>
<li>The <i>StaffTracker</i> service is the front-end web application that shows an overview of staff members and their locations.</li>
<li>The <i>StaffService</i> service is web service with a SOAP interface that provides read and write access to the staff records. The staff records are stored in the <i>staff</i> database.</li>
<li>The <i>RoomService</i> service provides read access to the rooms records, that are stored in a separate <i>rooms</i> database.</li>
<li>The <i>ZipcodeService</i> service provides read access to zip codes, that are stored in a separate <i>zipcodes</i> database.</li>
<li>The <i>GeolocationService</i> infers the location of a staff member from its IP address using the GeoIP service.</li>
</ul>
<br />
To deploy the system shown above, we need a target machine that provides Apache Tomcat (for managing the web application front-end and web services) and MySQL (for managing the databases) as container provider services:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, spoolDir ? "${stateDir}/spool"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
ids = if builtins.pathExists ./ids-tomcat-mysql.nix then (import ./ids-tomcat-mysql.nix).ids else {};
constructors = import ../../services-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir spoolDir forceDisableUserChange processManager ids;
};
containerProviderConstructors = import ../../service-containers-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir spoolDir forceDisableUserChange processManager ids;
};
in
rec {
sshd = {
pkg = constructors.sshd {
extraSSHDConfig = ''
UsePAM yes
'';
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
dbus-daemon = {
pkg = constructors.dbus-daemon {
services = [ disnix-service ];
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
tomcat = containerProviderConstructors.simpleAppservingTomcat {
commonLibs = [ "${pkgs.mysql_jdbc}/share/java/mysql-connector-java.jar" ];
webapps = [
pkgs.tomcat9.webapps # Include the Tomcat example and management applications
];
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
mysql = containerProviderConstructors.mysql {
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
disnix-service = {
pkg = constructors.disnix-service {
inherit dbus-daemon;
containerProviders = [ tomcat mysql ];
};
requiresUniqueIdsFor = [ "gids" ];
};
}
</pre>
<br />
The process model above is an extension of the previous processes model, adding two container provider services:<br />
<br />
<ul>
<li><i>tomcat</i> is the Apache Tomcat server. The constructor function: <i>simpleAppServingTomcat</i> composes a configuration for a supported process manager, such as supervisord.<br />
<br />
Moreover, it bundles a Dysnomia container configuration file, and a Dysnomia module: <i>tomcat-webapplication</i> that can be used to manage the life-cycles of Java web applications embedded in the servlet container.</li>
<li><i>mysql</i> is the MySQL DBMS server. The constructor function also creates a process manager configuration file, and bundles a Dysnomia container configuration file and module that manages the life-cycles of databases.</li>
<li>The container services above are propagated as <i>containerProviders</i> to the <i>disnix-service</i>. This function parameter is used to update the search paths for container configuration and modules, so that services can be deployed to these containers by Disnix.</li>
</ul>
<br />
After deploying the above processes model, we should see the following infrastructure model after capturing it:<br />
<br />
<pre style="overflow: auto;">
$ disnix-capture-infra infra-bootstrap.nix
{
"test1" = {
properties = {
"hostname" = "192.168.2.1";
"system" = "x86_64-linux";
};
containers = {
echo = {
};
fileset = {
};
process = {
};
supervisord-program = {
"supervisordTargetDir" = "/etc/supervisor/conf.d";
};
wrapper = {
};
tomcat-webapplication = {
"tomcatPort" = "8080";
"catalinaBaseDir" = "/var/tomcat";
};
mysql-database = {
"mysqlPort" = "3306";
"mysqlUsername" = "root";
"mysqlPassword" = "";
"mysqlSocket" = "/var/run/mysqld/mysqld.sock";
};
};
"system" = "x86_64-linux";
};
}
</pre>
<br />
As may be observed, the <i>tomcat-webapplication</i> and <i>mysql-database</i> containers (with their relevant configuration properties) were added to the infrastructure model.<br />
<br />
With the following command we can deploy the example system's services to the containers in the network:<br />
<br />
<pre style="overflow: auto;">
$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix
</pre>
<br />
resulting in a fully functional system:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_hpzjQu_BmVmn4Eqnm6AorCejW-eWTbGe49fHCgTflEA5D8fpX0zk216RV7jbcjxiF7s9sJsZnQzuF8-4s_QA-s3udcoMOty-QfqLT2rwoJTF6VXmGfcWrvP9lfEUErTGUXBmtz4PKy5t/s816/stafftracker.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="706" data-original-width="816" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg_hpzjQu_BmVmn4Eqnm6AorCejW-eWTbGe49fHCgTflEA5D8fpX0zk216RV7jbcjxiF7s9sJsZnQzuF8-4s_QA-s3udcoMOty-QfqLT2rwoJTF6VXmGfcWrvP9lfEUErTGUXBmtz4PKy5t/s600/stafftracker.png"/></a></div>
<br />
<h2>Deploying multiple container provider instances</h2>
<br />
As explained in the introduction, a limitation of the NixOS module system is that it is only possible to construct one instance of a service on a machine.<br />
<br />
Process instances in a processes model deployed by the Nix process management framework as well as services in a Disnix services model are instantiated from functions that make it possible to deploy <strong>multiple instances</strong> of the same service to the same machine, by making conflicting properties configurable.<br />
<br />
The following processes model was modified from the previous example to deploy two MySQL servers and two Apache Tomcat servers to the same machine:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, spoolDir ? "${stateDir}/spool"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
ids = if builtins.pathExists ./ids-tomcat-mysql-multi-instance.nix then (import ./ids-tomcat-mysql-multi-instance.nix).ids else {};
constructors = import ../../services-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir spoolDir forceDisableUserChange processManager ids;
};
containerProviderConstructors = import ../../service-containers-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir spoolDir forceDisableUserChange processManager ids;
};
in
rec {
sshd = {
pkg = constructors.sshd {
extraSSHDConfig = ''
UsePAM yes
'';
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
dbus-daemon = {
pkg = constructors.dbus-daemon {
services = [ disnix-service ];
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
tomcat-primary = containerProviderConstructors.simpleAppservingTomcat {
instanceSuffix = "-primary";
httpPort = 8080;
httpsPort = 8443;
serverPort = 8005;
ajpPort = 8009;
commonLibs = [ "${pkgs.mysql_jdbc}/share/java/mysql-connector-java.jar" ];
webapps = [
pkgs.tomcat9.webapps # Include the Tomcat example and management applications
];
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
tomcat-secondary = containerProviderConstructors.simpleAppservingTomcat {
instanceSuffix = "-secondary";
httpPort = 8081;
httpsPort = 8444;
serverPort = 8006;
ajpPort = 8010;
commonLibs = [ "${pkgs.mysql_jdbc}/share/java/mysql-connector-java.jar" ];
webapps = [
pkgs.tomcat9.webapps # Include the Tomcat example and management applications
];
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
mysql-primary = containerProviderConstructors.mysql {
instanceSuffix = "-primary";
port = 3306;
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
mysql-secondary = containerProviderConstructors.mysql {
instanceSuffix = "-secondary";
port = 3307;
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
disnix-service = {
pkg = constructors.disnix-service {
inherit dbus-daemon;
containerProviders = [ tomcat-primary tomcat-secondary mysql-primary mysql-secondary ];
};
requiresUniqueIdsFor = [ "gids" ];
};
}
</pre>
<br />
In the above processes model, we made the following changes:<br />
<br />
<ul>
<li>We have configured two Apache Tomcat instances: <i>tomcat-primary</i> and <i>tomcat-secondary</i>. Both instances can co-exist because they have been configured in such a way that they listen to unique TCP ports and have a unique instance name composed from the <i>instanceSuffix</i>.</li>
<li>We have configured two MySQL instances: <i>mysql-primary</i> and <i>mysql-secondary</i>. Similar to Apache Tomcat, they can both co-exist because they listen to unique TCP ports (e.g. <i>3306</i> and <i>3307</i>) and have a unique instance name.</li>
<li>Both the primary and secondary instances of the above services are propagated to the <i>disnix-service</i> (with the <i>containerProviders</i> parameter) making it possible for a client to discover them.</li>
</ul>
<br />
After deploying the above processes model, we can run the following command to discover the machine's configuration:<br />
<br />
<pre>
$ disnix-capture-infra infra-bootstrap.nix
{
"test1" = {
properties = {
"hostname" = "192.168.2.1";
"system" = "x86_64-linux";
};
containers = {
echo = {
};
fileset = {
};
process = {
};
supervisord-program = {
"supervisordTargetDir" = "/etc/supervisor/conf.d";
};
wrapper = {
};
tomcat-webapplication-primary = {
"tomcatPort" = "8080";
"catalinaBaseDir" = "/var/tomcat-primary";
};
tomcat-webapplication-secondary = {
"tomcatPort" = "8081";
"catalinaBaseDir" = "/var/tomcat-secondary";
};
mysql-database-primary = {
"mysqlPort" = "3306";
"mysqlUsername" = "root";
"mysqlPassword" = "";
"mysqlSocket" = "/var/run/mysqld-primary/mysqld.sock";
};
mysql-database-secondary = {
"mysqlPort" = "3307";
"mysqlUsername" = "root";
"mysqlPassword" = "";
"mysqlSocket" = "/var/run/mysqld-secondary/mysqld.sock";
};
};
"system" = "x86_64-linux";
};
}
</pre>
<br />
As may be observed, the infrastructure model contains two Apache Tomcat instances and two MySQL instances.<br />
<br />
With the following distribution model (<i>distribution.nix</i>), we can divide each database and web application over the two container instances:<br />
<br />
<pre>
{infrastructure}:
{
GeolocationService = {
targets = [
{ target = infrastructure.test1;
container = "tomcat-webapplication-primary";
}
];
};
RoomService = {
targets = [
{ target = infrastructure.test1;
container = "tomcat-webapplication-secondary";
}
];
};
StaffService = {
targets = [
{ target = infrastructure.test1;
container = "tomcat-webapplication-primary";
}
];
};
StaffTracker = {
targets = [
{ target = infrastructure.test1;
container = "tomcat-webapplication-secondary";
}
];
};
ZipcodeService = {
targets = [
{ target = infrastructure.test1;
container = "tomcat-webapplication-primary";
}
];
};
rooms = {
targets = [
{ target = infrastructure.test1;
container = "mysql-database-primary";
}
];
};
staff = {
targets = [
{ target = infrastructure.test1;
container = "mysql-database-secondary";
}
];
};
zipcodes = {
targets = [
{ target = infrastructure.test1;
container = "mysql-database-primary";
}
];
};
}
</pre>
<br />
Compared to the previous distribution model, the above model uses a more verbose notation for mapping services.<br />
<br />
As explained in <a href="https://sandervanderburg.blogspot.com/2016/05/mapping-services-to-containers-with.html">an earlier blog post</a>, in deployments in which only a single container is deployed, services are <strong>automapped</strong> to the container that has the same name as the service's type. When multiple instances exist, we need to manually specify the container where the service needs to be deployed to.<br />
<br />
After deploying the system with the following command:<br />
<br >
<pre>
$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix
</pre>
<br />
we will get a running system with the following deployment architecture:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiLJvQ3-uWU1w2cisqMmmVlCER0l-rNfdoNDRGW95vmYeuiem-lW0IPUsxXqWhQLxCpk4lHkqc5x3v2FLj3iJc5FxjadAUDWmhUygUThwC3emHhYwgC4jELpbpNV_rtZTlJe6K_8z77dRj/s1113/multicontainerarch.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="500" data-original-height="395" data-original-width="1113" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhiLJvQ3-uWU1w2cisqMmmVlCER0l-rNfdoNDRGW95vmYeuiem-lW0IPUsxXqWhQLxCpk4lHkqc5x3v2FLj3iJc5FxjadAUDWmhUygUThwC3emHhYwgC4jELpbpNV_rtZTlJe6K_8z77dRj/s600/multicontainerarch.png"/></a></div>
<br />
<h2>Using the Disnix web service for executing remote deployment operations</h2>
<br />
By default, Disnix uses SSH to communicate to target machines in the network. Disnix has a modular architecture and is also capable of communicating to target machines by other means, for example via NixOps, the backdoor client, D-Bus, and directly executing tasks on a local machine.<br />
<br />
There is also an external package: <i>DisnixWebService</i> that remotely exposes all deployment operations from a web service with a SOAP API.<br />
<br />
To use the <i>DisnixWebService</i>, we must deploy a Java servlet container (such as Apache Tomcat) with the <i>DisnixWebService</i> application, configured in such a way that it can connect to the <i>disnix-service</i> over the D-Bus system bus.<br />
<br />
The following processes model is an extension of the non-multi containers Staff Tracker example, with an Apache Tomcat service that bundles the <i>DisnixWebService</i>:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, spoolDir ? "${stateDir}/spool"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
ids = if builtins.pathExists ./ids-tomcat-mysql.nix then (import ./ids-tomcat-mysql.nix).ids else {};
constructors = import ../../services-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir spoolDir forceDisableUserChange processManager ids;
};
containerProviderConstructors = import ../../service-containers-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir spoolDir forceDisableUserChange processManager ids;
};
in
rec {
sshd = {
pkg = constructors.sshd {
extraSSHDConfig = ''
UsePAM yes
'';
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
dbus-daemon = {
pkg = constructors.dbus-daemon {
services = [ disnix-service ];
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
tomcat = containerProviderConstructors.disnixAppservingTomcat {
commonLibs = [ "${pkgs.mysql_jdbc}/share/java/mysql-connector-java.jar" ];
webapps = [
pkgs.tomcat9.webapps # Include the Tomcat example and management applications
];
enableAJP = true;
inherit dbus-daemon;
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
apache = {
pkg = constructors.basicAuthReverseProxyApache {
dependency = tomcat;
serverAdmin = "admin@localhost";
targetProtocol = "ajp";
portPropertyName = "ajpPort";
authName = "DisnixWebService";
authUserFile = pkgs.stdenv.mkDerivation {
name = "htpasswd";
buildInputs = [ pkgs.apacheHttpd ];
buildCommand = ''
htpasswd -cb ./htpasswd admin secret
mv htpasswd $out
'';
};
requireUser = "admin";
};
requiresUniqueIdsFor = [ "uids" "gids" ];
};
mysql = containerProviderConstructors.mysql {
properties.requiresUniqueIdsFor = [ "uids" "gids" ];
};
disnix-service = {
pkg = constructors.disnix-service {
inherit dbus-daemon;
containerProviders = [ tomcat mysql ];
authorizedUsers = [ tomcat.name ];
dysnomiaProperties = {
targetEPR = "http://$(hostname)/DisnixWebService/services/DisnixWebService";
};
};
requiresUniqueIdsFor = [ "gids" ];
};
}
</pre>
<br />
The above processes model contains the following changes:<br />
<br />
<ul>
<li>The Apache Tomcat process instance is constructed with the <i>containerProviderConstructors.disnixAppservingTomcat</i> constructor function automatically deploying the <i>DisnixWebService</i> and providing the required configuration settings so that it can communicate with the <i>disnix-service</i> over the D-Bus system bus.<br />
<br />
Because the <i>DisnixWebService</i> requires the presence of the D-Bus system daemon, it is configured as a <i>dependency</i> for Apache Tomcat ensuring that it is started before Apache Tomcat.</li>
<li>Connecting to the Apache Tomcat server including the <i>DisnixWebService</i> requires no authentication. To secure the web applications and the <i>DisnixWebService</i>, I have configured an <i>apache</i> reverse proxy that forwards connections to Apache Tomcat using the AJP protocol.<br />
<br />
Moreover, the reverse proxy protects incoming requests by using HTTP basic authentication requiring a username and password.</li>
</ul>
<br />
We can use the following bootstrap infrastructure model to discover the machine's configuration:<br />
<br />
<pre style="overflow: auto;">
{
test1.properties.targetEPR = "http://192.168.2.1/DisnixWebService/services/DisnixWebService";
}
</pre>
<br />
The difference between this bootstrap infrastructure model and the previous is that it uses a different connection property (<i>targetEPR</i>) that refers to the URL of the <i>DisnixWebService</i>.<br />
<br />
By default, Disnix uses the <i>disnix-ssh-client</i> to communicate to target machines. To use a different client, we must set the following environment variables:<br />
<br />
<pre>
$ export DISNIX_CLIENT_INTERFACE=disnix-soap-client
$ export DISNIX_TARGET_PROPERTY=targetEPR
</pre>
<br />
The above environment variables instruct Disnix to use the <i>disnix-soap-client</i> executable and the <i>targetEPR</i> property from the infrastructure model as a connection string.<br />
<br />
To authenticate ourselves, we must set the following environment variables with a username and password:<br />
<br />
<pre>
$ export DISNIX_SOAP_CLIENT_USERNAME=admin
$ export DISNIX_SOAP_CLIENT_PASSWORD=secret
</pre>
<br />
The following command makes it possible to discover the machine's configuration using the <i>disnix-soap-client</i> and <i>DisnixWebService</i>:<br />
<br />
<pre style="overflow: auto;">
$ disnix-capture-infra infra-bootstrap.nix
{
"test1" = {
properties = {
"hostname" = "192.168.2.1";
"system" = "x86_64-linux";
"targetEPR" = "http://192.168.2.1/DisnixWebService/services/DisnixWebService";
};
containers = {
echo = {
};
fileset = {
};
process = {
};
supervisord-program = {
"supervisordTargetDir" = "/etc/supervisor/conf.d";
};
wrapper = {
};
tomcat-webapplication = {
"tomcatPort" = "8080";
"catalinaBaseDir" = "/var/tomcat";
"ajpPort" = "8009";
};
mysql-database = {
"mysqlPort" = "3306";
"mysqlUsername" = "root";
"mysqlPassword" = "";
"mysqlSocket" = "/var/run/mysqld/mysqld.sock";
};
};
"system" = "x86_64-linux";
}
;
}
</pre>
<br />
After capturing the full infrastructure model, we can deploy the system with <i>disnix-env</i> if desired, using the <i>disnix-soap-client</i> to carry out all necessary remote deployment operations.<br />
<br />
<h2>Miscellaneous: using Docker containers as light-weight virtual machines</h2>
<br />
As explained earlier in this blog post, the Nix process management framework is only a partial infrastructure deployment solution -- you still need to somehow obtain physical or virtual machines with a software distribution running the Nix package manager.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">In a blog post written some time ago</a>, I have explained that Docker containers are not virtual machines or even light-weight virtual machines.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2021/02/deploying-mutable-multi-process-docker.html">In my previous blog post</a>, I have shown that we can also deploy mutable Docker multi-process containers in which process instances can be upgraded without stopping the container.<br />
<br />
The deployment workflow for upgrading mutable containers, is very machine-like -- NixOS has a similar workflow that consists of updating the machine configuration (<i>/etc/nixos/configuration.nix</i>) and running a single command-line instruction to upgrade machine (<i>nixos-rebuild switch</i>).<br />
<br />
We can actually start using containers as VMs by adding another ingredient in the mix -- <a href="https://stackoverflow.com/questions/27937185/assign-static-ip-to-docker-container">we can also assign static IP addresses to Docker containers</a>.<br />
<br />
With the following Nix expression, we can create a Docker image for a mutable container, using any of the processes models shown previously as the "machine's configuration":<br />
<br />
<pre style="overflow: auto;">
let
pkgs = import <nixpkgs> {};
createMutableMultiProcessImage = import ../nix-processmgmt/nixproc/create-image-from-steps/create-mutable-multi-process-image-universal.nix {
inherit pkgs;
};
in
createMutableMultiProcessImage {
name = "disnix";
tag = "test";
contents = [ pkgs.mc pkgs.disnix ];
exprFile = ./processes.nix;
interactive = true;
manpages = true;
processManager = "supervisord";
}
</pre>
<br />
The <i>exprFile</i> in the above Nix expression refers to a previously shown processes model, and the <i>processManager</i> the desired process manager to use, such as supervisord.<br />
<br />
With the following command, we can build the image with Nix and load it into Docker:<br />
<br />
<pre>
$ nix-build
$ docker load -i result
</pre>
<br />
With the following command, we can create a network to which our containers (with IP addresses) should belong:<br />
<br />
<pre>
$ docker network create --subnet=192.168.2.0/8 disnixnetwork
</pre>
<br />
The above command creates a subnet with a prefix: <i>192.168.2.0</i> and allocates an 8-bit block for host IP addresses.<br />
<br />
We can create and start a Docker container named: <i>containervm</i> using our previously built image, and assign it an IP address:<br />
<br />
<pre>
$ docker run --network disnixnetwork --ip 192.168.2.1 \
--name containervm disnix:test
</pre>
<br />
By default, Disnix uses SSH to connect to remote machines. With the following commands we can create a public-private key pair and copy the public key to the container:<br />
<br />
<pre>
$ ssh-keygen -t ed25519 -f id_test -N ""
$ docker exec containervm mkdir -m0700 -p /root/.ssh
$ docker cp id_test.pub containervm:/root/.ssh/authorized_keys
$ docker exec containervm chmod 600 /root/.ssh/authorized_keys
$ docker exec containervm chown root:root /root/.ssh/authorized_keys
</pre>
<br />
On the coordinator machine, that carries out the deployment, we must add the private key to the SSH agent and configure the <i>disnix-ssh-client</i> to connect to the <i>disnix-service</i>:<br />
<br />
<pre>
$ ssh-add id_test
$ export DISNIX_REMOTE_CLIENT=disnix-client
</pre>
<br />
By executing all these steps, <i>containervm</i> can be (mostly) used as if it were a virtual machine, including connecting to it with an IP address over SSH.<br />
<br />
<h2>Conclusion</h2>
<br />
In this blog post, I have described how the Nix process management framework can be used as a partial infrastructure deployment solution for Disnix. It can be used both for deploying the <i>disnix-service</i> (to facilitate multi-user installations) as well as deploying container providers: services that manage the life-cycles of services deployed by Disnix.<br />
<br />
Moreover, the Nix process management framework makes it possible to do these deployments on all kinds of software distributions that can use the Nix package manager, including NixOS, conventional Linux distributions and other operating systems, such as macOS and FreeBSD.<br />
<br />
If I had developed this solution a couple of years ago, it would probably have saved me many hours of preparation work for my first demo in <a href="https://sandervanderburg.blogspot.com/2015/11/deploying-services-to-heterogeneous.html">my NixCon 2015 talk</a> in which I wanted demonstrate that it is possible to deploy services to a heterogeneous network that consists of a NixOS, Ubuntu and Windows machine. Back then, I had to do all the infrastructure deployment tasks manually.<br />
<br />
I also have to admit (but this statement is mostly based on my personal preferences, not facts), is that I find the functional style that the framework uses is IMO far more intuitive than the NixOS module system for certain service configuration aspects, especially for configuring container services and exposing them with Disnix and Dysnomia:<br />
<br />
<ul>
<li>Because every process instance is constructed from a constructor function that makes all instance parameters explicit, you are guarded against common configuration errors such as undeclared dependencies.<br />
<br />
For example, the <i>DisnixWebService</i>-enabled Apache Tomcat service requires access to the <i>dbus-service</i> providing the system bus. Not having this service in the processes model, causes a missing function parameter error.</li>
<li>Function parameters in the processes model make it more clear that a process depends on another process and what that relationship may be. For example, with the <i>containerProviders</i> parameter it becomes IMO really clear that the <i>disnix-service</i> uses them as potential deployment targets for services deployed by Disnix.<br />
<br />
In comparison, the implementations of the Disnix and Dysnomia NixOS modules are far more complicated and monolithic -- the Dysnomia module has to figure for all potential container services deployed as part of a NixOS configuration, their properties, convert them to Dysnomia configuration files, and configure the systemd configuration for the <i>disnix-service</i> for proper activation ordering.<br />
<br />
The <i>wants</i> parameter (used for activation ordering) is just a list of strings, not knowing whether it contains valid references to services that have been deployed already.</li>
</ul>
<br />
<h2>Availability</h2>
<br />
The constructor functions for the services as well as the deployment examples described in this blog post can be found in the <a href="https://github.com/svanderburg/nix-processmgmt-services">Nix process management services repository</a>.<br />
<br />
<h2>Future work</h2>
<br />
Slowly more and more of my personal use cases are getting supported by the Nix process management framework.<br />
<br />
Moreover, the services repository is steadily growing. To ensure that all the services that I have packaged so far do not break, I really need to focus my work on a service test solution.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-4155698244385190892021-02-24T22:46:00.001+01:002021-02-25T10:41:52.221+01:00Deploying mutable multi-process Docker containers with the Nix process management framework (or running Hydra in a Docker container)In a blog post written several months ago, I have shown that the <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html">Nix process management framework</a> can also be used to conveniently <a href="https://sandervanderburg.blogspot.com/2020/10/building-multi-process-docker-images.html">construct multi-process Docker images</a>.<br />
<br />
Although Docker is primarily used for managing single root application process containers, multi-process containers can sometimes be useful to deploy systems that consist of multiple, tightly coupled, processes.<br />
<br />
The Docker manual has a section that describes <a href="https://docs.docker.com/config/containers/multi-service_container">how to construct images for multi-process containers</a>, but IMO the configuration process is a bit tedious and cumbersome.<br />
<br />
To make this process more convenient, I have built a <strong>wrapper</strong> function: <i>createMultiProcessImage</i> around the <i>dockerTools.buildImage</i> function (provided by Nixpkgs) that does the following:<br />
<br />
<ul>
<li>It constructs an image that runs a Linux and Docker compatible process manager as an entry point. Currently, it supports <i>supervisord</i>, <i>sysvinit</i>, <i>disnix</i> and <i>s6-rc</i>.</li>
<li>The Nix process management framework is used to build a configuration for a system that consists of multiple processes, that will be managed by any of the supported process managers.</li>
</ul>
<br />
Although the framework makes the construction of multi-process images convenient, a big drawback of multi-process Docker containers is <strong>upgrading</strong> them -- for example, for Debian-based containers you can imperatively upgrade packages by connecting to the container:<br />
<br />
<pre>
$ docker exec -it mycontainer /bin/bash
</pre>
<br />
and upgrade the desired packages, such as <i>file</i>:<br />
<br />
<pre>
$ apt install file
</pre>
<br />
The upgrade instruction above is <strong>not reproducible</strong> -- <i>apt</i> may install <i>file</i> version 5.38 today, and 5.39 tomorrow.<br />
<br />
To cope with these kinds of side-effects, Docker works with <strong>images</strong> that snapshot the outcomes of all the installation steps. Constructing a container from the same image will always provide the same versions of all dependencies.<br />
<br />
As a consequence, to perform a <strong>reproducible</strong> container <strong>upgrade</strong>, it is required to construct a new image, <strong>discard</strong> the container and <strong>reconstruct</strong> the container from the new image version, causing the system as a whole to be terminated, including the processes that have not changed.<br />
<br />
For a while, I have been thinking about this limitation and developed a solution that makes it possible to upgrade multi-process containers without stopping and discarding them. The only exception is the process manager.<br />
<br />
To make deployments reproducible, it combines the reproducibility properties of Docker and Nix.<br />
<br />
In this blog post, I will describe how this solution works and how it can be used.<br />
<br />
<h2>Creating a function for building mutable Docker images</h2>
<br />
As explained in an earlier blog post, <a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">that compares the deployment properties of Nix and Docker</a>, both solutions support reproducible deployment, albeit for different application domains.<br />
<br />
Moreover, their reproducibility properties are built around different concepts:<br />
<br />
<ul>
<li>Docker containers are reproducible, because they are constructed from <strong>images</strong> that consist of immutable layers identified by hash codes derived from their contents.</li>
<li><a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix package builds are reproducible</a>, because they are stored in <strong>isolation</strong> in a Nix store and made immutable (the files' permissions are set read-only). In the construction process of the packages, many <strong>side effects</strong> are <strong>mitigated</strong>.<br />
<br />
As a result, when the hash code prefix of a package (derived from all build inputs) is the same, then the build output is also (nearly) bit-identical, regardless of the machine on which the package was built.</li>
</ul>
<br />
By taking these reproducibilty properties into account, we can create a reproducible deployment process for upgradable containers by using a specific separation of responsibilities.<br />
<br />
<h3>Deploying the base system</h3>
<br />
For the deployment of the <strong>base system</strong> that includes the <strong>process manager</strong>, we can stick ourselves to the traditional Docker deployment workflow based on images (the only unconventional aspect is that we use Nix to build a Docker image, instead of <i>Dockerfile</i>s).<br />
<br />
The process manager that the image provides deploys its configuration from a dynamic configuration directory.<br />
<br />
To support supervisord, we can invoke the following command as the container's entry point:<br />
<br />
<pre>
supervisord --nodaemon \
--configuration /etc/supervisor/supervisord.conf \
--logfile /var/log/supervisord.log \
--pidfile /var/run/supervisord.pid
</pre>
<br />
The above command starts the supervisord service (in foreground mode), using the <i>supervisord.conf</i> configuration file stored in <i>/etc/supervisord</i>.<br />
<br />
The <i>supervisord.conf</i> configuration file has the following structure:<br />
<br />
<pre>
[supervisord]
[include]
files=conf.d/*
</pre>
<br />
The above configuration automatically loads all program definitions stored in the <i>conf.d</i> directory. This directory is writable and initially empty. It can be populated with configuration files generated by the Nix process management framework.<br />
<br />
For the other process managers that the framework supports (<i>sysvinit</i>, <i>disnix</i> and <i>s6-rc</i>), we follow a similar strategy -- we configure the process manager in such a way that the configuration is loaded from a source that can be dynamically updated.<br />
<br />
<h3>Deploying process instances</h3>
<br />
Deployment of the <strong>process instances</strong> is not done in the construction of the image, but by the Nix process management framework and the Nix package manager running in the container.<br />
<br />
To allow a processes model deployment to refer to packages in the Nixpkgs collection and install binary substitutes, we must configure a Nix channel, such as the unstable Nixpkgs channel:<br />
<br />
<pre>
$ nix-channel --add https://nixos.org/channels/nixpkgs-unstable
$ nix-channel --update
</pre>
<br />
(As a sidenote: it is also possible to subscribe to a stable Nixpkgs channel or a specific Git revision of Nixpkgs).<br />
<br />
The processes model (and relevant sub models, such as <i>ids.nix</i> that contains <a href="https://sandervanderburg.blogspot.com/2020/09/assigning-unique-ids-to-services-in.html">numeric ID assignments</a>) are copied into the Docker image.<br />
<br />
We can deploy the processes model for supervisord as follows:<br />
<br />
<pre>
$ nixproc-supervisord-switch
</pre>
<br />
The above command will deploy the processes model in the <i>NIXPROC_PROCESSES</i> environment variable, which defaults to: <i>/etc/nixproc/processes.nix</i>:<br />
<br />
<ul>
<li>First, it builds supervisord configuration files from the processes model (this step also includes deploying all required packages and service configuration files)</li>
<li>It creates symlinks for each configuration file belonging to a process instance in the writable <i>conf.d</i> directory</li>
<li>It instructs <i>supervisord</i> to reload the configuration so that only obsolete processes get deactivated and new services activated, causing unchanged processes to remain untouched.</li>
</ul>
<br />
(For the other process managers, we have equivalent tools: <i>nixproc-sysvinit-switch</i>, <i>nixproc-disnix-switch</i> and <i>nixproc-s6-rc-switch</i>).<br />
<br />
<h3>Initial deployment of the system</h3>
<br />
Because only the process manager is deployed as part of the image (with an initially empty configuration), the system is not yet usable when we start a container.<br />
<br />
To solve this problem, we must perform an <strong>initial deployment</strong> of the system on first startup.<br />
<br />
I used my lessons learned from the chainloading techniques in <a href="https://skarnet.org/software/s6/">s6</a> (in the <a href="https://sandervanderburg.blogspot.com/2021/02/developing-s6-rc-backend-for-nix.html">previous blog post</a>) and developed hacky generated <strong>bootstrap script</strong> (<i>/bin/bootstrap</i>) that serves as the container's entry point:<br />
<br />
<pre style="overflow: auto;">
cat > /bin/bootstrap <<EOF
#! ${pkgs.stdenv.shell} -e
# Configure Nix channels
nix-channel --add ${channelURL}
nix-channel --update
# Deploy the processes model (in a child process)
nixproc-${input.processManager}-switch &
# Overwrite the bootstrap script, so that it simply just
# starts the process manager the next time we start the
# container
cat > /bin/bootstrap <<EOR
#! ${pkgs.stdenv.shell} -e
exec ${cmd}
EOR
# Chain load the actual process manager
exec ${cmd}
EOF
chmod 755 /bin/bootstrap
</pre>
<br />
The generated bootstrap script does the following:<br />
<br />
<ul>
<li>First, a Nix channel is configured and updated so that we can install packages from the Nixpkgs collection and obtain substitutes.</li>
<li>The next step is deploying the processes model by running the <i>nixproc-*-switch</i> tool for a supported process manager. This process is started in the background (as a child process) -- we can use this trick to force the managing bash shell to load our desired process supervisor as soon as possible.<br />
<br />
Ultimately, we want the process manager to become responsible for supervising any other process running in the container.</li>
<li>After the deployment process is started in the background, the bootstrap script is overridden by a bootstrap script that becomes our real entry point -- the process manager that we want to use, such as <i>supervisord</i>.<br />
<br />
Overriding the bootstrap script makes sure that the next time we start the container, it will start instantly without attempting to deploy the system again.</li>
<li>Finally, the bootstrap script "execs" into the real process manager, becoming the new PID 1 process. When the deployment of the system is done (the <i>nixproc-*-switch</i> process that still runs in the background), the process manager becomes responsible for reaping it.</li>
</ul>
<br />
With the above script, the workflow of deploying an upgradable/mutable multi-process container is the same as deploying an ordinary container from a Docker image -- the only (minor) difference is that the first time that we start the container, it may take some time before the services become available, because the multi-process system needs to be deployed by Nix and the Nix process management framework.<br />
<br />
<h2>A simple usage scenario</h2>
<br />
Similar to my previous blog posts about the Nix process management framework, I will use the trivial web application system to demonstrate how the functionality of the framework can be used.<br />
<br />
The web application system consists of one or more <i>webapp</i> processes (with an embedded HTTP server) that only return static HTML pages displaying their identities.<br />
<br />
An Nginx reverse proxy forwards incoming requests to the appropriate <i>webapp</i> instance -- each <i>webapp</i> service can be reached by using its unique virtual host value.<br />
<br />
To construct a mutable multi-process Docker image with Nix, we can write the following Nix expression (<i>default.nix</i>):<br />
<br />
<pre style="overflow: auto;">
let
pkgs = import <nixpkgs> {};
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
createMutableMultiProcessImage = import "${nix-processmgmt}/nixproc/create-image-from-steps/create-mutable-multi-process-image-universal.nix" {
inherit pkgs;
};
in
createMutableMultiProcessImage {
name = "multiprocess";
tag = "test";
contents = [ pkgs.mc ];
exprFile = ./processes.nix;
idResourcesFile = ./idresources.nix;
idsFile = ./ids.nix;
processManager = "supervisord"; # sysvinit, disnix, s6-rc are also valid options
}
</pre>
<br />
The above Nix expression invokes the <i>createMutableMultiProcessImage</i> function that constructs a Docker image that provides a base system with a process manager, and a bootstrap script that deploys the multi-process system:<br />
<br />
<ul>
<li>The <i>name</i>, <i>tag</i>, and <i>contents</i> parameters specify the image name, tag and the packages that need to be included in the image.</li>
<li>The <i>exprFile</i> parameter refers to a <strong>processes model</strong> that captures the configurations of the process instances that need to be deployed.</li>
<li>The <i>idResources</i> parameter refers to an <strong>ID resources</strong> model that specifies from which resource pools unique IDs need to be selected.</li>
<li>The <i>idsFile</i> parameter refers to an <strong>IDs model</strong> that contains the unique ID assignments for each process instance. Unique IDs resemble TCP/UDP port assignments, user IDs (UIDs) and group IDs (GIDs).</li>
<li>We can use the <i>processManager</i> parameter to select the process manager we want to use. In the above example it is <i>supervisord</i>, but other options are also possible.</li>
</ul>
<br />
We can use the following processes model (<i>processes.nix</i>) to deploy a small version of our example system:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
sharedConstructors = import "${nix-processmgmt}/examples/services-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir cacheDir tmpDir forceDisableUserChange processManager ids;
};
constructors = import "${nix-processmgmt}/examples/webapps-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager ids;
};
in
rec {
webapp = rec {
port = ids.webappPorts.webapp or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
requiresUniqueIdsFor = [ "webappPorts" "uids" "gids" ];
};
nginx = rec {
port = ids.nginxPorts.nginx or 0;
pkg = sharedConstructors.nginxReverseProxyHostBased {
webapps = [ webapp ];
inherit port;
} {};
requiresUniqueIdsFor = [ "nginxPorts" "uids" "gids" ];
};
}
</pre>
<br />
The above Nix expression configures two process instances, one <i>webapp</i> process that returns a static HTML page with its identity and an Nginx reverse proxy that forwards connections to it.<br />
<br />
A notable difference between the expression shown above and the processes models of the same system shown in my previous blog posts, is that this expression does not contain any references to files on the local filesystem, with the exception of the ID assignments expression (<i>ids.nix</i>).<br />
<br />
We obtain all required functionality from the Nix process management framework by invoking <i>builtins.fetchGit</i>. Eliminating local references is required to allow the processes model to be copied into the container and deployed from within the container.<br />
<br />
We can build a Docker image as follows:<br />
<br />
<pre>
$ nix-build
</pre>
<br />
load the image into Docker:<br />
<br />
<pre>
$ docker load -i result
</pre>
<br />
and create and start a Docker container:<br />
<br />
<pre style="overflow: auto;">
$ docker run -it --name webapps --network host multiprocess:test
unpacking channels...
warning: Nix search path entry '/nix/var/nix/profiles/per-user/root/channels' does not exist, ignoring
created 1 symlinks in user environment
2021-02-21 15:29:29,878 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message.
2021-02-21 15:29:29,878 WARN No file matches via include "/etc/supervisor/conf.d/*"
2021-02-21 15:29:29,897 INFO RPC interface 'supervisor' initialized
2021-02-21 15:29:29,897 CRIT Server 'inet_http_server' running without any HTTP authentication checking
2021-02-21 15:29:29,898 INFO supervisord started with pid 1
these derivations will be built:
/nix/store/011g52sj25k5k04zx9zdszdxfv6wy1dw-credentials.drv
/nix/store/1i9g728k7lda0z3mn1d4bfw07v5gzkrv-credentials.drv
/nix/store/fs8fwfhalmgxf8y1c47d0zzq4f89fz0g-nginx.conf.drv
/nix/store/vxpm2m6444fcy9r2p06dmpw2zxlfw0v4-nginx-foregroundproxy.sh.drv
/nix/store/4v3lxnpapf5f8297gdjz6kdra8g7k4sc-nginx.conf.drv
/nix/store/mdldv8gwvcd5fkchncp90hmz3p9rcd99-builder.pl.drv
/nix/store/r7qjyr8vr3kh1lydrnzx6nwh62spksx5-nginx.drv
/nix/store/h69khss5dqvx4svsc39l363wilcf2jjm-webapp.drv
/nix/store/kcqbrhkc5gva3r8r0fnqjcfhcw4w5il5-webapp.conf.drv
/nix/store/xfc1zbr92pyisf8lw35qybbn0g4f46sc-webapp.drv
/nix/store/fjx5kndv24pia1yi2b7b2bznamfm8q0k-supervisord.d.drv
these paths will be fetched (78.80 MiB download, 347.06 MiB unpacked):
...
</pre>
<br />
As may be noticed by looking at the output, on first startup the Nix process management framework is invoked to deploy the system with Nix.<br />
<br />
After the system has been deployed, we should be able to connect to the <i>webapp</i> process via the Nginx reverse proxy:<br />
<br />
<pre>
$ curl -H 'Host: webapp.local' http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5000
</body>
</html>
</pre>
<br />
When it is desired to upgrade the system, we can change the system's configuration by connecting to the container instance:<br />
<br />
<pre>
$ docker exec -it webapps /bin/bash
</pre>
<br />
In the container, we can edit the <i>processes.nix</i> configuration file:<br />
<br />
<pre>
$ mcedit /etc/nixproc/processes.nix
</pre>
<br />
and make changes to the configuration of the system. For example, we can change the processes model to include a second <i>webapp</i> process:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
sharedConstructors = import "${nix-processmgmt}/examples/services-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir cacheDir tmpDir forceDisableUserChange processManager ids;
};
constructors = import "${nix-processmgmt}/examples/webapps-agnostic/constructors/constructors.nix" {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager ids;
};
in
rec {
webapp = rec {
port = ids.webappPorts.webapp or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
requiresUniqueIdsFor = [ "webappPorts" "uids" "gids" ];
};
webapp2 = rec {
port = ids.webappPorts.webapp2 or 0;
dnsName = "webapp2.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "2";
};
requiresUniqueIdsFor = [ "webappPorts" "uids" "gids" ];
};
nginx = rec {
port = ids.nginxPorts.nginx or 0;
pkg = sharedConstructors.nginxReverseProxyHostBased {
webapps = [ webapp webapp2 ];
inherit port;
} {};
requiresUniqueIdsFor = [ "nginxPorts" "uids" "gids" ];
};
}
</pre>
<br />
In the above process model model, a new process instance named: <i>webapp2</i> was added that listens on a unique port that can be reached with the <i>webapp2.local</i> virtual host value.<br />
<br />
By running the following command, the system in the container gets upgraded:<br />
<br />
<pre>
$ nixproc-supervisord-switch
</pre>
<br />
resulting in two <i>webapp</i> process instances running in the container:<br />
<br />
<pre>
$ supervisorctl
nginx RUNNING pid 847, uptime 0:00:08
webapp RUNNING pid 459, uptime 0:05:54
webapp2 RUNNING pid 846, uptime 0:00:08
supervisor>
</pre>
<br />
The first instance: <i>webapp</i> was left untouched, because its configuration was not changed.<br />
<br />
The second instance: <i>webapp2</i> can be reached as follows:<br />
<br />
<pre>
$ curl -H 'Host: webapp2.local' http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5001
</body>
</html>
</pre>
<br />
After upgrading the system, the new configuration should also get reactivated after a container restart.<br />
<br />
<h2>A more interesting example: Hydra</h2>
<br />
As explained earlier, to create upgradable containers we require a fully functional Nix installation in a container. This observation made a think about a more interesting example than the trivial web application system.<br />
<br />
A prominent example of a system that requires Nix and is composed out of multiple tightly integrated process is <a href="https://sandervanderburg.blogspot.com/2013/04/setting-up-hydra-build-cluster-for.html">Hydra: the Nix-based continuous integration service</a>.<br />
<br />
To make it possible to deploy a minimal Hydra service in a container, I have packaged all its relevant components for the Nix process management framework.<br />
<br />
The processes model looks as follows:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
nix-processmgmt-services = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt-services.git;
ref = "master";
};
constructors = import "${nix-processmgmt-services}/services-agnostic/constructors.nix" {
inherit nix-processmgmt pkgs stateDir runtimeDir logDir tmpDir cacheDir forceDisableUserChange processManager;
};
instanceSuffix = "";
hydraUser = hydraInstanceName;
hydraInstanceName = "hydra${instanceSuffix}";
hydraQueueRunnerUser = "hydra-queue-runner${instanceSuffix}";
hydraServerUser = "hydra-www${instanceSuffix}";
in
rec {
nix-daemon = {
pkg = constructors.nix-daemon;
};
postgresql = rec {
port = 5432;
postgresqlUsername = "postgresql";
postgresqlPassword = "postgresql";
socketFile = "${runtimeDir}/postgresql/.s.PGSQL.${toString port}";
pkg = constructors.simplePostgresql {
inherit port;
authentication = ''
# TYPE DATABASE USER ADDRESS METHOD
local hydra all ident map=hydra-users
'';
identMap = ''
# MAPNAME SYSTEM-USERNAME PG-USERNAME
hydra-users ${hydraUser} ${hydraUser}
hydra-users ${hydraQueueRunnerUser} ${hydraUser}
hydra-users ${hydraServerUser} ${hydraUser}
hydra-users root ${hydraUser}
# The postgres user is used to create the pg_trgm extension for the hydra database
hydra-users postgresql postgresql
'';
};
};
hydra-server = rec {
port = 3000;
hydraDatabase = hydraInstanceName;
hydraGroup = hydraInstanceName;
baseDir = "${stateDir}/lib/${hydraInstanceName}";
inherit hydraUser instanceSuffix;
pkg = constructors.hydra-server {
postgresqlDBMS = postgresql;
user = hydraServerUser;
inherit nix-daemon port instanceSuffix hydraInstanceName hydraDatabase hydraUser hydraGroup baseDir;
};
};
hydra-evaluator = {
pkg = constructors.hydra-evaluator {
inherit nix-daemon hydra-server;
};
};
hydra-queue-runner = {
pkg = constructors.hydra-queue-runner {
inherit nix-daemon hydra-server;
user = hydraQueueRunnerUser;
};
};
apache = {
pkg = constructors.reverseProxyApache {
dependency = hydra-server;
serverAdmin = "admin@localhost";
};
};
}
</pre>
<br />
In the above processes model, each process instance represents a component of a Hydra installation:<br />
<br />
<ul>
<li>The <i>nix-daemon</i> process is a service that comes with Nix package manager to facilitate multi-user package installations. The <i>nix-daemon</i> carries out builds on behalf of a user.<br />
<br />
Hydra requires it to perform builds as an unprivileged Hydra user and uses the Nix protocol to more efficiently orchestrate large builds.</li>
<li>Hydra uses a PostgreSQL database backend to store data about projects and builds.<br />
<br />
The <i>postgresql</i> process refers to the PostgreSQL database management system (DBMS) that is configured in such a way that the Hydra components are authorized to manage and modify the Hydra database.</li>
<li><i>hydra-server</i> is the front-end of the Hydra service that provides a web user interface. The initialization procedure of this service is responsible for initializing the Hydra database.</li>
<li>The <i>hydra-evaluator</i> regularly updates the repository checkouts and evaluates the Nix expressions to decide which packages need to be built.</li>
<li>The <i>hydra-queue-runner</i> builds all jobs that were evaluated by the <i>hydra-evaluator</i>.</li>
<li>The <i>apache</i> server is used as a reverse proxy server forwarding requests to the <i>hydra-server</i>.</li>
</ul>
<br />
With the following commands, we can build the image, load it into Docker, and deploy a container that runs Hydra:<br />
<br />
<pre>
$ nix-build hydra-image.nix
$ docker load -i result
$ docker run -it --name hydra-test --network host hydra:test
</pre>
<br />
After deploying the system, we can connect to the container:<br />
<br />
<pre>
$ docker exec -it hydra-test /bin/bash
</pre>
<br />
and observe that all processes are running and managed by supervisord:<br />
<br />
<pre>
$ supervisorctl
apache RUNNING pid 1192, uptime 0:00:42
hydra-evaluator RUNNING pid 1297, uptime 0:00:38
hydra-queue-runner RUNNING pid 1296, uptime 0:00:38
hydra-server RUNNING pid 1188, uptime 0:00:42
nix-daemon RUNNING pid 1186, uptime 0:00:42
postgresql RUNNING pid 1187, uptime 0:00:42
supervisor>
</pre>
<br />
With the following commands, we can create our initial admin user:<br />
<br />
<pre>
$ su - hydra
$ hydra-create-user sander --password secret --role admin
creating new user `sander'
</pre>
<br />
We can connect to the Hydra front-end in a web browser by opening <i>http://localhost</i> (this works because the container uses host networking):<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpuhkSMlLOhy1AcbHG7RogIbLKaRJdAqHUgRuQ5aqApYowc62xof6ILfqCVKnjOTTdlOZzmzl-A9uYuq3qaLOnceJYfnFi2q2DpnT8Sp9ToU3HPSoTXsFGsKD_h27khpsx3VAonYpG325E/s1145/hydraoverview.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="813" data-original-width="1145" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgpuhkSMlLOhy1AcbHG7RogIbLKaRJdAqHUgRuQ5aqApYowc62xof6ILfqCVKnjOTTdlOZzmzl-A9uYuq3qaLOnceJYfnFi2q2DpnT8Sp9ToU3HPSoTXsFGsKD_h27khpsx3VAonYpG325E/s600/hydraoverview.png"/></a></div>
<br />
and configure a job set to a build a project, such as <a href="https://sandervanderburg.blogspot.com/2017/01/some-programming-patterns-for-multi.html">libprocreact</a>:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtJ4MIWORvF7UPB-mb3jbWuP0S5ozmMiD_q3-pxsp1wfe1is_kY237U2y2Phcarxxv_-gaXkFy6imFzkXm2u7BUqOWjbWThob29n-ZFGLEFFUNeLCNl3BTfv3jlvPgDVAL7OEElLzonwoy/s1145/hydrajobset.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="813" data-original-width="1145" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgtJ4MIWORvF7UPB-mb3jbWuP0S5ozmMiD_q3-pxsp1wfe1is_kY237U2y2Phcarxxv_-gaXkFy6imFzkXm2u7BUqOWjbWThob29n-ZFGLEFFUNeLCNl3BTfv3jlvPgDVAL7OEElLzonwoy/s600/hydrajobset.png"/></a></div>
<br />
Another nice bonus feature of having multiple process managers supported is that if we build Hydra's <a href="https://sandervanderburg.blogspot.com/2020/06/using-disnix-as-simple-and-minimalistic.html">Nix process management configuration for Disnix</a>, we can also visualize the deployment architecture of the system with <i>disnix-visualize</i>:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPtseJq_wT5JC2j7Hu5eg8Boc4jqubuad3e_oAYnnpq6XPvwWPByAzaKCHmfu7jYE40pIsihuDWibnibM30NXar5BmLAsyOqOsuFBbs63wAW0qj7vBKCIwLyRFq6i2vOTdmfBI9Y523JCl/s0/deploymentarch.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="376" data-original-width="736" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgPtseJq_wT5JC2j7Hu5eg8Boc4jqubuad3e_oAYnnpq6XPvwWPByAzaKCHmfu7jYE40pIsihuDWibnibM30NXar5BmLAsyOqOsuFBbs63wAW0qj7vBKCIwLyRFq6i2vOTdmfBI9Y523JCl/s0/deploymentarch.png"/></a></div>
<br />
The above diagram displays the following properties:<br />
<br />
<ul>
<li>The outer box indicates that we are deploying to a single machine: <i>localhost</i></li>
<li>The inner box indicates that all components are managed as processes</li>
<li>The ovals correspond to process instances in the processes model and the arrows denote dependency relationships.<br />
<br />
For example, the <i>apache</i> reverse proxy has a dependency on <i>hydra-server</i>, meaning that the latter process instance should be deployed first, otherwise the reverse proxy is not able to forward requests to it.</li>
</ul>
<br />
<h2>Building a Nix-enabled container image</h2>
<br />
As explained in the previous section, mutable Docker images require a fully functional Nix package manager in the container.<br />
<br />
Since this may also be an interesting sub use case, I have created a convenience function: <i>createNixImage</i> that can be used to build an image whose only purpose is to provide a working Nix installation:<br />
<br />
<pre style="overflow: auto;">
let
pkgs = import <nixpkgs> {};
nix-processmgmt = builtins.fetchGit {
url = https://github.com/svanderburg/nix-processmgmt.git;
ref = "master";
};
createNixImage = import "${nix-processmgmt}/nixproc/create-image-from-steps/create-nix-image.nix" {
inherit pkgs;
};
in
createNixImage {
name = "foobar";
tag = "test";
contents = [ pkgs.mc ];
}
</pre>
<br />
The above Nix expression builds a Docker image with a working Nix setup and a custom package: the <a href="http://midnight-commander.org">Midnight Commander</a>.<br />
<br />
<h2>Conclusions</h2>
<br />
In this blog post, I have described a new function in the Nix process management framework: <i>createMutableMultiProcessImage</i> that creates reproducible mutable multi-process container images, by combining the reproducibility properties of Docker and Nix. With the exception of the process manager, process instances in a container can be upgraded without bringing the entire container down.<br />
<br />
With this new functionality, the deployment workflow of a multi-process container configuration has become very similar to how physical and virtual machines are managed with <a href="https://sandervanderburg.blogspot.com/2011/01/nixos-purely-functional-linux.html">NixOS</a> -- you can edit a declarative specification of a system and run a single command-line instruction to deploy the new configuration.<br />
<br />
Moreover, this new functionality allows us to deploy a complex, tightly coupled multi-process system, such as Hydra: the Nix-based continuous integration service. In the Hydra example case, we are using Nix for three deployment aspects: constructing the Docker image, deploying the multi-process system configuration and building the projects that are configured in Hydra.<br />
<br />
A big drawback of mutable multi-process images is that there is no sharing possible between multiple multi-process containers. Since the images are not built from common layers, the Nix store is private to each container and all packages are deployed in the writable custom layer, this may lead to substantial disk and RAM overhead per container instance.<br />
<br />
Deploying the processes model to a container instance can probably be made more convenient by using <a href="https://nixos.wiki/wiki/Flakes">Nix flakes</a> -- a new Nix feature that is still experimental. With flakes we can easily deploy an arbitrary number of Nix expressions to a container and pin the deployment to a specific version of Nixpkgs.<br />
<br />
Another interesting observation is the word: mutable. I am not completely sure if it is appropriate -- both the layers of a Docker image, as well as the Nix store paths are immutable and never change after they have been built. For both solutions, immutability is an important ingredient in making sure that a deployment is reproducible.<br />
<br />
I have decided to still call these deployments mutable, because I am looking at the problem from a Docker perspective -- the writable layer of the container (that is mounted on top of the immutable layers of an image) is modified each time that we upgrade a system.<br />
<br />
<h2>Future work</h2>
<br />
Although I am quite happy with the ability to create mutable multi-process containers, there is still quite a bit of work that needs to be done to make the Nix process management framework more usable.<br />
<br />
Most importantly, trying to deploy Hydra revealed all kinds of regressions in the framework. To cope with all these breaking changes, a structured testing approach is required. Currently, such an approach is completely absent.<br />
<br />
I could also (in theory) automate the still missing parts of Hydra. For example, I have not automated the process that updates the garbage collector roots, which needs to run in a timely manner. To solve this, I need to use a <i>cron</i> service or systemd timer units, which is beyond the scope of my experiment.<br />
<br />
<h2>Availability</h2>
<br />
The <i>createMutableMultiProcessImage</i> function is part of the experimental <a href="https://github.com/svanderburg/nix-processmgmt">Nix process management framework GitHub repository</a> that is still under heavy development.<br />
<br />
Because the amount of services that can be deployed with the framework has grown considerably, I have moved all non-essential services (not required for testing) into a <a href="https://github.com/svanderburg/nix-processmgmt-services">separate repository</a>. The Hydra constructor functions can be found in this repository as well.<br />Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-39705574089553686212021-02-01T22:29:00.001+01:002021-02-24T11:44:51.566+01:00Developing an s6-rc backend for the Nix process management frameworkOne of my major blog topics last year was <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html">my experimental Nix process management framework</a>, that is still under heavy development.<br />
<br />
As explained in many of my earlier blog posts, one of its major objectives is to facilitate <strong>high-level deployment specifications</strong> of running processes that can be translated to configurations for all kinds of process managers and deployment solutions.<br />
<br />
The backends that I have implemented so far, were picked for the following reasons:<br />
<br />
<ul>
<li><strong>Multiple operating systems support</strong>. The most common process management service was chosen for each operating system: On Linux, <i>sysvinit</i> (because this used to be the most common solution) and <i>systemd</i> (because it is used by many conventional Linux distributions today), <i>bsdrc</i> on FreeBSD, <i>launchd</i> for macOS, and <i>cygrunsrv</i> for Cygwin.</li>
<li><strong>Supporting unprivileged user deployments</strong>. To supervise processes without requiring a service that runs on PID 1, that also works for unprivileged users, <i>supervisord</i> is very convenient because it was specifically designed for this purpose.</li>
<li><a href="https://sandervanderburg.blogspot.com/2020/08/experimenting-with-nix-and-service.html"><strong>Docker</strong> was selected</a> because it is a very popular solution for managing services, and process management is one of its sub responsibilities.</li>
<li><strong>Universal process management</strong>. <a href="https://sandervanderburg.blogspot.com/2020/06/using-disnix-as-simple-and-minimalistic.html">Disnix was selected</a> because it can be used as a primitive process management solution that works on any operating system supported by the Nix package manager. Moreover, the Disnix services model is a super set of the processes model used by the process management framework.</li>
</ul>
<br />
Not long after writing my blog post about the process manager-agnostic abstraction layer, somebody opened <a href="https://github.com/svanderburg/nix-processmgmt/issues/1">an issue on GitHub</a> with the suggestion to also support <i>s6-rc</i>. Although I was already aware that more process/service management solutions exist, <i>s6-rc</i> was a solution that I did not know about.<br />
<br />
Recently, I have implemented the suggested <i>s6-rc</i> backend. Although deploying <i>s6-rc</i> services now works quite conveniently, getting to know <i>s6-rc</i> and its companion tools was somewhat challenging for me.<br />
<br />
In this blog post, I will elaborate about my learning experiences and explain how the <i>s6-rc</i> backend was implemented.<br />
<br />
<h2>The s6 tool suite</h2>
<br />
<a href="https://skarnet.org/software/s6-rc"><i>s6-rc</i></a> is a software projected published on <a href="https://skarnet.org">skarnet</a> and part of a bigger <a href="https://skarnet.org/software">tool ecosystem</a>. <i>s6-rc</i> is a companion tool of <a href="https://skarnet.org/software/s6">s6</a>: skarnet.org's small & secure supervision software suite.<br />
<br />
On Linux and many other UNIX-like systems, the initialization process (typically <i>/sbin/init</i>) is a <strong>highly critical</strong> program:<br />
<br />
<ul>
<li>It is the first program loaded by the kernel and responsible for setting the remainder of the boot procedure in motion. This procedure is responsible for mounting additional file systems, loading device drivers, and starting essential system services, such as SSH and logging services.</li>
<li>The PID 1 process supervises all processes that were directly loaded by it, as well as indirect child processes that get orphaned -- when this happens they get automatically adopted by the process that runs as PID 1.<br />
<br />
As explained in <a href="https://sandervanderburg.blogspot.com/2020/01/writing-well-behaving-daemon-in-c.html">an earlier blog post</a>, traditional UNIX services that daemonize on their own, deliberately orphan themselves so that they remain running in the background.</li>
<li>When a child process terminates, the parent process must take notice or the terminated process will stay behind as a zombie process.<br />
<br />
Because the PID 1 process is the common ancestor of all other processes, it is required to automatically reap all relevant zombie processes that become a child of it.</li>
<li>The PID 1 process runs with root privileges and, as a result, has full access to the system. When the security of the PID 1 process gets compromised, the entire system is at risk.</li>
<li>If the PID 1 process crashes, the kernel crashes (and hence the entire system) with a kernel panic.</li>
</ul>
<br />
There are many kinds of programs that you can use as a system's PID 1. For example, you can directly use a shell, such as <i>bash</i>, but is far more common to use an init system, such as <a href="https://savannah.nongnu.org/projects/sysvinit"><i>sysvinit</i></a> or <a href="https://www.freedesktop.org/wiki/Software/systemd"><i>systemd</i></a>.<br />
<br />
According to the author of <i>s6</i>, <a href="https://skarnet.org/software/s6-linux-init/why.html">an init system is made out of four parts</a>:<br />
<br />
<blockquote>
<ol>
<li><i>/sbin/init</i>: the first userspace program that is run by the kernel at boot time (not counting an initramfs).</li>
<li>pid 1: the program that will run as process 1 for most of the lifetime of the machine. This is not necessarily the same executable as <i>/sbin/init</i>, because <i>/sbin/init</i> can exec into something else.</li>
<li>a process supervisor.</li>
<li>a service manager.</li>
</ol>
</blockquote>
<br />
In the <i>s6</i> tool eco-system, most of these parts are implemented by separate tools:<br />
<br />
<ul>
<li>The first userspace program: <i>s6-linux-init</i> takes care of the coordination of the initialization process. It does a variety of one-time boot things: for example, it traps the ctrl-alt-del keyboard combination, it starts the shutdown daemon (that is responsible for eventually shutting down the system), and runs the initial boot script (<i>rc.init</i>).<br />
<br />
(As a sidenote: this is almost true -- the <i>/sbin/init</i> process is a wrapper script that "execs" into <i>s6-linux-linux-init</i> with the appropriate parameters).</li>
<li>When the initialization is done, <i>s6-linux-init</i> execs into a process called <i>s6-svscan</i> provided by the <i>s6</i> toolset. <i>s6-svscan</i>'s task is to supervise an entire process supervision tree, which I will explain later.</li>
<li>Starting and stopping services is done by a separate service manager started from the <i>rc.init</i> script. <i>s6-rc</i> is the most prominent option (that we will use in this blog post), but also other tools can be used.</li>
</ul>
<br />
Many conventional init systems, implement most (or sometimes all) of these aspects in a single executable.<br />
<br />
In particular, the <i>s6</i> author is highly critical of systemd: the init system that is widely used by many conventional Linux distributions today -- he dedicated <a href="https://skarnet.org/software/systemd.html">an entire page with criticisms about it</a>.<br />
<br />
The author of <i>s6</i> advocates a number of design principles for his tool eco-system (that systemd violates in many ways):<br />
<br />
<ul>
<li>The Unix philosophy: do one job and do it well.</li>
<li>Doing less instead of more (preventing feature creep).</li>
<li>Keeping tight quality control over every tool by only opening up repository access to small teams only (or rather a single person).</li>
<li>Integration support: he is against the <a href="http://www.catb.org/esr/writings/cathedral-bazaar">bazaar</a> approach on project level, but in favor of the bazaar approach on an eco-system level in which everybody can write their own tools that integrate with existing tools.</li>
</ul>
<br />
The concepts implemented by the <i>s6</i> tool suite were not completely "invented" from scratch. <a href="http://cr.yp.to/daemontools.html">daemontools</a> is what the author considers the ancestor of s6 (if you look at the web page then you will notice that the concept of a "supervision tree" was pioneered there and that some of the tools listed resemble the same tools in the <i>s6</i> tool suite), and <a href="http://smarden.org/runit">runit</a> its cousin (that is also heavily inspired by daemontools).<br />
<br />
<h2>A basic usage scenario of s6 and s6-rc</h2>
<br />
Although it is possible to use Linux distributions in which the init system, supervisor and service manager are all provided by skarnet tools, a sub set of <i>s6</i> and <i>s6-rc</i> can also be used on any Linux distribution and other supported operating systems, such as the BSDs.<br />
<br />
Root privileges are not required to experiment with these tools.<br />
<br />
For example, with the following command we can use the Nix package manager to deploy the <i>s6</i> supervision toolset in a development shell session:<br />
<br />
<pre>
$ nix-shell -p s6
</pre>
<br />
In this development shell session, we can start the <i>s6-svscan</i> service as follows:<br />
<br />
<pre>
$ mkdir -p $HOME/var/run/service
$ s6-svscan $HOME/var/run/service
</pre>
<br />
The <i>s6-svscan</i> is a service that supervises an entire process supervision tree, including processes that may accidentally become a child of it, such as orphaned processes.<br />
<br />
The directory parameter is a <strong>scan directory</strong> that maintains the configurations of the processes that are currently supervised. So far, no supervised process have been deployed yet.<br />
<br />
We can actually deploy services by using the <i>s6-rc</i> toolset.<br />
<br />
For example, I can easily configure <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html">my trivial example system</a> used in previous blog posts that consists of one or multiple web application processes (with an embedded HTTP server) returning static HTML pages and an Nginx reverse proxy that forwards requests to one of the web application processes based on the appropriate virtual host header.<br />
<br />
Contrary to the other process management solutions that I have investigated earlier, <i>s6-rc</i> does not have an elaborate configuration language. It does not implement a parser (<a href="https://skarnet.org/software/s6-rc/faq.html">for very good reasons as explained by the author</a>, because it introduces extra complexity and bugs).<br />
<br />
Instead, you have to create directories with text files, in which each file represents a configuration property.<br />
<br />
With the following command, I can spawn a development shell with all the required utilities to work with <i>s6-rc</i>:<br />
<br />
<pre>
$ nix-shell -p s6 s6-rc execline
</pre>
<br />
The following shell commands create an <i>s6-rc</i> service configuration directory and a configuration for a single <i>webapp</i> process instance:<br />
<br />
<pre>
$ mkdir -p sv/webapp
$ cd sv/webapp
$ echo "longrun" > type
$ cat > run <<EOF
$ #!$(type -p execlineb) -P
envfile $HOME/envfile
exec $HOME/webapp/bin/webapp
EOF
</pre>
<br />
The above shell script creates a configuration directory for a service named: <i>webapp</i> with the following properties:<br />
<br />
<ul>
<li>It creates a service with <strong>type</strong>: <i>longrun</i>. A long run service deploys a process that runs in the foreground that will get supervised by <i>s6</i>.</li>
<li>The <i>run</i> file refers to an <strong>executable</strong> that <strong>starts</strong> the service. For <i>s6-rc</i> services it is common practice to implement wrapper scripts using <a href="https://skarnet.org/software/execline/"><i>execline</i></a>: a non-interactive scripting language.<br />
<br />
The execline script shown above loads an environment variable config file with the following content: <i>PORT=5000</i>. This environment variable is used to configure the TCP port number to which the service should bind to and then "execs" into a new process that runs the <i>webapp</i> process.<br />
<br />
(As a sidenote: although it is a common habit to use <i>execline</i> for writing wrapper scripts, this is not a hard requirement -- any executable implemented in any language can be used. For example, we could also write the above <i>run</i> wrapper script as a bash script).</li>
</ul>
<br />
We can also configure the Nginx reverse proxy service in a similar way:<br />
<br />
<pre style="overflow: auto;">
$ mkdir -p ../nginx
$ cd ../nginx
$ echo "longrun" > type
$ echo "webapp" > dependencies
$ cat > run <<EOF
$ #!$(type -p execlineb) -P
foreground { mkdir -p $HOME/var/nginx/logs $HOME/var/cache/nginx }
exec $(type -p nginx) "-p" "$HOME/var/nginx" "-c" "$HOME/nginx/nginx.conf" "-g" "daemon off;"
EOF
</pre>
<br />
The above shell script creates a configuration directory for a service named: <i>nginx</i> with the following properties:<br />
<br />
<ul>
<li>It again creates a service of <strong>type</strong>: <i>longrun</i> because Nginx should be started as a foreground process.</li>
<li>It declares the <i>webapp</i> service (that we have configured earlier) a <strong>dependency</strong> ensuring that <i>webapp</i> is started before <i>nginx</i>. This dependency relationship is important to prevent Nginx doing a redirect to a non-existent service.</li>
<li>The <i>run</i> script first creates all mandatory state directories and finally execs into the Nginx process, with a configuration file using the above state directories, and turning off daemon mode so that it runs in the foreground.</li>
</ul>
<br />
In addition to configuring the above services, we also want to deploy the system as a whole. This can be done by creating <strong>bundles</strong> that encapsulate collections of services:<br />
<br />
<pre>
mkdir -p ../default
cd ../default
echo "bundle" > type
cat > contents <<EOF
webapp
nginx
EOF
</pre>
<br />
The above shell instructions create a bundle named: <strong>default</strong> referring to both the <i>webapp</i> and <i>nginx</i> reverse proxy service that we have configured earlier.<br />
<br />
Our <i>s6-rc</i> configuration directory structure looks as follows:<br />
<br />
<pre>
$ find ./sv
./sv
./sv/default
./sv/default/contents
./sv/default/type
./sv/nginx/run
./sv/nginx/type
./sv/webapp/dependencies
./sv/webapp/run
./sv/webapp/type
</pre>
<br />
If we want to deploy the service directory structure shown above, we first need to <strong>compile</strong> it into a <strong>configuration database</strong>. This can be done with the following command:<br />
<br />
<pre>
$ mkdir -p $HOME/etc/s6/rc
$ s6-rc-compile $HOME/etc/s6/rc/compiled-1 $HOME/sv
</pre>
<br />
The above command creates a compiled database file in: <i>$HOME/etc/s6/rc/compiled-1</i> stored in: <i>$HOME/sv</i>.<br />
<br />
With the following command we can <strong>initialize</strong> the <i>s6-rc</i> system with our compiled configuration database:<br />
<br />
<pre>
$ s6-rc-init -c $HOME/etc/s6/rc/compiled-1 -l $HOME/var/run/s6-rc \
$HOME/var/run/service
</pre>
<br />
The above command generates a "live directory" in: <i>$HOME/var/run/s6-rc</i> containing the state of <i>s6-rc</i>.<br />
<br />
With the following command, we can start all services in the: <i>default</i> bundle:<br />
<br />
<pre>
$ s6-rc -l $HOME/var/run/s6-rc -u change default
</pre>
<br />
The above command deploys a running system with the following process tree:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbsi4UW5ur5iZsFUat6v0nhC4HA5FQVk-evrugeJ74i9U3wASJ_kdsTBLAFTVmWO6fvU4hT2zdnSCGbGGMk676MrIrmWGdYBuNWzbQywBAb4nDqjhg19WbMLfNIb_A_H2pe4V0QuJ-dgmC/s0/processtree.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="258" data-original-width="302" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhbsi4UW5ur5iZsFUat6v0nhC4HA5FQVk-evrugeJ74i9U3wASJ_kdsTBLAFTVmWO6fvU4hT2zdnSCGbGGMk676MrIrmWGdYBuNWzbQywBAb4nDqjhg19WbMLfNIb_A_H2pe4V0QuJ-dgmC/s0/processtree.png"/></a></div>
<br />
As as can be seen in the diagram above, the entire process tree is supervised by <i>s6-svscan</i> (the program that we have started first). Every <i>longrun</i> service deployed by <i>s6-rc</i> is supervised by a process named: <i>s6-supervise</i>.<br />
<br />
<h2>Managing service logging</h2>
<br />
Another important property of <i>s6</i> and <i>s6-rc</i> is the way it handles logging. By default, all output that the supervised processes produce on the standard output and standard error are captured by <i>s6-svscan</i> and written to a single log stream (in our case, it will be redirected to the terminal).<br />
<br />
When it is desired to capture the output of a service into its own dedicated log file, you need to configure the service in such a way that it writes all relevant information to a pipe. A companion <strong>logging service</strong> is required to capture the data that is sent over the pipe.<br />
<br />
The following command-line instructions modify the <i>webapp</i> service (that we have created earlier) to let it send its output to another service:<br />
<br />
<pre>
$ cd sv
$ mv webapp webapp-srv
$ cd webapp-srv
$ echo "webapp-log" > producer-for
$ cat > run <<EOF
$ #!$(type -p execlineb) -P
envfile $HOME/envfile
fdmove -c 2 1
exec $HOME/webapp/bin/webapp
EOF
</pre>
<br />
In the script above, we have changed the <i>webapp</i> service configuration as follows:<br />
<br />
<ul>
<li>We <strong>rename</strong> the service from: <i>webapp</i> to <i>webapp-srv</i>. Using suffixes is a convention commonly used for <i>s6-rc</i> services that also have a log companion service.</li>
<li>With the <i>producer-for</i> property, we specify that the <i>webapp-srv</i> is a service that <strong>produces</strong> output for another service named: <i>webapp-log</i>. We will configure this service later.</li>
<li>We create a new <i>run</i> script that <strong>adds</strong> the following command: <i>fdmove -c 2 1</i>.<br />
<br />
The purpose of this added instruction is to redirect all output that is sent over the standard error (file descriptor: 2) to the standard output (file descriptor: 1). This redirection makes it possible that all data can be captured by the log companion service.</li>
</ul>
<br />
We can configure the log companion service: <i>webapp-log</i> with the following command-line instructions:<br />
<br />
<pre>
$ mkdir ../webapp-log
$ cd ../webapp-log
$ echo "longrun" > type
$ echo "webapp-srv" > consumer-for
$ echo "webapp" > pipeline-name
$ echo 3 > notification-fd
$ cat > run <<EOF
#!$(type -p execlineb) -P
foreground { mkdir -p $HOME/var/log/s6-log/webapp }
exec -c s6-log -d3 $HOME/var/log/s6-log/webapp
EOF
</pre>
<br />
The service configuration created above does the following:<br />
<br />
<ul>
<li>We create a service named: <i>webapp-log</i> that is a <strong>long running</strong> service.</li>
<li>We declare the service to be a <strong>consumer</strong> for the <i>webapp-srv</i> (earlier, we have already declared the companion service: <i>webapp-srv</i> to be a producer for this logging service).</li>
<li>We configure a <strong>pipeline name</strong>: <i>webapp</i> causing <i>s6-rc</i> to automatically generate a bundle with the name: <i>webapp</i> in which all involved services are its contents.<br />
<br />
This generated bundle allows us to always manage the service and logging companion as a single deployment unit.</li>
<li>The <i>s6-log</i> service supports <strong>readiness notifications</strong>. File descriptor: <i>3</i> is configured to receive that notification.</li>
<li>The <i>run</i> script creates the log directory in which the output should be stored and starts the <i>s6-log</i> service to capture the output and store the data in the corresponding log directory.<br />
<br />
The <i>-d3</i> parameter instructs it to send a readiness notification over file descriptor 3.</li>
</ul>
<br />
After modifying the configuration files in such a way that each <i>longrun</i> service has a logging companion, we need to compile a new database that provides <i>s6-rc</i> our new configuration:<br />
<br />
<pre>
$ s6-rc-compile $HOME/etc/s6/rc/compiled-2 $HOME/sv
</pre>
<br />
The above command creates a database with a new filename in: <i>$HOME/etc/s6/rc/compiled-2</i>. We are required to give it a new name -- the old configuration database (<i>compiled-1</i>) must be retained to make the upgrade process work.<br />
<br />
With the following command, we can upgrade our running configuration:<br />
<br />
<pre>
$ s6-rc-update -l $HOME/var/run/s6-rc $HOME/etc/s6/rc/compiled-2
</pre>
<br />
The result is the following process supervision tree:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi28k1GST-YA4GCCb2Za-N2obS-gVvTAdx-VV7FEnztnXecrKlSPJl5yUqkU4H-cvJkJYMMB8tePImIQDb43IEi1Ak8Iusk9nauAsro3mIOSxUgbEX1uUOAmyf-I9dgf4vrE4EnzHBWfbUL/s0/processtreewithloggers.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="271" data-original-width="536" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi28k1GST-YA4GCCb2Za-N2obS-gVvTAdx-VV7FEnztnXecrKlSPJl5yUqkU4H-cvJkJYMMB8tePImIQDb43IEi1Ak8Iusk9nauAsro3mIOSxUgbEX1uUOAmyf-I9dgf4vrE4EnzHBWfbUL/s0/processtreewithloggers.png"/></a></div>
<br />
As you may observe by looking at the diagram above, every service has a companion <i>s6-log</i> service that is responsible for capturing and storing its output.<br />
<br />
The log files of the services can be found in <i>$HOME/var/log/s6-log/webapp</i> and <i>$HOME/var/log/s6-log/nginx</i>.<br />
<br />
<h2>One shot services</h2>
<br />
In addition to <i>longrun</i> services that are useful for managing system services, more aspects need to be automated in a boot process, such as mounting file systems.<br />
<br />
These kinds of tasks can be automated with <i>oneshot</i> services, that execute an <i>up</i> script on startup, and optionally, a <i>down</i> script on shutdown.<br />
<br />
The following service configuration can be used to mount the kernel's <i>/proc</i> filesystem:<br />
<br />
<pre>
mkdir -p ../mount-proc
cd ../mount-proc
echo "oneshot" > type
cat > run <<EOF
$ #!$(type -p execlineb) -P
foreground { mount -t proc proc /proc }
EOF
</pre>
<br />
<h2>Chain loading</h2>
<br />
The <i>execline</i> scripts shown in this blog post resemble shell scripts in many ways. One particular aspect that sets execline scripts apart from shell scripts is that all commands make intensive use of a concept called <strong><a href="https://en.wikipedia.org/wiki/Chain_loading">chain loading</a></strong>.<br />
<br />
Every instruction in an execline script executes a task, may imperatively modify the environment (e.g. by changing environment variables, or changing the current working directory etc.) and then "execs" into a new chain loading task.<br />
<br />
The last parameter of each command-line instruction refers to the command-line instruction that it needs to "execs into" -- typically this command-line instruction is put on the next line.<br />
<br />
The <i>execline</i> package, as well as many packages in the <i>s6</i> ecosystem, contain many programs that support chain loading.<br />
<br />
It is also possible to implement custom chain loaders that follow the same protocol.<br />
<br />
<h2>Developing s6-rc function abstractions for the Nix process management framework</h2>
<br />
In the Nix process management framework, I have added function abstractions for each <i>s6-rc</i> service type: <i>longrun</i>, <i>oneshot</i> and <i>bundle</i>.<br />
<br />
For example, with the following Nix expression we can generate an <i>s6-rc</i> <i>longrun</i> configuration for the <i>webapp</i> process:<br />
<br />
<pre>
{createLongRunService, writeTextFile, execline, webapp}:
let
envFile = writeTextFile {
name = "envfile";
text = ''
PORT=5000
'';
};
in
createLongRunService {
name = "webapp";
run = writeTextFile {
name = "run";
executable = true;
text = ''
#!${execline}/bin/execlineb -P
envfile ${envFile}
fdmove -c 2 1
exec ${webapp}/bin/webapp
'';
};
autoGenerateLogService = true;
}
</pre>
<br />
Evaluating the Nix expression above does the following:<br />
<br />
<ul>
<li>It generates a service directory that corresponds to the: <i>name</i> parameter with a <i>longrun</i> <i>type</i> property file.</li>
<li>It generates a <i>run</i> execline script, that uses a generated <i>envFile</i> for configuring the service's port number, redirects the standard error to the standard output and starts the <i>webapp</i> process (that runs in the foreground).</li>
<li>The <i>autoGenerateLogService</i> parameter is a concept I introduced myself, to conveniently configure a companion log service, because this a very common operation -- I cannot think of any scenario in which you do not want to have a dedicated log file for a long running service.<br />
<br />
Enabling this option causes the service to automatically become a producer for the log companion service (having the same name with a <i>-log</i> suffix) and automatically configures a logging companion service that consumes from it.</li>
</ul>
<br />
In addition to constructing long run services from Nix expressions, there are also abstraction functions to create one shots: <i>createOneShotService</i> and bundles: <i>createServiceBundle</i>.<br />
<br />
The function that generates a log companion service can also be directly invoked with: <i>createLogServiceForLongRunService</i>, if desired.<br />
<br />
<h2>Generating a s6-rc service configuration from a process-manager agnostic configuration</h2>
<br />
The following Nix expression is a process manager-agnostic configuration for the <i>webapp</i> service, that can be translated to a configuration for any supported process manager in the Nix process management framework:<br />
<br />
<pre>
{createManagedProcess, tmpDir}:
{port, instanceSuffix ? "", instanceName ? "webapp${instanceSuffix}"}:
let
webapp = import ../../webapp;
in
createManagedProcess {
name = instanceName;
description = "Simple web application";
inherit instanceName;
process = "${webapp}/bin/webapp";
daemonArgs = [ "-D" ];
environment = {
PORT = port;
};
overrides = {
sysvinit = {
runlevels = [ 3 4 5 ];
};
};
}
</pre>
<br />
The Nix expression above specifies the following high-level configuration concepts:<br />
<br />
<ul>
<li>The <i>name</i> and <i>description</i> attributes are just meta data. The <i>description</i> property is ignored by the <i>s6-rc</i> generator, because <i>s6-rc</i> has no equivalent configuration property for capturing a description.</li>
<li>A process manager-agnostic configuration can specify both how the service can be started as a <strong>foreground process</strong> or as a process that <strong>daemonizes</strong> itself.<br />
<br />
In the above example, the <i>process</i> attribute specifies that the same executable needs to invoked for both a <i>foregroundProcess</i> and <i>daemon</i>. The <i>daemonArgs</i> parameter specifies the command-line arguments that need to be propagated to the executable to let it daemonize itself.<br />
<br />
<i>s6-rc</i> has a preference for managing foreground processes, because these can be more reliably managed. When a <i>foregroundProcess</i> executable can be inferred, the generator will automatically compose a <i>longrun</i> service making it possible for <i>s6</i> to supervise it.<br />
<br />
If only a <i>daemon</i> can be inferred, the generator will compose a <i>oneshot</i> service that starts the daemon with the <i>up</i> script, and on shutdown, terminates the daemon by dereferencing the PID file in the <i>down</i> script.
</li>
<li>The <i>environment</i> attribute set parameter is automatically translated to an <i>envfile</i> that the generated <i>run</i> script consumes.</li>
<li>Similar to the <i>sysvinit</i> backend, it is also possible to override the generated arguments for the <i>s6-rc</i> backend, if desired.</li>
</ul>
<br />
As already explained in the blog post that covers the framework's concepts, the Nix expression above needs to be complemented with a <strong>constructors</strong> expression that composes the common parameters of every process configuration and a <strong>processes</strong> model that constructs process instances that need to be deployed.<br />
<br />
The following processes model can be used to deploy a <i>webapp</i> process and an <i>nginx</i> reverse proxy instance that connects to it:<br />
<br />
<pre>
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir;
inherit forceDisableUserChange processManager;
};
in
rec {
webapp = rec {
port = 5000;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
};
nginx = rec {
port = 8080;
pkg = constructors.nginxReverseProxyHostBased {
webapps = [ webapp ];
inherit port;
} {};
};
}
</pre>
<br />
With the following command-line instruction, we can automatically create a scan directory and start <i>s6-svscan</i>:<br />
<br />
<pre>
$ nixproc-s6-svscan --state-dir $HOME/var
</pre>
<br />
The <i>--state-dir</i> causes the scan directory to be created in the user's home directory making unprivileged deployments possible.<br />
<br />
With the following command, we can deploy the entire system, that will get supervised by the <i>s6-svscan</i> service that we just started:<br />
<br />
<pre>
$ nixproc-s6-rc-switch --state-dir $HOME/var \
--force-disable-user-change processes.nix
</pre>
<br />
The <i>--force-disable-user-change</i> parameter prevents the deployment system from creating users and groups and changing user privileges, allowing the deployment as an unprivileged user to succeed.<br />
<br />
The result is a running system that allows us to connect to the <i>webapp</i> service via the Nginx reverse proxy:<br />
<br />
<pre>
$ curl -H 'Host: webapp.local' http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5000
</body>
</html>
</pre>
<br />
<h2>Constructing multi-process Docker images supervised by s6</h2>
<br />
Another feature of the Nix process management framework is constructing <strong>multi-process Docker images</strong> in which multiple process instances are supervised by a process manager of choice.<br />
<br />
<i>s6</i> can also be used as a supervisor in a container. To accomplish this, we can use <i>s6-linux-init</i> as an entry point.<br />
<br />
The following attribute generates a skeleton configuration directory:<br />
<br />
<pre>
let
skelDir = pkgs.stdenv.mkDerivation {
name = "s6-skel-dir";
buildCommand = ''
mkdir -p $out
cd $out
cat > rc.init <<EOF
#! ${pkgs.stdenv.shell} -e
rl="\$1"
shift
# Stage 1
s6-rc-init -c /etc/s6/rc/compiled /run/service
# Stage 2
s6-rc -v2 -up change default
EOF
chmod 755 rc.init
cat > rc.shutdown <<EOF
#! ${pkgs.stdenv.shell} -e
exec s6-rc -v2 -bDa change
EOF
chmod 755 rc.shutdown
cat > rc.shutdown.final <<EOF
#! ${pkgs.stdenv.shell} -e
# Empty
EOF
chmod 755 rc.shutdown.final
'';
};
</pre>
<br />
The skeleton directory generated by the above sub expression contains three configuration files:<br />
<br />
<ul>
<li><i>rc.init</i> is the script that the init system starts, right after starting the supervisor: <i>s6-svscan</i>. It is responsible for initializing the <i>s6-rc</i> system and starting all services in the <i>default</i> bundle.</li>
<li><i>rc.shutdown</i> script is executed on shutdown and stops all previously started services by <i>s6-rc</i>.</li>
<li><i>rc.shutdown.final</i> runs at the very end of the shutdown procedure, after all processes have been killed and all file systems have been unmounted. In the above expression, it does nothing.</li>
</ul>
<br />
In the initialization process of the image (the <i>runAsRoot</i> parameter of <i>dockerTools.buildImage</i>), we need to execute a number of dynamic initialization steps.<br />
<br />
First, we must initialize <i>s6-linux-init</i> to read its configuration files from <i>/etc/s6/current</i> using the skeleton directory (that we have configured in the sub expression shown earlier) as its initial contents (the <i>-f</i> parameter) and run the init system in container mode (the <i>-C</i> parameter):<br />
<br />
<pre style="overflow: auto;">
mkdir -p /etc/s6
s6-linux-init-maker -c /etc/s6/current -p /bin -m 0022 -f ${skelDir} -N -C -B /etc/s6/current
mv /etc/s6/current/bin/* /bin
rmdir etc/s6/current/bin
</pre>
<br />
<i>s6-linux-init-maker</i> generates an <i>/bin/init</i> script, that we can use as the container's entry point.<br />
<br />
I want the logging services to run as an unprivileged user (<i>s6-log</i>) requiring me to create the user and corresponding group first:<br />
<br />
<pre>
groupadd -g 2 s6-log
useradd -u 2 -d /dev/null -g s6-log s6-log
</pre>
<br />
We must also compile a database from the <i>s6-rc</i> configuration files, by running the following command-line instructions:<br />
<br />
<pre>
mkdir -p /etc/s6/rc
s6-rc-compile /etc/s6/rc/compiled ${profile}/etc/s6/sv
</pre>
<br />
As can be seen in the <i>rc.init</i> script that we have generated earlier, the compiled database: <i>/etc/s6/rc/compiled</i> is propagated to <i>s6-rc-init</i> as a command-line parameter.<br />
<br />
With the following Nix expression, we can build an <i>s6-rc</i> managed multi-process Docker image that deploys all the process instances in the processes model that we have written earlier:<br />
<br />
<pre style="overflow: auto;">
let
pkgs = import <nixpkgs> {};
createMultiProcessImage = import ../../nixproc/create-multi-process-image/create-multi-process-image-universal.nix {
inherit pkgs system;
inherit (pkgs) dockerTools stdenv;
};
in
createMultiProcessImage {
name = "multiprocess";
tag = "test";
exprFile = ./processes.nix;
stateDir = "/var";
processManager = "s6-rc";
}
</pre>
<br />
With the following command, we can build the image:<br />
<br />
<pre>
$ nix-build
</pre>
<br />
and load the image into Docker with the following command:<br />
<br />
<pre>
$ docker load -i result
</pre>
<br />
<h2>Discussion</h2>
<br />
With the addition of the <i>s6-rc</i> backend in the Nix process management framework, we have a modern alternative to systemd at our disposal.<br />
<br />
We can easily let services be managed by <i>s6-rc</i> using the same agnostic high-level deployment configurations that can also be used to target other process management backends, including systemd.<br />
<br />
What I particularly like about the <i>s6</i> tool ecosystem (and this also applies in some extent to its ancestor: <i>daemontools</i> and cousin project: <i>runit</i>) is the idea to construct the entire system's initialization process and its sub concerns (process supervision, logging and service management) from separate tools, each having clear/fixed scopes.<br />
<br />
This kind of design reminds me of <a href="https://en.wikipedia.org/wiki/Microkernel">microkernels</a> -- in a microkernel design, the kernel is basically split into multiple collaborating processes each having their own responsibilities (e.g. file systems, drivers).<br />
<br />
The microkernel is the only process that has full access to the system and typically only has very few responsibilities (e.g. memory management, task scheduling, interrupt handling).<br />
<br />
When a process crashes, such as a driver, this failure should not tear the entire system down. Systems can even recover from problems, by restarting crashed processes.<br />
<br />
Furthermore, these non-kernel processes typically have very few privileges. If a process' security gets compromised (such as a leaky driver), the system as a whole will not be affected.<br />
<br />
Aside from a number of functional differences compared to systemd, there are also some non-functional differences as well.<br />
<br />
systemd can only be used on Linux using glibc as the system's libc, <i>s6</i> can also be used on different operating systems (e.g. the BSDs) with different libc implementations, such as <a href="https://musl.libc.org/">musl</a>.<br />
<br />
Moreover, the supervisor service (<i>s6-svscan</i>) <a href="https://skarnet.org/software/s6/s6-svscan-not-1.html">can also be used as a user-level supervisor that does not need to run as PID 1</a>. Although systemd supports user sessions (allowing service deployments from unprivileged users), it still has the requirement to have systemd as an init system that needs to run as the system's PID 1.<br />
<br />
<h2>Improvement suggestions</h2>
<br />
Although the <i>s6</i> ecosystem provides useful tools and has all kinds of powerful features, I also have a number of improvement suggestions. They are mostly usability related:<br />
<br />
<ul>
<li>I have noticed that the command-line tools have very <strong>brief help pages</strong> -- they only enumerate the available options, but they do not provide any additional information explaining what these options do.<br />
<br />
I have also noticed that there are no official manpages, but there is a <a href="https://github.com/flexibeast/s6-man-pages">third-party initiative</a> that seems to provide them.<br />
<br />
The "official" source of reference are the HTML pages. For me personally, it is not always convenient to access HTML pages on limited machines with no Internet connection and/or only terminal access.</li>
<li>Although each individual tool is well documented (albeit in HTML), I was having quite a few difficulties figuring out <strong>how to use them together</strong> -- because every tool has a very specific purpose, you typically need to combine them in interesting ways to do something meaningful.<br />
<br />
For example, I could not find any clear documentation on skarnet describing typical combined usage scenarios, such as how to use <i>s6-rc</i> on a conventional Linux distribution that already has a different service management solution.<br />
<br />
Fortunately, I discovered a Linux distribution that turned out to be immensely helpful: <a href="https://artixlinux.org">Artix Linux</a>. Artix Linux provides <i>s6</i> as one of its supported process management solutions. I ended up installing Artix Linux in a virtual machine and reading <a href="https://wiki.artixlinux.org/Main/S6">their documentation</a>.<br />
<br />
This kind of unclarity seems to be somewhat analogous to common criticisms of microkernels: <a href="https://yarchive.net/comp/microkernels.html">one of Linus Torvalds' criticisms</a> is that in microkernel designs, the pieces are simplified, but the coordination of the entire system is more difficult.</li>
<li><strong>Updating</strong> existing service configurations is <strong>difficult</strong> and <strong>cumbersome</strong>. Each time I want to change something (e.g. adding a new service), then I need to compile a new database, make sure that the newly compiled database co-exists with the previous database, and then run <i>s6-rc-update</i>.<br />
<br />
It is very easy to make mistakes. For example, I ended up overwriting the previous database several times. When this happens, the upgrade process gets stuck.<br />
<br />
systemd, on the other hand, allows you to put a new service configuration file in the configuration directory, such as: <i>/etc/systemd/system</i>. We can conveniently reload the configuration with a single command-line instruction:<br />
<br />
<pre>
$ systemctl daemon-reload
</pre>
I believe that the updating process can still be somewhat simplified in <i>s6-rc</i>. Fortunately, I have managed to hide that complexity in the <i>nixproc-s6-rc-deploy</i> tool.</li>
<li>It was also difficult to find out all the available configuration properties for <i>s6-rc</i> services -- I ended up looking at the <a href="https://github.com/skarnet/s6-rc/tree/master/examples">examples</a> and studying the documentation pages for <a href="https://skarnet.org/software/s6-rc/s6-rc-compile.html"><i>s6-rc-compile</i></a>, <a href="https://skarnet.org/software/s6/s6-supervise.html"><i>s6-supervise</i></a> and <a href="https://skarnet.org/software/s6/servicedir.html">service directories</a>.<br />
<br />
I think that it could be very helpful to write a dedicated documentation page that describes all configurable properties of <i>s6-rc</i> services.</li>
<li>I believe it is also very common that for each <i>longrun</i> service (with a <i>-srv</i> suffix), that you want a companion logging service (with a <i>-log</i> suffix).<br />
<br />
As a matter of fact, I can hardly think of a situation in which you do not want this. Maybe it helps to introduce a convenience property to automatically facilitate the generation of log companion services.</li>
</ul>
<br />
<h2>Availability</h2>
<br />
The <i>s6-rc</i> backend described in this blog post is part of the current development version of the Nix process management framework, that is still under heavy development.<br />
<br />
The framework can be obtained from <a href="https://github.com/svanderburg/nix-processmgmt">my GitHub page</a>.<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com1tag:blogger.com,1999:blog-1397115249631682228.post-26579064844530045012020-12-31T17:13:00.003+01:002020-12-31T17:21:41.028+01:00Annual blog reflection over 2020In <a href="https://sandervanderburg.blogspot.com/2020/12/blog-reflection-over-last-decade.html">my previous blog post</a> that I wrote yesterday, I celebrated my blog's 10th anniversary and did a reflection over the last decade. However, I did not elaborate much about 2020.<br />
<br />
Because 2020 is a year for the history books, I have decided to also do an annual reflection over the last year (similar to my previous annual blog reflections).<br />
<br />
<h2>A summary of blog posts written in 2020</h2>
<br />
Nearly all of the blog posts that I have written this year were in service of only two major goals: developing the <strong>Nix process management framework</strong> and implementing <strong>service container</strong> support in Disnix.<br />
<br />
Both of them took a substantial amount of development effort. Much more than I initially anticipated.<br />
<br />
<h3>Investigating process management</h3>
<br />
I already started working on this topic last year. In November 2019, <a href="https://sandervanderburg.blogspot.com/2019/11/a-nix-based-functional-organization-for.html">I wrote a blog post</a> about packaging sysvinit scripts and a Nix-based functional organization for configuring process instances, that could potentially also be applied to other process management solutions, such as systemd and supervisord.<br />
<br />
After building my first version of a framework (which was already a substantial leap in reaching my full objective), I thought it would not take me that much time to get all details that I originally planned finished. It turns out that I heavily underestimated the complexity.<br />
<br />
To test my framework, I needed a simple test program that could daemonize on its own, which was (and still is) a common practice for running services on Linux and many other UNIX-like operating systems.<br />
<br />
I thought writing such a tool that daemonizes would be easy, but after some research, I discovered that it is actually quite complicated to do it properly. <a href="https://sandervanderburg.blogspot.com/2020/01/writing-well-behaving-daemon-in-c.html">I wrote a blog post about my findings</a>.<br />
<br />
It took me roughly three months to finish <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html">the first implementation of the process manager-agnostic abstraction layer</a> that makes it possible to write a high-level specification of a running process, that could universally target all kinds of process managers, such as sysvinit, systemd, supervisord and launchd.<br />
<br />
After completing the abstraction layer, I also discovered that a sufficiently high-level deployment specification of running processes could also target other kinds deployment solutions.<br />
<br />
I have developed two additional backends for the Nix process management framework: one that uses <a href="https://sandervanderburg.blogspot.com/2020/08/experimenting-with-nix-and-service.html">Docker</a> and another using <a href="https://sandervanderburg.blogspot.com/2020/06/using-disnix-as-simple-and-minimalistic.html">Disnix</a>. Both solutions are technically not qualified as process managers, but they can still be used as such by only using a limited set of features of these tools.<br />
<br />
To be able to develop the Docker backend, I needed to dive deep into the underlying concepts of Docker. <a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">I wrote a blog post about the relevant Docker deployment concepts</a>, and also gave a presentation about it at Mendix.<br />
<br />
While implementing more examples, I also realized that to more securely run long-running services, they typically need to run as unprivileged users. To get predictable results, these unprivileged users require stable user IDs and group IDs.<br />
<br />
Several years ago, I have already worked on a port assigner tool that could already assign unique TCP/UDP port numbers to services, so that multiple instances can co-exist.<br />
<br />
I have extended the port assigner tool to <a href="https://sandervanderburg.blogspot.com/2020/09/assigning-unique-ids-to-services-in.html">assign arbitrary numeric IDs</a> to generically solve this problem. In turns out that implementing this tool was much more difficult than expected -- the Dynamic Disnix toolset was originally developed under very high time pressure and had a substantial amount of technical debt.<br />
<br />
In order to implement the numeric ID assigner tool, I needed to revise the model parsing libraries, that broke the implementations of some of the deployment planner algorithms.<br />
<br />
To fix them, I was forced to study how these algorithms worked again. <a href="https://sandervanderburg.blogspot.com/2020/10/transforming-disnix-models-to-graphs.html">I wrote a blog post</a> about the graph-based deployment planning algorithms and a new implementation that should be better maintainable. Retrospectively, I wish I did my homework better at the time when I wrote the original implementation.<br />
<br />
In September, I gave a talk about the Nix process management framework at <a href="https://2020.nixcon.org">NixCon 2020</a>, that was held online.<br />
<br />
I pretty much reached all my objectives that I initially set for the Nix process management framework, but there is still some leftover work to bring it at an acceptable usability level -- to be able to more easily add new backends (somebody gave me <a href="https://skarnet.org/software/s6">s6</a> as an option) the transformation layer needs to be standardized.<br />
<br />
Moreover, I still need to develop a test strategy for services so that you can be (reasonably) sure that they work with a variety of process managers and under a variety of conditions (e.g. unprivileged user deployments).<br />
<br />
<h3>Exposing services as containers in Disnix</h3>
<br />
Disnix is a Nix-based distributed service deployment tool. Services can basically be any kind of deployment unit whose life-cycle can be managed by a companion tool called Dysnomia.<br />
<br />
There is one practical problem though, in order to deploy a service-oriented system with Disnix, it typically requires the presence of already deployed <strong>containers</strong> (not be confused with Linux containers), that are environments in which services are managed by another service.<br />
<br />
Some examples of container providers and corresponding services are:<br />
<br />
<ul>
<li>The MySQL DBMS (as a container) and multiple hosted MySQL databases (as services)</li>
<li>Apache Tomcat (as a container) and multiple hosted Java web applications (as services)</li>
<li>systemd (as a container) and multiple hosted systemd unit configuration files (as services)</li>
</ul>
<br />
Disnix deploys the services (as described above), but not the containers. These need to be deployed by other means first.<br />
<br />
In the past, I have been working on solutions that manage the underlying infrastructure of services as well (I typically used to call this problem domain: <strong>infrastructure deployment</strong>). For example, NixOps can deploy a network of NixOS machines that also expose container services that can be used by Disnix. It is also possible to deploy the containers as services, in a separate deployment layer managed by Disnix.<br />
<br />
When the Nix process management framework became more usable, I wanted to make the deployment of container providers also a more accessible feature. I heavily revised Disnix with a new feature that makes it possible to <a href="https://sandervanderburg.blogspot.com/2020/04/deploying-container-and-application.html">expose services as container providers</a>, making it possible to deploy both the container services and application services from a single deployment model.<br />
<br />
To make this feature work reliably, I was again forced to <a href="https://sandervanderburg.blogspot.com/2020/07/a-new-input-model-transformation.html">revise the model transformation pipeline</a>. This time I concluded that the lack of references in the Nix expression language was an impediment.<br />
<br />
Another nice feature by combining the Nix process management framework and Disnix is that you can <a href="https://sandervanderburg.blogspot.com/2020/05/deploying-heterogeneous-service.html">more easily deploy a heterogeneous system locally</a>.<br />
<br />
I have released a new version of Disnix: version 0.10, that provides all these new features.<br />
<br />
<h3>The Monitoring playground</h3>
<br />
Besides working on the two major topics shown above, the only other thing I did was a Mendix crafting project in which I developed a <a href="https://sandervanderburg.blogspot.com/2020/11/constructing-simple-alerting-system.html">monitoring playground</a>, allowing me to locally experiment with alerting scripts, including the visualization and testing.<br />
<br />
<h2>Some thoughts</h2>
<br />
From a blogging perspective, I am happy what I have accomplished this year -- not only have I managed to reach my usual level of productivity again (<a href="https://sandervanderburg.blogspot.com/2019/12/9th-annual-blog-reflection.html">last year was somewhat disappointing</a>), I also managed to both develop a working Nix process management framework (making it possible to use all kinds of process managers), and use Disnix to deploy both container and application services. Both of these features are on my wish list for many years.<br />
<br />
In the Nix community, having the ability to also use other process managers than systemd, is something we have been discussing already since late 2014.<br />
<br />
However, there are also two major things that kept me mentally occupied in the last year.<br />
<br />
<h3>Open source work</h3>
<br />
Many blog posts are about the open source work I do. Some of my open source work is done as part of my day time job as a software engineer -- sometimes we can write a new useful feature, make an extension to an existing project that may come in handy, or do something as part of an investigation/learning project.<br />
<br />
However, the majority of my open source work is done in my spare time -- in many cases, my motivation is not as <a href="https://www.merriam-webster.com/dictionary/altruistic">altruistic</a> as people may think: typically I need something to solve my own problems or there is some technical concept that I would like to explore. However, I still do a substantial amount of work to help other people or for the "greater good".<br />
<br />
Open source projects are typically quite satisfying the work on, but they also have negative aspects (typically the negative aspects are negligible in the early stages of a project). Sadly as projects get more popular and gain more exposure, the negativity attached to them also grows.<br />
<br />
For example, although I got quite a few positive reactions on my work on the Nix process management framework (especially at NixCon 2020), I know that not everybody is or will be happy about it.<br />
<br />
I have worked with people in the past, who consider this kind of work a complete waste of time -- in their opinion, we already have <a href="https://kubernetes.io/">Kubernetes</a> that has already solved all relevant service management problems (<a href="https://www.reddit.com/r/kubernetes/comments/dtsg4z/dilbert_on_kubernetes/">some people even think it is a solution to all problems</a>).<br />
<br />
I have to admit that, while Kubernetes can be used to solve similar kind of problems (and what is not supported as a first-class feature, can still be scripted in ad-hoc ways), there is still much to think about:<br />
<br />
<ul>
<li>As explained in my blog post about Docker concepts, the Nix store typically supports much more <strong>efficient sharing</strong> of common dependencies between packages than layered Docker images, resulting in much lower disk space consumption and RAM consumption of processes that have common dependencies.</li>
<li>Docker containers support the deployment of so-called microservices because of a common denominator: <strong>processes</strong>. Almost all modern operating systems and programming-languages have a notion of processes.<br />
<br />
As a consequence, lots of systems nowadays typically get constructed in such a way that they can be easily decomposed into processes (translating to container instances), imposing substantial overhead on each process instance (because these containers typically need to embed common sub services).<br />
<br />
Services can also be more efficiently deployed (in terms of storage and RAM usage) as units by managed by a common runtime (e.g. multiple Java web applications managed by Apache Tomcat or multple PHP applications managed by the Apache HTTP server).<br />
<br />
The latter form of reuse is now slowly disappearing, because it does not fit nicely in a container model. In Disnix, this form of reuse is a first-class concept.</li>
<li>Microservices managed by Docker (somewhat) support <strong>technology diversity</strong>, because of the fact that all major programming languages support the concept of processes.<br />
<br />
However, one particular kind of technology that you cannot choose is the the <strong>operating system</strong> -- Docker/Kubernetes relies on non-standardized Linux-only concepts.<br />
<br />
I have also been thinking about the option to pick your operating system as well: you need security, then pick: OpenBSD, you want performance, then pick: Linux etc. The Nix process management framework allows you to also target process managers on different operating systems than Linux, such as BSD rc scripts and Apple's launchd.</li>
</ul>
<br />
I personally believe that these goals are still important, and that keeps me motivated to work on it.<br />
<br >
Furthermore, I also believe that it is important to have <strong>multiple implementations</strong> of tools that solve the <strong>same</strong> or similar kind of <strong>problems</strong> -- in the open source world, there are lot of "battles" between communities about which technology should be the standard for a certain problem.<br />
<br />
My favourite example of such a battle is the system's process manager -- many Linux distributions nowadays have adopted <a href="https://www.freedesktop.org/wiki/Software/systemd/">systemd</a>, but this is not without any controversy, such as in the <a href="https://debian.org">Debian project</a>.<br />
<br />
It took them many years to come to the decision to adopt it, and still there are people who want to discuss "init system diversity". Likewise, there are people who find the systemd-adoption decision unacceptable, and have forked Debian into <a href="https://www.devuan.org">Devuan</a>, providing a Debian-based distribution without systemd.<br />
<br />
With the Nix process management framework the fact that systemd exists (and may not be everybody's first choice) is not a big deal -- you can actually switch to other solutions, if desired. A battle between service managers is not required. A sufficiently high-level specification of a well understood problem allows you to target multiple solutions.<br />
<br />
Another problem I face is that these two projects are not the only projects I have been working on or maintain. There are many other projects I have been working on the past.<br />
<br />
Sadly, I am also a very bad multitasker. If there are problems reported with my other projects, and the fix is straight forward, or there is a straight forward pull request, then it is typically no big deal to respond.<br />
<br />
However, I also learned that some for some of the problems other people face, there is no quick fix. Sometimes I get pull requests that partially solves a problem, or in other cases: fix a specific problem, but breaks others features. These pull requests cannot always be directly accepted and also need a substantial amount of my time for reviewing.<br />
<br />
For certain kinds of reported problems, I need to work on a <strong>fundamental revision</strong> that requires a substantial amount of development effort -- however, it is impossible to pick up such a task while working on another "major project".<br />
<br />
Alternatively, I need to make the decision to abandon what I am currently working on and make the switch. However, this option also does not have my preference because I know it will significantly delay my original goal.<br />
<br />
I have noticed that lots of people get dissatisfied and frustrated, including myself. Moreover, I also consider it a bad thing to feel pressure on the things I am working on in my spare time.<br />
<br />
So what to do about it? Maybe I can write a separate blog post on this subject.<br />
<br />
Anyway, I was not planning to abandon or stop anything. Eventually, I will pick up these other problems as well -- my strategy for now, is to do it when I am ready. People simply have to wait (so if you are reading this and waiting for something: yes, I will pick it up eventually, just be patient).<br />
<br />
<h3>The COVID-19 crisis</h3>
<br />
The other subject that kept me (and pretty much everybody in the world) busy is the COVID-19 crisis.<br />
<br />
I still remember the beginning of 2020 -- for me personally, it started out very well. I visited some friends that I have not seen in a long time, and then <a href="http://fosdem.org">FOSDEM</a> came, the Free and Open Source Developers European Meeting.<br />
<br />
Already in January, I heard about this new virus that was rapidly spreading in the Wuhan region on the news. At that time, nobody in the Netherlands (or in Europe) was really worried yet. Even to questions, such as: "what will happen when it reaches Europe?", people typically responded with: "ah yes, well influenza has a impact on people too, it will not be worse!".<br />
<br />
A few weeks later, it started to spread to countries close to Europe. The first problematic country I heard about was Iran, and a couple of weeks later it reached Italy. In Italy, it spread so rapidly that within only a few weeks, the intensive care capacity was completely drained, forcing medical personnel to make choices who could be helped and who could not.<br />
<br />
By then, it sounded pretty serious to me. Furthermore, I was already quite sure that it was only a matter of time before it would reach the Netherlands. And indeed, at the end of February, the first COVID-19 case was reported. Apparently this person contracted the virus in Italy.<br />
<br />
Then the spreading went quickly -- every day, more and more COVID-19 cases were reported and this amount grew exponentially. Similar to other countries, we also slowly ran into capacity problems in hospitals (materials, equipment, personnel, intensive care capacity etc.). In particular, the intensive care capacity reached at a very critical level. Fortunately, there were hospitals in Germany willing to help us out.<br />
<br />
In March, a country-wide lockdown was announced -- basically all group activities were forbidden, schools and non-essential shops were closed, and everybody who is capable of working from should work from home. As a consequence, since March, I have been permanently working from home.<br />
<br />
As with pretty much everybody in the world, COVID-19 has negative consequences for me as well. Fortunately, I have not much to complain about -- I did not get unemployed, I did not get sick, and also nobody in my direct neighbourhood ran into any serious problems.<br />
<br />
The biggest implication of the COVID-19 pandemic for me is social contacts -- despite the lockdown I still regularly meet up with family and frequent acquaintances, but I have barely met any new people. For example, at Mendix, I typically came in contact with all kinds of people in the company, especially those that do not work in the same team.<br />
<br />
Moreover, I also learned that quite a few of my contacts got isolated because of all group activities that were canceled -- for example I did not have any music rehearsals in a while, causing me not to see or speak to any of my friends there.<br />
<br />
Same thing with conferences and meet ups -- because most of them were canceled or turned into online events, it is very difficult to have good interactions with new people.<br />
<br />
I also did not do any traveling -- my summer holiday was basically a staycation. Fortunately, in the summer, we have managed to minimize the amount of infections, making it possible to open up public places. I visited some touristic places in the Netherlands, that are normally crowded by people from abroad. That by itself was quite interesting -- I normally tend to neglect national touristic sites.<br />
<br />
Although the COVID-19 pandemic brought all kinds of negative things, there were also a couple of things that I consider a good thing:<br />
<br />
<ul>
<li>At Mendix, we have an open office space that typically tends to be very crowded and noisy. It is not that I cannot work in such an environment, but I also realize that I do appreciate silence, especially for programming tasks that require concentration. At home, it is quiet, I have much fewer distractions and I also typically feel much less tired after a busy work day.</li>
<li>I also typically used to neglect home improvements a lot. The COVID-19 crisis helped me to finally prioritize some non-urgent home improvements tasks -- for example, on the attic, where my musical instruments are stored, I finally took the time to organize everything in such a way that I can rehearse conveniently.</li>
<li>Besides the fact that rehearsals and concerts were cancelled, I actually practiced a lot -- I even studied many advanced solo pieces that I have not looked at in years. Playing music became a standard activity between my programming tasks, to clear my mind. Normally, I would use this time to talk to people at the coffee machine in the office.</li>
<li>During busy times I also used to tend to neglect house keeping tasks a lot. I still remember (many years ago) when I just moved into my first house, doing the dishes was already a problem (I had no dish washer at that time). When working from home, it is not a problem to keep everything tidy.</li>
<li>It is also much easier to maintain healthy daily habits. In the first lockdown (that was in spring), cycling/walking/running was a daily routine that I could maintain with ease.</li>
</ul>
<br />
In the Netherlands, we have managed to overcome the first lockdown in just a matter of weeks by social distancing. Sadly, after the restrictions were relaxed we got sloppy and at the end of the summer the infection rate started to grow. We also ran into all kinds of problems to mitigate the infections -- limited test capacity, people who got tired of all the countermeasures not following the rules, illegal parties etc.<br />
<br />
Since a couple of weeks we are in our second lockdown with a comparable level of strictness -- again, the schools and non-essentials shops are closed etc. The second lockdown feels a lot worse than the first -- now it is in the winter, people are no longer motivated (the amount of people that revolt in the Netherlands have grown substantially, including people spreading news that everything is a Hoax and/or a one big scheme organized by left-wing politicians) and it is already taking much longer than the first.<br />
<br />
Fortunately, there is a tiny light at the end of the tunnel. In Europe, one vaccine (the <a href="https://www.nhg.org/actueel/nieuws/biontechpfizer-vaccin-relevante-informatie-voor-huisartsen">Pfizer vaccine</a>) has been approved and more submissions are pending (with good results). By Monday, the authorities will start to vaccinate people in the Netherlands.<br />
<br />
If we can keep the infection rates and the mutations under control (<a href="https://www.bbc.com/news/health-55388846">such as the mutation that appeared in England</a>) then we will eventually build up the required group immunity to finally get the situation under control (this probably is going to take many more months, but at least it is a start).<br />
<br />
<h2>Conclusion</h2>
<br />
This elaborate reflection blog post (that is considerably longer than all my previous yearly reflections combined) reflects over 2020 that is probably a year that will no go unnoticed in the history books.<br />
<br />
I hope everybody remains in good health and stays motivated to do what it is needed to get the virus under control.<br />
<br />
Moreover, when the crisis is over, I also hope we can retain the positive things learned in this crisis, such as making it more a habit to allow people to work (at least partially) from home. The open-source complaint in this blog post is just a minor inconvenience compared to the COVID-19 crisis and the impact that it has on many people in the world.<br />
<br />
The final thing I would like to say is:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYJj0TPnPQUmcKqToUiXVvUeIFriQlA-UNKUHzy3ezqNknq4qKoCHS4E533Mj0K_ulKOEE3qJ8QwC3ZAsHLRS3bozaRTT8T6mISWxQf7pEP1CMLqfMVem3__pEgeFmdRyiRXbwjgzR74I8/s1000/fireworks_rotterdam.jpeg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="667" data-original-width="1000" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYJj0TPnPQUmcKqToUiXVvUeIFriQlA-UNKUHzy3ezqNknq4qKoCHS4E533Mj0K_ulKOEE3qJ8QwC3ZAsHLRS3bozaRTT8T6mISWxQf7pEP1CMLqfMVem3__pEgeFmdRyiRXbwjgzR74I8/s600/fireworks_rotterdam.jpeg"/></a></div>
<br />
HAPPY NEW YEAR!!!!!!!!!!!!<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-79455348932125318422020-12-30T14:33:00.001+01:002020-12-30T14:33:25.736+01:00Blog reflection over the last decade<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Z54tsfshoBnOakb0_wgPQTas29ph4d0UY-0lYVEBjCA1BjyxYxlEvttdTmOFcf3Dd9Tujb4Wk-BHOyRttdm9CAzBiI5vGwFGRc5rjX30snCNSU-UuqhpvK_k6UofmYkl2GJE3IXBe4tg/s0/notebook.jpg" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="350" data-original-height="500" data-original-width="500" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh-Z54tsfshoBnOakb0_wgPQTas29ph4d0UY-0lYVEBjCA1BjyxYxlEvttdTmOFcf3Dd9Tujb4Wk-BHOyRttdm9CAzBiI5vGwFGRc5rjX30snCNSU-UuqhpvK_k6UofmYkl2GJE3IXBe4tg/s0/notebook.jpg"/></a></div>
<br />
Today it is exactly <strong>ten years</strong> ago that I started this blog. As with previous years, I will do a reflection, but this time it will be over the <strong>last decade</strong>.<br />
<br />
<h2>What was holding me back</h2>
<br />
The idea to have my own blog was already there for a long time. I always thought it was an interesting medium. For example, I considered it a good instrument to express my thoughts on the technical work I do, and in particular, I liked having the ability to get feedback.<br />
<br />
The main reason why it still took me so long to start was because I never considered it "the right time". For example, I was already close to starting a blog 15 years ago (while I was still early in my studies) when web development was still one of my main technical interests, but still refrained from doing so.<br />
<br />
At that time I did some interesting "discoveries" and I had some random ideas I could elaborate about, but these ideas never materialized enough so that I could write a story about it.<br />
<br />
Moreover, I also did not feel comfortable enough yet to express myself, because I did not have much writing experience in English. Retrospectively, I learned that there is a never a right time for having a blog, I should just start.<br />
<br />
<h2>Entering the research domain</h2>
<br />
A couple of years later, while I was working on my master's thesis, I made the decision to go for a PhD degree, because I was genuinely interested in the research domain of my master's thesis: <strong>software deployment</strong>, mostly because of my prior experience in industry and building <a href="https://linuxfromscratch.org">Linux distributions from scratch</a>.<br />
<br />
Even before starting my PhD, I already knew that writing is an important component in research -- as a researcher, you have to regularly report about your work by means of <strong>research papers</strong> that typically need to be anonymously peer reviewed.<br />
<br />
In most scientific disciplines, academic papers are published in journals. In the computer science domain, it is more common to publish papers in conference proceedings.<br />
<br />
Only a certain percentage of paper submissions that are considered good quality (as judged by the peer reviews) are accepted for publication. Rejection of a paper typically requires you make revisions and submitting that paper to a different conference.<br />
<br />
For top general conferences in the software engineering domain the <a href="https://taoxie.cs.illinois.edu/seconferences.htm">acceptance rate</a> is typically lower than 20% (this ratio used be even lower, close to 15%).<br />
<br />
In my PhD, I had a very quick publication start -- in the first month, a paper about atomic upgrades for distributed systems was accepted that covered an important topic of my master's thesis.<br />
<br />
Roughly half a year later, me and my co-workers published a research paper about the objectives of the Pull Deployment of Services (PDS) research project (in which my research was of the sub topics) funded by <a href="http://jacquard.nl">NWO/Jacquard</a>.<br />
<br />
Although I had a very good start, I slowly started to learn (the hard way) that you cannot simply publish research papers about all the work you do -- as a matter of fact, it only represents a modest sub set of your daily work.<br />
<br />
To write a good research paper, it takes quite a bit of time and effort to decide about the topic (including the paper's title) and to get all the details right. I had all kinds of interesting ideas but many of these ideas were not considered <strong>novel</strong> -- they were interesting engineering efforts but they did not add interesting new (significant) scientific knowledge.<br />
<br />
Moreover, in a research paper, you also need to put your contribution in <strong>context</strong> (e.g. explain/show how it compares to similar work and how it expands existing knowledge), and provide <strong>validation</strong> (this can be a proof, but in most cases you <strong>evaluate</strong> in what degree your contribution meets its claims, for example, by providing empirical data).<br />
<br />
After instant acceptance of the first two papers, things did not work out that smoothly anymore. I had several paper rejections in a row -- one paper was badly rejected because I did not put it into the right context (for example, I ignored some important related work) and I did not make my contribution very clear (I basically left it open to the interpretation of the reader, which is a bad thing).<br />
<br />
Fortunately, I learned a lot from this rejection. The reviewers even suggested me an alternative conference where I could submit my revised paper to. After addressing the reviewers' criticisms, the paper got accepted.<br />
<br />
Another paper was rejected twice in a row for IMO very weak reasons. Most notably, it turned out that many reviewers believed that the subject was not really software engineering related (which is strange, because software deployment is explicitly listed as one of the subjects in the conference's <a href="http://www.sbs.co.za/icse2010/">call for papers</a>).<br />
<br />
When I explained this peculiarity to Eelco Visser (one of my supervisors and co-promotor), he suggested that I should have more frequent interaction with the scientific community and write about the subject on my blog. Software deployment is generally a neglected subject in the software engineering research community.<br />
<br />
Eventually, we have managed to publish the problematic papers (one is about Disnix, the tool implementation of my research) and the other about the testing aspect of the previously rejected paper.<br />
<br />
After that problematic period, I have managed to publish two more papers that got instantly accepted bringing me to all kinds of interesting conferences.<br />
<br />
<h2>The decision to start my blog</h2>
<br />
Although having a 3 paper acceptance streak and traveling to the conferences to present them felt nice for a while, I still was not too happy.<br />
<br />
In late 2010, one day before new years eve (I typically reflect over things in the past year at new year's eve) I realized that research papers alone is just a very narrow representation of the work that I do as a researcher (although the amount of papers and their impact are typically used as the only metric to judge the performance of a researcher).<br />
<br />
In addition to getting research papers accepted and doing the required writing, there is much more that the work of an academic researcher (and in particular the software engineering domain) is about:<br />
<br />
<ul>
<li>Research in software engineering is about <strong>constructing tools</strong>. For example, the paper: <a href="http://dl.acm.org/citation.cfm?id=807694">Research Paradigms in Computer Science</a>' by <a href="http://www.cs.brown.edu/~pw">Peter Wegner</a> from Brown University says:<br />
<br />
<blockquote>
Research in engineering is directed towards the efficient accomplishment of specific tasks and towards the development of tools that will enable classes of tasks to be accomplished more efficiently.
</blockquote>
<br />
In addition to the problems that tools try solve or optimize, the construction of these tools is typically also very challenging, similar to conventional software development projects.<br />
<br />
Although the construction aspects of tools may not always be novel and too detailed for a research paper (that typically has a page limit), it is definitely useful to work towards a good and stable design and implementation. Writing about these aspects can be very useful for yourself, your colleagues and peers in the field.<br />
<br />
Moreover, having a tool that is usable and works also mattered to me and to the people in my research group. For example, my deployment research was built on top of the Nix package manager, that in addition to research, was also used to solve our internal deployment problems.<br />
<br />
</li>
<li>
I did not completely start all the development work of my tooling from scratch -- I was building my deployment tooling on top of the Nix package manager that was both a research project, and an open source project (more accurately called a community project) with a small group of external contributors.<br />
<br />
(As a sidenote: the Nix package manager was started by Eelco Dolstra who was a Postdoc in the same research project and one my university supervisors).<br />
<br />
I considered my blog a good instrument to communicate with the Nix community about ideas and implementation aspects.
</li>
<li>
Research is also about having <strong>frequent interaction</strong> with your peers that work for different universities, companies and/or research institutes.<br />
<br />
A research paper is useful to get feedback, but at the same time, it is also quite an inaccessible medium -- people can obtain a copy from publishers (typically behind a paywall) or from your personal homepage and communicate by e-mail, but the barrier is typically high.
</li>
<li>
I was also frequently in touch with software engineering practitioners, such as former study friends, open source communities and people from our research project's industry partner: <a href="https://www.usa.philips.com/healthcare">Philips Healthcare</a>.<br />
<br />
I regularly received all kinds of interesting questions related to the practical aspects of my work. For example, how to apply our research tools to industry problems or how our research tools compare to conventional tools.<br />
<br />
Not all of these questions can be transformed into research papers, but were definitely useful to investigate and write about.
</li>
<li>
Being in academia is <strong>more than</strong> just working on <strong>publications</strong>. You also travel to conferences, get involved in all kinds of different (and sometimes related) research subjects of your colleagues and peers and you may also help in teaching. These subjects are also worth writing about.
</li>
</ul>
<br />
Because of the above reasons, I was finally convinced that the time was right to start my blog.<br />
<br />
<h2>The beginning: catching up with my research papers</h2>
<br />
Since I was already working on my PhD research for more than 2 years, there was still a lot of catching up I had to do. It did not make sense to just randomly start writing about something technical or research related. Basically, I wanted all information on my blog "to fit together".<br />
<br />
For the first half year, my blog was basically about writing things down I had already done and published about.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEho7swTYEBJDpJGXvRgPPfztxkamtKHN_NAtYV2Rn1_YJHaQEaoF4jstHiA5pkPCIS49sPDThUqAoSZ2ZnO-1f0zy4nZGdRh5alW8Cbvi0kl0uO3-mkPXZpU2ixwT89k-0Uft20AQv_Zdvp/s1600/doctor.png" imageanchor="1" style="clear:left; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="182" width="200" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEho7swTYEBJDpJGXvRgPPfztxkamtKHN_NAtYV2Rn1_YJHaQEaoF4jstHiA5pkPCIS49sPDThUqAoSZ2ZnO-1f0zy4nZGdRh5alW8Cbvi0kl0uO3-mkPXZpU2ixwT89k-0Uft20AQv_Zdvp/s200/doctor.png" /></a></div>
<br />
After <a href="https://sandervanderburg.blogspot.com/2010/12/first-blog-post.html">my blog announcement</a>, I started explaining what the <a href="https://sandervanderburg.blogspot.com/2010/12/pull-deployment-of-services.html">Pull Deployment of Services research project</a> is about, then explaining the <a href="https://sandervanderburg.blogspot.com/2011/01/nix-package-manager.html">Nix package manager</a> that serves as the fundamental basis of all the deployment tooling that I was developing, followed by <a href="https://sandervanderburg.blogspot.com/2011/01/nixos-purely-functional-linux.html">NixOS</a>, a Linux distribution that is entirely managed by the Nix package manager that can be deployed from a single declarative specification.<br />
<br />
The next blog post was about <a href="https://sandervanderburg.blogspot.com/2011/02/using-nixos-for-declarative-deployment.html">declarative deployment and testing with NixOS</a>. It was used as an ingredient for a research paper that already got published, and a talk with the same title for FOSDEM: the free and open source's European meeting in Brussels. Writing about the subject on my blog was a useful preparation session for my talk.<br />
<br />
After giving my talk at FOSDEM, there was more catching up work to do. After explaining the basic Nix concepts, I could finally elaborate about <a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a>, the tool I have been developing as part of my research that uses Nix to extend deployment to the domain of service-oriented systems.<br />
<br />
After writing about the <a href="https://sandervanderburg.blogspot.com/2011/03/self-adaptive-deployment-with-disnix.html">self-adaptive deployment framework</a> built on top of Disnix (I have submitted my paper at the beginning that year, and it got accepted shortly before writing the corresponding blog post), I was basically up-to-date with all research aspects.<br />
<br />
<h2>Using my blog for research</h2>
<br />
After my catch up phase was completed, I could finally start writing about things that were not directly related to any research papers already written in the past.<br />
<br />
One of the things I have been struggling with for a while was making our tools work with .NET technology. The Nix package manager (and sister projects, such as Disnix) were primarily developed for UNIX-like operating systems (most notably Linux) and technologies that run on these operating systems.<br />
<br />
Our industry partner: Philips Healthcare, mostly uses Microsoft technologies in their development stack ranging from .NET as a runtime, C# for coding, SQL server for storage, and IIS as web server.<br />
<br />
At that time, .NET was heavily tied to the Windows eco-system (<a href="https://www.mono-project.com">Mono</a> already existed that provided a somewhat compatible runtime for other operating systems than Windows, but it did not provide compatible implementations of all libraries to work with the Philips platform).<br />
<br />
With some small modifications, I could use Nix on Cygwin to build .NET projects. However, running .NET applications that rely on shared libraries (called library assemblies in .NET terminology) was still a challenge. I could only provide a number of very sub optimal solutions, of which none was ideal.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2011/09/deploying-net-applications-with-nix.html">I wrote about it on my blog</a>, and during my trip to ICSE 2011 in Hawaii I learned from a discussion with a co-attendee that you could also use an event listener that triggers when a library assembly is missing. The reflection API can be used in this event handler to load these missing assemblies, making it possible to <a href="https://sandervanderburg.blogspot.com/2011/09/deploying-net-applications-with-nix_14.html">efficiently solve my dependency problem</a> making it possible to use both Nix and <a href="https://sandervanderburg.blogspot.com/2011/10/deploying-net-services-with-disnix.html">Disnix to deploy .NET services</a> on Windows without any serious obstacles.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWTy4_EFXZWzBpHQB1fDUg332qXKEYtPIh0hFdf4LLOtzgXbY6OJk0m0ImKon77WxZaQuo-ZtPR9QPs-1eqnsQrGQ-U2ilIHm6UgkGU2mbf6dSQOsizYibKq464mREoTIR7YVZZGSh9iYd/s1600/building.jpg" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="300" width="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiWTy4_EFXZWzBpHQB1fDUg332qXKEYtPIh0hFdf4LLOtzgXbY6OJk0m0ImKon77WxZaQuo-ZtPR9QPs-1eqnsQrGQ-U2ilIHm6UgkGU2mbf6dSQOsizYibKq464mREoTIR7YVZZGSh9iYd/s400/building.jpg" /></a></div>
<br />
I have also managed to discuss one of my biggest frustrations in the research community: the fact that <a href="https://sandervanderburg.blogspot.com/2011/10/software-deployment-complexity.html">software deployment is a neglected subject</a>. Thanks to spreading the blog post on <a href="https://twitter.com">Twitter</a> (that in turn got retweeted by all kinds of people in the research community) it attracted quite a few visitors and a large number helpful comments. I even got in touch with a company that develops a software deployment automation solution as their main product.<br />
<br />
Another investigation that I did as part of my blog (without publishing in mind) was addressing a common criticism from various communities, <a href="https://lists.debian.org/debian-devel/2008/12/msg01007.html">such as the Debian community</a>, that Nix would not qualify itself as a viable package management solution because <a href="https://sandervanderburg.blogspot.com/2011/11/on-nix-nixos-and-filesystem-hierarchy.html">it does not comply to the Filesystem Hierarchy Standard (FHS)</a>.<br />
<br />
I also did <a href="https://sandervanderburg.blogspot.com/2011/12/evaluation-and-comparison-of-gobolinux.html">a comparison with the deployment properties of GoboLinux</a>, another Linux distribution that deliberately deviates from the FHS to show that a different filesystem organisation has clear benefits for making deployments more reliable and reproducible. The GoboLinux blog post appeared on <a href="http://reddit.com">Reddit</a> (both the NixOS and Linux channels) and attracted quite a few visitors.<br />
<br />
From these practical investigations I wrote a blog post that draws some <a href="https://sandervanderburg.blogspot.com/2011/12/techniques-and-lessons-for-improvement.html">general conclusions</a>.<br />
<br />
<h2>Reaching the end of my PhD research</h2>
<br />
After an interesting year, both from a research and blogging perspective, I was reaching the final year of my PhD research (in the Netherlands, a contract of a PhD student is typically only valid for 4 years).<br />
<br />
I had already slowly started with writing my PhD thesis, but there was still some unfinished business. There were four (!!!) more research ideas that I wanted to publish about (which was retrospectively looking, a very overambitious goal).<br />
<br />
One of these papers was a collaboration project in which we combined our knowledge about software deployment and construction with license compliance engineering to determine which source files are actually used in a binary so that we could detect whether it meets the terms and conditions of <a href="https://sandervanderburg.blogspot.com/2011/02/free-and-open-source-software.html">free and open-source licenses</a>.<br />
<br />
Although our contribution looked great and we were able to detect a compliance issue in <a href="https://ffmpeg.org/">FFmpeg</a>, a widely used open source project, the paper was rejected twice in a row. The second time the reviews were really vague and not helpful at all. One of my co-authors called the reviewers extremely incompetent.<br />
<br />
After the second rejection, I was (sort of) done with it and extremely disappointed. I did not even want to revise it and submit it anywhere else. Nonetheless, I have published the paper as a technical report, <a href="https://sandervanderburg.blogspot.com/2012/04/dynamic-analysis-of-build-processes-to.html">reported about it on my blog</a>, and added it as a chapter to my PhD thesis.<br />
<br />
(As a sidenote: more than 2 years later, we did another attempt to resurrect the paper. The revisions were quite a bit of work, but the third version finally got accepted at ASE 2014: one of the top general conferences in the software engineering domain.<br />
<br />
This was a happy moment for me -- I was so disappointed about the process, and I was happy to see that there were people who could motivate and convince me that we should not give up).<br />
<br />
Another research idea was formalizing infrastructure deployment. Sadly, the idea was not really considered novel -- it was mostly just an incremental improvement over our earlier work. As a result, I got two paper rejections in a row. After the second rejection, I have abolished the idea to publish about it, but I still wrote a chapter about it in my PhD thesis.<br />
<br />
All the above rejections (and the corresponding reviews) really started to negatively impact my motivation. I wrote two blog posts about my observations: one blog post was about a common reason for rejecting a paper: <a href="https://sandervanderburg.blogspot.com/2012/01/engineering-versus-science.html">the complaint that a contribution is engineering, but not science</a> (which is quite weird for research in software engineering). Another blog post was about <a href="https://sandervanderburg.blogspot.com/2012/04/software-engineering-fractions.html">the difficulties in connecting academic research with software engineering practice</a>. From my experiences thus far, I concluded that there is a huge gap between the two.<br />
<br />
Fortunately, I still managed to gather enough energy to finish my third idea. I already had a proof-of-concept implementation for <a href="https://sandervanderburg.blogspot.com/2012/03/deployment-of-mutable-components.html">managing state of services deployed by Disnix</a> for a while. By pulling out a few all nighters, I managed to write a research paper (all by myself) and submitted it to HotSWUp 2012. That paper got instantly accepted, which was a good boost for my motivation.<br />
<br />
In the last few months, the only thing I could basically do is finishing up my PhD thesis. To still keep my blog somewhat active, I have written a number of posts about <a href="https://sandervanderburg.blogspot.com/2012/07/a-review-of-conferences-in-2008-2009.html">my</a> <a href="https://sandervanderburg.blogspot.com/2012/08/a-review-of-conferences-in-2010.html">conference</a> <a href="https://sandervanderburg.blogspot.com/2012/09/a-review-of-conferences-in-2011-2012.html">experiences</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOsG9Mw5LCORnQ9q628EXK970XwtAORAQ6C702jYYyy_Am2UpWIw8ko5UtxiZ_aWjoj52c-x5ewi4b6p2PQ68vQJP5QDvoksiDvAHNcuA7sVQ4WPFO2Ar4TdvtxOwLTa0VGAAqTKvIzl9l/s1600/P1050706.JPG" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="213" width="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgOsG9Mw5LCORnQ9q628EXK970XwtAORAQ6C702jYYyy_Am2UpWIw8ko5UtxiZ_aWjoj52c-x5ewi4b6p2PQ68vQJP5QDvoksiDvAHNcuA7sVQ4WPFO2Ar4TdvtxOwLTa0VGAAqTKvIzl9l/s320/P1050706.JPG" /></a></div>
<br />
Although I already had a very early proof-of-concept implementation, I never managed to finish my fourth research paper idea. This was not a problem for finishing my PhD thesis as I already had enough material to complete it, but still I consider it one the more interesting research ideas that I never got to finish. As of today, I still have not finished or published about it (neither on my blog or in a research paper).<br />
<br />
<h2>Leaving academia, working for industry</h2>
<br />
A couple of weeks before my contract with the university was about to expire, I finished the first draft of my PhD thesis and submitted it to the reading committee for review.<br />
<br />
Although the idea of having an academic research career crossed my mind several times, I ultimately decided that this was not something I wanted to pursue, for a variety of reasons. Most notably, the discrepancy between topics suitable for publishing and things that could be applied in practice was one of the major reasons.<br />
<br />
All that was left was looking for a new job. After <a href="https://sandervanderburg.blogspot.com/2012/10/my-post-phd-carreer-aka-leaving-academia.html">an interesting job searching month</a> I joined Conference Compass, a startup company that consisted of fewer than 10 people when I joined.<br />
<br />
One of the interesting technical challenges they were facing was setting up a product-line for their mobile conference apps. My past experience with deployment technologies turned out to come in quite handy.<br />
<br />
The Nix project did not disappear after all involved people in the PDS project left the university (besides me, Eelco Dolstra (the author of the Nix package manager) and Rob Vermaas also joined an industrial company) -- the project moved to GitHub, increasing its popularity and the number of contributors.<br />
<br />
The fact that the Nix project continued and that blogging had so many advantages for me personally, I decided to resume my blog. The only thing that changed is that my blog was no longer in service of a research project, but just a personal means to dive into technical subjects.<br />
<br />
<h2>Reintroducing Nix to different audiences</h2>
<br />
Almost at the same time that the Nix project moved to GitHub, the <a href="https://guix.gnu.org">GNU Guix project</a> was announced: GNU Guix is a package manager with similar objectives to the Nix package manager, but with some notable differences too: instead of the Nix expression language, it uses Scheme as a configuration language.<br />
<br />
Moreover, the corresponding software packages distribution: GuixSD, exclusively provides free software.<br />
<br />
GNU Guix reuses the Nix daemon, and related components such as the Nix store from the Nix package manager to organize and isolate software packages.<br />
<br />
I wrote a <a href="https://sandervanderburg.blogspot.com/2012/11/on-nix-and-gnu-guix.html">comparison blog post</a>, that was posted on Reddit and <a href="http://news.ycombinator.com">Hackernews</a> attracting a huge number of visitors. The amount of visitors was several orders of magnitude higher than all the blog posts I have written before that. As of today, this blog post is still in my overall top 10.<br />
<br />
One of the things I did in the first month at Conference Compass is explaining the Nix package manager to my colleagues who did not have much system administration experience or knowledge about package managers.<br />
<br />
I have decided to use <a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">a programming language-centered Nix explanation recipe</a>, as opposed to a system administration-centered explanation. In many ways, I consider this explanation recipe the better of the three that I wrote.<br />
<br />
This blog post also got posted on Reddit and Hackernews attracting a huge number of visitors. In only one month, with two blog posts, I attracted more visitors to my blog than all my previous blog posts combined.<br />
<br />
<h2>Developing an app building infrastructure</h2>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtfjmT617foVmHJj3A2kxVaJItoZLIPpbhRLwFG5GKuchvH3_dU9lk1MUqDOicHCNS3xRf7DKzkpCovdsaKtVvTr704RFVPfvfxsj4lr5Mf2kvIqow5UVjY3A6nQn5u3l1ASib_lIRm23J/s1600/scr.jpg" imageanchor="1" style="margin-left:1em; margin-right:1em"><img border="0" height="400" width="312" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtfjmT617foVmHJj3A2kxVaJItoZLIPpbhRLwFG5GKuchvH3_dU9lk1MUqDOicHCNS3xRf7DKzkpCovdsaKtVvTr704RFVPfvfxsj4lr5Mf2kvIqow5UVjY3A6nQn5u3l1ASib_lIRm23J/s400/scr.jpg" /></a></div>
<br />
As explained earlier, Conference Compass was looking into developing a product-line for mobile conference apps.<br />
<br />
I did some of the work in the open, by using a variety of tools from the Nix project and making contributions to the Nix project.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2012/11/building-android-applications-with-nix.html">I have packaged many components of the Android SDK and developed a function abstraction that automatically builds Android APKs</a>. Similarly, I also built <a href="https://sandervanderburg.blogspot.com/2012/12/deploying-ios-applications-with-nix.html">a function for iOS apps</a> (that works both with the simulator and real devices), and for <a href="https://sandervanderburg.blogspot.com/2014/01/building-appcelerator-titanium-apps.html">Appcelerator Titanium</a>: a JavaScript-based cross platform framework allowing you target a variety of mobile platforms including Android and iOS.<br />
<br />
In addition to the Nix-based app building infrastructure, I have also described how you can <a href="https://sandervanderburg.blogspot.com/2013/04/setting-up-hydra-build-cluster-for.html">set up Hydra: a Nix-based continuous integration service</a> to automatically build mobile apps and other software projects.<br />
<br />
It turns out that in addition to ordinary software projects, Hydra also works well for distributing bleeding edge builds of mobile apps -- for example, you can use your phone or tablet's web browser to automatically download and install any bleeding edge build that you want.<br />
<br />
The only thing that was a bit of a challenge was <a href="https://sandervanderburg.blogspot.com/2014/08/wireless-ad-hoc-distributions-of-ios.html">distributing apps to iOS devices with Hydra</a>, but with some workarounds that was also possible.<br />
<br />
I have also developed <a href="https://sandervanderburg.blogspot.com/2017/12/controlling-hydra-server-from-nodejs.html">a Node.js package to conveniently integrate custom application with Hydra</a>.<br />
<br />
<h2>Finishing up my PhD and defending my thesis</h2>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidWejka2ZeDXteVKmcxTDQ2kPFHYpEMZm_bFipInJlB1GQQV05u4RR2pb-v5HT16oL1AeM88zX_YAvrG_BHGw2uqUtcqpO7fNw6N-BOe4ohPtnxBYa6J32vkXiaekQTcmv-k2MPiusw395/s1600/pds.png" imageanchor="1"><img border="0" height="420" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEidWejka2ZeDXteVKmcxTDQ2kPFHYpEMZm_bFipInJlB1GQQV05u4RR2pb-v5HT16oL1AeM88zX_YAvrG_BHGw2uqUtcqpO7fNw6N-BOe4ohPtnxBYa6J32vkXiaekQTcmv-k2MPiusw395/s640/pds.png" width="520" /></a>
<br />
Although I left academia, the transition to industry was actually very gradual -- as explained earlier, while being employed at Conference Compass, I still had to finish and defend my PhD thesis.<br />
<br />
Several weeks before my planned defence date, I received feedback from my reading committee about my draft that I finished in my last month at the university. This was a very stressful period -- in addition to making revisions to my PhD thesis, I also had to arrange the printing and the logistics of the ceremony.<br />
<br />
I also wrote three more blog posts about my thesis and the defence process: I provided a <a href="https://sandervanderburg.blogspot.com/2013/05/a-reference-architecture-for.html">summary of my PhD thesis as a blog post</a>, I wrote about the <a href="https://sandervanderburg.blogspot.com/2013/06/dr-sander.html">defence ceremony</a>, and about <a href="https://sandervanderburg.blogspot.com/2013/06/my-phd-thesis-propositions-and-some.html">my PhD thesis propositions</a>.<br />
<br />
Writing thesis propositions is also a tradition in the Netherlands. Earlier that year, my former colleague <a href="http:/felienne.com">Felienne Hermans</a> decided to blog and tweet about her PhD thesis propositions, and I did the same thing.<br />
<br />
PhD thesis propositions are typically not supposed to have a direct relationship to your PhD thesis, but they should be defendable. In addition to your thesis, the committee members are also allowed to ask you questions about your propositions.<br />
<br />
The blog post about my PhD thesis propositions (as of today) still regularly attracts visitors. The amount of visitors of this blog post heavily outnumbers the summary blog post about my PhD thesis.<br />
<br />
In addition to my PhD thesis, there were more interesting post-academia research events: a journal paper submission finally got officially published (4 years after submitting the first draft!) and we have managed to get our paper about discovering license compliance inconsistencies accepted at ASE 2014, that was previously rejected twice.<br />
<br />
<h2>Learning Node.js and more about JavaScript</h2>
<br />
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtnD7AG1YjcmVad6Wn5jt9OfaUXsT9FD3FIHSZV-6TmLwkz5tEjUy9EtQW1zJUXZmridWq19RIgjEjkZTnMSFaM8D2bGKyk3IXqTdmxjNKa_CF8HZrIISlGM4MuFyvnr-47453yINj7sEn/s1600/shapeprototypes.png" imageanchor="1"><img border="0" height="318" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjtnD7AG1YjcmVad6Wn5jt9OfaUXsT9FD3FIHSZV-6TmLwkz5tEjUy9EtQW1zJUXZmridWq19RIgjEjkZTnMSFaM8D2bGKyk3IXqTdmxjNKa_CF8HZrIISlGM4MuFyvnr-47453yINj7sEn/s400/shapeprototypes.png" width="520" /></a>
<br />
In addition to the app building infrastructure at Conference Compass, I have also spend considerable amounts of time learning things about <a href="http://nodejs.org">Node.js</a> and its underlying concepts: the asynchronous event loop. Although I already had some JavaScript programming experience, all my knowledge thus far was limited to the web browser.<br />
<br />
I learned about all kinds of new concepts, such as <a href="https://sandervanderburg.blogspot.com/2013/07/asynchronous-programming-with-javascript.html">callbacks</a> (and function-level scoping), <a href="https://sandervanderburg.blogspot.com/2013/12/asynchronous-programming-with.html">promises</a>, <a href="https://sandervanderburg.blogspot.com/2014/03/structured-asynchronous-programming.html">asynchronous programming (in general)</a> and <a href="https://sandervanderburg.blogspot.com/2016/01/integrating-callback-and-promise-based.html">mixing callbacks with promises</a>. Moreover, I also learned that (despite my earlier experiences in the <a href="https://sandervanderburg.blogspot.com/2011/06/concepts-of-programming-languages.html">concepts of programming languages course</a>) working with prototypes in JavaScript was more difficult than expected. I have decided to address my earlier shortcomings in my teachings with <a href="https://sandervanderburg.blogspot.com/2013/02/yet-another-blog-post-about-object.html">a blog post</a>.<br />
<br />
With Titanium (the cross-platform mobile app development framework that uses JavaScript as an implementation language), beyond regular development work, I investigated how we can <a href="https://sandervanderburg.blogspot.com/2016/08/porting-node-simple-xmpp-from-nodejs.html">port a Node.js-based XMPP library to Titanium</a> and how we can <a href="https://sandervanderburg.blogspot.com/2017/02/mvc-lessons-in-titaniumalloy.html">separate concerns well enough to make a simple, reliable chat application</a>.<br />
<br />
<h2>Building a service management platform and implementing major Disnix improvements</h2>
<br />
At Conference Compass, somewhere in the middle of 2013, we decided to shift away from a single monolithic backend application for all our apps, to a more modular approach in which each app has their own backend and their own storage.<br />
<br />
After a couple of brief experiments with Heroku, we shifted to a Nix-based approach in mid 2014. NixOps was used to automatically deploy virtual machines in the cloud (using Amazon's EC2 service), and Disnix became responsible for deploying all services to these virtual machines.<br />
<br />
In the Nix community, there was quite a bit of confusion about these two tools, because both use the Nix package manager and are designed for distributed deployment. <a href="https://sandervanderburg.blogspot.com/2015/03/on-nixops-disnix-service-deployment-and.html">I wrote a blog post to explain in what ways they are similar and different</a>.<br />
<br />
Over the course of 2015, most of my company work was concentrated on the service management platform. In addition to automating the deployment of all machines and services, I also implemented the following functionality:<br />
<br />
<ul>
<li>Backup support (<a href="https://sandervanderburg.blogspot.com/2015/07/deploying-state-with-disnix.html">using the experimental state management facilities of Dysnomia</a>)</li>
<li>Monitoring support with Datadog</li>
<li><a href="https://sandervanderburg.blogspot.com/2015/10/setting-up-basic-software-configuration.html">A general configuration management framework to organize and document all relevant configuration items</a></li>
<li>Various optimizations: <a href="https://sandervanderburg.blogspot.com/2015/08/deploying-target-specific-services-with.html">target-specific services that do not require unnecessary reconfigurations</a> and <a href="https://sandervanderburg.blogspot.com/2015/12/on-demand-service-activation-and-self.html">on-demand activation and self-termination of services</a>, to save RAM.</li>
</ul>
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVGVZlUHRw9JUsqDfPFI2x1u4VHd0KtxqYbf7KqRFAyXNn5zs_i8b1FvddN1LebkAnqnPwsbU1TLT2MuIN035Y15cqPKpCBALztmCHACRczHk_BjHHOTMgXoLgW5usW-k42w9ORCi8g-xB/s1600/visualize.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjVGVZlUHRw9JUsqDfPFI2x1u4VHd0KtxqYbf7KqRFAyXNn5zs_i8b1FvddN1LebkAnqnPwsbU1TLT2MuIN035Y15cqPKpCBALztmCHACRczHk_BjHHOTMgXoLgW5usW-k42w9ORCi8g-xB/s1600/visualize.png" width="510" /></a></div>
<br />
In late 2015, the first NixCon conference was organized, in which <a href="https://sandervanderburg.blogspot.com/2015/11/deploying-services-to-heterogeneous.html">I gave a presentation about Disnix and explained how it can be used for the deployment of microservices</a>. I received all kinds of useful feedback that I implemented in the first half of 2016:<br />
<br />
<ul>
<li>Most notably, <a href="https://sandervanderburg.blogspot.com/2016/05/mapping-services-to-containers-with.html">I changed the internal model of Disnix to also work with the notion of <strong>containers</strong></a> (environments that manage services), a feature that Dysnomia already supported, but could not be directly controlled in Disnix.</li>
<li><a href="https://sandervanderburg.blogspot.com/2016/06/deploying-containers-with-disnix-as.html">You can also manage multiple instances of container services on a single machine</a>.</li>
</ul>
<br />
Over time, I did many more interesting Disnix developments:<br />
<br />
<ul>
<li>I made modifications to use it as a <a href="https://sandervanderburg.blogspot.com/2016/06/using-disnix-as-remote-package-deployer.html">remote package deployer</a></li>
<li>I worked on <a href="https://sandervanderburg.blogspot.com/2017/01/some-programming-patterns-for-multi.html">an abstraction layer to more easily deal with concurrency</a></li>
<li>I built a tool that can <a href="https://sandervanderburg.blogspot.com/2017/03/reconstructing-disnix-deployment.html">reconstruct the Disnix deployment models from a network of already deployed services</a>.</li>
<li>I built <a href="https://sandervanderburg.blogspot.com/2018/01/diagnosing-problems-and-running.html">a tool that helps diagnosing problems</a></li>
<li>I created more public examples (based on <a href="https://sandervanderburg.blogspot.com/2018/02/deploying-systems-with-circular.html">Chord</a> and <a href="https://sandervanderburg.blogspot.com/2018/02/a-more-realistic-public-disnix-example.html">my own web framework</a>).</li>
</ul>
<br />
Furthermore, the Dynamic Disnix framework (an extension toolset that I developed for a research paper many years ago), also got all kinds of updates. For example, it was extended to <a href="https://sandervanderburg.blogspot.com/2015/07/assigning-port-numbers-to-microservices.html">automatically assign TCP/UDP port numbers</a> and to <a href="https://sandervanderburg.blogspot.com/2016/08/an-extended-self-adaptive-deployment.html">work with state migrations</a>.<br />
<br />
While working on the service management platform, five new Disnix versions were released (<a href="https://sandervanderburg.blogspot.com/2015/03/disnix-03-release-announcement.html">the first was 0.3</a>, the last 0.8). I wrote <a href="https://sandervanderburg.blogspot.com/2016/01/disnix-05-release-announcement-and-some.html">a blog post for the 0.5 release that explains all previously released versions, including the first two prototype iterations</a>.<br />
<br />
<h2>Brief return to web technology</h2>
<br />
As explained in the introduction, I already had the idea to start my blog while I was still actively doing web development.<br />
<br />
At some point I needed to make some updates to web applications that I had developed for my voluntary work that still use pieces of my old custom web framework.<br />
<br />
I already release some pieces (most notably <a href="https://sandervanderburg.blogspot.com/2014/03/implementing-consistent-layouts-for.html">the layout manager</a>) of it on my GitHub page as a side project, but at some point I have also decided to release <a href="https://sandervanderburg.blogspot.com/2017/07/some-reflections-on-my-experiences-with.html">the remainder of the components</a>.<br />
<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUT6AwR_xqtFKNTMidy6Piu8nW8dDqWfeBwy2s_E9k18F1T3SU75i1uhmDAKwmRoB3M7OpdxaMhku6nET5TuLBlNAsdJ_1zYcprlKVp6ZIn7w7CZi-PoqhMEiCMnh5XGQUXhbJEC3qPyjB/s1600/desktop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" data-original-height="718" data-original-width="1298" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjUT6AwR_xqtFKNTMidy6Piu8nW8dDqWfeBwy2s_E9k18F1T3SU75i1uhmDAKwmRoB3M7OpdxaMhku6nET5TuLBlNAsdJ_1zYcprlKVp6ZIn7w7CZi-PoqhMEiCMnh5XGQUXhbJEC3qPyjB/s640/desktop.png" width="500" /></a></div>
<br />
I also wrote a blog post about <a href="https://sandervanderburg.blogspot.com/2017/08/a-checklist-of-minimalistic-layout.html">my struggles composing a decent layout</a> and some pointers on "rational" layout decisions.<br />
<br />
<h2>Working on Nix generators</h2>
<br />
In addition to JavaScript development at Conference Compass, I was also using Nix-related tools for automating deployments of Node.js projects.<br />
<br />
Eventually, <a href="https://sandervanderburg.blogspot.com/2014/10/deploying-npm-packages-with-nix-package.html">I created node2nix</a> to make deployments with the NPM package manager possible in Nix expressions (at the time this was already possible with npm2nix, but node2nix was developed to address important shortcomings of npm2nix, such as circular dependencies).<br />
<br />
Over time, I faced many more challenges that were node2nix/NPM related:<br />
<br />
<ul>
<li>To make NPM more compatible on Windows, the NPM authors introduced <a href="https://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html">a "flattening strategy" that required a substantial rewrite of node2nix</a>.</li>
<li><a href="https://sandervanderburg.blogspot.com/2016/09/simulating-npm-global-package.html">Simulating global NPM package installations</a> in Nix expressions.</li>
<li><a href="https://sandervanderburg.blogspot.com/2017/03/subsituting-impure-version-specifiers.html">Substituting impure version specifiers</a> that may trigger accidental remote network requests.</li>
<li><a href="https://sandervanderburg.blogspot.com/2017/12/bypassing-npms-content-addressable.html">Bypassing NPM's content-addressable cache for local installations</a> (that is fundamentally incompatible with how NPM installations are done in Nix expressions).</li>
</ul>
<br />
When I released my custom web framework I also did the same for PHP. <a href="https://sandervanderburg.blogspot.com/2017/10/deploying-php-composer-packages-with.html">I have created composer2nix</a> to allow PHP composer projects to be deployed with the Nix package manager.<br />
<br />
In addition to building these generators, I also heavily invested in working towards identifying common concepts and implementation decisions for both node2nix and composer2nix.<br />
<br />
Both tools use an internal DSL to generate Nix expressions (<a href="https://sandervanderburg.blogspot.com/2013/01/nijs-internal-dsl-for-nix-in-javascript.html">NiJS for JavaScript</a>, and <a href="https://sandervanderburg.blogspot.com/2017/09/pndp-internal-dsl-for-nix-in-php.html">PNDP for PHP</a>) as opposed to using strings.<br />
<br />
Both tools implement a domain model (that is close to NPM and composer concepts) that get <a href="https://sandervanderburg.blogspot.com/2017/11/creating-custom-object-transformations.html">translated to an object structure in the Nix expression language</a> with a generic translation strategy.<br />
<br />
<h2>Joining Mendix, working on Nix concepts and improvements</h2>
<br />
Slightly over 2 years ago I <a href="https://sandervanderburg.blogspot.com/2018/05/a-new-challenge.html">joined</a> <a href="https://mendix.com">Mendix</a>, a company that develops a low-code application development platform and related services.<br />
<br />
While I was learning about Mendix, I wrote <a href="https://sandervanderburg.blogspot.com/2018/06/my-introduction-to-mendix-and-low-code.html">a blog post that explains its basic concepts</a>.<br />
<br />
In addition, as a crafting project, I also <a href="https://sandervanderburg.blogspot.com/2018/08/automating-mendix-application.html">automated the deployment of Mendix applications with Nix</a> technologies (and even <a href="https://sandervanderburg.blogspot.com/2018/07/automating-mendix-application.html">wrote about it</a> on the <a href="https://www.mendix.com/blog/">public Mendix blog</a>).<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYU00VF1K-iDJxhFehgkiopgJ_u79ry_E_WaULmSCHQ-2f9JMs5kP4nLKcKwMusH0mwXyf3ONMkW7GDOu0Ege-Dj1Jx5MKflOB8fdby_97re8q3mXwn2wXeCkJXSyal_hFBBt6-wLv5S4K/s1600/blog-nix-deployment-header%25402x-2.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjYU00VF1K-iDJxhFehgkiopgJ_u79ry_E_WaULmSCHQ-2f9JMs5kP4nLKcKwMusH0mwXyf3ONMkW7GDOu0Ege-Dj1Jx5MKflOB8fdby_97re8q3mXwn2wXeCkJXSyal_hFBBt6-wLv5S4K/s640/blog-nix-deployment-header%25402x-2.png" width="500" data-original-width="1600" data-original-height="805" /></a></div>
<br />
While learning about the Mendix cloud deployment platform, I also got heavily involved in documenting its architecture. I wrote <a href="https://sandervanderburg.blogspot.com/2019/01/a-minimalistic-discovery-and.html">a blog post about my practices</a> (the notation that I used was inspired by the diagrams that I generate with Disnix). <a href="https://sandervanderburg.blogspot.com/2019/02/generating-functional-architecture.html">I even implemented some of these ideas in the Dynamic Disnix toolset</a>.<br />
<br />
When I just joined Mendix, I was mostly learning about the company and their development stack. In my spare time, I made quite a few random Nix contributions:<br />
<br />
<ul>
<li>As a personal learning exercise and attempt to make the <i>stdenv.mkDerivation</i> function abstraction in Nix more understandable, I wrote <a href="https://sandervanderburg.blogspot.com/2018/07/layered-build-function-abstractions-for.html">layered build function abstractions</a>.</li>
<li>I also extended the lessons for building these abstractions to <a href="https://sandervanderburg.blogspot.com/2018/09/creating-nix-build-function.html">automate the deployment of SDKs with Nix</a>, most notably the Android SDK.</li>
<li>I also worked on <a href="https://sandervanderburg.blogspot.com/2018/10/auto-patching-prebuilt-binary-software.html">automating the process of patching prebuilt ELF binaries</a> so that these programs can be conveniently deployed by Nix. Most notably, it came in handy for the Android SDK.</li>
</ul>
<br />
Furthermore, I made some structural Disnix improvements as well:<br />
<br />
<ul>
<li><a href="https://sandervanderburg.blogspot.com/2019/05/a-nix-friendly-xml-based-data-exchange.html">I wrote a data exchange library</a> to more reliably consume Disnix deployment models (that are generated by Nix expressions), that also provides much better error reporting.</li>
<li><a href="https://sandervanderburg.blogspot.com/2019/08/a-new-input-model-transformation.html">I revised the Disnix model transformation pipeline</a> to improve maintainability. In addition, it also provides more configuration properties and a new model: the packages model.</li>
</ul>
<br />
<h2>Side projects</h2>
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGJmawIzZI07BZ4Iv1npG1J1rDb6XKLRFseWDw0a1RrkFPjXV8wcE9YsEGdvYhKh7Os59E920_KfqyyLlz8UosQwBZ-9rcEZw87Mi8nIBl_XDP7lxf2fkWY5lb9ibTyTDxKhoS56yXyBeU/s1600/nightflight.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgGJmawIzZI07BZ4Iv1npG1J1rDb6XKLRFseWDw0a1RrkFPjXV8wcE9YsEGdvYhKh7Os59E920_KfqyyLlz8UosQwBZ-9rcEZw87Mi8nIBl_XDP7lxf2fkWY5lb9ibTyTDxKhoS56yXyBeU/s1600/nightflight.png" /></a></div>
<br />
In addition to all the major themes above, there are also many in between projects and blog posts about all kinds of random subjects.<br />
<br />
For example, one of my long-running side projects is the <a href="https://sandervanderburg.blogspot.com/2012/06/iff-file-format-experiments.html">IFF file format experiments project</a> (a container file format commonly used on the <a href="https://sandervanderburg.blogspot.com/2011/07/second-computer.html">Commodore Amiga</a>) that I already started in the middle of my PhD.<br />
<br />
In addition to the viewer, I also developed a hacky <a href="https://sandervanderburg.blogspot.com/2012/01/porting-software-to-amigaos.html">Nix function to build software projects on AmigaOS</a>, <a href="https://sandervanderburg.blogspot.com/2013/11/emulating-amiga-display-modes.html">explained how to emulate the Amiga graphics modes</a>, <a href="https://sandervanderburg.blogspot.com/2014/05/rendering-8-bit-palettized-surfaces-in.html">ported the project to SDL 2.0</a>, and to <a href="https://sandervanderburg.blogspot.com/2014/06/porting-gnu-autotools-projects-to.html">Visual Studio so that it could run on Windows</a>.<br />
<br />
I also wrote many general Nix-related blog posts between major projects, such as:<br />
<br />
<ul>
<li><a href="https://sandervanderburg.blogspot.com/2013/06/setting-up-multi-user-nix-installation.html">How to deploy a multi-user Nix installation on conventional Linux distributions</a>.</li>
<li><a href="https://sandervanderburg.blogspot.com/2013/09/managing-user-environments-with-nix.html">Managing user environments with Nix</a>
<li><a href="https://sandervanderburg.blogspot.com/2013/09/composing-fhs-compatible-chroot.html">How to deploy games with Steam in NixOS</a></li>
<li><a href="https://sandervanderburg.blogspot.com/2013/12/using-nix-while-doing-development.html">Using Nix while doing development.</a></li>
<li><a href="https://sandervanderburg.blogspot.com/2014/07/managing-private-nix-packages-outside.html">How to easily build packages with Nix outside the Nixpkgs repository</a></li>
<li><a href="https://sandervanderburg.blogspot.com/2015/10/deploying-prebuilt-binary-software-with.html">How to patch prebuilt ELF binaries so that they can be deployed with Nix.</a></li>
<li><a href="https://sandervanderburg.blogspot.com/2015/02/a-sales-pitch-explanation-of-nixos.html">A sales pitch explanation of NixOS</a></li>
<li><a href="https://sandervanderburg.blogspot.com/2015/04/an-evaluation-and-comparison-of-snappy.html">Comparing the deployment aspects of Snappy Ubuntu with Nix</a></li>
<li><a href="https://sandervanderburg.blogspot.com/2016/10/push-and-pull-deployment-of-nix-packages.html">Push and pull deployment of Nix packages</a>
<li><a href="https://sandervanderburg.blogspot.com/2018/01/syntax-highlighting-nix-expressions-in.html">Syntax highlighting Nix expressions in mcedit</a></li>
</ul>
<br />
And covered developer's cultural aspects, such as <a href="https://sandervanderburg.blogspot.com/2015/01/agile-software-development-my.html">my experiences with Agile software development and Scrum</a>, and <a href="https://sandervanderburg.blogspot.com/2019/10/on-motivation-and-purpose.html">developer motivation</a>.<br />
<br />
<h2>Some thoughts</h2>
<br />
In this blog post, I have explained my motivation for starting my blog 10 years ago, and covered all the major projects I have been working including most of the blog posts that I have written.<br />
<br />
If you are a PhD student or a more seasoned researcher, then I would definitely encourage you to start a blog -- it gave me the following benefits:<br />
<br />
<ul>
<li>It makes your job much <strong>more interesting</strong>. All aspects of your research and teaching get attention, not just the papers, that typically only reflect over a modest sub set of your work.</li>
<li>It is a good and <strong>more accessible</strong> means to get frequent interaction with peers, practitioners, and outsiders who might have an interest in your work.</li>
<li>It <strong>improves</strong> your <strong>writing skills</strong>, which is also useful for writing papers.</li>
<li>It helps me to <strong>structure</strong> my <strong>work</strong>, by working on focused goals one at the time. You can use some of these pieces as ingredient for a research paper and/or your PhD thesis.</li>
<li>It may attract <strong>more visitors</strong> than research papers.</li>
</ul>
<br />
About the last benefit: in academia, there all kinds of metrics to measure the impact of a researcher, such as the <a href="https://en.wikipedia.org/wiki/G-index">G-index</a>, and <a href="https://en.wikipedia.org/wiki/H-index">H-index</a>. These metrics are sometimes taken very seriously, for example, by organizations that decide whether you can get a research grant or not.<br />
<br />
To give you a comparison: my most "popular" research paper titled: "Software deployment in a dynamic cloud: From device to service orientation in a hospital environment" was only downloaded (at the time of writing this blog post) 625 times from the <a href="https://dl.acm.org/doi/10.1109/CLOUD.2009.5071534">ACM digital library</a> and 240 times from <a href="https://ieeexplore.ieee.org/abstract/document/5071534">IEEE Xplore</a>. According to <a href="https://scholar.google.com/citations?user=E4JzksUAAAAJ&hl=en#d=gs_md_cita-d&u=%2Fcitations%3Fview_op%3Dview_citation%26hl%3Den%26user%3DE4JzksUAAAAJ%26citation_for_view%3DE4JzksUAAAAJ%3AzYLM7Y9cAGgC%26tzom%3D-60">Google Scholar</a>, it got 28 citations.<br />
<br />
My most popular blog post (that I wrote as an ingredient for my PhD research) is: <a href="https://sandervanderburg.blogspot.com/2011/11/on-nix-nixos-and-filesystem-hierarchy.html">On Nix, NixOS and the Filesystem Hierarchy Standard (FHS)</a> that attracted 5654 views, which is several orders of magnitude higher than my most popular research paper. In addition, I wrote several more research-related blog posts that got a comparable number of views, such as the blog post about <a href="https://sandervanderburg.blogspot.com/2013/06/my-phd-thesis-propositions-and-some.html">my PhD thesis propositions</a>.<br />
<br />
After completing my PhD research, I wrote blog posts that attracted even several orders of magnitude more visitors than the two blog posts mentioned above.<br />
<br />
(As a sidenote: I personally am not a big believer in the relevance of these numbers. What matters to me is the quality of my work, not quantity).<br />
<br />
Regularly writing for yourself as part of your job is not an observation that is unique to me. For example, the famous computer scientist <a href="https://www.cs.utexas.edu/users/EWD/">Edsger Dijkstra</a>, wrote more than 1300 manuscripts (called EWDs) about topics that he considered important, without publishing in mind.<br />
<br />
In <a href="https://www.cs.utexas.edu/users/EWD/ewd10xx/EWD1000.PDF">EWD 1000</a>, he says:<br />
<br />
<blockquote>
If there is one "scientific" discovery I am proud of, it is the discovery of the habit of writing without publication in mind. I experience it as a liberating habit: without it, doing the work becomes one thing and writing it down becomes another one, which is often viewed as an unpleasant burden. When working and writing have merged, that burden has been taken away.
</blockquote>
<br />
If you feel hesitant to start your blog, he says the following about a writer's block:<br />
<br />
<blockquote>
I only freed myself from that writer's block by the conscious decision not to write for a target audience but to write primarily to please myself.
</blockquote>
<br />
For software engineering practitioners (which I effectively became after leaving academia) a blog has benefits too:<br />
<br />
<ul>
<li>I consider good writing skills important for practitioners as well, for example to write specifications, API documentation, other technical documentation and end-user documentation. A blog helps you developing them.</li>
<li>Structuring your thoughts and work is also useful for software development projects, in particular free and open source projects.</li>
<li>It is also a good instrument to get in touch with development and open source communities. In addition to the Nix community, I also got a bit of attention in the Titanium community (with <a href="https://sandervanderburg.blogspot.com/2016/08/porting-node-simple-xmpp-from-nodejs.html">my XMPP library porting project</a>), the JavaScript community (for example, with <a href="https://sandervanderburg.blogspot.com/2013/02/yet-another-blog-post-about-object.html">my blog post about prototypes</a>) and more recently: the InfluxData community (with <a href="https://sandervanderburg.blogspot.com/2020/11/constructing-simple-alerting-system.html">my monitoring playground project</a>).</li>
</ul>
<br />
<h2>Concluding remarks</h2>
<br />
In this blog post, I covered most of my blog post written in the last decade, but I did not elaborate much about 2020. Since 2020 is a year that will definitely not go unnoticed in the history books, I will write (as an exception) an annual reflection over 2020 tomorrow.<br />
<br />
Moreover, after browsing over all my blog posts since the beginning of my blog, I also realized that it is a bit hard to find relevant old information.<br />
<br />
To alleviate that problem, I have reorganized/standardized all my labels so that you can more easily search on subjects. <a href="http://sandervanderburg.nl/index.php/blog">On my homepage</a>, I have added an overview of all labels that I am currently using.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-44941197530371707352020-11-29T21:57:00.017+01:002020-12-27T17:25:51.173+01:00Constructing a simple alerting system with well-known open source projects<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWxW6g1UXgRO_FY2UDPlZ5Yt9norTP20YlaqTlDrWYbieB9dCxiF3vpaMs3gftyCyphXDPzYlS6-nvnzYLoncprWBMmGIipHYgq40D2U2ZBmnaVG_i3DQooCjTNYSreL48W9O7trs7cE6B/s0/alertingexperiment.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" border="0" width="520" data-original-height="1152" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjWxW6g1UXgRO_FY2UDPlZ5Yt9norTP20YlaqTlDrWYbieB9dCxiF3vpaMs3gftyCyphXDPzYlS6-nvnzYLoncprWBMmGIipHYgq40D2U2ZBmnaVG_i3DQooCjTNYSreL48W9O7trs7cE6B/s0/alertingexperiment.png"/></a></div>
<br />
Some time ago, I have been experimenting with all kinds of monitoring and alerting technologies. For example, with the following technologies, I can develop a simple alerting system with relative ease:<br />
<br />
<ul>
<li><a href="https://www.influxdata.com/time-series-platform/telegraf/">Telegraf</a> is an agent that can be used to gather measurements and transfer the corresponding data to all kinds of storage solutions.</li>
<li><a href="https://www.influxdata.com/">InfluxDB</a> is a <a href="https://en.wikipedia.org/wiki/Time_series_database">time series</a> database platform that can store, manage and analyze timestamped data.</li>
<li><a href="https://www.influxdata.com/time-series-platform/kapacitor/">Kapacitor</a> is a real-time streaming data process engine, that can be used for a variety of purposes. I can use Kapacitor to analyze measurements and see if a threshold has been exceeded so that an alert can be triggered.</li>
<li><a href="https://alerta.io">Alerta</a> is a monitoring system that can store, de-duplicate alerts, and arrange black outs.</li>
<li><a href="https://grafana.com">Grafana</a> is a multi-platform open source analytics and interactive visualization web application.</li>
</ul>
<br />
These technologies appear to be quite straight forward to use. However, as I was learning more about them, I discovered a number of oddities, that may have big implications.<br />
<br />
Furthermore, testing and making incremental changes also turns out to be much more challenging than expected, making it very hard to diagnose and fix problems.<br />
<br />
In this blog post, I will describe how I built a simple monitoring and alerting system, and elaborate about my learning experiences.<br />
<br />
<h2>Building the alerting system</h2>
<br />
As described in the introduction, I can combine several technologies to create an alerting system. I will explain them more in detail in the upcoming sections.<br />
<br />
<h3>Telegraf</h3>
<br />
Telegraf is a pluggable agent that gathers measurements from a variety of <strong>inputs</strong> (such as system metrics, platform metrics, database metrics etc.) and sends them to a variety of <strong>outputs</strong>, typically storage solutions (database management systems such as InfluxDB, PostgreSQL or MongoDB). Telegraf has a large <a href="https://docs.influxdata.com/telegraf/v1.14/plugins/plugin-list/">plugin eco-system</a> that provides all kinds integrations.<br />
<br />
In this blog post, I will use InfluxDB as an output storage backend. For the inputs, I will restrict myself to capturing a sub set of system metrics only.<br />
<br />
With the following <i>telegraf.conf</i> configuration file, I can capture a variety of system metrics every 10 seconds:<br />
<br />
<pre>
[agent]
interval = "10s"
[[outputs.influxdb]]
urls = [ "http://test1:8086" ]
database = "sysmetricsdb"
username = "sysmetricsdb"
password = "sysmetricsdb"
[[inputs.system]]
# no configuration
[[inputs.cpu]]
## Whether to report per-cpu stats or not
percpu = true
## Whether to report total system cpu stats or not
totalcpu = true
## If true, collect raw CPU time metrics.
collect_cpu_time = false
## If true, compute and report the sum of all non-idle CPU states.
report_active = true
[[inputs.mem]]
# no configuration
</pre>
<br />
With the above configuration file, I can collect the following metrics:<br />
<ul>
<li>System metrics, such as the hostname and system load.</li>
<li>CPU metrics, such as how much the CPU cores on a machine are utilized, including the total CPU activity.</li>
<li>Memory (RAM) metrics.</li>
</ul>
<br />
The data will be stored in an InfluxDB database name: <i>sysmetricsdb</i> hosted on a remote machine with host name: <i>test1</i>.<br />
<br />
<h3>InfluxDB</h3>
<br />
As explained earlier, InfluxDB is a timeseries platform that can store, manage and analyze timestamped data. In many ways, InfluxDB resembles relational databases, but there are also some notable differences.<br />
<br />
The query language that InfluxDB uses is called <a href="https://docs.influxdata.com/influxdb/v1.8/query_language/">InfluxQL</a> (that shares many similarities with SQL).<br />
<br />
For example, with the following query I can retrieve the first three data points from the <i>cpu</i> measurement, that contains the CPU-related measurements collected by Telegraf:<br />
<br />
<pre>
> precision rfc3339
> select * from "cpu" limit 3
</pre>
<br />
providing me the following result set:<br />
<br />
<pre style="overflow: auto;">
name: cpu
time cpu host usage_active usage_guest usage_guest_nice usage_idle usage_iowait usage_irq usage_nice usage_softirq usage_steal usage_system usage_user
---- --- ---- ------------ ----------- ---------------- ---------- ------------ --------- ---------- ------------- ----------- ------------ ----------
2020-11-16T15:36:00Z cpu-total test2 10.665258711721098 0 0 89.3347412882789 0.10559662090813073 0 0 0.10559662090813073 0 8.658922914466714 1.79514255543822
2020-11-16T15:36:00Z cpu0 test2 10.665258711721098 0 0 89.3347412882789 0.10559662090813073 0 0 0.10559662090813073 0 8.658922914466714 1.79514255543822
2020-11-16T15:36:10Z cpu-total test2 0.1055966209080346 0 0 99.89440337909197 0 0 0 0.10559662090813073 0 0 0
</pre>
<br />
As you may probably notice by looking at the output above, every data point has a timestamp and a number of fields capturing CPU metrics:<br />
<br />
<ul>
<li><i>cpu</i> identifies the CPU core.</li>
<li><i>host</i> contains the host name of the machine.</li>
<li>The remainder of the fields contain all kinds of CPU metrics, e.g. how much CPU time is consumed by the system (<i>usage_system</i>), the user (<i>usage_user</i>), by waiting for IO (<i>usage_iowait</i>) etc.</li>
<li>The <i>usage_active</i> field contains the total CPU activity percentage, which is going to be useful to develop an alert that will warn us if there is too much CPU activity for a long period of time.</li>
</ul>
<br />
Aside from the fact that all data is timestamp based, data in InfluxDB has another notable difference compared to relational databases: an InfluxDB database is <strong>schemaless</strong>. You can add an arbitrary number of fields and tags to a data point without having to adjust the database structure (and migrating existing data to the new database structure).<br />
<br />
Fields and tags can contain arbitrary data, such as numeric values or strings. Tags are also <strong>indexed</strong> so that you can search for these values more efficiently. Furthermore, tags can be used to group data.<br />
<br />
For example, the <i>cpu</i> measurement collection has the following tags:<br />
<br />
<pre>
> SHOW TAG KEYS ON "sysmetricsdb" FROM "cpu";
name: cpu
tagKey
------
cpu
host
</pre>
<br />
As shown in the above output, the <i>cpu</i> and <i>host</i> fields are tags in the <i>cpu</i> measurement.<br />
<br />
We can use these tags to search for all data points related to a CPU core and/or host machine. Moreover, we can use these tags for grouping allowing us to compute aggregate values, sch as the mean value per CPU core and host.<br />
<br />
Beyond storing and retrieving data, InfluxDB has many useful additional features:<br />
<br />
<ul>
<li>You can also automatically <a href="https://docs.influxdata.com/influxdb/v1.8/query_language/sample-data/"><strong>sample</strong> data</a> and run <a href="https://docs.influxdata.com/influxdb/v1.8/query_language/continuous_queries/"><strong>continuous queries</strong></a> that generate and store sampled data in the background.</li>
<li>Configure <a href="https://docs.influxdata.com/influxdb/v1.8/guides/downsample_and_retain/"><strong>retention policies</strong></a> so that data is no longer stored for an indefinite amount of time. For example, you can configure a retention policy to drop raw data after a certain amount of time, but retain the corresponding sampled data.</li>
</ul>
<br />
InfluxDB has a "open core" development model. The free and open source edition (FOSS) of InfluxDB server (that is <a href="https://opensource.org/licenses/MIT">MIT licensed</a>) allows you to host multiple databases on a multiple servers.<br />
<br />
However, if you also want <strong>horizontal scalability</strong> and/or <strong>high assurance</strong>, then you need to switch to the hosted InfluxDB versions -- data in InfluxDB is partitioned into so-called <strong>shards</strong> of a fixed size (the default shard size is 168 hours).<br />
<br />
These shards can be distributed over multiple InfluxDB servers. It is also possible to deploy multiple <strong>read replicas</strong> of the same shard to multiple InfluxDB servers improving read speed.<br />
<br />
<h3>Kapacitor</h3>
<br />
Kapacitor is a real-time streaming data process engine developed by InfluxData -- the same company that also develops InfluxDB and Telegraf.<br />
<br />
It can be used for all kinds of purposes. In my example cases, I will only use it to determine whether some threshold has been exceeded and an alert needs to be triggered.<br />
<br />
Kapacitor works with customly implemented <strong>tasks</strong> that are written in a domain-specific language called the <a href="https://docs.influxdata.com/kapacitor/v1.5/tick/introduction/">TICK script language</a>. There are two kinds of tasks: <strong>stream</strong> and <strong>batch</strong> tasks. <a href="https://www.influxdata.com/blog/batch-processing-vs-stream-processing/">Both task types have advantages and disadvantages</a>.<br />
<br />
We can easily develop an alert that gets triggered if the CPU activity level is high for a relatively long period of time (more than 75% on average over 1 minute).<br />
<br />
To implement this alert as a stream job, we can write the following TICK script:<br />
<br />
<pre style="overflow: auto;">
dbrp "sysmetricsdb"."autogen"
stream
|from()
.measurement('cpu')
.groupBy('host', 'cpu')
.where(lambda: "cpu" != 'cpu-total')
|window()
.period(1m)
.every(1m)
|mean('usage_active')
|alert()
.message('Host: {{ index .Tags "host" }} has high cpu usage: {{ index .Fields "mean" }}')
.warn(lambda: "mean" > 75.0)
.crit(lambda: "mean" > 85.0)
.alerta()
.resource('{{ index .Tags "host" }}/{{ index .Tags "cpu" }}')
.event('cpu overload')
.value('{{ index .Fields "mean" }}')
</pre>
<br />
A stream job is built around the following principles:<br />
<br />
<ul>
<li>A stream task does not execute queries on an InfluxDB server. Instead, it creates a <strong>subscription</strong> to InfluxDB -- whenever a data point gets inserted into InfluxDB, the data points gets forwarded to Kapacitor as well.<br />
<br />
To make subscriptions work, both InfluxDB and Kapacitor need to be able to connect to each other with a public IP address.</li>
<li>A stream task defines a <strong>pipeline</strong> consisting of a number of <strong>nodes</strong> (connected with the <i>|</i> operator). Each node can <strong>consume</strong> data points, filter, transform, aggregate, or execute arbitrary operations (such as calling an external service), and <strong>produce</strong> new data points that can be propagated to the next node in the pipeline.</li>
<li>Every node also has <strong>property methods</strong> (such as <i>.measurement('cpu')</i>) making it possible to configure parameters.</li>
</ul>
<br />
The TICK script example shown above does the following:<br />
<br />
<ul>
<li>The <i>from</i> node consumes <i>cpu</i> data points from the InfluxDB subscription, groups them by <i>host</i> and <i>cpu</i> and filters out data points with the the <i>cpu-total</i> label, because we are only interested in the CPU consumption per core, not the total amount.</li>
<li>The <i>window</i> node states that we should aggregate data points over the last 1 minute and pass the resulting (aggregated) data points to the next node after one minute in time has elapsed. To aggregate data, Kapacitor will buffer data points in memory.</li>
<li>The <i>mean</i> node computes the mean value for <i>usage_active</i> for the aggregated data points.</li>
<li>The <i>alert</i> node is used to trigger an alert of a specific severity level (WARNING if the mean activity percentage is bigger than 75%) and (CRITICAL if the mean activity percentage is bigger than 85%). In the remainder of the case, the status is considered OK. The alert is sent to Alerta.</li>
</ul>
<br />
It is also possible to write a similar kind of alerting script as a batch task:<br />
<br />
<pre style="overflow: auto;">
dbrp "sysmetricsdb"."autogen"
batch
|query('''
SELECT mean("usage_active")
FROM "sysmetricsdb"."autogen"."cpu"
WHERE "cpu" != 'cpu-total'
''')
.period(1m)
.every(1m)
.groupBy('host', 'cpu')
|alert()
.message('Host: {{ index .Tags "host" }} has high cpu usage: {{ index .Fields "mean" }}')
.warn(lambda: "mean" > 75.0)
.crit(lambda: "mean" > 85.0)
.alerta()
.resource('{{ index .Tags "host" }}/{{ index .Tags "cpu" }}')
.event('cpu overload')
.value('{{ index .Fields "mean" }}')
</pre>
<br />
The above TICK script looks similar to the stream task shown earlier, but instead of using a subscription, the script queries the InfluxDB database (with an InfluxQL query) for data points over the last minute with a <i>query</i> node.<br />
<br />
Which approach for writing a CPU alert is best, you may wonder? Each of these two approaches have their pros and cons:<br />
<br />
<ul>
<li>Stream tasks offer <strong>low latency</strong> responses -- when a data point appears, a stream task can immediately respond, whereas a batch task needs to query every minute all the data points to compute the mean percentage over the last minute.</li>
<li>Stream tasks maintain a buffer for aggregating the data points making it possible to only send <strong>incremental</strong> updates to Alerta. Batch tasks are stateless. As a result, they need to update the status of all hosts and CPUs every minute.</li>
<li>Processing data points is done synchronously and in sequential order -- if an update round to Alerta takes too long (which is more likely to happen with a batch task), then the next processing run may overlap with the previous, causing all kinds of unpredictable results.<br />
<br />
It may also cause Kapacitor to eventually crash due to growing resource consumption.</li>
<li>Batch tasks may also <strong>miss</strong> data points -- while querying data over a certain time window, it may happen that a new data point gets inserted in that time window (that is being queried). This new data point will not be picked up by Kapacitor.<br />
<br />
A subscription made by a stream task, however, will never miss any data points.</li>
<li>Stream tasks can only work with data points that appear from the moment Kapacitor is started -- it <strong>cannot work</strong> with data points in the <strong>past</strong>.<br />
<br />
For example, if Kapacitor is restarted and some important event is triggered in the restart time window, Kapacitor will not notice that event, causing the alert to remain in its previous state.<br />
<br />
To work effectively with stream tasks, a <strong>continuous</strong> data stream is required that frequently reports on the status of a resource. Batch tasks, on the other hand, can work with historical data.</li>
<li>The fact that nodes maintain a buffer may also cause the <strong>RAM consumption</strong> of Kapacitor to grow considerably, if the data volumes are big.<br />
<br />
A batch task on the other hand, does not buffer any data and is more memory efficient.<br />
<br />
Another compelling advantage of batch tasks over stream tasks is that InfluxDB does all the work. The hosted version of InfluxDB can also horizontally scale.</li>
<li>Batch tasks can also <strong>aggregate</strong> data more efficiently (e.g. computing the mean value or sum of values over a certain time period).</li>
</ul>
<br />
I consider neither of these script types the optimal solution. However, for implementing the alerts I tend to have a slight preference for stream jobs, because of its low latency, and incremental update properties.<br />
<br />
<h3>Alerta</h3>
<br />
As explained in the introduction, Alerta is a monitoring system that can store and de-duplicate alerts, and arrange black outs.<br />
<br />
The Alerta server provides a REST API that can be used to query and modify alerting data and uses MongoDB or PostgreSQL as a storage database.<br />
<br />
There are also a variety of Alerta clients: there is the <i>alerta-cli</i> allows you to control the service from the command-line. There is also a web user interface that I will show later in this blog post.<br />
<br />
<h2>Running experiments</h2>
<br />
With all the components described above in place, we can start running experiments to see if the CPU alert will work as expected. To gain better insights in the process, I can install Grafana that allows me to visualize the measurements that are stored in InfluxDB.<br />
<br />
Configuring a dashboard and panel for visualizing the CPU activity rate was straight forward. I configured a new dashboard, with the following variables:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaT1tw7nTL0PFgaAGjgAfYyZyWJiehm-k7Cyv5H85GwjAM38UeUYub3GnbAO9-gLl2UOuIaoplmHuinnGMIxdllbdsHlza4YwhUKLP9woy0FVGSILY2FwUAY5Fjc0bBXKA-pS-278s1XqR/s0/variables.png" style="display: block; padding: 1em 0; text-align: center; "imageanchor="1"><img alt="" border="0" width="520" data-original-height="969" data-original-width="1387" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiaT1tw7nTL0PFgaAGjgAfYyZyWJiehm-k7Cyv5H85GwjAM38UeUYub3GnbAO9-gLl2UOuIaoplmHuinnGMIxdllbdsHlza4YwhUKLP9woy0FVGSILY2FwUAY5Fjc0bBXKA-pS-278s1XqR/s0/variables.png"/></a></div>
<br />
The above variables allow me to select for each machine in the network, which CPU core's activity percentage I want to visualize.<br />
<br />
I have configured the CPU panel as follows:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiTA9hyp-JCEEjNtxkyohmR2Nbk_mx6KaElifZ4zJ3gJr95-I26Cg_s_9NbRhZF_KCwUoQtkqQ0xX1m9vOyHp-uCK-WJJ1yJ0XxrYAaxZSsTudTlqGE0sM2GDUQrXph2oNpABcBn0UP49I/s0/panelconfig.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" border="0" width="520" data-original-height="969" data-original-width="1387" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjiTA9hyp-JCEEjNtxkyohmR2Nbk_mx6KaElifZ4zJ3gJr95-I26Cg_s_9NbRhZF_KCwUoQtkqQ0xX1m9vOyHp-uCK-WJJ1yJ0XxrYAaxZSsTudTlqGE0sM2GDUQrXph2oNpABcBn0UP49I/s0/panelconfig.png"/></a></div>
<br />
In the above configuration, I query the <i>usage_activity</i> from the <i>cpu</i> measurement collection, using the dashboard variables: <i>cpu</i> and <i>host</i> to filter for the right target machine and CPU core.<br />
<br />
I have also configured the field unit to be a percentage value (between 0 and 100).<br />
<br />
When running the following command-line instruction on a test machine that runs Telegraf (<i>test2</i>), <a href="https://stackoverflow.com/questions/2925606/how-to-create-a-cpu-spike-with-a-bash-command">I can deliberately hog the CPU</a>:<br />
<br />
<pre>
$ dd if=/dev/zero of=/dev/null
</pre>
<br />
The above command reads zero bytes (one-by-one) and discards them by sending them to <i>/dev/null</i>, causing the CPU to remain utilized at a high level:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMXjeTxn8eRwJEWfB5znmYCpNdWLMNXR4-WFeIJVO4NtDWQM36YgZvZszlti1ytj14jq5adkcCsFwxniEKK0dLbWQ72dt4W01W1KS9tLev-FRY7yhsb7zdBlCDnCnytVDISeKG_ldLf1cW/s0/hogcpu.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" border="0" width="520" data-original-height="409" data-original-width="852" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiMXjeTxn8eRwJEWfB5znmYCpNdWLMNXR4-WFeIJVO4NtDWQM36YgZvZszlti1ytj14jq5adkcCsFwxniEKK0dLbWQ72dt4W01W1KS9tLev-FRY7yhsb7zdBlCDnCnytVDISeKG_ldLf1cW/s0/hogcpu.png"/></a></div>
<br />
In the graph shown above, it is clearly visible that CPU core 0 on the <i>test2</i> machine remains utilized at 100% for several minutes.<br />
<br />
(As a sidenote, we can also <a href="https://stackoverflow.com/questions/20200982/how-to-generate-a-memory-shortage-using-bash-script/34755981">hog both the CPU and consume RAM at the same time</a> with a simple command line instruction).<br />
<br />
If we keep hogging the CPU and wait for at least a minute, the Alerta web interface dashboard will show a CRITICAL alert:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEig4lcebuMOo7wiSg381nk-0qPnuLPBJ0EG9w2LaXKnCWTS3AJQKohFoY23h3cqFH0DSJB-Sdrc4rHIGbIypGr0hABWjd0SrVTJw6YAJmJ9P4s9VwEkqobog3LMMe7uajXA0Gk7AN05xK1P/s0/criticalalert.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" width="520" border="0" data-original-height="745" data-original-width="1253" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEig4lcebuMOo7wiSg381nk-0qPnuLPBJ0EG9w2LaXKnCWTS3AJQKohFoY23h3cqFH0DSJB-Sdrc4rHIGbIypGr0hABWjd0SrVTJw6YAJmJ9P4s9VwEkqobog3LMMe7uajXA0Gk7AN05xK1P/s0/criticalalert.png"/></a></div>
<br />
If we stop the <i>dd</i> command, then the TICK script should eventually notice that the mean percentage drops below the WARNING threshold causing the alert to go back into the <i>OK</i> state and disappearing from the Alerta dashboard.<br />
<br />
<h2>Developing test cases</h2>
<br />
Being able to trigger an alert with a simple command-line instruction is useful, but not always convenient or effective -- one of the inconveniences is that we always have to wait at least one minute to get feedback.<br />
<br />
Moreover, when an alert does not work, it is not always easy to find the root cause. I have encountered the following problems that contribute to a failing alert:<br />
<br />
<ul>
<li>Telegraf may not be running and, as a result, not capturing the data points that need to be analyzed by the TICK script.</li>
<li>A subscription cannot be established between InfluxDB and Kapacitor. This may happen when Kapacitor cannot be reached through a public IP address.</li>
<li>There are data points collected, but only the wrong kinds of measurements.</li>
<li>The TICK script is functionally incorrect.</li>
</ul>
<br />
Fortunately, for stream tasks it is relatively easy to quickly find out whether an alert is functionally correct or not -- we can generate test cases that almost instantly trigger each possible outcome with a minimal amount of data points.<br />
<br />
An interesting property of stream tasks is that they have no notion of time -- the <i>.window(1m)</i> property may suggest that Kapacitor computes the mean value of the data points every minute, but that is not what it actually does. Instead, Kapacitor only looks at the timestamps of the data points that it receives.<br />
<br />
When Kapacitor sees that the timestamps of the data points fit in the 1 minute time window, then it keeps buffering. As soon as a data point appears that is outside this time window, the <i>window</i> node relays an aggregated data point to the next node (that computes the mean value, than in turn is consumed by the alert node deciding whether an alert needs to be raised or not).<br />
<br />
We can exploit that knowledge, to create a very minimal bash test script that triggers every possible outcome: OK, WARNING and CRITICAL:<br />
<br />
<pre style="font-size: 90%; overflow: auto;">
influxCmd="influx -database sysmetricsdb -host test1"
export ALERTA_ENDPOINT="http://test1"
### Trigger CRITICAL alert
# Force the average CPU consumption to be 100%
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=100 0000000000"
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=100 60000000000"
# This data point triggers the alert
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=100 120000000000"
sleep 1
actualSeverity=$(alerta --output json query | jq '.[0].severity')
if [ "$actualSeverity" != "critical" ]
then
echo "Expected severity: critical, but we got: $actualSeverity" >&2
false
fi
### Trigger WARNING alert
# Force the average CPU consumption to be 80%
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=80 180000000000"
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=80 240000000000"
# This data point triggers the alert
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=80 300000000000"
sleep 1
actualSeverity=$(alerta --output json query | jq '.[0].severity')
if [ "$actualSeverity" != "warning" ]
then
echo "Expected severity: warning, but we got: $actualSeverity" >&2
false
fi
### Trigger OK alert
# Force the average CPU consumption to be 0%
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=0 300000000000"
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=0 360000000000"
# This data point triggers the alert
$influxCmd -execute "INSERT cpu,cpu=cpu0,host=test2 usage_active=0 420000000000"
sleep 1
actualSeverity=$(alerta --output json query | jq '.[0].severity')
if [ "$actualSeverity" != "ok" ]
then
echo "Expected severity: ok, but we got: $actualSeverity" >&2
false
fi
</pre>
<br />
The shell script shown above automatically triggers all three possible outcomes of the CPU alert:<br />
<br />
<ul>
<li>CRITICAL is triggered by generating data points that force a mean activity percentage of 100%.</li>
<li>WARNING is triggered by a mean activity percentage of 80%.</li>
<li>OK is triggered by a mean activity percentage of 0%.</li>
</ul>
<br />
It uses the Alerta CLI to connect to the Alerta server to check whether the alert's severity level has the expected value.<br />
<br />
We need three data points to trigger each alert type -- the first two data points are on the boundaries of the 1 minute window (0 seconds and 60 seconds), forcing the mean value to become the specified CPU activity percentage.<br />
<br />
The third data point is deliberately outside the time window (of 1 minute), forcing the alert node to be triggered with a mean value over the previous two data points.<br />
<br />
Although the above test strategy works to quickly validate all possible outcomes, one impractical aspect is that the timestamps in the above example start with 0 (meaning 0 seconds after the epoch: January 1st 1970 00:00 UTC).<br />
<br />
If we also want to observe the data points generated by the above script in Grafana, we need to configure the panel to go back in time 50 years.<br />
<br />
Fortunately, I can also easily adjust the script to start with a base timestamp, that is 1 hour in the past:<br />
<br />
<pre>
offset="$(($(date +%s) - 3600))"
</pre>
<br />
With this tiny adjustment, we should see the following CPU graph (displaying data points from the last hour) after running the test script:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiF30n7_ebGzq8EpRgxB0rM4V4z734eevDwVCCwJ5BHSklQpebiyI1mg9driz7Clo5cJk-imXw3CI8a2XIKeqlIXJsiaXN756fKeO9b8wHrS6QiuXSxZlQ_rSOn1JS7WAc9NACVKM4GPxrI/s0/testcpugraph.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" border="0" width="520" data-original-height="412" data-original-width="722" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiF30n7_ebGzq8EpRgxB0rM4V4z734eevDwVCCwJ5BHSklQpebiyI1mg9driz7Clo5cJk-imXw3CI8a2XIKeqlIXJsiaXN756fKeO9b8wHrS6QiuXSxZlQ_rSOn1JS7WAc9NACVKM4GPxrI/s0/testcpugraph.png"/></a></div>
<br />
As you may notice, we can see that the CPU activity level quickly goes from 100%, to 80%, to 0%, using only 9 data points.<br />
<br />
Although testing stream tasks (from a functional perspective) is quick and convenient, testing batch tasks in a similar way is difficult. Contrary to the stream task implementation, the <i>query</i> node in the batch task does have a notion of time (because of the <i>WHERE</i> clause that includes the <i>now()</i> expression).<br />
<br />
Moreover, the embedded InfluxQL query evaluates the mean values every minute, but the test script does not exactly know when this event triggers.<br />
<br />
The only way I could think of to (somewhat reliably) validate the outcomes is by creating a test script that continuously inserts data points for at least double the time window size (2 minutes) until Alerta reports the right alert status (if it does not after a while, I can conclude that the alert is incorrectly implemented).<br />
<br />
<h2>Automating the deployment</h2>
<br />
As you may probably have already guessed, to be able to conveniently experiment with all these services, and to reliably run tests in isolation, some form of <strong>deployment automation</strong> is an absolute must-have.<br />
<br />
Most people who do not know anything about my deployment technology preferences, will probably go for <a href="https://docker.com">Docker</a> or <a href="https://docs.docker.com/compose/">docker-compose</a>, but I have decided to use a variety of solutions from the <a href="https://nixos.org">Nix project</a>.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2015/03/on-nixops-disnix-service-deployment-and.html">NixOps</a> is used to automatically deploy a network of <a href="https://sandervanderburg.blogspot.com/2011/01/nixos-purely-functional-linux.html">NixOS</a> machines -- I have created a logical and physical NixOps configuration that deploys two VirtualBox virtual machines.<br />
<br />
With the following command I can create and deploy the virtual machines:<br />
<br />
<pre>
$ nixops create network.nix network-virtualbox.nix -d test
$ nixops deploy -d test
</pre>
<br />
The first machine: <i>test1</i> is responsible for hosting the entire monitoring infrastructure (InfluxDB, Kapacitor, Alerta, Grafana), and the second machine (<i>test2</i>) runs Telegraf and the load tests.<br />
<br />
<a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a> (my own deployment tool) is responsible for deploying all services, such as InfluxDB, Kapacitor, Alarta, and the database storage backends. Contrary to docker-compose, Disnix does not work with containers (or other Docker objects, such as networks or volumes), but with arbitrary deployment units that are managed with a plugin system called <a href="https://sandervanderburg.blogspot.com/2012/03/deployment-of-mutable-components.html">Dysnomia</a>.<br />
<br />
Moreover, Disnix can also be used for distributed deployment in a network of machines.<br />
<br />
I have packaged all the services and captured them in a Disnix services model that specifies all deployable services, their types, and their inter-dependencies.<br />
<br />
If I combine the services model with the NixOps network models, and a distribution model (that maps Telegraf and the test scripts to the <i>test2</i> machine and the remainder of the services to the first: <i>test1</i>), I can deploy the entire system:<br />
<br />
<pre>
$ export NIXOPS_DEPLOYMENT=test
$ export NIXOPS_USE_NIXOPS=1
$ disnixos-env -s services.nix \
-n network.nix \
-n network-virtualbox.nix \
-d distribution.nix
</pre>
<br />
The following diagram shows a possible deployment scenario of the system:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOwqpehzJJWlAEs0OzHRjd8uEt8iI4EiQw3JkxQw-qCL0HxrSymUsWkOyYeT_YVAG1I8DHbARtwTi2cmV9LcgdhUDzKtIJ_RTHDEFc4UYr_Qjyw0Lf6gG9oqJvyxUi9Zb9bwwnfYTdgDBh/s0/deploymentarch.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" border="0" width="520" data-original-height="322" data-original-width="2311" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhOwqpehzJJWlAEs0OzHRjd8uEt8iI4EiQw3JkxQw-qCL0HxrSymUsWkOyYeT_YVAG1I8DHbARtwTi2cmV9LcgdhUDzKtIJ_RTHDEFc4UYr_Qjyw0Lf6gG9oqJvyxUi9Zb9bwwnfYTdgDBh/s0/deploymentarch.png"/></a></div>
<br />
The above diagram describes the following properties:<br />
<br />
<ul>
<li>The light-grey colored boxes denote <strong>machines</strong>. In the above diagram, we have two of them: <i>test1</i> and <i>test2</i> that correspond to the VirtualBox machines deployed by NixOps.</li>
<li>The dark-grey colored boxes denote <a href="https://sandervanderburg.blogspot.com/2016/05/mapping-services-to-containers-with.html"><strong>containers</strong> in a Disnix-context</a> (not to be confused with Linux or Docker containers). These are environments that manage other services.<br />
<br />
For example, a container service could be the PostgreSQL DBMS managing a number of PostgreSQL databases or the Apache HTTP server managing web applications.</li>
<li>The ovals denote <strong>services</strong> that could be any kind of deployment unit. In the above example, we have services that are running processes (managed by systemd), databases and web applications.</li>
<li>The arrows denote <strong>inter-dependencies</strong> between services. When a service has an inter-dependency on another service (i.e. the arrow points from the former to the latter), then the latter service needs to be activated first. Moreover, the former service also needs to know how the latter can be reached.</li>
<li>Services can also be <a href="https://sandervanderburg.blogspot.com/2020/04/deploying-container-and-application.html"><strong>container providers</strong></a> (as denoted by the arrows in the labels), stating that other services can be embedded inside this service.<br />
<br />
As already explained, the PostgreSQL DBMS is an example of such a service, because it can host multiple PostgreSQL databases.</li>
</ul>
<br />
Although the process components in the diagram above can also be conveniently deployed with Docker-based solutions (i.e. as I have explained in an earlier blog post, <a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">containers are somewhat confined and restricted processes</a>), the non-process integrations need to be managed by other means, such as writing extra shell instructions in Dockerfiles.<br />
<br />
In addition to deploying the system to machines managed by NixOps, it is also possible to use the <a href="https://sandervanderburg.blogspot.com/2011/02/using-nixos-for-declarative-deployment.html">NixOS test driver</a> -- the NixOS test driver automatically generates QEMU virtual machines with a shared Nix store, so that no disk images need to be created, making it possible to quickly spawn networks of virtual machines, with very small storage footprints.<br />
<br />
I can also create a minimal distribution model that only deploys the services required to run the test scripts -- Telegraf, Grafana and the front-end applications are not required, resulting in a much smaller deployment:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3t4Z5JfHl6O0TfPTvVlY34W2ijm-yEiO1StMpzKonfD9t-xZnqurNcJOXHvVzN8baXy5XFQCKpPkLCMPseqyLaziH_hnNNHvtrGczt4lPnaGNpJ8RaTv0-pz4gy4wRJBG32rL-HCQ3lOQ/s0/deploymentarch-minimal.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" border="0" width="520" data-original-height="429" data-original-width="1053" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEh3t4Z5JfHl6O0TfPTvVlY34W2ijm-yEiO1StMpzKonfD9t-xZnqurNcJOXHvVzN8baXy5XFQCKpPkLCMPseqyLaziH_hnNNHvtrGczt4lPnaGNpJ8RaTv0-pz4gy4wRJBG32rL-HCQ3lOQ/s0/deploymentarch-minimal.png"/></a></div>
<br />
As can be seen in the above diagram, there are far fewer components required.<br />
<br />
In this virtual network that runs a minimal system, we can run automated tests for rapid feedback. For example, the following test driver script (implemented in Python) will run my test shell script shown earlier:<br />
<br />
<pre>
test2.succeed("test-cpu-alerts")
</pre>
<br />
With the following command I can automatically run the tests on the terminal:<br />
<br />
<pre>
$ nix-build release.nix -A tests
</pre>
<br />
<h2>Availability</h2>
<br />
The deployment recipes, test scripts and documentation describing the configuration steps are stored in the <a href="https://github.com/svanderburg/monitoring-playground">monitoring playground repository</a> that can be obtained from my GitHub page.<br />
<br />
Besides the CPU activity alert described in this blog post, I have also developed a memory alert that triggers if too much RAM is consumed for a longer period of time.<br />
<br />
In addition to virtual machines and services, there is also deployment automation in place allowing you also easily deploy Kapacitor TICK scripts and Grafana dashboards.<br />
<br />
To deploy the system, you need to use the very latest version of Disnix (version 0.10) that was released very recently.<br />
<br />
<h2>Acknowledgements</h2>
<br />
I would like to thank my employer: <a href="https://mendix.com">Mendix</a> for writing this blog post. Mendix allows developers to work two days per month on research projects, making projects like these possible.<br />
<br />
<h2>Presentation</h2>
<br />
I have given a presentation about this subject at Mendix. For convienence, I have embedded the slides:<br />
<br />
<iframe src="//www.slideshare.net/slideshow/embed_code/key/10fJKf8ywJ2HKS" width="595" height="485" frameborder="0" marginwidth="0" marginheight="0" scrolling="no" style="border:1px solid #CCC; border-width:1px; margin-bottom:5px; max-width: 100%;" allowfullscreen> </iframe> <div style="margin-bottom:5px"> <strong> <a href="//www.slideshare.net/sandervanderburg/the-monitoring-playground" title="The Monitoring Playground" target="_blank">The Monitoring Playground</a> </strong> from <strong><a href="https://www.slideshare.net/sandervanderburg" target="_blank">Sander van der Burg</a></strong> </div>
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-56019576714365362812020-10-31T16:05:00.003+01:002020-12-19T01:15:47.865+01:00Building multi-process Docker images with the Nix process management frameworkSome time ago, I have described <a href="https://sandervanderburg.blogspot.com/2019/11/a-nix-based-functional-organization-for.html">my experimental Nix-based process management framework</a> that makes it possible to automatically <strong>deploy</strong> running <strong>processes</strong> (sometimes also ambiguously called services) from declarative specifications written in the <a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix expression language</a>.<br />
<br />
The framework is built around two concepts. As its name implies, the <a href="https://sandervanderburg.blogspot.com/2011/01/nix-package-manager.html"><strong>Nix package manager</strong></a> is used to deploy all required packages and static artifacts, and a <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html"><strong>process manager</strong> of choice</a> (e.g. sysvinit, systemd, supervisord and others) is used to manage the life-cycles of the processes.<br />
<br />
Moreover, it is built around <strong>flexible concepts</strong> allowing integration with solutions that are not qualified as process managers (but can still be used as such), such as <a href="https://docker.com">Docker</a> -- <a href="https://sandervanderburg.blogspot.com/2020/08/experimenting-with-nix-and-service.html">each process instance can be deployed as a Docker container</a> with a shared Nix store using the host system's network.<br />
<br />
As explained in <a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">an earlier blog post</a>, Docker has become such a popular solution that it has become a standard for deploying (micro)services (often as a utility in the <a href="https://kubernetes.io">Kubernetes</a> solution stack).<br />
<br />
When deploying a system that consists of multiple services with Docker, a typical strategy (and recommended practice) is to use multiple containers that have <a href="https://runnable.com/docker/rails/run-multiple-processes-in-a-container">only one root application process</a>. Advantages of this approach is that Docker can control the life-cycles of the applications, and that each process is (somewhat) isolated/protected from other processes and the host system.<br />
<br />
By default, containers are isolated, but if they need to interact with other processes, then they can use all kinds of <strong>integration</strong> facilities -- for example, they can share namespaces, or use shared volumes.<br />
<br />
In some situations, it may also be desirable to <strong>deviate</strong> from the one root process per container practice -- for some systems, processes may need to interact quite intensively (e.g. with IPC mechanisms, shared files or shared memory, or a combination these) in which the container boundaries introduce more inconveniences than benefits.<br />
<br />
Moreover, when running multiple processes in a single container, common dependencies can also typically be more efficiently shared leading to lower disk and RAM consumption.<br />
<br />
As explained in my previous blog post (that explores various Docker concepts), sharing dependencies between containers only works if containers are constructed from images that share the same layers with the same shared libraries. In practice, this form of sharing is not always as efficient as we want it to be.<br />
<br />
Configuring a Docker image to run multiple application processes is somewhat cumbersome -- <a href="https://docs.docker.com/config/containers/multi-service_container/">the official Docker documentation</a> describes two solutions: one that relies on a <strong>wrapper</strong> script that starts multiple processes in the background and a loop that waits for the "main process" to terminate, and the other is to use a <strong>process manager</strong>, such as supervisord.<br />
<br />
I realised that I could solve this problem much more conveniently by combining the <i>dockerTools.buildImage {}</i> function in Nixpkgs (that builds Docker images with the Nix package manager) with the Nix process management abstractions.<br />
<br />
I have created my own abstraction function: <i>createMultiProcessImage</i> that builds multi-process Docker images, managed by any supported process manager that works in a Docker container.<br />
<br />
In this blog post, I will describe how this function is implemented and how it can be used.<br />
<br />
<h2>Creating images for single root process containers</h2>
<br />
As shown in earlier blog posts, creating a Docker image with Nix for a single root application process is very straight forward.<br />
<br />
For example, we can build an image that launches a trivial web application service with an embedded HTTP server (as shown in many of my previous blog posts), as follows:<br />
<br />
<pre>
{dockerTools, webapp}:
dockerTools.buildImage {
name = "webapp";
tag = "test";
runAsRoot = ''
${dockerTools.shadowSetup}
groupadd webapp
useradd webapp -g webapp -d /dev/null
'';
config = {
Env = [ "PORT=5000" ];
Cmd = [ "${webapp}/bin/webapp" ];
Expose = {
"5000/tcp" = {};
};
};
}
</pre>
<br />
The above Nix expression (<i>default.nix</i>) invokes the <i>dockerTools.buildImage</i> function to automatically construct an image with the following properties:<br />
<br />
<ul>
<li>The image has the following name: <i>webapp</i> and the following version tag: <i>test</i>.</li>
<li>The web application service requires some <strong>state</strong> to be initialized before it can be used. To configure state, we can run instructions in a QEMU virual machine with root privileges (<i>runAsRoot</i>).<br />
<br />
In the above deployment Nix expression, we create an unprivileged user and group named: <i>webapp</i>. For production deployments, it is typically recommended to drop root privileges, for security reasons.</li>
<li>The <i>Env</i> directive is used to configure environment variables. The <i>PORT</i> environment variable is used to configure the TCP port where the service should bind to.</li>
<li>The <i>Cmd</i> directive starts the <i>webapp</i> process in foreground mode. The life-cycle of the container is bound to this application process.</li>
<li><i>Expose</i> exposes TCP port 5000 to the public so that the service can respond to requests made by clients.</li>
</ul>
<br />
We can build the Docker image as follows:<br />
<br />
<pre>
$ nix-build
</pre>
<br />
load it into Docker with the following command:<br />
<br />
<pre>
$ docker load -i result
</pre>
<br />
and launch a container instance using the image as a template:<br />
<br />
<pre>
$ docker run -it -p 5000:5000 webapp:test
</pre>
<br />
If the deployment of the container succeeded, we should get a response from the <i>webapp</i> process, by running:<br />
<br />
<pre>
$ curl http://localhost:5000
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5000
</body>
</html>
</pre>
<br />
<h2>Creating multi-process images</h2>
<br />
As shown in previous blog posts, the <i>webapp</i> process is part of a bigger system, namely: a web application system with an Nginx reverse proxy forwarding requests to multiple <i>webapp</i> instances:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
sharedConstructors = import ../services-agnostic/constructors.nix {
inherit pkgs stateDir runtimeDir logDir cacheDir tmpDir forceDisableUserChange processManager;
};
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager;
};
in
rec {
webapp = rec {
port = 5000;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
};
nginx = rec {
port = 8080;
pkg = sharedConstructors.nginxReverseProxyHostBased {
webapps = [ webapp ];
inherit port;
} {};
};
}
</pre>
<br />
The Nix expression above shows a simple <strong>processes model</strong> variant of that system, that consists of only two process instances:<br />
<br />
<ul>
<li>The <i>webapp</i> process is (as shown earlier) an application that returns a static HTML page.</li>
<li><i>nginx</i> is configured as a reverse proxy to forward incoming connections to multiple <i>webapp</i> instances using the virtual host header property (<i>dnsName</i>).<br />
<br />
If somebody connects to the <i>nginx</i> server with the following host name: <i>webapp.local</i> then the request is forwarded to the <i>webapp</i> service.</li>
</ul>
<br />
<h3>Configuration steps</h3>
<br />
To allow all processes in the process model shown to be deployed to a single container, we need to execute the following steps in the construction of an image:<br />
<br />
<ul>
<li>Instead of deploying a single package, such as <i>webapp</i>, we need to refer to a collection of packages and/or configuration files that can be managed with a process manager, such as sysvinit, systemd or supervisord.<br />
<br />
The Nix process management framework provides all kinds of Nix function abstractions to accomplish this.<br />
<br />
For example, the following function invocation builds a configuration profile for the sysvinit process manager, containing a collection of <i>sysvinit</i> scripts (also known as <a href="https://wiki.debian.org/LSBInitScripts">LSB Init</a> compliant scripts):<br />
<br />
<pre style="overflow: auto;">
profile = import ../create-managed-process/sysvinit/build-sysvinit-env.nix {
exprFile = ./processes.nix;
stateDir = "/var";
};
</pre>
<br />
</li>
<li>Similar to single root process containers, we may also need to initialize state. For example, we need to create common FHS state directories (e.g. <i>/tmp</i>, <i>/var</i> etc.) in which services can store their relevant state files (e.g. log files, temp files).<br />
<br />
This can be done by running the following command:<br />
<br />
<pre>
nixproc-init-state --state-dir /var
</pre>
</li>
<li>Another property that multiple process containers have in common is that they may also require the presence of unprivileged users and groups, for security reasons.<br />
<br />
With the following commands, we can automatically generate all required users and groups specified in a deployment profile:<br />
<br />
<pre>
${dysnomia}/bin/dysnomia-addgroups ${profile}
${dysnomia}/bin/dysnomia-addusers ${profile}
</pre>
</li>
<li>Instead of starting a (single root) application process, we need to start a process manager that manages the processes that we want to deploy. As already explained, the framework allows you to pick multiple options.</li>
</ul>
<br />
<h3>Starting a process manager as a root process</h3>
<br />
From all process managers that the framework currently supports, the most straight forward option to use in a Docker container is: supervisord.<br />
<br />
To use it, we can create a symlink to the supervisord configuration in the deployment profile:<br />
<br />
<pre>
ln -s ${profile} /etc/supervisor
</pre>
<br />
and then start supervisord as a root process with the following command directive:<br />
<br />
<pre>
Cmd = [
"${pkgs.pythonPackages.supervisor}/bin/supervisord"
"--nodaemon"
"--configuration" "/etc/supervisor/supervisord.conf"
"--logfile" "/var/log/supervisord.log"
"--pidfile" "/var/run/supervisord.pid"
];
</pre>
<br />
(As a sidenote: creating a symlink is not strictly required, but makes it possible to control running services with the <i>supervisorctl</i> command-line tool).<br />
<br />
Supervisord is not the only option. We can also use sysvinit scripts, but doing so is a bit tricky. As explained earlier, the life-cycle of container is bound to a running root process (in foreground mode).<br />
<br />
sysvinit scripts do not run in the foreground, but start processes that daemonize and terminate immediately, leaving daemon processes behind that remain running in the background.<br />
<br />
As described in an earlier blog post about translating high-level process management concepts, it is also possible to run "daemons in the foreground" by creating a proxy script. We can also make a similar foreground proxy for a collection of daemons:<br />
<br />
<pre>
#!/bin/bash -e
_term()
{
nixproc-sysvinit-runactivity -r stop ${profile}
kill "$pid"
exit 0
}
nixproc-sysvinit-runactivity start ${profile}
# Keep process running, but allow it to respond to the TERM and INT
# signals so that all scripts are stopped properly
trap _term TERM
trap _term INT
tail -f /dev/null & pid=$!
wait "$pid"
</pre>
<br />
The above proxy script does the following:<br />
<br />
<ul>
<li>It first starts all sysvinit scripts by invoking the <i>nixproc-sysvinit-runactivity start</i> command.</li>
<li>Then it registers a signal handler for the <i>TERM</i> and <i>INT</i> signals. The corresponding callback triggers a shutdown procedure.</li>
<li>We invoke a dummy command that keeps running in the foreground without consuming too many system resources (<i>tail -f /dev/null</i>) and we wait for it to terminate.</li>
<li>The signal handler properly deactivates all processes in reverse order (with the <i>nixproc-sysvinit-runactivity -r stop</i> command), and finally terminates the dummy command causing the script (and the container) to stop.</li>
</ul>
<br />
In addition supervisord and sysvinit, we can also use <a href="https://sandervanderburg.blogspot.com/2020/06/using-disnix-as-simple-and-minimalistic.html">Disnix as a process manager</a> by using a similar strategy with a foreground proxy.<br />
<br />
<h3>Other configuration properties</h3>
<br />
The above configuration properties suffice to get a multi-process container running. However, to make working with such containers more practical from a user perspective, we may also want to:<br />
<br />
<ul>
<li>Add basic shell utilities to the image, so that you can control the processes, investigate log files (in case of errors), and do other maintenance tasks.</li>
<li>Add a <i>.bashrc</i> configuration file to make file coloring working for the <i>ls</i> command, and to provide a decent prompt in a shell session.</li>
</ul>
<br />
<h2>Usage</h2>
<br />
The configuration steps described in the previous section are wrapped into a function named: <i>createMultiProcessImage</i>, which itself is a thin wrapper around the <i>dockerTools.buildImage</i> function in Nixpkgs -- it accepts the same parameters with a number of additional parameters that are specific to multi-process configurations.<br />
<br />
The following function invocation builds a multi-process container deploying our example system, using supervisord as a process manager:<br />
<br />
<pre style="overflow: auto;">
let
pkgs = import <nixpkgs> {};
createMultiProcessImage = import ../../nixproc/create-multi-process-image/create-multi-process-image.nix {
inherit pkgs system;
inherit (pkgs) dockerTools stdenv;
};
in
createMultiProcessImage {
name = "multiprocess";
tag = "test";
exprFile = ./processes.nix;
stateDir = "/var";
processManager = "supervisord";
}
</pre>
<br />
After building the image, and deploying a container, with the following commands:<br />
<br />
<pre>
$ nix-build
$ docker load -i result
$ docker run -it --network host multiprocessimage:test
</pre>
<br />
we should be able to connect to the <i>webapp</i> instance via the <i>nginx</i> reverse proxy:<br />
<br />
<pre>
$ curl -H 'Host: webapp.local' http://localhost:8080
<!DOCTYPE html>
<html>
<head>
<title>Simple test webapp</title>
</head>
<body>
Simple test webapp listening on port: 5000
</body>
</html>
</pre>
<br />
As explained earlier, the constructed image also provides extra command-line utilities to do maintenance tasks, and control the life-cycle of the individual processes.<br />
<br />
For example, we can "connect" to the running container, and check which processes are running:<br />
<br />
<pre>
$ docker exec -it mycontainer /bin/bash
# supervisorctl
nginx RUNNING pid 11, uptime 0:00:38
webapp RUNNING pid 10, uptime 0:00:38
supervisor>
</pre>
<br />
If we change the <i>processManager</i> parameter to <i>sysvinit</i>, we can deploy a multi-process image in which the foreground proxy script is used as a root process (that starts and stops sysvinit scripts).<br />
<br />
We can control the life-cycle of each individual process by directly invoking the sysvinit scripts in the container:<br />
<br />
<pre>
$ docker exec -it mycontainer /bin/bash
$ /etc/rc.d/init.d/webapp status
webapp is running with Process ID(s) 33.
$ /etc/rc.d/init.d/nginx status
nginx is running with Process ID(s) 51.
</pre>
<br />
Although having extra command-line utilities to do administration tasks is useful, a disadvantage is that they considerably increase the size of the image.<br />
<br />
To save storage costs, it is also possible to disable <i>interactive</i> mode to exclude these packages:<br />
<br />
<pre style="overflow: auto;">
let
pkgs = import <nixpkgs> {};
createMultiProcessImage = import ../../nixproc/create-multi-process-image/create-multi-process-image.nix {
inherit pkgs system;
inherit (pkgs) dockerTools stdenv;
};
in
createMultiProcessImage {
name = "multiprocess";
tag = "test";
exprFile = ./processes.nix;
stateDir = "/var";
processManager = "supervisord";
interactive = false; # Do not install any additional shell utilities
}
</pre>
<br />
<h2>Discussion</h2>
<br />
In this blog post, I have described a new utility function in the Nix process management framework: <i>createMultiProcessImage</i> -- a thin wrapper around the <i>dockerTools.buildImage</i> function that can be used to convienently build multi-process Docker images, using any Docker-capable process manager that the Nix process management framework supports.<br />
<br />
Besides the fact that we can convienently construct multi-process images, this function also has the advantage (similar to the <i>dockerTools.buildImage</i> function) that Nix is only required for the construction of the image. To deploy containers from a multi-process image, Nix is not a requirement.<br />
<br />
There is also a drawback: similar to "ordinary" multi-process container deployments, when it is desired to upgrade a process, the entire container needs to be redeployed, also requiring a user to terminate all other running processes.<br />
<br />
<h2>Availability</h2>
<br />
The <i>createMultiProcessImage</i> function is part of the current development version of the <a href="https://github.com/svanderburg/nix-processmgmt">Nix process management framework</a> that can be obtained from my GitHub page.<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-69085703025015002962020-10-08T23:29:00.003+02:002020-12-19T01:13:35.879+01:00Transforming Disnix models to graphs and visualizing themIn <a href="https://sandervanderburg.blogspot.com/2020/09/assigning-unique-ids-to-services-in.html">my previous blog post</a>, I have described a new tool in the Dynamic Disnix toolset that can be used to automatically assign unique numeric IDs to services in a <a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a> service model. Unique numeric IDs can represent all kinds of useful resources, such as TCP/UDP port numbers, user IDs (UIDs), and group IDs (GIDs).<br />
<br />
Although I am quite happy to have this tool at my disposal, implementing it was much more difficult and time consuming than I expected. Aside from the fact that the problem is not as obvious as it may sound, the main reason is that the Dynamic Disnix toolset was originally developed as a proof of concept implementation for a research paper under very high time pressure. As a result, it has accumulated quite a bit of <strong>technical debt</strong>, that as of today, is still at a fairly high level (but much better than it was when I completed the PoC).<br />
<br />
For the ID assigner tool, I needed to make changes to the foundations of the tools, such as the model parsing libraries. As a consequence, all kinds of related aspects in the toolset started to break, such as the deployment planning algorithm implementations.<br />
<br />
Fixing some of these algorithm implementations was much more difficult than I expected -- they were not properly documented, not decomposed into functions, had little to no reuse of common concepts and as a result, were difficult to understand and change. I was forced to re-read the papers that I used as a basis for these algorithms.<br />
<br />
To prevent myself from having to go through such a painful process again, I have decided to <strong>revise</strong> them in such a way that they are better understandable and maintainable.<br />
<br />
<h2>Dynamically distributing services</h2>
<br />
The deployment models in the core Disnix toolset are <strong>static</strong>. For example, the distribution of services to machines in the network is done in a <strong>distribution model</strong> in which the user has to manually map services in the services model to target machines in the infrastructure model (and optionally to <a href="https://sandervanderburg.blogspot.com/2016/06/deploying-containers-with-disnix-as.html">container services</a> hosted on the target machines).<br />
<br />
Each time a condition changes, e.g. the system needs to scale up or a machine crashes and the system needs to recover, a new distribution model must be configured and the system must be redeployed. For big complex systems that need to be reconfigured frequently, manually specifying new distribution models becomes very impractical.<br />
<br />
As I have already explained in older blog posts, to cope with the limitations of static deployment models (and other static configuration aspects), <a href="https://sandervanderburg.blogspot.com/2011/03/self-adaptive-deployment-with-disnix.html">I have developed Dynamic Disnix</a>, in which various configuration aspects can be automated, including the distribution of services to machines.<br />
<br />
A strategy for dynamically distributing services to machines can be specified in a <strong>QoS model</strong>, that typically consists of two phases:<br />
<br />
<ul>
<li>First, a <strong>candidate</strong> target <strong>selection</strong> must be made, in which for each service the appropriate candidate target machines are selected.<br />
<br />
Not all machines are capable of hosting a certain service for functional and non-functional reasons -- for example, a <i>i686-linux</i> machine is not capable of running a binary compiled for a <i>x86_64-linux</i> machine.<br />
<br />
A machine can also be exposed to the public internet, and as a result, may not be suitable to host a service that exposes privacy-sensitive information.</li>
<li>After the suitable candidate target machines are known for each service, we must decide to which candidate machine each service gets <strong>distributed</strong>.<br />
<br />
This can be done in many ways. The strategy that we want to use is typically based on all kinds of non-functional requirements.<br />
<br />
For example, we can optimize a system's reliability by minimizing the amount of network links between services, requiring a strategy in which services that depend on each other are mapped to the same machine, as much as possible.</li>
</ul>
<br />
<h2>Graph-based optimization problems</h2>
<br />
In the Dynamic Disnix toolset, I have implemented various kinds of distribution algorithms/strategies for all kinds of purposes.<br />
<br />
I did not "invent" most of them -- for some, I got inspiration from papers in the academic literature.<br />
<br />
Two of the more advanced deployment planning algorithms are graph-based, to accomplish the following goals:<br />
<br />
<ul>
<li><strong>Reliable deployment</strong>. Network links are a potential source making a distributed system unreliable -- connections may fail, become slow, or could be interrupted frequently. By minimizing the amount of network links between services (by co-locating them on the same machine), their impact can be reduced. To not make deployments not too expensive, it should be done with a minimal amount of machines.<br />
<br />
As described in the paper: "<a href="https://gsd.uwaterloo.ca/publications/view/266">Reliable Deployment of Component-based Applications into Distributed Environments</a>" by A. Heydarnoori and F. Mavaddat, this problem can be transformed into a graph problem: the multiway cut problem (which is <a href="https://en.wikipedia.org/wiki/NP-hardness">NP-hard</a>).<br />
<br />
It can only be solved in polynomial time with an approximation algorithm that comes close to the optimal solution, <a href="https://en.wikipedia.org/wiki/P_versus_NP_problem">unless a proof that <i>P = NP</i> exists</a>.</li>
<li><strong>Fragile deployment</strong>. Inspired by the above deployment problem, I also came up with the opposite problem (as my own "invention") -- how can we make any connection between a service a true network link (not local), so that we can test a system for robustness, using a minimal amount of machines?<br />
<br />
This problem can be modeled as a graph coloring problem (that is a NP-hard problem as well). I used one of the approximation algorithms described in the paper: "<a href="https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.399.396&rep=rep1&type=pdf">New Methods to Color the Vertices of a Graph</a>" by D. Brélaz to implement a solution.</li>
</ul>
<br />
To work with these graph-based algorithms, I originally did not apply any transformations -- because of time pressure, I directly worked with objects from the Disnix models (e.g. services, target machines) and somewhat "glued" these together with generic data structures, such as lists and hash tables.<br />
<br />
As a result, when looking at the implementation, it is very hard to get an understanding of the process and how an implementation aspect relates to a concept described in the papers shown above.<br />
<br />
In my revised version, I have implemented a <strong>general purpose</strong> graph library that can be used to solve all kinds of general graph related problems.<br />
<br />
Aside from using a general graph library, I have also separated the graph-based generation processes into the following steps:<br />
<br />
<ul>
<li>After opening the Disnix input models (such as the services, infrastructure, and distribution models) I <strong>transform</strong> the models to a graph representing an instance of the problem domain.</li>
<li>After the graph has been generated, I <strong>apply</strong> the approximation <strong>algorithm</strong> to the graph data structure.</li>
<li>Finally, I <strong>transform</strong> the resolved graph <strong>back</strong> to a distribution model that should provide our desired distribution outcome.</li>
</ul>
<br />
This new organization provides better separation of concerns, common concepts can be reused (such as graph operations), and as a result, the implementations are much closer to the approximation algorithms described in the papers.<br />
<br />
<h2>Visualizing the generation process</h2>
<br />
Another advantage of having a reusable graph implementation is that we can easily extend it to <strong>visualize</strong> the problem graphs.<br />
<br />
When I combine these features together with my earlier work that <a href="https://sandervanderburg.blogspot.com/2019/02/generating-functional-architecture.html">visualizes services models</a>, and a new tool that visualizes infrastructure models, I can make the entire generation process transparent.<br />
<br />
For example, the following services model:<br />
<br />
<pre>
{system, pkgs, distribution, invDistribution}:
let
customPkgs = import ./pkgs { inherit pkgs system; };
in
rec {
testService1 = {
name = "testService1";
pkg = customPkgs.testService1;
type = "echo";
};
testService2 = {
name = "testService2";
pkg = customPkgs.testService2;
dependsOn = {
inherit testService1;
};
type = "echo";
};
testService3 = {
name = "testService3";
pkg = customPkgs.testService3;
dependsOn = {
inherit testService1 testService2;
};
type = "echo";
};
}
</pre>
<br />
can be visualized as follows:<br />
<br />
<pre>
$ dydisnix-visualize-services -s services.nix
</pre>
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3yicERasasfXo3wf3KuJkAjXUemBp_UUmD7uhPnd6Sw6gogmpWxcCmdGC5gu4Q3FSvxcjvHGwrju7pagQW3sGQ-ymKYpR_YBz_XYF-FcAcb79cEipgXjAkqvCKBrgibghVCZWSIF2WsL0/s0/services.png" style="display: block; padding: 1em 0; text-align: center; " imageanchor="1"><img alt="" border="0" data-original-height="322" data-original-width="277" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEi3yicERasasfXo3wf3KuJkAjXUemBp_UUmD7uhPnd6Sw6gogmpWxcCmdGC5gu4Q3FSvxcjvHGwrju7pagQW3sGQ-ymKYpR_YBz_XYF-FcAcb79cEipgXjAkqvCKBrgibghVCZWSIF2WsL0/s0/services.png"/></a></div>
<br />
The above services model and corresponding visualization capture the following properties:<br />
<br />
<ul>
<li>They describe three <strong>services</strong> (as denoted by ovals).</li>
<li>The arrows denote <strong>inter-dependency relationships</strong> (the <i>dependsOn</i> attribute in the services model).<br />
<br />
When a service has an inter-dependency on another service means that the latter service has to be activated first, and that the dependent service needs to know how to reach the former.<br />
<br />
<i>testService2</i> depends on <i>testService1</i> and <i>testService3</i> depends on both the other two services.</li>
</ul>
<br />
We can also visualize the following infrastructure model:<br />
<br />
<pre>
{
testtarget1 = {
properties = {
hostname = "testtarget1";
};
containers = {
mysql-database = {
mysqlPort = 3306;
};
echo = {};
};
};
testtarget2 = {
properties = {
hostname = "testtarget2";
};
containers = {
mysql-database = {
mysqlPort = 3306;
};
};
};
testtarget3 = {
properties = {
hostname = "testtarget3";
};
};
}
</pre>
<br />
with the following command:<br />
<br />
<pre>
$ dydisnix-visualize-infra -i infrastructure.nix
</pre>
<br />
resulting in the following visualization:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo4zLVMHwPKgOlp-AKkeISXO13Wfx2rhjUPGE_6O1-spqCPStPsEgQjW689rVOkWdg5va_-885JTeIqoymhzW6TlZnUe31MkdRKnYLPq5P_qL3JqCqr28xaZW9xdnVRq_sw7oHKgqJWdyv/s0/infrastructure.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" width="520" border="0" data-original-height="185" data-original-width="827" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjo4zLVMHwPKgOlp-AKkeISXO13Wfx2rhjUPGE_6O1-spqCPStPsEgQjW689rVOkWdg5va_-885JTeIqoymhzW6TlZnUe31MkdRKnYLPq5P_qL3JqCqr28xaZW9xdnVRq_sw7oHKgqJWdyv/s0/infrastructure.png"/></a></div>
<br />
The above infrastructure model declares three machines. Each target machine provides a number of container services (such as a MySQL database server, and <i>echo</i> that acts as a testing container).<br />
<br />
With the following command, we can generate a problem instance for the graph coloring problem using the above services and infrastructure models as inputs:<br />
<br />
<pre>
$ dydisnix-graphcol -s services.nix -i infrastructure.nix \
--output-graph
</pre>
<br />
resulting in the following graph:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDlWRWdWGF9p9WUPkamzX2v9sYsLwMiLGrLY2trdsN6EMvA3Tnjb1_4g9HBuP6pD0Yg_2T6aV9Y0vdiliR_0Bc9XeKt7CXcUKRcfw-sbfDKd83irkbC_1VJlpd9bwooAz7by3ThVaNukC0/s0/graphcol-instance.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="255" data-original-width="204" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhDlWRWdWGF9p9WUPkamzX2v9sYsLwMiLGrLY2trdsN6EMvA3Tnjb1_4g9HBuP6pD0Yg_2T6aV9Y0vdiliR_0Bc9XeKt7CXcUKRcfw-sbfDKd83irkbC_1VJlpd9bwooAz7by3ThVaNukC0/s0/graphcol-instance.png"/></a></div>
<br />
The graph shown above captures the following properties:<br />
<br />
<ul>
<li>Each service translates to a node</li>
<li>When an inter-dependency relationship exists between services, it gets translated to a (bi-directional) link representing a network connection (the rationale is that a service that has an inter-dependency on another service, interact with each other by using a network connection).</li>
</ul>
<br />
Each target machine translates to a color, that we can represent with a numeric index -- <i>0</i> is <i>testtarget1</i>, <i>1</i> is <i>testtarget2</i> and so on.<br />
<br />
The following command generates the resolved problem instance graph in which each vertex has a color assigned:<br />
<br />
<pre>
$ dydisnix-graphcol -s services.nix -i infrastructure.nix \
--output-resolved-graph
</pre>
<br />
resulting in the following visualization:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7NwqMju2MOfPp1B06TXTOq9vk9E9Xr-cMVKhR49NlRssy1BogP3MF3wKSfagNKfmZH47yb4kbwq5oTCJKjPWiipdm348GOj_TT36sQnbIgVeXd4I6r6Fe-uR3QjrlhsuAIYHqz_zhHr7g/s0/graphcol-resolved.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" data-original-height="255" data-original-width="248" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEj7NwqMju2MOfPp1B06TXTOq9vk9E9Xr-cMVKhR49NlRssy1BogP3MF3wKSfagNKfmZH47yb4kbwq5oTCJKjPWiipdm348GOj_TT36sQnbIgVeXd4I6r6Fe-uR3QjrlhsuAIYHqz_zhHr7g/s0/graphcol-resolved.png"/></a></div>
<br />
(As a sidenote: in the above graph, colors are represented by numbers. In theory, I could also use real colors, but if I also want that the graph to remain visually appealing, I need to solve a color picking problem, which is beyond the scope of my refactoring objective).<br />
<br />
The resolved graph can be translated back into the following distribution model:<br />
<br />
<pre>
$ dydisnix-graphcol -s services.nix -i infrastructure.nix
{
"testService2" = [
"testtarget2"
];
"testService1" = [
"testtarget1"
];
"testService3" = [
"testtarget3"
];
}
</pre>
<br />
As you may notice, every service is distributed to a separate machine, so that every network link between a service is a real network connection between machines.<br />
<br />
We can also visualize the problem instance of the multiway cut problem. For this, we also need a distribution model that, declares for each service, which target machine is a candidate.<br />
<br />
The following distribution model makes all three target machines in the infrastructure model a candidate for every service:<br />
<br />
<pre style="overflow: auto;">
{infrastructure}:
{
testService1 = [ infrastructure.testtarget1 infrastructure.testtarget2 infrastructure.testtarget3 ];
testService2 = [ infrastructure.testtarget1 infrastructure.testtarget2 infrastructure.testtarget3 ];
testService3 = [ infrastructure.testtarget1 infrastructure.testtarget2 infrastructure.testtarget3 ];
}
</pre>
<br />
With the following command we can generate a problem instance representing a host-application graph:<br />
<br />
<pre>
$ dydisnix-multiwaycut -s services.nix -i infrastructure.nix \
-d distribution.nix --output-graph
</pre>
<br />
providing me the following output:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEho0Gy2W6DEo3JUfwvqa7jmNgdpvsIzroQsMPaNNUIM5Wt0F1fNysUKL45dKGu9vxhRQuJWEp6DW9YGnJPWzQaNd-NKL2C2xl9_riMpkAcndlAt8P3T51QL_LvmQHgmeHiD8u2nkanb4oOq/s0/multiwaycut-instance.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="407" data-original-width="739" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEho0Gy2W6DEo3JUfwvqa7jmNgdpvsIzroQsMPaNNUIM5Wt0F1fNysUKL45dKGu9vxhRQuJWEp6DW9YGnJPWzQaNd-NKL2C2xl9_riMpkAcndlAt8P3T51QL_LvmQHgmeHiD8u2nkanb4oOq/s0/multiwaycut-instance.png"/></a></div>
<br />
The above problem graph has the following properties:<br />
<br />
<ul>
<li>Each service translates to an <strong>app node</strong> (prefixed with <i>app:</i>) and each candidate target machine to a <strong>host node</strong> (prefixed with <i>host:</i>).</li>
<li>When a network connection between two services exists (implicitly derived from having an inter-dependency relationship), an edge is generated with a weight of <i>1</i>.</li>
<li>When a target machine is a candidate target for a service, then an edge is generated with a weight of <i>n<sup>2</sup></i> representing a very large number.</li>
</ul>
<br />
The objective of solving the multiway cut problem is to cut edges in the graph in such a way that each terminal (host node) is disconnected from the other terminals (host nodes), in which the total weight of the cuts is minimized.<br />
<br />
When applying the approximation algorithm in the paper to the above graph:<br />
<br />
<pre>
$ dydisnix-multiwaycut -s services.nix -i infrastructure.nix \
-d distribution.nix --output-resolved-graph
</pre>
<br />
we get the following resolved graph:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCyAw8iCW5fGAbuCkU87uyY4EaThWPU_eOw0q0uP8n_kddSsikFXupl-C_0ZX_jlSWwbZ7Zl911L02pqL0NSiowymflUs_Xnml9SxJR7wZFr4Px-HJER1stF428XG0w3jDsBe8jTCFWxFR/s0/multiwaycut-resolved.png" style="display: block; padding: 1em 0; text-align: center; "><img alt="" width="520" border="0" data-original-height="407" data-original-width="916" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgCyAw8iCW5fGAbuCkU87uyY4EaThWPU_eOw0q0uP8n_kddSsikFXupl-C_0ZX_jlSWwbZ7Zl911L02pqL0NSiowymflUs_Xnml9SxJR7wZFr4Px-HJER1stF428XG0w3jDsBe8jTCFWxFR/s0/multiwaycut-resolved.png"/></a></div>
<br />
that can be transformed back into the following distribution model:<br />
<br />
<pre>
$ dydisnix-multiwaycut -s services.nix -i infrastructure.nix \
-d distribution.nix
{
"testService2" = [
"testtarget1"
];
"testService1" = [
"testtarget1"
];
"testService3" = [
"testtarget1"
];
}
</pre>
<br />
As you may notice by looking at the resolved graph (in which the terminals: <i>testtarget2</i> and <i>testtarget3</i> are disconnected) and the distribution model output, all services are distributed to the same machine: <i>testtarget1</i> making all connections between the services local connections.<br />
<br />
In this particular case, the solution is not only close to the optimal solution, but it is the optimal solution.<br />
<br />
<h2>Conclusion</h2>
<br />
In this blog post, I have described how I have revised the deployment planning algorithm implementations in the Dynamic Disnix toolset. Their concerns are now much better separated, and the graph-based algorithms now use a general purpose graph library, that can also be used for generating visualizations of the intermediate steps in the generation process.<br />
<br />
This revision was not on my short-term planned features list, but I am happy that I did the work. Retrospectively, I regret that I never took the time to finish things up properly after the submission of the paper. Although Dynamic Disnix's quality is still not where I want it to be, it is quite a step forward in making the toolset more usable.<br />
<br />
Sadly, it is almost 10 years ago that I started Dynamic Disnix and still there is no offical release yet and the technical debt in Dynamic Disnix is one of the important reasons that I never did an official release. Hopefully, with this step I can do it some day. :-)<br />
<br />
The good news is that I made the paper submission deadline and that the paper got accepted for presentation. It brought me to the <a href="https://www.hpi.uni-potsdam.de/giese/public/selfadapt/seams/">SEAMS 2011</a> conference (co-located with <a href="http://2011.icse-conferences.org/">ICSE 2011</a>) in Honolulu, Hawaii, allowing me to take pictures such as this one:<br />
<br />
<div class="separator" style="clear: both;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkXrcObq0GDXEN2cn6gMluenRE0a_3uOmnh3OyLvGzZ0QJZ-jbJ5MKI1RAda0wxxeqJO1AbUibVOiM9Ygtvi_kPiHqOS0Ej_77zfe4joxHdMflQ7CZ0t-o1TbUvxDsjv-qWzLfm6XqirHP/s2048/P1040527.JPG" style="display: block; padding: 1em 0; text-align: center; "><img alt="" border="0" width="520" data-original-height="1365" data-original-width="2048" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEgkXrcObq0GDXEN2cn6gMluenRE0a_3uOmnh3OyLvGzZ0QJZ-jbJ5MKI1RAda0wxxeqJO1AbUibVOiM9Ygtvi_kPiHqOS0Ej_77zfe4joxHdMflQ7CZ0t-o1TbUvxDsjv-qWzLfm6XqirHP/s600/P1040527.JPG"/></a></div>
<br />
<h2>Availability</h2>
<br />
The graph library and new implementations of the deployment planning algorithms described in this blog post are part of the current development version of <a href="https://github.com/svanderburg/dydisnix">Dynamic Disnix</a>.<br />
<br />
The paper: "A Self-Adaptive Deployment Framework for Service-Oriented Systems" describes the Dynamic Disnix framework (developed 9 years ago) and can be obtained from <a href="http://sandervanderburg.nl/index.php/publications">my publications</a> page.<br />
<br />
<h2>Acknowledgements</h2>
<br />
To generate the visualizations I used the <a href="https://graphviz.org/">Graphviz</a> toolset.<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0tag:blogger.com,1999:blog-1397115249631682228.post-12669392321035098592020-09-24T20:24:00.004+02:002020-12-19T01:13:23.168+01:00Assigning unique IDs to services in Disnix deployment modelsAs described in some of my recent blog posts, one of the more advanced features of <a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a> as well as <a href="https://sandervanderburg.blogspot.com/2019/11/a-nix-based-functional-organization-for.html">the experimental Nix process management</a> framework is to deploy <a href="https://sandervanderburg.blogspot.com/2016/06/deploying-containers-with-disnix-as.html"><strong>multiple instances</strong></a> of the same service to the same machine.<br />
<br />
To make running multiple service instances on the same machine possible, these tools rely on <strong>conflict avoidance</strong> rather than isolation (typically used for <a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">containers</a>). To allow multiple services instances to co-exist on the same machine, they need to be configured in such a way that they do not allocate any conflicting resources.<br />
<br />
Although for small systems it is doable to configure multiple instances by hand, this process gets tedious and time consuming for larger and more technologically diverse systems.<br />
<br />
One particular kind of conflicting resource that could be configured automatically are <strong>numeric IDs</strong>, such as TCP/UDP port numbers, user IDs (UIDs), and group IDs (GIDs).<br />
<br />
In this blog post, I will describe how multiple service instances are configured (in Disnix and the process management framework) and how we can automatically assign unique numeric IDs to them.<br />
<br />
<h2>Configuring multiple service instances</h2>
<br />
To facilitate conflict avoidance in Disnix and the Nix process management framework, services are configured as follows:<br />
<br />
<pre style="overflow: auto;">
{createManagedProcess, tmpDir}:
{port, instanceSuffix ? "", instanceName ? "webapp${instanceSuffix}"}:
let
webapp = import ../../webapp;
in
createManagedProcess {
name = instanceName;
description = "Simple web application";
inherit instanceName;
# This expression can both run in foreground or daemon mode.
# The process manager can pick which mode it prefers.
process = "${webapp}/bin/webapp";
daemonArgs = [ "-D" ];
environment = {
PORT = port;
PID_FILE = "${tmpDir}/${instanceName}.pid";
};
user = instanceName;
credentials = {
groups = {
"${instanceName}" = {};
};
users = {
"${instanceName}" = {
group = instanceName;
description = "Webapp";
};
};
};
overrides = {
sysvinit = {
runlevels = [ 3 4 5 ];
};
};
}
</pre>
<br />
The Nix expression shown above is a nested function that describes how to deploy a simple self-contained REST web application with an embedded HTTP server:<br />
<br />
<ul>
<li>The <strong>outer function header</strong> (first line) specifies all common build-time dependencies and configuration properties that the service needs:<br />
<br />
<ul>
<li><i>createManagedProcess</i> is a function that can be used to <a href="https://sandervanderburg.blogspot.com/2019/11/a-nix-based-functional-organization-for.html">define process manager agnostic configurations</a> that can be translated to configuration files for a variety of process managers (e.g. <i>systemd</i>, <i>launchd</i>, <i>supervisord</i> etc.).</li>
<li><i>tmpDir</i> refers to the temp directory in which temp files are stored.</li>
</ul>
</li>
<li>The <strong>inner function header</strong> (second line) specifies all instance parameters -- these are the parameters that must be configured in such a way that conflicts with other process instances are avoided:<br />
<br />
<ul>
<li>The <i>instanceName</i> parameter (that can be derived from the <i>instanceSuffix</i>) is a value used by some of the process management backends (e.g. the ones that invoke the <i>daemon</i> command) to derive a unique PID file for the process. When running multiple instances of the same process, each of them requires a unique PID file name.</li>
<li>The <i>port</i> parameter specifies to which TCP port the service binds to. Binding the service to a port that is already taken by another service, causes the deployment of this service to fail.</li>
</ul>
</li>
<li>
In the function <strong>body</strong>, we invoke the <i>createManagedProcess</i> function to construct configuration files for all supported process manager backends to run the <i>webapp</i> process:<br />
<br />
<ul>
<li>As explained earlier, the <i>instanceName</i> is used to configure the <i>daemon</i> executable in such a way that it allocates a unique PID file.</li>
<li>The <i>process</i> parameter specifies which executable we need to run, both as a foreground process or daemon.</li>
<li>The <i>daemonArgs</i> parameter specifies which command-line parameters need to be propagated to the executable when the process should <a href="https://sandervanderburg.blogspot.com/2020/01/writing-well-behaving-daemon-in-c.html">daemonize on its own</a>.</li>
<li>The <i>environment</i> parameter specifies all environment variables. The <i>webapp</i> service uses these variables for runtime property configuration.</li>
<li>The <i>user</i> parameter is used to specify that the process should run as an unprivileged user. The <i>credentials</i> parameter is used to configure the creation of the user account and corresponding user group.</li>
<li>The <i>overrides</i> parameter is used to override the process manager-agnostic parameters with process manager-specific parameters. For the <i>sysvinit</i> backend, we configure the runlevels in which the service should run.</li>
</ul>
</li>
</ul>
<br />
Although the convention shown above makes it possible to avoid conflicts (assuming that all potential conflicts have been identified and exposed as function parameters), these parameters are typically configured manually:<br />
<br />
<pre style="overflow: auto;">
{ pkgs, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "sysvinit"
, ...
}:
let
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager;
};
processType = import ../../nixproc/derive-dysnomia-process-type.nix {
inherit processManager;
};
in
rec {
webapp1 = rec {
name = "webapp1";
port = 5000;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "1";
};
type = processType;
};
webapp2 = rec {
name = "webapp2";
port = 5001;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "2";
};
type = processType;
};
}
</pre>
<br />
The above Nix expression shows both a valid Disnix <strong>services</strong> as well as a valid <strong>processes</strong> model that composes two web application process instances that can run concurrently on the same machine by invoking the nested constructor function shown in the previous example:<br />
<br />
<ul>
<li>Each <i>webapp</i> instance has its own unique instance name, by specifying a unique numeric <i>instanceSuffix</i> that gets appended to the service name.</li>
<li>Every <i>webapp</i> instance binds to a unique TCP port (5000 and 5001) that should not conflict with system services or other process instances.</li>
</ul>
<br />
<h2>Previous work: assigning port numbers</h2>
<br />
Although configuring two process instances is still manageable, the configuration process becomes more tedious and time consuming when the amount and the kind of processes (each having their own potential conflicts) grow.<br />
<br />
Five years ago, I already identified a resource that could be automatically assigned to services: <strong>port numbers</strong>.<br />
<br />
I have created <a href="https://sandervanderburg.blogspot.com/2015/07/assigning-port-numbers-to-microservices.html">a very simple port assigner tool</a> that allows you to specify a global ports pool and a target-specific pool pool. The former is used to assign globally unique port numbers to all services in the network, whereas the latter assigns port numbers that are unique to the target machine where the service is deployed to (this is to cope with the scarcity of port numbers).<br />
<br />
Although the tool is quite useful for systems that do not consist of too many different kinds of components, I ran into a number limitations when I want to manage a more diverse set of services:<br />
<br />
<ul>
<li>Port numbers are not the only numeric IDs that services may require. When deploying systems that consist of self-contained executables, you typically want to run them as unprivileged users for security reasons. User accounts on most UNIX-like systems require unique <strong>user IDs</strong>, and the corresponding users' groups require unique <strong>group IDs</strong>.</li>
<li>We typically want to manage <strong>multiple</strong> resource <strong>pools</strong>, for a variety of reasons. For example, when we have a number of HTTP server instances and a number of database instances, then we may want to pick port numbers in the 8000-9000 range for the HTTP servers, whereas for the database servers we want to use a different pool, such as 5000-6000.</li>
</ul>
<br />
<h2>Assigning unique numeric IDs</h2>
<br />
To address these shortcomings, I have developed a replacement tool that acts as a generic numeric ID assigner.<br />
<br />
This new ID assigner tool works with ID <strong>resource configuration</strong> files, such as:<br />
<br />
<pre>
rec {
ports = {
min = 5000;
max = 6000;
scope = "global";
};
uids = {
min = 2000;
max = 3000;
scope = "global";
};
gids = uids;
}
</pre>
<br />
The above ID resource configuration file (<i>idresources.nix</i>) defines three resource pools: <i>ports</i> is a resource that represents port numbers to be assigned to the webapp processes, <i>uids</i> refers to user IDs and <i>gids</i> to group IDs. The group IDs' resource configuration is identical to the users' IDs configuration.<br />
<br />
Each resource attribute refers the following configuration properties:<br />
<br />
<ul>
<li>The <i>min</i> value specifies the <strong>minimum</strong> ID to hand out, <i>max</i> the <strong>maximum</strong> ID.</li>
<li>The <i>scope</i> value specifies the <strong>scope</strong> of the resource pool. <i>global</i> (which is the default option) means that the IDs assigned from this resource pool to services are globally unique for the entire system.<br />
<br />
The <i>machine</i> scope can be used to assign IDs that are unique for the machine where a service is distributed to. When the latter option is used, services that are distributed two separate machines may have the same ID.</li>
</ul>
<br />
We can adjust the services/processes model in such a way that every service will use dynamically assigned IDs and that each service specifies for which resources it requires a unique ID:<br />
<br />
<pre style="overflow: auto;">
{ pkgs, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "sysvinit"
, ...
}:
let
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager ids;
};
processType = import ../../nixproc/derive-dysnomia-process-type.nix {
inherit processManager;
};
in
rec {
webapp1 = rec {
name = "webapp1";
port = ids.ports.webapp1 or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "1";
};
type = processType;
requiresUniqueIdsFor = [ "ports" "uids" "gids" ];
};
webapp2 = rec {
name = "webapp2";
port = ids.ports.webapp2 or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "2";
};
type = processType;
requiresUniqueIdsFor = [ "ports" "uids" "gids" ];
};
}
</pre>
<br />
In the above services/processes model, we have made the following changes:<br />
<br />
<ul>
<li>In the beginning of the expression, we <strong>import</strong> the dynamically generated <i>ids.nix</i> expression that provides ID assignments for each resource. If the <i>ids.nix</i> file does not exists, we generate an empty attribute set. We implement this construction (in which the absence of <i>ids.nix</i> can be tolerated) to allow the ID assigner to bootstrap the ID assignment process.</li>
<li>Every hardcoded <i>port</i> attribute of every service is replaced by a <strong>reference</strong> to the <i>ids</i> attribute set that is dynamically generated by the ID assigner tool. To allow the ID assigner to open the services model in the first run, we provide a fallback port value of 0.</li>
<li>Every service specifies for which resources it <strong>requires</strong> a unique ID through the <i>requiresUniqueIdsFor</i> attribute. In the above example, both service instances require unique IDs to assign a port number, user ID to the user and group ID to the group.</li>
</ul>
<br />
The port assignments are propagated as function parameters to the constructor functions that configure the services (as shown earlier in this blog post).<br />
<br />
We could also implement a similar strategy with the UIDs and GIDs, but a more convenient mechanism is to compose the function that creates the credentials, so that it transparently uses our <i>uids</i> and <i>gids</i> assignments.<br />
<br />
As shown in the expression above, the <i>ids</i> attribute set is also propagated to the constructors expression. The constructors expression indirectly composes the <i>createCredentials</i> function as follows:<br />
<br />
<pre>
{pkgs, ids ? {}, ...}:
{
createCredentials = import ../../create-credentials {
inherit (pkgs) stdenv;
inherit ids;
};
...
}
</pre>
<br />
The <i>ids</i> attribute set is propagated to the function that composes the <i>createCredentials</i> function. As a result, it will automatically assign the UIDs and GIDs in the <i>ids.nix</i> expression when the user configures a user or group with a name that exists in the <i>uids</i> and <i>gids</i> resource pools.<br />
<br />
To make these UIDs and GIDs assignments go smoothly, it is recommended to give a process instance the same process name, instance name, user and group names.<br />
<br />
<h2>Using the ID assigner tool</h2>
<br />
By combining the ID resources specification with the three Disnix models: a <strong>services model</strong> (that defines all distributable services, shown above), an <strong>infrastructure model</strong> (that captures all available target machines) and their properties and a <strong>distribution model</strong> (that maps services to target machines in the network), we can automatically generate an ids configuration that contains all ID assignments:<br />
<br />
<pre style="overflow: auto;">
$ dydisnix-id-assign -s services.nix -i infrastructure.nix \
-d distribution.nix \
--id-resources idresources.nix --output-file ids.nix
</pre>
<br />
The above command will generate an ids configuration file (<i>ids.nix</i>) that provides, for each resource in the ID resources model, a unique assignment to services that are distributed to a target machine in the network. (Services that are not distributed to any machine in the distribution model will be skipped, to not waste too many resources).<br />
<br />
The output file (<i>ids.nix</i>) has the following structure:<br />
<br />
<pre>
{
"ids" = {
"gids" = {
"webapp1" = 2000;
"webapp2" = 2001;
};
"uids" = {
"webapp1" = 2000;
"webapp2" = 2001;
};
"ports" = {
"webapp1" = 5000;
"webapp2" = 5001;
};
};
"lastAssignments" = {
"gids" = 2001;
"uids" = 2001;
"ports" = 5001;
};
}
</pre>
<br />
<ul>
<li>The <i>ids</i> attribute contains for each resource (defined in the ID resources model) the unique ID assignments per service. As shown earlier, both service instances require unique IDs for <i>ports</i>, <i>uids</i> and <i>gids</i>. The above attribute set stores the corresponding ID assignments.</li>
<li>The <i>lastAssignments</i> attribute memorizes the last ID assignment per resource. Once an ID is assigned, it will not be immediately reused. This is to allow roll backs and to prevent data to incorrectly get owned by the wrong user accounts. Once the maximum ID limit is reached, the ID assigner will start searching for a free assignment from the beginning of the resource pool.</li>
</ul>
<br />
In addition to assigning IDs to services that are distributed to machines in the network, it is also possible to assign IDs to all services (regardless whether they have been deployed or not):<br />
<br />
<pre style="overflow: auto;">
$ dydisnix-id-assign -s services.nix \
--id-resources idresources.nix --output-file ids.nix
</pre>
<br />
Since the above command does not know anything about the target machines, it only works with an ID resources configuration that defines global scope resources.<br />
<br />
When you intend to upgrade an existing deployment, you typically want to retain already assigned IDs, while obsolete ID assignment should be removed, and new IDs should be assigned to services that have none yet. This is to prevent unnecessary redeployments.<br />
<br />
When removing the first webapp service and adding a third instance:<br />
<br />
<pre style="overflow: auto;">
{ pkgs, system
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager ? "sysvinit"
, ...
}:
let
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir forceDisableUserChange processManager ids;
};
processType = import ../../nixproc/derive-dysnomia-process-type.nix {
inherit processManager;
};
in
rec {
webapp2 = rec {
name = "webapp2";
port = ids.ports.webapp2 or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "2";
};
type = processType;
requiresUniqueIdsFor = [ "ports" "uids" "gids" ];
};
webapp3 = rec {
name = "webapp3";
port = ids.ports.webapp3 or 0;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
instanceSuffix = "3";
};
type = processType;
requiresUniqueIdsFor = [ "ports" "uids" "gids" ];
};
}
</pre>
<br />
And running the following command (that provides the current <i>ids.nix</i> as a parameter):<br />
<br />
<pre>
$ dydisnix -s services.nix -i infrastructure.nix -d distribution.nix \
--id-resources idresources.nix --ids ids.nix --output-file ids.nix
</pre>
<br />
we will get the following ID assignment configuration:<br />
<br />
<pre>
{
"ids" = {
"gids" = {
"webapp2" = 2001;
"webapp3" = 2002;
};
"uids" = {
"webapp2" = 2001;
"webapp3" = 2002;
};
"ports" = {
"webapp2" = 5001;
"webapp3" = 5002;
};
};
"lastAssignments" = {
"gids" = 2002;
"uids" = 2002;
"ports" = 5002;
};
}
</pre>
<br />
As may be observed, since the <i>webapp2</i> process is in both the current and the previous configuration, its ID assignments will be retained. <i>webapp1</i> gets removed because it is no longer in the services model. <i>webapp3</i> gets the next numeric IDs from the resources pools.<br />
<br />
Because the configuration of <i>webapp2</i> stays the same, it does not need to be redeployed.<br />
<br />
The models shown earlier are valid Disnix services models. As a consequence, they can be used with Dynamic Disnix's ID assigner tool: <i>dydisnix-id-assign</i>.<br />
<br />
Although these Disnix services models are also valid processes models (used by the Nix process management framework) not every processes model is guaranteed to be compatible with a Disnix service model.<br />
<br />
For process models that are not compatible, it is possible to use the <i>nixproc-id-assign</i> tool that acts as a wrapper around <i>dydisnix-id-assign</i> tool:<br />
<br />
<pre>
$ nixproc-id-assign --id-resources idresources.nix processes.nix
</pre>
<br />
Internally, the <i>nixproc-id-assign</i> tool converts a processes model to a Disnix service model (augmenting the process instance objects with missing properties) and propagates it to the <i>dydisnix-id-assign</i> tool.<br />
<br />
<h2>A more advanced example</h2>
<br />
The <i>webapp</i> processes example is fairly trivial and only needs unique IDs for three kinds of resources: port numbers, UIDs, and GIDs.<br />
<br />
I have also developed a more complex example for the Nix process management framework that exposes several commonly used system services on Linux systems, such as:<br />
<br />
<pre style="overflow: auto;">
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
ids = if builtins.pathExists ./ids.nix then (import ./ids.nix).ids else {};
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir cacheDir forceDisableUserChange processManager ids;
};
in
rec {
apache = rec {
port = ids.httpPorts.apache or 0;
pkg = constructors.simpleWebappApache {
inherit port;
serverAdmin = "root@localhost";
};
requiresUniqueIdsFor = [ "httpPorts" "uids" "gids" ];
};
postgresql = rec {
port = ids.postgresqlPorts.postgresql or 0;
pkg = constructors.postgresql {
inherit port;
};
requiresUniqueIdsFor = [ "postgresqlPorts" "uids" "gids" ];
};
influxdb = rec {
httpPort = ids.influxdbPorts.influxdb or 0;
rpcPort = httpPort + 2;
pkg = constructors.simpleInfluxdb {
inherit httpPort rpcPort;
};
requiresUniqueIdsFor = [ "influxdbPorts" "uids" "gids" ];
};
}
</pre>
<br />
The above processes model exposes three service instances: an Apache HTTP server (that works with a simple configuration that serves web applications from a single virtual host), PostgreSQL and InfluxDB. Each service requires a unique user ID and group ID so that their privileges are separated.<br />
<br />
To make these services more accessible/usable, we do not use a shared ports resource pool. Instead, each service type consumes port numbers from their own resource pools.<br />
<br />
The following ID resources configuration can be used to provision the unique IDs to the services above:<br />
<br />
<pre>
rec {
uids = {
min = 2000;
max = 3000;
};
gids = uids;
httpPorts = {
min = 8080;
max = 8085;
};
postgresqlPorts = {
min = 5432;
max = 5532;
};
influxdbPorts = {
min = 8086;
max = 8096;
step = 3;
};
}
</pre>
<br />
The above ID resources configuration defines a shared UIDs and GIDs resource pool, but separate ports resource pools for each service type. This has the following implications if we deploy multiple instances of each service type:<br />
<br />
<ul>
<li>All Apache HTTP server instances get a TCP port assignment between 8080-8085.</li>
<li>All PostgreSQL server instances get a TCP port assignment between 5432-5532.</li>
<li>All InfluxDB server instances get a TCP port assignment between 8086-8096. Since an InfluxDB allocates two port numbers: one for the HTTP server and one for the RPC service (the latter's port number is the base port number + 2). We use a step count of 3 so that we can retain this convention for each InfluxDB instance.</li>
</ul>
<br />
<h2>Conclusion</h2>
<br />
In this blog post, I have described a new tool: <i>dydisnix-id-assign</i> that can be used to automatically assign unique numeric IDs to services in Disnix service models.<br />
<br />
Moreover, I have described: <i>nixproc-id-assign</i> that acts a thin wrapper around this tool to automatically assign numeric IDs to services in the Nix process management framework's processes model.<br />
<br />
This tool replaces the old <i>dydisnix-port-assign</i> tool in the <a href="https://sandervanderburg.blogspot.com/2016/08/an-extended-self-adaptive-deployment.html">Dynamic Disnix toolset</a> (described in the blog post written five years ago) that is much more limited in its capabilities.<br />
<br />
<h2>Availability</h2>
<br />
The <i>dydisnix-id-assign</i> tool is available in the current development version of <a href="https://github.com/svanderburg/dydisnix">Dynamic Disnix</a>. The <i>nixproc-id-assign</i> is part of the current implementation of the <a href="https://github.com/svanderburg/nix-processmgmt">Nix process management framework prototype</a>.<br />
<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com2tag:blogger.com,1999:blog-1397115249631682228.post-48090080118774471652020-08-11T21:18:00.004+02:002020-12-19T01:15:27.794+01:00Experimenting with Nix and the service management properties of DockerIn <a href="https://sandervanderburg.blogspot.com/2020/07/on-using-nix-and-docker-as-deployment.html">the previous blog post</a>, I have analyzed <a href="https://nixos.org/nix">Nix</a> and <a href="https://www.docker.com">Docker</a> as deployment solutions and described in what ways these solutions are similar and different.<br />
<br />
To summarize my findings:<br />
<br />
<ul>
<li><a href="https://sandervanderburg.blogspot.com/2012/11/an-alternative-explaination-of-nix.html">Nix</a> is a <strong>source-based package manager</strong> responsible for obtaining, installing, configuring and upgrading packages in a reliable and reproducible manner and facilitating the construction of packages from source code and their dependencies.</li>
<li>Docker's purpose is to fully <strong>manage</strong> the life-cycle of <strong>applications</strong> (services and ordinary processes) in a reliable and reproducible manner, including their deployments.</li>
</ul>
<br />
As explained in my previous blog post, two prominent goals both solutions have in common is to facilitate <strong>reliable</strong> and <strong>reproducible</strong> deployment. They both use different kinds of techniques to accomplish these goals.<br />
<br />
Although Nix and Docker can be used for a variety of comparable use cases (such as constructing images, deploying test environments, and constructing packages from source code), one prominent feature that the Nix package manager does not provide is <strong>process</strong> (or service) <strong>management</strong>.<br />
<br />
In a Nix-based workflow you need to augment Nix with another solution that can facilitate process management.<br />
<br />
In this blog post, I will investigate how Docker could fulfill this role -- it is pretty much the opposite goal of the combined use cases scenarios I have shown in the previous blog post, in which Nix can overtake the role of a conventional package manager in supplying packages in the construction process of an image and even the complete construction process of images.<br />
<br />
<h2>Existing Nix integrations with process management</h2>
<br />
Although Nix does not do any process management, there are sister projects that can, such as:<br />
<br />
<ul>
<li><a href="https://sandervanderburg.blogspot.com/2011/01/nixos-purely-functional-linux.html"><strong>NixOS</strong></a> builds entire machine configurations from a single declarative deployment specification and uses the Nix package manager to deploy and isolate all static artifacts of a system. It will also automatically generate and deploy <a href="https://freedesktop.org/wiki/Software/systemd/">systemd</a> units for services defined in a NixOS configuration.</li>
<li><a href="https://github.com/LnL7/nix-darwin"><strong>nix-darwin</strong></a> can be used to specify a collection of services in a deployment specification and uses the Nix package manager to deploy all services and their corresponding <a href="https://www.launchd.info/">launchd</a> configuration files.</li>
</ul>
<br />
Although both projects do a great job (e.g. they both provide a big collection of deployable services) what I consider a disadvantage is that they are <strong>platform specific</strong> -- both solutions only work on a single operating system (Linux and macOS) and a single process management solution (systemd and launchd).<br />
<br />
If you are using Nix in a different environment, such as a different operating system, a conventional (non-NixOS) Linux distribution, or a different process manager, then there is no off-the-shelf solution that will help you managing services for packages provided by Nix.<br />
<br />
<h2>Docker functionality</h2>
<br />
Docker could be considered a multi-functional solution for application management. I can categorize its functionality as follows:<br />
<br />
<ul>
<li><strong>Process management</strong>. The life-cycle of a container is bound to the life-cycle of a root process that needs to be started or stopped.</li>
<li><strong>Dependency management</strong>. To ensure that applications have all the dependencies that they need and that no dependency is missing, Docker uses <strong>images</strong> containing a complete root filesystem with all required files to run an application.</li>
<li><strong>Resource isolation</strong> is heavily used for a variety of different reasons:<br />
<ul>
<li>Foremost, to ensure that the root filesystem of the container does not conflict with the host system's root filesystem.</li>
<li>It is also used to prevent conflicts with other kinds of resources. For example, the isolated network interfaces allow services to bind to the same TCP ports that may also be in use by the host system or other containers.</li>
<li>It offers some degree of protection. For example, a malicious process will not be able to see or control a process belonging to the host system or a different container.</li>
</ul>
</li>
<li><strong>Resource restriction</strong> can be used to limit the amount of system resources that a process can consume, such as the amount of RAM.<br />
<br />
Resource restriction can be useful for a variety of reasons, for example, to prevent a service from eating up all the system's resources affecting the stability of the system as a whole.</li>
<li><strong>Integrations</strong> with the host system (e.g. volumes) and other services.</li>
</ul>
<br />
As described in the previous blog post, Docker uses a number of key concepts to implement the functionality shown above, such as layers, <a href="https://man7.org/linux/man-pages/man7/namespaces.7.html"><strong>namespaces</strong></a> and <a href="https://man7.org/linux/man-pages/man7/cgroups.7.html"><strong>cgroups</strong></a>.<br />
<br />
<h2>Developing a Nix-based process management solution</h2>
<br />
For quite some time, <a href="https://sandervanderburg.blogspot.com/2019/11/a-nix-based-functional-organization-for.html">I have been investigating the process management domain</a> and worked on a <a href="https://github.com/svanderburg/nix-processmgmt">prototype solution</a> to provide a more generalized infrastructure that complements Nix with process management -- I came up with an experimental Nix-based process manager-agnostic framework that has the following objectives:<br />
<br />
<ul>
<li>It uses Nix to <strong>deploy</strong> all required <strong>packages</strong> and other <strong>static artifacts</strong> (such as configuration files) that a service needs.</li>
<li>It integrates with a <strong>variety</strong> of process managers on a variety of operating systems. So far, it can work with: sysvinit scripts, BSD rc scripts, supervisord, systemd, cygrunsrv and launchd.<br />
<br />
In addition to process managers, it can also <a href="https://sandervanderburg.blogspot.com/2020/05/deploying-heterogeneous-service.html">automatically convert a processes model to deployment specifications that Disnix can consume</a>.</li>
<li>It uses <strong>declarative</strong> specifications to define functions that construct managed processes and process instances.<br />
<br />
Processes can be declared in a process-manager specific and <a href="https://sandervanderburg.blogspot.com/2020/02/a-declarative-process-manager-agnostic.html">process-manager agnostic</a> way. The latter makes it possible to target all six supported process managers with the same declarative specification, albeit with a limited set of features.</li>
<li>It allows you to run <strong>multiple instances</strong> of processes, by introducing a convention to cope with potential resource conflicts between process instances -- instance properties and potential conflicts can be configured with function parameters and can be changed in such a way that they do not conflict.</li>
<li>It can facilitate <strong>unprivileged</strong> user deployments by using Nix's ability to perform unprivileged package deployments and introducing a convention that allows you to disable user switching.</li>
</ul>
<br />
To summarize how the solution works from a user point of view, we can write a process manager-agnostic constructor function as follows:<br />
<br />
<pre>
{createManagedProcess, tmpDir}:
{port, instanceSuffix ? "", instanceName ? "webapp${instanceSuffix}"}:
let
webapp = import ../../webapp;
in
createManagedProcess {
name = instanceName;
description = "Simple web application";
inherit instanceName;
process = "${webapp}/bin/webapp";
daemonArgs = [ "-D" ];
environment = {
PORT = port;
PID_FILE = "${tmpDir}/${instanceName}.pid";
};
user = instanceName;
credentials = {
groups = {
"${instanceName}" = {};
};
users = {
"${instanceName}" = {
group = instanceName;
description = "Webapp";
};
};
};
overrides = {
sysvinit = {
runlevels = [ 3 4 5 ];
};
};
}
</pre>
<br />
The Nix expression above is a nested function that defines in a process manager-agnostic way a configuration for a web application process containing an embedded web server serving a static HTML page.<br />
<br />
<ul>
<li>The <strong>outer function header</strong> (first line) refers to parameters that are <strong>common</strong> to all process instances: <i>createManagedProcess</i> is a function that can construct process manager configurations and <i>tmpDir</i> refers to the directory in which temp files are stored (which is <i>/tmp</i> in conventional Linux installations).</li>
<li>The <strong>inner function header</strong> (second line) refers to <strong>instance parameters</strong> -- when it is desired to construct multiple instances of this process, we must make sure that we have configured these parameters in such as a way that they do not conflict with other processes.<br />
<br />
For example, when we assign a unique TCP port and a unique instance name (a property used by the <a href="http://www.libslack.org/daemon/"><i>daemon</i></a> tool to create unique PID files) we can safely have multiple instances of this service co-existing on the same system.</li>
<li>In the body, we invoke the <i>createManagedProcess</i> function to generate configurations files for a process manager.</li>
<li>The <i>process</i> parameter specifies the executable that we need to run to start the process.</li>
<li>The <i>daemonArgs</i> parameter specifies command-line instructions passed to the the process executable, when the process should daemonize itself (the <i>-D</i> parameter instructs the webapp process to daemonize).</li>
<li>The <i>environment</i> parameter specifies all environment variables. Environment variables are used as a generic configuration facility for the service.</li>
<li>The <i>user</i> parameter specifies the name the process should run as (each process instance has its own user and group with the same name as the instance).</li>
<li>The <i>credentials</i> parameter is used to automatically create the group and user that the process needs.</li>
<li>The <i>overrides</i> parameter makes it possible to override the parameters generated by the <i>createManagedProcess</i> function with process manager-specific overrides, to configure features that are not universally supported.<br />
<br />
In the example above, we use an override to configure the <a href="https://wiki.debian.org/RunLevel">runlevels</a> in which the service should run (runlevels 3-5 are typically used to boot a system that is network capable). Runlevels are a sysvinit-specific concept.</li>
</ul>
<br />
In addition to defining constructor functions allowing us to construct zero or more process instances, we also need to construct process instances. These can be defined in a <strong>processes model</strong>:<br />
<br />
<pre>
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
, processManager
}:
let
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir;
inherit forceDisableUserChange processManager;
};
in
rec {
webapp = rec {
port = 5000;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
};
nginxReverseProxy = rec {
port = 8080;
pkg = constructors.nginxReverseProxyHostBased {
webapps = [ webapp ];
inherit port;
} {};
};
}
</pre>
<br />
The above Nix expressions defines two process instances and uses the following conventions:<br />
<br />
<ul>
<li>The first line is a function header in which the function parameters correspond to ajustable properties that apply to all process instances:
<ul>
<li><i>stateDir</i> allows you to globally override the base directory in which all state is stored (the default value is: <i>/var</i>).</li>
<li>We can also change the locations of each individual state directories: <i>tmpDir</i>, <i>cacheDir</i>, <i>logDir</i>, <i>runtimeDir</i> etc.) if desired.</li>
<li><i>forceDisableUserChange</i> can be enabled to prevent the process manager to change user permissions and create users and groups. This is useful to facilitate unprivileged user deployments in which the user typically has no rights to change user permissions.</li>
<li>The <i>processManager</i> parameter allows you to pick a process manager. All process configurations will be automatically generated for the selected process manager.<br />
<br />
For example, if we would pick: <i>systemd</i> then all configurations get translated to systemd units. <i>supervisord</i> causes all configurations to be translated to supervisord configuration files.</li>
</ul>
</li>
<li>To get access to constructor functions, we import a <strong>constructors expression</strong> that composes all constructor functions by calling them with their common parameters (not shown in this blog post).<br />
<br />
The constructors expression also contains a reference to the Nix expression that deploys the webapp service, shown in our previous example.</li>
<li>The processes model defines two processes: a <i>webapp</i> instance that listens to TCP port 5000 and Nginx that acts as a reverse proxy forwarding requests to <i>webapp</i> process instances based on the virtual host name.</li>
<li><i>webapp</i> is declared a <strong>dependency</strong> of the <i>nginxReverseProxy</i> service (by passing <i>webapp</i> as a parameter to the constructor function of Nginx). This causes <i>webapp</i> to be activated before the <i>nginxReverseProxy</i>.</li>
</ul>
<br />
To deploy all process instances with a process manager, we can invoke a variety of tools that are bundled with the experimental Nix process management framework.<br />
<br />
The process model can be deployed as sysvinit scripts for an unprivileged user, with the following command:<br />
<br />
<pre>
$ nixproc-sysvinit-switch --state-dir /home/sander/var \
--force-disable-user-change processes.nix
</pre>
<br />
The above command automatically generates sysvinit scripts, changes the base directory of all state folders to a directory in the user's home directory: <i>/home/sander/var</i> and disables user changing (and creation) so that an unprivileged user can run it.<br />
<br />
The following command uses systemd as a process manager with the default parameters, for production deployments:<br />
<br />
<pre>
$ nixproc-systemd-switch processes.nix
</pre>
<br />
The above command automatically generates systemd unit files and invokes systemd to deploy the processes.<br />
<br />
In addition to the examples shown above, the framework contains many more tools, such as: <i>nixproc-supervisord-switch</i>, <i>nixproc-launchd-switch</i>, <i>nixproc-bsdrc-switch</i>, <i>nixproc-cygrunsrv-switch</i>, and <i>nixproc-disnix-switch</i> that all work with the same processes model.<br />
<br />
<h2>Integrating Docker into the process management framework</h2>
<br />
Both Docker and the Nix-based process management framework are multi-functional solutions. After comparing the functionality of Docker and the process management framework, I realized that it is possible to integrate Docker into this framework as well, if I would use it in an unconventional way, by disabling or substituting some if its conflicting features.<br />
<br />
<h3>Using a shared Nix store</h3>
<br />
As explained in the beginning of this blog post, Docker's primary means to provide dependencies is by using images that are self-contained root file systems containing all necessary files (e.g. packages, configuration files) to allow an application to work.<br />
<br />
In the previous blog post, I have also demonstrated that instead of using traditional <i>Dockerfile</i>s to construct images, we can also use the Nix package manager as a replacement. A Docker image built by Nix is typically smaller than a conventional Docker image built from a base Linux distribution, because it only contains the runtime dependencies that an application actually needs.<br />
<br />
A major disadvantage of using Nix constructed Docker images is that they only consist of one layer -- as a result, there is no reuse between container instances running different services that use common libraries. To alleviate this problem, Nix can also build layered images, in which common dependencies are isolated in separate layers as much as possible.<br />
<br />
There is even a more optimal reuse strategy possible -- when running Docker on a machine that also has Nix installed, we do not need to put anything that is in the Nix store in a disk image. Instead, we can <strong>share</strong> the host system's Nix store between Docker containers.<br />
<br />
This may sound scary, but as I have explained in the previous blog post, paths in the Nix store are prefixed with SHA256 hash codes. When two Nix store paths with identical hash codes are built on two different machines, their build results should be (nearly) bit-identical. As a result, it is safe to share the same Nix store path between multiple machines and containers.<br />
<br />
A hacky solution to build a container image, without actually putting any of the Nix built packages in the container, can be done with the following expression:<br />
<br />
<pre>
with import <nixpkgs> {};
let
cmd = [ "${nginx}/bin/nginx" "-g" "daemon off;" "-c" ./nginx.conf ];
in
dockerTools.buildImage {
name = "nginxexp";
tag = "test";
runAsRoot = ''
${dockerTools.shadowSetup}
groupadd -r nogroup
useradd -r nobody -g nogroup -d /dev/null
mkdir -p /var/log/nginx /var/cache/nginx /var/www
cp ${./index.html} /var/www/index.html
'';
config = {
Cmd = map (arg: builtins.unsafeDiscardStringContext arg) cmd;
Expose = {
"80/tcp" = {};
};
};
}
</pre>
<br />
The above expression is quite similar to the Nix-based Docker image example shown in the previous blog post, that deploys Nginx serving a static HTML page.<br />
<br />
The only difference is how I configure the start command (the <i>Cmd</i> parameter). In the Nix expression language, <a href="https://shealevy.com/blog/2018/08/05/understanding-nixs-string-context">strings have <strong>context</strong></a> -- if a string with context is passed to a build function (any string that contains a value that evaluates to a Nix store path), then the corresponding Nix store paths automatically become a dependency of the package that the build function builds.<br />
<br />
By using the unsafe <i>builtins.unsafeDiscardStringContext</i> function I can discard the context of strings. As a result, the Nix packages that the image requires are still built. However, because their context is discarded they are no longer considered dependencies of the Docker image. As a consequence, they will not be integrated into the image that the <i>dockerTools.buildImage</i> creates.<br />
<br />
(As a sidenote: there are still two Nix store paths that end-up in the image, namely <i>bash</i> and <i>glibc</i> that is a runtime dependency of <i>bash</i>. This is caused by the fact that the internals of the <i>dockerTools.buildImage</i> function make a reference to <i>bash</i> without discarding its context. In theory, it is also possible to eliminate this dependency as well).<br />
<br />
To run the container and make sure that the required Nix store paths are available, I can mount the host system's Nix store as a shared volume:<br />
<br />
<pre>
$ docker run -p 8080:80 -v /nix/store:/nix/store -it nginxexp:latest
</pre>
<br />
By mounting the host system's Nix store (with the <i>-v</i> parameter), Nginx should still behave as expected -- it is not provided by the image, but referenced from the shared Nix store.<br />
<br />
(As a sidenote: mounting the host system's Nix store for sharing is not a new idea. It has already been intensively used by the <a href="https://sandervanderburg.blogspot.com/2011/02/using-nixos-for-declarative-deployment.html">NixOS test driver</a> for many years to rapidly create QEMU virtual machines for system integration tests).<br />
<br />
<h3>Using the host system's network</h3>
<br />
As explained in the previous blog post, every Docker container by default runs in its own private network namespace making it possible for services to bind to any port without conflicting with the services on the host system or services provided by any other container.<br />
<br />
The Nix process management framework does not work with private networks, because it is not a generalizable concept (i.e. namespaces are a Linux-only feature). Aside from Docker, the only other process manager supported by the framework that can work with namespaces is systemd.<br />
<br />
To prevent ports and other dynamic resources from conflicting with each other, the process management framework makes it possible to configure them through instance function parameters. If the instance parameters have unique values, they will not conflict with other process instances (based on the assumption that the packager has identified all possible conflicts that a process might have).<br />
<br />
Because we already have a framework that prevents conflicts, we can also instruct Docker to use the host system's network with the <i>--network host</i> parameter:<br />
<br />
<pre>
$ docker run -v /nix/store:/nix/store --network host -it nginxexp:latest
</pre>
<br />
The only thing the framework cannot provide you is protection -- malicious services in a private network namespace cannot connect to ports used by other containers or the host system, but the framework cannot protect you from that.<br />
<br />
<h3>Mapping a base directory for storing state</h3>
<br />
Services that run in containers are not always stateless -- they may rely on data that should be persistently stored, such as databases. <a href="https://developers.redhat.com/blog/2016/02/24/10-things-to-avoid-in-docker-containers/">The Docker recommendation to handle persistent state</a> is not to store it in a container's writable layer, but on a shared volume on the host system.<br />
<br />
Data stored outside the container makes it possible to reliably upgrade a container -- when it is desired to install a newer version of an application, the container can be discarded and recreated from a new image.<br />
<br />
For the Nix process management framework, integration with a state directory outside the container is also useful. With an extra shared volume, we can mount the host system's state directory:<br />
<br />
<pre>
$ docker run -v /nix/store:/nix/store \
-v /var:/var --network host -it nginxexp:latest
</pre>
<br />
<h3>Orchestrating containers</h3>
<br />
The last piece in the puzzle is to orchestrate the containers: we must create or discard them, and start or stop them, and perform all required steps in the right order.<br />
<br />
Moreover, to prevent the Nix packages that a containers needs from being garbage collected, we need to make sure that they are a dependency of a package that is registered as in use.<br />
<br />
I came up with my own convention to implement the container deployment process. When building the processes model for the <i>docker</i> process manager, the following files are generated that help me orchestrating the deployment process:<br />
<br />
<pre>
01-webapp-docker-priority
02-nginx-docker-priority
nginx-docker-cmd
nginx-docker-createparams
nginx-docker-settings
webapp-docker-cmd
webapp-docker-createparams
webapp-docker-settings
</pre>
<br />
In the above list, we have the following kinds of files:<br />
<br />
<ul>
<li>The files that have a <i>-docker-settings</i> suffix contain general properties of the container, such as the image that needs to be used a template.</li>
<li>The files that have a <i>-docker-createparams</i> suffix contain the command line parameters that are propagated to <i>docker create</i> to create the container. If a container with the same name already exists, the container creation is skipped and the existing instance is used instead.</li>
<li>To prevent the Nix packages that a Docker container needs from being garbage collected the generator creates a file with a <i>-docker-cmd</i> suffix containing the <i>Cmd</i> instruction including the full Nix store paths of the packages that a container needs.<br />
<br />
Because the strings' contexts are not discarded in the generation process, the packages become a dependency of the configuration file. As long as this configuration file is deployed, the packages will not get garbage collected.</li>
<li>To ensure that the containers are activated in the right order we have two files that are prefixed with two numeric digits that have a <i>-container-priority</i> suffix. The numeric digits determine in which order the containers should be activated -- in the above example the webapp process gets activated before Nginx (that acts as a reverse proxy).</li>
</ul>
<br />
With the following command, we can automatically generate the configuration files shown above for all our processes in the processes model, and use it to automatically create and start docker containers for all process instances:<br />
<br />
<pre style="overflow: auto;">
$ nixproc-docker-switch processes.nix
55d833e07428: Loading layer [==================================================>] 46.61MB/46.61MB
Loaded image: webapp:latest
f020f5ecdc6595f029cf46db9cb6f05024892ce6d9b1bbdf9eac78f8a178efd7
nixproc-webapp
95b595c533d4: Loading layer [==================================================>] 46.61MB/46.61MB
Loaded image: nginx:latest
b195cd1fba24d4ec8542c3576b4e3a3889682600f0accc3ba2a195a44bf41846
nixproc-nginx
</pre>
<br />
The result is two running Docker containers that correspond to the process instances shown in the processes model:<br />
<br />
<pre style="overflow: auto;">
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
b195cd1fba24 nginx:latest "/nix/store/j3v4fz9h…" 15 seconds ago Up 14 seconds nixproc-nginx
f020f5ecdc65 webapp:latest "/nix/store/b6pz847g…" 16 seconds ago Up 15 seconds nixproc-webapp
</pre>
<br />
and we should be able to access the example HTML page, by opening the following URL: <i>http://localhost:8080</i> in a web browser.<br />
<br />
<h2>Deploying Docker containers in a heterogeneous and/or distributed environment</h2>
<br />
As explained in my previous blog posts about the experimental Nix process management framework, the processes model is a sub set of a <a href="https://sandervanderburg.blogspot.com/2011/02/disnix-toolset-for-distributed.html">Disnix</a> <strong>services</strong> model. When it is desired to deploy processes to a network of machines or combine processes with other kinds of services, we can easily turn a processes model into a services model.<br />
<br />
For example, I can change the processes model shown earlier into a services model that deploys Docker containers:<br />
<br />
<pre>
{ pkgs ? import <nixpkgs> { inherit system; }
, system ? builtins.currentSystem
, stateDir ? "/var"
, runtimeDir ? "${stateDir}/run"
, logDir ? "${stateDir}/log"
, cacheDir ? "${stateDir}/cache"
, tmpDir ? (if stateDir == "/var" then "/tmp" else "${stateDir}/tmp")
, forceDisableUserChange ? false
}:
let
constructors = import ./constructors.nix {
inherit pkgs stateDir runtimeDir logDir tmpDir;
inherit forceDisableUserChange;
processManager = "docker";
};
in
rec {
webapp = rec {
name = "webapp";
port = 5000;
dnsName = "webapp.local";
pkg = constructors.webapp {
inherit port;
};
type = "docker-container";
};
nginxReverseProxy = rec {
name = "nginxReverseProxy";
port = 8080;
pkg = constructors.nginxReverseProxyHostBased {
webapps = [ webapp ];
inherit port;
} {};
type = "docker-container";
};
}
</pre>
<br />
In the above example, I have added a <i>name</i> attribute to each process (a required property for Disnix service models) and a <i>type</i> attribute referring to: <i>docker-container</i>.<br />
<br />
In Disnix, a service could take any form. A plugin system (named <a href="https://sandervanderburg.blogspot.com/2015/07/deploying-state-with-disnix.html">Dysnomia</a>) is responsible for managing the life-cycle of a service, such as activating or deactivating it. The <i>type</i> attribute is used to tell Disnix that we should use the <i>docker-container</i> Dysnomia module. This module will automatically create and start the container on activation, and stop and discard the container on deactivation.<br />
<br />
To deploy the above services to a network of machines, we require an <strong>infrastructure model</strong> (that captures the available machines and their relevant deployment properties):<br />
<br />
<pre>
{
test1.properties.hostname = "test1";
}
</pre>
<br />
The above infrastructure model contains only one target machine: <i>test1</i> with a hostname that is identical to the machine name.<br />
<br />
We also require a <strong>distribution model</strong> that maps services in the services model to machines in the infrastructure model:<br />
<br />
<pre>
{infrastructure}:
{
webapp = [ infrastructure.test1 ];
nginxReverseProxy = [ infrastructure.test1 ];
}
</pre>
<br />
In the above distribution model, we map the all the processes in the services model to the <i>test1</i> target machine in the infrastructure model.<br />
<br />
With the following command, we can deploy our Docker containers to the remote <i>test1</i> target machine:<br />
<br />
<pre>
$ disnix-env -s services.nix -i infrastructure.nix -d distribution.nix
</pre>
<br />
When the above command succeeds, the <i>test1</i> target machine provides running <i>webapp</i> and <i>nginxReverseProxy</i> containers.<br />
<br />
(As a sidenote: to make Docker container deployments work with Disnix, the Docker service already needs to be predeployed to the target machines in the infrastructure model, or the Docker daemon needs to be deployed as a <a href="https://sandervanderburg.blogspot.com/2020/04/deploying-container-and-application.html">container provider</a>).<br />
<br />
<h2>Deploying conventional Docker containers with Disnix</h2>
<br />
The nice thing about the <i>docker-container</i> Dysnomia module is that it is generic enough to also work with conventional Docker containers (that work with images, not a shared Nix store).<br />
<br />
For example, we can deploy Nginx as a regular container built with the <i>dockerTools.buildImage</i> function:<br />
<br />
<pre style="font-size: 90%; overflow: auto;">
{dockerTools, stdenv, nginx}:
let
dockerImage = dockerTools.buildImage {
name = "nginxexp";
tag = "test";
contents = nginx;
runAsRoot = ''
${dockerTools.shadowSetup}
groupadd -r nogroup
useradd -r nobody -g nogroup -d /dev/null
mkdir -p /var/log/nginx /var/cache/nginx /var/www
cp ${./index.html} /var/www/index.html
'';
config = {
Cmd = [ "${nginx}/bin/nginx" "-g" "daemon off;" "-c" ./nginx.conf ];
Expose = {
"80/tcp" = {};
};
};
};
in
stdenv.mkDerivation {
name = "nginxexp";
buildCommand = ''
mkdir -p $out
cat > $out/nginxexp-docker-settings <<EOF
dockerImage=${dockerImage}
dockerImageTag=nginxexp:test
EOF
cat > $out/nginxexp-docker-createparams <<EOF
-p
8080:80
EOF
'';
}
</pre>
<br />
In the above example, instead of using the process manager-agnostic <i>createManagedProcess</i>, I directly construct a Docker-based Nginx image (by using the <i>dockerImage</i> attribute) and container configuration files (in the <i>buildCommand</i> parameter) to make the container deployments work with the <i>docker-container</i> Dysnomia module.<br />
<br />
It is also possible to deploy containers from images that are constructed with <i>Dockerfile</i>s. After we have built an image in the traditional way, we can export it from Docker with the following command:<br />
<br />
<pre>
$ docker save nginx-debian -o nginx-debian.tar.gz
</pre>
<br />
and then we can use the following Nix expression to deploy a container using our exported image:<br />
<br />
<pre>
{dockerTools, stdenv, nginx}:
stdenv.mkDerivation {
name = "nginxexp";
buildCommand = ''
mkdir -p $out
cat > $out/nginxexp-docker-settings <<EOF
dockerImage=${./nginx-debian.tar.gz}
dockerImageTag=nginxexp:test
EOF
cat > $out/nginxexp-docker-createparams <<EOF
-p
8080:80
EOF
'';
}
</pre>
<br />
In the above expression, the <i>dockerImage</i> property refers to our exported image.<br />
<br />
Although Disnix is flexible enough to also orchestrate Docker containers (thanks to its generalized plugin architecture), I did not develop the <i>docker-container</i> Dysnomia module to make Disnix compete with existing container orchestration solutions, such as <a href="https://kubernetes.io/">Kubernetes</a> or <a href="https://docs.docker.com/engine/swarm/">Docker Swarm</a>.<br />
<br />
Disnix is a heterogeneous deployment tool that can be used to integrate units that have all kinds of shapes and forms on all kinds of operating systems -- having a <i>docker-container</i> module makes it possible to mix Docker containers with other service types that Disnix and Dysnomia support.<br />
<br />
<h2>Discussion</h2>
<br />
In this blog post, I have demonstrated that we can integrate Docker as a process management backend option into the experimental Nix process management framework, by substituting some of its conflicting features.<br />
<br />
Moreover, because a Disnix service model is a superset of a processes model, we can also use Disnix as a simple Docker container orchestrator and integrate Docker containers with other kinds of services.<br />
<br />
Compared to Docker, the Nix process management framework supports a number of features that Docker does not:<br />
<br />
<ul>
<li>Docker is heavily developed around Linux-specific concepts, such as namespaces and cgroups. As a result, it can only be used to deploy software built for Linux.<br />
<br />
The Nix process management framework should work on any operating system that is supported by the Nix package manager (e.g. Nix also has first class support for macOS, and can also be used on other UNIX-like operating systems such as FreeBSD). The same also applies to Disnix.</li>
<li>The Nix process management framework can work with <i>sysvinit</i>, <i>BSD rc</i> and Disnix process scripts, that do not require any external service to manage a process' life-cycle. This is convenient for local unprivileged user deployments. To deploy Docker containers, you need to have the Docker daemon installed first.</li>
<li>Docker has an experimental rootless deployment mode, but in the Nix process management framework facilitating unprivileged user deployments is a first class concept.</li>
</ul>
<br />
On the other hand, the Nix process management framework does not take over all responsibilities of Docker:<br />
<br />
<ul>
<li>Docker heavily relies on namespaces to prevent resource conflicts, such as overlapping TCP ports and global state directories. The Nix process management framework solves conflicts by avoiding them (i.e. configuring properties in such a way that they are unique). The conflict avoidance approach works as long as a service is well-specified. Unfortunately, preventing conflicts is not a hard guarantee that the tool can provide you.</li>
<li>Docker also provides some degree of protection by using namespaces and cgroups. The Nix process management framework does not support this out of the box, because these concepts are not generalizable over all the process management backends it supports. (As a sidenote: it is still possible to use these concepts by defining process manager-specific overrides).</li>
</ul>
<br />
From a functionality perspective, <a href="https://docs.docker.com/compose"><i>docker-compose</i></a> comes close to the features that the experimental Nix process management framework supports. <i>docker-compose</i> allows you to declaratively define container instances and their dependencies, and automatically deploy them.<br />
<br />
However, as its name implies <i>docker-compose</i> is specifically designed for deploying Docker containers whereas the Nix process management framework is more general -- it should work with all kinds of process managers, uses Nix as the primary means to provide dependencies, it uses the Nix expression language for configuration and it should work on a variety of operating systems.<br />
<br />
The fact that Docker (and containers in general) are multi-functional solutions is not an observation only made by me. For example, <a href="https://iximiuz.com/en/posts/you-dont-need-an-image-to-run-a-container">this blog post</a> also demonstrates that containers can work without images.<br />
<br />
<h2>Availability</h2>
<br />
The Docker backend has been integrated into the <a href="https://github.com/svanderburg/nix-processmgmt">latest development version</a> of the Nix process management framework.<br />
<br />
To use the <i>docker-container</i> Dysnomia module (so that Disnix can deploy Docker containers), you need to install the latest development version of Dysnomia.<br />
Sander van der Burghttp://www.blogger.com/profile/12718166966821611609noreply@blogger.com0