For some reason, I need to perform deployment activities from JavaScript programs. To do that, I could (re)implement the deployment mechanisms that I need from scratch, such a feature that builds packages and a feature that fetches dependencies from external repositories.
What happens in practice is that people exactly do this -- they create programming language specific package managers, implementing features that are already well supported by generic tools. Nowadays, almost every modern programming language environment has one, such as the Node.js Package Manager, CPAN, HackageDB, Eclipse plugin manager etc.
Each language-specific package manager implements deployment activities in their own way. Some of these package managers have good deployment properties, others have annoying drawbacks. Some of them can easily integrate or co-exist with the host's systems package manager, others cannot. Most importantly, they don't offer the non-functional properties I care about, such as reliable and reproducible deployment.
For me, Nix offers the stuff I want. Therefore, I want to integrate it with my programs implemented in a general purpose programming language.
A common integration solution is to generate Nix expressions through string manipulation and to invoke the nix-build (or nix-env) processes to build it. For every deployment operation or configuration step, a developer is required to generate an expression that builds or configures something (by string manipulation, that is unparsed and unchecked) and pass that to Nix, which is inelegant, laborious, tedious, error-prone, and results in more code that needs to be maintained.
As a solution for this, I have created NiJS: An internal DSL for Nix in JavaScript.
Calling Nix functions from JavaScript
As explained ealier, manually generating Nix expressions as strings have various drawbacks. In earlier blog posts, I have explained that in Nix (which is a package manager borrowing concepts from purely functional programming languages) every build action is modeled as a function and the expressions that we typically want to evaluate (or generate) are function invocations.
To me, it looks like a better and more elegant idea to be able to call these Nix functions through function calls from the implementation language, rather than generating these function invocations as strings. This is how the idea of NiJS was born.
For example, it would be nice to have the following JavaScript function invocation that calls the stdenv.mkDerivation function of Nixpkgs, which is used to build a particular package from source code and its given build-time dependencies:
stdenv().mkDerivation ({
name : "hello-2.8",
src : args.fetchurl({
url : "mirror://gnu/hello/hello-2.8.tar.gz",
sha256 : "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6"
}),
doCheck : true,
meta : {
description : "A program that produces a familiar, friendly greeting",
homepage : {
_type : "url",
value : "http://www.gnu.org/software/hello/manual"
},
license : "GPLv3+"
}
});
The above JavaScript example, defines how to build GNU Hello -- a trivial example package -- from source code. We can easily translate that function call to the following Nix expression containing a function call to stdenv.mkDerivation:
let
pkgs = import <nixpkgs> {};
in
pkgs.stdenv.mkDerivation {
name = "hello-2.8";
src = pkgs.fetchurl {
url = "mirror://gnu/hello/hello-2.8.tar.gz";
sha256 = "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6";
};
doCheck = true;
meta = {
description = "A program that produces a familiar, friendly greeting";
homepage = http://www.gnu.org/software/hello/manual;
license = "GPLv3+";
}
}
The above expression can be passed to nix-build to actually build GNU Hello.To actually make the JavaScript function call work, I had to define the mkDerivation() JavaScript function as follows:
var mkDerivation = function(args) {
return {
_type : "nix",
value : "pkgs.stdenv.mkDerivation "+nijs.jsToNix(args)
};
}
The function takes an object as a parameter and returns an object with the type property set to nix (more details on this later), and a value property that is a string containing the generated Nix expression.As you may notice from the code example, only little string manipulation is done. The Nix expression is generated almost automatically. We compose the Nix expression that needs to be evaluated, by putting the name of the Nix function that we want to call in a string and we append the output of the jsToNix() function invocation to args, the argument of the JavaScript function.
The jsToNix() function is a very powerful one -- it takes objects from the JavaScript language and translates them to Nix language objects, in a generic and straight forward manner. For example, it translates a JavaScript object into a Nix attribute set (with the same properties), a JavaScript array into an expression having a list, and so on.
The resulting object (with the nix type) can be passed to the callNixBuild() function, which adds the import <nixpkgs> {}; statement to the beginning of the expression and invokes nix-build to evaluate it, which as a side-effect, builds GNU Hello.
Defining packages in NiJS
We have just shown how to invoke a Nix function from JavaScript, by creating a trivial proxy that translates the JavaScript function call to a string with a Nix function call.
I have intentionally used stdenv.mkDerivation as an example. While I could have picked another example, such as writeTextFile that writes a string to a text file, I did not do this.
The stdenv.mkDerivation function is a very important function in Nix -- it's directly and indirectly called from almost every Nix expression building a package. By creating a proxy to this function from JavaScript, we have created a new range of possibilities -- we can also use this proxy to write package build recipes and their compositions inside JavaScript instead of the Nix expression language.
As explained in earlier blog posts, a Nix package description is a file that defines a function taking its required build-inputs as function arguments. The body of the function describes how to build the package from source code and its build-time dependencies.
In JavaScript, we can also do something like this. We can define each package as a separate CommonJS module that exports a pkg property. The pkg property refers to a function declaration, in which we describe how to build a package from source code and its dependencies provided as function arguments:
exports.pkg = function(args) {
return args.stdenv().mkDerivation ({
name : "hello-2.8",
src : args.fetchurl({
url : "mirror://gnu/hello/hello-2.8.tar.gz",
sha256 : "0wqd8sjmxfskrflaxywc7gqw7sfawrfvdxd9skxawzfgyy0pzdz6"
}),
doCheck : true,
meta : {
description : "A program that produces a familiar, friendly greeting",
homepage : {
_type : "url",
value : "http://www.gnu.org/software/hello/manual"
},
license : "GPLv3+"
}
});
};
The result of the function call is a string that contains our generated Nix expression.
As explained in earlier blog posts about Nix, we cannot use these function declarations to build a package directly, but we have to compose them by calling the function with it arguments. These function arguments provide a particular version of a dependency. In NiJS, we can compose packages in a CommonJS module that looks like this:
var pkgs = {
stdenv : function() {
return require('./pkgs/stdenv.js').pkg;
},
fetchurl : function(args) {
return require('./pkgs/fetchurl.js').pkg(args);
},
hello : function() {
return require('./pkgs/hello.js').pkg({
stdenv : pkgs.stdenv,
fetchurl : pkgs.fetchurl
});
},
zlib : function() {
return require('./pkgs/zlib.js').pkg;
},
file : function() {
return require('./pkgs/file.js').pkg({
stdenv : pkgs.stdenv,
fetchurl : pkgs.fetchurl,
zlib : pkgs.zlib
});
},
...
};
exports.pkgs = pkgs;
The above module defines a pkgs property that contains an object in which each member refers to a function. Each function invokes a CommonJS module that builds a package, such as the GNU Hello example that we have shown earlier, and passes its required dependencies as function arguments. The dependencies are defined in the same composition object, such as stdenv and fetchurl.
Apart from the language, the major difference between this composition module and the top-level composition expression in Nix, is that we have added an extra function indirection for each object member. In Nix, the composition expression is an attribute set with function calls. Because Nix is a lazy-language, these function calls only get evaluated when they are needed.
JavaScript is an eager language and will evaluate all function invocations when the composition object is generated. To prevent this, we have wrapped these invocations in functions that need to be called. This also explains why we refer to stdenv (and any other package) as a function call in the GNU Hello example.
By importing the composition expression shown earlier and by passing the result of the application of one of its function members to callNixBuild(), a package such as GNU Hello can be built:
var nijs = require('nijs');
var pkgs = require('pkgs.js').pkgs;
nijs.callNixBuild({
nixObject : pkgs.hello(),
onSuccess : function(result) {
process.stdout.write(result + "\n");
},
onFailure : function(code) {
process.exit(code);
}
});
The above fragment asynchronously builds GNU Hello and writes its resulting Nix store path to the standard output (done by the onSuccess() callback function). As building individual packages from a composition specification is a common use case, I have created a command-line utility: nijs-build that can automatically do this, which is convenient for testing:
$ nijs-build pkgs.js -A hello /nix/store/xkbqlb0w5snmrxqi6ysixfszx1wc7mqd-hello-2.8
The above command-line instruction builds GNU Hello defined in our composition JavaScript specification.
Translating JavaScript objects to Nix expression language objects
We have seen that the jsToNix() function performs most of the "magic". Most of the mappings from JavaScript to Nix are straight forward, mostly by using JavaScript's typeof operator:
- Boolean and number values can be converted verbatim
- Strings and XML objects can be almost taken verbatim, but quotes must be placed around them and they must be properly escaped.
- Objects in JavaScript are a bit trickier, as arrays are also objects. We must use the Array.isArray() method to check for this. We can recursively convert objects into either Nix lists or Nix attribute sets.
- null values can't be determined by type. We must check for the null reference explicitly.
- Objects that have an undefined type will throw an exception.
- Recursive attribute sets can be generated by adding the: _recursive = true member to an object.
- URLs can be defined by creating an object with the: _type = "url" member and a value member containing the URL in a string.
- Files can be defined by creating an object with the: _type = "file" member and a value containing the file path. File names with spaces are also allowed and require a special trick in the Nix expression language to allow it to work properly. Another tricky part is that file names can be absolute or relative. In order to make the latter case work, we have to know the path of to the CommonJS module that is referencing a file. Fortunately the module.filename property inside a CommonJS module will exactly tell us that. By passing the module parameter to the file object, we can make this work.
We also need to distinguish between parts that have already been converted. The proxies, such as the one to stdenv.mkDerivation are just pieces of Nix expression code that must be used without conversion. For that, I have introduced objects with the nix type, as shown earlier that are just placed into the generated expression verbatim.
Then there is still one open question. JavaScript also has objects that have the function type. What to do with these? Should we disallow them, or is there a way in which we can allow them to be used in our internal DSL?
Converting JavaScript functions to Nix expressions
I'm not aware of any Nix expression language construct that is semantically equivalent (or similar) to a JavaScript function. The Nix expression language has functions, but these are executed lazily and can only use primops in the Nix expression language. Therefore, we cannot really "compile" JavaScript functions to Nix functions.
However, I do know a way to call arbitrary processes from Nix expressions (through a derivation) and to expose them as functions that return Nix language objects, by converting the process' output to Nix expressions that are imported. I have used this trick earlier in our SEAMS paper to integrate deployment planning algorithms. I can also use the same trick to allow JavaScript functions to be called from Nix expressions:
{stdenv, nodejs}:
{function, args}:
let
nixToJS = ...
in
import (stdenv.mkDerivation {
name = "function-proxy";
buildInputs = [ nodejs ];
buildCommand = ''
(
cat <<EOF
var nijs = require('${./nijs.js}');
var fun = ${function};
var args = [
${stdenv.lib.concatMapStrings (arg: nixToJS arg+",\n") args}
];
var result = fun.apply(this, args);
process.stdout.write(nijs.jsToNix(result));
EOF
) | node > $out
'';
})
The above code fragment shows how the nijsFunProxy function is defined that can be used to create a proxy to a JavaScript function and allows somebody to call it as if it was a Nix function.The proxy takes a JavaScript function definition in a string and a list of function arguments that can be of any Nix expression language type. Then the parameters are converted to JavaScript objects (through our inverse nixToJS() function) and a JavaScript file is generated that performs the function invocation with the converted parameters. Finally, the resulting JavaScript object returned by the function is converted to a Nix expression and written to the Nix store. By importing the generated Nix expression we can return an equivalent Nix language object.
In JavaScript, every function definition is in fact an object that is an instance of the Function prototype. The length property will tell us the number of command-line arguments, the toString() method will dump a string representation of the function definition, and apply() can be used to evaluate the function object with an array of arguments.
By using these properties, we can generate a Nix function wrapper with an invocation to nijsFunProxy that looks like this:
fun =
arg0: arg1:
nijsFunProxy {
function = ''
function sumTest(a, b) {
return a + b;
}
'';
args = [ arg0 arg1 ];
};
By creating such wrappers calling the nijsFunProxy, we can "translate" JavaScript functions to Nix and call them from Nix expressions. However, there are a number of caveats:
- We cannot use variables outside the scope of the function, e.g. global variables.
- We must always return something. If nothing is returned, we will have an undefined object, which cannot be converted. Nix 'void functions' don't exist.
- Functions with a variable number of positional arguments are not supported, as Nix functions don't support this.
It may be a bit inconvenient to use self-contained JavaScript functions, as we cannot access anything outside the scope of JavaScript function. Fortunately, the proxy can also include specified CommonJS modules, that provide standard functionality. See the package's documentation for more details on this.
Calling JavaScript functions from Nix expressions
Of course, the nijsFunProxy can also be used directly from ordinary Nix expressions, if desired. The following expression uses a JavaScript function that adds two integers. It writes the result of an addition to a file in the Nix store:
{stdenv, nijsFunProxy}:
let
sum = a: b: nijsFunProxy {
function = ''
function sum(a, b) {
return a + b;
}
'';
args = [ a b ];
};
in
stdenv.mkDerivation {
name = "sum";
buildCommand = ''
echo ${toString (sum 1 2)} > $out
'';
}
Conclusion
In this blog post, I have described NiJS: an internal DSL for Nix in JavaScript. It offers the following features:
- A way to easily generate function invocations to Nix functions from JavaScript.
- A translation function that maps JavaScript language objects to Nix expression language objects.
- A way to define and compose Nix packages in JavaScript.
- A way to invoke JavaScript functions from Nix expressions
I'd also like to point out that NiJS is not supposed to be a Nix alternative. It's rather a convenient means to use Nix from a general purpose language (in this case: JavaScript). Some of the additional use cases were implications of having a proxy and interesting to experiment with. :-)
Finally, I think NiJS may also give people that are unfamiliar with the Nix expression language the feeling that they don't have to learn it, because packages can be directly created from JavaScript in an environment that they are used to. I think this is a false sense of security. Although we can build packages from JavaScript, under the hood still Nix expressions are being generated that may yield errors. Errors from NiJS objects are much harder to debug. In such cases it's still required to know what happens.
Moreover, I find JavaScript and the asynchronous programming model of Node.js ugly and very error prone, as the language does not restrict you of doing all kinds of harmful things. But that's a different discussion. People may also find this blog post about Node.js interesting to read.
Related work
Maybe this internal DSL approach on top of an external DSL approach sounds crazy, but it's not really unique to NiJS, nor is NiJS the only internal DSL approach for Nix:
- GNU Guix is an internal DSL for Nix in Scheme (through GNU Guile). Guix is a more sophisticated approach with the intention of deploying a complete GNU distribution. It also generates lower-level Nix store derivation files, instead of Nix expressions. NiJS has a much simpler implementation, fewer use-cases and a different goal.
- ORM systems such as Hibernate can also be considered an internal DSL (Java) on-top-of an external DSL (SQL) approach. They allow developers to treat records in database tables as (collections of) objects in a general purpose programming language. They offer various advantages, such as less boilerplate code, but also disadvantages, such as the ORM mismatch, resulting in issues, such as performance penalties. NiJS has also issues related due to mismatches between the source and target language, such as debugging issues.
Availability
NiJS can be obtained from the NiJS GitHub page and used under the MIT license. The package also includes a number of example packages in JavaScript and a number of example JavaScript function invocations from Nix expressions.
No comments:
Post a Comment