For example, an interesting consideration is that when a different programming language than Nix's expression language is used, an internal DSL can be used to make integration with the Nix expression language more convenient and safe.
Another problem that might surface when developing a custom solution is the scale at which builds are carried out. When it is desired to carry out builds on a large scale, additional concerns must be addressed beyond the solutions that the Nix package manager provides, such as managing a build queue, timing out builds that appear to be stuck, and sending notifications in case of a success or failure.
In such cases, it may be very tempting to address these concerns in your own custom build solution. However, some of these concerns are quite difficult to implement, such as build queue management.
There is already a Nix project that solves many of these problems, namely: Hydra, the Nix-based continuous integration service. One of Hydra's lesser known features is that it also exposes many of its operations through a REST API. Apart from a very small section in the Hydra manual, the API is not very well documented and -- as a result -- a bit cumbersome to use.
In this blog post, I will describe a recently developed Node.js package that exposes most of Hydra's functionality with an API that is documented and convenient to use. It can be used to conveniently integrate a Node.js application with Hydra's build management services. To demonstrate its usefulness, I have developed a simple command-line utility that can be used to remotely control a Hydra instance.
Finding relevant API calls
The Hydra API is not extensively documented. The manual has a small section titled: "Using the external API" that demonstrates some basic usage scenarios, such as querying data in JSON format.
Querying data is basically a nearly identical operation to opening the Hydra web front-end in a web browser. The only difference is that the GET request should include a header field stating that the output format should be displayed in JSON format.
For example, the following request fetches an overview of projects in JSON format (as opposed to HTML which is the default):
$ curl -H "Accept: application/json" https://hydra.nixos.org
In addition to querying data in JSON format, Hydra supports many additional REST operations, such as creating, updating or deleting projects and jobsets. Unfortunately, these operations are not documented anywhere in the manual.
By analyzing the Hydra source code and running the following script I could (sort of) semi-automatically derive the REST operations that are interesting to invoke:
$ cd hydra/src/lib/Hydra/Controller
$ find . -type f | while read i
do
echo "# $i"
grep -E '^sub [a-zA-Z0-9]+[ ]?\:' $i
echo
done
Running the script shows me the following (partial) output:
# ./Project.pm
sub projectChain :Chained('/') :PathPart('project') :CaptureArgs(1) {
sub project :Chained('projectChain') :PathPart('') :Args(0) :ActionClass('REST::ForBrowsers') { }
sub edit : Chained('projectChain') PathPart Args(0) {
sub create : Path('/create-project') {
...
The web front-end component of Hydra uses Catalyst, a Perl-based MVC framework. One of the conventions that Catalyst uses is that every request invokes an annotated Perl subroutine.
The above script scans the controller modules, and shows for each module annotated subroutines that may be of interest. In particular, the sub routines with a :ActionClass('REST::ForBrowsers') annotation are relevant -- they encapsulate create, read, update and delete operations for various kinds of data.
For example, the project subroutine shown above supports the following REST operations:
sub project_GET {
...
}
sub project_PUT {
...
}
sub project_DELETE {
...
}
The above operations will query the properties a project (GET), create a new or update an existing project (PUT) or delete a project (DELETE).
With the manual, the extracted information and reading the source code a bit (which is unavoidable since many details are missing such as formal function parameters), I was able to develop a client API supporting a substantial amount of Hydra features.
API usage
Using client the API is relatively straight forward. The general idea is to instantiate the HydraConnector prototype to connect to a Hydra server, such as a local test instance:
var HydraConnector = require('hydra-connector').HydraConnector;
var hydraConnector = new HydraConnector("http://localhost");
and then invoke any of its supported operations:
hydraConnector.queryProjects(function(err, projects) {
if(err) {
console.log("Some error occurred: "+err);
} else {
for(var i = 0; i < projects.length; i++) {
var project = projects[i];
console.log("Project: "+project.name);
}
}
});
The above code fragment fetches all projects and displays their names.
Write operations require a user to be logged in with the appropriate permissions. By invoking the login() method, we can authenticate ourselves:
hydraConnector.login("admin", "myverysecretpassword", function(err) {
if(err) {
console.log("Login succeeded!");
} else {
console.log("Some error occurred: "+err);
}
});
Besides logging in, the client API also implements a logout() operation to relinquish write operation rights.
As explained in an earlier blog post, write operations require user authentication but all data is publicly readable. When it is desired to restrict read access, Hydra can be placed behind a reverse proxy that requires HTTP basic authentication.
The client API also supports HTTP basic authentication to support this usage scenario:
var hydraConnector = new HydraConnector("http://localhost",
"sander",
"12345"); // HTTP basic credentials
Using the command-line client
To demonstrate the usefulness of the API, I have created a utility that serves as a command-line equivalent of Hydra's web front-end. The following command shows an overview of projects:
$ hydra-connect --url https://hydra.nixos.org --projects
As may be observed in the screenshot above, the command-line utility provides suggestions for additional command-line instructions that could be relevant, such as querying more detailed information or modifying data.
The following command shows the properties of an individual project:
$ hydra-connect --url https://hydra.nixos.org --project disnix
By default, the command-line utility will show a somewhat readable textual representation of the data. It is also possible to display the "raw" JSON data of any request for debugging purposes:
$ hydra-connect --url https://hydra.nixos.org --project disnix --json
We can also use the command-line utility to create or edit data, such as a project:
$ hydra-connect --url https://hydra.nixos.org --login $ export HYDRA_SESSION=... $ hydra-connect --url https://hydra.nixos.org --project disnix --modify
The application will show command prompts asking the user to provide all the relevant project properties.
When changes have been triggered, the active builds in the queue can be inspected by running:
$ hydra-connect --url https://hydra.nixos.org --status
Many other operations are supported. For example, you can inspect the properties of an individual build:
$ hydra-connect --url https://hydra.nixos.org --build 65100054
And request various kinds of build related artifacts, such as the raw build log of a build:
$ hydra-connect --url https://hydra.nixos.org --build 65100054 --raw-log
or download one of its build products:
$ hydra-connect --url https://hydra.nixos.org --build 65100054 \ --build-product 4 > /home/sander/Download/disnix-0.7.2.tar.gz
Conclusion
In this blog post, I have shown the node-hydra-connector package that can be used to remotely control a Hydra server from a Node.js application. The package can be obtained from the NPM registry and the corresponding GitHub repository.
No comments:
Post a Comment