Securing Hydra with a reverse proxy (Setting up a Hydra cluster part 3)

In two earlier blog posts, I have given a general description about Hydra and described how Hydra can be set up.

Another important aspect is to know that most of the facilities of Hydra are publicly accessible by anyone, except for the administration tasks, such as maintaining projects and jobsets for which it is required to have a user account.

A way to prevent Hydra from having public user access is to secure the reverse proxy that is in front of it. In this blog post, I will describe how to set up an HTTPS virtual domain with password authentication for this purpose. Moreover, the approach is not restricted to Hydra -- it can be used to secure any web application by means of a reverse proxy.

Generating a self-signed SSL certificate

To be able to set up HTTPS connections we need an SSL certificate. The easiest way to get one is to create a self-signed SSL certificate. The following command-line instruction suffices for me to generate a private key:

$ openssl genrsa -des3 -out hydra.example.com.key 1024

The above command asks you to provide a passphrase. Then we need to generate a certificate signing request (CSR) file to sign the certificate with our own identity:

$ openssl req -new -key hydra.example.com.key \
    -out hydra.example.com.csr

The above command asks you some details about your identity, such as the company name, state or province and country. After the CSR has been created, we can generate a certificate by running:

$ openssl x509 -req -days 365 -in hydra.example.com.csr \
    -signkey hydra.example.com.key -out hydra.example.com.orig.crt

To prevent the web server from asking me for the passphrase on every start, I ran the following to adapt the certificate:

$ openssl rsa -in hydra.example.com.orig.crt \
    -out hydra.example.com.crt

Now we have successfully generated a self-signed certificate allowing us to encrypt the remote connection to our Hydra instance.

Obviously, if you really care about security, it's better to obtain a cross-signed SSL certificate, since that provides you (some sort of) guarantee that a remote host can be trusted, whereas our self-signed SSL certificate forces users to create a security exception in their browsers.

Creating user accounts

A simple way to secure an Apache HTTP server with user authentication is by using the htpasswd facility. I did the following to create a user account for myself:

$ htpasswd -c /etc/nixos/htpasswd sander

The above command creates a user account named: sander, asks me for a password and stores the resulting htpasswd file in /etc/nixos. The above command can be repeated to add more user accounts.

Adapting the NixOS system configuration

As a final step, the reverse proxy must be reconfigured to accept HTTPS connections. The following fragment shows how the Apache HTTPD NixOS service can be configured to act as a reverse proxy having an unsecured HTTP virtual domain, and a secured HTTPS virtual domain requiring users to authenticate with a username and password:

services.httpd = {
  enable = true;
  adminAddr = "sander@example.com";
  hostName = "hydra.example.com";
  sslServerCert = "/etc/nixos/hydra.example.com.crt";
  sslServerKey = "/etc/nixos/hydra.example.com.key";
      
  virtualHosts = [
    { hostName = "localhost";
      extraConfig = ''
        <Proxy *>
          Order deny,allow
          Allow from all
        </Proxy>
            
        ProxyRequests     Off
        ProxyPreserveHost On
        ProxyPass         /    http://localhost:3000/ retry=5 disablereuse=on
        ProxyPassReverse  /    http://localhost:3000/
      '';
    }
        
    { hostName = "localhost";
      extraConfig = ''
        <Proxy *>
          Order deny,allow
          Allow from all
          AuthType basic
          AuthName "Hydra"
          AuthBasicProvider file
          AuthUserFile /etc/nixos/htpasswd
          Require user sander
        </Proxy>
            
        ProxyRequests     Off
        ProxyPreserveHost On
        ProxyPass         /    http://localhost:3000/ retry=5 disablereuse=on
        ProxyPassReverse  /    http://localhost:3000/
        RequestHeader set X-Forwarded-Proto https
        RequestHeader set X-Forwarded-Port 443
      '';
      enableSSL = true;
    }
  ];
};

There is one important aspect that I had to take into account in the configuration of the HTTPS virtual host. Without providing the RequestHeader properties, links in Hydra are not properly generated, as Catalyst (the MVC framework used by Hydra) does not know that links should use the https:// scheme instead of http://.

Restricting the unsecured HTTP virtual domain

It may also desired to prevent others from having access to the insecure HTTP virtual host. In a NixOS system configuration, you can set the firewall configuration to only accept HTTPS connections by adding the following lines:

networking.firewall.enabled = true;
networking.firewall.allowedTCPPorts = [ 443 ];

Conclusion

The tricks described in this blog post allowed me to secure Hydra with an HTTPS connection and password authentication, so that I can open a web browser and use: https://hydra.example.com to access the secured virtual host.

The major disadvantage of this approach is that you cannot use Nix's channel mechanism to automatically obtain substitutes from Hydra. Perhaps, in the future Nix can be adapted to also support connections with user authentication.