[Nix-dev] Sharing the Nix store between systems

[Rickard Nilsson]

Hi,

I would like to put /nix/store in a central NFS share, and let all my
NixOS computers mount it over the network. This would save disk space and
make new computer deploys simple.

However there are some things that I don't know how to handle.

* How can I in NixOS specify which user that should do the builds? I would
like to use users and groups defined in a central LDAP catalog, and not
let NixOS create any local build users/groups.

* How would I stop Nix on one host from removing contents from the store
that is used by other hosts?

* Is there a way to also throw a build server into the picture, so that
NixOS computers can ask the build server to build store paths, which would
then be put on the NFS share directly by the build server?


Best regards,
Rickard Nilsson

[Marc Weber]

Post by Rickard Nilsson
* How can I in NixOS specify which user that should do the builds? I would
like to use users and groups defined in a central LDAP catalog, and not
let NixOS create any local build users/groups.

grep in nix source for nixbldX

Post by Rickard Nilsson
* How would I stop Nix on one host from removing contents from the store
that is used by other hosts?

patch nix or put references in /nix/var/nix/gcroots/..

Post by Rickard Nilsson
* Is there a way to also throw a build server into the picture, so that
NixOS computers can ask the build server to build store paths, which would
then be put on the NFS share directly by the build server?

If you have a client server setup there are remote builders. That's what
currently works. See manual and nix.buildMachines.
If you want to share the store so that you don't have to copy store
paths first before building I don't know about existing solutions yet -
but I'm interested in finding one as well.

Marc Weber

[Rickard Nilsson]

Hi,

I decided to solve the problem of store-sharing by
letting the nix-worker on localhost forward everything
to the nix-worker on another host. This way, I can
let a "client host" mount the store read-only, but
it is still possible to perform builds and installation,
since it is really the master host that does all
writing to the shared store. This is in my opinion
a very nice solution for thin disk-less clients.

One thing to note about this solution is that the
master of course don't add any gcroots for the new
builds, so the builds might be garbage collected
by the Nix master even though a client host uses
them. I intend to solve this by sharing some
subdirectory of /nix/var/nix/profiles on the master
to the hosts. That is orthogonal to the nix-worker
forwarding, though.

The forwarding was done like this:

jobs.nixRemoteDaemon = {
name = "nix-remote-daemon";
startOn = "startup";
script = ''
export
PATH=${pkgs.openssh}/bin:${pkgs.socat}/bin:${pkgs.coreutils}/bin
NIX_SOCKET=/nix/var/nix/daemon-socket/socket
NIX_REMOTE_SSHKEY=/etc/secrets/nix-remote-daemon.id_dsa
NIX_REMOTE_USER=nixremote
NIX_REMOTE_HOST=mynixmaster
mkdir -p $(dirname "$NIX_SOCKET")
rm -f "$NIX_SOCKET"
exec socat UNIX-LISTEN:$NIX_SOCKET,fork EXEC:"ssh -i
$NIX_REMOTE_SSHKEY $NIX_REMOTE_USER@$NIX_REMOTE_HOST socat -
UNIX-CONNECT\:$NIX_SOCKET"
'';
};

I simply define a new job that listens on the nix-worker
socket and forwards everything via ssh to the nix-worker
socket on the master. This works flawlessly for me.

I want to ask for recommendations on how to package this
and submit it to Nix/NixOS. I my opinion, this should really
be a feature of nix-worker itself, maybe something like this:

nix-worker --remote --remote-host ... --remote-user .. --remote-key ..

The simplest implementation of this is to just write a bash
wrapper around the existing nix-worker that uses socat and
ssh like above.

Is this a good idea, or should it rather be packaged as a separate
nix-remote-worker daemon?


Best regards,
Rickard Nilsson

Post by Marc Weber

Post by Rickard Nilsson
* How can I in NixOS specify which user that should do the builds? I would
like to use users and groups defined in a central LDAP catalog, and not
let NixOS create any local build users/groups.

grep in nix source for nixbldX

Post by Rickard Nilsson
* How would I stop Nix on one host from removing contents from the store
that is used by other hosts?

patch nix or put references in /nix/var/nix/gcroots/..

Post by Rickard Nilsson
* Is there a way to also throw a build server into the picture, so that
NixOS computers can ask the build server to build store paths, which would
then be put on the NFS share directly by the build server?

If you have a client server setup there are remote builders. That's what
currently works. See manual and nix.buildMachines.
If you want to share the store so that you don't have to copy store
paths first before building I don't know about existing solutions yet -
but I'm interested in finding one as well.
Marc Weber
_______________________________________________
nix-dev mailing list
http://lists.science.uu.nl/mailman/listinfo/nix-dev