[Nix-dev] Installing /nix on nfs-mounted volume possible?

[Ben Booth]

Hi all,

I would like to use nix to install software and run in on a variety of linux machines at my work. I tried installing nix to my NFS-mounted home folder as outlined here[1], but I had trouble with both proot[2] and nix-user-chroot[3], since they both mess up some permissions and break sudo. I should also mention that I really don't want to build packages from source if I can avoid it.

The next thing I tried was installing nix to an automounted NFS volume mounted at /nix. This seems to work great so far, but now my nix store is visible to all users. I don't really mind this, if other users want to run my software that's fine. But I've read that for multi-user installations, I need to designate a build user and do some other steps for building software from source. I haven't tried that yet since I've just been downloading packages from the binary cache. Is this something I need to worry about?

I've also heard that installing nix on NFS-mounted volumes can cause corruption of the nix store because of locking issues. Is this still an issue for modern NFSv4 implementations? Our NFS server is running Centos 7 with a 3.10.0 kernel. Is there some other way to accomplish what I'm trying to do that doesn't have these issues?


One last question: I installed python into my nix store and would like to install some PyPI packages using pip. How can I make sure that, for compiled python packages that reference native libraries, that they only look in the nix store for native libraries, and not in /usr/lib, etc? I saw the `nix-shell` command, but it looks like I would have to add a `-p package` argument for each package I want in the environment. Is there any way to tell `nix-shell` that I want all my nix packages in the environment? I guess I could create a shell script to automate adding a `-p` argument for each package in my nix store, but is there an easier way?

Thanks!
Ben Booth

[1] https://nixos.org/wiki/How_to_install_nix_in_home_\(on_another_distribution\)
[2] http://proot.me/
[3] https://github.com/lethalman/nix-user-chroot

[Wout Mertens]

The problem with nfs nix stores is that the metadata is stored in sqlite,
which can't guarantee proper locking on nfs. However, if you only write
from one host, there is no problem.

As for the python stuff, you can run it in a proot, or use nixos which
doesn't have /usr :-)

Post by Ben Booth
Hi all,
I would like to use nix to install software and run in on a variety of
linux machines at my work. I tried installing nix to my NFS-mounted home
folder as outlined here[1], but I had trouble with both proot[2] and
nix-user-chroot[3], since they both mess up some permissions and break
sudo. I should also mention that I really don't want to build packages from
source if I can avoid it.
The next thing I tried was installing nix to an automounted NFS volume
mounted at /nix. This seems to work great so far, but now my nix store is
visible to all users. I don't really mind this, if other users want to run
my software that's fine. But I've read that for multi-user installations, I
need to designate a build user and do some other steps for building
software from source. I haven't tried that yet since I've just been
downloading packages from the binary cache. Is this something I need to
worry about?
I've also heard that installing nix on NFS-mounted volumes can cause
corruption of the nix store because of locking issues. Is this still an
issue for modern NFSv4 implementations? Our NFS server is running Centos 7
with a 3.10.0 kernel. Is there some other way to accomplish what I'm trying
to do that doesn't have these issues?
One last question: I installed python into my nix store and would like to
install some PyPI packages using pip. How can I make sure that, for
compiled python packages that reference native libraries, that they only
look in the nix store for native libraries, and not in /usr/lib, etc? I saw
the `nix-shell` command, but it looks like I would have to add a `-p
package` argument for each package I want in the environment. Is there any
way to tell `nix-shell` that I want all my nix packages in the environment?
I guess I could create a shell script to automate adding a `-p` argument
for each package in my nix store, but is there an easier way?
Thanks!
Ben Booth
[1]
https://nixos.org/wiki/How_to_install_nix_in_home_\(on_another_distribution\)
[2] http://proot.me/
[3] https://github.com/lethalman/nix-user-chroot
_______________________________________________
nix-dev mailing list
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Ben Booth]

The problem with nfs nix stores is that the metadata is stored in sqlite, which can't guarantee proper locking on nfs. However, if you only write from one host, there is no problem.
As for the python stuff, you can run it in a proot, or use nixos which doesn't have /usr :-)

Thanks for your help! I've thought of a new approach: create a NixOS VM, and then export its /nix store as a read-only NFS mount. That way, if a user wants to install software, they can just ssh into the NixOS VM, install it there in an isolated environment, with no NFS locking issues, then that software can be used on any other VM instance. Can you think of any reason that would not work?

Thanks!
Ben

[deCube.net | Danny Wilson]

I’ve used a shared nix store between VMs in the past, until the sqlite database got corrupted…

If I’d do it again, I would configure a single build VM ( nix.buildMachines ) and set nix.maxJobs to 0. That guarantees all builds are executed on this one machine. Any nix store copying after the build will then be a noop, since the store is completely shared :-)


What I did to get a bootable /nix store, was this:

https://gist.github.com/vizanto/7568026

Note the per-hostname profile directory: /mnt-root/nix/vm-specific/`hostname`/var/nix/profiles
This is done to have intra-vm garbage collection working, since all profiles are known.

Post by Ben Booth

The problem with nfs nix stores is that the metadata is stored in sqlite, which can't guarantee proper locking on nfs. However, if you only write from one host, there is no problem.
As for the python stuff, you can run it in a proot, or use nixos which doesn't have /usr :-)

Thanks for your help! I've thought of a new approach: create a NixOS VM, and then export its /nix store as a read-only NFS mount. That way, if a user wants to install software, they can just ssh into the NixOS VM, install it there in an isolated environment, with no NFS locking issues, then that software can be used on any other VM instance. Can you think of any reason that would not work?
Thanks!
Ben
_______________________________________________
nix-dev mailing list
http://lists.science.uu.nl/mailman/listinfo/nix-dev

[Eelco Dolstra]

Hi,

The problem with nfs nix stores is that the metadata is stored in sqlite, which
can't guarantee proper locking on nfs. However, if you only write from one host,
there is no problem.

IIRC, SQLite *should* work if you disable WAL mode (by adding "use-sqlite-wal =
false" to nix.conf). But I haven't tested that.

Another possibility, also untested, would be to mount /nix/store via NFS, and
then forward the Nix daemon socket over the network in some way. E.g., you could
use some socat magic to connect /nix/var/nix/daemon-socket/socket on the clients
to the same file on the server. We do something like that for NixOS containers:
the containers have a read-only /nix/store bind mount, but they can still do Nix
operations because the host's /nix/var/nix/daemon-socket/socket is bind-mounted
into the container.