[Nix-dev] In multi-user Nix, let the daemon handle creation of GC roots

[s***@catern.com]

My understanding is that currently GC roots (symlinks in
profiles/gcroots) are created and deleted directly by the various Nix
tools, even in multi-user configurations. (whether on NixOS or on
another Linux distribution)

It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.

The features that come to mind:
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently

The network-storage-/nix use case may be the most important, since there
seems to be a lot of people who want to put /nix on NFS.

Thoughts? Has this been considered?

Thanks for Nix!

[Wout Mertens]

Note, sharing /nix is already not really possible because the metadata is
stored in sqlite and its locking does not play nice with nfs. (*)
Another issue is that right now, nix does not /require/ the daemon to work,
and this proposal would change that.

However, you can totally share /nix between multiple hosts, you just have
to pinkie-promise not to write to it from multiple hosts at the same time.

Wout.

(*): the reason is that fnctl() locking is broken on many implementations.
If this testing project
https://sourceforge.net/projects/locktests/files/?source=navbar says it's
not broken, you can totally use nix on nfs.

Post by s***@catern.com
My understanding is that currently GC roots (symlinks in
profiles/gcroots) are created and deleted directly by the various Nix
tools, even in multi-user configurations. (whether on NixOS or on
another Linux distribution)
It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently
The network-storage-/nix use case may be the most important, since there
seems to be a lot of people who want to put /nix on NFS.
Thoughts? Has this been considered?
Thanks for Nix!
_______________________________________________
nix-dev mailing list
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Jörg Thalheim]

Note, sharing /nix is already not really possible because the metadata is stored in sqlite and its locking does not play nice with nfs. (*)
Another issue is that right now, nix does not /require/ the daemon to work, and this proposal would change that.
However, you can totally share /nix between multiple hosts, you just have to pinkie-promise not to write to it from multiple hosts at the same time.
Wout.

I used this in a single-user setup:

https://github.com/nixos-users/wiki/wiki/NFS#nix-store-on-nfs

I did not experienced problems so far

(*): the reason is that fnctl() locking is broken on many implementations. If this testing project https://sourceforge.net/projects/locktests/files/?source=navbar says it's not broken, you can totally use nix on nfs.
My understanding is that currently GC roots (symlinks in
profiles/gcroots) are created and deleted directly by the various Nix
tools, even in multi-user configurations. (whether on NixOS or on
another Linux distribution)
It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently
The network-storage-/nix use case may be the most important, since there
seems to be a lot of people who want to put /nix on NFS.
Thoughts? Has this been considered?
Thanks for Nix!
_______________________________________________
nix-dev mailing list
https://mailman.science.uu.nl/mailman/listinfo/nix-dev
_______________________________________________
nix-dev mailing list
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[s***@catern.com]

Post by Wout Mertens
Another issue is that right now, nix does not /require/ the daemon to
work, and this proposal would change that.

To clarify I'm just talking about this for multi-user setups, where the
daemon is already required.

Post by Wout Mertens
However, you can totally share /nix between multiple hosts, you just
have to pinkie-promise not to write to it from multiple hosts at the
same time.

Yes, this way (if the daemon also had the ability to talk to the
database) you don't need that pinkie promise.

But this is just one benefit of letting the daemon handle GC roots, I'm
interested also in the other benefits.

[Adrien Devresse]

Post by Wout Mertens
Note, sharing /nix is already not really possible because the metadata
is stored in sqlite and its locking does not play nice with nfs. (*)

Sharing is possible if you use a distributed file system that handle
consistency correctly, like GPFS, Lustre or similar.

We use Nix in shared model in production everyday in my organization.

Post by Wout Mertens
Another issue is that right now, nix does not /require/ the daemon to
work, and this proposal would change that.

It is not really an issue. It could be done the same way it is done
currently. The client does the GC management if configured in single
user mode, or does it through the daemon if configure in multi user mode.

The strong point here is that only ONE user should write to /nix :
- Yourself in single user mode
- The nix-daemon in multi user mode.

This is not the case currently.

Post by Wout Mertens
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently
The network-storage-/nix use case may be the most important, since there
seems to be a lot of people who want to put /nix on NFS.
Thoughts? Has this been considered?

I strongly support your idea.

The roots / profile implementation is currently hacky, not really
reliable, and potentially a security issue.


Regards,
Adev

Post by Wout Mertens
Note, sharing /nix is already not really possible because the metadata
is stored in sqlite and its locking does not play nice with nfs. (*)
Another issue is that right now, nix does not /require/ the daemon to
work, and this proposal would change that.
However, you can totally share /nix between multiple hosts, you just
have to pinkie-promise not to write to it from multiple hosts at the
same time.
Wout.
(*): the reason is that fnctl() locking is broken on many
implementations. If this testing
project https://sourceforge.net/projects/locktests/files/?source=navbar says
it's not broken, you can totally use nix on nfs.
My understanding is that currently GC roots (symlinks in
profiles/gcroots) are created and deleted directly by the various Nix
tools, even in multi-user configurations. (whether on NixOS or on
another Linux distribution)
It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently
The network-storage-/nix use case may be the most important, since there
seems to be a lot of people who want to put /nix on NFS.
Thoughts? Has this been considered?
Thanks for Nix!
_______________________________________________
nix-dev mailing list
https://mailman.science.uu.nl/mailman/listinfo/nix-dev
_______________________________________________
nix-dev mailing list
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Maarten Hoogendoorn]

There appear to be sufficient people in favor of this feature. Time for a
RFC?

Post by Wout Mertens
Note, sharing /nix is already not really possible because the metadata is
stored in sqlite and its locking does not play nice with nfs. (*)
Sharing is possible if you use a distributed file system that handle
consistency correctly, like GPFS, Lustre or similar.
We use Nix in shared model in production everyday in my organization.
Another issue is that right now, nix does not /require/ the daemon to
work, and this proposal would change that.
It is not really an issue. It could be done the same way it is done
currently. The client does the GC management if configured in single user
mode, or does it through the daemon if configure in multi user mode.
- Yourself in single user mode
- The nix-daemon in multi user mode.
This is not the case currently.
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently
The network-storage-/nix use case may be the most important, since there
seems to be a lot of people who want to put /nix on NFS.
Thoughts? Has this been considered?
I strongly support your idea.
The roots / profile implementation is currently hacky, not really
reliable, and potentially a security issue.
Regards,
Adev
Note, sharing /nix is already not really possible because the metadata is
stored in sqlite and its locking does not play nice with nfs. (*)
Another issue is that right now, nix does not /require/ the daemon to
work, and this proposal would change that.
However, you can totally share /nix between multiple hosts, you just have
to pinkie-promise not to write to it from multiple hosts at the same time.
Wout.
(*): the reason is that fnctl() locking is broken on many implementations.
If this testing project https://sourceforge.net/projects/locktests/files/?
source=navbar says it's not broken, you can totally use nix on nfs.

Post by s***@catern.com
My understanding is that currently GC roots (symlinks in
profiles/gcroots) are created and deleted directly by the various Nix
tools, even in multi-user configurations. (whether on NixOS or on
another Linux distribution)
It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently
The network-storage-/nix use case may be the most important, since there
seems to be a lot of people who want to put /nix on NFS.
Thoughts? Has this been considered?
Thanks for Nix!
_______________________________________________
nix-dev mailing list
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

_______________________________________________
_______________________________________________
nix-dev mailing list
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Kevin Cox]

Post by s***@catern.com
It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently

I'm not sure just this will solve your problem, however it might be a
good start. For example if the daemon only created indirect roots then
each root is accounted for (based on where it points) and can be cleaned
up by either the system or the user.

That being said I'm not sure this is necssary. You can already
inspect/delete roots by a given user to see how much space they are
"using". I'm curious what you would want to do to help with containers,
or to help with putting nix on network storage. I don't see how this is
required.

Basically I don't have any fanstic reason this shouldn't be done (except
simplicity) however I also don't really understand any of your arguments
for why it should be done. I think some sort of design-document for
solving one (or more) of these problems would be useful. This way there
is a concrete example of why the current situation wouldn't work and
provide a place for people to suggest concrete alternatives and debate
the merits.

[s***@catern.com]

Post by Kevin Cox

Post by s***@catern.com
It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.
- Allows later implementing policy about GC roots/space consumption
- Allows avoiding complicated locking around doing GC
- Allows /nix to be put on network storage transparently
- Allows /nix to be shared between containers transparently

I'm not sure just this will solve your problem, however it might be a
good start. For example if the daemon only created indirect roots then
each root is accounted for (based on where it points) and can be cleaned
up by either the system or the user.
That being said I'm not sure this is necssary. You can already
inspect/delete roots by a given user to see how much space they are
"using".

Yes, but that's a post-hoc check which can't really be made to work
reliably. What do you do if a user is using too much space? Delete their
stuff without any warning? That's unworkable, instead there should be an
error at the time that the user tries to consume excess
space. (Admittedly the policy around this will be tricky, but currently
we don't even have the mechanism to write such policy).

Post by Kevin Cox
I'm curious what you would want to do to help with containers,
or to help with putting nix on network storage. I don't see how this is
required.

It helps because it allows moving away from user-based control over GC
roots. Imagine two separate independent services running as the same
user in containers built with Nix. The two services have to run as the
same user because we don't have privileges on the system to create new
users. Those two services shouldn't be allowed to remove each others' GC
roots or add GC roots using each others' credentials. We should be able
to give the services conditional access to creating GC roots, without
full access to a per-user directory; we can do that if GC roots are
created through IPC.

A similar issue exists for multiple hosts sharing a store. Those hosts
shouldn't need to have the exact same set of users, just to share a
store. I should be able to publish my store so it is publicly accessible
to the world at large, and then allow people to connect in
(authenticating in some way) to the nix-daemon managing that store and
register new GC roots so they can maintain deployments out of that
global store. This is a lot easier than requiring the use of NFS with
POSIX user accounts which authenticate via Kerberos (or don't
authenticate at all, in traditional NFS).

The basic issue is tying control over GC roots to Unix user accounts.
Per-user software deployment is great and a big advance over root-only
software deployment, but Unix users have two fatal practical flaws: They
require privileges to create and they don't straightforwardly work
across the network. This hurts using a single store for multiple
containers on a single box, as well as using a single store across an
untrusting network.

And the more fundamental issue is that Unix users aren't compatible with
a capability-security model, which hopefully Nix can help push the world
towards. I would like to be able to give out a capability for creating
GC roots; that's not possible with the current user-based mechanism.

Hopefully that makes sense? I am fairly new to Nix, but I'm coming at
this from a capability-security mindset, and want to make sure Nix is as
compatible as possible with cap-security.

[Kevin Cox]

Post by s***@catern.com

Post by Kevin Cox
That being said I'm not sure this is necssary. You can already
inspect/delete roots by a given user to see how much space they are
"using".

Yes, but that's a post-hoc check which can't really be made to work
reliably. What do you do if a user is using too much space? Delete their
stuff without any warning? That's unworkable, instead there should be an
error at the time that the user tries to consume excess
space. (Admittedly the policy around this will be tricky, but currently
we don't even have the mechanism to write such policy).

So are you also going to stop the user from building new things? You
would also have the problem if the user was inside their quota then
started building something very large. I guess it wouldn't be bad to
start allowing this sort of enforcement but I'm not sure how useful it
would be in practice.

Post by s***@catern.com

Post by Kevin Cox
I'm curious what you would want to do to help with containers,
or to help with putting nix on network storage. I don't see how this is
required.

It helps because it allows moving away from user-based control over GC
roots. Imagine two separate independent services running as the same
user in containers built with Nix. The two services have to run as the
same user because we don't have privileges on the system to create new
users. Those two services shouldn't be allowed to remove each others' GC
roots or add GC roots using each others' credentials. We should be able
to give the services conditional access to creating GC roots, without
full access to a per-user directory; we can do that if GC roots are
created through IPC.

If you have two requests from the same user how do you segment them into
container or some other division? If you can restrict the containers in
some way couldn't you do this yourself by putting the links somewhere
the container can't access?

Post by s***@catern.com
A similar issue exists for multiple hosts sharing a store. Those hosts
shouldn't need to have the exact same set of users, just to share a
store. I should be able to publish my store so it is publicly accessible
to the world at large, and then allow people to connect in
(authenticating in some way) to the nix-daemon managing that store and
register new GC roots so they can maintain deployments out of that
global store. This is a lot easier than requiring the use of NFS with
POSIX user accounts which authenticate via Kerberos (or don't
authenticate at all, in traditional NFS).
The basic issue is tying control over GC roots to Unix user accounts.
Per-user software deployment is great and a big advance over root-only
software deployment, but Unix users have two fatal practical flaws: They
require privileges to create and they don't straightforwardly work
across the network. This hurts using a single store for multiple
containers on a single box, as well as using a single store across an
untrusting network.
And the more fundamental issue is that Unix users aren't compatible with
a capability-security model, which hopefully Nix can help push the world
towards. I would like to be able to give out a capability for creating
GC roots; that's not possible with the current user-based mechanism.
Hopefully that makes sense? I am fairly new to Nix, but I'm coming at
this from a capability-security mindset, and want to make sure Nix is as
compatible as possible with cap-security.

Same question as above, these sound like great goals but I don't see how
the Nix daemon is going to be able to divide the requests according to
capability.

[s***@catern.com]

Post by Kevin Cox
So are you also going to stop the user from building new things? You
would also have the problem if the user was inside their quota then
started building something very large.

Hmm, in that situation the build should fail, and that failure would be
propagated back up to the user. Though that could be difficult to
implement, I agree, because builds don't necessarily take place using
the user's resource quota.

Things can get better if/when we use an intensional store, where the
user can perform the build themselves rather than have the nix-daemon
perform it on their behalf.

Post by Kevin Cox
I guess it wouldn't be bad to start allowing this sort of enforcement
but I'm not sure how useful it would be in practice.

It's true, thinking it over more, I guess Unix traditionally doesn't
control creating new GC roots: You can hardlink anything you can read,
and dup() any file descriptor at all. So this kind of store quota policy
would be somewhat unnatural in Unix. I don't actually need/want it
myself, it was just an idea that I thought would be neat to support.
(The other uses of creating-store-roots-over-IPC still stand)

Post by Kevin Cox
If you have two requests from the same user how do you segment them into
container or some other division? If you can restrict the containers in
some way couldn't you do this yourself by putting the links somewhere
the container can't access?

You would pass in a proxy to the container. That is, you'd mount the
store into the container, as well as a socket which points to a
policy-implementing daemon. That policy-implementing daemon would
forward requests on to the real daemon.

Post by Kevin Cox
Same question as above, these sound like great goals but I don't see how
the Nix daemon is going to be able to divide the requests according to
capability.

Again, we could do this through proxying requests through a
policy-implementing daemon, which could identify requests based on
whatever authentication mechanism it wants.

To be clear I'm not suggesting we actually add a ton of policy-logic to
nix-daemon itself, sorry for not saying that at the start. But if we
added the ability for GC roots to be created through IPC requests, that
would allow others to put policy in between, through whatever means they
wish.

[Tuomas Tynkkynen]

Post by s***@catern.com
My understanding is that currently GC roots (symlinks in
profiles/gcroots) are created and deleted directly by the various Nix
tools, even in multi-user configurations. (whether on NixOS or on
another Linux distribution)
It seems to me that it would be useful for the daemon to handle making
GC roots, and forbid users to directly create GC roots.

How would deleting indirect roots work? Keep in mind that currently
users can simply delete any result symlinks from nix-build with 'rm' or
any other file management tool to remove the root. That won't work
over NFS because such an indirect root could be created on a different
filesystem (e.g. any local fs) that the machine running nix-daemon can't
access to verify whether the symlink is still there.

Additionally there's the problem of paths being in use by running processes
on other machines being garbage collected under their feet.

[s***@catern.com]

Post by Tuomas Tynkkynen
How would deleting indirect roots work? Keep in mind that currently
users can simply delete any result symlinks from nix-build with 'rm' or
any other file management tool to remove the root. That won't work
over NFS because such an indirect root could be created on a different
filesystem (e.g. any local fs) that the machine running nix-daemon can't
access to verify whether the symlink is still there.
Additionally there's the problem of paths being in use by running processes
on other machines being garbage collected under their feet.

Both of these could be handled client-side. Instead of having a per-user
roots directory in /nix/var/nix, we would have some truly per-user
directory (in /home/user/.var/nix/ or something) where roots are
put. We'd also have some kind of state tracking for what roots we've
told the nix-daemon about. Then when we one runs nix-store --gc, it
would:

1. Clean up broken links in .var/nix/roots
2. Search for runtime roots as the current user (this, incidentally, is
a critical feature on its own; my systems run with an unprivileged
nix-daemon and hidepid=2, so the nix-daemon (as it should be) cannot
determine what programs other users are running)
3. Tell the nix-daemon to remove any roots that were previously
registered and are no longer present, excluding roots that are currently
runtime roots.

Unfortunately, this would mean that if a user runs a program out of the
store, without first contacting the daemon and registering a (temporary)
root, the store paths for that program could be GC'd out from under
it. But I think that makes sense, anyway - the runtime root stuff
inherently requires a lot of privileges for the nix-daemon, which are
impractical/inadvisable to grant it in containerized/multi-host setups.
Much better to move runtime root detection to be as local as possible.

This would be an optional mode of operation, of course, global runtime
roots are still helpful if you can get them.

[s***@catern.com]

Two other observations about adding such a feature, where GC roots are
added and removed by sending messages to the daemon:

- Since the daemon would now know exactly when a GC root was removed,
this would allow for a time-based garbage collection policy, where a
GC only collects things if they have been unreferenced for at least an
hour, or a week, or any other period of time.

- I believe this would make our GC approach be an example of "partial
tracing" as described in Bacon 2004. "A Unified Theory of Garbage
Collection", https://www.cs.virginia.edu/~cs415/reading/bacon-garbage.pdf
which would be interestingly novel, since they say "We know of no
implementation of partial tracing."