Discussion:
GSoC ideas
(too old to reply)
Pjotr Prins
2016-02-06 11:38:02 UTC
Permalink
I have posted a GNU Guix project idea to SciRuby Foundation:

https://groups.google.com/forum/?fromgroups#!topic/sciruby-dev/CZt75MuUxhg

Basically a student can work on

- Add JRuby support to Guix
- Add rbx support to Guix
- Add all testing frameworks to Guix (e.g. cucumber)
- Add all web development frameworks to Guix (e.g. RoR)
- Add Sciruby and related modules to Guix
- Provide support to Travis-CI (probably through Docker)
- Provide general Docker support
- Integrate with IRuby notebook and nyaplot

Guix has support for containers built-in, so we can even live without
Docker.

If you are interested, sign up with the SciRuby mailing list (sorry
they use Google groups).

Also I encourage other packagers to post project ideas to relevant
organisations (R, SciPy anyone?).

Pj.
Ludovic Courtès
2016-02-06 12:36:04 UTC
Permalink
Post by Pjotr Prins
https://groups.google.com/forum/?fromgroups#!topic/sciruby-dev/CZt75MuUxhg
Nice! (Well, except for Google Groups. ;-))

For the record, GNU is applying as an organization. Guix will be
participating under the aegis of GNU as in previous years, so interested
parties are welcome to start brainstorming!

Ludo’.
Christopher Allan Webber
2016-02-06 18:54:34 UTC
Permalink
Maybe we should have a place to collect ideas?

Anyway, here would be my top items for GSoC:

- An installer wizard (written in Guile of course!)
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
- Perhaps some kind of non-emacs, non-cli package (and system
management?) interface. Maybe GTK based?
Efraim Flashner
2016-02-07 10:16:37 UTC
Permalink
On Sat, 06 Feb 2016 10:54:34 -0800
Post by Christopher Allan Webber
Maybe we should have a place to collect ideas?
- An installer wizard (written in Guile of course!)
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
- Perhaps some kind of non-emacs, non-cli package (and system
management?) interface. Maybe GTK based?
In the interest of full disclosure, I should note that in all likelyhood I
will still qualify as a student for this years GSoC.

The three ideas that I had so far that are floating around in a text file are:

- arm64 port -- This could probably be extended into getting a working system
into place to have a u-boot -> grub loader, and/or using kexec to get a
booted system to reboot into an arm libre-kernel. (for armv7 and arm64?)

- ncurses installer -- my idea was Debian's installer can be considered ugly,
but it does have everything needed to install a system.

- GaPMaCt - Guix's amazing package manager and configure tool -- Same
non-emacs package & system manager as above. Maybe a backend with a GTK or
ncurses frontend.
--
Efraim Flashner <***@flashner.co.il> א׀ךים ׀לשנך
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
Ludovic Courtès
2016-02-09 20:52:31 UTC
Permalink
Post by Efraim Flashner
- arm64 port -- This could probably be extended into getting a working system
into place to have a u-boot -> grub loader, and/or using kexec to get a
booted system to reboot into an arm libre-kernel. (for armv7 and arm64?)
For aarch64 the first task would be to port Guix (not GuixSD) to that
architecture (info "(guix) Bootstrapping"). If everything goes well, it
can be relatively simple, but sometimes things don’t go well. ;-)

Otherwise it’s surely an eligible project IMO.
Post by Efraim Flashner
- ncurses installer -- my idea was Debian's installer can be considered ugly,
but it does have everything needed to install a system.
Right, that would be nice.
Post by Efraim Flashner
- GaPMaCt - Guix's amazing package manager and configure tool -- Same
non-emacs package & system manager as above. Maybe a backend with a GTK or
ncurses frontend.
+1!

Ludo’.
myglc2
2016-03-25 20:32:45 UTC
Permalink
Efraim Flashner <***@flashner.co.il> writes:
[...]
Post by Efraim Flashner
- ncurses installer -- my idea was Debian's installer can be considered ugly,
but it does have everything needed to install a system.
Couldn't agree more. Don't know if you are still considering this idea,
but reading your comment and a couple more recent GSoC GuixSD installer
proposals prompted me to write some general thoughts about Guix
installation which you might find interesting:

http://article.gmane.org/gmane.comp.gnu.guix.devel/18206

Best - George
Tomáš Čech
2016-02-08 10:45:30 UTC
Permalink
Post by Christopher Allan Webber
Maybe we should have a place to collect ideas?
- An installer wizard (written in Guile of course!)
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
What would be purpose of that? Do you plan to use some other store path so it
won't collide with Guix maintained installation on the same machine?

OTOH it may be interesting to generate only (in my case) RPM metadata
so native system's package manager can track Guix generated files and
use pre/post install scripts for interactions with Guix (to
install/remove/protect them). That would glue Guix to system even
better. Just wild idea when thinking about your idea...

S_W
Efraim Flashner
2016-02-08 11:37:31 UTC
Permalink
On Mon, 8 Feb 2016 11:45:30 +0100
Post by Tomáš Čech
Post by Christopher Allan Webber
Maybe we should have a place to collect ideas?
- An installer wizard (written in Guile of course!)
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
What would be purpose of that? Do you plan to use some other store path so it
won't collide with Guix maintained installation on the same machine?
OTOH it may be interesting to generate only (in my case) RPM metadata
so native system's package manager can track Guix generated files and
use pre/post install scripts for interactions with Guix (to
install/remove/protect them). That would glue Guix to system even
better. Just wild idea when thinking about your idea...
S_W
like checkinstall? It would be nice if a foreign distro to have graphical
programs just work, integrated into the system like their own native programs.
--
Efraim Flashner <***@flashner.co.il> א׀ךים ׀לשנך
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
Christopher Allan Webber
2016-02-08 17:24:54 UTC
Permalink
Post by Tomáš Čech
Post by Christopher Allan Webber
Maybe we should have a place to collect ideas?
- An installer wizard (written in Guile of course!)
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
What would be purpose of that? Do you plan to use some other store path so it
won't collide with Guix maintained installation on the same machine?
OTOH it may be interesting to generate only (in my case) RPM metadata
so native system's package manager can track Guix generated files and
use pre/post install scripts for interactions with Guix (to
install/remove/protect them). That would glue Guix to system even
better. Just wild idea when thinking about your idea...
S_W
Sorry, I was unclear. What I'm talking about is Guix *as a package
manager* to be able to be available on foreigh distros so I can
encourage friends to do:

apt-get install guix

... and then do user-space hacking with Guix, and use "guix environment"
for hacking on my projects... I think we'd increase our userbase
dramatically by getting Guix as a user-space pacakge manager into
various distros.

The gexp part was because of the gexp that (iirc) generates the "tarball
install" method of things as inspiration... could we build something
similar that builds a .deb or .rpm of Guix as an install method?

- Chris
Thompson, David
2016-02-08 18:45:28 UTC
Permalink
On Mon, Feb 8, 2016 at 12:24 PM, Christopher Allan Webber
Post by Christopher Allan Webber
What I'm talking about is Guix *as a package
manager* to be able to be available on foreigh distros so I can
apt-get install guix
... and then do user-space hacking with Guix, and use "guix environment"
for hacking on my projects... I think we'd increase our userbase
dramatically by getting Guix as a user-space pacakge manager into
various distros.
Such packages would be convenient for users of these distros, but
getting them into Debian and Fedora proper is unlikely because Guix
does not conform to the FHS.

- Dave
Christopher Allan Webber
2016-02-08 19:47:31 UTC
Permalink
Post by Thompson, David
On Mon, Feb 8, 2016 at 12:24 PM, Christopher Allan Webber
Post by Christopher Allan Webber
What I'm talking about is Guix *as a package
manager* to be able to be available on foreigh distros so I can
apt-get install guix
... and then do user-space hacking with Guix, and use "guix environment"
for hacking on my projects... I think we'd increase our userbase
dramatically by getting Guix as a user-space pacakge manager into
various distros.
Such packages would be convenient for users of these distros, but
getting them into Debian and Fedora proper is unlikely because Guix
does not conform to the FHS.
- Dave
So, I talked about this with Stefano Zacchiroli while at FOSDEM. He
suggested that we could do /opt/g/ or something like that and we *could*
get Guix in as a package manager to Debian. Sure, it wouldn't be able
to use the Hydra packages, but it's probably okayish.

(We *might* be able to get Guix in even using the Hydra packages if the
Guix daemon could have a wrapper which bind-mounts it to /gnu/ or
something, when it starts up? That might be a bit over the top, but
maybe would allow us to have both. I'm not so sure about this strategy
though, I've just been bouncing it around in my head.)

We could also build apt / rpm packages which we just provide ourselves,
on the Guix website, or by providing some PPA'ish thing. I would have
preferred this myself when I started running Debian on Guix... and in
this route we could keep everything as it already is. (But people will
have to take an extra step to "get to" those nice packages.)

- Chris
Pjotr Prins
2016-02-08 20:43:50 UTC
Permalink
Post by Christopher Allan Webber
So, I talked about this with Stefano Zacchiroli while at FOSDEM. He
suggested that we could do /opt/g/ or something like that and we *could*
get Guix in as a package manager to Debian. Sure, it wouldn't be able
to use the Hydra packages, but it's probably okayish.
Unless users get binaries they won't be happy.
Post by Christopher Allan Webber
We could also build apt / rpm packages which we just provide ourselves,
on the Guix website, or by providing some PPA'ish thing. I would have
preferred this myself when I started running Debian on Guix... and in
this route we could keep everything as it already is. (But people will
have to take an extra step to "get to" those nice packages.)
That looks like the way forward to me. Such a package can setup and
start the daemon - which is enough. No need to get the blessing from
the distributions themselves (will take time, but it will come - there
really is no difference with allowing foreign packages to work anyway).

Pj.
Ludovic Courtès
2016-02-09 10:36:24 UTC
Permalink
Post by Pjotr Prins
Post by Christopher Allan Webber
So, I talked about this with Stefano Zacchiroli while at FOSDEM. He
suggested that we could do /opt/g/ or something like that and we *could*
get Guix in as a package manager to Debian. Sure, it wouldn't be able
to use the Hydra packages, but it's probably okayish.
Unless users get binaries they won't be happy.
+1
Post by Pjotr Prins
Post by Christopher Allan Webber
We could also build apt / rpm packages which we just provide ourselves,
on the Guix website, or by providing some PPA'ish thing. I would have
preferred this myself when I started running Debian on Guix... and in
this route we could keep everything as it already is. (But people will
have to take an extra step to "get to" those nice packages.)
That looks like the way forward to me. Such a package can setup and
start the daemon - which is enough. No need to get the blessing from
the distributions themselves (will take time, but it will come - there
really is no difference with allowing foreign packages to work anyway).
Yeah, I think you’re right.

There are already people who wrote Arch/Parabola and RPM (openSuSE)
packages, AFAIK. I’m not sure what the status is?

If someone familiar with dpkg could chime in, we could host the
infrastructure to build the .deb somewhere.

Ludo’.
Tomáš Čech
2016-02-10 08:53:01 UTC
Permalink
Post by Ludovic Courtès
There are already people who wrote Arch/Parabola and RPM (openSuSE)
packages, AFAIK. I’m not sure what the status is?
For openSUSE ecosystem I maintain package in system:packagemanagers
project of our Open Build Service. It means it doesn't live in the
main package repository but it is easy to install.

https://software.opensuse.org/download/package?project=system:packagemanager&package=guix

I'm trying to push that to the main distribution but for that the
package may need some more work because I need to get rid of foreign
binaries (the ones in
$PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
guix' for all openSUSE users is still my goal.

Any help on that from Guix side would be appreciated.

Besides this, thanks to our the OBS I should be able to setup build
for other RPM distributions (Fedora, CentOS) but it wasn't my focus
now.

Best regards,

Tomas
Ricardo Wurmus
2016-02-10 09:42:55 UTC
Permalink
Post by Tomáš Čech
Post by Ludovic Courtès
There are already people who wrote Arch/Parabola and RPM (openSuSE)
packages, AFAIK. I’m not sure what the status is?
For openSUSE ecosystem I maintain package in system:packagemanagers
project of our Open Build Service. It means it doesn't live in the
main package repository but it is easy to install.
https://software.opensuse.org/download/package?project=system:packagemanager&package=guix
I'm trying to push that to the main distribution but for that the
package may need some more work because I need to get rid of foreign
binaries (the ones in
$PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
guix' for all openSUSE users is still my goal.
The bootstrap binaries form the roots of the dependency graph.
Modifying them results in different hashes for all packages. I learned
this when a colleague accidentally modified these files in our
deployment, which made Guix refuse all of the substitutes from Hydra and
instead rebuilt the world locally.

I don’t think you can realistically get rid of them.

~~ Ricardo
Jookia
2016-02-10 12:44:52 UTC
Permalink
Post by Tomáš Čech
I'm trying to push that to the main distribution but for that the
package may need some more work because I need to get rid of foreign
binaries (the ones in
$PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
guix' for all openSUSE users is still my goal.
Any help on that from Guix side would be appreciated.
Besides this, thanks to our the OBS I should be able to setup build
for other RPM distributions (Fedora, CentOS) but it wasn't my focus
now.
Best regards,
Tomas
I brought this up a bit earlier in IRC but you could rebuild the bootstrap
binaries- but then they'd just be identical to the ones shipped. So are they
really foreign then?

Jookia.
Ricardo Wurmus
2016-02-11 05:44:38 UTC
Permalink
Post by Jookia
Post by Tomáš Čech
I'm trying to push that to the main distribution but for that the
package may need some more work because I need to get rid of foreign
binaries (the ones in
$PREFIX/share/guile/site/2.0/gnu/packages/bootstrap/). So `zypper in
guix' for all openSUSE users is still my goal.
Any help on that from Guix side would be appreciated.
Besides this, thanks to our the OBS I should be able to setup build
for other RPM distributions (Fedora, CentOS) but it wasn't my focus
now.
Best regards,
Tomas
I brought this up a bit earlier in IRC but you could rebuild the bootstrap
binaries- but then they'd just be identical to the ones shipped. So are they
really foreign then?
I wonder: can you *really* rebuild them? Would that not require
reproducible builds, something that we try to achieve with Guix? These
tarballs come from their upstream release servers, if I’m not mistaken.

Since the bootstrapping step starts with statically linked binaries of
tar (and some other tools) and the tarballs that are then unpacked with
that binary of tar, I don’t see how these binaries could possibly
be reproduced *exactly* on a foreign distro and seamlessly replace the
bundled binaries.

It would be amazing and quite a surprise if you *were* able to rebuild
them and thus reproduce them bit for bit.

~~

I wished we could reduce the number of bootstrap binaries down to little
more than just Guile and an old C compiler, i.e. implement a subset of
tar in Guile and use that to unpack the tarballs, then use the simple C
compiler to compile an old version of GCC (without C++), then use that
version of GCC to compile a GCC capable of building C++, then use that
GCC to build a modern GCC...

Since Guile is mostly written in Guile (and it is probably somewhat
easier to come to a point to trust it) there may be value in trying to
bootstrap from even less.

~~

Practically speaking, though, you probably cannot get around the
bootstrap binaries — not unless there was a way to arbitrarily
manipulate hashes or sneakily “seed” hashes at a certain level into the
graph (e.g. tell Guix that the binaries of GCC that are in the system
always have a particular hash even if they do not).

I don’t think a system like this would be desirable in general as it
undermines reproducibility and trust. (I do see that something like
this might be useful for a shoddy port to MacOS, but ... ugh! What’s
the point in that!)

~~ Rambling Ricardo
Ludovic Courtès
2016-02-11 10:54:31 UTC
Permalink
[...]
Post by Ricardo Wurmus
Post by Jookia
I brought this up a bit earlier in IRC but you could rebuild the bootstrap
binaries- but then they'd just be identical to the ones shipped. So are they
really foreign then?
I wonder: can you *really* rebuild them?
First, you’d need to build them as explained at
<http://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries>,
but starting from the exact same commit that was used to build them.
Post by Ricardo Wurmus
Would that not require reproducible builds, something that we try to
achieve with Guix?
Indeed, some of the packages in the bootstrap binaries are reproducible,
some are not. See
<https://lists.gnu.org/archive/html/guix-devel/2013-09/msg00159.html>
for a previous experiment in that area.
Post by Ricardo Wurmus
I wished we could reduce the number of bootstrap binaries down to little
more than just Guile and an old C compiler, i.e. implement a subset of
tar in Guile and use that to unpack the tarballs, then use the simple C
compiler to compile an old version of GCC (without C++), then use that
version of GCC to compile a GCC capable of building C++, then use that
GCC to build a modern GCC...
Since Guile is mostly written in Guile (and it is probably somewhat
easier to come to a point to trust it) there may be value in trying to
bootstrap from even less.
Definitely! Just like we have cpio, FTP and HTTP clients, etc. in
Scheme, we could incrementally have tiny replacements for some of the
things currently found in the bootstrap binaries. They don’t need to be
full-featured.

This is one of the topics discussed as the reproducible summit:

https://lists.gnu.org/archive/html/guix-devel/2015-12/msg00107.html
https://reproducible-builds.org/events/athens2015/bootstrapping/

Ludo’.
Diane Trout
2016-02-23 23:00:22 UTC
Permalink
Post by Pjotr Prins
That looks like the way forward to me. Such a package can setup and
start the daemon - which is enough. No need to get the blessing from
the distributions themselves (will take time, but it will come - there
really is no difference with allowing foreign packages to work
anyway).
I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.

https://github.com/detrout/debian-guix

Currently I've only split the guix package into the emacs components
and everything else. I'd thought about splitting the daemon out into
its own package, but I wasn't sure what the daemon depended on. 

The daemon is still using the default /gnu/store path, and the user
needs to manually run guix authorize if they want to use hydra
binaries. The package is currently based on the stable 0.9.0 release,
and I'm not sure how security updates make it into a guix store if you
without updating the scheme packaging source tree.

It might be nice to prompt the user if they wanted to authorize hydra
on install but that's not implemented.

Currently its unlikely to go into Debian because Debian policy requires
everything to be built from source, and currently the Guix build
process downloads some bootstrap binaries.

However with the current packaging "guix environment --pure bash --
bash" does give me a clean guix environment, and the guix info docs get
installed when Debian emacs can see them.

Diane
Christopher Allan Webber
2016-02-23 23:52:30 UTC
Permalink
Post by Diane Trout
Post by Pjotr Prins
That looks like the way forward to me. Such a package can setup and
start the daemon - which is enough. No need to get the blessing from
the distributions themselves (will take time, but it will come - there
really is no difference with allowing foreign packages to work anyway).
I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.
https://github.com/detrout/debian-guix
Currently I've only split the guix package into the emacs components
and everything else. I'd thought about splitting the daemon out into
its own package, but I wasn't sure what the daemon depended on.
The daemon is still using the default /gnu/store path, and the user
needs to manually run guix authorize if they want to use hydra
binaries. The package is currently based on the stable 0.9.0 release,
and I'm not sure how security updates make it into a guix store if you
without updating the scheme packaging source tree.
It might be nice to prompt the user if they wanted to authorize hydra
on install but that's not implemented.
Currently its unlikely to go into Debian because Debian policy requires
everything to be built from source, and currently the Guix build
process downloads some bootstrap binaries.
However with the current packaging "guix environment --pure bash --
bash" does give me a clean guix environment, and the guix info docs get
installed when Debian emacs can see them.
Diane
Great work Diane!

Are those bootstrapping binaries really necessary for getting Guix
going? I guess for some reason I thought if you did the whole
configure/make/etc dance it wouldn't be but maybe I'm wrong.

Maybe this is a good step towards getting a Guix .deb we self-host on
the Guix website?

- Chris
Leo Famulari
2016-02-24 00:02:34 UTC
Permalink
Post by Christopher Allan Webber
Post by Diane Trout
Post by Pjotr Prins
That looks like the way forward to me. Such a package can setup and
start the daemon - which is enough. No need to get the blessing from
the distributions themselves (will take time, but it will come - there
really is no difference with allowing foreign packages to work anyway).
I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.
https://github.com/detrout/debian-guix
Currently I've only split the guix package into the emacs components
and everything else. I'd thought about splitting the daemon out into
its own package, but I wasn't sure what the daemon depended on.
The daemon is still using the default /gnu/store path, and the user
needs to manually run guix authorize if they want to use hydra
binaries. The package is currently based on the stable 0.9.0 release,
and I'm not sure how security updates make it into a guix store if you
without updating the scheme packaging source tree.
It might be nice to prompt the user if they wanted to authorize hydra
on install but that's not implemented.
Currently its unlikely to go into Debian because Debian policy requires
everything to be built from source, and currently the Guix build
process downloads some bootstrap binaries.
However with the current packaging "guix environment --pure bash --
bash" does give me a clean guix environment, and the guix info docs get
installed when Debian emacs can see them.
Diane
Great work Diane!
Are those bootstrapping binaries really necessary for getting Guix
going? I guess for some reason I thought if you did the whole
configure/make/etc dance it wouldn't be but maybe I'm wrong.
My understanding is that if you alter the bootstrap binaries, the entire
dependency graph will change, forcing a rebuild of everything. And of
course, the altered binaries may present different interfaces, breaking
things as well.
Ricardo Wurmus
2016-02-24 09:03:34 UTC
Permalink
Post by Leo Famulari
Post by Christopher Allan Webber
Are those bootstrapping binaries really necessary for getting Guix
going? I guess for some reason I thought if you did the whole
configure/make/etc dance it wouldn't be but maybe I'm wrong.
My understanding is that if you alter the bootstrap binaries, the entire
dependency graph will change, forcing a rebuild of everything. And of
course, the altered binaries may present different interfaces, breaking
things as well.
This is correct. Back then we ran into trouble with our Guix
installation at work when someone modified permission bits on the
bootstrap binaries, causing a rebuild of everything. It took us a while
to find out the cause and revert the change.

Some of the bootstrap binaries can be reproduced from source (if you
make sure to follow the Guix recipes), but others (like Guile IIRC)
don’t have reproducible build systems, so reproducing the exact same
binaries without using Guix is going to be very challenging.

I don’t know if it is possible and if it would make sense to cheat,
i.e. just lie to Guix about the hashes of the bootstrap binaries.

~~ Ricardo
Efraim Flashner
2016-02-24 09:16:51 UTC
Permalink
On Wed, 24 Feb 2016 10:03:34 +0100
Post by Ricardo Wurmus
[...]
[...]
This is correct. Back then we ran into trouble with our Guix
installation at work when someone modified permission bits on the
bootstrap binaries, causing a rebuild of everything. It took us a while
to find out the cause and revert the change.
Some of the bootstrap binaries can be reproduced from source (if you
make sure to follow the Guix recipes), but others (like Guile IIRC)
don’t have reproducible build systems, so reproducing the exact same
binaries without using Guix is going to be very challenging.
I don’t know if it is possible and if it would make sense to cheat,
i.e. just lie to Guix about the hashes of the bootstrap binaries.
~~ Ricardo
What about taking it a step further and having a multi-level bootstrap
process like when we have the core-updates? If we bootstrap away enough times
would we end up with the bootstrap binaries we have now?
--
Efraim Flashner <***@flashner.co.il> א׀ךים ׀לשנך
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
Jookia
2016-02-24 09:36:58 UTC
Permalink
Post by Efraim Flashner
What about taking it a step further and having a multi-level bootstrap
process like when we have the core-updates? If we bootstrap away enough times
would we end up with the bootstrap binaries we have now?
From what I understand the bootstrap binaries aren't reproducible yet. If they
were, it would solve this issue as we could build them again.
Ludovic Courtès
2016-02-25 18:15:13 UTC
Permalink
Post by Jookia
Post by Efraim Flashner
What about taking it a step further and having a multi-level bootstrap
process like when we have the core-updates? If we bootstrap away enough times
would we end up with the bootstrap binaries we have now?
From what I understand the bootstrap binaries aren't reproducible yet.
Depends on what kind of reproducibility we’re talking about. It’s
simple to build bootstrap binaries:

https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries

I think they are all bit-for-bit reproducible (that is, you can use
--rounds=3 and everything is fine), except for Guile due to
<http://bugs.gnu.org/20272>.)

However, if you build them today, you’ll obviously get something
different from the bootstrap binaries we currently use, which were from
Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.

HTH,
Ludo’.
Christopher Allan Webber
2016-02-25 20:26:29 UTC
Permalink
Post by Ludovic Courtès
Post by Jookia
Post by Efraim Flashner
What about taking it a step further and having a multi-level bootstrap
process like when we have the core-updates? If we bootstrap away enough times
would we end up with the bootstrap binaries we have now?
From what I understand the bootstrap binaries aren't reproducible yet.
Depends on what kind of reproducibility we’re talking about. It’s
https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
I think they are all bit-for-bit reproducible (that is, you can use
--rounds=3 and everything is fine), except for Guile due to
<http://bugs.gnu.org/20272>.)
However, if you build them today, you’ll obviously get something
different from the bootstrap binaries we currently use, which were from
Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
HTH,
Ludo’.
It seems like a good idea, once that bug in Guile is fixed, to move over
to a new set of bootstrap binaries... even if this involves some
difficulty for Guix users today. It would certainly be a good thing to
do before we hit 1.0, whenever that is.
Ludovic Courtès
2016-02-26 23:19:04 UTC
Permalink
Post by Christopher Allan Webber
Post by Ludovic Courtès
Post by Jookia
Post by Efraim Flashner
What about taking it a step further and having a multi-level bootstrap
process like when we have the core-updates? If we bootstrap away enough times
would we end up with the bootstrap binaries we have now?
From what I understand the bootstrap binaries aren't reproducible yet.
Depends on what kind of reproducibility we’re talking about. It’s
https://www.gnu.org/software/guix/manual/html_node/Bootstrapping.html#Building-the-Bootstrap-Binaries
I think they are all bit-for-bit reproducible (that is, you can use
--rounds=3 and everything is fine), except for Guile due to
<http://bugs.gnu.org/20272>.)
However, if you build them today, you’ll obviously get something
different from the bootstrap binaries we currently use, which were from
Guile 2.0.9, libc 2.19, GCC 4.7.2, some old Coreutils, etc.
HTH,
Ludo’.
It seems like a good idea, once that bug in Guile is fixed, to move over
to a new set of bootstrap binaries... even if this involves some
difficulty for Guix users today. It would certainly be a good thing to
do before we hit 1.0, whenever that is.
I prefer to change those binaries as rarely as possible. Intuitively
(and unscientifically), it gives more confidence to keep using the same
old binaries wrt. Ken Thompson attacks.

Ludo’.
Jookia
2016-02-28 10:51:35 UTC
Permalink
Post by Ludovic Courtès
I prefer to change those binaries as rarely as possible. Intuitively
(and unscientifically), it gives more confidence to keep using the same
old binaries wrt. Ken Thompson attacks.
I'm not sure about that, if we could establish the binaries could be
reproducibly built using the current bootstrap binaries it sounds like it could
be fine. Having reproducible bootstrap binaries seems like something incredibly
useful especially for packagers that for whatever reason want to verify that the
binaries can be built with Guix before signing them.
Post by Ludovic Courtès
Ludo’.
Jookia.
Ludovic Courtès
2016-02-28 15:08:00 UTC
Permalink
Post by Jookia
Post by Ludovic Courtès
I prefer to change those binaries as rarely as possible. Intuitively
(and unscientifically), it gives more confidence to keep using the same
old binaries wrt. Ken Thompson attacks.
I'm not sure about that, if we could establish the binaries could be
reproducibly built using the current bootstrap binaries it sounds like it could
be fine. Having reproducible bootstrap binaries seems like something incredibly
useful especially for packagers that for whatever reason want to verify that the
binaries can be built with Guix before signing them.
We would have to update them every time we change GCC, Guile, Coreutils,
etc. or one of their dependencies, which sounds impractical or even
infeasible to me.

Ludo’.
Jookia
2016-02-28 15:10:29 UTC
Permalink
Post by Ludovic Courtès
We would have to update them every time we change GCC, Guile, Coreutils,
etc. or one of their dependencies, which sounds impractical or even
infeasible to me.
Perhaps there's some miscommunication here- the problem right now is that the
binaries aren't reproducible at all. Having them switched to reproducible builds
once and then tagging the Guix commit used would probably be enough to fix the
problem. I don't think we need to rebuild them every time.
Post by Ludovic Courtès
Ludo’.
Jookia.
Christopher Allan Webber
2016-02-29 05:22:23 UTC
Permalink
Post by Jookia
Post by Ludovic Courtès
We would have to update them every time we change GCC, Guile, Coreutils,
etc. or one of their dependencies, which sounds impractical or even
infeasible to me.
Perhaps there's some miscommunication here- the problem right now is that the
binaries aren't reproducible at all. Having them switched to reproducible builds
once and then tagging the Guix commit used would probably be enough to fix the
problem. I don't think we need to rebuild them every time.
Yes, this is what I was originally suggesting!
Ludovic Courtès
2016-02-29 10:01:44 UTC
Permalink
Post by Jookia
Post by Ludovic Courtès
We would have to update them every time we change GCC, Guile, Coreutils,
etc. or one of their dependencies, which sounds impractical or even
infeasible to me.
Perhaps there's some miscommunication here- the problem right now is that the
binaries aren't reproducible at all. Having them switched to reproducible builds
once and then tagging the Guix commit used would probably be enough to fix the
problem. I don't think we need to rebuild them every time.
Ah sure, I agree! Though to be on the safe side, we’d need to do
diverse double compilation (DDC) of these binaries, à la Wheeler.

Ludo’.
Diane Trout
2016-02-24 00:32:30 UTC
Permalink
Post by Christopher Allan Webber
Great work Diane!
Are those bootstrapping binaries really necessary for getting Guix
going?  I guess for some reason I thought if you did the whole
configure/make/etc dance it wouldn't be but maybe I'm wrong.
Maybe this is a good step towards getting a Guix .deb we self-host on
the Guix website?
The bootstrap binaries are used to compute the hashes for Guix's
initial packages which then influence all of the downstream hashes.

I know that shipping sourceless binaries is against Debian policy. I do
wonder if there's a way to produce the same bootstrap binaries from
Debian components.

Also one of your other messages suggested the /gnu directory is also
against Debian policy.

If there's a way resolve those things, there's probably a much better
chance to get it into Debian.

But other than that I think my postrm script is incomplete and
certainly needs some more testing but this can certainly turn into
.debs hosted by guix.

Diane
Pjotr Prins
2016-02-24 07:04:09 UTC
Permalink
Post by Diane Trout
Also one of your other messages suggested the /gnu directory is also
against Debian policy.
If there's a way resolve those things, there's probably a much better
chance to get it into Debian.
Yes, this in old discussion (also in Nix). For sure, if guix built in
/usr/local and $HOME it would be much easier to get by the policy
folks (also in HPC environments).

There is, however, a good reason to make it a blatant /gnu outside the
FHS. The store contains everything that normally goes into the full
FHS. It lives outside the FHS. If we make it part of the FHS (even in
one subdirectory) it would confuse things badly. I think the clarity
matters to system administrators AND users.

Also, for true reproducibility, we need one path. Arguably it could be
a different one, but to make sure a package is exactly the same in its
binary form it needs to be running from a root point.

It took me quite some time to come round to this idea. But now I am
really convinced it is the only way forward. Administrators can use
chroot and containers/VM to mount /gnu - so, there is no real
technical concern.

Guix is not tied to one server. If they think it important, the Debian
people can, at some point, decide to take Guix, create their own
caching server, and distribute /usr/local/guix. They have done far
more intrusive things ;)

For us acceptance in Debian is not a prime concern either. An
installable .deb package is good enough to help regular users.
Therefore I don't think you need to fix the binary bootstrapping as
long as we can make it part of the automated build farm.
Post by Diane Trout
But other than that I think my postrm script is incomplete and
certainly needs some more testing but this can certainly turn into
.debs hosted by guix.
Diane
Thanks for the great work! A .deb package would be very useful for
people to start using Guix.

Pj.
--
Thompson, David
2016-02-24 17:20:44 UTC
Permalink
Post by Pjotr Prins
For us acceptance in Debian is not a prime concern either. An
installable .deb package is good enough to help regular users.
Therefore I don't think you need to fix the binary bootstrapping as
long as we can make it part of the automated build farm.
+1

Getting a Guix package into Debian is really not important, especially
given the amount of nasty hacks it would require to attempt to appease
the Debian developers.

Hosting a .deb file on our own server that users could download and
install with dpkg would be perfect for us.

Thanks for the hard work, Diane!

- Dave
Diane Trout
2016-02-24 19:36:52 UTC
Permalink
Post by Thompson, David
Hosting a .deb file on our own server that users could download and
install with dpkg would be perfect for us.
Actually the best thing to do would be to put the debs into a signed
repository, for example https://mirrorer.alioth.debian.org/ is a utility that
lets you create your own apt-gettable repositories.

The guix on debian instructions would then be:

apt-key add (signing key)
add "deb http://<server>/debian guix main" to
/etc/apt/sources.list.d/guix.list
apt-get update
apt-get install guix

this allows apt-get update to get new versions.

The nix hydra paper implies that it can build packages and installers for non-
nix systems. I assume the guix version can as well?
Post by Thompson, David
Thanks for the hard work, Diane!
You're welcome, I'm glad this may be useful to others. :)

Diane
Efraim Flashner
2016-02-25 10:30:47 UTC
Permalink
Post by Diane Trout
Post by Thompson, David
Hosting a .deb file on our own server that users could download and
install with dpkg would be perfect for us.
Actually the best thing to do would be to put the debs into a signed
repository, for example https://mirrorer.alioth.debian.org/ is a utility that
lets you create your own apt-gettable repositories.
apt-key add (signing key)
add "deb http://<server>/debian guix main" to
/etc/apt/sources.list.d/guix.list
apt-get update
apt-get install guix
this allows apt-get update to get new versions.
Another option would be to include in the pre/post install script to add
the repo if it's not already there. I know some third party repos delete
and recreate their files, my `tor+http` setting keeps on getting
overwritten.
Post by Diane Trout
The nix hydra paper implies that it can build packages and installers for non-
nix systems. I assume the guix version can as well?
Post by Thompson, David
Thanks for the hard work, Diane!
You're welcome, I'm glad this may be useful to others. :)
Diane
I haven't looked too closely at the code yet, so appologies if you've
already taken care of it. In the guix manual it says not to install the
binary installer over an existing install, does the deb you've created
make sure to not do that?
--
Efraim Flashner <***@flashner.co.il> א׀ךים ׀לשנך
GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted
Diane Trout
2016-02-26 01:27:03 UTC
Permalink
Post by Efraim Flashner
Another option would be to include in the pre/post install script to add
the repo if it's not already there. I know some third party repos delete
and recreate their files, my `tor+http` setting keeps on getting
overwritten.
I know some packages automatically install repository keys, I don't really
like it.
Post by Efraim Flashner
I haven't looked too closely at the code yet, so appologies if you've
already taken care of it. In the guix manual it says not to install the
binary installer over an existing install, does the deb you've created
make sure to not do that?
No. Other than the basic checks from dpkg where it will complain if it tries
to overwrite a file.

This package is somewhat inferior to the binary installation instructions as
it currently stands the guix scheme libraries go into
/usr/share/guile/site/2.0/guix.

I discovered this causes trouble by trying to use a guix git tree I'd
compiled. That didn't work right until after I'd rebuilt it with:
$ guix environment guix
$ ./bootstrap ; ./configure --prefix=/usr --localstatedir=/var

I'm wondering if the debian build script should build guix, then and then do

$ guix package --no-substitutes -i guix

And install what ended up in /gnu/store

However I haven't checked to see if there's a way to start the guix-daemon in
temporary location. The Debian build utilities assume it can cause things to
be built in a subdirectory of the source tree by manipulating environment
variables.

Diane
Jan Nieuwenhuizen
2016-02-24 19:14:21 UTC
Permalink
Post by Diane Trout
I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.
https://github.com/detrout/debian-guix
Thank you, this is /really/ useful. I want to introduce Guix at work
and was thinking about a debian package.

Greetings, Jan
--
Jan Nieuwenhuizen <***@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.nl
Jan Nieuwenhuizen
2016-02-24 21:24:27 UTC
Permalink
Post by Jan Nieuwenhuizen
Post by Diane Trout
I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.
https://github.com/detrout/debian-guix
Thank you, this is /really/ useful. I want to introduce Guix at work
and was thinking about a debian package.
For Ubuntu, I used the patch below.

Greetings,
Jan

diff --git a/README.rst b/README.rst
index 7092389..0e6b228 100644
--- a/README.rst
+++ b/README.rst
@@ -22,7 +22,7 @@ To build you need to do something like the following:
cd guix
git clone https://github.com/detrout/debian-guix.git debian
uscan --download-current-version
- tar xavf ../guix-$(dpkg-parsechangelog -S Version | cut -f 1 -d -).orig.tar.gz --strip-components=1
+ tar xavf ../guix_$(dpkg-parsechangelog -S Version | cut -f 1 -d -).orig.tar.gz --strip-components=1
sudo apt-get install build-essential dh-autoreconf dh-systemd autotools-dev graphviz guile-2.0-dev guile-json help2man libgcrypt20-dev libsqlite3-dev libbz2-dev texinfo
dpkg-buildpackage
sudo dpkg -i ../guix_0.9.0-1_amd64.deb ../emacs-guix_0.9.0-1_all.deb
diff --git a/watch b/watch
index 5e7fecb..972498e 100644
--- a/watch
+++ b/watch
@@ -1,3 +1,2 @@
-version=4
-opts="pgpsigurlmangle=s/$/.sig/" \
+version=3
http://alpha.gnu.org/gnu/guix/ guix-([\d\.]+)\.tar\.gz debian uupdate
--
Jan Nieuwenhuizen <***@gnu.org> | GNU LilyPond http://lilypond.org
Freelance IT http://JoyofSource.com | Avatar® http://AvatarAcademy.nl
Diane Trout
2016-02-24 21:56:57 UTC
Permalink
Post by Jan Nieuwenhuizen
For Ubuntu, I used the patch below.
Thank you.

I fixed the typo and switched to using a uscan version=3 config file with the
Ludovic Courtès signing key in upstream/signing-key.asc so the uscan key
validation works correctly.

Does the updated uscan file work correctly on *buntu?

I also committed a change to the postrm script as I learned of a few more
directories that should be removed on --purge.

Diane
Ludovic Courtès
2016-02-25 18:10:39 UTC
Permalink
Post by Diane Trout
I wrote a basic Debian recipe to build guix, create the build users,
and install the systemd config file.
https://github.com/detrout/debian-guix
Nice!
Post by Diane Trout
Currently I've only split the guix package into the emacs components
and everything else. I'd thought about splitting the daemon out into
its own package, but I wasn't sure what the daemon depended on. 
What you describe sounds good enough.
Post by Diane Trout
Currently its unlikely to go into Debian because Debian policy requires
everything to be built from source, and currently the Guix build
process downloads some bootstrap binaries.
Without a fully automated process to build .deb for several distros, I
don’t think we can offer to distribute .deb ourselves. :-/

If you’re versed in Guix, you might be interested in writing a
derivation that runs Debian in QEMU and invokes the commands that build
.deb files, as Chris suggested. That would be awesome.

Ludo’.
Diane Trout
2016-02-26 01:28:13 UTC
Permalink
Post by Ludovic Courtès
If you’re versed in Guix, you might be interested in writing a
derivation that runs Debian in QEMU and invokes the commands that build
.deb files, as Chris suggested. That would be awesome.
I don't currently know enough about guix to do that, but I think it'd be great
if I did.

Diane
myglc2
2016-03-25 20:52:44 UTC
Permalink
Post by Christopher Allan Webber
Sorry, I was unclear. What I'm talking about is Guix *as a package
manager* to be able to be available on foreigh distros so I can
apt-get install guix
... and then do user-space hacking with Guix, and use "guix environment"
for hacking on my projects... I think we'd increase our userbase
dramatically by getting Guix as a user-space pacakge manager into
various distros.
I strongly agree with this. To my mind this is the key selling point of
Guix and we need to make it easy to experience this.

But you know, given how well the Guix Binary install + a lame makefile
has worked to install/uninstall on my Debian servers, and the way Guix
kind of includes it's own world, I'm not so sure a .deb package
necessary.
Post by Christopher Allan Webber
The gexp part was because of the gexp that (iirc) generates the "tarball
install" method of things as inspiration... could we build something
similar that builds a .deb or .rpm of Guix as an install method?
- Chris
Maybe a downloadable install package that does a binary install and is
routinely verified to run on 3 or 4 popular distros is just as good and
has more band for the buck.

Anyway, reading your comments and a couple other recent GSoC GuixSD
installer proposals prompted me to write some general thoughts about
Guix installation which you might find interesting:

http://lists.gnu.org/archive/html/guix-devel/2016-03/msg01024.html

Best - George

Ludovic Courtès
2016-02-09 20:49:30 UTC
Permalink
Post by Christopher Allan Webber
Maybe we should have a place to collect ideas?
What about creating a page at <https://libreplanet.org/wiki/Group:Guix>?
Post by Christopher Allan Webber
- An installer wizard (written in Guile of course!)
For GuixSD I guess? Sounds like good idea, of course.
Post by Christopher Allan Webber
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
That would be useful, and CheckInstall may be good enough for that. I
don’t think it’s “enough” for GSoC, though. WDYT?
Post by Christopher Allan Webber
- Perhaps some kind of non-emacs, non-cli package (and system
management?) interface. Maybe GTK based?
Definitely. Both a guile-curses-based and a GTK+-based thing would be
nice. For the latter we need to check whether PackageKit can be of any
use (this has been discussed in the past and my impression was that it
is biased towards “traditional” distros with just a single root-managed
profile.)

Thanks,
Ludo’.
Christopher Allan Webber
2016-02-09 22:17:08 UTC
Permalink
Post by Ludovic Courtès
Post by Christopher Allan Webber
Maybe we should have a place to collect ideas?
What about creating a page at <https://libreplanet.org/wiki/Group:Guix>?
https://libreplanet.org/wiki/Group:Guix/GSoC-2016

Here you go!
Post by Ludovic Courtès
Post by Christopher Allan Webber
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
That would be useful, and CheckInstall may be good enough for that. I
don’t think it’s “enough” for GSoC, though. WDYT?
If the student was really to package Guix for many distributions, it
might be enough. Especially if that student was to explore packaging
with upstream. Anyway, the payout could be *huge*.

If they finish early, I'm sure we could find other things for them to
do? :)

- Chris
Pjotr Prins
2016-02-10 06:12:54 UTC
Permalink
Post by Christopher Allan Webber
Post by Ludovic Courtès
Post by Christopher Allan Webber
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
That would be useful, and CheckInstall may be good enough for that. I
don’t think it’s “enough” for GSoC, though. WDYT?
If the student was really to package Guix for many distributions, it
might be enough. Especially if that student was to explore packaging
with upstream. Anyway, the payout could be *huge*.
One thing to keep in mind is that GSoC is about programming. So, we
have to be clear about that in the proposal.
Post by Christopher Allan Webber
If they finish early, I'm sure we could find other things for them to
do? :)
Better to put a secondary job in - such as improving the web-ui or
creating a tool for importing deb packages, or something. Students
will come if they think the job challenging/interesting enough.

Pj.
Christopher Allan Webber
2016-02-11 02:11:33 UTC
Permalink
Post by Pjotr Prins
Post by Christopher Allan Webber
Post by Ludovic Courtès
Post by Christopher Allan Webber
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
That would be useful, and CheckInstall may be good enough for that. I
don’t think it’s “enough” for GSoC, though. WDYT?
If the student was really to package Guix for many distributions, it
might be enough. Especially if that student was to explore packaging
with upstream. Anyway, the payout could be *huge*.
One thing to keep in mind is that GSoC is about programming. So, we
have to be clear about that in the proposal.
Post by Christopher Allan Webber
If they finish early, I'm sure we could find other things for them to
do? :)
Better to put a secondary job in - such as improving the web-ui or
creating a tool for importing deb packages, or something. Students
will come if they think the job challenging/interesting enough.
Pj.
So, while we're on the topic of hopes and dreams that I might
*personally* love to see solved through GSoC, but maybe people will be
like "That's not enough of a GSoC programming project"....

npm packaging and importer. Even better: enough npm packaging and
importing to where we package pump.io :)

It might not be the most fun GSoC project though?
Ludovic Courtès
2016-02-11 10:01:00 UTC
Permalink
Post by Christopher Allan Webber
Post by Pjotr Prins
Post by Christopher Allan Webber
Post by Ludovic Courtès
Post by Christopher Allan Webber
- g-expressions which generate packages of the Guix package manager for
.deb and .rpm based distros, and work to get those upstream in Debian
and Fedora
That would be useful, and CheckInstall may be good enough for that. I
don’t think it’s “enough” for GSoC, though. WDYT?
If the student was really to package Guix for many distributions, it
might be enough. Especially if that student was to explore packaging
with upstream. Anyway, the payout could be *huge*.
One thing to keep in mind is that GSoC is about programming. So, we
have to be clear about that in the proposal.
Post by Christopher Allan Webber
If they finish early, I'm sure we could find other things for them to
do? :)
Better to put a secondary job in - such as improving the web-ui or
creating a tool for importing deb packages, or something. Students
will come if they think the job challenging/interesting enough.
Pj.
So, while we're on the topic of hopes and dreams that I might
*personally* love to see solved through GSoC, but maybe people will be
like "That's not enough of a GSoC programming project"....
We could frame it in terms of programming: providing a Guix API for the
creation of Debian etc. packages.

There is (used to be?) something in Nixpkgs that produces a derivation
to build stuff in a Debian VM, build the .deb with CheckInstall, and
return that .deb as the derivation’s output. Something like this is
definitely programming, and possibly fun as well. :-)

Could you maybe add more details for this one on the wiki page?
Post by Christopher Allan Webber
npm packaging and importer. Even better: enough npm packaging and
importing to where we package pump.io :)
Woow, why not! Point the candidate to your blog post on that topic.
;-)

Ludo’.
Christopher Allan Webber
2016-02-11 17:59:05 UTC
Permalink
Post by Ludovic Courtès
Post by Christopher Allan Webber
So, while we're on the topic of hopes and dreams that I might
*personally* love to see solved through GSoC, but maybe people will be
like "That's not enough of a GSoC programming project"....
We could frame it in terms of programming: providing a Guix API for the
creation of Debian etc. packages.
There is (used to be?) something in Nixpkgs that produces a derivation
to build stuff in a Debian VM, build the .deb with CheckInstall, and
return that .deb as the derivation’s output. Something like this is
definitely programming, and possibly fun as well. :-)
Could you maybe add more details for this one on the wiki page?
Post by Christopher Allan Webber
npm packaging and importer. Even better: enough npm packaging and
importing to where we package pump.io :)
Woow, why not! Point the candidate to your blog post on that topic.
;-)
Ludo’.
Okay, I updated the page:

https://libreplanet.org/wiki/Group:Guix/GSoC-2016
Ludovic Courtès
2016-02-21 21:07:27 UTC
Permalink
[...]
Post by Christopher Allan Webber
Post by Ludovic Courtès
Post by Christopher Allan Webber
npm packaging and importer. Even better: enough npm packaging and
importing to where we package pump.io :)
Woow, why not! Point the candidate to your blog post on that topic.
;-)
Ludo’.
https://libreplanet.org/wiki/Group:Guix/GSoC-2016
Cool, thanks!

Now is really the time to add new ideas or expound items already on this
page!

There are a couple of things regarding the Shepherd that could be done,
notably
<http://www.gnu.org/software/soc-projects/ideas-2015.html#dmd_units>.

It could be useful to continue on the GNUnet idea at
<http://www.gnu.org/software/soc-projects/ideas-2015.html#guix>, or to
look at alternative approaches or P2P stacks.

Someone wants add new items to the wiki page starting from that? :-)

Thanks,
Ludo’.
myglc2
2016-03-23 19:07:50 UTC
Permalink
Post by Ludovic Courtès
Post by Christopher Allan Webber
Maybe we should have a place to collect ideas?
What about creating a page at <https://libreplanet.org/wiki/Group:Guix>?
Post by Christopher Allan Webber
- An installer wizard (written in Guile of course!)
For GuixSD I guess? Sounds like good idea, of course.
Possibly out of scope, but, I have found that there are many ways to
install Guix. It would help to simplify this. what if this installer
could do something like this:

Examine the environment & determine the possible ways that Guix and/or
GuixSD can be installed and then walk the user through choices:

- netboot?
- fdisk & GuixSD?
- GNU/Linux system?
- ?root user?
- install Guix?
- put GuixSD netboot on USB?
- install GuixSD to disk?
- install GuixSD dual boot?
- running in user space?
- guix installed?
- set up git clone?
- guix not present?
- can you sudo?
- yes - got to '?root user' above
- no
- install Guix in user space?
- put GuixSD netboot on USB?

At the end of the process, the user-lever user should have a functioning
Guix setup in which all guix features described in the manual are
working, (except those requiring root, e.g. guix system reconfigure).
Manolis Ragkousis
2016-02-11 11:46:30 UTC
Permalink
Hello everyone,

As I am still a student I would like to suggest continuing the project
of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
having a working GuixSD/Hurd by the end of the summer.

After discussions with Justus and Samuel in FOSDEM about the Guix-Hurd
port I am starting to have a good idea on what is left to be done and
the solution needed for the problems we have.

So here's a preliminary timeline as well as a list of things to be done:

Before May:

1) Merge wip-hurd branch.
2) Make the daemon handle chroot builds on the Hurd.
Note here that on the Hurd, one does not need to be root to achieve
isolation, so I should change the daemon to use this capability.
3) Instead of using the Linux syscall guile wrappers, I should modify
Guix to use a more Hurdish way (i.e settrans) so later on we can handle
translators and bootstrap a GNU/Hurd system.
4) Better understand how GuixSD startup works.

May-August:
1) Implement the mechanisms for creating and mounting the initial
filesystem and starting the system.
2) Implement the mechanisms to handle the Hurd servers in /hurd. I
remember I was told that there may be an issue when the servers are not
actually there (i.e all the binaries are in the /gnu/store). Would love
if somebody could tell us more on that.
3) Isolate Linuxisms around Guix.
4) David Michael has made an excelent work on getting Shepherd working
with Hurd so I don't think we will have any serious issues with the init
daemon.
5) Others that will come up.. In the following weeks I will work closely
with the Hurd guys to make sure we get a better understanding of
everything and a detailed timeline.

End of GSoC:
Have a working GuixSD with Hurd as the kernel.

Currently Justus (and others soon :-)) are testing Guix on their
machines. We have some issues but we are working on them.

Please feel free to add/correct anything from the above. Your
comment/opinion may help to get things up and running faster :-)

Finally I want to say that I will continue my work on this, regardless
of GSoC or not, but I would like it if it was. :-)

Thank you,
Manolis
Andreas Enge
2016-02-11 17:32:15 UTC
Permalink
Post by Manolis Ragkousis
As I am still a student I would like to suggest continuing the project
of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
having a working GuixSD/Hurd by the end of the summer.
Sounds great, I hope it will go through!

Andreas
Alex Sassmannshausen
2016-02-12 11:48:49 UTC
Permalink
Post by Andreas Enge
Post by Manolis Ragkousis
As I am still a student I would like to suggest continuing the project
of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
having a working GuixSD/Hurd by the end of the summer.
Sounds great, I hope it will go through!
Andreas
+1 :-)
Christopher Allan Webber
2016-02-11 17:36:36 UTC
Permalink
Post by Manolis Ragkousis
Hello everyone,
As I am still a student I would like to suggest continuing the project
of porting Guix to GNU/Hurd in this GSoC as well, with the objective of
having a working GuixSD/Hurd by the end of the summer.
After discussions with Justus and Samuel in FOSDEM about the Guix-Hurd
port I am starting to have a good idea on what is left to be done and
the solution needed for the problems we have.
1) Merge wip-hurd branch.
2) Make the daemon handle chroot builds on the Hurd.
Note here that on the Hurd, one does not need to be root to achieve
isolation, so I should change the daemon to use this capability.
3) Instead of using the Linux syscall guile wrappers, I should modify
Guix to use a more Hurdish way (i.e settrans) so later on we can handle
translators and bootstrap a GNU/Hurd system.
4) Better understand how GuixSD startup works.
1) Implement the mechanisms for creating and mounting the initial
filesystem and starting the system.
2) Implement the mechanisms to handle the Hurd servers in /hurd. I
remember I was told that there may be an issue when the servers are not
actually there (i.e all the binaries are in the /gnu/store). Would love
if somebody could tell us more on that.
3) Isolate Linuxisms around Guix.
4) David Michael has made an excelent work on getting Shepherd working
with Hurd so I don't think we will have any serious issues with the init
daemon.
5) Others that will come up.. In the following weeks I will work closely
with the Hurd guys to make sure we get a better understanding of
everything and a detailed timeline.
Have a working GuixSD with Hurd as the kernel.
Currently Justus (and others soon :-)) are testing Guix on their
machines. We have some issues but we are working on them.
Please feel free to add/correct anything from the above. Your
comment/opinion may help to get things up and running faster :-)
Finally I want to say that I will continue my work on this, regardless
of GSoC or not, but I would like it if it was. :-)
Thank you,
Manolis
I can't speak for everyone else but it feels like this would be a great
project for you to contininue. The guile-emacs project ran for a few
years, so I don't see any blocker on continuing, right?
Samuel Thibault
2016-02-28 22:12:25 UTC
Permalink
Hello,

Just sending my big +1 on this project :)

Samuel
Richard Braun
2016-02-28 23:33:57 UTC
Permalink
Post by Samuel Thibault
Just sending my big +1 on this project :)
Same here.

I'd like to note that I'm personally pushing so that we have less
students but with better projects. This one definitely fits.
--
Richard Braun
Ludovic Courtès
2016-03-02 10:09:28 UTC
Permalink
Hi!

I realize I hadn’t responded to Manolis. Another +1 from me!
Post by Manolis Ragkousis
1) Merge wip-hurd branch.
2) Make the daemon handle chroot builds on the Hurd.
Note here that on the Hurd, one does not need to be root to achieve
isolation, so I should change the daemon to use this capability.
I think an ideal situation would be if libc provides ‘mount’ and
‘umount’, with MS_BIND implemented in terms of firmlinks.

I remember Roland was not thrilled with the idea of adding ‘mount’ and
‘umount’, but it would clearly help porting. I don’t know if this task
should be part of the GSoC project though, and you may need guidance
from the Hurd folks. What do people think?
Post by Manolis Ragkousis
3) Instead of using the Linux syscall guile wrappers, I should modify
Guix to use a more Hurdish way (i.e settrans) so later on we can handle
translators and bootstrap a GNU/Hurd system.
Possibly, depending on whether libc provides ‘mount’ and ‘umount’, which
would make things simpler.
Post by Manolis Ragkousis
Have a working GuixSD with Hurd as the kernel.
\o/

Cheers,
Ludo’.
Samuel Thibault
2016-03-02 10:12:27 UTC
Permalink
Post by Ludovic Courtès
Post by Manolis Ragkousis
1) Merge wip-hurd branch.
2) Make the daemon handle chroot builds on the Hurd.
Note here that on the Hurd, one does not need to be root to achieve
isolation, so I should change the daemon to use this capability.
I think an ideal situation would be if libc provides ‘mount’ and
‘umount’, with MS_BIND implemented in terms of firmlinks.
I remember Roland was not thrilled with the idea of adding ‘mount’ and
‘umount’, but it would clearly help porting.
It can both help and hurt. Roland's concern is that we will probably not
want to provide a mount() that behaves like Linux, because in some cases
it may just not even make sense. So the "ported" software that would try
to use it would just fail.

Samuel
Ludovic Courtès
2016-03-02 21:31:41 UTC
Permalink
Post by Samuel Thibault
Post by Ludovic Courtès
Post by Manolis Ragkousis
1) Merge wip-hurd branch.
2) Make the daemon handle chroot builds on the Hurd.
Note here that on the Hurd, one does not need to be root to achieve
isolation, so I should change the daemon to use this capability.
I think an ideal situation would be if libc provides ‘mount’ and
‘umount’, with MS_BIND implemented in terms of firmlinks.
I remember Roland was not thrilled with the idea of adding ‘mount’ and
‘umount’, but it would clearly help porting.
It can both help and hurt. Roland's concern is that we will probably not
want to provide a mount() that behaves like Linux, because in some cases
it may just not even make sense. So the "ported" software that would try
to use it would just fail.
Do you have examples of GNU/Linux uses that would make no sense?

Other options include calling out to the ‘settrans’ command (inelegant
IMO), writing Guile bindings for some of the Hurd libraries, and/or some
sort of a MiG in Scheme (neat but takes some time!).

Ludo’.
Samuel Thibault
2016-03-02 22:06:54 UTC
Permalink
Post by Ludovic Courtès
Post by Samuel Thibault
It can both help and hurt. Roland's concern is that we will probably not
want to provide a mount() that behaves like Linux, because in some cases
it may just not even make sense. So the "ported" software that would try
to use it would just fail.
Do you have examples of GNU/Linux uses that would make no sense?
I haven't investigated, I just report what kind of fear Roland was
having.

Samuel
Manolis Ragkousis
2016-03-21 12:09:36 UTC
Permalink
Hello everyone,

Although I have uploaded and shared my draft to the GSoC website, I am
also resending it to the lists as per Ludovic's instruction.

I am sharing it as a txt + pdf.

Here is also the link that I shared to the GSoC website as per Google's
instructions.
https://docs.google.com/document/d/1NEEz_PHM2YZTm8okd2BgPRSc4c89I1fGnwbyeeEUrhw/edit?usp=sharing

This is the time to share your ideas/corrections :-)

Manolis
Samuel Thibault
2016-03-22 00:49:03 UTC
Permalink
Hello,
Post by Manolis Ragkousis
Although I have uploaded and shared my draft to the GSoC website, I am
also resending it to the lists as per Ludovic's instruction.
Looks very promising :)

Samuel
Ludovic Courtès
2016-03-23 13:40:38 UTC
Permalink
Hello!
Post by Manolis Ragkousis
Although I have uploaded and shared my draft to the GSoC website, I am
also resending it to the lists as per Ludovic's instruction.
Yeah, the GSoC website makes it possible to provide a link to a text
file, so let’s do that instead of using their SaaSS.
Post by Manolis Ragkousis
2. The Project
The project consists of four main stages
1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core.
2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
3. Successfully boot into one such system using GNU Shepherd with pid 1.
4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module.
This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system.
In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the
relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
for example.
Note that technically the ‘file_set_translator’ function is in libc:

--8<---------------cut here---------------start------------->8---
scheme@(guile-user)> (dynamic-func "file_set_translator" (dynamic-link))
$1 = #<pointer 0x1675660>
--8<---------------cut here---------------end--------------->8---
Post by Manolis Ragkousis
This module will be called (guix build hurd). This allows us to
re-implement the functionality of the 'settrans' command, as described
here[1], in Scheme and use that in place of 'mount()', where
applicable.
Good! (We might as well call it (hurd …) in fact: (hurd fs) would
correspond to fs.defs, (hurd io) for io.defs, and so on.)
Post by Manolis Ragkousis
Additionally, we need to make sure that all modules in (guix build *) and (gnu build *) can offer the same
functionalities on a GNU/Hurd system. For example (gnu build vm) relies on QEMU's '-kernel' command-line
option, which is Linux-specific. On the other hand, in the case of modules like (gnu build linux-boot) or
(gnu build linux-initrd), which by design are Linux-specific, we will need to provide a separate module with
equivalent functionality. (gnu system *) modules will require changes as well, in order to be able to use
modifications that will happen in the (guix build *) and (gnu build *) modules.
I think it’s important to think about:

1. How functionality equivalent to linux-{initrd,boot} will be
implemented on the Hurd.

In my experience the equivalent thing is simpler on the Hurd,
esp. if relying on passive translators (see daemons/runsystem.sh in
the Hurd), though here we’ll probably explicitly activate
translators at boot time.

2. How to structure GuixSD code in a way that abstracts the underlying
OS kernel.

For instance, we could have a (guix build os) module providing an
API that works for both kernels, probably close to the Linux one,
and that would dispatch to the right implementation, the Linux or
Hurd one.

Both of these are quite a bit of design and implementation work that we
should not underestimate.
Post by Manolis Ragkousis
Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific
mechanisms.
1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer
isolation similar to Linux containers as described here[3].
This is really super optional IMO. This module is only used by ‘guix
environment --container’, which is an optional feature.
Post by Manolis Ragkousis
2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root
privileges in order to setup chrooted environments for the builds. In the Hurd this can be done by setting up a
translator as described here[4].
This is more important, and already non-trivial work. If libc provided
‘mount’ with support for MS_BIND (which I think it should), it would
help a bit, but there’s still the problem of the other Linux name spaces
that are used (the CLONE_NEW* clone(2) flags.)

Thus it may make more sense to write this functionality in guix-daemon
using directly the Hurd interfaces. Separate PID name spaces, UID name
spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
“just” a matter of giving the child process ports to separate proc,
auth, etc. translators.

In itself this is also a bit of work. I wonder what the Hurd folks
think about priorities here?
Post by Manolis Ragkousis
3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory
structure from all the files in the individual package directories.
Fun but optional. ;-)
Post by Manolis Ragkousis
3. Estimated Project Timeline
Before March 31
Finish merging the wip-hurd branch to upstream.
Note that this task depends on, ahem, other people as well. ;-)
Post by Manolis Ragkousis
Write a (guix build hurd) module which will contain Guile bindings to the RPC stubs like hurd/fs.defs or hurd/exec.defs .
April 1 - April 15
Package missing dependencies (Hurd libs).
Re-implement 'settrans' in scheme.
Don’t just straight into low-level “details.” I’d recommend
familiarizing yourself with GuixSD on GNU/Linux, fiddling with bits of
code and so on (you can try safely in a VM with ‘guix system vm’), and
then, familiarizing yourself with the GNU/Hurd startup process (looking
at daemons/runsystem.sh already gives a good idea.)
Post by Manolis Ragkousis
April 16 - May 1
At this point we will have the tools needed to build a Hurd based file-system. (Milestone 1)
“Hurd-based file system”?
Post by Manolis Ragkousis
May 2 - May 22
Create (gnu build hurd-boot) and (gnu build hurd-initrd).
Start working on describing the GNU/Hurd system in (gnu system).
I know Debian at some point added initrd support to GNU Mach for some
reason, but fundamentally, the whole point of multiboot is to provide a
solution more flexible than initrds. So, hopefully, no initrds here.
Since there’s no initrd, there’s also no ‘switch_root’ dance (on
Linux-based system the initrd is the initial root file system, so you
need to switch to the real root file system as soon as you’re able to
mount it), which simplifies things.

Justus, Samuel, WDYT?
Post by Manolis Ragkousis
May 23 - 12 June
Modify (gnu system *) modules as needed.
All the modules (guix build *) and (gnu build *) will be working as expected by now.
Try building a GuixSD image. (Milestone 2)
This is the crux of the matter. :-)

Before starting, it would be nice to have a rough idea of how GuixSD
startup will work on the Hurd, and what changes this entails.

For debugging purposes, it would be very helpful to say the least to
have a working ‘guix system vm’: it would allow you to test your changes
very quickly, without rebooting and so on.

This in itself requires some thought: currently (guix system vm) relies
on QEMU’s ‘-kernel’ option to boot the kernel Linux. For GNU/Hurd, we
instead need to boot a complete image with GRUB, like ‘guix system
vm-image’ does. You’ll have to investigate how to port this.
Post by Manolis Ragkousis
13 June - 23 June
Solve any problems with booting into the system and running GNU Shepherd.
24 June - 9 July
Have a fully working GNU/Hurd system. (Milestone 3)
Make sure all the services/packages run correctly on the new GuixSD/Hurd system and solve any issues.
10 July - 8 August
Start working on getting Hurd specific mechanisms integrated to Guix.
What do you mean?
Post by Manolis Ragkousis
9 August - 23 August
By now all the objectives will have been achieved.
Merge patches, code refactoring, documentation.
Deliver a working GuixSD system image with Hurd as the kernel.
Victory! :-)

I think this is super cool, and also super ambitious! I’d expect that
we’d be able to reach milestone #2 if everything goes well (and assuming
we don’t try to address isolation in guix-daemon), but milestone #3 is
most likely for later.

What do people think?

The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts. I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.

Justus, Samuel: could you add yourselves as official co-mentors on
Google’s web site, if not already done? :-)

Thanks, Manolis!

Ludo’.
Justus Winter
2016-03-23 16:55:50 UTC
Permalink
Hi,

Quoting Ludovic Courtès (2016-03-23 14:40:38)
Post by Ludovic Courtès
Post by Manolis Ragkousis
2. The Project
The project consists of four main stages
1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core.
2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
3. Successfully boot into one such system using GNU Shepherd with pid 1.
4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
For me, 4. is the most important bit, so we can build packages in
isolation.
Post by Ludovic Courtès
Post by Manolis Ragkousis
Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module.
This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system.
In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the
relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
for example.
--8<---------------cut here---------------start------------->8---
$1 = #<pointer 0x1675660>
--8<---------------cut here---------------end--------------->8---
Post by Manolis Ragkousis
This module will be called (guix build hurd). This allows us to
re-implement the functionality of the 'settrans' command, as described
here[1], in Scheme and use that in place of 'mount()', where
applicable.
Right. In fact, what settrans (or mount) does isn't that hard to
reproduce, though I wouldn't mind moving the c implementation to
libhurdutil or the like and binding to that.
Post by Ludovic Courtès
1. How functionality equivalent to linux-{initrd,boot} will be
implemented on the Hurd.
In my experience the equivalent thing is simpler on the Hurd,
esp. if relying on passive translators (see daemons/runsystem.sh in
the Hurd), though here we’ll probably explicitly activate
translators at boot time.
Currently, there is not really a reason to have an initrd-like
solution on the Hurd. Yes, one has to start the default pager
explicitly, but that's about it.
Post by Ludovic Courtès
Post by Manolis Ragkousis
Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific
mechanisms.
1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer
isolation similar to Linux containers as described here[3].
This is really super optional IMO. This module is only used by ‘guix
environment --container’, which is an optional feature.
Yes, subhurds are more on the experimental side imho.
Post by Ludovic Courtès
Post by Manolis Ragkousis
2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root
privileges in order to setup chrooted environments for the builds. In the Hurd this can be done by setting up a
translator as described here[4].
This is more important, and already non-trivial work. If libc provided
‘mount’ with support for MS_BIND (which I think it should), it would
help a bit, but there’s still the problem of the other Linux name spaces
that are used (the CLONE_NEW* clone(2) flags.)
Thus it may make more sense to write this functionality in guix-daemon
using directly the Hurd interfaces. Separate PID name spaces, UID name
spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
“just” a matter of giving the child process ports to separate proc,
auth, etc. translators.
In itself this is also a bit of work. I wonder what the Hurd folks
think about priorities here?
I'd go for specializing guix-daemon over trying too hard to implement
Linux-compatible interfaces in the libc. I consider the
filesystem-isolation part easy, UID isolation relatively easy, PID
isolation is a bit tricky. Currently one cannot simply start another
proc server and expect it to work as it needs privileged kernel
interfaces. I created an RPC to allow nested proc servers for
unprivileged subhurds. That should do the trick, it is merely a
matter of making it actually work I guess.
Post by Ludovic Courtès
Post by Manolis Ragkousis
3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory
structure from all the files in the individual package directories.
Fun but optional. ;-)
Agreed.
Post by Ludovic Courtès
Post by Manolis Ragkousis
May 2 - May 22
Create (gnu build hurd-boot) and (gnu build hurd-initrd).
Start working on describing the GNU/Hurd system in (gnu system).
I know Debian at some point added initrd support to GNU Mach for some
reason, but fundamentally, the whole point of multiboot is to provide a
solution more flexible than initrds. So, hopefully, no initrds here.
Since there’s no initrd, there’s also no ‘switch_root’ dance (on
Linux-based system the initrd is the initial root file system, so you
need to switch to the real root file system as soon as you’re able to
mount it), which simplifies things.
Justus, Samuel, WDYT?
Debian has the initrd for the live cds, but that is a bit hacky. I
don't believe we need an initrd at this point.
Post by Ludovic Courtès
Post by Manolis Ragkousis
May 23 - 12 June
Modify (gnu system *) modules as needed.
All the modules (guix build *) and (gnu build *) will be working as expected by now.
Try building a GuixSD image. (Milestone 2)
This is the crux of the matter. :-)
Before starting, it would be nice to have a rough idea of how GuixSD
startup will work on the Hurd, and what changes this entails.
For debugging purposes, it would be very helpful to say the least to
have a working ‘guix system vm’: it would allow you to test your changes
very quickly, without rebooting and so on.
This in itself requires some thought: currently (guix system vm) relies
on QEMU’s ‘-kernel’ option to boot the kernel Linux. For GNU/Hurd, we
instead need to boot a complete image with GRUB, like ‘guix system
vm-image’ does. You’ll have to investigate how to port this.
qemu can boot multiboot operating systems.
Post by Ludovic Courtès
Post by Manolis Ragkousis
Deliver a working GuixSD system image with Hurd as the kernel.
Hurd is not a kernel.
Post by Ludovic Courtès
Victory! :-)
I think this is super cool, and also super ambitious! I’d expect that
we’d be able to reach milestone #2 if everything goes well (and assuming
we don’t try to address isolation in guix-daemon), but milestone #3 is
most likely for later.
What do people think?
The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts. I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.
I consider isolation more important.
Post by Ludovic Courtès
Justus, Samuel: could you add yourselves as official co-mentors on
Google’s web site, if not already done? :-)
I clicked on 'want to mentor', do I need to do more?

Justus
Manolis Ragkousis
2016-03-24 11:18:25 UTC
Permalink
Hello Justus, Hello Ludo
Post by Justus Winter
Quoting Ludovic Courtès (2016-03-23 14:40:38)
Post by Ludovic Courtès
Post by Manolis Ragkousis
2. The Project
The project consists of four main stages
1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core.
2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
3. Successfully boot into one such system using GNU Shepherd with pid 1.
4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
For me, 4. is the most important bit, so we can build packages in
isolation.
I think our priority should be to first get GuixSD working and then
concentrate to achieving isolation. From what I understand none of the
two is trivial but in the long run the ability to spawn GNU/Hurd system
vms on the fly will make it easier to work on it.
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
Currently the tools Guix uses to interact with the filesystem exist inside the (guix build syscalls) module.
This module provides bindings to libc's syscall wrappers, which are only available on a GNU/Linux system.
In order to offer the same functionality on a GNU/Hurd system we must first write Guile bindings for the
relevant Hurd functions, like 'file_set_translator' in hurd/fs.defs
for example.
--8<---------------cut here---------------start------------->8---
$1 = #<pointer 0x1675660>
--8<---------------cut here---------------end--------------->8---
Post by Manolis Ragkousis
This module will be called (guix build hurd). This allows us to
re-implement the functionality of the 'settrans' command, as described
here[1], in Scheme and use that in place of 'mount()', where
applicable.
Right. In fact, what settrans (or mount) does isn't that hard to
reproduce, though I wouldn't mind moving the c implementation to
libhurdutil or the like and binding to that.
Then I think it will be wiser to move the c implementation of settrans
to libhurdutil and binding to that. Then we will only need a (guix
build hurd) module which will include that binding. I will update the
proposal if you agree.
Post by Justus Winter
Post by Ludovic Courtès
1. How functionality equivalent to linux-{initrd,boot} will be
implemented on the Hurd.
In my experience the equivalent thing is simpler on the Hurd,
esp. if relying on passive translators (see daemons/runsystem.sh in
the Hurd), though here we’ll probably explicitly activate
translators at boot time.
Currently, there is not really a reason to have an initrd-like
solution on the Hurd. Yes, one has to start the default pager
explicitly, but that's about it.
Then it seems we only need a hurd-boot module which will take care of
activating translators and booting the system.
Post by Justus Winter
2. How to structure GuixSD code in a way that abstracts the
underlying OS kernel.
For instance, we could have a (guix build os) module providing an
API that works for both kernels, probably close to the Linux one,
and that would dispatch to the right implementation, the Linux or
Hurd one.
This module sounds like a great idea. I will add it to the proposal.
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
Finally, once GuixSD is successfully ported, we can cater to the last stage of taking advantage of Hurd specific
mechanisms.
1)Replacing (gnu build linux-container) with (gnu build subhurd) when on a GNU/Hurd system. Subhurds can offer
isolation similar to Linux containers as described here[3].
This is really super optional IMO. This module is only used by ‘guix
environment --container’, which is an optional feature.
Yes, subhurds are more on the experimental side imho.
So Adding subhurd support -> if there is time.
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
2)Modify the guix-daemon to run without root privileges by utilizing the Hurd architecture. The daemon needs root
privileges in order to setup chrooted environments for the builds. In the Hurd this can be done by setting up a
translator as described here[4].
This is more important, and already non-trivial work. If libc provided
‘mount’ with support for MS_BIND (which I think it should), it would
help a bit, but there’s still the problem of the other Linux name spaces
that are used (the CLONE_NEW* clone(2) flags.)
Thus it may make more sense to write this functionality in guix-daemon
using directly the Hurd interfaces. Separate PID name spaces, UID name
spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
“just” a matter of giving the child process ports to separate proc,
auth, etc. translators.
In itself this is also a bit of work. I wonder what the Hurd folks
think about priorities here?
I'd go for specializing guix-daemon over trying too hard to implement
Linux-compatible interfaces in the libc. I consider the
filesystem-isolation part easy, UID isolation relatively easy, PID
isolation is a bit tricky. Currently one cannot simply start another
proc server and expect it to work as it needs privileged kernel
interfaces. I created an RPC to allow nested proc servers for
unprivileged subhurds. That should do the trick, it is merely a
matter of making it actually work I guess.
As I said above I believe that first getting GuixSD to work will make it
easier to work on this and test any changes.
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
3)Guix uses symlink trees for user profiles. Instead we can use stowfs[5]. Stowfs creates a traditional Unix directory
structure from all the files in the individual package directories.
Fun but optional. ;-)
Agreed.
Stowfs support -> if there is time.
Post by Justus Winter
Post by Ludovic Courtès
Before March 31
Finish merging the wip-hurd branch to upstream.
Note that this task depends on, ahem, other people as well. ;-)
Well one can only hope :-)
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
May 2 - May 22
Create (gnu build hurd-boot) and (gnu build hurd-initrd).
Start working on describing the GNU/Hurd system in (gnu system).
I know Debian at some point added initrd support to GNU Mach for some
reason, but fundamentally, the whole point of multiboot is to provide a
solution more flexible than initrds. So, hopefully, no initrds here.
Since there’s no initrd, there’s also no ‘switch_root’ dance (on
Linux-based system the initrd is the initial root file system, so you
need to switch to the real root file system as soon as you’re able to
mount it), which simplifies things.
Justus, Samuel, WDYT?
Debian has the initrd for the live cds, but that is a bit hacky. I
don't believe we need an initrd at this point.
(gnu build hurd-initrd) -> dropped
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
May 23 - 12 June
Modify (gnu system *) modules as needed.
All the modules (guix build *) and (gnu build *) will be working as expected by now.
Try building a GuixSD image. (Milestone 2)
This is the crux of the matter. :-)
Before starting, it would be nice to have a rough idea of how GuixSD
startup will work on the Hurd, and what changes this entails.
Currently I am working on familiarizing myself with both how Hurd and
GuixSD booting works. I am already using GuixSD as a system and I am
going though the source code so I can better understand what is going on.
Post by Justus Winter
Post by Ludovic Courtès
For debugging purposes, it would be very helpful to say the least to
have a working ‘guix system vm’: it would allow you to test your changes
very quickly, without rebooting and so on.
This in itself requires some thought: currently (guix system vm) relies
on QEMU’s ‘-kernel’ option to boot the kernel Linux. For GNU/Hurd, we
instead need to boot a complete image with GRUB, like ‘guix system
vm-image’ does. You’ll have to investigate how to port this.
qemu can boot multiboot operating systems.
Justus is correct here. I found this
https://www.gnu.org/software/hurd/hurd/running/qemu.html#index9h1 which
explains how to make qemu start with gnumach. This way guix system vm
can work, just with the proper modifications.
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
Deliver a working GuixSD system image with Hurd as the kernel.
Hurd is not a kernel.
I will rephrase it correctly. I am sorry again.
Post by Justus Winter
Post by Ludovic Courtès
The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts. I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.
I consider isolation more important.
I understand that isolation is more important but I think first getting
GuixSD will be more beneficial in the long run. If everything goes
well, even though we may not be able to fully implement isolation in the
duration of this gsoc, we can bootstrap the process so we continue after
the end of the summer.

WDYT?

Thank you :-)
Manolis
Justus Winter
2016-03-24 11:38:32 UTC
Permalink
Quoting Manolis Ragkousis (2016-03-24 12:18:25)
Post by Manolis Ragkousis
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
The project consists of four main stages
1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core.
2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
3. Successfully boot into one such system using GNU Shepherd with pid 1.
4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
For me, 4. is the most important bit, so we can build packages in
isolation.
I think our priority should be to first get GuixSD working and then
concentrate to achieving isolation. From what I understand none of the
two is trivial but in the long run the ability to spawn GNU/Hurd system
vms on the fly will make it easier to work on it.
Otoh if we could properly build packages, we could provide the
substitutes, so people could try Guix on the Hurd without going
through the 12h+ bootstrap procedure.
Post by Manolis Ragkousis
Currently I am working on familiarizing myself with both how Hurd and
GuixSD booting works. I am already using GuixSD as a system and I am
going though the source code so I can better understand what is going on.
Here is an overview of the early server bootstrap in the Hurd. It is
slightly outdated, but still the best description that I know of:

http://teythoon.cryptobitch.de/posts/bootstrapping-the-hurd/
Post by Manolis Ragkousis
Post by Justus Winter
Post by Ludovic Courtès
For debugging purposes, it would be very helpful to say the least to
have a working ‘guix system vm’: it would allow you to test your changes
very quickly, without rebooting and so on.
This in itself requires some thought: currently (guix system vm) relies
on QEMU’s ‘-kernel’ option to boot the kernel Linux. For GNU/Hurd, we
instead need to boot a complete image with GRUB, like ‘guix system
vm-image’ does. You’ll have to investigate how to port this.
qemu can boot multiboot operating systems.
Justus is correct here. I found this
https://www.gnu.org/software/hurd/hurd/running/qemu.html#index9h1 which
explains how to make qemu start with gnumach. This way guix system vm
can work, just with the proper modifications.
Hmmm, so one still needs a filesystem, right? That's going to be a
bit tricky too, since whatever tool you use for that purpose, it
surely does not support creating hurdish passive translator records.
Without passive translator records things get indeed more interesting.
I have a patch for some tool for creating ext2 filesystems that could
help, or we create all the passive translator records on first boot
similar to how Samuels Debian/Hurd live cds deal with that. Or I
finish my bootshell work that can boot from filesystems without
passive translator records.

Justus
Manolis Ragkousis
2016-03-24 12:36:04 UTC
Permalink
Post by Justus Winter
Otoh if we could properly build packages, we could provide the
substitutes, so people could try Guix on the Hurd without going
through the 12h+ bootstrap procedure.
Here the problem is not isolation but the fact that we don't have a
substitutes server. The machine I had at my house crashed and I cannot
bring it back up remotely. When I get back to my home island and revive
it, this will not be an issue.
Post by Justus Winter
Here is an overview of the early server bootstrap in the Hurd. It is
http://teythoon.cryptobitch.de/posts/bootstrapping-the-hurd/
It definitely is!! Nice one!!
Post by Justus Winter
Hmmm, so one still needs a filesystem, right? That's going to be a
bit tricky too, since whatever tool you use for that purpose, it
surely does not support creating hurdish passive translator records.
Without passive translator records things get indeed more interesting.
I have a patch for some tool for creating ext2 filesystems that could
help, or we create all the passive translator records on first boot
similar to how Samuels Debian/Hurd live cds deal with that. Or I
finish my bootshell work that can boot from filesystems without
passive translator records.
Where can I find more info on those?

Manolis
Justus Winter
2016-03-24 12:49:23 UTC
Permalink
Quoting Manolis Ragkousis (2016-03-24 13:36:04)
Post by Manolis Ragkousis
Post by Justus Winter
Hmmm, so one still needs a filesystem, right? That's going to be a
bit tricky too, since whatever tool you use for that purpose, it
surely does not support creating hurdish passive translator records.
Without passive translator records things get indeed more interesting.
I have a patch for some tool for creating ext2 filesystems that could
help, or we create all the passive translator records on first boot
similar to how Samuels Debian/Hurd live cds deal with that. Or I
finish my bootshell work that can boot from filesystems without
passive translator records.
Where can I find more info on those?
https://www.gnu.org/software/hurd/hurd-paper.html#translator

Passive translator records are stored in ext2 using the Hurd on-disk
format:

http://git.sceen.net/hurd/hurd.git/blob/HEAD:/ext2fs/ext2_fs.h#l252

Justus
Ludovic Courtès
2016-03-24 13:22:53 UTC
Permalink
Hi Justus,
Post by Justus Winter
Quoting Ludovic Courtès (2016-03-23 14:40:38)
Post by Ludovic Courtès
Post by Manolis Ragkousis
2. The Project
The project consists of four main stages
1. Modify Guix so it will be able to create and mount the file-system needed to boot into a system with Hurd at its core.
2. Modify Guix so it can produce a working image, while isolating any cases of Linux assumptions.
3. Successfully boot into one such system using GNU Shepherd with pid 1.
4. Modify the new Guix system to take advantage of Hurd specific mechanisms.
For me, 4. is the most important bit, so we can build packages in
isolation.
I guess you mean isolation in guix-daemon.
Post by Justus Winter
Post by Ludovic Courtès
This is more important, and already non-trivial work. If libc provided
‘mount’ with support for MS_BIND (which I think it should), it would
help a bit, but there’s still the problem of the other Linux name spaces
that are used (the CLONE_NEW* clone(2) flags.)
Thus it may make more sense to write this functionality in guix-daemon
using directly the Hurd interfaces. Separate PID name spaces, UID name
spaces, mount name spaces, etc. are fairly natural on the Hurd: it’s
“just” a matter of giving the child process ports to separate proc,
auth, etc. translators.
In itself this is also a bit of work. I wonder what the Hurd folks
think about priorities here?
I'd go for specializing guix-daemon over trying too hard to implement
Linux-compatible interfaces in the libc.
So this would become maybe half of the GSoC coding part maybe. WDYT?
Post by Justus Winter
I consider the filesystem-isolation part easy, UID isolation
relatively easy, PID isolation is a bit tricky. Currently one cannot
simply start another proc server and expect it to work as it needs
privileged kernel interfaces. I created an RPC to allow nested proc
servers for unprivileged subhurds. That should do the trick, it is
merely a matter of making it actually work I guess.
“Merely”, hmm… :-)

So, let’s say PID isolation will be optional, and we can always adjust
later on. Sounds good?

Manolis, make sure to read about how the various Hurd servers provides
these parts of POSIX personality: file systems, UIDs, PIDs, networking,
and so on.

As far as code integration code, I think we won’t bother syncing this
work with nix-daemon; guix-daemon has already diverged, and for instance
it does not have OS X sandbox support.

You’ll have to arrange to have the Hurd-specific bits in a separate
file, so that ‘#ifdef HURD’ are not scattered all over the place.

This is C(++) as you know. WDYT, Manolis?
Post by Justus Winter
Post by Ludovic Courtès
This in itself requires some thought: currently (guix system vm) relies
on QEMU’s ‘-kernel’ option to boot the kernel Linux. For GNU/Hurd, we
instead need to boot a complete image with GRUB, like ‘guix system
vm-image’ does. You’ll have to investigate how to port this.
qemu can boot multiboot operating systems.
Great, didn’t know that.
Post by Justus Winter
Post by Ludovic Courtès
Post by Manolis Ragkousis
Deliver a working GuixSD system image with Hurd as the kernel.
Hurd is not a kernel.
I think we use “kernel” to mean “the thing(s) that libc talks to.”
Post by Justus Winter
Post by Ludovic Courtès
The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts. I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.
I consider isolation more important.
OK.

So, Manolis, what about reframing the agenda such that porting
guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the
programming effort), followed by GuixSD stuff?

Thanks,
Ludo’.
Manolis Ragkousis
2016-03-24 13:55:31 UTC
Permalink
Hey
Post by Ludovic Courtès
So, let’s say PID isolation will be optional, and we can always adjust
later on. Sounds good?
Manolis, make sure to read about how the various Hurd servers provides
these parts of POSIX personality: file systems, UIDs, PIDs, networking,
and so on.
Already started. :-)
Post by Ludovic Courtès
As far as code integration code, I think we won’t bother syncing this
work with nix-daemon; guix-daemon has already diverged, and for instance
it does not have OS X sandbox support.
You’ll have to arrange to have the Hurd-specific bits in a separate
file, so that ‘#ifdef HURD’ are not scattered all over the place.
This is C(++) as you know. WDYT, Manolis?
I have to start studying the daemon's code more. From what I know the
part that handles builds is in libstore/build.cc. I will start there.

I think I can do it.
Post by Ludovic Courtès
Post by Justus Winter
Post by Ludovic Courtès
The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts. I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.
I consider isolation more important.
OK.
Isolation first it is then.
Post by Ludovic Courtès
So, Manolis, what about reframing the agenda such that porting
guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the
programming effort), followed by GuixSD stuff?
Current objectives then:
1) Achieve build isolation in the daemon on the Hurd.
2) Modify Guix so it can produce a working image, while isolating any
cases of Linux assumptions.
3) Boot to GuixSD

Ludo, Justus do you agree with this?

Manolis
Justus Winter
2016-03-24 14:30:14 UTC
Permalink
Hi,

Quoting Manolis Ragkousis (2016-03-24 14:55:31)
Post by Manolis Ragkousis
Post by Ludovic Courtès
Post by Justus Winter
Post by Ludovic Courtès
The main question is whether you should implement build isolation in
guix-daemon, in which case that would leave little time for the GuixSD
parts. I think I would rather let you focus on the GuixSD stuff as you
wrote, but I’d like to hear what the Hurd folks think.
I consider isolation more important.
OK.
Isolation first it is then.
Post by Ludovic Courtès
So, Manolis, what about reframing the agenda such that porting
guix-daemon to GNU/Hurd comes first (I’d consider it roughly half of the
programming effort), followed by GuixSD stuff?
1) Achieve build isolation in the daemon on the Hurd.
2) Modify Guix so it can produce a working image, while isolating any
cases of Linux assumptions.
3) Boot to GuixSD
Well, I didn't ment to stomp over your priorities like that, it is just
that I personally consider it more important and from my point of view
it should be easier to achieve.

Justus
Manolis Ragkousis
2016-03-04 15:09:06 UTC
Permalink
Hey
Post by Ludovic Courtès
Do you have examples of GNU/Linux uses that would make no sense?
After a quick look in mount's man page I think that maybe Roland was
referring to some flags (like relatime for example) that when passed to
mount, expect the Linux kernel to handle the filesystem in a specific way.
Post by Ludovic Courtès
Other options include calling out to the ‘settrans’ command (inelegant
IMO), writing Guile bindings for some of the Hurd libraries, and/or some
sort of a MiG in Scheme (neat but takes some time!).
We could have a guile wrapper for the ‘settrans’ command that could
cover our needs. Isn't this feasible? WDYT?

Manolis
Ludovic Courtès
2016-03-05 22:02:15 UTC
Permalink
[...]
Post by Manolis Ragkousis
Post by Ludovic Courtès
Other options include calling out to the ‘settrans’ command (inelegant
IMO), writing Guile bindings for some of the Hurd libraries, and/or some
sort of a MiG in Scheme (neat but takes some time!).
We could have a guile wrapper for the ‘settrans’ command that could
cover our needs. Isn't this feasible? WDYT?
It’s definitely feasible, and probably the easiest option. It scores
poorly in terms of elegance though. :-)

Ludo’.
Ben Woodcroft
2016-02-06 23:53:00 UTC
Permalink
On 06/02/16 21:38, Pjotr Prins wrote:
[..]
Post by Pjotr Prins
- Add all testing frameworks to Guix (e.g. cucumber)
Hey just fyi I think I just got cucumber working here
https://github.com/wwood/guix_mine/blob/master/ben/packages/rails.scm

but the (7?) patches need finalisation (updating to newest
versions/synopses/description/lint/reproducibility etc.) before
submission here.

For Ruby, circular dependencies seem to be an ongoing issue. I'm not
sure how hard this is, but an error message more useful than "stack
overflow" or whatever it is would be helpful, perhaps this is something
a student could work on.
Pjotr Prins
2016-02-07 05:51:48 UTC
Permalink
Post by Ben Woodcroft
[..]
Post by Pjotr Prins
- Add all testing frameworks to Guix (e.g. cucumber)
Hey just fyi I think I just got cucumber working here
https://github.com/wwood/guix_mine/blob/master/ben/packages/rails.scm
but the (7?) patches need finalisation (updating to newest
versions/synopses/description/lint/reproducibility etc.) before
submission here.
Cool! Your packaging of Rails comes a long way too.

Of course there is a lot more to package still ;)
Post by Ben Woodcroft
For Ruby, circular dependencies seem to be an ongoing issue. I'm not
sure how hard this is, but an error message more useful than "stack
overflow" or whatever it is would be helpful, perhaps this is
something a student could work on.
Interesting point. Question is whether it is all laziness of upstream,
or that it is a form of bootstrapping the test systems ;). I'll
factor it in if anyone is interested in this project in the SciRuby
world.

Pj.


--
Continue reading on narkive:
Loading...