Discussion:
the importance of rust-build-system [Fwd: [tor-dev] Tor in a safer language: Network team update from Amsterdam]
(too old to reply)
ng0
2017-04-01 07:58:41 UTC
Permalink
Raw Message
tor is seriously considering to switch to rust. I have my own open
bugs / roadmap points which depend on our rust-build-system working.
tor will not switch immediately, but it shows the importance of having a
working rust-build-system. Firefox will switch at some point.
Danny, could you list what's left for completion? Is it just circular
dependencies? I'm about to publish my project page within the next few
days. I hope you don't mind if I list you as a go-to person for getting
involved in upstream (Guix) to fix up the rust-build-system.
If you do mind, please let me know. I will not publish your email
address, I'll rather point to a git commit.

----- Forwarded message from Sebastian Hahn -----

From: Sebastian Hahn
To: tor-***@lists.torproject.org
Subject: [tor-dev] Tor in a safer language: Network team update from Amsterdam

Hi there tor-dev,

as an update to those who didn't have the chance to meet with us in
Amsterdam or those who haven't followed the efforts to rely on C less,
here's what happened at the "let's not fight about Go versus Rust, but
talk about how to migrate Tor to a safer language" session and what
happened after.

Notes from session:

We didn't fight about Rust or Go or modern C++. Instead, we focused on
identifying goals for migrating Tor to a memory-safe language, and how
to get there. With that frame of reference, Rust emerged as a extremely
strong candidate for the incremental improvement style that we
considered necessary. We were strongly advised to not use cgo, by people
who have used it extensively.

As there are clearly a lot of unknowns with this endeavor, and a lot
that we will learn/come up against along the way, we feel that Rust is a
compelling option to start with, with the caveat that we will first
experiment, learn from the experience, and then build on what we learn.

You can also check out the session notes on the wiki (submitted, but not
posted yet).[1]

The real fun part started after the session. We got together to actually
make a plan for an experiment and to give Rust a serious chance. We
quickly got a few trivial things working like statically linking Rust
into Tor, integrating with the build system to call out to cargo for the
Rust build, and using Tor's allocator from Rust.

We're planning to write up a blog post summarizing our experiences so
far while hopefully poking the Rust developers to prioritize the missing
features so we can stop using nightly Rust soon (~months, instead of
years).

We want to have a patch merged into tor soon so you can all play with
your dev setup to help identify any challenges. We want to stress that
this is an optional experiment for now, we would love feedback but
nobody is paid to work on this and nobody is expected to spend more
time than they have sitting around.

We have committed to reviewing any patch that includes any Rust code to
provide feedback, get experience to develop a style, and actually make
use of this experiment. This means we're not ready to take on big
patches that add lots of tricky stuff quite now, we want to take it slow
and learn from this.

We would like to do a session at the next dev meeting to give updates on
this effort, but in the meantime, if team members would like to start
learning Rust and helping us identify/implement small and well-isolated
areas to begin migration, or new pieces of functionality that we can
build immediately in Rust, that would be really great.

So, for a TLDR:

What has already been done:
- Rust in Tor build
- Putting together environment setup instructions and a (very small)
initial draft for coding standards
- Initial work to identify good candidates for migration (not tightly
interdependent)

What we think are next steps:
- Define conventions for the API boundary between Rust and C
- Add a non-trivial Rust API and deploy with a flag to optionally use
(to test support with a safe fallback)
- Learn from similar projects
- Add automated tooling for Rust, such as linting and testing


Cheers
Alex, Chelsea, Sebastian

[1]: Will be visible here https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
_______________________________________________
tor-dev mailing list
tor-***@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev

----- End forwarded message -----
Leo Famulari
2017-04-01 18:58:24 UTC
Permalink
Raw Message
Post by ng0
tor is seriously considering to switch to rust. I have my own open
bugs / roadmap points which depend on our rust-build-system working.
tor will not switch immediately, but it shows the importance of having a
working rust-build-system. Firefox will switch at some point.
It's immediately important because the latest version of librsvg (used
to create the GuixSD GRUB background image) is written in Rust. We could
even update that package "the hard way", without a full
rust-build-system.
Ludovic Courtès
2017-04-01 22:22:14 UTC
Permalink
Raw Message
Post by Leo Famulari
Post by ng0
tor is seriously considering to switch to rust. I have my own open
bugs / roadmap points which depend on our rust-build-system working.
tor will not switch immediately, but it shows the importance of having a
working rust-build-system. Firefox will switch at some point.
It's immediately important because the latest version of librsvg (used
to create the GuixSD GRUB background image) is written in Rust. We could
even update that package "the hard way", without a full
rust-build-system.
For librsvg and Tor, a switch to Rust is both good news and bad news:
it’s good news because using a memory-safe language is indeed a wise
decision, but it’s bad news because it introduces a single point of
trust (opaque Rust binaries built by Mozilla) in our dependency graph.

Ludo’.
ng0
2017-12-08 10:04:29 UTC
Permalink
Raw Message
Post by Leo Famulari
Post by ng0
tor is seriously considering to switch to rust. I have my own open
bugs / roadmap points which depend on our rust-build-system working.
tor will not switch immediately, but it shows the importance of having a
working rust-build-system. Firefox will switch at some point.
It's immediately important because the latest version of librsvg (used
to create the GuixSD GRUB background image) is written in Rust. We could
even update that package "the hard way", without a full
rust-build-system.
it’s good news because using a memory-safe language is indeed a wise
decision, but it’s bad news because it introduces a single point of
trust (opaque Rust binaries built by Mozilla) in our dependency graph.
Ludo’.
It gets even worse: https://www.mercurial-scm.org/wiki/OxidationPlan
So being able to handle rust will be necessary not only for Guix, but
for GNU (iirc we had some Mercurial repos somewhere in GNU, at least
GNU Octave and GNU Health)!
--
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://c.n0.is/ng0_pubkeys/tree/keys
WWW: https://n0.is
ng0
2017-04-02 13:02:25 UTC
Permalink
Raw Message
Oh sweet damn.... My friends and myself so ignorant of this April Fools
annoyance that I would say: Please double check this message if
torproject
does not have the same opinion on aprils fools as I do.

But: the message of tor was merely an opener to my question. I'm still
curious in what needs to be done so that I can get people to join on the
rust parts of my project ;)
Post by ng0
tor is seriously considering to switch to rust. I have my own open
bugs / roadmap points which depend on our rust-build-system working.
tor will not switch immediately, but it shows the importance of having a
working rust-build-system. Firefox will switch at some point.
Danny, could you list what's left for completion? Is it just circular
dependencies? I'm about to publish my project page within the next few
days. I hope you don't mind if I list you as a go-to person for getting
involved in upstream (Guix) to fix up the rust-build-system.
If you do mind, please let me know. I will not publish your email
address, I'll rather point to a git commit.
----- Forwarded message from Sebastian Hahn -----
From: Sebastian Hahn
Subject: [tor-dev] Tor in a safer language: Network team update from Amsterdam
Hi there tor-dev,
as an update to those who didn't have the chance to meet with us in
Amsterdam or those who haven't followed the efforts to rely on C less,
here's what happened at the "let's not fight about Go versus Rust, but
talk about how to migrate Tor to a safer language" session and what
happened after.
We didn't fight about Rust or Go or modern C++. Instead, we focused on
identifying goals for migrating Tor to a memory-safe language, and how
to get there. With that frame of reference, Rust emerged as a extremely
strong candidate for the incremental improvement style that we
considered necessary. We were strongly advised to not use cgo, by people
who have used it extensively.
As there are clearly a lot of unknowns with this endeavor, and a lot
that we will learn/come up against along the way, we feel that Rust is a
compelling option to start with, with the caveat that we will first
experiment, learn from the experience, and then build on what we learn.
You can also check out the session notes on the wiki (submitted, but not
posted yet).[1]
The real fun part started after the session. We got together to actually
make a plan for an experiment and to give Rust a serious chance. We
quickly got a few trivial things working like statically linking Rust
into Tor, integrating with the build system to call out to cargo for the
Rust build, and using Tor's allocator from Rust.
We're planning to write up a blog post summarizing our experiences so
far while hopefully poking the Rust developers to prioritize the missing
features so we can stop using nightly Rust soon (~months, instead of
years).
We want to have a patch merged into tor soon so you can all play with
your dev setup to help identify any challenges. We want to stress that
this is an optional experiment for now, we would love feedback but
nobody is paid to work on this and nobody is expected to spend more
time than they have sitting around.
We have committed to reviewing any patch that includes any Rust code to
provide feedback, get experience to develop a style, and actually make
use of this experiment. This means we're not ready to take on big
patches that add lots of tricky stuff quite now, we want to take it slow
and learn from this.
We would like to do a session at the next dev meeting to give updates on
this effort, but in the meantime, if team members would like to start
learning Rust and helping us identify/implement small and well-isolated
areas to begin migration, or new pieces of functionality that we can
build immediately in Rust, that would be really great.
- Rust in Tor build
- Putting together environment setup instructions and a (very small)
initial draft for coding standards
- Initial work to identify good candidates for migration (not tightly
interdependent)
- Define conventions for the API boundary between Rust and C
- Add a non-trivial Rust API and deploy with a flag to optionally use
(to test support with a safe fallback)
- Learn from similar projects
- Add automated tooling for Rust, such as linting and testing
Cheers
Alex, Chelsea, Sebastian
[1]: Will be visible here https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
_______________________________________________
tor-dev mailing list
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
----- End forwarded message -----
ng0
2017-04-02 13:05:46 UTC
Permalink
Raw Message
Post by ng0
Oh sweet damn.... My friends and myself so ignorant of this April Fools
annoyance that I would say: Please double check this message if
torproject
does not have the same opinion on aprils fools as I do.
https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
and
https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes/MemorySafeLanguagesandTor

suggest however that it really is *not* a joke as it happened long
before April 1st.
Post by ng0
But: the message of tor was merely an opener to my question. I'm still
curious in what needs to be done so that I can get people to join on the
rust parts of my project ;)
Post by ng0
tor is seriously considering to switch to rust. I have my own open
bugs / roadmap points which depend on our rust-build-system working.
tor will not switch immediately, but it shows the importance of having a
working rust-build-system. Firefox will switch at some point.
Danny, could you list what's left for completion? Is it just circular
dependencies? I'm about to publish my project page within the next few
days. I hope you don't mind if I list you as a go-to person for getting
involved in upstream (Guix) to fix up the rust-build-system.
If you do mind, please let me know. I will not publish your email
address, I'll rather point to a git commit.
----- Forwarded message from Sebastian Hahn -----
From: Sebastian Hahn
Subject: [tor-dev] Tor in a safer language: Network team update from Amsterdam
Hi there tor-dev,
as an update to those who didn't have the chance to meet with us in
Amsterdam or those who haven't followed the efforts to rely on C less,
here's what happened at the "let's not fight about Go versus Rust, but
talk about how to migrate Tor to a safer language" session and what
happened after.
We didn't fight about Rust or Go or modern C++. Instead, we focused on
identifying goals for migrating Tor to a memory-safe language, and how
to get there. With that frame of reference, Rust emerged as a extremely
strong candidate for the incremental improvement style that we
considered necessary. We were strongly advised to not use cgo, by people
who have used it extensively.
As there are clearly a lot of unknowns with this endeavor, and a lot
that we will learn/come up against along the way, we feel that Rust is a
compelling option to start with, with the caveat that we will first
experiment, learn from the experience, and then build on what we learn.
You can also check out the session notes on the wiki (submitted, but not
posted yet).[1]
The real fun part started after the session. We got together to actually
make a plan for an experiment and to give Rust a serious chance. We
quickly got a few trivial things working like statically linking Rust
into Tor, integrating with the build system to call out to cargo for the
Rust build, and using Tor's allocator from Rust.
We're planning to write up a blog post summarizing our experiences so
far while hopefully poking the Rust developers to prioritize the missing
features so we can stop using nightly Rust soon (~months, instead of
years).
We want to have a patch merged into tor soon so you can all play with
your dev setup to help identify any challenges. We want to stress that
this is an optional experiment for now, we would love feedback but
nobody is paid to work on this and nobody is expected to spend more
time than they have sitting around.
We have committed to reviewing any patch that includes any Rust code to
provide feedback, get experience to develop a style, and actually make
use of this experiment. This means we're not ready to take on big
patches that add lots of tricky stuff quite now, we want to take it slow
and learn from this.
We would like to do a session at the next dev meeting to give updates on
this effort, but in the meantime, if team members would like to start
learning Rust and helping us identify/implement small and well-isolated
areas to begin migration, or new pieces of functionality that we can
build immediately in Rust, that would be really great.
- Rust in Tor build
- Putting together environment setup instructions and a (very small)
initial draft for coding standards
- Initial work to identify good candidates for migration (not tightly
interdependent)
- Define conventions for the API boundary between Rust and C
- Add a non-trivial Rust API and deploy with a flag to optionally use
(to test support with a safe fallback)
- Learn from similar projects
- Add automated tooling for Rust, such as linting and testing
Cheers
Alex, Chelsea, Sebastian
[1]: Will be visible here https://trac.torproject.org/projects/tor/wiki/org/meetings/2017Amsterdam/Notes
_______________________________________________
tor-dev mailing list
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-dev
----- End forwarded message -----
Danny Milosavljevic
2017-04-02 22:35:24 UTC
Permalink
Raw Message
Hi ng0,

On Sat, 1 Apr 2017 07:58:41 +0000
Post by ng0
Danny, could you list what's left for completion? Is it just circular
dependencies?
Very little is missing:
- Rustc and cargo should be disentangled. Right now they have to be updated in lockstep.
- Rust has a new optional maker (instead of makefiles) called rustbuild; for some reason I didn't get it to work. Future versions will drop the makefiles, so we have to get it to work eventually.
- Eventually we could try to bootstrap a Rust compiler from the original ocaml sources. I'm trying to do that now but it's still broken. It would be fine to make it memory-safe by just not freeing anything ever - since it would only be used to compile the Rust compiler.
- Earlier we had an heuristic in that if there's a Cargo.lock file present we assume that it's an application, and if there isn't one we assume that it's a library. Not sure how safe that heuristic is. What's the official way to find out what it is?
- There's a design limitation in that our rust build system doesn't API support version ranges but cargo does. If multiple libraries depend on the same library with differing API version ranges, cargo uses an version range intersection algorithm in order to find out which implementation version to use. We don't do that - although we do use cargo, so it will fail in that case (and not do something invalid silently).

Note that Rust itself makes a distinction between stable features that are guaranteed to stay and stay the same in the future and unstable features that don't. Many Rust crates use unstable features, among them very basic crates. That makes us unable to use them, and rightfully so.

Other than that I've got a big number of (unpolished) Rust packages that do work.
Post by ng0
days. I hope you don't mind if I list you as a go-to person for getting
involved in upstream (Guix) to fix up the rust-build-system.
Okay.
ng0
2017-12-07 20:59:21 UTC
Permalink
Raw Message
Hi,

my apologies. Occasionally I have a tendency for
very late replies in some threads, as you might have noticed.
Post by Danny Milosavljevic
Hi ng0,
On Sat, 1 Apr 2017 07:58:41 +0000
Post by ng0
Danny, could you list what's left for completion? Is it just circular
dependencies?
- Rustc and cargo should be disentangled. Right now they have to be updated in lockstep.
That's not really a problem for producing functional rust crate binaries.
I consider this optional. Or am I wrong?
Post by Danny Milosavljevic
- Rust has a new optional maker (instead of makefiles) called rustbuild;
for some reason I didn't get it to work. Future versions will drop
the makefiles, so we have to get it to work eventually.
Can you share the issues you had with this? How far did you come?
Post by Danny Milosavljevic
- Eventually we could try to bootstrap a Rust compiler from the
original ocaml sources. I'm trying to do that now but it's still
broken. It would be fine to make it memory-safe by just not
freeing anything ever - since it would only be used to compile the
Rust compiler.
Wouldn't it be easier to help the mrustc project, which is aimed at just that (bootstrapping rustc) in the long run?
Is it even possible at this point to bootstrap rustc using OCaml without wasting too much resources?
They've come a long way and many releases since they've gone selfhosted.

Can you share your findings and work on this?
Post by Danny Milosavljevic
- Earlier we had an heuristic in that if there's a Cargo.lock file present we assume
that it's an application, and if there isn't one we assume that it's a library.
Not sure how safe that heuristic is. What's the official way to find out what it is?
- There's a design limitation in that our rust build system doesn't API support
version ranges but cargo does. If multiple libraries depend on the same library
with differing API version ranges, cargo uses an version range intersection
algorithm in order to find out which implementation version to use.
We don't do that - although we do use cargo, so it will fail in that case (and
not do something invalid silently).
Can you explain this a bit more in detail? You seem to have put more time to
work/think about these issues than I have done so far.
I need to understand most of the issues, so that I can communicate them to
people who'd work on them. At least to get them started on the issues.
Post by Danny Milosavljevic
Note that Rust itself makes a distinction between stable features that
are guaranteed to stay and stay
the same in the future and unstable features that don't.
Many Rust crates use unstable features, among them very basic crates.
That makes us unable to use them, and rightfully so.
You mean fundamental basic crates required by many applications?
Do you have some insight into the implications of this?
I would provide a rustc-nightly outside of Guix master/official,
that's not an issue for me.
Post by Danny Milosavljevic
Other than that I've got a big number of (unpolished) Rust packages that do work.
Can you share these package definitions somewhere? I'd like
to help out and see how many intersections (duplicates) we
both have in our unfinished rust crates work.
Post by Danny Milosavljevic
Post by ng0
days. I hope you don't mind if I list you as a go-to person for getting
involved in upstream (Guix) to fix up the rust-build-system.
Okay.
--
GnuPG: A88C8ADD129828D7EAC02E52E22F9BBFEE348588
GnuPG: https://c.n0.is/ng0_pubkeys/tree/keys
WWW: https://n0.is
Loading...