Should python-build-system packages have native-inputs?

[Chris Marusich]

Hi Guix,

I've noticed that a fair number of packages in gnu/packages/python.scm
using the python-build-system declare native-inputs. I suspect that in
every case, these should actually just be inputs. I also suspect that
this is benign, except perhaps for the fact that it may confuse
Pythonistas who (like myself) initially started out by looking at these
packages as examples of how to get started defining packages in Guix.

The python-build-system's "lower" procedure (in
guix/build-system/python.scm) explicitly forbids cross-compilation:

--8<---------------cut here---------------start------------->8---
(define* (lower name
#:key source inputs native-inputs outputs system target
(python (default-python))
#:allow-other-keys
#:rest arguments)
"Return a bag for NAME."
(define private-keywords
'(#:source #:target #:python #:inputs #:native-inputs))

(and (not target) ;XXX: no cross-compilation
(bag
(name name)
(system system)
(host-inputs `(,@(if source
`(("source" ,source))
'())
,@inputs

;; Keep the standard inputs of 'gnu-build-system'.
,@(standard-packages)))
(build-inputs `(("python" ,python)
,@native-inputs))
(outputs outputs)
(build python-build)
(arguments (strip-keyword-arguments private-keywords arguments)))))
--8<---------------cut here---------------end--------------->8---

As for the native-inputs, they get stored in the bag's build-inputs,
which eventually find their way to the "inputs" keyword argument used on
the build side by the various build phases. In fact, the inputs,
propagated-inputs, and native-inputs of any package that uses the
python-build-system are all put into this "inputs" keyword argument.

With this in mind, I have two questions:

* Should we change these native-inputs to inputs to prevent confusion?
I can personally vouch for the fact that the presence of native-inputs
in python-build-system packages confused the heck out of me at first!

* Are there any circumstances under which it actually WOULD make sense
to cross-compile a Python package?

For now, I think the answers to these questions are "sure" and "probably
not", respectively. I'm very curious to hear your thoughts about the
second question, in particular!

[Fis Trivial]

Post by Chris Marusich
Hi Guix,
I've noticed that a fair number of packages in gnu/packages/python.scm
using the python-build-system declare native-inputs. I suspect that in
every case, these should actually just be inputs. I also suspect that
this is benign, except perhaps for the fact that it may confuse
Pythonistas who (like myself) initially started out by looking at these
packages as examples of how to get started defining packages in Guix.
The python-build-system's "lower" procedure (in
--8<---------------cut here---------------start------------->8---
(define* (lower name
#:key source inputs native-inputs outputs system target
(python (default-python))
#:allow-other-keys
#:rest arguments)
"Return a bag for NAME."
(define private-keywords
'(#:source #:target #:python #:inputs #:native-inputs))
(and (not target) ;XXX: no cross-compilation
(bag
(name name)
(system system)
`(("source" ,source))
'())
;; Keep the standard inputs of 'gnu-build-system'.
(build-inputs `(("python" ,python)
(outputs outputs)
(build python-build)
(arguments (strip-keyword-arguments private-keywords arguments)))))
--8<---------------cut here---------------end--------------->8---
As for the native-inputs, they get stored in the bag's build-inputs,
which eventually find their way to the "inputs" keyword argument used on
the build side by the various build phases. In fact, the inputs,
propagated-inputs, and native-inputs of any package that uses the
python-build-system are all put into this "inputs" keyword argument.
* Should we change these native-inputs to inputs to prevent confusion?
I can personally vouch for the fact that the presence of native-inputs
in python-build-system packages confused the heck out of me at first!
* Are there any circumstances under which it actually WOULD make sense
to cross-compile a Python package?
For now, I think the answers to these questions are "sure" and "probably
not", respectively. I'm very curious to hear your thoughts about the
second question, in particular!

I'm confused, some native-inputs are for testing. They shouldn't be
installed in normal case.

[Mark H Weaver]

Post by Fis Trivial
I'm confused, some native-inputs are for testing. They shouldn't be
installed in normal case.

native-inputs are not only for testing. More generally, when
cross-compiling, any programs that must be run on the build machine must
be 'native-inputs'. Any programs that must be run on the target machine
must be 'inputs'.

Mark

[Hartmut Goebel]

Post by Chris Marusich
* Should we change these native-inputs to inputs to prevent confusion?
I can personally vouch for the fact that the presence of native-inputs
in python-build-system packages confused the heck out of me at first!

 
As Fis already wrote:  These native-inputs are for testing and shouldn't
be installed in normal case. Please see "Python Modules" in the manual:

Python packages required only at build time---e.g., those listed with
the @code{setup_requires} keyword in @file{setup.py}---or only for
testing---e.g., those in @code{tests_require}---go into
@code{native-inputs}.  The rationale is that (1) they do not need to be
propagated because they are not needed at run time, and (2) in a
cross-compilation context, it's the ``native'' input that we'd want.

Post by Chris Marusich
* Are there any circumstances under which it actually WOULD make sense
to cross-compile a Python package?

Of course: Pure-python packages should be able to be cross-compiled
without any problems, sicne the bytes-code is the same for all
platforms. And for extension modules it would allow compiling on a
faster environment (e.g. x86 vs. ARMv4).

(I was not aware of python packages are not cross-compiled, thus I can
only guess the reason why this is not possible: Python distutils may not
be able to *cross*-compile extension modules. Maybe we could work on this.)

[Chris Marusich]

Hi Fis and Hartmut,

Thank you for the quick response!

As Fis already wrote:  These native-inputs are for testing and shouldn't
be installed in normal case.

It's true that for some of the packages that use the
python-build-system, we have been putting the dependencies required for
testing (such as python-pytest) into the package's native-inputs.
However, whether such dependencies are inputs or native-inputs does not
matter. Because the python-build-system never cross-compiles, all of
the inputs, propagated-inputs, and native-inputs will be included in the
single list that gets passed to each of the build phases via the
#:inputs keyword argument. You can verify this yourself by inserting
debug statements in the build phases.

In other words, it doesn't matter if we put python-pytest in a package's
inputs or its native-inputs. The end result is the same. If the
python-build-system actually did support cross-compilation, then this
might be a different story. However, the python-build-system doesn't
cross-compile. As a result, native-inputs and inputs are treated the
same in all of the phases defined in guix/build/python-build-system.scm.

Python packages required only at build time---e.g., those listed with
@code{native-inputs}.  The rationale is that (1) they do not need to be
propagated because they are not needed at run time, and (2) in a
cross-compilation context, it's the ``native'' input that we'd want.

Thank you for mentioning the manual; I had forgotten that we include
explicit guidance for Python modules. I've just reviewed the "Python
Modules" section. I think we should not be advising people to use
native-inputs in packages that use the python-build-system. There is no
meaningful difference between "native-inputs" and "inputs" in this case,
so asking people to contemplate the difference is like asking them a
kōan. It's just going to cause confusion.

This is confusing. And that is precisely why I think we should stop
declaring native-inputs for packages that use the python-build-system.
My understanding is that the concept of "native-inputs" for a package
only makes sense when that package uses a build system that can
cross-compile, such as the gnu-build-system. Because the
python-build-system never cross-compiles, it doesn't make sense to
declare native-inputs for a package that uses the python-build-system.
Instead, those dependencies should just be declared as inputs.

Post by Chris Marusich
* Are there any circumstances under which it actually WOULD make sense
to cross-compile a Python package?

Of course: Pure-python packages should be able to be cross-compiled
without any problems, sicne the bytes-code is the same for all
platforms.

I'm not sure that's the same thing as cross compilation. When cross
compiling a program for a different architecture, the output of the
build is different for each architecture. If Python's bytecode is the
same for all platforms, then it sounds like no cross-compilation is
necessary, which suggests that the notion of "cross compilation" does
not make sense for Python code. Did I misinterpret what you meant?

And for extension modules it would allow compiling on a faster
environment (e.g. x86 vs. ARMv4).
(I was not aware of python packages are not cross-compiled, thus I can
only guess the reason why this is not possible: Python distutils may not
be able to *cross*-compile extension modules. Maybe we could work on this.)

I am curious about extension modules. I understand they are tied
closely to the underlying architecture, but I have little experience
with them, so I'm not sure how they relate to cross compilation. In any
case, it doesn't change the fact that today, the python-build-system
does not cross-compile.

[Hartmut Goebel]

Am 28.04.2018 um 12:11 schrieb Chris Marusich:

I understand your concerns, and I understand why this is hard to get for
a Pythonista. But this is exactly why we added this section to the manual.

Post by Chris Marusich
Because the python-build-system never cross-compiles,

This is an implementation detail which might might change. And if we
remove all inputs now, we need to add again them later. This is a lot of
work, I know since I've cleaned this up for all Python modules. IMHO
it's not a good idea for drop this knowledge from the code.

Post by Chris Marusich
If the
python-build-system actually did support cross-compilation, then this
might be a different story.

Maybe this is going to change somewhen :-) We should aim to the top, not
the status quo :-)

Post by Chris Marusich
My understanding is that the concept of "native-inputs" for a package
only makes sense when that package uses a build system that can
cross-compile,

This is my understanding, too. But the python-build-system might be able
to cross-compile somewhen and then this information is essential.

Post by Chris Marusich

And for extension modules it would allow compiling on a faster
environment (e.g. x86 vs. ARMv4).
(I was not aware of python packages are not cross-compiled, thus I can
only guess the reason why this is not possible: Python distutils may not
be able to *cross*-compile extension modules. Maybe we could work on this.)

I am curious about extension modules. I understand they are tied
closely to the underlying architecture, but I have little experience
with them, so I'm not sure how they relate to cross compilation.

Extension modules are simply modules or libraries  written in C/C++ or
other languages. Even modules written in Cython would be counted in
here, since they are translated to C and then compiled into platform
dependent code.

Post by Chris Marusich
In any
case, it doesn't change the fact that today, the python-build-system
does not cross-compile.

In any case, this is a current limitation only :-)

[Chris Marusich]

Post by Hartmut Goebel

the python-build-system does not cross-compile.

In any case, this is a current limitation only :-)

I see. If we actually do plan on implementing some kind of
cross-compilation support for the python-build-system, then I can
understand why it makes sense to proactively put into the native-inputs
those things which might possibly need to be native-inputs when that
time comes. Can we at least mention in the manual that the
python-build-system doesn't currently cross-compile, so native-inputs
will be treated the same as inputs for now, but we still recommend
putting the "build-only dependencies" in native-inputs in order to
future-proof our package definitions? That alone would have helped
clarify things for me when I was starting out.

After reading about Python extension modules a little more, it seems
that they can in fact be cross-compiled. I didn't look into Python 2,
and I don't know what it would take to enable such cross-compilation in
the python-build-system.

For those following along, here are some related links.

An explanation of Python extension modules:
https://docs.python.org/3/extending/building.html#distributing-your-extension-modules

Some Python open bugs that mention cross-compilation:
https://bugs.python.org/issue?%40columns=id%2Cactivity%2Ctitle%2Ccreator%2Cassignee%2Cstatus&%40sort=-activity&%40group=priority&%40filter=status&status=-1%2C1%2C3&%40search_text=cross-compile&submit=search+in+open+issues

[Mark H Weaver]

Post by Hartmut Goebel

Post by Chris Marusich
Because the python-build-system never cross-compiles,

This is an implementation detail which might might change. And if we
remove all inputs now, we need to add again them later. This is a lot of
work, I know since I've cleaned this up for all Python modules. IMHO
it's not a good idea for drop this knowledge from the code.

I agree.

Mark