GNU bug report logs - #48463
gnu: Add j.

Previous Next

To reply to this bug, email your comments to 48463 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: guix-patches <at> gnu.org
Subject: gnu: Add j.
Date: Sun, 16 May 2021 19:54:03 +0900

[Message part 1 (text/plain, inline)]

Hey Guix,

This packages up the J programming lanuage. The build scripts that upstream
provides are pretty hostile to package maintainers, so we have to jump through
quite a few hoops to get the build working.

I have had this in my personal, private channel for a while, tweaking it over
time, so I believe it works fairly well in practice. Here are the high-level
design decisions I have made:

1) Provide multiple J versions;

It is extremely common for J users to have multiple versions installed in
parallel since there are often major, breaking changes between
(non-patch-level) releases.

In addition, a very large fraction of J users keep the beta installed as well,
so I have elected to provide this too.

2) Bundle together all variants of the interpreter along with a
wrapper script that selects the most featureful one at runtime;

There are 3 versions of the interpreter, supporting different CPU feature-sets:
vanilla, AVX, AVX2. This package elects to build all these variants and
provides a wrapper script that launches the "most AVX-y" version possible at
runtime.

Essentially, this is trying to make up for upstream's lack of a fat binary.

3) Try to mirror the FHS standard for installation locations;

For the most part, J uses idiosyncratic defaults for where it looks for library
files and the like. This package elects to configure things so files sit in
more standard locations---e.g. runtime scripts sit under .../share/j;
configuration files sit under .../etc/j; etc.

4) Point the J package manager to $HOME/.j/<version>.

J maintains its own mini package repository for "addons", which are simply
officially-supported J scripts. This package sets things up so that these
packages get installed to .j/<version>/addons. In addition, other related
directories are gathered under .j/<version>.


Items on the TODO list:

1) Write an importer and create packages for J addons;

This is currently in progress.

2) Create a something like a jsoftware-union procedure to bundle up J with
user-selectable addons;

This is necessary since the interpreter looks for addons at a specified path.
We probably want to mimick something like texlive and provide both an "all
batteries included" package along with one where users can select which addons
they want.

3) Support more older versions of J.

J is open source starting with j801, so it'd be nice to have packages for the
entire j80x series too. Unfortunately, those use an entirely different set of
build scripts from the current j90x series, so it will require munging-work
similar to what we already have.


Cheers!

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #8 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Leo Prikler <leo.prikler <at> student.tugraz.at>
To: elaexuotee <at> wilsonb.com, 48463 <at> debbugs.gnu.org
Subject: Re: gnu: Add j.
Date: Sun, 16 May 2021 16:30:48 +0200

Hi elaexuotee,

Am Sonntag, den 16.05.2021, 19:54 +0900 schrieb elaexuotee <at> wilsonb.com:
> Hey Guix,
> 
> This packages up the J programming lanuage. The build scripts that
> upstream provides are pretty hostile to package maintainers, so we
> have to jump through quite a few hoops to get the build working.
Yup, that much is visible from the package description.

> I have had this in my personal, private channel for a while, tweaking
> it over time, so I believe it works fairly well in practice. Here are
> the high-level design decisions I have made:
> 
> 1) Provide multiple J versions;
> 
> It is extremely common for J users to have multiple versions
> installed in parallel since there are often major, breaking changes
> between (non-patch-level) releases.
> 
> In addition, a very large fraction of J users keep the beta installed
> as well, so I have elected to provide this too.
That should be fine, we provide multiple versions for other packages
where applicable as well.  Do note, that we should try to keep the
number limited, though.  You may perhaps want to export make-jlib, so
that the user can build their own.

> 2) Bundle together all variants of the interpreter along with a
> wrapper script that selects the most featureful one at runtime;
> 
> There are 3 versions of the interpreter, supporting different CPU
> feature-sets: vanilla, AVX, AVX2. This package elects to build all
> these variants and provides a wrapper script that launches the "most
> AVX-y" version possible at runtime.
> 
> Essentially, this is trying to make up for upstream's lack of a fat
> binary.
Are fat binaries the accepted HPC way?  I'm not up-to-date to this.

> 3) Try to mirror the FHS standard for installation locations;
> 
> For the most part, J uses idiosyncratic defaults for where it looks
> for library files and the like. This package elects to configure
> things so files sit in more standard locations---e.g. runtime scripts
> sit under .../share/j; configuration files sit under .../etc/j; etc.
> 
> 4) Point the J package manager to $HOME/.j/<version>.
> 
> J maintains its own mini package repository for "addons", which are
> simply officially-supported J scripts. This package sets things up so
> that these packages get installed to .j/<version>/addons. In
> addition, other related directories are gathered under .j/<version>.
Those should be fine.

> Items on the TODO list:
> 
> 1) Write an importer and create packages for J addons;
> 
> This is currently in progress.
> 
> 2) Create a something like a jsoftware-union procedure to bundle up J
> with user-selectable addons;
> 
> This is necessary since the interpreter looks for addons at a
> specified path. We probably want to mimick something like texlive and
> provide both an "all batteries included" package along with one where
> users can select which addons they want.
Both sound like good ideas, although I'm a little less sure about the
texlive one.  Would a meta-package like gnome also be acceptable?

> 3) Support more older versions of J.
> 
> J is open source starting with j801, so it'd be nice to have packages
> for the entire j80x series too. Unfortunately, those use an entirely
> different set of build scripts from the current j90x series, so it
> will require munging-work similar to what we already have.
As above, I'd prefer if it was one procedure and one package pointing
to the latest j80x, assuming all of our added patches can also be
applied to earlier versions.

> +(define-record-type* <jlib-build-configuration>
> +  jlib-build-configuration make-jlib-build-configuration
> +  jlib-build-configuration?
> +  (builder      jlib-build-configuration-builder (default
> "guix.gnu.org"))
> +  (version      jlib-build-configuration-version (default #f))
> +  (revision     jlib-build-configuration-revision (default #f))
> +  (hash         jlib-build-configuration-hash (default #f))
> +  (type         jlib-build-configuration-type (default 'release))
> +  (patches      jlib-build-configuration-patches (default '()))
> +  (extra-inputs jlib-build-configuration-extra-inputs (default '()))
> +  (extra-envars jlib-build-configuration-extra-envars (default
> '())))
> +
> +(define make-jlib
> +  (match-lambda
> +    (($ <jlib-build-configuration>
> +        builder jversion revision hash type patches extra-inputs
> extra-envars)
Please try to use keyword arguments.

> +       (properties `((jversion . ,jversion)
> +                     (jrevision . ,revision)
> +                     (jtype . ,type)))
Are those used anywhere?  Can jversion and jrevision not be parsed from
(package-version jlib)?

Regards,
Leo

Message #11 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com, x <at> wilsonb.com
To: Leo Prikler <leo.prikler <at> student.tugraz.at>
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: gnu: Add j.
Date: Mon, 24 May 2021 22:37:41 +0900

Thanks for taking a look!

Leo Prikler <leo.prikler <at> student.tugraz.at> wrote:
> That should be fine, we provide multiple versions for other packages
> where applicable as well.  Do note, that we should try to keep the
> number limited, though.  You may perhaps want to export make-jlib, so
> that the user can build their own.

Cool. I like the idea of exporting the build procedures.

> Are fat binaries the accepted HPC way?  I'm not up-to-date to this.

I believe so? Don't quote me on that though. For this particular package the
overhoad is just ~8MB, so I don't think it's much of an issue either way. It
would probably be more of a hassle to split j into a package for each variant.

> Both sound like good ideas, although I'm a little less sure about the
> texlive one.  Would a meta-package like gnome also be acceptable?

A metapackage is certainly practical. The entire set of J addons currently
weighs in at a whopping 50MB. hehe. However, there are about 125 addons in
total, so it simply feels "more correct" to let the user also choose just the
addons they want, which I think would require something like jsoftware-union.

That said, either way, an "all batteries include" package would be good to
have, and since this is way easier than packaging the addons separately, I'll
definitely work on the metapackage first.

> As above, I'd prefer if it was one procedure and one package pointing
> to the latest j80x, assuming all of our added patches can also be
> applied to earlier versions.

Yeah, that would definitely be ideal.

Unfortunately, it's not so straigthforward. The the "major versions" are the
xxx part of jxxx-y, with large changes between each, e.g. j902 introduced and
non-compatible API change over j901. So, from a pure J user perspective, having
all jxxx versions available is ideal.

However, each version seems to require its own set of build flags and sometimes
version-specific patches (like in the j901 case). We probably don't want to
push those settings out into user manifest specs.

On the other hand, there are already 10 versions from j801 to j903.

> > +(define make-jlib
> > +  (match-lambda
> > +    (($ <jlib-build-configuration>
> > +        builder jversion revision hash type patches extra-inputs
> > extra-envars)
> Please try to use keyword arguments.

Actually, maybe these "build configuration" records could solve the above
"too many versions" problem. We could offer only the latest J (and J beta?)
packages in the repo, but let (gnu packages jsoftware) export a set of "default
configurations" for old versions to be used with the make-j procedure.

Any particular reason to avoid using records though? Currently, its letting us
share configuration options between j902 and j903 that don't work in j901.
Personally, I thought this felt like a nice declarative, scheme-y way to go,
but my Scheme is still very amateurish. Definitely wanting to rectify any
strange ideas I may have.

> > +       (properties `((jversion . ,jversion)
> > +                     (jrevision . ,revision)
> > +                     (jtype . ,type)))
> Are those used anywhere?  Can jversion and jrevision not be parsed from
> (package-version jlib)?

The make-j procedure uses them. We'd have to parse out these from both the
version string and package name with a what's essentially the inverse of the
jname procedure. I found it a lot more readable and less hassle to just
propagate this data directly.

However, if there's a particular reason why using properties is problematic,
I'll try to see what I can do.

Message #14 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Leo Prikler <leo.prikler <at> student.tugraz.at>
To: elaexuotee <at> wilsonb.com, x <at> wilsonb.com
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: gnu: Add j.
Date: Mon, 24 May 2021 16:39:57 +0200

Hi,

Am Montag, den 24.05.2021, 22:37 +0900 schrieb elaexuotee <at> wilsonb.com:
> > Are fat binaries the accepted HPC way?  I'm not up-to-date to this.
> 
> I believe so? Don't quote me on that though. For this particular
> package the
> overhoad is just ~8MB, so I don't think it's much of an issue either
> way. It
> would probably be more of a hassle to split j into a package for each
> variant.
In that case it will probably be fine.

> > Both sound like good ideas, although I'm a little less sure about
> > the
> > texlive one.  Would a meta-package like gnome also be acceptable?
> 
> A metapackage is certainly practical. The entire set of J addons
> currently weighs in at a whopping 50MB. hehe. However, there are
> about 125 addons in total, so it simply feels "more correct" to let
> the user also choose just the addons they want, which I think would
> require something like jsoftware-union.
> 
> That said, either way, an "all batteries include" package would be
> good to have, and since this is way easier than packaging the addons
> separately, I'll definitely work on the metapackage first.
stuff-union usually implies, that stuff can't just be put into a
profile and expected to work (like texlive), whereas in other cases
(e.g. the gnome metapackage) people can use its parts on their own
provided that they have the right combination (which typically implies
having icons etc.)

> > As above, I'd prefer if it was one procedure and one package
> > pointing to the latest j80x, assuming all of our added patches can
> > also be applied to earlier versions.
> 
> Yeah, that would definitely be ideal.
> 
> Unfortunately, it's not so straigthforward. The the "major versions"
> are the xxx part of jxxx-y, with large changes between each, e.g.
> j902 introduced and non-compatible API change over j901. So, from a
> pure J user perspective, having all jxxx versions available is ideal.
> 
> However, each version seems to require its own set of build flags and
> sometimes version-specific patches (like in the j901 case). We
> probably don't want to push those settings out into user manifest
> specs.
> 
> On the other hand, there are already 10 versions from j801 to j903.
That's less than the number of Rust versions we need just for
bootstrap, but still a few too many in my opinion.  Are all of those
still used in production or would it be wiser to to put some of them
into Guix-Past [1]?

> > > +(define make-jlib
> > > +  (match-lambda
> > > +    (($ <jlib-build-configuration>
> > > +        builder jversion revision hash type patches extra-inputs
> > > extra-envars)
> > Please try to use keyword arguments.
> 
> Actually, maybe these "build configuration" records could solve the
> above "too many versions" problem. We could offer only the latest J
> (and J beta?) packages in the repo, but let (gnu packages jsoftware)
> export a set of "default configurations" for old versions to be used
> with the make-j procedure.
I don't think there's much to be gained if we provide packages without
packages.

> Any particular reason to avoid using records though? Currently, its
> letting us share configuration options between j902 and j903 that
> don't work in j901.
> Personally, I thought this felt like a nice declarative, scheme-y way
> to go, but my Scheme is still very amateurish. Definitely wanting to
> rectify any strange ideas I may have.
Having keyword lists is also nice and declarative, but more
importantly, it lets you call the function as a normal function without
having to worry about constructing some record in a particular way. 
The configuration you're using doesn't have a specific setting?  Just
override it with your own.  It's as simple as calling (append old-
config extra-config).

> > > +       (properties `((jversion . ,jversion)
> > > +                     (jrevision . ,revision)
> > > +                     (jtype . ,type)))
> > Are those used anywhere?  Can jversion and jrevision not be parsed
> > from (package-version jlib)?
> 
> The make-j procedure uses them. We'd have to parse out these from
> both the version string and package name with a what's essentially
> the inverse of the jname procedure. I found it a lot more readable
> and less hassle to just propagate this data directly.
> 
> However, if there's a particular reason why using properties is
> problematic, I'll try to see what I can do.
I find this way of mistreating the version field very weird.  You're
not adding anything new by doing this and you're not making anything
more secure (I'd argue, that it's less secure, because you could update
the version and forget about the property or vice versa).

First of all, do you even need all this info?  `j' is an objectively
bad name for a package.  `j-beta' is not better in any way, it only
avoids the user installing a potentially unstable "J" package.  (Note,
that we typically use "-next" for that, however).  However, this info
is meaningless when hardcoded into a procedure.  You can just inherit
from the package that's generated and override the name as needed.

Next, should ijconsole not simply be a package like jlib (both properly
prefixed with jsoftware- of course)?  Then you can take whatever
version bits you need from the package version itself.

Regards,
Leo

[1] https://gitlab.inria.fr/guix-hpc/guix-past

Message #17 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Leo Prikler <leo.prikler <at> student.tugraz.at>
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: gnu: Add j.
Date: Tue, 25 May 2021 12:57:36 +0900

As always, thanks for the quick turnaround.

Leo Prikler <leo.prikler <at> student.tugraz.at> wrote:
> stuff-union usually implies, that stuff can't just be put into a
> profile and expected to work (like texlive), whereas in other cases
> (e.g. the gnome metapackage) people can use its parts on their own
> provided that they have the right combination (which typically implies
> having icons etc.)

Oh, okay.  For some reason, I was just imagining "metapackage" to mean "all
batteries included", but your description makes a lot more sense. In this
case, having addons without the J interpreter is pretty useless, so IIUC a
jsoftware-union makes more sense than a metapackage in this case?

Just to be clear, J looks for addons at a path burned into
$store/...-jlib-$version/etc/j/profilex.ijs.  Currently, that points to
$HOME/.j/$jversion/addons, but we'll need to change that to a single store
output path that contains the union of all desired addons, hence why I was
thinking the *-union approach is needed.

For now, instead of messing with the union, would it make sense to just create
a (non-public?) jsoftware-addons package that contains all addons and point
jlib at those?

> That's less than the number of Rust versions we need just for
> bootstrap, but still a few too many in my opinion.  Are all of those
> still used in production or would it be wiser to to put some of them
> into Guix-Past [1]?

Oh cool. Guix Past might be a better solution! Thanks for sharing.

I don't have any solid data, since "in production" J is mostly dominated by the
financial sector from what I hear, but just going by my impression from lurking
on the J mailing list, the j80x series (and even earlier non-free versions)
does still have a user base.

Personally, I just want to make these older versions available to the
community but am agnostic about the *how* so will defer to whatever you (and
others) think is best.

> Having keyword lists is also nice and declarative, but more
> importantly, it lets you call the function as a normal function without
> having to worry about constructing some record in a particular way. 
> The configuration you're using doesn't have a specific setting?  Just
> override it with your own.  It's as simple as calling (append old-
> config extra-config).

Oh! (append old-config extra-config) is exactly the kind of thing I was trying
to achieve; however, I'm not sure I quite grok the usage you are describing.
Something like this?

(define* (proc #:key foo bar) ...)
(define old-config '(#:foo 0))
(define extra-config '(#:bar 1))
(apply proc (append old-config extra-config))

> I find this way of mistreating the version field very weird.  You're
> not adding anything new by doing this and you're not making anything
> more secure (I'd argue, that it's less secure, because you could update
> the version and forget about the property or vice versa).

I agree the version stuff is a bit crappy.  It's something that got munged
badly over time and deserves more thought.  Here are the issues I'm trying to
solve:

  1) The source URLs look like ../j<version>-<type>-<patch> or
     ../j<version>-<type>, where <type> is either "beta" or "release", and
     depending on whether a patch-level exists;
  2) The type string ("release" or "beta") is a build setting; and
  3) ijconsole installs to something like ../bin/ijconsole-902, without
     including the patch-level, like ../bin/ijconsole-902-b.
  4) Users should probably be able to easily install the latest J release as
     well as J beta concurrently without caring about exactly which version
     these are.

Those imply that <version>, <type>, and <patch> are indeed handled separately,
no matter concrete method utilized.

My thinking was that 4 means we want a separate *-beta package, meaning that
the version field should probably just look like "<version>-<patch>".
Additionally, relating back to the above discussion about multiple versions

   5) It would be nice to be able to install the latest j901 without having to
      know that this corresponds to patch level f, i.e. currently j <at> 901-f.

> First of all, do you even need all this info?  `j' is an objectively bad name
> for a package.  `j-beta' is not better in any way, it only avoids the user
> installing a potentially unstable "J" package.  

Yeah, "j" is pretty short but it does mirror "r".  I'm agnostic about package
name, though.  Suggestions welcome.

> (Note, that we typically use "-next" for that, however).

Thanks for pointing out this convention.

> However, this info is meaningless when hardcoded into a procedure.  You can
> just inherit from the package that's generated and override the name as
> needed.

I see what you mean in a general sense but not quite how to apply it in this
case. I am all ears if you have a simpler, cleaner solution that addresses the
constraints listed above!

> Next, should ijconsole not simply be a package like jlib (both properly
> prefixed with jsoftware- of course)?

This is effectively what make-j does, no? ijconsole needs to know the path to
jlib, so make-j points it at the correct one and wraps everything up into a
package.


Anyway, thanks again for taking a look!

Message #22 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: gnu: Add j.
Date: Fri, 01 Oct 2021 17:31:11 +0200

Hi,

Am Dienstag, den 25.05.2021, 12:57 +0900 schrieb
elaexuotee <at> wilsonb.com:
> As always, thanks for the quick turnaround.
> 
> Leo Prikler <leo.prikler <at> student.tugraz.at> wrote:
> > stuff-union usually implies, that stuff can't just be put into a
> > profile and expected to work (like texlive), whereas in other cases
> > (e.g. the gnome metapackage) people can use its parts on their own
> > provided that they have the right combination (which typically
> > implies having icons etc.)
> 
> Oh, okay.  For some reason, I was just imagining "metapackage" to
> mean "all batteries included", but your description makes a lot more
> sense. In this case, having addons without the J interpreter is
> pretty useless, so IIUC a jsoftware-union makes more sense than a
> metapackage in this case?
> 
> Just to be clear, J looks for addons at a path burned into
> $store/...-jlib-$version/etc/j/profilex.ijs.  Currently, that points
> to $HOME/.j/$jversion/addons, but we'll need to change that to a
> single store output path that contains the union of all desired
> addons, hence why I was thinking the *-union approach is needed.
> 
> For now, instead of messing with the union, would it make sense to
> just create a (non-public?) jsoftware-addons package that contains
> all addons and point jlib at those?
Are those addons data packages or can they be compiled without needing
jlib?  If they're pure data, then you can pack them up and refer them
as you wanted to, if not, you first have to create jlib-minimal without
them, then package up everything and finally do jlib.

Oh, and one thing that was previously ignored is that using XDG
standards for things we can't simply put in the store (like
configuration if it needs that) is preferable to having a ~/.j
directory.

> > That's less than the number of Rust versions we need just for
> > bootstrap, but still a few too many in my opinion.  Are all of
> > those still used in production or would it be wiser to to put some
> > of them into Guix-Past [1]?
> 
> Oh cool. Guix Past might be a better solution! Thanks for sharing.
> 
> I don't have any solid data, since "in production" J is mostly
> dominated by the financial sector from what I hear, but just going by
> my impression from lurking on the J mailing list, the j80x series
> (and even earlier non-free versions) does still have a user base.
> 
> Personally, I just want to make these older versions available to the
> community but am agnostic about the *how* so will defer to whatever
> you (and others) think is best.
People are also still running ancient versions of Debian.  Python 2 has
officially reached end of life, yet it is used as well.  At some point
we do have to declare software that people are still using as old :)

> > Having keyword lists is also nice and declarative, but more
> > importantly, it lets you call the function as a normal function
> > without having to worry about constructing some record in a
> > particular way.  The configuration you're using doesn't have a
> > specific setting?  Just override it with your own.  It's as simple
> > as calling (append old-config extra-config).
> 
> Oh! (append old-config extra-config) is exactly the kind of thing I
> was trying
> to achieve; however, I'm not sure I quite grok the usage you are
> describing.
> Something like this?
> 
> (define* (proc #:key foo bar) ...)
> (define old-config '(#:foo 0))
> (define extra-config '(#:bar 1))
> (apply proc (append old-config extra-config))
Yup.

> > I find this way of mistreating the version field very
> > weird.  You're not adding anything new by doing this and you're not
> > making anything more secure (I'd argue, that it's less secure,
> > because you could update the version and forget about the property
> > or vice versa).
> 
> I agree the version stuff is a bit crappy.  It's something that got
> munged badly over time and deserves more thought.  Here are the
> issues I'm trying to solve:
> 
>   1) The source URLs look like ../j<version>-<type>-<patch> or
>      ../j<version>-<type>, where <type> is either "beta" or
> "release", and
>      depending on whether a patch-level exists;
>   2) The type string ("release" or "beta") is a build setting; and
>   3) ijconsole installs to something like ../bin/ijconsole-902,
> without
>      including the patch-level, like ../bin/ijconsole-902-b.
>   4) Users should probably be able to easily install the latest J
> release as
>      well as J beta concurrently without caring about exactly which
> version
>      these are.
> 
> Those imply that <version>, <type>, and <patch> are indeed handled
> separately, no matter concrete method utilized.
Those four issues do call for separate keyword arguments, but they need
not necessarily be reflected in the version string.  You can have two
packages with the same version, but different hashes in Guix.  

> My thinking was that 4 means we want a separate *-beta package,
> meaning that the version field should probably just look like
> "<version>-<patch>".
> Additionally, relating back to the above discussion about multiple
> versions
Sounds good, though <version>.<patch> would be equally acceptable. 
Your call.

>    5) It would be nice to be able to install the latest j901 without
> having to
>       know that this corresponds to patch level f, i.e. currently 
> j <at> 901-f.
I think `guix build j <at> 901' ought to resolve this automatically.

> > First of all, do you even need all this info?  `j' is an
> > objectively bad name
> > for a package.  `j-beta' is not better in any way, it only avoids
> > the user installing a potentially unstable "J" package.  
> 
> Yeah, "j" is pretty short but it does mirror "r".  I'm agnostic about
> package name, though.  Suggestions welcome.
Fair enough, though this remains a problem for single letter programs
in which others would likely want to claim the letter as well.  R has
the benefit of being old and well-known enough, it would probably be
rlang otherwise (not to be confused with erlang).

> > (Note, that we typically use "-next" for that, however).
> 
> Thanks for pointing out this convention.
> 
> > However, this info is meaningless when hardcoded into a
> > procedure.  You can just inherit from the package that's generated
> > and override the name as needed.
> 
> I see what you mean in a general sense but not quite how to apply it
> in this case. I am all ears if you have a simpler, cleaner solution
> that addresses the constraints listed above!
I think the cleaner solution here is to not store those properties in
the package, but still having them as parameters to the make-j function
call.

> > Next, should ijconsole not simply be a package like jlib (both
> > properly prefixed with jsoftware- of course)?
> 
> This is effectively what make-j does, no? ijconsole needs to know the
> path to jlib, so make-j points it at the correct one and wraps
> everything up into a package.
I don't really understand why those needs to be done in two steps
however.  ijconsole could already contain the fully functioning j
program.  Or partially functioning if we account for addons.

Cheers!

Message #25 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: gnu: Add j.
Date: Wed, 12 Jan 2022 18:47:52 +0900

[Message part 1 (text/plain, inline)]

Here's a long-needed update.

> > For now, instead of messing with the union, would it make sense to
> > just create a (non-public?) jsoftware-addons package that contains
> > all addons and point jlib at those?
>
> Are those addons data packages or can they be compiled without needing
> jlib?  If they're pure data, then you can pack them up and refer them
> as you wanted to, if not, you first have to create jlib-minimal without
> them, then package up everything and finally do jlib.

They're pure data, which is nice.
The J package manager also lets users install arbitrary addons from GitHub, so
packaging up the official addons would lock the `addons/' directory under the
non-writable store, meaning that we probably also want an importer.

I am working on that, but instead of keeping this package in lingo much longer,
maybe we can release sonner and consider the addon package code a separate
issue?

> Oh, and one thing that was previously ignored is that using XDG
> standards for things we can't simply put in the store (like
> configuration if it needs that) is preferable to having a ~/.j
> directory.

Excellent point. Fixed.

> > > That's less than the number of Rust versions we need just for
> > > bootstrap, but still a few too many in my opinion.  Are all of
> > > those still used in production or would it be wiser to to put some
> > > of them into Guix-Past [1]?
>
> People are also still running ancient versions of Debian.  Python 2 has
> officially reached end of life, yet it is used as well.  At some point
> we do have to declare software that people are still using as old :)

Sounds sane.

I will probably eventually work on getting the j80x series in Guix and the
really ancient ones in Guix Past. However, that will be a future task for
myself.

> > > Having keyword lists is also nice and declarative, but more
> > > importantly, it lets you call the function as a normal function
> > > without having to worry about constructing some record in a
> > > particular way.  The configuration you're using doesn't have a
> > > specific setting?  Just override it with your own.  It's as simple
> > > as calling (append old-config extra-config).
> > ...
> Yup.

Now using keyword lists.

> > > I find this way of mistreating the version field very
> > > weird.  You're not adding anything new by doing this and you're not
> > > making anything more secure (I'd argue, that it's less secure,
> > > because you could update the version and forget about the property
> > > or vice versa).

The code overall is a lot more straightforward now. How does the versioning
stuff look now?

> > My thinking was that 4 means we want a separate *-beta package,
> > meaning that the version field should probably just look like
> > "<version>-<patch>".
> > Additionally, relating back to the above discussion about multiple
> > versions
> Sounds good, though <version>.<patch> would be equally acceptable. 
> Your call.
>
> >    5) It would be nice to be able to install the latest j901 without
> > having to
> >       know that this corresponds to patch level f, i.e. currently 
> > j <at> 901-f.
> I think `guix build j <at> 901' ought to resolve this automatically.

It turns out that the j <at> 901 specification only gets recognized with versions
strings of the form <version>.<patch>, so going with that.

> > > First of all, do you even need all this info?  `j' is an
> > > objectively bad name
> > > for a package.  `j-beta' is not better in any way, it only avoids
> > > the user installing a potentially unstable "J" package.  
> > 
> > Yeah, "j" is pretty short but it does mirror "r".  I'm agnostic about
> > package name, though.  Suggestions welcome.
> Fair enough, though this remains a problem for single letter programs
> in which others would likely want to claim the letter as well.  R has
> the benefit of being old and well-known enough, it would probably be
> rlang otherwise (not to be confused with erlang).

J has been around from the 1990's :D
That said, I went ahead and named it `jsoftware-j', since despite age, J is
still pretty obscure, and there's not much of a good reason to monopolize a
single letter name.

> > I see what you mean in a general sense but not quite how to apply it
> > in this case. I am all ears if you have a simpler, cleaner solution
> > that addresses the constraints listed above!
> I think the cleaner solution here is to not store those properties in
> the package, but still having them as parameters to the make-j function
> call.

Happily, the new package re-organization was able to get rid of these
properties.

> > > Next, should ijconsole not simply be a package like jlib (both
> > > properly prefixed with jsoftware- of course)?
> > 
> > This is effectively what make-j does, no? ijconsole needs to know the
> > path to jlib, so make-j points it at the correct one and wraps
> > everything up into a package.
> I don't really understand why those needs to be done in two steps
> however.  ijconsole could already contain the fully functioning j
> program.  Or partially functioning if we account for addons.

As you can see, there's no need for a separate ijconsole package any more.
Instead, `ijconsole' is just defined as a program-file that becomes an
input to jsoftware-j.


It is also worth pointing out the convenience procedure
`jsoftware-ijconsole-symlink'. Basically, it p


Anyway, let me know what you think!

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #28 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: elaexuotee <at> wilsonb.com, Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Wed, 12 Jan 2022 11:06:50 +0000

[Message part 1 (text/plain, inline)]

elaexuotee--- via Guix-patches via schreef op wo 12-01-2022 om 18:47 [+0900]:
+                  (if ,(target-64bit?)
+                    (parallel
+                      (invoke "env" jplat jbit-avx "./build_libj.sh")
+                      (invoke "env" jplat jbit-avx2
+                              "./build_libj.sh"))))))))

I don't think avx and avx2 are available on aarch64.
Maybe you can use ,(target-x86-64?) instead?

> +    `(#:tests? #f
>          [...]
> +        (replace 'check
> +          (lambda* (#:key tests? #:allow-other-keys)
> +            (when tests? [...]))))

I think #:tests? #false should be removed, otherwise this
check phase would never do anything.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #31 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: elaexuotee <at> wilsonb.com, Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Wed, 12 Jan 2022 11:10:27 +0000

[Message part 1 (text/plain, inline)]

Hi,
> +                ;; The jlibrary/dev directory only sometimes exists,
> but needs
> +                ;; to be copied into the ~system directory when it
> does.
> +                (for-each
> +                  (lambda (dev-dir)
> +                    (if (access? dev-dir R_OK)
> +                      (copy-recursively dev-dir
> "test/bin/system/dev")))


Are you testing for file permissions, or for the existence of the file?
If the latter, I'd recommend using 'file-exists?' instead.

> +                  (if (access? source-dev R_OK)


Likewise.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #34 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Maxime Devos <maximedevos <at> telenet.be>
Cc: Liliana Marie Prikler <liliana.prikler <at> gmail.com>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Wed, 12 Jan 2022 21:07:45 +0900

[Message part 1 (text/plain, inline)]

> > +    `(#:tests? #f
> >          [...]
> > +        (replace 'check
> > +          (lambda* (#:key tests? #:allow-other-keys)
> > +            (when tests? [...]))))
> 
> I think #:tests? #false should be removed, otherwise this
> check phase would never do anything.

Nice catch! Thanks. Disabled tests when tweaking builds and forgot to remove.

> > +                ;; The jlibrary/dev directory only sometimes exists,
> > but needs
> > +                ;; to be copied into the ~system directory when it
> > does.
> > +                (for-each
> > +                  (lambda (dev-dir)
> > +                    (if (access? dev-dir R_OK)
> > +                      (copy-recursively dev-dir
> > "test/bin/system/dev")))
> 
> 
> Are you testing for file permissions, or for the existence of the file?
> If the latter, I'd recommend using 'file-exists?' instead.
> 
> > +                  (if (access? source-dev R_OK)

Just existence. Thanks for the pointer. Changed.

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #37 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com, Maxime Devos <maximedevos <at> telenet.be>
Cc: 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Wed, 12 Jan 2022 20:55:21 +0100

Hi,

> +(define-module (gnu packages jsoftware)
> +  #:use-module ((guix build utils))
> + [...]

Double bracketing is pointless, use it only when needed.

> +(define* (make-j #:key
> +                 (builder "guix.gnu.org")
> +                 vername
> +                 revision
> +                 hash
> +                 (type 'release)
> +                 commit
> +                 (patches '())
> +                 (extra-inputs '())
> +                 (extra-envars '()))
> + (package
> +   (name (jname "jsoftware-j" type))
> +   (version (jversion->string vername revision))
> +   (source
> +    (origin
> +      (method git-fetch)
> +      (uri (git-reference
> +            (url "https://github.com/jsoftware/jsource")
> +            (commit (or commit (jinfo->git-tag vername type
> revision))))
Vername sounds a little weird, make that version-base or something
clearer.  Also, the argument #:commit is used in an unclear fashion --
if you were to pass an actual commit hash to it, it'd still be treated
as a release and not be using git-version. 

On a related note
> +(define (jversion->string version revision)
> +  "Return a string representation of a J version and (optional)
> revision pair."
> +  (let ((postfix (if (not revision) ""
> +                   (string-append "." revision))))
> +    (string-append version postfix)))
should also take commit and revision should probably be dashed.  In
that way, when packaging commits between releases we can use
"jrevision.guix-revision" as the complete revision.

In short, I'd add a #:tag argument to override the tag and treat commit
like a let-bound commit.

> +    `(#:modules (((ice-9 ftw) #:select (scandir))
> +                 ((ice-9 popen) #:select (open-pipe* close-pipe))
> +                 ((ice-9 regex) #:select (match:substring string-
> match))
> +                 ((ice-9 threads) #:select (parallel par-for-each))
> +                 ((srfi srfi-26) #:select (cut))
> +                 ((srfi srfi-1) #:select (fold))
> +                 ,@%gnu-build-system-modules)
It's nice that you annotated all those, but note that it probably
wouldn't have been needed.  If you notice this list getting longer and
longer as you update, consider dropping the #:selects.

> +        (replace 'build
> +          (lambda _
> +            (setenv "USE_OPENMP" "1")
> +            (setenv "USE_THREAD" "1")
> +            (for-each (lambda (var-val) (apply setenv var-val))
> +                      (quote ,extra-envars))
> +            ;; The build scripts assume that PWD is make2.
> +            (with-directory-excursion "make2"
> +              (let* ((platform ,(if (target-arm?) "raspberry"
> "linux"))
> +                     (jplat (string-append "jplatform=" platform))
> +                     (target-bit ,(if (target-64bit?) "64" "32"))
> +                     (jbit (string-append "j64x=" "j" target-bit))
> +                     (jbit-avx (string-append jbit "avx"))
> +                     (jbit-avx2 (string-append jbit "avx2")))
> +                (parallel
> +                  ;; Since jconsole doesn't depend on AVX features,
> we just
> +                  ;; build it once.
> +                  (invoke "env" jplat jbit "./build_jconsole.sh")
> +                  (invoke "env" jplat jbit "./build_libj.sh")
> +                  (if ,(target-64bit?)
> +                    (parallel
> +                      (invoke "env" jplat jbit-avx
> "./build_libj.sh")
> +                      (invoke "env" jplat jbit-avx2
> +                              "./build_libj.sh"))))))))
Maxime already made a comment w.r.t. 32bit AVX here, but I think this
would be a prime example to use the CPU tuning that was recently
landed.  

Most of the above (except the semantics of the make-j keyword
arguments) are not blockers in my opinion.

Cheers

Message #40 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Thu, 13 Jan 2022 16:51:08 +0900

[Message part 1 (text/plain, inline)]

> > +(define-module (gnu packages jsoftware)
> > +  #:use-module ((guix build utils))
> > + [...]
> 
> Double bracketing is pointless, use it only when needed.

Ah, nice catch. I had a bunch of #:select keys and forgot to kill the parens
when removing.

> > +(define* (make-j #:key
> > +                 (builder "guix.gnu.org")
> > +                 vername
> > +                 revision
> > +                 hash
> > +                 (type 'release)
> > +                 commit
> > +                 (patches '())
> > +                 (extra-inputs '())
> > +                 (extra-envars '()))
> > + (package
> > +   (name (jname "jsoftware-j" type))
> > +   (version (jversion->string vername revision))
> > +   (source
> > +    (origin
> > +      (method git-fetch)
> > +      (uri (git-reference
> > +            (url "https://github.com/jsoftware/jsource")
> > +            (commit (or commit (jinfo->git-tag vername type
> > revision))))
> Vername sounds a little weird, make that version-base or something
> clearer.  Also, the argument #:commit is used in an unclear fashion --
> if you were to pass an actual commit hash to it, it'd still be treated
> as a release and not be using git-version. 

Cool. I had a similar sense, but our ideas are a lot sharper than the ones I
had. This actually prompted me to do some code cleanup, leveraging now-me who
has a bit more Guile experience than past-me. At the very least, variable names
should be more descriptive and consistent, overall.

> On a related note
> > +(define (jversion->string version revision)
> > +  "Return a string representation of a J version and (optional)
> > revision pair."
> > +  (let ((postfix (if (not revision) ""
> > +                   (string-append "." revision))))
> > +    (string-append version postfix)))
> should also take commit and revision should probably be dashed.  In
> that way, when packaging commits between releases we can use
> "jrevision.guix-revision" as the complete revision.
> 
> In short, I'd add a #:tag argument to override the tag and treat commit
> like a let-bound commit.

Done.

> > +    `(#:modules (((ice-9 ftw) #:select (scandir))
> > +                 ((ice-9 popen) #:select (open-pipe* close-pipe))
> > +                 ((ice-9 regex) #:select (match:substring string-
> > match))
> > +                 ((ice-9 threads) #:select (parallel par-for-each))
> > +                 ((srfi srfi-26) #:select (cut))
> > +                 ((srfi srfi-1) #:select (fold))
> > +                 ,@%gnu-build-system-modules)
> It's nice that you annotated all those, but note that it probably
> wouldn't have been needed.  If you notice this list getting longer and
> longer as you update, consider dropping the #:selects.
> 
> > +        (replace 'build
> > +          (lambda _
> > +            (setenv "USE_OPENMP" "1")
> > +            (setenv "USE_THREAD" "1")
> > +            (for-each (lambda (var-val) (apply setenv var-val))
> > +                      (quote ,extra-envars))
> > +            ;; The build scripts assume that PWD is make2.
> > +            (with-directory-excursion "make2"
> > +              (let* ((platform ,(if (target-arm?) "raspberry"
> > "linux"))
> > +                     (jplat (string-append "jplatform=" platform))
> > +                     (target-bit ,(if (target-64bit?) "64" "32"))
> > +                     (jbit (string-append "j64x=" "j" target-bit))
> > +                     (jbit-avx (string-append jbit "avx"))
> > +                     (jbit-avx2 (string-append jbit "avx2")))
> > +                (parallel
> > +                  ;; Since jconsole doesn't depend on AVX features,
> > we just
> > +                  ;; build it once.
> > +                  (invoke "env" jplat jbit "./build_jconsole.sh")
> > +                  (invoke "env" jplat jbit "./build_libj.sh")
> > +                  (if ,(target-64bit?)
> > +                    (parallel
> > +                      (invoke "env" jplat jbit-avx
> > "./build_libj.sh")
> > +                      (invoke "env" jplat jbit-avx2
> > +                              "./build_libj.sh"))))))))
> Maxime already made a comment w.r.t. 32bit AVX here, but I think this
> would be a prime example to use the CPU tuning that was recently
> landed.  

Good idea. Upstream's build scripts condition a *lot* of behaviour on the
j64avx environment variable, so it might not be straightforward, but I will put
this on the to-do list of future improvements.

(Note, the code block quoted here got much simplified in the current patch.)

> Most of the above (except the semantics of the make-j keyword
> arguments) are not blockers in my opinion.

Cheers!

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #43 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Thu, 13 Jan 2022 18:51:14 +0100

[Message part 1 (text/plain, inline)]

Hi,

Am Donnerstag, dem 13.01.2022 um 16:51 +0900 schrieb
elaexuotee <at> wilsonb.com:
> > 
> > > +(define* (make-j #:key
> > > +                 (builder "guix.gnu.org")
> > > +                 vername
> > > +                 revision
> > > +                 hash
> > > +                 (type 'release)
> > > +                 commit
> > > +                 (patches '())
> > > +                 (extra-inputs '())
> > > +                 (extra-envars '()))
> > > + (package
> > > +   (name (jname "jsoftware-j" type))
> > > +   (version (jversion->string vername revision))
> > > +   (source
> > > +    (origin
> > > +      (method git-fetch)
> > > +      (uri (git-reference
> > > +            (url "https://github.com/jsoftware/jsource")
> > > +            (commit (or commit (jinfo->git-tag vername type
> > > revision))))
> > Vername sounds a little weird, make that version-base or something
> > clearer.  Also, the argument #:commit is used in an unclear fashion
> > --
> > if you were to pass an actual commit hash to it, it'd still be
> > treated
> > as a release and not be using git-version. 
> 
> Cool. I had a similar sense, but our ideas are a lot sharper than the
> ones I had. This actually prompted me to do some code cleanup,
> leveraging now-me who has a bit more Guile experience than past-me.
> At the very least, variable names should be more descriptive and
> consistent, overall.
Note, that the current version still ignores commit.  I've attached a
proof of concept patch, which fixes that, plus some other minor things,
but it now uses a pair for version.  Also it's untested, so treat with
care.


> Good idea. Upstream's build scripts condition a *lot* of behaviour on
> the j64avx environment variable, so it might not be straightforward,
> but I will put this on the to-do list of future improvements.
> 
> (Note, the code block quoted here got much simplified in the current
> patch.)
Reading /proc/cpuinfo is not nice.  Please use (guix cpu), which also
provides you the flags that you need.

Cheers

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #46 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Sat, 15 Jan 2022 23:05:24 +0900

[Message part 1 (text/plain, inline)]

Thanks for staying on top of this patch review.

Liliana Marie Prikler <liliana.prikler <at> gmail.com> wrote:
> Hi,
> 
> Note, that the current version still ignores commit.  I've attached a
> proof of concept patch, which fixes that, plus some other minor things,
> but it now uses a pair for version.  Also it's untested, so treat with
> care.

Oh! I see. You're wanting the version string to have optional REVISION and
COMMIT parts, like `git-version'. That makes a lot of sense.

There was a small issue with `jversion->string' in your patch, which produces
MAJOR-REVISION-COMMIT if no MINOR value is provided. I want to make sure that
users can always have specifications like "jsoftware-j <at> MAJOR" in their profile
manifest, but guix doesn't treat hyphen as a version separator.

I just changed the logic to always keep a dot after MAJOR, and then use hyphens
after that.

> > Good idea. Upstream's build scripts condition a *lot* of behaviour on
> > the j64avx environment variable, so it might not be straightforward,
> > but I will put this on the to-do list of future improvements.
> > 
> > (Note, the code block quoted here got much simplified in the current
> > patch.)
> Reading /proc/cpuinfo is not nice.  Please use (guix cpu), which also
> provides you the flags that you need.

Oh nice. A new, shiny module! This is exactly what I was looking for when
originally writing the script. Updated.

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #49 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Sat, 15 Jan 2022 16:08:13 +0100

Hi,

Am Samstag, dem 15.01.2022 um 23:05 +0900 schrieb
elaexuotee <at> wilsonb.com:
> Thanks for staying on top of this patch review.
> 
> Liliana Marie Prikler <liliana.prikler <at> gmail.com> wrote:
> > Hi,
> > 
> > Note, that the current version still ignores commit.  I've attached
> > a proof of concept patch, which fixes that, plus some other minor
> > things, but it now uses a pair for version.  Also it's untested, so
> > treat with care.
> 
> Oh! I see. You're wanting the version string to have optional
> REVISION and COMMIT parts, like `git-version'. That makes a lot of
> sense.
> 
> There was a small issue with `jversion->string' in your patch, which
> produces MAJOR-REVISION-COMMIT if no MINOR value is provided. I want
> to make sure that users can always have specifications like
> "jsoftware-j <at> MAJOR" in their profile manifest, but guix doesn't treat
> hyphen as a version separator.
> 
> I just changed the logic to always keep a dot after MAJOR, and then
> use hyphens after that.
I'm not quite sure if I agree with that decision, but you're right that
@MAJOR ought to be supported.  How about enforcing that MINOR exists if
REVISION and COMMIT are used and setting it to "0" if there hasn't been
an "a" beta or release yet?

> > 
> > > Good idea. Upstream's build scripts condition a *lot* of
> > > behaviour on he j64avx environment variable, so it might not be
> > > straightforward, but I will put this on the to-do list of future
> > > improvements.
> > > 
> > > (Note, the code block quoted here got much simplified in the
> > > current patch.)
> > Reading /proc/cpuinfo is not nice.  Please use (guix cpu), which
> > also provides you the flags that you need.
> 
> Oh nice. A new, shiny module! This is exactly what I was looking for
> when originally writing the script. Updated.
LGTM.

> +   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
> +   (inputs (cons* `("libedit" ,libedit)
> +                  `("libomp" ,libomp)
> +                  `("ijconsole" ,(ijconsole))
> +                  extra-inputs))
My variant already had these translated to the new style.  If you're
not using old style on purpose – and I don't think you do – you might
want to make your life easier by dropping these labels and going with
plain (list)s.

> +              (call-with-output-file "jlibrary/bin/profilex.ijs"
> +                (lambda (port)
> +                  (display
> +                    (string-join
> +                      (list
> +                        "share=. '/share/j',~ ({.~ _2 { I.@:=&'/')
> BINPATH"
> +                        "system=. share,'/system'"
> +                        "tools=. share,'/tools'"
> +                        ;; Upstream defaults to spamming $HOME with
> unhidden
> +                        ;; userdata directories.  Set this to be
> +                        ;; $HOME/.config/j/<jversion> instead
> +                        "'jtype jversion'=. (3&{,{.) <;._2 ,&'/'
> 9!:14''"
> +                        "jversion=. ({.~ i.&'-') jversion"
> +                        "user=. home,'/.config/j/',jversion"
> +                        "addons=. user,'/addons'"
> +                        "break=. user,'/break'"
> +                        "config=. user,'/config'"
> +                        "install=. user,'/install'"
> +                        "snap=. user,'/snap'"
> +                        "temp=. user,'/temp'"
> +                        "\n")
> +                      "\n")
> +                    port)))
You might want to use an aux-file for that.  Name it
jsoftware/profile.ijs if it's just a plain file (which I assume).  I
recently pushed a commit towards renpy which replaces a large format
block by fetching an auxiliary file and substitute*, which you can take
as reference if you're unsure.  Note that renpy still uses old-style
inputs, so the assoc-ref makes sense in that case; however, you should
be able to also express this pattern in terms of search-input-file
somehow (or otherwise express it as gexp).

Cheers

Message #52 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Sun, 16 Jan 2022 14:29:58 +0900

[Message part 1 (text/plain, inline)]

Good day.

> > I just changed the logic to always keep a dot after MAJOR, and then
> > use hyphens after that.
> I'm not quite sure if I agree with that decision, but you're right that
> @MAJOR ought to be supported.  How about enforcing that MINOR exists if
> REVISION and COMMIT are used and setting it to "0" if there hasn't been
> an "a" beta or release yet?

Interesting idea. How about just always forcing a MINOR part, setting to "0" if
upstream doesn't have one?

> > +   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
> > +   (inputs (cons* `("libedit" ,libedit)
> > +                  `("libomp" ,libomp)
> > +                  `("ijconsole" ,(ijconsole))
> > +                  extra-inputs))
> My variant already had these translated to the new style.  If you're
> not using old style on purpose – and I don't think you do – you might
> want to make your life easier by dropping these labels and going with
> plain (list)s.

Yeah, I had trouble getting that to work nicely with the ijconsole input. See
below for more details.

> You might want to use an aux-file for that.  Name it
> jsoftware/profile.ijs if it's just a plain file (which I assume).  I
> recently pushed a commit towards renpy which replaces a large format
> block by fetching an auxiliary file and substitute*, which you can take
> as reference if you're unsure.  Note that renpy still uses old-style
> inputs, so the assoc-ref makes sense in that case; however, you should
> be able to also express this pattern in terms of search-input-file
> somehow (or otherwise express it as gexp).

Oooh. Neat. That makes adding and testing changes their much nicer.

Note, I wasn't able to find a nice way to get this working together with the
new inputs syntax, since `search-input-files' only searches under input paths
which are directories.

The ijconsole and profilex input are regular files, so it raises a search-error.
One workaronud is to put ijconsole and profilex together under a file-union
input, but I found that almost worse than just using old-style syntax.

If you have a better idea, I am all ears.


Cheers!

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #55 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Sun, 16 Jan 2022 09:04:06 +0100

Am Sonntag, dem 16.01.2022 um 14:29 +0900 schrieb
elaexuotee <at> wilsonb.com:
> Good day.
> 
> > 
> > > I just changed the logic to always keep a dot after MAJOR, and
> > > then use hyphens after that.
> > I'm not quite sure if I agree with that decision, but you're right
> > that @MAJOR ought to be supported.  How about enforcing that MINOR
> > exists if REVISION and COMMIT are used and setting it to "0" if
> > there hasn't been an "a" beta or release yet?
> 
> Interesting idea. How about just always forcing a MINOR part, setting
> to "0" if upstream doesn't have one?
That'd declare regular releases as MAJOR.0 in the version field, which
I'm not sure if we want that.  In the case of random commits I'm less
reserved, as they don't correspond to releases anyway.

> > 
> > > +   (native-inputs `(("clang-toolchain" ,clang-toolchain)))
> > > +   (inputs (cons* `("libedit" ,libedit)
> > > +                  `("libomp" ,libomp)
> > > +                  `("ijconsole" ,(ijconsole))
> > > +                  extra-inputs))
> > My variant already had these translated to the new style.  If
> > you're not using old style on purpose – and I don't think you do –
> > you might want to make your life easier by dropping these labels
> > and going with plain (list)s.
> 
> Yeah, I had trouble getting that to work nicely with the ijconsole
> input. See below for more details.
> 
> > You might want to use an aux-file for that.  Name it
> > jsoftware/profile.ijs if it's just a plain file (which I assume). 
> > I recently pushed a commit towards renpy which replaces a large
> > format block by fetching an auxiliary file and substitute*, which
> > you can take as reference if you're unsure.  Note that renpy still
> > uses old-style inputs, so the assoc-ref makes sense in that case;
> > however, you should be able to also express this pattern in terms
> > of search-input-file somehow (or otherwise express it as gexp).
> 
> Oooh. Neat. That makes adding and testing changes their much nicer.
> 
> Note, I wasn't able to find a nice way to get this working together
> with the new inputs syntax, since `search-input-files' only searches
> under input paths which are directories.
> 
> The ijconsole and profilex input are regular files, so it raises a
> search-error.  One workaronud is to put ijconsole and profilex
> together under a file-union input, but I found that almost worse than
> just using old-style syntax.
> 
> If you have a better idea, I am all ears.
You could define that file-union right after ijconsole.  If you want to
golf even more, you could define ijconsole inside that file-union, i.e.
  (define jsoftware-aux-files
    (file-union "jsoftware-aux-files"
      `(("profile.ijs" ,(search-aux-file ...)
        ("ijconsole" ,(program-file ...))))

I'm not quite sure if you want to use jsoftware-aux-files directly as
input or whether it's wiser to stuff it into another union like 
(file-union "jsoftware-aux-input" `(("aux" ,jsoftware-aux-files))).
search-input-file will probably do the right thing regardless.
The new style should also still work with assoc-ref, it'd just be
weirder to look at.  Lastly, you could code up a (search-file-input)
just in case; I'm not sure if we have one already.

Cheers

Message #58 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Mon, 17 Jan 2022 00:19:18 +0900

[Message part 1 (text/plain, inline)]

> > Interesting idea. How about just always forcing a MINOR part, setting
> > to "0" if upstream doesn't have one?
> That'd declare regular releases as MAJOR.0 in the version field, which
> I'm not sure if we want that.  In the case of random commits I'm less
> reserved, as they don't correspond to releases anyway.

I see your point. In fact, upstream releases start with MINOR part "a" and
"count up" through the letters over the course of a year. It's a pretty simple
scheme. For example, J 901 started at "j901-release-a" and ended at
"j901-release-f".

When I originally wrote this package, I noticed that the release tag for the
first J 902 was "j902-release", and so treaded MINOR as potentially optional.
However, after checking upstream's forums, this looks to just be a git tag
mishap.

How about we just go ahead and treat MINOR as mandatory as well?


> > If you have a better idea, I am all ears.
> You could define that file-union right after ijconsole.  If you want to
> golf even more, you could define ijconsole inside that file-union, i.e.
>   (define jsoftware-aux-files
>     (file-union "jsoftware-aux-files"
>       `(("profile.ijs" ,(search-aux-file ...)
>         ("ijconsole" ,(program-file ...))))
> 
> I'm not quite sure if you want to use jsoftware-aux-files directly as
> input or whether it's wiser to stuff it into another union like 
> (file-union "jsoftware-aux-input" `(("aux" ,jsoftware-aux-files))).
> search-input-file will probably do the right thing regardless.
> The new style should also still work with assoc-ref, it'd just be
> weirder to look at.  Lastly, you could code up a (search-file-input)
> just in case; I'm not sure if we have one already.

The file-union seems like a cludgy workaround to me. What we really want is an
easy, direct way to get handles on the input files. Heck, program-file objects
already have a name property; why can't we use that? Attached patches are a
proof-of-concept.

That said, if this is going to turn into a big rabbit hole, can we just munge
the J package inputs into whatever you think is best?

[0001-packages-Expand-range-of-objects-add-input-label-can.patch (text/x-patch, attachment)]

[0002-gnu-Add-j.patch (text/x-patch, attachment)]

Message #61 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Sun, 16 Jan 2022 20:53:29 +0100

Hi,

Am Montag, dem 17.01.2022 um 00:19 +0900 schrieb
elaexuotee <at> wilsonb.com:
> > > Interesting idea. How about just always forcing a MINOR part,
> > > setting to "0" if upstream doesn't have one?
> > That'd declare regular releases as MAJOR.0 in the version field,
> > which I'm not sure if we want that.  In the case of random commits
> > I'm less reserved, as they don't correspond to releases anyway.
> 
> I see your point. In fact, upstream releases start with MINOR part "a"
> and "count up" through the letters over the course of a year. It's a
> pretty simple scheme. For example, J 901 started at "j901-release-a"
> and ended at "j901-release-f".
> 
> When I originally wrote this package, I noticed that the release tag
> for the first J 902 was "j902-release", and so treaded MINOR as
> potentially optional.
> However, after checking upstream's forums, this looks to just be a git
> tag mishap.
> 
> How about we just go ahead and treat MINOR as mandatory as well?
In other words minor should always have a value >= "a" and 902 was just
missing it?  Fair enough, in that case making version mandatory sounds
like the way to go.

> > > If you have a better idea, I am all ears.
> > You could define that file-union right after ijconsole.  If you want
> > to
> > golf even more, you could define ijconsole inside that file-union,
> > i.e.
> >   (define jsoftware-aux-files
> >     (file-union "jsoftware-aux-files"
> >       `(("profile.ijs" ,(search-aux-file ...)
> >         ("ijconsole" ,(program-file ...))))
> > 
> > I'm not quite sure if you want to use jsoftware-aux-files directly as
> > input or whether it's wiser to stuff it into another union like 
> > (file-union "jsoftware-aux-input" `(("aux" ,jsoftware-aux-files))).
> > search-input-file will probably do the right thing regardless.
> > The new style should also still work with assoc-ref, it'd just be
> > weirder to look at.  Lastly, you could code up a (search-file-input)
> > just in case; I'm not sure if we have one already.
> 
> The file-union seems like a cludgy workaround to me. What we really
> want is an easy, direct way to get handles on the input files. Heck,
> program-file objects already have a name property; why can't we use
> that? Attached patches are a proof-of-concept.
That proof of concept does look nice, but for one we're trying to move
away from labels, and for the other, it's on a scale that I don't want
to decide as part of a package addition.  If you feel it has value
outside of the proposed usage for j, discussing it under a different
number or perhaps on guix-devel might be worth it.

> That said, if this is going to turn into a big rabbit hole, can we just
> munge the J package inputs into whatever you think is best?
As said in my previous mail, that'd be
> >   (define jsoftware-aux-files
> >     (file-union "jsoftware-aux-files"
> >       `(("profile.ijs" ,(search-aux-file ...)
> >         ("ijconsole" ,(program-file ...))))
In my personal opinion, you can then simply add jsoftware-aux-files as
input and (search-input-file "ijconsole") instead of the current assoc-
ref.  WDYT?

Don't worry, I don't plan to drag this out too long, but I also don't
planning on pushing this today.  There are definitely some stylistic
choices that I want to make under the considerations here (basically,
we can just merge major and minor into a single base that'd be "903.a",
for example), but it's almost bedtime and I want to relax a little
before going to sleep.

Cheers

Message #64 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Mon, 17 Jan 2022 10:24:49 +0900

[Message part 1 (text/plain, inline)]

Again, thanks for the consistent quick turn-around, lately.

> > How about we just go ahead and treat MINOR as mandatory as well?
> In other words minor should always have a value >= "a" and 902 was just
> missing it?  Fair enough, in that case making version mandatory sounds
> like the way to go.

Exactly.

Thanks to your close reading of the patch and your opinionated suggestions, we
were able to catch this

> > The file-union seems like a cludgy workaround to me. What we really
> > want is an easy, direct way to get handles on the input files. Heck,
> > program-file objects already have a name property; why can't we use
> > that? Attached patches are a proof-of-concept.
> That proof of concept does look nice, but for one we're trying to move
> away from labels, and for the other, it's on a scale that I don't want
> to decide as part of a package addition.  If you feel it has value
> outside of the proposed usage for j, discussing it under a different
> number or perhaps on guix-devel might be worth it.

Of course. It would be kind of ridiculous to merge some random, only vaguely
related patch deep in the internals of the system, as part of a simple package
addendum.

We're not writing bills for the Senate here!

Whether or not the patch is valuable, I could learn a lot from any ensuing
discussion, so I might take up your idea to post separately.

> > That said, if this is going to turn into a big rabbit hole, can we just
> > munge the J package inputs into whatever you think is best?
> As said in my previous mail, that'd be
> > >   (define jsoftware-aux-files
> > >     (file-union "jsoftware-aux-files"
> > >       `(("profile.ijs" ,(search-aux-file ...)
> > >         ("ijconsole" ,(program-file ...))))
> In my personal opinion, you can then simply add jsoftware-aux-files as
> input and (search-input-file "ijconsole") instead of the current assoc-
> ref.  WDYT?

Sounds clear to me!

However, for some reason, right now if 'search-auxiliary-file' is inside a
file-union, I'm getting ENOENT on the file somewhere:

    Backtrace:
               2 (primitive-load "/gnu/store/fk7mr923n47r7wj7xqlfmh80jc5?")
    In ice-9/eval.scm:
        619:8  1 (_ #f)
    In unknown file:
               0 (stat "/home/x/devel/org.gnu.savannah/guix/gnu/package?" ?)

    ERROR: In procedure stat:
    In procedure stat: No such file or directory: "/home/x/devel/org.gnu.savannah/guix/gnu/packages/aux-files/jsoftware/profilex.ijs"
    builder for `/gnu/store/4zhrg7g17bqpmlgp5i58vbsc5g8xsl1s-jsoftware-aux-files.drv' failed with exit code 1
    build of /gnu/store/4zhrg7g17bqpmlgp5i58vbsc5g8xsl1s-jsoftware-aux-files.drv failed
    View build log at '/var/log/guix/drvs/4z/hrg7g17bqpmlgp5i58vbsc5g8xsl1s-jsoftware-aux-files.drv.bz2'.
    cannot build derivation `/gnu/store/ax3nwc5xybqcirxadm4ynz99jsq3l3j7-jsoftware-j-903.a.drv': 1 dependencies couldn't be built
    guix build: error: build of `/gnu/store/ax3nwc5xybqcirxadm4ynz99jsq3l3j7-jsoftware-j-903.a.drv' failed

Running `stat <path>` from the command line on the offending path succeeds as
expected, and moving the 'search-auxiliary-file' out of the file-union and
into the package inputs lets the build proceed.

I'm stumped. Attached is the package definition, for reference.


> Don't worry, I don't plan to drag this out too long, but I also don't
> planning on pushing this today.  There are definitely some stylistic
> choices that I want to make under the considerations here (basically,
> we can just merge major and minor into a single base that'd be "903.a",
> for example), but it's almost bedtime and I want to relax a little
> before going to sleep.

Sure. I'm mostly just way out of my depth here, getting sent on deep
rabbit-holes every time you respond give me some feedback to chew on. :P

Definitely down for some stylistic improvements. Let me know!

Hope you got some nice relaxation time in!

[jsoftware.scm (text/plain, attachment)]

Message #67 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Mon, 17 Jan 2022 22:12:52 +0100

[Message part 1 (text/plain, inline)]

Hi,

I tried to get J ready for packaging, but thanks to my CPU not
supporting AVX2 and some helpful discussion in IRC, I was able to avoid
an error here.

The (guix cpu) code is not meant to be used at runtime as the comment
to ijconsole implies it wants it to be used.  Rather, you should use it
at compile time to bake in the correct tuning parameters, and it should
ideally also support "cross-tuning" (though in that case you probably
need to disable the tests because your CPU won't make sense of the
code).  ijconsole not only fails that, but it also fails at an even
more basic task in Guix' launcher scripts, which is actually knowing
the thing you launch.  (We hardcode everything, period.)

Now I could just disable everything AVX-related in J and push something
that works on x86 and amd64, but since you do claim that AVX is
important to J, there are also other options one could take here.  One
is to implement tuning for this package the correct way, though since
it reinvents build systems, that's probably going to be a hard one. 
The other would be to define package variants (e.g. jsoftware-j-with-
avx) and use those -- there ought to be an old blog post on Guix HPC
detailing the rationale behind doing that.

WDYT?  Is AVX worth the trouble?

[0001-gnu-Add-j.patch (text/x-patch, attachment)]

Message #70 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Tue, 18 Jan 2022 13:01:50 +0900

> I tried to get J ready for packaging, but thanks to my CPU not
> supporting AVX2 and some helpful discussion in IRC, I was able to avoid
> an error here.

The black hole has extended it's grip...

> The (guix cpu) code is not meant to be used at runtime as the comment
> to ijconsole implies it wants it to be used.  Rather, you should use it
> at compile time to bake in the correct tuning parameters, and it should
> ideally also support "cross-tuning" (though in that case you probably
> need to disable the tests because your CPU won't make sense of the
> code).

Hrm... Mind if I ask about the rationale? We could go back to scraping
/proc/cpuinfo, but I suspect that's not addressing the issue you have in mind.

> ijconsole not only fails that, but it also fails at an even more basic task
> in Guix' launcher scripts, which is actually knowing the thing you launch.
> (We hardcode everything, period.)

Okay. I see the logic in that.

The original patch indeed hard-coded paths in the ijconsole script, but that
required having ijconsole be a wrapper package. I thought I was being clever by
eliminating a "needless" wrapper, but it sounds like we can just bring it
back.

> Now I could just disable everything AVX-related in J and push something
> that works on x86 and amd64, but since you do claim that AVX is
> important to J, there are also other options one could take here.  One
> is to implement tuning for this package the correct way, though since
> it reinvents build systems, that's probably going to be a hard one. 
> The other would be to define package variants (e.g. jsoftware-j-with-
> avx) and use those -- there ought to be an old blog post on Guix HPC
> detailing the rationale behind doing that.

Not including AVX would be a hard hit.

Morally, the ijconsole script just wraps the libj*.so libs into a "fat binary".
From what I understand, fat binaries are (were?) an accepted solution in the
HPC world. Heck, the libj*.so total up to a whopping 15Mb, which feels more
like slim that fat, anyway.

Questions floating around in my head:

- What is wrong with detecting CPU features in a wrapper vs in the binary?
- If a wrapper is theoretically okay, how can we fix the current one?
- Is such a "proper fix" massively more subtle and bug-prone than the naive
  solution? Why?

> WDYT?  Is AVX worth the trouble?

Personally, I really do not like the jsoftware-j-with-* variants workaround. If
you say that tuning support is the only other acceptable solution, then I'll
bite the bullet and see what is possible.


P.S. Just throwing this out there, but if you'd like, I'd be willing to do a
mob session together. It seems like we are in compatible timezones.

Message #73 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Tue, 18 Jan 2022 09:04:42 +0100

Hi,

Am Dienstag, dem 18.01.2022 um 13:01 +0900 schrieb
elaexuotee <at> wilsonb.com:
> 
> 
> > The (guix cpu) code is not meant to be used at runtime as the
> > comment to ijconsole implies it wants it to be used.  Rather, you
> > should use it at compile time to bake in the correct tuning
> > parameters, and it should ideally also support "cross-tuning"
> > (though in that case you probably need to disable the tests because
> > your CPU won't make sense of the code).
> 
> Hrm... Mind if I ask about the rationale? We could go back to
> scraping /proc/cpuinfo, but I suspect that's not addressing the issue
> you have in mind.
> 
> > ijconsole not only fails that, but it also fails at an even more
> > basic task in Guix' launcher scripts, which is actually knowing the
> > thing you launch. (We hardcode everything, period.)
> 
> Okay. I see the logic in that.
> 
> The original patch indeed hard-coded paths in the ijconsole script,
> but that required having ijconsole be a wrapper package. I thought I
> was being clever by eliminating a "needless" wrapper, but it sounds
> like we can just bring it back.
As far as I can see, you still need a wrapper to provide a symlink
stripped of the version.  I think you can combine the CPU detection
code with that (see below), so you can have a wrapper procedure, which
strips the version LOGICAL OR detects the CPU.

> > Now I could just disable everything AVX-related in J and push
> > something that works on x86 and amd64, but since you do claim that
> > AVX is important to J, there are also other options one could take
> > here.  One is to implement tuning for this package the correct way,
> > though since it reinvents build systems, that's probably going to
> > be a hard one. The other would be to define package variants (e.g.
> > jsoftware-j-with-avx) and use those -- there ought to be an old
> > blog post on Guix HPC detailing the rationale behind doing that.
> 
> Not including AVX would be a hard hit.
> 
> Morally, the ijconsole script just wraps the libj*.so libs into a
> "fat binary".  From what I understand, fat binaries are (were?) an
> accepted solution in the HPC world. Heck, the libj*.so total up to a
> whopping 15Mb, which feels more like slim that fat, anyway.
> 
> Questions floating around in my head:
> 
> - What is wrong with detecting CPU features in a wrapper vs in the
> binary?
> - If a wrapper is theoretically okay, how can we fix the current one?
> - Is such a "proper fix" massively more subtle and bug-prone than the
> naive solution? Why?
FWIW I don't quite think that fat binaries are the issue here, but that
building AVX blows up the check phase in the way we currently do.  It's
a similar issue w.r.t. check? for cross-compiling.  In my opinion only
the base feature set can be checked unless the CPU supports the same
features as the target.

I think if we want to do fat binaries, the solution would be to build
all three variants from make-j and tests only the base variants, and
then have a non-substitutable wrapper package, that takes that as an
input and simply provides a wrapper tuned to the current CPU features.
If you want, you can then rerun the correct tests in the wrapper
package.  The package returned by make-j could itself also provide
three binaries just in case someone doesn't want to generate the
wrapper package, but knows they can run ijconsole-avx2 just fine. 
WDYT?

> > WDYT?  Is AVX worth the trouble?
> 
> Personally, I really do not like the jsoftware-j-with-* variants
> workaround.  If you say that tuning support is the only other
> acceptable solution, then I'll bite the bullet and see what is
> possible.
> 
> P.S. Just throwing this out there, but if you'd like, I'd be willing
> to do a mob session together. It seems like we are in compatible
> timezones.
I'm in UTC+1 and have a day job from 8am to 4pm (plus delta and bus
routes), which normally make me unable to reply from roughly 6:30am to
8pm.  I'm also a little shy when it comes to letting strangers hear my
voice and experiencing healthy vaccine side effects atm.

Cheers

Message #76 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: elaexuotee <at> wilsonb.com
To: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Tue, 18 Jan 2022 19:40:06 +0900

> FWIW I don't quite think that fat binaries are the issue here, but that
> building AVX blows up the check phase in the way we currently do.  It's
> a similar issue w.r.t. check? for cross-compiling.  In my opinion only
> the base feature set can be checked unless the CPU supports the same
> features as the target.

Oh, good. Apparently, I mis-interpreted your original concerns.

> I think if we want to do fat binaries, the solution would be to build
> all three variants from make-j and tests only the base variants, and
> then have a non-substitutable wrapper package, that takes that as an
> input and simply provides a wrapper tuned to the current CPU features.
> If you want, you can then rerun the correct tests in the wrapper
> package.  The package returned by make-j could itself also provide
> three binaries just in case someone doesn't want to generate the
> wrapper package, but knows they can run ijconsole-avx2 just fine. 
> WDYT?

Slick. Let me check my understanding against yours with some specifics:

- Name make-j's package 'jsoftware-j-lib' which
  1. Builds all upstream stuff, and
  2. Only runs libj.so (non-avx) tests;

- Create non-substitutable 'jsoftware-j' wrapper package which
  1. Detects SSE extensions at build time and specializes the 'ijconsole'
     script accordingly, and
  2. Runs tests if and only if an avx or avx2 variant.

Does those points jibe with your thoughts on the fat binaries approach?


Glancing around at the CPU tuning stuff, it looks like tunable packages end up
getting a 'cpu-tuning' property which holds the microartecture name passed to
-march. AVX first landed in Sandy Bridge, and AVX2 in Haswell, so assuming
cpu-tuning is accessible at build time, it should be easy enough to condition
the build phase on those.

Regarding the check phase, here's a relevant comment in
guix/transformations.scm(tuned-package):552:

    ;; The machine building this package may or may not be able to run code
    ;; for MICRO-ARCHITECTURE.  Because of that, skip tests; they are run for
    ;; the "baseline" variant anyway.

Which I read to mean that the check phase should explicitly use libj.so,
ignoring libj-avx.so and libj-avx2.so. Running specialized tests in a
non-substitutable wrapper seems potentially better in this particular case.


Thoughts?


> I'm in UTC+1 and have a day job from 8am to 4pm (plus delta and bus
> routes), which normally make me unable to reply from roughly 6:30am to
> 8pm.  I'm also a little shy when it comes to letting strangers hear my
> voice and experiencing healthy vaccine side effects atm.

UTC+9 here. Okay. That's unfortunate, but I'll keep the offer open if you
change your mind.

Hope you get to feeling better soon!

Cheers,
B. Wilson

Message #79 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Liliana Marie Prikler <liliana.prikler <at> gmail.com>
To: elaexuotee <at> wilsonb.com
Cc: Maxime Devos <maximedevos <at> telenet.be>, 48463 <at> debbugs.gnu.org
Subject: Re: [bug#48463] gnu: Add j.
Date: Tue, 18 Jan 2022 12:33:38 +0100

Am Dienstag, dem 18.01.2022 um 19:40 +0900 schrieb
elaexuotee <at> wilsonb.com:
> > FWIW I don't quite think that fat binaries are the issue here, but
> > that building AVX blows up the check phase in the way we currently
> > do. It's a similar issue w.r.t. check? for cross-compiling.  In my
> > opinion only the base feature set can be checked unless the CPU
> > supports the same features as the target.
> 
> Oh, good. Apparently, I mis-interpreted your original concerns.
> 
> > I think if we want to do fat binaries, the solution would be to
> > build all three variants from make-j and tests only the base
> > variants, and then have a non-substitutable wrapper package, that
> > takes that as an input and simply provides a wrapper tuned to the
> > current CPU features.
> > If you want, you can then rerun the correct tests in the wrapper
> > package.  The package returned by make-j could itself also provide
> > three binaries just in case someone doesn't want to generate the
> > wrapper package, but knows they can run ijconsole-avx2 just fine. 
> > WDYT?
> 
> Slick. Let me check my understanding against yours with some
> specifics:
> 
> - Name make-j's package 'jsoftware-j-lib' which
>   1. Builds all upstream stuff, and
>   2. Only runs libj.so (non-avx) tests;
> 
> - Create non-substitutable 'jsoftware-j' wrapper package which
>   1. Detects SSE extensions at build time and specializes the
> 'ijconsole'
>      script accordingly, and
>   2. Runs tests if and only if an avx or avx2 variant.
> 
> Does those points jibe with your thoughts on the fat binaries
> approach?
That's roughly what I was suggesting, yes.

> Glancing around at the CPU tuning stuff, it looks like tunable
> packages end up getting a 'cpu-tuning' property which holds the
> microartecture name passed to -march. AVX first landed in Sandy
> Bridge, and AVX2 in Haswell, so assuming cpu-tuning is accessible at
> build time, it should be easy enough to condition the build phase on
> those.
> 
> Regarding the check phase, here's a relevant comment in
> guix/transformations.scm(tuned-package):552:
> 
>     ;; The machine building this package may or may not be able to
> run code
>     ;; for MICRO-ARCHITECTURE.  Because of that, skip tests; they are
> run for
>     ;; the "baseline" variant anyway.
> 
> Which I read to mean that the check phase should explicitly use
> libj.so, ignoring libj-avx.so and libj-avx2.so. Running specialized
> tests in a non-substitutable wrapper seems potentially better in this
> particular case.
> 
> Thoughts?
I can't say that I agree with your reasoning here.  Essentially, what
you (or rather the J authors) are doing here is trying to outsmart the
compiler, while Guix assumes, that -march=WHATEVER ought to both
suffice in terms of performance and still be correct.  I'm leaning more
towards the assumption that Guix makes here.


If you want to enable tests on your local machine, you can still do so
by inheriting the transformed package.  Perhaps we should add a "--
with-tests" transformation as a counterpart to the already existing --
without-tests, which would forcefully enable checks that have been
disabled elsewhere.

Furthermore, tuned-package would still support substitution, you just
need to tell your CI to build for particular micro-architectures.  If
you're part of a sufficiently large dev team that uses J with Guix
tooling, that makes a difference between everyone running checks Monday
in the morning or simply fetching a build that has already succeeded on
Sunday.

Cheers

Message #82 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: Maxime Devos <maximedevos <at> telenet.be>
To: 48463 <at> debbugs.gnu.org
Subject: Re: gnu: Add j
Date: Sat, 28 May 2022 14:44:45 +0200

[Message part 1 (text/plain, inline)]

Hi,

There is now a mechanism for tuning packages for particular
subarchitectures without resorting to irreproducible and
unsubstitutable packages, see
<https://hpc.guix.info/blog/2022/01/tuning-packages-for-a-cpu-micro-architecture/>.
 Might be useful here.

Greetings,
Maxime.

[signature.asc (application/pgp-signature, inline)]

Message #85 received at 48463 <at> debbugs.gnu.org (full text, mbox):

From: jgart <jgart <at> dismail.de>
To: 48463 <at> debbugs.gnu.org
Cc: elaexuotee <at> wilsonb.com, Leo Prikler <leo.prikler <at> student.tugraz.at>,
 Maxime Devos <maximedevos <at> telenet.be>,
 Liliana Marie Prikler <liliana.prikler <at> gmail.com>
Subject: Re: gnu: Add j.
Date: Tue, 02 Jul 2024 10:19:27 -0500

Hi!

Is anyone interested in picking this up again?

-- 
all the best,
jgart

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.