GNU bug report logs - #32463
27.0.50; (logior -1) => 4611686018427387903

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 32463 in the body.
You can then email your comments to 32463 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Katsumi Yamaoka <yamaoka <at> jpl.org>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 12:29:45 +0900

Hi,

What do I have to do to get -1 by `(logior -1)' ?  Otherwise,
is it just a bug?  Setting `binary-as-unsigned' has no effect.

I'm using an old input method sj3-egg[1] but it got not to work.
The following Lisp snippet shows what it does first when opening
the connection to the sj3 server:

(with-temp-buffer
  (set-buffer-multibyte nil)
  (let ((pt (point-min)))
    (insert "\377\377\377\376")
    (logior
     (lsh (- (logxor (char-after pt) 128) 128) 24)
     (lsh (char-after (+ pt 1)) 16)
     (lsh (char-after (+ pt 2)) 8)
     (lsh (char-after (+ pt 3)) 0))))

It should return -2, but 4611686018427387902 now.
(Oh, it's doubled most-positive-fixnum!)

Thanks.

[1] http://www.jpl.org/ftp/pub/elisp/sj3-egg-0.8.5.tar.gz

In GNU Emacs 27.0.50 (build 1, x86_64-unknown-cygwin, GTK+ Version 3.22.28)
 of 2018-08-17 built on localhost
Windowing system distributor 'The Cygwin/X Project', version 11.0.12000000

Message #8 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: yamaoka <at> jpl.org
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 05:59:14 +0000

[Message part 1 (text/plain, inline)]

Can you try the attached patch? It fixes a few things that I reported
to Tom yesterday:

---


None of these are necessarily bugs, but:

* I find the behavior of `lsh', `logand', `logior', and `logxor', when
given negative arguments, surprising. I think it would make most sense
to treat negative numbers as the infinite bitstream consisting of all
ones to the left of the specified value:

(logand -1 -1) would be interpreted as ...1111111 & ...1111111 =
...1111111, so it would be -1 (rather than 2 * most-positive-fixnum +
1).
(lsh (- (lsh -1 64) 1) -1) would be ...1110111...1111 shifted to the
right by one digit, an odd number, rather than the even number
currently produced. (I believe lsh and ash should behave identically.)

* the documentation of `random' still refers to representable integers
* I think we should rename `random' to `random-fixnum' and add a Lisp
function `random' which accepts positive fixnum, bignum, and float
arguments, doing the right thing for each. `cl-random' similarly needs
updating, or documentation of its current 32-bit nature.
* there appears to be a most-positive-bignum; on x86-64, it consists
of 0x7fffffff 8-byte words of one bits, 16 GiB. Operating on integers
larger than that will currently abort emacs with an error message:
"gmp: overflow in mpz type".
* long-running bignum operations appear not to be interruptible.

Please consider something like the attached patch?

Thanks!
On Fri, Aug 17, 2018 at 3:31 AM Katsumi Yamaoka <yamaoka <at> jpl.org> wrote:
>
> Hi,
>
> What do I have to do to get -1 by `(logior -1)' ?  Otherwise,
> is it just a bug?  Setting `binary-as-unsigned' has no effect.
>
> I'm using an old input method sj3-egg[1] but it got not to work.
> The following Lisp snippet shows what it does first when opening
> the connection to the sj3 server:
>
> (with-temp-buffer
>   (set-buffer-multibyte nil)
>   (let ((pt (point-min)))
>     (insert "\377\377\377\376")
>     (logior
>      (lsh (- (logxor (char-after pt) 128) 128) 24)
>      (lsh (char-after (+ pt 1)) 16)
>      (lsh (char-after (+ pt 2)) 8)
>      (lsh (char-after (+ pt 3)) 0))))
>
> It should return -2, but 4611686018427387902 now.
> (Oh, it's doubled most-positive-fixnum!)
>
> Thanks.
>
> [1] http://www.jpl.org/ftp/pub/elisp/sj3-egg-0.8.5.tar.gz
>
> In GNU Emacs 27.0.50 (build 1, x86_64-unknown-cygwin, GTK+ Version 3.22.28)
>  of 2018-08-17 built on localhost
> Windowing system distributor 'The Cygwin/X Project', version 11.0.12000000
>
>
>

[Minor-bignum-tweaks.patch (text/x-patch, attachment)]

Message #11 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Katsumi Yamaoka <yamaoka <at> jpl.org>
To: Pip Cet <pipcet <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 16:40:55 +0900

On Fri, 17 Aug 2018 05:59:14 +0000, Pip Cet wrote:
> Can you try the attached patch? It fixes a few things that I reported
> to Tom yesterday:

Oh, the patch solves my problem.  sj3-egg now works as before.
Thank you!

Message #14 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 10:27:25 +0100

On Fri 17 Aug 2018, Pip Cet wrote:

> Can you try the attached patch? It fixes a few things that I reported
> to Tom yesterday:
>
> ---
>
>
> None of these are necessarily bugs, but:
>
> * I find the behavior of `lsh', `logand', `logior', and `logxor', when
> given negative arguments, surprising. I think it would make most sense
> to treat negative numbers as the infinite bitstream consisting of all
> ones to the left of the specified value:
>
> (logand -1 -1) would be interpreted as ...1111111 & ...1111111 =
> ...1111111, so it would be -1 (rather than 2 * most-positive-fixnum +
> 1).
> (lsh (- (lsh -1 64) 1) -1) would be ...1110111...1111 shifted to the
> right by one digit, an odd number, rather than the even number
> currently produced. (I believe lsh and ash should behave identically.)

> @@ -3383,8 +3383,6 @@ ash_lsh_impl (Lisp_Object value, Lisp_Object count, bool lsh)
>        mpz_init (result);
>        if (XFIXNUM (count) >= 0)
>  	mpz_mul_2exp (result, XBIGNUM (value)->value, XFIXNUM (count));
> -      else if (lsh)
> -	mpz_tdiv_q_2exp (result, XBIGNUM (value)->value, - XFIXNUM (count));
>        else
>  	mpz_fdiv_q_2exp (result, XBIGNUM (value)->value, - XFIXNUM (count));
>        val = make_number (result);
> @@ -3401,14 +3399,7 @@ ash_lsh_impl (Lisp_Object value, Lisp_Object count, bool lsh)

Please add more test cases to test/src/data-tests.el to ensure that the
logical operations have the expected behaviour for positive and negative
arguments, and for both bignum and fixnum arguments. The tdiv/fdiv were
added to give expected results. Pay particular attention to values around
most-positive-fixnum and most-negative-fixnum.

>        if (XFIXNUM (count) >= 0)
>  	mpz_mul_2exp (result, result, XFIXNUM (count));
> -      else if (lsh)
> -	{
> -	  if (mpz_sgn (result) > 0)
> -	    mpz_fdiv_q_2exp (result, result, - XFIXNUM (count));
> -	  else
> -	    mpz_fdiv_q_2exp (result, result, - XFIXNUM (count));
> -	}
> -      else /* ash */
> +      else
>  	mpz_fdiv_q_2exp (result, result, - XFIXNUM (count));

This part is an obvious simplification and is fine (I should have
spotted it before sending my patch that Tom committed).

    AndyM

Message #17 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: andrewjmoreton <at> gmail.com, eggert <at> cs.ucla.edu
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 11:36:06 +0000

On Fri, Aug 17, 2018 at 9:34 AM Andy Moreton <andrewjmoreton <at> gmail.com> wrote:
>
> On Fri 17 Aug 2018, Pip Cet wrote:
>
> > Can you try the attached patch? It fixes a few things that I reported
> > to Tom yesterday:
> >
> > ---
> >
> >
> > None of these are necessarily bugs, but:
> >
> > * I find the behavior of `lsh', `logand', `logior', and `logxor', when
> > given negative arguments, surprising. I think it would make most sense
> > to treat negative numbers as the infinite bitstream consisting of all
> > ones to the left of the specified value:
> >
> > (logand -1 -1) would be interpreted as ...1111111 & ...1111111 =
> > ...1111111, so it would be -1 (rather than 2 * most-positive-fixnum +
> > 1).
> > (lsh (- (lsh -1 64) 1) -1) would be ...1110111...1111 shifted to the
> > right by one digit, an odd number, rather than the even number
> > currently produced. (I believe lsh and ash should behave identically.)
>
> > @@ -3383,8 +3383,6 @@ ash_lsh_impl (Lisp_Object value, Lisp_Object count, bool lsh)
> >        mpz_init (result);
> >        if (XFIXNUM (count) >= 0)
> >       mpz_mul_2exp (result, XBIGNUM (value)->value, XFIXNUM (count));
> > -      else if (lsh)
> > -     mpz_tdiv_q_2exp (result, XBIGNUM (value)->value, - XFIXNUM (count));
> >        else
> >       mpz_fdiv_q_2exp (result, XBIGNUM (value)->value, - XFIXNUM (count));
> >        val = make_number (result);
> > @@ -3401,14 +3399,7 @@ ash_lsh_impl (Lisp_Object value, Lisp_Object count, bool lsh)
>
> Please add more test cases to test/src/data-tests.el to ensure that the
> logical operations have the expected behaviour for positive and negative
> arguments, and for both bignum and fixnum arguments.

Paul committed a patch in the meantime (independently, I think?) which
does add tests. I'll try to write some more.

> The tdiv/fdiv were
> added to give expected results. Pay particular attention to values around
> most-positive-fixnum and most-negative-fixnum.

I don't think they do give the expected results. We should discuss
that in more detail, but first, can we agree that lsh and ash behave
the same for bignums? If so, clearly one branch of the code you quoted
is incorrect, and I think it's the tdiv one.

I ran:

(require 'cl)

(let ((i 0))
  (while (< i 128)
    (message "%d %x" i (lsh (- (lsh -1 i) 1) -1))
    (incf i)))

and got this output:

[...]
57 -100000000000001
58 -200000000000001
59 -400000000000001
60 -800000000000001
61 -1000000000000000
62 -2000000000000000
63 -4000000000000000
64 -8000000000000000
[...]

Something is wrong there. The expression certainly shouldn't switch
from being odd to being even just because we're leaving the fixnum
range. If I do the calculation by hand on a piece of paper, I get the
results that correspond to the fixnum case, not the bignum case.

Message #20 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: andrewjmoreton <at> gmail.com, eggert <at> cs.ucla.edu
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 11:53:45 +0000

[Message part 1 (text/plain, inline)]

On Fri, Aug 17, 2018 at 11:36 AM Pip Cet <pipcet <at> gmail.com> wrote:
> and got this output:

I forgot to mention this is on an x86-64 machine with a Linux kernel
and Debian GNU/Linux, and external libgmp.

I'm attaching the patch that I think should still be applied, which
includes some tests.

[Minor-bignum-tweaks-002.patch (text/x-patch, attachment)]

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 14:24:02 +0100

On Fri 17 Aug 2018, Pip Cet wrote:

> On Fri, Aug 17, 2018 at 9:34 AM Andy Moreton <andrewjmoreton <at> gmail.com> wrote:
> Paul committed a patch in the meantime (independently, I think?) which
> does add tests. I'll try to write some more.

Thanks. I should have done that with my patches to fix some bignum bugs.

>> The tdiv/fdiv were
>> added to give expected results. Pay particular attention to values around
>> most-positive-fixnum and most-negative-fixnum.
>
> I don't think they do give the expected results. We should discuss
> that in more detail, but first, can we agree that lsh and ash behave
> the same for bignums? If so, clearly one branch of the code you quoted
> is incorrect, and I think it's the tdiv one.

I agree that for bignums lsh and ash should behave the same way. I can
easily belive that there are bugs in this code, as I was fighting
problems with 64bit Windows having 32bit long at the same time.

> I ran:
>
> (require 'cl)
>
> (let ((i 0))
>   (while (< i 128)
>     (message "%d %x" i (lsh (- (lsh -1 i) 1) -1))
>     (incf i)))

Or equivalently:
  (dotimes (i 128) (message "%d %x" i (lsh (1- (lsh -1 i)) -1)))

> and got this output:
>
> [...]
> 57 -100000000000001
> 58 -200000000000001
> 59 -400000000000001
> 60 -800000000000001
> 61 -1000000000000000
> 62 -2000000000000000
> 63 -4000000000000000
> 64 -8000000000000000
> [...]
>
> Something is wrong there.

Yes, this is clearly not behaving correctly and needs to be fixed.
Thanks for investigating this.

    AndyM

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Fri, 17 Aug 2018 14:27:32 +0100

On Fri 17 Aug 2018, Pip Cet wrote:

> On Fri, Aug 17, 2018 at 11:36 AM Pip Cet <pipcet <at> gmail.com> wrote:
>> and got this output:
>
> I forgot to mention this is on an x86-64 machine with a Linux kernel
> and Debian GNU/Linux, and external libgmp.
>
> I'm attaching the patch that I think should still be applied, which
> includes some tests.

I have tried this patch on a Windows MSYS2 mingw64 x86_64 build [1], and
the updated data-tests pass. Thanks for fixing the bugs I introduced.

    AndyM

[1] This needs a patched gmp.h to avoid a different problem with
incorrect DLL imports from libgmp-10.dll that break Flogcount.

Message #29 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pip Cet <pipcet <at> gmail.com>, andrewjmoreton <at> gmail.com
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 11:48:11 -0700

Pip Cet wrote:

> Paul committed a patch in the meantime (independently, I think?) which
> does add tests. I'll try to write some more.

Yes, I noticed the logior etc. problem separately and fixed it in master without 
knowing about this bug report. There are some other bignum problems too that 
need fixing and are in my pipeline.

> can we agree that lsh and ash behave
> the same for bignums?

It would be weird for lsh to act one way for negative bignums, and a different 
and incompatible way for negative fixnums. Instead, I suggest that we deprecate 
lsh, as it doesn't make sense any more now that integers have unbounded size. 
While we're deprecating it, we can make (lsh A B) signal an error if A is a 
bignum and B is negative, since there's nothing we can do there that is 
reasonable and is compatible with the fixnum behavior.

Message #32 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: andrewjmoreton <at> gmail.com, pipcet <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 21:59:49 +0300

> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Sat, 18 Aug 2018 11:48:11 -0700
> Cc: 32463 <at> debbugs.gnu.org
> 
> It would be weird for lsh to act one way for negative bignums, and a different 
> and incompatible way for negative fixnums. Instead, I suggest that we deprecate 
> lsh, as it doesn't make sense any more now that integers have unbounded size. 

It is IMO absurd for us to deprecate a valid and useful operation just
because we added bignums.  If we cannot agree on its semantics for
bignums (which would surprise me), then it is better to make it not
work for bignums at all than deprecate it for fixnums.

> While we're deprecating it, we can make (lsh A B) signal an error if A is a 
> bignum and B is negative, since there's nothing we can do there that is 
> reasonable and is compatible with the fixnum behavior.

If that's the best we can do, fine.  But it doesn't require
deprecating lsh while we are at it.

Message #35 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 20:45:36 +0100

On Sat 18 Aug 2018, Paul Eggert wrote:

> Pip Cet wrote:
>
>> Paul committed a patch in the meantime (independently, I think?) which
>> does add tests. I'll try to write some more.
>
> Yes, I noticed the logior etc. problem separately and fixed it in master
> without knowing about this bug report. There are some other bignum problems
> too that need fixing and are in my pipeline.

I'm not sure what is on your list of remaining issues, but here is mine:

a) A bug in bignumcompare for 64bit Windows. Patch is here:
   http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00487.html

b) fmod_float has a bug:
   http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00442.html

c) Extend Fexpt to support bignums. Patch is here:
   http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00503.html

d) Extend Fceiling, Ffloor, Fround and Ftruncate to support bignums by
   updating rounding_driver.

Please describe any additional issues you have discovered.

    AndyM

Message #38 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: andrewjmoreton <at> gmail.com, pipcet <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 12:59:07 -0700

Eli Zaretskii wrote:

>> we can make (lsh A B) signal an error if A is a
>> bignum and B is negative, since there's nothing we can do there that is
>> reasonable and is compatible with the fixnum behavior.

> If that's the best we can do, fine.
OK, let's go that route.

> It is IMO absurd for us to deprecate a valid and useful operation just
> because we added bignums.

It would indeed be absurd if lsh were still valid and useful. However, because 
lsh assumes fixed-width integers its overall utility is negative for new Elisp 
code because it mostly just introduces opportunities for confusion. This is why 
Common Lisp and Scheme don't have lsh. Backward compatibility is the only reason 
Emacs Lisp should have lsh. (Obviously we can't simply remove lsh.)

Message #41 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: eliz <at> gnu.org
Cc: eggert <at> cs.ucla.edu, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 19:58:40 +0000

On Sat, Aug 18, 2018 at 7:00 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Paul Eggert <eggert <at> cs.ucla.edu>
> > Date: Sat, 18 Aug 2018 11:48:11 -0700
> > Cc: 32463 <at> debbugs.gnu.org
> >
> > It would be weird for lsh to act one way for negative bignums, and a different
> > and incompatible way for negative fixnums. Instead, I suggest that we deprecate
> > lsh, as it doesn't make sense any more now that integers have unbounded size.
>
> It is IMO absurd for us to deprecate a valid and useful operation just
> because we added bignums.  If we cannot agree on its semantics for
> bignums (which would surprise me), then it is better to make it not
> work for bignums at all than deprecate it for fixnums.

The recent code changes made `lsh' behave the same as `ash' for
fixnums, if I understand correctly. Are you suggesting we revert to
the previous behavior, and try to come up with an interpretation for
bignums that somehow extends the previous behavior?

(In any case, the current code for bignums is inconsistent for the
low-order bits that should be unaffected by whatever convention we
choose).

Message #44 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pip Cet <pipcet <at> gmail.com>, eliz <at> gnu.org
Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 15:27:40 -0700

[Message part 1 (text/plain, inline)]

Pip Cet wrote:
> Are you suggesting we revert to
> the previous behavior, and try to come up with an interpretation for
> bignums that somehow extends the previous behavior?

I think Eli was suggesting reverting lsh to the traditional behavior for 
fixnums, for backwards-compatibility reasons.

There doesn't seem to be a good way to extend this behavior for bignums, so I 
installed the attached patch that simply makes it an error to invoke (lsh A B) 
where A is a negative bignum and B is negative. This patch also adds some test 
cases inspired by one of your previous emails (thanks).

[0001-Restore-traditional-lsh-behavior-on-fixnums.txt (text/plain, attachment)]

Message #47 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pip Cet <pipcet <at> gmail.com>, andrewjmoreton <at> gmail.com
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 15:43:54 -0700

[Message part 1 (text/plain, inline)]

Pip Cet wrote:
> I'm attaching the patch that I think should still be applied, which
> includes some tests.

Thanks. I think most of that patch was addressed by my previous patch, except 
for the doc changes for 'random' for which I installed the attached.

[0001-Document-that-random-is-limited-to-fixnums.txt (text/plain, attachment)]

Message #50 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 15:56:55 -0700

> I'm not sure what is on your list of remaining issues, but here is mine:
> 
> a) A bug in bignumcompare for 64bit Windows. Patch is here:
>    http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00487.html

That patch doesn't go far enough, I'm afraid, as there are other bugs in bignum 
comparison. For example, NaNs vs bignums don't always work. And while we're in 
the neighborhood one can more cheaply compare a fixnum to a bignum by simply 
looking at the bignum's sign, as the numeric values don't matter in that case. 
This item is first on my list of things to fix partly because it's relatively easy.

> b) fmod_float has a bug:
>    http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00442.html
> 
> c) Extend Fexpt to support bignums. Patch is here:
>    http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00503.html
> 
> d) Extend Fceiling, Ffloor, Fround and Ftruncate to support bignums by
>    updating rounding_driver.

These are all news to me; thanks for the list.

There are some relatively minor things involving removal of assumption that 
there are no padding bits (OK, so Emacs is not likely to run on Crays any time 
soon, but it's easy to port).

My bigger concern is memory management, along with integer overflow in size or 
bitcount calculation. Copies are made of bignums when not needed, behavior is 
dicey if memory is exhausted during bignum computation, and I'm afraid C-g will 
have problems when bignums get large. I don't have a good handle on this stuff 
yet. I have put in some sanity checks (e.g., see check_bignum_size in emacs.c) 
but I suspect more are needed. In particular, Fexpt will need to be careful as 
it is a good way to explode a bignum's size.

Message #53 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sat, 18 Aug 2018 16:17:29 -0700

[Message part 1 (text/plain, inline)]

Paul Eggert wrote:
> This item is first on my list of things to fix partly because it's relatively easy.

No time like the present, so I installed the attached.

[0001-Improve-bignum-comparison-Bug-32463-50.fix (text/plain, attachment)]

Message #56 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 11:34:55 +0100

On Sat 18 Aug 2018, Paul Eggert wrote:

> My bigger concern is memory management, along with integer overflow in size or
> bitcount calculation. Copies are made of bignums when not needed, behavior is
> dicey if memory is exhausted during bignum computation, and I'm afraid C-g
> will have problems when bignums get large. I don't have a good handle on this
> stuff yet. I have put in some sanity checks (e.g., see check_bignum_size in
> emacs.c) but I suspect more are needed. In particular, Fexpt will need to be
> careful as it is a good way to explode a bignum's size.

Indeed. There are many places that create a temporary bignum, and thus
call mpz_init/mpz_clear frequently. I haven't profiled this, but it may
be an opportunuty for some optimization.

Fixing C-g will require awkward changes to either GMP and/or emacs.

    AndyM

Message #59 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Live System User <nyc4bos <at> aol.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 06:43:39 -0400

Andy Moreton <andrewjmoreton <at> gmail.com> writes:

> On Sat 18 Aug 2018, Paul Eggert wrote:
>
>> Pip Cet wrote:
>>
>>> Paul committed a patch in the meantime (independently, I think?) which
>>> does add tests. I'll try to write some more.
>>
>> Yes, I noticed the logior etc. problem separately and fixed it in master
>> without knowing about this bug report. There are some other bignum problems
>> too that need fixing and are in my pipeline.
>
> I'm not sure what is on your list of remaining issues, but here is mine:
>
> a) A bug in bignumcompare for 64bit Windows. Patch is here:
>    http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00487.html
>
> b) fmod_float has a bug:
>    http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00442.html
>
> c) Extend Fexpt to support bignums. Patch is here:
>    http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00503.html
>
> d) Extend Fceiling, Ffloor, Fround and Ftruncate to support bignums by
>    updating rounding_driver.
>
> Please describe any additional issues you have discovered.

  There also appears to be a bug dealing with date/time
  convesion (maybe a float issue?):

         M-: (date-to-time "Tue, 06 Mar 2018 11:17:02 -0500")
         Invalid date: Tue, 06 Mar 2018 11:17:02 -0500


  When the date/time is being parsed you eroneouse get:

        M-: (parse-time-string "Tue, 06 Mar 2018 11:17:02 -0500")
        (nil nil nil 6 3 2018 2 nil nil)


  With Emacs 26.1, you get:

         (2 17 11 6 3 2018 2 nil -18000)

  See bug#32443

  Thanks.


>
>     AndyM

Message #62 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: eggert <at> cs.ucla.edu
Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 10:48:19 +0000

On Sat, Aug 18, 2018 at 10:58 PM Paul Eggert <eggert <at> cs.ucla.edu> wrote:
 My bigger concern is memory management, along with integer overflow in size or
> bitcount calculation. Copies are made of bignums when not needed, behavior is
> dicey if memory is exhausted during bignum computation, and I'm afraid C-g will
> have problems when bignums get large.

Even if memory isn't exhausted, creating a bignum larger than 16 GB
(our most-positive-bignum) results in an immediate crash with external
libgmp (Linux, x86_64), and that appears not to be easy to fix without
modifying gmp.

> I don't have a good handle on this stuff
> yet. I have put in some sanity checks (e.g., see check_bignum_size in emacs.c)
> but I suspect more are needed. In particular, Fexpt will need to be careful as
> it is a good way to explode a bignum's size.

That and left shifts are probably the ones to worry about for now.
Creating a large bignum by repeated multiplication will require at
least some intermediate bignums, which need to be allocated and copied
and thus probably alert the user to something going on.

Message #65 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 03:52:55 -0700

[Message part 1 (text/plain, inline)]

>> a) A bug in bignumcompare for 64bit Windows. Patch is here:
>> http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00487.html
>> ...
>> b) fmod_float has a bug:
>> http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00442.html
>>
>> c) Extend Fexpt to support bignums. Patch is here:
>> http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00503.html
>>
>> d) Extend Fceiling, Ffloor, Fround and Ftruncate to support bignums by
>>    updating rounding_driver.

I worked on these and installed patches to master that should do (a), (b), and 
(c). For (d) I wrote the attached patch, and plan to test it a bit more before 
installing, as it's the hairiest.

[rounding.diff (text/x-patch, attachment)]

Message #68 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pip Cet <pipcet <at> gmail.com>
Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 03:59:21 -0700

Pip Cet wrote:

> Even if memory isn't exhausted, creating a bignum larger than 16 GB
> (our most-positive-bignum) results in an immediate crash with external
> libgmp (Linux, x86_64), and that appears not to be easy to fix without
> modifying gmp.

Is there a libgmp bug report for this? or is there a reasonable way to 
characterize this arbitrary limitation in libgmp, so that Emacs does not go over 
the limit and crash? I've already put in one limit, and we can tighten that 
limit (or add more checks) if we know what libgmp's limits are.

> That and left shifts are probably the ones to worry about for now.
> Creating a large bignum by repeated multiplication will require at
> least some intermediate bignums, which need to be allocated and copied
> and thus probably alert the user to something going on.

expt does bignums now too, so that's one more point of failure in this area.

Message #71 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: eggert <at> cs.ucla.edu
Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 11:32:19 +0000

On Sun, Aug 19, 2018 at 10:59 AM Paul Eggert <eggert <at> cs.ucla.edu> wrote:
> Pip Cet wrote:
> > Even if memory isn't exhausted, creating a bignum larger than 16 GB
> > (our most-positive-bignum) results in an immediate crash with external
> > libgmp (Linux, x86_64), and that appears not to be easy to fix without
> > modifying gmp.
>
> Is there a libgmp bug report for this? or is there a reasonable way to
> characterize this arbitrary limitation in libgmp, so that Emacs does not go over
> the limit and crash? I've already put in one limit, and we can tighten that
> limit (or add more checks) if we know what libgmp's limits are.

libgmp stores mpzs as an int (this is a known bug, and plans to work
around it are on the GMP site) giving their current size in words (8
bytes each on x86_64). So the maximum bignum is 0x7fffffff * 64 one
bits. When that limit is reached, _mpz_realloc calls abort rather than
the user-provided reallocation function.

Of course, on POSIX systems, we can catch SIGABRT, but then we'd need
a flag to mark whether we're in a GMP function and another flag to
mark that we're calling abort from a signal handler which might have
interrupted the GMP function, and a third flag to mark that stack
overflow occurred in a signal handler which interrupted a GMP function
so we longjmp()ed out of GMP. On most ELF systems, we can probably
preload a library redirecting abort() for GMP. These are all ugly
solutions to what is ultimately a GMP limitation that should probably
be fixed, making GMP interruptible and abort-safe, which essentially
all interactive applications of it require.

I think we can be clever and wrap calls to mpz_mul_2exp (which can
create arbitrary bignums) and whatever Fexpt uses and make our
allocation function signal for bignums >= sqrt(most-positive-bignum)
(such numbers cannot produce the overflow condition if combined with
multiplication, addition, or subtraction). In fact, I'd suggest a much
lower limit until/unless the C-g issue is fixed, perhaps overridable
by a user preference if people really want to use big bignums.

(I've looked at implementing big rational numbers as an Emacs type
using GMP, too, and I think performance issues are more pronounced for
those, since even (absolutely) small numbers can have a huge
representation.)

> > That and left shifts are probably the ones to worry about for now.
> > Creating a large bignum by repeated multiplication will require at
> > least some intermediate bignums, which need to be allocated and copied
> > and thus probably alert the user to something going on.
>
> expt does bignums now too, so that's one more point of failure in this area.

Yes, that's what "that" referred to. Are there other operations that
create large bignums that I've missed?

I'm not sure what a reasonable limit would be, but I think a global
limit of bignum size to something that allows for "immediate"
computations would be best.

Message #74 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Pip Cet <pipcet <at> gmail.com>
Cc: eggert <at> cs.ucla.edu, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 18:03:30 +0300

> From: Pip Cet <pipcet <at> gmail.com>
> Date: Sat, 18 Aug 2018 19:58:40 +0000
> Cc: eggert <at> cs.ucla.edu, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
> 
> > It is IMO absurd for us to deprecate a valid and useful operation just
> > because we added bignums.  If we cannot agree on its semantics for
> > bignums (which would surprise me), then it is better to make it not
> > work for bignums at all than deprecate it for fixnums.
> 
> The recent code changes made `lsh' behave the same as `ash' for
> fixnums, if I understand correctly. Are you suggesting we revert to
> the previous behavior, and try to come up with an interpretation for
> bignums that somehow extends the previous behavior?

I indeed missed the change in the behavior of lsh, as it wasn't
accompanied by any documentation changes.  However, today's commits by
Paul already reverted lsh to its previous behavior, I believe.

Message #77 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 19:00:43 +0100

On Sun 19 Aug 2018, Paul Eggert wrote:

>>> a) A bug in bignumcompare for 64bit Windows. Patch is here:
>>> http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00487.html
>>> ...
>>> b) fmod_float has a bug:
>>> http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00442.html
>>>
>>> c) Extend Fexpt to support bignums. Patch is here:
>>> http://lists.gnu.org/archive/html/emacs-devel/2018-08/msg00503.html
>>>
>>> d) Extend Fceiling, Ffloor, Fround and Ftruncate to support bignums by
>>>    updating rounding_driver.
>
> I worked on these and installed patches to master that should do (a), (b), and
> (c). For (d) I wrote the attached patch, and plan to test it a bit more before
> installing, as it's the hairiest.

Thanks for fixing up all of these issues. Hopefully with some added test
cases the patch for (d) will also work too. Two more issues:

e) Grepping for FIXED_OR_FLOATP shows various places where code either
   assumes that all types are covered (no longer true now there are
   bignums), or uses incorrect accessors for the value in the object.
   For example, Fdefine_coding_system_internal has:

      else if (FIXED_OR_FLOATP (tmp))
        {
          dim2 = CHARSET_DIMENSION (CHARSET_FROM_ID (XFIXNAT (tmp)));

   Which should probably be:

      else if (NUMBERP (tmp))
        {
          dim2 = CHARSET_DIMENSION (CHARSET_FROM_ID (XFLOATINT (tmp)));

f) Users of CONS_TO_INTEGER and INTEGER_TO_CONS could be converted to
   use bignums.

Message #80 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Pip Cet <pipcet <at> gmail.com>
Cc: eggert <at> cs.ucla.edu, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 23:02:00 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

What exactly is the problem with lsh and bignums?  Is it for negative
numbers because there is no specific width?

I think that limiting lsh to fixnums is better than altering its behavior
for fixnums.  But there are other solutions.

One possible solution is to give lsh an optional third argument to
specify the nominal width of the first argument.  The default could be
the width of a fixnum on your platform.

How does Scheme handle the issue?


-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #83 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: rms <at> gnu.org, Pip Cet <pipcet <at> gmail.com>
Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Sun, 19 Aug 2018 20:47:29 -0700

Richard Stallman wrote:
> What exactly is the problem with lsh and bignums?  Is it for negative
> numbers because there is no specific width?

Yes, that's it.

> One possible solution is to give lsh an optional third argument to
> specify the nominal width of the first argument.  The default could be
> the width of a fixnum on your platform.

Although we discussed something along those lines and could easily implement 
something, it's a bit trickier than it might sound at first, because of corner 
cases where the semantics are unclear. Here's one possible implementation (there 
are other approaches of course):

   (defun lsh (value count &optional width)
     (when (and (< value 0) (< count 0))
       (let ((lo (if width (ash 1 (1- width)) most-negative-fixnum)))
	 (when (< value lo)
	   (signal 'args-out-of-range (list value count width)))
	 (setq value (logand (ash value -1) (- -1 lo)))
	 (setq count (1+ count))))
     (ash value count))

I am skeptical whether the complexity of this extension is worth the effort to 
maintain and document. In a language with bignums, if you need a mask of bits 
you simply use a nonnegative integer, which means you can use ash without 
worrying about the corner cases that invariably afflict lsh.

> How does Scheme handle the issue?

Scheme does not have logical shifts, only arithmetic shifts. Logical shifts 
don't make that much sense once you have bignums.

Message #86 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Richard Stallman <rms <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: andrewjmoreton <at> gmail.com, pipcet <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Mon, 20 Aug 2018 23:37:23 -0400

[[[ To any NSA and FBI agents reading my email: please consider    ]]]
[[[ whether defending the US Constitution against all enemies,     ]]]
[[[ foreign or domestic, requires you to follow Snowden's example. ]]]

Maybe you are right about lsh and bignums, but we must keep lsh for
fixnums compatibility.  It could give an error if the first argument
is a negative bignum.  Or it could give an error for any bignum.

-- 
Dr Richard Stallman
President, Free Software Foundation (https://gnu.org, https://fsf.org)
Internet Hall-of-Famer (https://internethalloffame.org)

Message #89 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Pip Cet <pipcet <at> gmail.com>
Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 02:40:34 -0700

[Message part 1 (text/plain, inline)]

Pip Cet wrote:

> I think we can be clever and wrap calls to mpz_mul_2exp (which can
> create arbitrary bignums) and whatever Fexpt uses....  I'd suggest a much
> lower limit until/unless the C-g issue is fixed, perhaps overridable
> by a user preference if people really want to use big bignums.

Yes, this sounds good. After stressing Emacs with bignums for a bit, I found 
that it was too easy to get Emacs to abort or hang by creating large bignums.

> I'm not sure what a reasonable limit would be, but I think a global
> limit of bignum size to something that allows for "immediate"
> computations would be best.

I installed the attached patch to do that. It tentatively defaults to a limit of 
2↑↑5 (i.e., 2**65536) for bignums, overrideable by setting a new variable 
'integer-width' that defaults to 65536. This default should be big enough for 
almost all Emacs applications and should avoid issues of aborts and hangs.

[0001-Avoid-libgmp-aborts-by-imposing-limits.patch (text/x-patch, attachment)]

Message #92 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 11:50:15 +0100

On Tue 21 Aug 2018, Paul Eggert wrote:

> Pip Cet wrote:
>> I'm not sure what a reasonable limit would be, but I think a global
>> limit of bignum size to something that allows for "immediate"
>> computations would be best.
>
> I installed the attached patch to do that. It tentatively defaults to a limit
> of 2↑↑5 (i.e., 2**65536) for bignums, overrideable by setting a new variable
> 'integer-width' that defaults to 65536. This default should be big enough for
> almost all Emacs applications and should avoid issues of aborts and hangs.

Have you checked a mini-gmp build to ensure that this patch works if
the GMP library is not installed ?

It might be slightly faster to use mpz_limbs_read in make_number instead
of mpz_getlimbn.

    AndyM

Message #95 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: andrewjmoreton <at> gmail.com, pipcet <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 17:36:50 +0300

> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Tue, 21 Aug 2018 02:40:34 -0700
> Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
> 
> I installed the attached patch to do that. It tentatively defaults to a limit of 
> 2↑↑5 (i.e., 2**65536) for bignums, overrideable by setting a new variable 
> 'integer-width' that defaults to 65536. This default should be big enough for 
> almost all Emacs applications and should avoid issues of aborts and hangs.

Should the default value be different on 32-bit platforms?

Message #98 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 15:52:23 +0100

On Tue 21 Aug 2018, Eli Zaretskii wrote:

>> From: Paul Eggert <eggert <at> cs.ucla.edu>
>> Date: Tue, 21 Aug 2018 02:40:34 -0700
>> Cc: andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
>> 
>> I installed the attached patch to do that. It tentatively defaults to a limit of 
>> 2↑↑5 (i.e., 2**65536) for bignums, overrideable by setting a new variable 
>> 'integer-width' that defaults to 65536. This default should be big enough for 
>> almost all Emacs applications and should avoid issues of aborts and hangs.
>
> Should the default value be different on 32-bit platforms?

For a 32bit platform using 32bit mp_limb_t, this is only 2048 limbs, so
well within range of mpz_t::_mp_size. Having the same limit on all
platforms should be fine.

Does this limit apply to bignum values in lisp objects, or to
intermediate values inside libgmp, which may require extra space ?
The documentation for `integer-width' should make this clear.

    AndyM

Message #101 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: andrewjmoreton <at> gmail.com, pipcet <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 10:24:04 -0700

Eli Zaretskii wrote:
> Should the default value [of integer-width] be different on 32-bit platforms?

My guess is not. The current default of 2**16 means that individual bignums are 
limited to around 8 KiB, and even 32-bit platforms can handle that. Of course as 
we gain experience we may want to change the default.

Message #104 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 19:29:31 -0700

[Message part 1 (text/plain, inline)]

>>> d) Extend Fceiling, Ffloor, Fround and Ftruncate to support bignums by
>>>    updating rounding_driver.
> 
> I worked on these and installed patches to master that should do (a), (b), and 
> (c). For (d) I wrote the attached patch, and plan to test it a bit more before 
> installing, as it's the hairiest.

It took me longer to write the test cases than the code, but the tests did find 
bugs so it was worth it. I installed the attached.

While we're on the subject I moved the definition of 'bignump' and 'fixnump' 
from C to Lisp, since they are easily implementable in Lisp and don't seem to be 
performance relevant. Hope you don't mind too much that I would rather minimize 
the low-level details that the C code exports.

Both patches attached.

[0001-Move-bignump-fixnump-from-C-to-Lisp.patch (text/x-patch, attachment)]

[0002-Add-bignum-support-to-floor-ceiling-etc.patch (text/x-patch, attachment)]

Message #107 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 19:34:36 -0700

> Does this limit apply to bignum values in lisp objects, or to
> intermediate values inside libgmp, which may require extra space ?
> The documentation for `integer-width' should make this clear.

integer-width applies only to Lisp objects. I'm not sure we should be exposing 
internal details of the interpreter to the Lisp user, which includes the sizes 
of its internal temporaries. If a need shows up for it we can do it, but I don't 
see the need.

Message #110 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Tue, 21 Aug 2018 19:56:29 -0700

> Have you checked a mini-gmp build to ensure that this patch works if
> the GMP library is not installed ?

I have now. :-)  It works, in the sense that Emacs builds and fails only the 
tests that it was already failing.

> It might be slightly faster to use mpz_limbs_read in make_number instead
> of mpz_getlimbn.

On Fedora 28 x86-64, mpz_getlimbn is inline whereas mpz_limbs_read is a function 
call, so it's not clear which should be faster. Another item on the list of 
things to do....

Message #113 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Wed, 22 Aug 2018 09:20:54 +0100

On Tue 21 Aug 2018, Paul Eggert wrote:

>> Have you checked a mini-gmp build to ensure that this patch works if
>> the GMP library is not installed ?
>
> I have now. :-)  It works, in the sense that Emacs builds and fails only the
> tests that it was already failing.

Thanks. I mentioned this because GMP_NUMB_BITS etc are not defined in
mini-gmp.h (which your patches handle), but I wasn't sure if there were
other symbols that needed similar treatment.

As GMP_NUMB_BITS is always defined now, uses of mp_bits_per_limb could
be replaced with GMP_NUMB_BITS.

    AndyM

Message #116 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Wed, 22 Aug 2018 09:39:50 +0100

On Tue 21 Aug 2018, Paul Eggert wrote:

>>>> d) Extend Fceiling, Ffloor, Fround and Ftruncate to support bignums by
>>>>    updating rounding_driver.
>>
>> I worked on these and installed patches to master that should do (a), (b),
>> and (c). For (d) I wrote the attached patch, and plan to test it a bit more
>> before installing, as it's the hairiest.
>
> It took me longer to write the test cases than the code, but the tests did
> find bugs so it was worth it. I installed the attached.

Thanks for all of your recent work on bignum issues, which has
siginificantly improved the bignum implementation.

> While we're on the subject I moved the definition of 'bignump' and 'fixnump'
> from C to Lisp, since they are easily implementable in Lisp and don't seem to
> be performance relevant. Hope you don't mind too much that I would rather
> minimize the low-level details that the C code exports.

I agree that performance is not an issue. However implementing these in lisp
assumes that bignums never represent values in fixnum range, rather than
checking that. We need some checks at the C level to ensure that this
property always holds.

    AndyM

Message #119 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Tom Tromey <tom <at> tromey.com>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: Andy Moreton <andrewjmoreton <at> gmail.com>, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Wed, 22 Aug 2018 10:56:56 -0600

>>>>> "Paul" == Paul Eggert <eggert <at> cs.ucla.edu> writes:

Paul> My bigger concern is memory management, along with integer overflow in
Paul> size or bitcount calculation. Copies are made of bignums when not
Paul> needed

One idea is to change make_number to take ownership of the passed-in mpz_t.
I didn't do this because it makes the API confusing to use (I put in a
comment to this effect) but perhaps efficiency outweighs this; and
anyway the function could be renamed to make this more clear.

Tom

Message #122 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Tom Tromey <tom <at> tromey.com>
Cc: Andy Moreton <andrewjmoreton <at> gmail.com>, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Wed, 22 Aug 2018 10:52:53 -0700

Tom Tromey wrote:
> One idea is to change make_number to take ownership of the passed-in mpz_t.
> I didn't do this because it makes the API confusing to use (I put in a
> comment to this effect) but perhaps efficiency outweighs this; and
> anyway the function could be renamed to make this more clear.

I plan to look into doing something along those lines. (Also, change its name to 
make_integer, since it doesn't make floats.)

Message #125 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: tom <at> tromey.com, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Wed, 22 Aug 2018 21:25:28 +0300

> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Wed, 22 Aug 2018 10:52:53 -0700
> Cc: Andy Moreton <andrewjmoreton <at> gmail.com>, 32463 <at> debbugs.gnu.org
> 
> Also, change its name to make_integer, since it doesn't make
> floats.

I'd rather we left the name alone.  It's a veteran name, burnt into
our muscle memory by many years of coding in Emacs.  Let's not change
it just because it isn't 100% accurate.

Message #128 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Andy Moreton <andrewjmoreton <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Thu, 23 Aug 2018 00:27:55 +0100

On Tue 21 Aug 2018, Paul Eggert wrote:

>> Does this limit apply to bignum values in lisp objects, or to
>> intermediate values inside libgmp, which may require extra space ?
>> The documentation for `integer-width' should make this clear.
>
> integer-width applies only to Lisp objects. I'm not sure we should be exposing
> internal details of the interpreter to the Lisp user, which includes the sizes
> of its internal temporaries. If a need shows up for it we can do it, but I
> don't see the need.

The current documentation is uninformative:

  Maximum number of bits in bignums.
  Integers outside the fixnum range are limited to absolute values less
  than 2**N, where N is this variable’s value.  N should be nonnegative.

This says plenty abut fixnums, but does not mention that anything larger
than fixnum range is represented as a bignum. For users who are new to
all of this, it is important to state that.

I think it is worth mentioning that intermediate computations not visible
to lisp may use slightly larger bignums, and to make it clear that this
is a soft limit intended to prevent misbehaviour of the runtime.

    AndyM

Message #131 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: tom <at> tromey.com, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Wed, 22 Aug 2018 17:28:49 -0700

Eli Zaretskii wrote:
> I'd rather we left the name alone.  It's a veteran name, burnt into
> our muscle memory by many years of coding in Emacs.

But the name 'make_number' means something completely different now; it has a 
different signature, it is a lot more expensive than it used to be, and it 
cannot be used the way the old name could be used. So now is a good time to 
change the name anyway, regardless of whether the name choice was a good one 
originally.

We already changed the name XINT and friends, even though their semantics didn't 
change. It would be odd if we didn't fix make_number too, while we're in the 
neighborhood, since the case for changing it is even stronger than the case for 
XINT etc.

Message #134 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: tom <at> tromey.com, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Thu, 23 Aug 2018 05:39:49 +0300

> Cc: tom <at> tromey.com, andrewjmoreton <at> gmail.com, 32463 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> Date: Wed, 22 Aug 2018 17:28:49 -0700
> 
> Eli Zaretskii wrote:
> > I'd rather we left the name alone.  It's a veteran name, burnt into
> > our muscle memory by many years of coding in Emacs.
> 
> But the name 'make_number' means something completely different now; it has a 
> different signature, it is a lot more expensive than it used to be, and it 
> cannot be used the way the old name could be used.

Then perhaps restore the old make_number and make a new function with
the extra functionality.

> We already changed the name XINT and friends, even though their semantics didn't 
> change.

And it's already a significant blow to coding habits.  I wonder when
this renaming spree will end, because it's beginning to look like a
whole new language to learn.

Message #137 received at 32463 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Andy Moreton <andrewjmoreton <at> gmail.com>
Cc: 32463 <at> debbugs.gnu.org
Subject: Re: bug#32463: 27.0.50; (logior -1) => 4611686018427387903
Date: Thu, 23 Aug 2018 17:05:31 +0300

> From: Andy Moreton <andrewjmoreton <at> gmail.com>
> Date: Thu, 23 Aug 2018 00:27:55 +0100
> 
> The current documentation is uninformative:
> 
>   Maximum number of bits in bignums.
>   Integers outside the fixnum range are limited to absolute values less
>   than 2**N, where N is this variable’s value.  N should be nonnegative.
> 
> This says plenty abut fixnums, but does not mention that anything larger
> than fixnum range is represented as a bignum. For users who are new to
> all of this, it is important to state that.
> 
> I think it is worth mentioning that intermediate computations not visible
> to lisp may use slightly larger bignums, and to make it clear that this
> is a soft limit intended to prevent misbehaviour of the runtime.

Feel free to propose a documentation patch, and let's take it from
there.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.