GNU bug report logs - #38753
27.0.60; cl--random-state uncontrolled growth due to bignums

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 38753 in the body.
You can then email your comments to 38753 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Christopher Wellons <wellons <at> nullprogram.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 27.0.60; cl--random-state uncontrolled growth due to bignums
Date: Thu, 26 Dec 2019 12:12:07 -0500

The cl-random generator was not written for bignums, so it misbehaves in
Emacs 27. After generating a few million numbers from any particular
state, it will only signal "Arithmetic overflow error". The generator
relies on fixnums wrapping via two's complement. In Emacs 27, fixnums
turn into bignums rather than wrap, so the state grows until reaching
the bignum limit, then breaks.

The cl-random function is a lagged Fibonacci generator. The output is
the difference of two integers at special "taps" in the state vector,
and one tap is replaced with the output. The output is properly
truncated using logand, but state update is not, and it soon fills with
growing bignums.

The fix is trivial: Move the logand truncation so that it applies to
both the output and the state update. The truncated bits are never used
so the output of the generator remains unchanged.

diff --git a/lisp/emacs-lisp/cl-extra.el b/lisp/emacs-lisp/cl-extra.el
index 7e9d8fe870..2e0b37c14d 100644
--- a/lisp/emacs-lisp/cl-extra.el
+++ b/lisp/emacs-lisp/cl-extra.el
@@ -469,7 +469,7 @@ cl-random
	  (while (< (setq i (1+ i)) 200) (cl-random 2 state))))
    (let* ((i (cl-callf (lambda (x) (% (1+ x) 55)) (cl--random-state-i state)))
	   (j (cl-callf (lambda (x) (% (1+ x) 55)) (cl--random-state-j state)))
-	   (n (logand 8388607 (aset vec i (- (aref vec i) (aref vec j))))))
+	   (n (aset vec i (logand 8388607 (- (aref vec i) (aref vec j))))))
      (if (integerp lim)
	  (if (<= lim 512) (% n lim)
	    (if (> lim 8388607) (setq n (+ (ash n 9) (cl-random 512 state))))

Message #10 received at 38753-done <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: 38753-done <at> debbugs.gnu.org
Subject: bug#38753: 27.0.60; cl--random-state uncontrolled growth due to
 bignums
Date: Sun, 29 Dec 2019 13:59:31 +0100

Thank you! The proposed fix indeed looks correct; applied to emacs-27.

Message #13 received at 38753 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: Christopher Wellons <wellons <at> nullprogram.com>
Cc: 38753 <at> debbugs.gnu.org
Subject: Re: bug#38753: 27.0.60;
 cl--random-state uncontrolled growth due to bignums
Date: Sun, 29 Dec 2019 13:31:30 +0000

On Thu, Dec 26, 2019 at 5:13 PM Christopher Wellons
<wellons <at> nullprogram.com> wrote:
> The cl-random generator was not written for bignums, so it misbehaves in
> Emacs 27.

Good catch.

I might be wrong, but it looks to me like cl-random and Frandom are
currently both limited to small integers (< 31 bits and fixnum range,
respectively) and we don't have a random number generator that will do
the right thing for
(random 100000000000000000000000000000000000000000000)

Any idea how easy it would be to fix either of them, or both?

Message #16 received at 38753 <at> debbugs.gnu.org (full text, mbox):

From: Philipp Stephani <p.stephani2 <at> gmail.com>
To: Pip Cet <pipcet <at> gmail.com>
Cc: Christopher Wellons <wellons <at> nullprogram.com>, 38753 <at> debbugs.gnu.org
Subject: Re: bug#38753: 27.0.60;
 cl--random-state uncontrolled growth due to bignums
Date: Sun, 29 Dec 2019 14:48:25 +0100

Am So., 29. Dez. 2019 um 14:33 Uhr schrieb Pip Cet <pipcet <at> gmail.com>:
>
> On Thu, Dec 26, 2019 at 5:13 PM Christopher Wellons
> <wellons <at> nullprogram.com> wrote:
> > The cl-random generator was not written for bignums, so it misbehaves in
> > Emacs 27.
>
> Good catch.
>
> I might be wrong, but it looks to me like cl-random and Frandom are
> currently both limited to small integers (< 31 bits and fixnum range,
> respectively) and we don't have a random number generator that will do
> the right thing for
> (random 100000000000000000000000000000000000000000000)
>
> Any idea how easy it would be to fix either of them, or both?


I haven't tried it out, but mpz_urandomm should be what we need. We'd
need to find a replacement for mini-gmp though.

Message #19 received at 38753 <at> debbugs.gnu.org (full text, mbox):

From: Mattias Engdegård <mattiase <at> acm.org>
To: Pip Cet <pipcet <at> gmail.com>, Christopher Wellons <wellons <at> nullprogram.com>, 
 Philipp <p.stephani2 <at> gmail.com>
Cc: 38753 <at> debbugs.gnu.org
Subject: Re: bug#38753: 27.0.60; cl--random-state uncontrolled growth due to
 bignums
Date: Sun, 29 Dec 2019 15:09:30 +0100

> I might be wrong, but it looks to me like cl-random and Frandom are currently both limited to small integers (< 31 bits and fixnum range, respectively) and we don't have a random number generator that will do the right thing for
> (random 100000000000000000000000000000000000000000000) 

Thank you for noticing this! It definitely merits a separate bug (or two).

Message #22 received at 38753 <at> debbugs.gnu.org (full text, mbox):

From: Christopher Wellons <wellons <at> nullprogram.com>
To: Pip Cet <pipcet <at> gmail.com>
Cc: 38753 <at> debbugs.gnu.org
Subject: Re: bug#38753: 27.0.60; cl--random-state uncontrolled growth due to
 bignums
Date: Sun, 29 Dec 2019 09:27:18 -0500

> Any idea how easy it would be to fix either of them, or both?

The reason I found that bug is because I was experimenting with 
addressing exactly that problem:

https://github.com/skeeto/lcg128

It's got the right features, including support for arbitrary limits (per 
your example), but I'm not satisfied with the performance. For arbitrary 
limits it uses the same rejection algorithm as Python — generate (logb 
lim) bits and reject if out of range — and cl-random could be updated to 
use the same technique. The LCG around 6x slower than cl-random in the 
common case (generating small numbers). Bignums aren't ideal for this 
since every operation requires an allocation, and the LCG throws away 
half of the work (the upper half of the multiplication result).

The lagged Fibonacci generator really does hit a sweet spot for Emacs 
Lisp, where, before bignums, even 32-bit integers weren't a reliable 
option. So it fits within Emacs' worst fixnum limitations and requires 
few bytecode instructions to generate a number.

The random built-in ought to simply be pcg32 across all platforms:

http://www.pcg-random.org/download.html

Currently it's whatever crummy host-provided PRNG it happens to find at 
compile time. It's certainly possible to implement pcg32 using Emacs 27 
bignums, but it would be even slower than my LCG, especially on 32-bit 
platforms. In C it's very, very fast.

If the pcg32 state, just one 64-bit integer, was an optional parameter, 
then cl-random could build on it. However, the state would nearly always 
be a bignum, even on 64-bit platforms, and that would really slow it 
down. I haven't yet thought of any obviously good options for this.

Message #25 received at 38753 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> gmail.com>
To: Christopher Wellons <wellons <at> nullprogram.com>
Cc: 38753 <at> debbugs.gnu.org
Subject: Re: bug#38753: 27.0.60;
 cl--random-state uncontrolled growth due to bignums
Date: Sun, 29 Dec 2019 17:33:59 +0000

On Sun, Dec 29, 2019 at 2:27 PM Christopher Wellons
<wellons <at> nullprogram.com> wrote:
> > Any idea how easy it would be to fix either of them, or both?
> The reason I found that bug is because I was experimenting with
> addressing exactly that problem:
>
> https://github.com/skeeto/lcg128
>
> It's got the right features, including support for arbitrary limits (per
> your example), but I'm not satisfied with the performance.

That looks excellent. Maybe we should be using a cryptographic RNG by
default, though? I really don't know the tradeoffs, though.

At first glance, there are two questions I have about lcg128:
 - should random floats have more than 53 significant bits if they're
< 0.5? I think the answer is yes, even though that will make the
implementation harder.
 - should (lcg128-range 2^n) call the generator twice, on average? I
think it's an important special case in which we only want to call the
generator once.

Message #28 received at 38753 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Pip Cet <pipcet <at> gmail.com>
Cc: wellons <at> nullprogram.com, 38753 <at> debbugs.gnu.org
Subject: Re: bug#38753: 27.0.60;
 cl--random-state uncontrolled growth due to bignums
Date: Sun, 29 Dec 2019 20:12:02 +0200

> From: Pip Cet <pipcet <at> gmail.com>
> Date: Sun, 29 Dec 2019 17:33:59 +0000
> Cc: 38753 <at> debbugs.gnu.org
> 
> That looks excellent. Maybe we should be using a cryptographic RNG by
> default, though?

Are you aware that we already init the random seed using system
entropy and/or cryptographic RNG?  See init_random.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.