Subject: bug#41194: closed (Re: bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory)

From: [email-masked] (GNU bug Tracking System)
Date: Sun, 14 Jun 2020 20:49:02 +0000

Your bug report

#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
41194: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=41194
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#41194: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory
From: Kévin Le Gouguec <[email-masked]>
To: Andrea Corallo <[email-masked]>
Cc: [email-masked]
Date: Sun, 14 Jun 2020 22:47:52 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Andrea Corallo <[email-masked]> writes:

> this is very good new thanks!  If I read correctly we are always under
> about 600MB as memory consumption is this correct?

Yep.  I can post the detailed data here if that's needed; here are the
10 files that took the most virtual memory (vsz):

 1. org/org.el              541.2 MB
 2. gnus/gnus-sum.el        459.3 MB
 3. emacs-lisp/comp.el      455.2 MB
 4. net/tramp.el            408.2 MB
 5. net/tramp-adb.el        390.4 MB
 6. net/tramp-sh.el         371.0 MB
 7. progmodes/cperl-mode.el 356.2 MB
 8. ../lisp/window.el       348.9 MB
 9. gnus/message.el         345.4 MB
10. net/soap-client.el      344.4 MB

And the 10 files that took the most resident memory (rss):

 1. org/org.el              361.6 MB
 2. emacs-lisp/comp.el      287.3 MB
 3. gnus/gnus-sum.el        286.7 MB
 4. net/tramp.el            236.8 MB
 5. net/tramp-adb.el        218.1 MB
 6. net/tramp-sh.el         199.0 MB
 7. progmodes/cperl-mode.el 185.0 MB
 8. ../lisp/window.el       181.6 MB
 9. gnus/message.el         174.0 MB
10. erc/erc.el              173.5 MB

> I suspect we are still looking at a speed 3 run.  How comp-speed was set
> for the test?
>
> I ask that because for now speed 3 is written into the stone within
> elisp-benchmarks.el, yeah... :/ and so has to be manually changed there.

Now that's just sneaky 😛 You're right, it was a speed 3 run.  Setting
comp-speed to 2 in elisp-benchmarks.el, I now get the attached results.

* Results
** master
#+TBLNAME: master
| test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|----------------+----------------+------------+---------+-------------+-----------------|
| bubble-no-cons |         118.41 |       0.23 |       3 |      118.63 |            0.36 |
| bubble         |          46.29 |      29.34 |     388 |       75.63 |            0.36 |
| dhrystone      |          74.82 |       0.00 |       0 |       74.82 |            0.34 |
| fibn-rec       |          59.50 |       0.00 |       0 |       59.50 |            0.28 |
| fibn-tc        |         147.39 |      96.86 |    1179 |      244.26 |            2.70 |
| fibn           |         320.33 |     276.23 |    3442 |      596.56 |            5.31 |
| flet           |         112.02 |       0.00 |       0 |      112.02 |            0.29 |
| inclist        |         166.30 |       0.00 |       0 |      166.30 |            0.67 |
| listlen-tc     |          76.97 |       0.00 |       0 |       76.97 |            0.53 |
| map-closure    |          81.86 |       0.00 |       0 |       81.86 |            0.31 |
| nbody          |          28.62 |      98.53 |    1303 |      127.14 |            0.21 |
| pcase          |         110.63 |       0.00 |       0 |      110.63 |            0.29 |
| pidigits       |          61.78 |      39.46 |     469 |      101.24 |            0.70 |
|----------------+----------------+------------+---------+-------------+-----------------|
| total          |        1404.92 |     540.65 |    6786 |     1945.57 |            6.13 |
** feature/native-comp
#+TBLNAME: feature/native-comp
| test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) | tot avg err (s) |
|----------------+----------------+------------+---------+-------------+-----------------|
| bubble-no-cons |          30.12 |       0.16 |       2 |       30.28 |            0.04 |
| bubble         |          18.05 |      27.91 |     352 |       45.96 |            0.33 |
| dhrystone      |          34.26 |       0.00 |       0 |       34.26 |            0.05 |
| fibn-rec       |          26.92 |       0.00 |       0 |       26.92 |            0.03 |
| fibn-tc        |          98.66 |      93.89 |    1096 |      192.55 |            1.66 |
| fibn           |         253.19 |     269.12 |    3192 |      522.31 |            2.17 |
| flet           |          22.92 |       0.00 |       0 |       22.92 |            0.04 |
| inclist        |          11.76 |       0.00 |       0 |       11.76 |            0.16 |
| listlen-tc     |          21.44 |       0.00 |       0 |       21.44 |            0.05 |
| map-closure    |          80.57 |       0.00 |       0 |       80.57 |            0.03 |
| nbody          |          15.62 |      95.66 |    1212 |      111.28 |            0.10 |
| pcase          |          17.64 |       0.00 |       0 |       17.64 |            0.03 |
| pidigits       |          59.82 |      38.86 |     436 |       98.68 |            0.39 |
|----------------+----------------+------------+---------+-------------+-----------------|
| total          |         690.98 |     525.59 |    6291 |     1216.57 |            2.79 |
** ratio
| test           | non-gc avg (s) | gc avg (s) | gcs avg | tot avg (s) |
|----------------+----------------+------------+---------+-------------|
| bubble-no-cons |           3.92 |       1.44 |     1.5 |        3.93 |
| bubble         |           2.56 |       1.05 |     1.1 |        1.64 |
| dhrystone      |           2.18 |            |         |        2.18 |
| fibn-rec       |           2.21 |            |         |        2.21 |
| fibn-tc        |           1.49 |       1.03 |    1.07 |        1.26 |
| fibn           |           1.26 |       1.03 |    1.08 |        1.14 |
| flet           |           4.89 |            |         |        4.89 |
| inclist        |           14.1 |            |         |        14.1 |
| listlen-tc     |            3.6 |            |         |         3.6 |
| map-closure    |           1.02 |            |         |        1.02 |
| nbody          |           1.83 |       1.03 |    1.07 |        1.14 |
| pcase          |           6.31 |            |         |        6.31 |
| pidigits       |           1.03 |       1.02 |    1.08 |        1.02 |
|----------------+----------------+------------+---------+-------------|
| total          |           2.03 |       1.03 |    1.08 |         1.6 |
#+TBLFM: @2$2..@>$> = if (remote(feature/native-comp,@@#$$#)!=0, remote(master,@@#$$#)/remote(feature/native-comp,@@#$$#), string("")); p3

Conclusion: I'm closing this report!  Again, thank you for your efforts.

(And thank you for this opportunity to learn about Org spreadsheets 😉)

Subject: 28.0.50; [feature/native-comp] fibn benchmark exhausts memory
From: Kévin Le Gouguec <[email-masked]>
To: [email-masked]
Date: Mon, 11 May 2020 22:47:33 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

As detailed in bug#41077, I compiled the feature/native-comp branch[1]
on a low-end-ish laptop[2], blacklisting char-fold.el since native
compilation took too much memory for this file.

As reported there, I get appreciable speedups on the benchmarks
distributed in the elisp-benchmarks ELPA package, except for one
specific test: fibn.  When native-compiled (with any value of
comp-speed), this test makes Emacs abort with the message:

> Memory exhausted--use C-x s then exit and restart Emacs

I've plotted some graphs showing fibn's memory consumption for every
value of comp-speed, as well as for emacs master[3] (where all 3
iterations run to completion):

See [4] for the methodology, source material and logs.  Out of curiosity
I've disassembled the compiled functions in fibn.el for comp-speed 2;
I'm not fluent enough in assembly to spot anything obviously wrong[5].

Let me know if there are things you would like me to try out in order to
investigate this.  

[1] Commit 92cf4bb8cc3da81f4877a734b9e9089ac4b89e85.

[2] Samsung NC10, 2GB RAM, 2GB HDD swap, on Debian Buster.

[3] Commit 9d8fc3a598090da518fcdd5c0503ed0f7faa41a9.

[4] .eln files compiled with:

    (dotimes (i 4) (rename-file (let ((comp-speed i)) (native-compile "fibn.el"))
                                (format "fibn-%d.eln" i)))

    Script used to collect measurements on branch native-comp:

    Script used to collect measurements on branch master:

    Script used to plot the graphs:

    Measurements:

    Logs for native-comp:

    Logs for master:

[5]