GNU bug report logs -
#19180
vacuum_weak_hash_table error
Previous Next
Full log
Message #107 received at 19180 <at> debbugs.gnu.org (full text, mbox):
Heya!
Andy Wingo <wingo <at> igalia.com> skribis:
> On Tue 31 Oct 2017 00:13, ludo <at> gnu.org (Ludovic Courtès) writes:
>
>> I built libguile with the patch (I haven’t yet taken the time to rebuild
>> all of Guile). It works, but unfortunately it still shows quick growth
>> of the heap on this example (<https://bugs.gnu.org/28590>):
>
> Fixed in attached patches (on top of the other one). This was a race
> between the periodic vacuum process, which should run after GC via a
> finalizer, and the mutator. If the mutator had the weak table lock, the
> vacuum would never be run. Of course in the test below, the mutator
> usually has the table lock, so we ended up often skipping the vacuum,
> which causes the table size to grow, which causes the active heap size
> to grow, which causes the bytes-allocated-before-GC to increase, and
> which ultimately is a vicious circle.
>
> In my tests this patch fixes the issue, though the level at which the
> heap stabilizes can vary slightly as there's a bit of nondeterministic
> concurrency as the mutator and the vacuum process still race a bit.
> Right now for me it stabilizes at about 6.2M of heap.
Yes. I can confirm that it fixes this issue.
One of my benchmarks is to ‘read’ the 26M file that contains the
CPS-as-sexps representation of gnu/packages/python.scm, with
source-location recording on (thus with a big source property table).
Compared to 2.2.2, execution time is almost divided by two; heap size is
usually 540M and 620M, whereas it varies between 560M and 920M with
2.2.2:
--8<---------------cut here---------------start------------->8---
ludo <at> ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.23 10.00 0.06 0.00 0.00 5.17
;;; (#<weak-table 1863537/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 780.03 MiB
ludo <at> ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.29 9.94 0.04 0.00 0.00 4.97
;;; (#<weak-table 1820831/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 660.36 MiB
ludo <at> ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.11 9.33 0.05 0.00 0.00 3.90
;;; (#<weak-table 1829710/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 562.21 MiB
ludo <at> ribbon ~/src/guix$ guile profile-cps-read.scm
clock utime stime cutime cstime gctime
8.23 9.90 0.08 0.00 0.00 5.22
;;; (#<weak-table 1828628/3595271>)
with Guile 2.2.2:
heap-size from 2.53 MiB to 918.09 MiB
ludo <at> ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.68 6.71 0.02 0.00 0.00 3.88
;;; (#<weak-table 1799062/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 540.54 MiB
ludo <at> ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.73 6.77 0.05 0.00 0.00 4.03
;;; (#<weak-table 1801035/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 620.22 MiB
ludo <at> ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.69 6.80 0.06 0.00 0.00 3.94
;;; (#<weak-table 1796895/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 573.36 MiB
ludo <at> ribbon ~/src/guix$ /data/src/guile-2.1/meta/guile profile-cps-read.scm
clock utime stime cutime cstime gctime
4.67 6.81 0.06 0.00 0.00 4.05
;;; (#<weak-table 1797102/3595271>)
with Guile 2.2.2.17-8069-dirty:
heap-size from 1.89 MiB to 547.39 MiB
--8<---------------cut here---------------end--------------->8---
Another benchmark is the ‘emit-bytecode’ procedure for this CPS, without
optimizations turned off (like ‘%lightweight-optimizations’ in Guix.)
The heap size after the ‘emit-bytecode’ call is still at 1.3G (338M
before the call), about the same as with 2.2.2. That’s not surprising
because I think memory consumption comes from the data structures used
at that compilation stage, as discussed before.
The wall-clock time is ~45s, whereas it’s ~54s with 2.2.2.
In other news, I’ve rebuilt 2.2.2 + these patches with Guix, and
everything went fine (Guile processes seem to peak at ~150M resident
when compiling).
So, all the lights are green on my side!
Thanks a whole lot for coming up with this solution!
Ludo’.
This bug report was last modified 7 years and 209 days ago.
Previous Next
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997,2003 nCipher Corporation Ltd,
1994-97 Ian Jackson.