GNU bug report logs - #68266
[PATCH 0/7] Memoize packages associated with cross building.

Previous Next

Full log

Message #47 received at 68266 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: Christopher Baines <mail <at> cbaines.net>
Cc: Josselin Poiret <dev <at> jpoiret.xyz>, Tobias Geerinckx-Rice <me <at> tobias.gr>,
 Simon Tournier <zimon.toutoune <at> gmail.com>, Mathieu Othacehe <othacehe <at> gnu.org>,
 68266 <at> debbugs.gnu.org, Ricardo Wurmus <rekado <at> elephly.net>,
 Christopher Baines <guix <at> cbaines.net>
Subject: Re: [bug#68266] [PATCH v2] guix: store: Add
 report-object-cache-duplication.
Date: Fri, 12 Jan 2024 15:22:07 +0100

Christopher Baines <mail <at> cbaines.net> skribis:

> This is intended to help with spotting duplication in the object cache, so
> where many keys, for example package records map to the same derivation. This
> represents an opportunity for improved performance if you can reduce this
> duplication in the cache, and better take advantage of the already present
> cache entries.

Another way to detect this is by looking at ‘add-data-to-store-cache’
stats:

--8<---------------cut here---------------start------------->8---
$ GUIX_PROFILING="add-data-to-store-cache object-cache" guix build --no-grafts greetd wlgreet du-dust circtools --target=aarch64-linux-gnu -d 
/gnu/store/mivzv83wryv9gp5bjncg5m1831dx2xwr-circtools-1.0.0.drv
/gnu/store/4xf7kh9mi0vpvs8m1ak4x8w1rpsdpv6z-du-dust-0.8.6.drv
/gnu/store/ayk54gvlbc1qam6irzf9kaig56dhzni0-wlgreet-0.4.1.drv
/gnu/store/r601i40cii9ic5w1k4hy5c2yngfayh64-greetd-0.9.0.drv
Object Cache:
  fresh caches:    22
  lookups:      40435
  hits:         36821 (91.1%)
  cache size:    3613 entries

'add-data-to-store' cache:
  lookups:       4090
  hits:           958 (23.4%)
  .drv files:    3062 (74.9%)
  Scheme files:   916 (22.4%)
$ GUIX_PROFILING="add-data-to-store-cache object-cache" ./pre-inst-env guix build --no-grafts greetd wlgreet du-dust circtools --target=aarch64-linux-gnu -d 
/gnu/store/1wsldmvigjb8w2gk418npbnfznlb0ck1-circtools-1.0.0.drv
/gnu/store/b5c73fawjdvkgy431qxz9l6l9y9a9lhz-du-dust-0.8.6.drv
/gnu/store/zwc7qzsbzf62dgbbzy74lki4hsr406bw-wlgreet-0.4.1.drv
/gnu/store/vjdd23hc82701afb132z1ajcqa7hfd74-greetd-0.9.0.drv
Object Cache:
  fresh caches:    22
  lookups:      37942
  hits:         34523 (91.0%)
  cache size:    3418 entries

'add-data-to-store' cache:
  lookups:       3895
  hits:           763 (19.6%)
  .drv files:    2957 (75.9%)
  Scheme files:   826 (21.2%)
--8<---------------cut here---------------end--------------->8---

Ideally, the hit rate there would be 0% and we could remove it.

If there’s a positive hit rate, it means we keep adding the same .drv
and/or *-builder files to the store, meaning that the object cache was
ineffective.

> +(define* (report-object-cache-duplication store #:key (threshold 10)
> +                                          (port (current-error-port)))

Do you have an example output of this?

How helpful does it look to you in practice?

I guess it we’d need to add a ‘register-profiling-hook!’ call somewhere?

If we were to add this, I’d suggest adding a docstring and following the
same style as the rest of the code (no long identifiers for local
variables and no car/cdr, for example).

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.