Subject: bug#30644: Cuirass runs out of build users

From: Andreas Enge <[email-masked]>
Date: Mon, 5 Mar 2018 16:53:51 +0100

Right now, I do not seem to be running out of build users, but I think
that cuirass nevertheless behaves strangely; so I am asking here instead
of opening a new bug (maybe there is no bug, actually).

I reconfigured bayfront this morning with the latest git checkout.
Right now, there are only two packages built in /tmp, if at all:
$ ll /tmp/
total 8
drwx------ 4 guixbuilder02 guixbuild 4096 Mar  5 16:06 guix-build-guile-emacs-20150512.41120e0.drv-0/
drwx------ 5 guixbuilder01 guixbuild 4096 Mar  5 15:05 guix-build-r-mgcv-1.8-23.drv-0/
(It is now 16:54, but "top" shows emacs and r activity, so my guess
is these packages are being built.)
But in any case, we are far from the 10 jobs I configured.

I am attaching the result of
$ grep -v evaluate cuirass.log|grep -v ^\`/gnu/store|grep -v spec|grep -v "next evaluation"|grep -v heap > ~/purged.log
which contains the non-trivial lines of /var/log/cuirass.log.

There is still one occurrence of this:
In ice-9/boot-9.scm:
    829:9  7 (catch _ _ #<procedure 7f49ffb6c9e0 at cuirass/base.sc?> ?)
In cuirass/base.scm:
   616:26  6 (_)
    548:2  5 (build-packages _ #<<sqlite-db> pointer: #<pointer 0x4?> ?)
   426:10  4 (spawn-builds #<build-daemon 256.97 42d2320> #<<sqlite?> ?)
In srfi/srfi-1.scm:
    640:9  3 (for-each #<procedure update! (drv)> _)
In cuirass/database.scm:
    69:18  2 (%sqlite-exec _ _ . _)
In ice-9/eval.scm:
   293:34  1 (_ #(#(#<directory (sqlite3) 1b91820> #<<sqlite-stm?> ?)))
    619:8  0 (_ #(#(#(#(#<directory (sqlite3) 1b91820> #<?>) #) 5) 5))
ice-9/eval.scm:619:8: Throw to key `sqlite-error' with args `(#f 5 "database is locked")'.

There is this, which looks like a problem, but not a fatal one:
;;; Failed to autoload make-session in (gnutls):
;;; missing interface for module (gnutls)
;;; Failed to autoload connection-end/client in (gnutls):
;;; missing interface for module (gnutls)

And what strikes me most is that between two lines of the kind
   building batch of 200 jobs
there are not 200 lines of successful or failed packages.

And my impression is that too many packages fail. What are these
"status 100" and "status 101"?

Do things look like this on berlin? (Of course, one difference is that
right now the machine builds everything itself without offloading.)

Andreas