GNU bug report logs - #44460
[PATCH] processes: Optionally normalize recutils output.

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 44460 in the body.
You can then email your comments to 44460 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: guix-patches <at> gnu.org
Subject: [PATCH] processes: Optionally normalize recutils output.
Date: Wed, 04 Nov 2020 20:31:01 -0800

[Message part 1 (text/plain, inline)]

Hi Guix,

Please let me know if there is already a way to do what I want without
this patch :). I really don't know a whole lot of recutils beyond what I
learned this morning.

I was trying to extract PIDs of the child processes of sessions from the
output of guix processes.  I ran into the following problem: the PID and
the command are on the same line.

---- Begin ----
$ guix processes
SessionPID: 4278
ClientPID: 4268
ClientCommand: /gnu/store/...
ChildProcess: 4435: /gnu/store/...
ChildProcess: 4554: /gnu/store/...
ChildProcess: 4646: guile --no-auto-compile ...
---- End ----

I ended up having to use sed to remove the trailing command for the
child process.  After reading the documentation for recutils I found out
that records can also be expressed in separate record sets that can be
joined together. 

This patch adds a --normalize flag that specifies the
recutils output be put in separate record sets for Locks, Sessions, and
ChildProcesses.  For example:

---- Begin ----
PAGER=cat ./pre-inst-env guix processes --normalize
%rec: Session
%type: PID int
%type: ClientPID int
%key: PID

PID: 4278
ClientPID: 4268
ClientCommand: /gnu/store/...

%rec: Lock
%type: Session rec Session

%rec: ChildProcess
%type: Session rec Session
%type: PID int
%key: PID

Session: 4278
PID: 4435
Command: /gnu/store/...

Session: 4278
PID: 4554
Command: /gnu/store/...

Session: 4278
PID: 4646
Command: guile --no-auto-compile ...

---- End ----

What do you think?

John

[0001-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #8 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: 44460 <at> debbugs.gnu.org
Subject: processes: Don't normalize Locks
Date: Thu, 05 Nov 2020 07:49:42 -0800

[Message part 1 (text/plain, inline)]

I got a little eager to normalize and put locks in their own record
set. That was unnecessary as records can have multiple of the same field
name. This new patch removes the Lock record set and puts the Lock in
the Session record.

- John

[0001-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #11 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: 44460 <at> debbugs.gnu.org
Subject: Fixup the output of Session
Date: Thu, 05 Nov 2020 08:01:40 -0800

[Message part 1 (text/plain, inline)]

I had too many newlines after the session recordset and too few between
session records.

- John

[0001-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #14 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: 44460 <at> debbugs.gnu.org
Subject: Add copyright lines
Date: Fri, 06 Nov 2020 15:34:13 -0800

[Message part 1 (text/plain, inline)]

Sorry, forgot to add my copyright information.

- John

[0001-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #17 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Soo <jsoo1 <at> asu.edu>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Tue, 10 Nov 2020 23:15:11 +0100

Hi John,

John Soo <jsoo1 <at> asu.edu> skribis:

>>From 48945997d9139ddc5e288512de3dda8d5accaf44 Mon Sep 17 00:00:00 2001
> From: John Soo <jsoo1 <at> asu.edu>
> Date: Wed, 4 Nov 2020 07:51:52 -0800
> Subject: [PATCH] processes: Optionally normalize recutils output.
>
> * guix/scripts/processes.scm: Add "normalize" flag

[...]

> +@table @code
> +@item --normalize
> +Normalize the output records into record sets (@pxref{Record Sets,,,
> +recutils, GNU recutils manual}).  Normalizing into record sets allows
> +joins across record types.
> +
> +@example
> +$ guix processes --normalize | \
> +    recsel \
> +    -j Session \
> +    -t ChildProcess \
> +    -p Session.PID,PID \
> +    -e 'Session.ClientCommand ~ build'
> +Session_PID: 4278
> +PID: 4435
> +
> +Session_PID: 4278
> +PID: 4554
> +
> +Session_PID: 4278
> +PID: 4646
> +@end example
> +@end table

Nice!  Right above the example, I’d suggest adding a sentence like “The
example below lists…” (what does it list actually? :-)).

In the default format, I wonder if we could already change split
‘ChildProcess’ into ‘ChildPID’ and ‘ChildCommand’, as you had initially
proposed on IRC; would that work?

> +(define (lock->record lock port)
> +  (format port "LockHeld: ~a~%" lock))

Maybe ‘lock->recutils’ for consistency and to avoid confusion with
Scheme “records”?

> +(define (format-single-record port)

Maybe ‘daemon-sessions->recutils’?  Should ‘sessions’ be a parameter for
clarity?

> +  "Display denormalized session information to PORT."
> +  (for-each (lambda (session)
> +              (daemon-session->recutils session port)
> +                (newline port))
                  ^
Indentation is off.

> +(define (child-process->normalized-record process port)
> +  "Display PROCESS record on PORT in normalized form"
> +  (format port "PID: ~a" (process-id process))
> +  (newline port)
> +  (format port "Command:~{ ~a~}" (process-command process)))
> +
> +(define (format-normalized port)

Please add a docstring.  Perhaps make ‘sessions’ a parameter?

> +  (define sessions (daemon-sessions))
> +
> +  (format port session-rec-type)

As reported by ‘-Wformat’, passing a non-literal format string is risky;
write this instead:

  (display session-rec-type port)

> +  (newline port)
> +  (newline port)
> +  (for-each
> +   (lambda (session)

Preferable indent ‘for-each’ like so:

  (for-each (lambda (session)

Likewise below.

> +  (display (G_ "
> +  --normalize            display results as normalized record sets"))

Should it be ‘--format=normalized’ (just like we have ‘--format’ in
‘guix describe’, for instance)?

Could you send an updated patch?

Thanks!

Ludo’.

Message #20 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Wed, 11 Nov 2020 09:51:08 -0800

[Message part 1 (text/plain, inline)]

Hello Ludo,

Ludovic Courtès <ludo <at> gnu.org> writes:

>> +@table @code
>> +@item --normalize
>> +Normalize the output records into record sets (@pxref{Record Sets,,,
>> +recutils, GNU recutils manual}).  Normalizing into record sets allows
>> +joins across record types.
>> +
>> +@example
>> +$ guix processes --normalize | \
>> +    recsel \
>> +    -j Session \
>> +    -t ChildProcess \
>> +    -p Session.PID,PID \
>> +    -e 'Session.ClientCommand ~ build'
>> +Session_PID: 4278
>> +PID: 4435
>> +
>> +Session_PID: 4278
>> +PID: 4554
>> +
>> +Session_PID: 4278
>> +PID: 4646
>> +@end example
>> +@end table
>
> Nice!  Right above the example, I’d suggest adding a sentence like “The
> example below lists…” (what does it list actually? :-)).
>
> In the default format, I wonder if we could already change split
> ‘ChildProcess’ into ‘ChildPID’ and ‘ChildCommand’, as you had initially
> proposed on IRC; would that work?

I think we could do that, but I had two reasons to use the normalized
format instead.

* Backwards incompatibility - I didn't want to break any existing scripts
  that may exist.

* Still not normalized - how can I search for just the child processes
  associated with a particular command?

I wouldn't be opposed to splitting ChildProcess into ChildPID and
ChildCommand.  I would like it best if that change was made in addition
to adding the normalized version, since the normalized version allows
more functionality.

>> +(define (lock->record lock port)
>> +  (format port "LockHeld: ~a~%" lock))
>
> Maybe ‘lock->recutils’ for consistency and to avoid confusion with
> Scheme “records”?

Done.

>> +(define (format-single-record port)
>
> Maybe ‘daemon-sessions->recutils’?  Should ‘sessions’ be a parameter for
> clarity?

Much better, thank you. I updated the normalized version too.

>
>> +  "Display denormalized session information to PORT."
>> +  (for-each (lambda (session)
>> +              (daemon-session->recutils session port)
>> +                (newline port))
>                   ^
> Indentation is off.

Fixed.

>> +(define (child-process->normalized-record process port)
>> +  "Display PROCESS record on PORT in normalized form"
>> +  (format port "PID: ~a" (process-id process))
>> +  (newline port)
>> +  (format port "Command:~{ ~a~}" (process-command process)))
>> +
>> +(define (format-normalized port)
>
> Please add a docstring.  Perhaps make ‘sessions’ a parameter?

Done.

>> +  (define sessions (daemon-sessions))
>> +
>> +  (format port session-rec-type)
>
> As reported by ‘-Wformat’, passing a non-literal format string is risky;
> write this instead:
>
>   (display session-rec-type port)

Done.

>> +  (newline port)
>> +  (newline port)
>> +  (for-each
>> +   (lambda (session)
>
> Preferable indent ‘for-each’ like so:
>
>   (for-each (lambda (session)

Done.

> Likewise below.
>
>> +  (display (G_ "
>> +  --normalize            display results as normalized record sets"))
>
> Should it be ‘--format=normalized’ (just like we have ‘--format’ in
> ‘guix describe’, for instance)?

That makes sense. What do you think of

Other changes:

* Updated the record descriptors to include the possible
  fields for sessions. I got some nice guidance from the recutils irc on
  that.

* Put the PID and Command first for the ChildProcess

* Add a --list-formats like guix describe has.

Thanks!

John

[0001-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #23 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Wed, 11 Nov 2020 09:59:34 -0800

[Message part 1 (text/plain, inline)]

I realized I slightly altered the --help format. That is fixed here.

Thanks again,

John

[0001-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #26 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Wed, 11 Nov 2020 10:47:39 -0800

[Message part 1 (text/plain, inline)]

Oops! I also forget to change the flag names in the docs and the commit
message. Fixing those here.

- John

[0001-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #29 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Soo <jsoo1 <at> asu.edu>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Thu, 12 Nov 2020 11:58:18 +0100

Hi John,

John Soo <jsoo1 <at> asu.edu> skribis:


[...]

>>> +Session_PID: 4278
>>> +PID: 4646
>>> +@end example
>>> +@end table
>>
>> Nice!  Right above the example, I’d suggest adding a sentence like “The
>> example below lists…” (what does it list actually? :-)).
>>
>> In the default format, I wonder if we could already change split
>> ‘ChildProcess’ into ‘ChildPID’ and ‘ChildCommand’, as you had initially
>> proposed on IRC; would that work?
>
> I think we could do that, but I had two reasons to use the normalized
> format instead.
>
> * Backwards incompatibility - I didn't want to break any existing scripts
>   that may exist.
>
> * Still not normalized - how can I search for just the child processes
>   associated with a particular command?

Like:

  guix processes | recsel -e 'ClientCommand ~ "xyz"' -p ChildProcess

?

Actually what does “normalized” mean in this context?

> I wouldn't be opposed to splitting ChildProcess into ChildPID and
> ChildCommand.  I would like it best if that change was made in addition
> to adding the normalized version, since the normalized version allows
> more functionality.

I would think it’s OK to break compatibility on just these
“ChildProcess” fields.

> * Updated the record descriptors to include the possible
>   fields for sessions. I got some nice guidance from the recutils irc on
>   that.
>
> * Put the PID and Command first for the ChildProcess
>
> * Add a --list-formats like guix describe has.

Great, I’ll take a look.  Thanks!

Ludo’.

Message #32 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Thu, 12 Nov 2020 07:37:39 -0800

Hi Ludo,

Ludovic Courtès <ludo <at> gnu.org> writes:

>> * Still not normalized - how can I search for just the child processes
>>   associated with a particular command?
>
> Like:
>
>   guix processes | recsel -e 'ClientCommand ~ "xyz"' -p ChildProcess
>
> ?
>
> Actually what does “normalized” mean in this context?

Excellent question. I was thinking along the lines of database
normalization. The default output has multi-valued fields for child
processes, so the idea is to make them their own record set.  Does that
make sense?

An aside - Probably to be entirely honest about normalizing the output,
locks really would be in a separate record set too.

Another challenge is making sure the user can understand what
"normalized" means.  I am not sure readers of the manual/cli help will
be able to infer what it means from context.  On the other hand, it is
such a small use case that it seems imbalanced to provide a lot of
background for the term "normal". What do you think?

>> I wouldn't be opposed to splitting ChildProcess into ChildPID and
>> ChildCommand.  I would like it best if that change was made in addition
>> to adding the normalized version, since the normalized version allows
>> more functionality.
>
> I would think it’s OK to break compatibility on just these
> “ChildProcess” fields.

Ok. Would it be ok if I put that in a separate commit?

Thanks again!

- John

Message #35 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Soo <jsoo1 <at> asu.edu>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Thu, 12 Nov 2020 21:29:19 +0100

Hi,

John Soo <jsoo1 <at> asu.edu> skribis:

> Ludovic Courtès <ludo <at> gnu.org> writes:
>
>>> * Still not normalized - how can I search for just the child processes
>>>   associated with a particular command?
>>
>> Like:
>>
>>   guix processes | recsel -e 'ClientCommand ~ "xyz"' -p ChildProcess
>>
>> ?
>>
>> Actually what does “normalized” mean in this context?
>
> Excellent question. I was thinking along the lines of database
> normalization. The default output has multi-valued fields for child
> processes, so the idea is to make them their own record set.  Does that
> make sense?

Yes it does!  Initially I wondered if it was a term used in recutils,
but apparently it’s not.

> An aside - Probably to be entirely honest about normalizing the output,
> locks really would be in a separate record set too.

Yeah.

> Another challenge is making sure the user can understand what
> "normalized" means.  I am not sure readers of the manual/cli help will
> be able to infer what it means from context.  On the other hand, it is
> such a small use case that it seems imbalanced to provide a lot of
> background for the term "normal". What do you think?

Sure.

Thinking more about it, to me the appeal of recutils is that it’s both
human- and machine-readable.  But here we end up having a specific
machine-readable variant.  But yeah, maybe that’s unavoidable.

>>> I wouldn't be opposed to splitting ChildProcess into ChildPID and
>>> ChildCommand.  I would like it best if that change was made in addition
>>> to adding the normalized version, since the normalized version allows
>>> more functionality.
>>
>> I would think it’s OK to break compatibility on just these
>> “ChildProcess” fields.
>
> Ok. Would it be ok if I put that in a separate commit?

Yes (you mean in addition to ‘-f normalized’, right?).

Thanks,
Ludo’.

Message #38 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Thu, 12 Nov 2020 21:33:52 -0800

[Message part 1 (text/plain, inline)]

Hi Ludo,

Ludovic Courtès <ludo <at> gnu.org> writes:

> Yes it does!  Initially I wondered if it was a term used in recutils,
> but apparently it’s not.

Do you think it is the best term to use?

>> An aside - Probably to be entirely honest about normalizing the output,
>> locks really would be in a separate record set too.
>
> Yeah.

I made Lock a separate record set. This exposes a limitation with
recutils. It is not possible to join more than two record sets together
(though they mentioned it as a goal on IRC).

>> Another challenge is making sure the user can understand what
>> "normalized" means.  I am not sure readers of the manual/cli help will
>> be able to infer what it means from context.  On the other hand, it is
>> such a small use case that it seems imbalanced to provide a lot of
>> background for the term "normal". What do you think?
>
> Sure.
>
> Thinking more about it, to me the appeal of recutils is that it’s both
> human- and machine-readable.  But here we end up having a specific
> machine-readable variant.  But yeah, maybe that’s unavoidable.

I suppose the normalized version is a little less human-readable.  It
would be behind a flag, though.  Is that a reasonable compromise?

After some thought I realize that the normalized version isn't that much
more useful than the default, but it does enable things like:

---- Example ----
$ guix processes -f normalized \
  | recsel \
    -t ChildProcess \
    -j Session \
    -p PID,Session.PID \
  | recfmt '{{PID}} {{Session.PID}}'
23607 2356724713 2356725002 23576
---- Example ----

This will format all the (PID, Session) pairs. Whereas the
non-normalized version would only print one PID given the following.

---- Example ----
$ guix processes \
  | recsel -p ChildPID,SessionPID \
  | recfmt '{{ChildPID}} {{SessionPID}}'
23607 23567
---

>>>> I wouldn't be opposed to splitting ChildProcess into ChildPID and
>>>> ChildCommand.  I would like it best if that change was made in addition
>>>> to adding the normalized version, since the normalized version allows
>>>> more functionality.
>>>
>>> I would think it’s OK to break compatibility on just these
>>> “ChildProcess” fields.
>>
>> Ok. Would it be ok if I put that in a separate commit?
>
> Yes (you mean in addition to ‘-f normalized’, right?).

I changed ChildProcess: pid: command to ChildPID and ChildCommand in the
default in a separate commit.

Thanks again,

John

[0001-processes-Put-ChildProcess-and-ChildPID-on-separate-.patch (text/x-patch, attachment)]

[0002-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #41 received at 44460 <at> debbugs.gnu.org (full text, mbox):

From: John Soo <jsoo1 <at> asu.edu>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: 44460 <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Fri, 13 Nov 2020 08:16:25 -0800

[Message part 1 (text/plain, inline)]

Hello again,

No big changes here, I just forgot to update the manual again.

- John

[0001-processes-Put-ChildProcess-and-ChildPID-on-separate-.patch (text/x-patch, attachment)]

[0002-processes-Optionally-normalize-recutils-output.patch (text/x-patch, attachment)]

Message #46 received at 44460-done <at> debbugs.gnu.org (full text, mbox):

From: Ludovic Courtès <ludo <at> gnu.org>
To: John Soo <jsoo1 <at> asu.edu>
Cc: 44460-done <at> debbugs.gnu.org
Subject: Re: [bug#44460] Add copyright lines
Date: Sun, 29 Nov 2020 23:49:56 +0100

[Message part 1 (text/plain, inline)]

Hi,

John Soo <jsoo1 <at> asu.edu> skribis:

>>From 6082c559d1200e632b3fb45eb0633d28829667a1 Mon Sep 17 00:00:00 2001
> From: John Soo <jsoo1 <at> asu.edu>
> Date: Thu, 12 Nov 2020 21:16:48 -0800
> Subject: [PATCH 1/2] processes: Put ChildProcess and ChildPID on separate
>  lines.
>
> * guix/scripts/processes.scm: Put child process information in
> separate fields.
> * doc/guix.texi: Document change in output of guix processes.

Applied.

>>From becf3a8fee4aea0a49dde47f5728410b97d94fbf Mon Sep 17 00:00:00 2001
> From: John Soo <jsoo1 <at> asu.edu>
> Date: Wed, 4 Nov 2020 07:51:52 -0800
> Subject: [PATCH 2/2] processes: Optionally normalize recutils output.
>
> * guix/scripts/processes.scm: Add "format" and "list-formats" flag.
> * doc/guix.texi: Document new flags.

Applied with the changes below.

I also tweaked the commit logs to list the entities added or modified.

Thanks!

Ludo’.

[Message part 2 (text/x-patch, inline)]

diff --git a/doc/guix.texi b/doc/guix.texi
index b739464853..fcaa2f2b63 100644
--- a/doc/guix.texi
+++ b/doc/guix.texi
@@ -12884,14 +12884,15 @@ Produce output in the specified @var{format}, one of:
 @table @code
 @item recutils
 The default option. It outputs a set of Session recutils records
-that include each ChildProcess as a field.
+that include each @code{ChildProcess} as a field.
 
 @item normalized
 Normalize the output records into record sets (@pxref{Record Sets,,,
 recutils, GNU recutils manual}).  Normalizing into record sets allows
 joins across record types.  The example below lists the PID of each
-ChildProcess and the associated PID for Session that spawned the
-ChildProcess where the Session was started using guix build.
+@code{ChildProcess} and the associated PID for @code{Session} that
+spawned the @code{ChildProcess} where the @code{Session} was started
+using @command{guix build}.
 
 @example
 $ guix processes --format=normalized | \
@@ -12899,7 +12900,7 @@ $ guix processes --format=normalized | \
     -j Session \
     -t ChildProcess \
     -p Session.PID,PID \
-    -e 'Session.ClientCommand ~ "guix build'"
+    -e 'Session.ClientCommand ~ "guix build"'
 PID: 4435
 Session_PID: 4278
 
diff --git a/guix/scripts/processes.scm b/guix/scripts/processes.scm
index bcc541badb..87e687852c 100644
--- a/guix/scripts/processes.scm
+++ b/guix/scripts/processes.scm
@@ -338,8 +338,10 @@ List the current Guix sessions and their processes."))
 (define-command (guix-processes . args)
   (category plumbing)
   (synopsis "list currently running sessions")
+
   (define options
-    (parse-command-line args %options (list %default-options)))
+    (parse-command-line args %options (list %default-options)
+                        #:build-options? #f))
 
   (with-paginated-output-port port
     (match (assoc-ref options 'format)

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.