GNU bug report logs - #23064
PostgreSQL logs to stdout/stderrr, which is lost

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: ludo <at> gnu.org (Ludovic Courtès)
Cc: tracker <at> debbugs.gnu.org
Subject: bug#23064: closed (PostgreSQL logs to stdout/stderrr, which is lost)
Date: Tue, 22 Mar 2016 23:25:02 +0000

[Message part 1 (text/plain, inline)]

Your message dated Wed, 23 Mar 2016 00:23:55 +0100
with message-id <87poumxfjo.fsf <at> gnu.org>
and subject line Re: bug#23064: herd has incomplete status reporting, not so helpful
has caused the debbugs.gnu.org bug report #23064,
regarding PostgreSQL logs to stdout/stderrr, which is lost
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
23064: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=23064
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Danny Milosavljevic <dannym <at> scratchpost.org>
To: bug-guix <at> gnu.org
Subject: herd has incomplete status reporting, not so helpful
Date: Sat, 19 Mar 2016 16:13:37 +0100

So there was a postgresql update where the data format was incompatible.

That happens regularily and is ... acceptable, I guess (even more acceptable in guix because I can easily switch back to the old release).

However, in order to find that out, I had to:

$ psql
... something about a socket not being reachable

$ sudo herd status postgres
Status of postgres:
  It is started.
  Running value is 18438.
  It is enabled.
  Provides (postgres).
  Requires (user-processes loopback).
  Conflicts with ().
  Will be respawned.

So it claimed that everything was just peachy.

$ sudo herd stop postgres
$ sudo herd start postgres

... which reported nothing out of the ordinary either.

$ ps -ef |grep postgres
[xxxx] <defunct>

Aha! finally an indication that something is amiss.

So I tried to find the log entries, to no avail so far.
(I found: /gnu/store/b073csgl56g2wnq7azj4wrswb6azsvc0-postgresql-9.5.1/bin/pg_ctl -D /var/lib/postgresql/data -l logfile start, not sure how that helps)
$ man pg_ctl
No manual entry for pg_ctl

In the mean time, I decided I can just start up the postgres server process manually and it will tell me.

$ postgres --config-file=/etc/postgresql.conf -D /var/lib/postgresql/data
FATAL:  data directory "/var/lib/postgresql/data" has wrong ownership
HINT:  The server must be started by the user that owns the data directory.

$ sudo -i
# su - postgres
Password:

errr... what password? 

# passwd postgres
...
# su - postgres
Password:
su: Authentication failure

# passwd -u postgres
# su - postgres
Password:
This account is currently not available.

Hmmm...

$ sudo -u postgres postgres --config-file=/etc/postgresql.conf -D /var/lib/postgresql/data
FATAL:  database files are incompatible with server
DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.5.1.
$ 

Aha! It says so right there in the standard output (or error?). Where did that message go in the shepherd case?

$ cd /var/log
$ grep -r incompatible .
$ 

???

Anyway, it works again now but we shouldn't have to do this kind of digging.

[Message part 3 (message/rfc822, inline)]

From: ludo <at> gnu.org (Ludovic Courtès)
To: Danny Milosavljevic <dannym <at> scratchpost.org>
Cc: 23064-done <at> debbugs.gnu.org
Subject: Re: bug#23064: herd has incomplete status reporting, not so helpful
Date: Wed, 23 Mar 2016 00:23:55 +0100

Danny Milosavljevic <dannym <at> scratchpost.org> skribis:

>> Could you try this and report back?
>
> $ patch -p1 < ...
> $ sudo ./pre-inst-env guix system reconfigure /etc/config.scm
> $ sudo herd stop postgres
> $ sudo herd start postgres
> Service postgres has been started.
> $ tail -f /var/log/messages 
> Mar 22 20:15:51 localhost nscd: 213 monitored file `/etc/services` was created, adding watch
> Mar 22 20:15:51 localhost nscd: 213 monitored file `/etc/services` was written to
> Mar 22 20:20:40 localhost postgres[2115]: [1-1] FATAL:  database files are incompatible with server
> Mar 22 20:20:40 localhost postgres[2115]: [1-2] DETAIL:  The data directory was initialized by PostgreSQL version 9.3, which is not compatible with this version 9.5.1.

Great, thanks for testing!  Pushed as
9b1cee97a5a75766cb52553111794c758c4f1651.

> Although it would be nice to have shepherd have a field in "herd status" which tells you when it respawned the service in < 1 s intervals (or maybe if it respawned it < 1 s after the initial start) - or maybe just how long ago it was (last) started and when the actual herd start command was :-)

In trying to implement that, I realized there’s a bug in respawn delay
handling.  More on that later.

Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.