GNU bug report logs - #66381
29.1; Auto-revert not polling files when notifications are enabled

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 66381 in the body.
You can then email your comments to 66381 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
To: bug-gnu-emacs <at> gnu.org
Subject: 29.1; Auto-revert not polling files when notifications are enabled
Date: Fri, 6 Oct 2023 13:13:40 -0400

I use a network filesystem which sometimes has to restart, e.g.  for
updates or when credentials expire. Normally, file notifications work
fine on this filesystem, but after a restart the old notifications will
never fire. Documentation says "By default, Auto Revert mode will poll
files for changes periodically even when file notifications are used." -
experimentally the file is never polled.

Ideally the notification would be recreated, but falling back to
polling would be
an improvement.

Reproduce from emacs -Q:

(global-auto-revert-mode t)
Open a file on the filesystem where notifications break
Append to the file externally -> emacs notices immediately
Restart the daemon, breaking notifications
Append to the file externally -> emacs never notices

This has been broken for a while. From auto-revert-handler:

         (revert
          (if buffer-file-name
              (and (or auto-revert-remote-files
                       (not (file-remote-p buffer-file-name)))
                   (or (not auto-revert-notify-watch-descriptor)
                       auto-revert-notify-modified-p)
                 ... eventually poll

Prior to bug#20943, there was a call to buffer-modified-p at the top
level that checked unconditionally whether the file had been modified.

Build:

In GNU Emacs 29.1 (build 1, x86_64-pc-linux-gnu, GTK+ Version 3.24.37,
 cairo version 1.16.0) of 2023-09-03, modified by Debian built on
 kokoro-ubuntu
System Description: Debian GNU/Linux rodete

There's a bunch of local bits in our Emacs build, but I'm ~mostly sure
they are unrelated
to this bug.

-- 
Thanks,
Daniel

Message #8 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>,
 Michael Albinus <michael.albinus <at> gmx.de>
Cc: 66381 <at> debbugs.gnu.org
Subject: Re: bug#66381: 29.1;
 Auto-revert not polling files when notifications are enabled
Date: Sat, 07 Oct 2023 09:17:01 +0300

> From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
> Date: Fri, 6 Oct 2023 13:13:40 -0400
> 
> I use a network filesystem which sometimes has to restart, e.g.  for
> updates or when credentials expire. Normally, file notifications work
> fine on this filesystem, but after a restart the old notifications will
> never fire. Documentation says "By default, Auto Revert mode will poll
> files for changes periodically even when file notifications are used." -
> experimentally the file is never polled.
> 
> Ideally the notification would be recreated, but falling back to
> polling would be
> an improvement.

Michael, can we reliably know when the watch handles are stale?  If
not, then users of such "restarting" network filesystems will need to
disable auto-revert-use-notify.

Message #11 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 66381 <at> debbugs.gnu.org, Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Sat, 07 Oct 2023 17:15:43 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

Hi Eli,

>> I use a network filesystem which sometimes has to restart, e.g.  for
>> updates or when credentials expire. Normally, file notifications work
>> fine on this filesystem, but after a restart the old notifications will
>> never fire. Documentation says "By default, Auto Revert mode will poll
>> files for changes periodically even when file notifications are used." -
>> experimentally the file is never polled.
>> 
>> Ideally the notification would be recreated, but falling back to
>> polling would be
>> an improvement.
>
> Michael, can we reliably know when the watch handles are stale?  If
> not, then users of such "restarting" network filesystems will need to
> disable auto-revert-use-notify.

We have `file-notify-valid-p'. It calls the backend specific funtion,
for example `inotify-valid-p' in the case of GNU/Linux using
inotify. The OP didn't tell us which backend he's using, so let's assume
inotify ATM.

This backend specific check does not ask the operating system, it checks
only some internal data structures. This isn't a robust check for a
stale watch handle.

However, inotify sends also an internal event IN_IGNORED, which is
raised according to inotify(7)

--8<---------------cut here---------------start------------->8---
           IN_IGNORED
                  Watch  was removed explicitly (inotify_rm_watch(2)) or auto‐
                  matically (file was deleted, or filesystem  was  unmounted).
                  See also BUGS.
--8<---------------cut here---------------end--------------->8---

filenotify.el translates this to the Emacs file-notify event `stopped',
and if this event arrives, the respective data structures are
updated. In case of a network filesystem unmount, everything shall work.

A recreation of file notification after a remount is not supported. This
would be a new feature.

However, the situation is more complex I fear. If I understand the OP
message correctly, file notification is used for auto-revert-mode. There
is the user option `auto-revert-notify-exclude-dir-regexp'. If a
directory located on a file name matching this regexp, file notification
is discarded. File names like  "/mnt", "/net/", or "/tmp_mnt/" match. So
in order to use file notification with auto-revert-mode on such a
location, this user option must be adapted.

I haven't tested the combination of watching a mounted file system, and
unmounting them, in autorevert.el. According to
`auto-revert-notify-handler', the file-notify event `stopped' is
handled, and auto-revert-mode continues with polling. I would test this
once I see a clear recipe (starting with

--8<---------------cut here---------------start------------->8---
emacs -Q --eval '(setq auto-revert-debug t file-notify-debug t)'
--8<---------------cut here---------------end--------------->8---

and telling all steps how to enable auto-revert-mode and how to unmount
the filesystem and what happens afterwards. The two debug options show
us the story in the *Messages* buffer.

Best regards, Michael.

Message #14 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when notifications
 are enabled
Date: Sat, 7 Oct 2023 12:14:52 -0400

emacs -Q --eval '(setq auto-revert-debug t file-notify-debug t)'
--eval '(global-auto-revert-mode t)' file.txt

Modify the file outside of Emacs:

file-notify-handle-event (file-notify ((1 . 0) (modify) "file.txt" 0)
file-notify--callback-inotify)
file-notify-callback (1 . 0) changed "/path/to/file.txt" nil
#s(file-notify--watch "/path/to" "file.txt"
auto-revert-notify-handler) "/path/to/file.txt" "/path/to/google3"
auto-revert-notify-handler ((1 . 0) changed "/path/to/file.txt")
Reverting buffer ‘file.txt’

Restart the network filesystem (sudo systemctl restart netfs). Modify
the file outside of Emacs again.

Nothing new in *Messages* for either the restart or the modification.

Hit a key in the file buffer:

file.txt changed on disk; really edit the buffer? (y, n, r or C-h)

Same story with inotify; there's no event when the filesystem
restarts. I'm not explicitly unmounting it, systemd kills and restarts
the job providing the fuse filesystem.

The path does not match auto-revert-notify-exclude-dir-regexp.

If supporting this is of general interest, I'd do it by (A)
re-enabling polling even when using notification; (B) removing and
re-adding the watch when polling reports changes that we weren't
notified about. If it's not of general interest, the best alternative
would be removing the bit of documentation that says "By default, Auto
Revert mode will poll files for changes periodically even when file
notifications are used."

For anyone with a similar problem, since the filesystem restarts are
typically overnight as part of system updates, I went with a simple
workaround:

(run-with-timer 1800 1800 (lambda () (progn
                                       (global-auto-revert-mode 'toggle)
                                       (global-auto-revert-mode 'toggle))))

On Sat, Oct 7, 2023 at 11:15 AM Michael Albinus <michael.albinus <at> gmx.de> wrote:
>
> Eli Zaretskii <eliz <at> gnu.org> writes:
>
> Hi Eli,
>
> >> I use a network filesystem which sometimes has to restart, e.g.  for
> >> updates or when credentials expire. Normally, file notifications work
> >> fine on this filesystem, but after a restart the old notifications will
> >> never fire. Documentation says "By default, Auto Revert mode will poll
> >> files for changes periodically even when file notifications are used." -
> >> experimentally the file is never polled.
> >>
> >> Ideally the notification would be recreated, but falling back to
> >> polling would be
> >> an improvement.
> >
> > Michael, can we reliably know when the watch handles are stale?  If
> > not, then users of such "restarting" network filesystems will need to
> > disable auto-revert-use-notify.
>
> We have `file-notify-valid-p'. It calls the backend specific funtion,
> for example `inotify-valid-p' in the case of GNU/Linux using
> inotify. The OP didn't tell us which backend he's using, so let's assume
> inotify ATM.
>
> This backend specific check does not ask the operating system, it checks
> only some internal data structures. This isn't a robust check for a
> stale watch handle.
>
> However, inotify sends also an internal event IN_IGNORED, which is
> raised according to inotify(7)
>
> --8<---------------cut here---------------start------------->8---
>            IN_IGNORED
>                   Watch  was removed explicitly (inotify_rm_watch(2)) or auto‐
>                   matically (file was deleted, or filesystem  was  unmounted).
>                   See also BUGS.
> --8<---------------cut here---------------end--------------->8---
>
> filenotify.el translates this to the Emacs file-notify event `stopped',
> and if this event arrives, the respective data structures are
> updated. In case of a network filesystem unmount, everything shall work.
>
> A recreation of file notification after a remount is not supported. This
> would be a new feature.
>
> However, the situation is more complex I fear. If I understand the OP
> message correctly, file notification is used for auto-revert-mode. There
> is the user option `auto-revert-notify-exclude-dir-regexp'. If a
> directory located on a file name matching this regexp, file notification
> is discarded. File names like  "/mnt", "/net/", or "/tmp_mnt/" match. So
> in order to use file notification with auto-revert-mode on such a
> location, this user option must be adapted.
>
> I haven't tested the combination of watching a mounted file system, and
> unmounting them, in autorevert.el. According to
> `auto-revert-notify-handler', the file-notify event `stopped' is
> handled, and auto-revert-mode continues with polling. I would test this
> once I see a clear recipe (starting with
>
> --8<---------------cut here---------------start------------->8---
> emacs -Q --eval '(setq auto-revert-debug t file-notify-debug t)'
> --8<---------------cut here---------------end--------------->8---
>
> and telling all steps how to enable auto-revert-mode and how to unmount
> the filesystem and what happens afterwards. The two debug options show
> us the story in the *Messages* buffer.
>
> Best regards, Michael.



-- 
Thanks,
Daniel

Message #17 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Sat, 07 Oct 2023 19:02:47 +0200

Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com> writes:

Hi Daniel,

> emacs -Q --eval '(setq auto-revert-debug t file-notify-debug t)'
> --eval '(global-auto-revert-mode t)' file.txt
>
> Modify the file outside of Emacs:
>
> file-notify-handle-event (file-notify ((1 . 0) (modify) "file.txt" 0)
> file-notify--callback-inotify)
> file-notify-callback (1 . 0) changed "/path/to/file.txt" nil
> #s(file-notify--watch "/path/to" "file.txt"
> auto-revert-notify-handler) "/path/to/file.txt" "/path/to/google3"
> auto-revert-notify-handler ((1 . 0) changed "/path/to/file.txt")
> Reverting buffer ‘file.txt’
>
> Restart the network filesystem (sudo systemctl restart netfs). Modify
> the file outside of Emacs again.
>
> Nothing new in *Messages* for either the restart or the modification.

I miss messages like

--8<---------------cut here---------------start------------->8---
file-notify-handle-event (file-notify ((2 . 0) stopped "/net/garfunkel/Multimedia/foo") auto-revert-notify-handler)
auto-revert-notify-handler ((2 . 0) stopped "/net/garfunkel/Multimedia/foo")
--8<---------------cut here---------------end--------------->8---

which I see in my test. "/net/garfunkel/Multimedia" is the mount point.

What is "/path/to/file.txt" exactly, and what is the related mount
point? You might send this information privately, if it is disclosed information.

Best regards, Michael.

Message #20 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Sat, 07 Oct 2023 19:59:36 +0200

Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com> writes:

Hi Daniel,

> It's /google/src/<censored>/file.txt, with the mount
> point on /google/src. Mount details:
>
> srcfsd /google/src fuse.srcfsd
> rw,nosuid,nodev,relatime,user_id=125,group_id=0,default_permissions,allow_other
> 0 0
>
> There's no stop notification generated, whether I restart or kill the
> daemon or just umount -l. inotifywait(1) agrees. I've got no idea what
> parts of this may be local kernel behavior versus FUSE in general, but
> I would guess that FUSE in general is more likely.

Yes, likely this is due to the srcfsd FUSE type. I don't use it, so I
cannot test.

Since inotify doesn't deliver IN_IGNORED (and the file-notify `stopped'
event doesn't appear therefore) we cannot do too much. We have no API
which sends heartbeat checks on watched file systems; such a check could
solve the problem. But there are no plans. Your workaround by restarting
global-auto-revert-mode every night might be the best you could use.

Best regards, Michael.

Message #23 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Cc: 66381 <at> debbugs.gnu.org, michael.albinus <at> gmx.de
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when notifications
 are enabled
Date: Sat, 07 Oct 2023 21:28:13 +0300

> From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
> Date: Sat, 7 Oct 2023 14:14:45 -0400
> Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
> 
> Coming back to my original suggestion: the documentation says that polling is used even if
> notifications are enabled. The implementation does not poll if there's a notification registered.

That last sentence is not true.

Message #26 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Cc: 66381 <at> debbugs.gnu.org, michael.albinus <at> gmx.de
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when notifications
 are enabled
Date: Sat, 07 Oct 2023 22:00:39 +0300

> From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
> Date: Sat, 7 Oct 2023 14:41:35 -0400
> Cc: michael.albinus <at> gmx.de, 66381 <at> debbugs.gnu.org
> 
> In auto-revert-handler:
> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/autorevert.el#n779
> 
>           (if buffer-file-name
>               (and (or auto-revert-remote-files
>                        (not (file-remote-p buffer-file-name)))
>                    (or (not auto-revert-notify-watch-descriptor)
>                        auto-revert-notify-modified-p)
>                    (if auto-revert-tail-mode
>                        (and (file-readable-p buffer-file-name)
>                             (/= auto-revert-tail-pos
>                                 (setq size
>                                       (file-attribute-size
>                                        (file-attributes buffer-file-name)))))
>                      (funcall (or buffer-stale-function
>                                   #'buffer-stale--default-function)
>                               t)))
> 
> When buffer-file-name, revert is true iff:
> 
> 1. auto-revert-remote-files or the file is not remote
> AND 2. there is no watch descriptor or a notification was received
> AND 3. some details about auto-revert-tail-mode OR t
> 
> If auto-revert-notify-watch-descriptor and not
> auto-revert-notify-modified-p, then the file won't be reverted.
> auto-revert-handler does get called by the polling timer, but it
> doesn't revert.

That's not what your sentence said, which I said wasn't true.  You
said something much more radical:

  The implementation does not poll if there's a notification registered.

Moreover, the documentation says that "polling is used even if
notifications are enabled", and that is true regardless of whether the
file is actually reverted or not.

IOW, the documentation describes the usual case, where notifications
are enabled and the watch descriptor is valid.  It says nothing at all
about your situation.

Message #29 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when notifications
 are enabled
Date: Sat, 7 Oct 2023 14:14:45 -0400

[Message part 1 (text/plain, inline)]

Coming back to my original suggestion: the documentation says that polling
is used even if notifications are enabled. The implementation does not poll
if there's a notification registered. Which behavior is intended?

On Sat, Oct 7, 2023 at 1:59 PM Michael Albinus <michael.albinus <at> gmx.de>
wrote:

> Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com> writes:
>
> Hi Daniel,
>
> > It's /google/src/<censored>/file.txt, with the mount
> > point on /google/src. Mount details:
> >
> > srcfsd /google/src fuse.srcfsd
> >
> rw,nosuid,nodev,relatime,user_id=125,group_id=0,default_permissions,allow_other
> > 0 0
> >
> > There's no stop notification generated, whether I restart or kill the
> > daemon or just umount -l. inotifywait(1) agrees. I've got no idea what
> > parts of this may be local kernel behavior versus FUSE in general, but
> > I would guess that FUSE in general is more likely.
>
> Yes, likely this is due to the srcfsd FUSE type. I don't use it, so I
> cannot test.
>
> Since inotify doesn't deliver IN_IGNORED (and the file-notify `stopped'
> event doesn't appear therefore) we cannot do too much. We have no API
> which sends heartbeat checks on watched file systems; such a check could
> solve the problem. But there are no plans. Your workaround by restarting
> global-auto-revert-mode every night might be the best you could use.
>
> Best regards, Michael.
>


-- 
Thanks,
Daniel

[Message part 2 (text/html, inline)]

Message #32 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 66381 <at> debbugs.gnu.org, michael.albinus <at> gmx.de
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when notifications
 are enabled
Date: Sat, 7 Oct 2023 14:41:35 -0400

Isn't it?

In auto-revert-handler:
https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/autorevert.el#n779

          (if buffer-file-name
              (and (or auto-revert-remote-files
                       (not (file-remote-p buffer-file-name)))
                   (or (not auto-revert-notify-watch-descriptor)
                       auto-revert-notify-modified-p)
                   (if auto-revert-tail-mode
                       (and (file-readable-p buffer-file-name)
                            (/= auto-revert-tail-pos
                                (setq size
                                      (file-attribute-size
                                       (file-attributes buffer-file-name)))))
                     (funcall (or buffer-stale-function
                                  #'buffer-stale--default-function)
                              t)))

When buffer-file-name, revert is true iff:

1. auto-revert-remote-files or the file is not remote
AND 2. there is no watch descriptor or a notification was received
AND 3. some details about auto-revert-tail-mode OR t

If auto-revert-notify-watch-descriptor and not
auto-revert-notify-modified-p, then the file won't be reverted.
auto-revert-handler does get called by the polling timer, but it
doesn't revert.


On Sat, Oct 7, 2023 at 2:28 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
>
> > From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
> > Date: Sat, 7 Oct 2023 14:14:45 -0400
> > Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
> >
> > Coming back to my original suggestion: the documentation says that polling is used even if
> > notifications are enabled. The implementation does not poll if there's a notification registered.
>
> That last sentence is not true.



-- 
Thanks,
Daniel

Message #35 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 66381 <at> debbugs.gnu.org, michael.albinus <at> gmx.de
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when notifications
 are enabled
Date: Sat, 7 Oct 2023 14:48:24 -0400

[Message part 1 (text/plain, inline)]

I definitely got that last bit about buffer-stale-function, but I think my
point still stands - the file isn't checked.

On Sat, Oct 7, 2023, 2:41 PM Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
wrote:

> Isn't it?
>
> In auto-revert-handler:
> https://git.savannah.gnu.org/cgit/emacs.git/tree/lisp/autorevert.el#n779
>
>           (if buffer-file-name
>               (and (or auto-revert-remote-files
>                        (not (file-remote-p buffer-file-name)))
>                    (or (not auto-revert-notify-watch-descriptor)
>                        auto-revert-notify-modified-p)
>                    (if auto-revert-tail-mode
>                        (and (file-readable-p buffer-file-name)
>                             (/= auto-revert-tail-pos
>                                 (setq size
>                                       (file-attribute-size
>                                        (file-attributes
> buffer-file-name)))))
>                      (funcall (or buffer-stale-function
>                                   #'buffer-stale--default-function)
>                               t)))
>
> When buffer-file-name, revert is true iff:
>
> 1. auto-revert-remote-files or the file is not remote
> AND 2. there is no watch descriptor or a notification was received
> AND 3. some details about auto-revert-tail-mode OR t
>
> If auto-revert-notify-watch-descriptor and not
> auto-revert-notify-modified-p, then the file won't be reverted.
> auto-revert-handler does get called by the polling timer, but it
> doesn't revert.
>
>
> On Sat, Oct 7, 2023 at 2:28 PM Eli Zaretskii <eliz <at> gnu.org> wrote:
> >
> > > From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
> > > Date: Sat, 7 Oct 2023 14:14:45 -0400
> > > Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
> > >
> > > Coming back to my original suggestion: the documentation says that
> polling is used even if
> > > notifications are enabled. The implementation does not poll if there's
> a notification registered.
> >
> > That last sentence is not true.
>
>
>
> --
> Thanks,
> Daniel
>

[Message part 2 (text/html, inline)]

Message #38 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 66381 <at> debbugs.gnu.org, Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Sun, 08 Oct 2023 10:24:32 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

Hi,

>>           (if buffer-file-name
>>               (and (or auto-revert-remote-files
>>                        (not (file-remote-p buffer-file-name)))
>>                    (or (not auto-revert-notify-watch-descriptor)
>>                        auto-revert-notify-modified-p)
>>                    (if auto-revert-tail-mode
>>                        (and (file-readable-p buffer-file-name)
>>                             (/= auto-revert-tail-pos
>>                                 (setq size
>>                                       (file-attribute-size
>>                                        (file-attributes buffer-file-name)))))
>>                      (funcall (or buffer-stale-function
>>                                   #'buffer-stale--default-function)
>>                               t)))
>>
>> When buffer-file-name, revert is true iff:
>>
>> 1. auto-revert-remote-files or the file is not remote
>> AND 2. there is no watch descriptor or a notification was received
>> AND 3. some details about auto-revert-tail-mode OR t
>>
>> If auto-revert-notify-watch-descriptor and not
>> auto-revert-notify-modified-p, then the file won't be reverted.
>> auto-revert-handler does get called by the polling timer, but it
>> doesn't revert.
>
> That's not what your sentence said, which I said wasn't true.  You
> said something much more radical:
>
>   The implementation does not poll if there's a notification registered.
>
> Moreover, the documentation says that "polling is used even if
> notifications are enabled", and that is true regardless of whether the
> file is actually reverted or not.
>
> IOW, the documentation describes the usual case, where notifications
> are enabled and the watch descriptor is valid.  It says nothing at all
> about your situation.

The problem is, that in Daniel's case we get no information that file
notification has ceased to work for the re-mounted file system (no
IN_IGNORED event from inotify). auto-revert-notify-watch-descriptor
still exists, and auto-revert-notify-modified-p isn't modified any
longer. Therefore the file is not reverted even while polling.

You have a wokaround which seems to work. Another approach would be use
polling only for the given mounted file system. Adding "^/google/src" to
auto-revert-notify-exclude-dir-regexp might work.

Best regards, Michael.

Message #41 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 66381 <at> debbugs.gnu.org, Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Sun, 08 Oct 2023 14:38:22 +0200

[Message part 1 (text/plain, inline)]

Michael Albinus <michael.albinus <at> gmx.de> writes:

Hi,

> The problem is, that in Daniel's case we get no information that file
> notification has ceased to work for the re-mounted file system (no
> IN_IGNORED event from inotify). auto-revert-notify-watch-descriptor
> still exists, and auto-revert-notify-modified-p isn't modified any
> longer. Therefore the file is not reverted even while polling.

I've debugged further, and it looks like there is a gap in forwarding
the IN_IGNORED event from inotify.c to filenotify.el. Oops.

Daniel, could you pls try the appended patch? It shall enable this to
work, meaning auto-revert-mode is informed about the unmounted file
system. I've tested with an NFS mount, 'tho.

Best regards, Michael.

[Message part 2 (text/x-patch, attachment)]

Message #44 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when notifications
 are enabled
Date: Mon, 9 Oct 2023 14:17:17 -0400

Thanks for the follow-up!

First of all, for anyone who stumbles on this, inotifywait's
documentation says it listens to all events by default. But it doesn't
request IN_UNMOUNT unless you use `-e unmount`. That confused all of
my test results - that's why I thought the event was never generated.
In possession of this new knowledge, I verified that an unmount event
is delivered to inotifywait when my filesystem daemon restarts. So
that's great progress.

I built a patched emacs and tried unmounting the filesystem. Emacs was notified:

file-notify-handle-event (file-notify ((1 . 0) (unmount isdir)
"/google/src/cloud/dmj/emacs/google3" 0)
file-notify--callback-inotify)
file-notify-handle-event (file-notify ((1 . 0) stopped
"/google/src/cloud/dmj/emacs/google3/file.txt")
auto-revert-notify-handler)
auto-revert-notify-handler ((1 . 0) stopped
"/google/src/cloud/dmj/emacs/google3/file.txt")
file-notify-handle-event (file-notify ((1 . 0) stopped
"/google/src/cloud/dmj/emacs/google3/file.txt")
auto-revert-notify-handler)
auto-revert-notify-handler ((1 . 0) stopped
"/google/src/cloud/dmj/emacs/google3/file.txt")
file-notify-handle-event (file-notify ((1 . 0) (ignored)
"/google/src/cloud/dmj/emacs/google3" 0)
file-notify--callback-inotify)

It works! Thanks!

On Sun, Oct 8, 2023 at 8:38 AM Michael Albinus <michael.albinus <at> gmx.de> wrote:
>
> Michael Albinus <michael.albinus <at> gmx.de> writes:
>
> Hi,
>
> > The problem is, that in Daniel's case we get no information that file
> > notification has ceased to work for the re-mounted file system (no
> > IN_IGNORED event from inotify). auto-revert-notify-watch-descriptor
> > still exists, and auto-revert-notify-modified-p isn't modified any
> > longer. Therefore the file is not reverted even while polling.
>
> I've debugged further, and it looks like there is a gap in forwarding
> the IN_IGNORED event from inotify.c to filenotify.el. Oops.
>
> Daniel, could you pls try the appended patch? It shall enable this to
> work, meaning auto-revert-mode is informed about the unmounted file
> system. I've tested with an NFS mount, 'tho.
>
> Best regards, Michael.
>


-- 
Thanks,
Daniel

Message #47 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com>
Cc: 66381 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Tue, 10 Oct 2023 20:03:48 +0200

Daniel Jacobowitz <daniel.jacobowitz <at> gmail.com> writes:

Hi Daniel,

> First of all, for anyone who stumbles on this, inotifywait's
> documentation says it listens to all events by default. But it doesn't
> request IN_UNMOUNT unless you use `-e unmount`.

Noted.

> I built a patched emacs and tried unmounting the filesystem. Emacs was notified:
>
> file-notify-handle-event (file-notify ((1 . 0) (unmount isdir)
> "/google/src/cloud/dmj/emacs/google3" 0)
> file-notify--callback-inotify)
> file-notify-handle-event (file-notify ((1 . 0) stopped
> "/google/src/cloud/dmj/emacs/google3/file.txt")
> auto-revert-notify-handler)
> auto-revert-notify-handler ((1 . 0) stopped
> "/google/src/cloud/dmj/emacs/google3/file.txt")
> file-notify-handle-event (file-notify ((1 . 0) stopped
> "/google/src/cloud/dmj/emacs/google3/file.txt")
> auto-revert-notify-handler)
> auto-revert-notify-handler ((1 . 0) stopped
> "/google/src/cloud/dmj/emacs/google3/file.txt")
> file-notify-handle-event (file-notify ((1 . 0) (ignored)
> "/google/src/cloud/dmj/emacs/google3" 0)
> file-notify--callback-inotify)
>
> It works! Thanks!

Thanks for the feedback.

I've checked the other file notification backends, and it turns out,
that none of them reports unmount events. So I have extended the patch,
the unmount event is propagated now in all backends (under different
event names), and it causes to stop the respective file notification
watch.

This works for all but the w32notify backend. I dont know whether this
library supports a kind of unmount events, and even in case it does, I
wouldn't be able to implement this. Eli, do we want to do something
here?

Everything is pushed to the master branch. Perhaps we could also add a
test case to filenotify-tests.el, but I have no idea how to emulate
unmounting in a test environment.

Best regards, Michael.

Message #50 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 66381 <at> debbugs.gnu.org, daniel.jacobowitz <at> gmail.com
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Fri, 13 Oct 2023 15:05:21 +0300

> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: Eli Zaretskii <eliz <at> gnu.org>,  66381 <at> debbugs.gnu.org
> Date: Tue, 10 Oct 2023 20:03:48 +0200
> 
> I've checked the other file notification backends, and it turns out,
> that none of them reports unmount events. So I have extended the patch,
> the unmount event is propagated now in all backends (under different
> event names), and it causes to stop the respective file notification
> watch.
> 
> This works for all but the w32notify backend. I dont know whether this
> library supports a kind of unmount events, and even in case it does, I
> wouldn't be able to implement this. Eli, do we want to do something
> here?

I don't know, I will have to test this.

Could you please help me by showing some Lisp I could use to see if
unmounting a volume gets reported in some way to the
file-notifications machinery?  I have very little free time these
days, for more than one good reason, but would like not to postpone
investigating this for too long, which will happen if I need to find
the recipe myself.  TIA.

Message #53 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 66381 <at> debbugs.gnu.org, daniel.jacobowitz <at> gmail.com
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Sun, 15 Oct 2023 15:58:56 +0200

Eli Zaretskii <eliz <at> gnu.org> writes:

Hi Eli,

>> I've checked the other file notification backends, and it turns out,
>> that none of them reports unmount events. So I have extended the patch,
>> the unmount event is propagated now in all backends (under different
>> event names), and it causes to stop the respective file notification
>> watch.
>>
>> This works for all but the w32notify backend. I dont know whether this
>> library supports a kind of unmount events, and even in case it does, I
>> wouldn't be able to implement this. Eli, do we want to do something
>> here?
>
> I don't know, I will have to test this.
>
> Could you please help me by showing some Lisp I could use to see if
> unmounting a volume gets reported in some way to the
> file-notifications machinery?  I have very little free time these
> days, for more than one good reason, but would like not to postpone
> investigating this for too long, which will happen if I need to find
> the recipe myself.  TIA.

I would apply the following recipe:

- Use a mounted share. In my environment, "z:" is a Samba mount.

- Call

--8<---------------cut here---------------start------------->8---
$ emacs -l filenotify --eval '(setq file-notify-debug t)' --eval '(file-notify-add-watch "z:/123" (quote (change attribute-change)) (quote ignore))'
--8<---------------cut here---------------end--------------->8---

- In another shell outside Emacs, call

--8<---------------cut here---------------start------------->8---
$ touch z:/123
--8<---------------cut here---------------end--------------->8---

- You'll see in Emacs the messages

--8<---------------cut here---------------start------------->8---
file-notify-handle-event (file-notify (541405157192 modified "123") file-notify--callback-w32notify)
file-notify-callback 541405157192 changed "z:/123" nil #s(file-notify--watch "z:/" "123" ignore) "z:/123" "z:/"
--8<---------------cut here---------------end--------------->8---

- Unmount the share. There shall be another file notification, but there
  isn't.

In a similar scenario I use with Emacs/inotify, I see the meesages

--8<---------------cut here---------------start------------->8---
file-notify-handle-event (file-notify ((1 . 0) (attrib) "123" 0) file-notify--callback-inotify)
file-notify-callback (1 . 0) attribute-changed "/tmp/tramp.sshfs.detlef/home/albinus/123" nil #s(file-notify--watch "/tmp/tramp.sshfs.detlef/home/albinus" "123" ignore) "/tmp/tramp.sshfs.detlef/home/albinus/123" "/tmp/tramp.sshfs.detlef/home/albinus"
file-notify-handle-event (file-notify ((1 . 0) (unmount isdir) "/tmp/tramp.sshfs.detlef/home/albinus" 0) file-notify--callback-inotify)
file-notify-handle-event (file-notify ((1 . 0) stopped "/tmp/tramp.sshfs.detlef/home/albinus/123") ignore)
file-notify-handle-event (file-notify ((1 . 0) (ignored) "/tmp/tramp.sshfs.detlef/home/albinus" 0) file-notify--callback-inotify)
--8<---------------cut here---------------end--------------->8---

"/tmp/tramp.sshfs.detlef" is the mount point in that case.

Best regards, Michael.

Message #56 received at 66381 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Michael Albinus <michael.albinus <at> gmx.de>
Cc: 66381 <at> debbugs.gnu.org, daniel.jacobowitz <at> gmail.com
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Mon, 16 Oct 2023 15:38:26 +0300

> From: Michael Albinus <michael.albinus <at> gmx.de>
> Cc: daniel.jacobowitz <at> gmail.com,  66381 <at> debbugs.gnu.org
> Date: Sun, 15 Oct 2023 15:58:56 +0200
> 
> - Use a mounted share. In my environment, "z:" is a Samba mount.
> 
> - Call
> 
> --8<---------------cut here---------------start------------->8---
> $ emacs -l filenotify --eval '(setq file-notify-debug t)' --eval '(file-notify-add-watch "z:/123" (quote (change attribute-change)) (quote ignore))'
> --8<---------------cut here---------------end--------------->8---
> 
> - In another shell outside Emacs, call
> 
> --8<---------------cut here---------------start------------->8---
> $ touch z:/123
> --8<---------------cut here---------------end--------------->8---
> 
> - You'll see in Emacs the messages
> 
> --8<---------------cut here---------------start------------->8---
> file-notify-handle-event (file-notify (541405157192 modified "123") file-notify--callback-w32notify)
> file-notify-callback 541405157192 changed "z:/123" nil #s(file-notify--watch "z:/" "123" ignore) "z:/123" "z:/"
> --8<---------------cut here---------------end--------------->8---
> 
> - Unmount the share. There shall be another file notification, but there
>   isn't.

On Windows, I cannot safely unmount the volume: Windows tells me that
it cannot be safely unmounted (because watching a filer on it has a
handle open on the volume).  If I unmount the volume forcibly, I see
what you expected me to see: nothing.

I guess this means the "cannot safely unmount" message will have to do
on MS-Windows.

Message #61 received at 66381-done <at> debbugs.gnu.org (full text, mbox):

From: Michael Albinus <michael.albinus <at> gmx.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 66381-done <at> debbugs.gnu.org, daniel.jacobowitz <at> gmail.com
Subject: Re: bug#66381: 29.1; Auto-revert not polling files when
 notifications are enabled
Date: Mon, 16 Oct 2023 16:29:51 +0200

Version: 30.1

Eli Zaretskii <eliz <at> gnu.org> writes:

Hi Eli,

> On Windows, I cannot safely unmount the volume: Windows tells me that
> it cannot be safely unmounted (because watching a filer on it has a
> handle open on the volume).  If I unmount the volume forcibly, I see
> what you expected me to see: nothing.
>
> I guess this means the "cannot safely unmount" message will have to do
> on MS-Windows.

Yes, might be OK. For inotify I have a similar issue with NFS
mounts. That's why I've used an FUSE mount in my example.

Nothing left to do, so I'm closing the bug.

Best regards, Michael.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.