GNU bug report logs - #75569
30.0.93; Large /proc/* file are read incompletely

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 75569 in the body.
You can then email your comments to 75569 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Daniel Mendler <mail <at> daniel-mendler.de>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.0.93; Large /proc/* file are read incompletely
Date: Wed, 15 Jan 2025 01:21:03 +0100

When opening a large file in /proc/* like /proc/cpuinfo in Emacs via
find-file, the buffer content is incomplete, when comparing the output
to the output produced by /bin/cat. The Eshell command eshell/cat has
the same problem.The proc file system doesn't report file sizes since
the files are dynamically generated. ls -l /proc/cpuinfo reports size 0.
Emacs doesn't handle this well, limiting its reads to 16KiB.

In Eshell:

cat /proc/cpuinfo > #<buffer bug>  ==> truncated buffer (exactly 16KiB)
*cat /proc/cpuinfo > #<buffer bug> ==> complete buffer (larger than 16KiB)

In GNU Emacs 30.0.93 (build 1, x86_64-pc-linux-gnu, X toolkit, cairo
 version 1.18.2, Xaw scroll bars) of 2025-01-14
Windowing system distributor 'The X.Org Foundation', version 11.0.12101015
System Description: Debian GNU/Linux trixie/sid

Configured using:
 'configure --with-tree-sitter
 --with-native-compilation --with-x-toolkit=athena --with-dbus
 --without-selinux --without-threads --without-gsettings --without-gpm
 --with-cairo --with-cairo-xcb --disable-gc-mark-trace --with-xinput2'

Configured features:
CAIRO DBUS FREETYPE GIF GLIB GMP GNUTLS HARFBUZZ JPEG LIBOTF LIBSYSTEMD
LIBXML2 MODULES NATIVE_COMP NOTIFY INOTIFY PDUMPER PNG RSVG SECCOMP
SOUND SQLITE3 TIFF TOOLKIT_SCROLL_BARS TREE_SITTER WEBP X11 XDBE XIM
XINPUT2 XPM LUCID ZLIB

Important settings:
  value of $LANG: en_US.UTF-8
  locale-coding-system: utf-8-unix

Message #8 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Daniel Mendler <mail <at> daniel-mendler.de>
Cc: 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Wed, 15 Jan 2025 09:19:10 +0200

> When opening a large file in /proc/* like /proc/cpuinfo in Emacs via
> find-file, the buffer content is incomplete, when comparing the output
> to the output produced by /bin/cat. The Eshell command eshell/cat has
> the same problem.The proc file system doesn't report file sizes since
> the files are dynamically generated. ls -l /proc/cpuinfo reports size 0.
> Emacs doesn't handle this well, limiting its reads to 16KiB.
>
> In Eshell:
>
> cat /proc/cpuinfo > #<buffer bug>  ==> truncated buffer (exactly 16KiB)
> *cat /proc/cpuinfo > #<buffer bug> ==> complete buffer (larger than 16KiB)

Strange, bug#9800 was closed as fixed, but I confirm the buffer is still
truncated.

Message #11 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Wed, 15 Jan 2025 17:07:56 +0200

> Cc: 75569 <at> debbugs.gnu.org
> From: Juri Linkov <juri <at> linkov.net>
> Date: Wed, 15 Jan 2025 09:19:10 +0200
> 
> > When opening a large file in /proc/* like /proc/cpuinfo in Emacs via
> > find-file, the buffer content is incomplete, when comparing the output
> > to the output produced by /bin/cat. The Eshell command eshell/cat has
> > the same problem.The proc file system doesn't report file sizes since
> > the files are dynamically generated. ls -l /proc/cpuinfo reports size 0.
> > Emacs doesn't handle this well, limiting its reads to 16KiB.
> >
> > In Eshell:
> >
> > cat /proc/cpuinfo > #<buffer bug>  ==> truncated buffer (exactly 16KiB)
> > *cat /proc/cpuinfo > #<buffer bug> ==> complete buffer (larger than 16KiB)
> 
> Strange, bug#9800 was closed as fixed, but I confirm the buffer is still
> truncated.

It was closed because it was fixed back then.  But maybe commits
8650b3308dd and/or 3fb2c174d3a7 broke that later?

The /proc/cpuinfo file on the system to which I have access is smaller
than 16KB, so could someone who can access a larger cpuinfo file
please see if those commits broke the fix, or maybe bisect?

Adding Paul in case he has comments or suggestions.

Message #14 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mail <at> daniel-mendler.de, Paul Eggert <eggert <at> cs.ucla.edu>,
 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Wed, 15 Jan 2025 19:08:03 +0200

>> > When opening a large file in /proc/* like /proc/cpuinfo in Emacs via
>> > find-file, the buffer content is incomplete, when comparing the output
>> > to the output produced by /bin/cat. The Eshell command eshell/cat has
>> > the same problem.The proc file system doesn't report file sizes since
>> > the files are dynamically generated. ls -l /proc/cpuinfo reports size 0.
>> > Emacs doesn't handle this well, limiting its reads to 16KiB.
>> >
>> > In Eshell:
>> >
>> > cat /proc/cpuinfo > #<buffer bug>  ==> truncated buffer (exactly 16KiB)
>> > *cat /proc/cpuinfo > #<buffer bug> ==> complete buffer (larger than 16KiB)
>> 
>> Strange, bug#9800 was closed as fixed, but I confirm the buffer is still
>> truncated.
>
> It was closed because it was fixed back then.

I'm sure I tested it when it was closed, so it was fixed back then indeed.

> But maybe commits 8650b3308dd and/or 3fb2c174d3a7 broke that later?

I confirm that reverting 8650b3308dd fixes it again after removing 'seekable'.

> The /proc/cpuinfo file on the system to which I have access is smaller
> than 16KB, so could someone who can access a larger cpuinfo file
> please see if those commits broke the fix, or maybe bisect?

To be able to test it you need at least 16 cpu cores.  If this is
not feasible, a simpler way would be just to try on another file
such as /proc/crypto.

Message #17 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Wed, 15 Jan 2025 22:07:02 +0200

> From: Juri Linkov <juri <at> linkov.net>
> Cc: Paul Eggert <eggert <at> cs.ucla.edu>,  mail <at> daniel-mendler.de,
>   75569 <at> debbugs.gnu.org
> Date: Wed, 15 Jan 2025 19:08:03 +0200
> 
> > But maybe commits 8650b3308dd and/or 3fb2c174d3a7 broke that later?
> 
> I confirm that reverting 8650b3308dd fixes it again after removing 'seekable'.

Thanks.  Can you try the patch below?

> > The /proc/cpuinfo file on the system to which I have access is smaller
> > than 16KB, so could someone who can access a larger cpuinfo file
> > please see if those commits broke the fix, or maybe bisect?
> 
> To be able to test it you need at least 16 cpu cores.  If this is
> not feasible, a simpler way would be just to try on another file
> such as /proc/crypto.

/proc/crypto here is also less than 16KB, so I still cannot test.

diff --git a/src/fileio.c b/src/fileio.c
index db7c491..a296970 100644
--- a/src/fileio.c
+++ b/src/fileio.c
@@ -4747,7 +4747,7 @@ DEFUN ("insert-file-contents", Finsert_file_contents, Sinsert_file_contents,
       goto handled;
     }
 
-  if (seekable || !NILP (end))
+  if ((seekable && st.size > 0) || !NILP (end))
     total = end_offset - beg_offset;
   else
     /* For a special file, all we can do is guess.  */

Message #20 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>, Juri Linkov <juri <at> linkov.net>
Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Wed, 15 Jan 2025 13:08:10 -0800

On 2025-01-15 12:07, Eli Zaretskii wrote:
> -  if (seekable || !NILP (end))
> +  if ((seekable && st.size > 0) || !NILP (end))

Presumably you meant st.st_size. However, POSIX does not specify the 
value of st.st_size for every seekable file. Instead, when reading Emacs 
should look at st.st_size only for regular files, shared memory objects, 
and typed memory objects (the last two are rare enough that we don't 
need to bother to optimize for them).

Also, there are other uses of 'seekable' in the code that probably need 
a similar treatment.

Message #23 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Thu, 16 Jan 2025 08:18:42 +0200

> Date: Wed, 15 Jan 2025 13:08:10 -0800
> Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> 
> On 2025-01-15 12:07, Eli Zaretskii wrote:
> > -  if (seekable || !NILP (end))
> > +  if ((seekable && st.size > 0) || !NILP (end))
> 
> Presumably you meant st.st_size.

(Blush) yes, sorry.

> However, POSIX does not specify the 
> value of st.st_size for every seekable file. Instead, when reading Emacs 
> should look at st.st_size only for regular files, shared memory objects, 
> and typed memory objects (the last two are rare enough that we don't 
> need to bother to optimize for them).

So you are saying that the condition should be

   if ((regular && st.st_size > 0) || seekable || !NILP (end))

is that right?

> Also, there are other uses of 'seekable' in the code that probably need 
> a similar treatment.

There are only two more:

	/* 'try' is reserved in some compilers (Microsoft C).  */
	ptrdiff_t trytry = min (gap_size, READ_BUF_SIZE);
	if (seekable || !NILP (end))
	  trytry = min (trytry, total - inserted);

	if (!seekable && NILP (end))
	  {

AFAIU, the additional conditions are not needed for these two uses,
because:

  . 'total' is already computed correctly at that point for the
    regular zero-size files
  . only non-regular files can be !seekable

Or am I missing something?

Thanks.

Message #26 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Wed, 15 Jan 2025 22:58:31 -0800

On 2025-01-15 22:18, Eli Zaretskii wrote:

> So you are saying that the condition should be
> 
>     if ((regular && st.st_size > 0) || seekable || !NILP (end))

regular implies seekable, so that doesn't look right either. I was 
thinking that rather than replace "seekable" with "(seekable && st.size 
> 0)" or with "(regular && st.st_size > 0) || seekable", one should 
replace "seekable" with "(regular && st.st_size > 0)".


> AFAIU, the additional conditions are not needed for these two uses,
> because:
> 
>    . 'total' is already computed correctly at that point for the
>      regular zero-size files
>    . only non-regular files can be !seekable
> 
> Or am I missing something?

You've looked it more carefully than I. All I was thinking is that if 
'total' was guessed to be READ_BUF_SIZE earlier, it's still a guess here.

Anyway, I suppose it doesn't matter much, as 'total' is a guess 
regardless of how it's computed. Even a regular file with nonzero size 
can change size before we read it.

There's another thing. On some platforms these /proc regular files have 
nonzero size. On at least one of these platforms (Solaris) the size is 
accurate (though it may change at any time); on others, I vaguely recall 
that it's a small nonzero fixed size (I don't remember which systems do 
that, unfortunately).

Message #29 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Thu, 16 Jan 2025 10:55:46 +0200

> Date: Wed, 15 Jan 2025 22:58:31 -0800
> Cc: juri <at> linkov.net, mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> 
> On 2025-01-15 22:18, Eli Zaretskii wrote:
> 
> > So you are saying that the condition should be
> > 
> >     if ((regular && st.st_size > 0) || seekable || !NILP (end))
> 
> regular implies seekable, so that doesn't look right either.

Then how about

  if ((regular && st.st_size > 0) || (!regular && seekable) || !NILP (end))

> I was 
> thinking that rather than replace "seekable" with "(seekable && st.size 
>  > 0)" or with "(regular && st.st_size > 0) || seekable", one should 
> replace "seekable" with "(regular && st.st_size > 0)".

That would exclude non-regular files that are seekable, which would be
a regression, I think.

> There's another thing. On some platforms these /proc regular files have 
> nonzero size. On at least one of these platforms (Solaris) the size is 
> accurate (though it may change at any time); on others, I vaguely recall 
> that it's a small nonzero fixed size (I don't remember which systems do 
> that, unfortunately).

If the size is non-zero and accurate, it should already work, because
this is just how "normal" files behave.  If the size is non-zero and
inaccurate, do we currently support that, e.g., if the file grows
while we read it?

Message #32 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Thu, 16 Jan 2025 12:51:11 -0800

On 2025-01-16 00:55, Eli Zaretskii wrote:

> Then how about
> 
>    if ((regular && st.st_size > 0) || (!regular && seekable) || !NILP (end))

That should work too but I expect we're overthinking it; see bellow.


> That would exclude non-regular files that are seekable, which would be
> a regression, I think.

I expect it's a bugfix not a regression. For such files, end_offset
is TYPE_MAXIUMUM (off_t), which Emacs will try to create an enormous buffer.


> If the size is non-zero and
> inaccurate, do we currently support that, e.g., if the file grows
> while we read it?

That's the crux of the bug. If we don't support that, the code is 
unreliable pretty much everywhere, not just in /proc. If we fix that, 
the /proc problem goes away.

In other words, when reading a file Emacs should treat st_size as merely 
a heuristic, not as authoritative.

Message #35 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: juri <at> linkov.net, mail <at> daniel-mendler.de, Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sat, 18 Jan 2025 12:44:30 +0200

> Date: Thu, 16 Jan 2025 12:51:11 -0800
> Cc: juri <at> linkov.net, mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> 
> On 2025-01-16 00:55, Eli Zaretskii wrote:
> 
> > Then how about
> > 
> >    if ((regular && st.st_size > 0) || (!regular && seekable) || !NILP (end))
> 
> That should work too but I expect we're overthinking it; see bellow.

Juri and Daniel, could you please try the change above?

> > That would exclude non-regular files that are seekable, which would be
> > a regression, I think.
> 
> I expect it's a bugfix not a regression. For such files, end_offset
> is TYPE_MAXIUMUM (off_t), which Emacs will try to create an enormous buffer.

Juri and Daniel, could you also try

  if ((regular && st.st_size > 0) || !NILP (end))  

instead?

> > If the size is non-zero and
> > inaccurate, do we currently support that, e.g., if the file grows
> > while we read it?
> 
> That's the crux of the bug. If we don't support that, the code is 
> unreliable pretty much everywhere, not just in /proc. If we fix that, 
> the /proc problem goes away.
> 
> In other words, when reading a file Emacs should treat st_size as merely 
> a heuristic, not as authoritative.

But does it do that now?  Regular files whose size changes as we read
them are pretty rare, so maybe we almost never bump into them.

Message #38 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mail <at> daniel-mendler.de, Paul Eggert <eggert <at> cs.ucla.edu>,
 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sat, 18 Jan 2025 19:31:34 +0200

>> > Then how about
>> > 
>> >    if ((regular && st.st_size > 0) || (!regular && seekable) || !NILP (end))
>> 
>> That should work too but I expect we're overthinking it; see bellow.
>
> Juri and Daniel, could you please try the change above?

There are many large files under /proc that you can try
such as /proc/kallsyms.

I tried the change above, and it works.

>> > That would exclude non-regular files that are seekable, which would be
>> > a regression, I think.
>> 
>> I expect it's a bugfix not a regression. For such files, end_offset
>> is TYPE_MAXIUMUM (off_t), which Emacs will try to create an enormous buffer.
>
> Juri and Daniel, could you also try
>
>   if ((regular && st.st_size > 0) || !NILP (end))  
>
> instead?

This works as well.

Message #41 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Paul Eggert <eggert <at> cs.ucla.edu>
To: Eli Zaretskii <eliz <at> gnu.org>, juri <at> linkov.net, mail <at> daniel-mendler.de
Cc: 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sat, 18 Jan 2025 09:35:34 -0800

On 2025-01-18 02:44, Eli Zaretskii wrote:
>> In other words, when reading a file Emacs should treat st_size as merely
>> a heuristic, not as authoritative.
> But does it do that now?  Regular files whose size changes as we read
> them are pretty rare, so maybe we almost never bump into them.

No, it doesn't do it now, and that's the main bug here. Fix that, and 
this other stuff isn't needed (and likely gets simplified away).

Message #44 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sun, 19 Jan 2025 12:52:39 +0200

> From: Juri Linkov <juri <at> linkov.net>
> Cc: mail <at> daniel-mendler.de,  Paul Eggert <eggert <at> cs.ucla.edu>,
>   75569 <at> debbugs.gnu.org
> Date: Sat, 18 Jan 2025 19:31:34 +0200
> 
> >> > Then how about
> >> > 
> >> >    if ((regular && st.st_size > 0) || (!regular && seekable) || !NILP (end))
> >> 
> >> That should work too but I expect we're overthinking it; see bellow.
> >
> > Juri and Daniel, could you please try the change above?
> 
> There are many large files under /proc that you can try
> such as /proc/kallsyms.

Ah, thanks.

> I tried the change above, and it works.
> 
> >> > That would exclude non-regular files that are seekable, which would be
> >> > a regression, I think.
> >> 
> >> I expect it's a bugfix not a regression. For such files, end_offset
> >> is TYPE_MAXIUMUM (off_t), which Emacs will try to create an enormous buffer.
> >
> > Juri and Daniel, could you also try
> >
> >   if ((regular && st.st_size > 0) || !NILP (end))  
> >
> > instead?
> 
> This works as well.

Thanks for testing.

Message #47 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: mail <at> daniel-mendler.de, 75569 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sun, 19 Jan 2025 12:54:25 +0200

> Date: Sat, 18 Jan 2025 09:35:34 -0800
> Cc: 75569 <at> debbugs.gnu.org
> From: Paul Eggert <eggert <at> cs.ucla.edu>
> 
> On 2025-01-18 02:44, Eli Zaretskii wrote:
> >> In other words, when reading a file Emacs should treat st_size as merely
> >> a heuristic, not as authoritative.
> > But does it do that now?  Regular files whose size changes as we read
> > them are pretty rare, so maybe we almost never bump into them.
> 
> No, it doesn't do it now, and that's the main bug here. Fix that, and 
> this other stuff isn't needed (and likely gets simplified away).

Patches welcome, then.

For now, I've installed a fix that Juri tested, to fix the regression.

Message #50 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Mendler <mail <at> daniel-mendler.de>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Paul Eggert <eggert <at> cs.ucla.edu>, 75569 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sun, 19 Jan 2025 11:58:26 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

>> Date: Sat, 18 Jan 2025 09:35:34 -0800
>> Cc: 75569 <at> debbugs.gnu.org
>> From: Paul Eggert <eggert <at> cs.ucla.edu>
>> 
>> On 2025-01-18 02:44, Eli Zaretskii wrote:
>> >> In other words, when reading a file Emacs should treat st_size as merely
>> >> a heuristic, not as authoritative.
>> > But does it do that now?  Regular files whose size changes as we read
>> > them are pretty rare, so maybe we almost never bump into them.
>> 
>> No, it doesn't do it now, and that's the main bug here. Fix that, and 
>> this other stuff isn't needed (and likely gets simplified away).
>
> Patches welcome, then.
>
> For now, I've installed a fix that Juri tested, to fix the regression.

Thank you! Are you going to install the fix on the emacs-30 branch if
this is a regression, or has it been broken in 29 too?

Daniel

Message #53 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: juri <at> linkov.net
Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sun, 19 Jan 2025 13:33:34 +0200

> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org
> Date: Sun, 19 Jan 2025 12:52:39 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> > From: Juri Linkov <juri <at> linkov.net>
> > Cc: mail <at> daniel-mendler.de,  Paul Eggert <eggert <at> cs.ucla.edu>,
> >   75569 <at> debbugs.gnu.org
> > Date: Sat, 18 Jan 2025 19:31:34 +0200
> > 
> > >> > Then how about
> > >> > 
> > >> >    if ((regular && st.st_size > 0) || (!regular && seekable) || !NILP (end))
> > >> 
> > >> That should work too but I expect we're overthinking it; see bellow.
> > >
> > > Juri and Daniel, could you please try the change above?
> > 
> > There are many large files under /proc that you can try
> > such as /proc/kallsyms.
> 
> Ah, thanks.
> 
> > I tried the change above, and it works.
> > 
> > >> > That would exclude non-regular files that are seekable, which would be
> > >> > a regression, I think.
> > >> 
> > >> I expect it's a bugfix not a regression. For such files, end_offset
> > >> is TYPE_MAXIUMUM (off_t), which Emacs will try to create an enormous buffer.
> > >
> > > Juri and Daniel, could you also try
> > >
> > >   if ((regular && st.st_size > 0) || !NILP (end))  
> > >
> > > instead?
> > 
> > This works as well.
> 
> Thanks for testing.

Actually... are you sure these changes fixed the problem?  I've
installed one of them, and /proc/kallsyms is not read in its entirety.
Which file did you try this with, and what is its real size?

Message #56 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Daniel Mendler <mail <at> daniel-mendler.de>
Cc: eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org, juri <at> linkov.net
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sun, 19 Jan 2025 13:34:54 +0200

> From: Daniel Mendler <mail <at> daniel-mendler.de>
> Cc: Paul Eggert <eggert <at> cs.ucla.edu>,  juri <at> linkov.net,  75569 <at> debbugs.gnu.org
> Date: Sun, 19 Jan 2025 11:58:26 +0100
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> Date: Sat, 18 Jan 2025 09:35:34 -0800
> >> Cc: 75569 <at> debbugs.gnu.org
> >> From: Paul Eggert <eggert <at> cs.ucla.edu>
> >> 
> >> On 2025-01-18 02:44, Eli Zaretskii wrote:
> >> >> In other words, when reading a file Emacs should treat st_size as merely
> >> >> a heuristic, not as authoritative.
> >> > But does it do that now?  Regular files whose size changes as we read
> >> > them are pretty rare, so maybe we almost never bump into them.
> >> 
> >> No, it doesn't do it now, and that's the main bug here. Fix that, and 
> >> this other stuff isn't needed (and likely gets simplified away).
> >
> > Patches welcome, then.
> >
> > For now, I've installed a fix that Juri tested, to fix the regression.
> 
> Thank you! Are you going to install the fix on the emacs-30 branch if
> this is a regression, or has it been broken in 29 too?

I'm not going to install this on emacs-30 for now.  First, its' risky,
and it's too late for such changes on the release branch.  And second,
it doesn't seem to be a solution for this case, see my other message.

Message #59 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: juri <at> linkov.net, mail <at> daniel-mendler.de
Cc: eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sun, 19 Jan 2025 14:22:08 +0200

> Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org
> Date: Sun, 19 Jan 2025 13:33:34 +0200
> From: Eli Zaretskii <eliz <at> gnu.org>
> 
> Actually... are you sure these changes fixed the problem?  I've
> installed one of them, and /proc/kallsyms is not read in its entirety.
> Which file did you try this with, and what is its real size?

I've now installed a followup fix, please test.

Message #62 received at 75569 <at> debbugs.gnu.org (full text, mbox):

From: Juri Linkov <juri <at> linkov.net>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 75569 <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sun, 19 Jan 2025 20:56:57 +0200

>> Actually... are you sure these changes fixed the problem?  I've
>> installed one of them, and /proc/kallsyms is not read in its entirety.
>> Which file did you try this with, and what is its real size?

I remember that both fixed the problem on /proc/kallsyms that is almost 18 MB.

> I've now installed a followup fix, please test.

I confirm that the installed change fixes the problem.

Message #67 received at 75569-done <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Juri Linkov <juri <at> linkov.net>
Cc: mail <at> daniel-mendler.de, eggert <at> cs.ucla.edu, 75569-done <at> debbugs.gnu.org
Subject: Re: bug#75569: 30.0.93; Large /proc/* file are read incompletely
Date: Sat, 01 Feb 2025 13:40:37 +0200

> From: Juri Linkov <juri <at> linkov.net>
> Cc: mail <at> daniel-mendler.de,  eggert <at> cs.ucla.edu,  75569 <at> debbugs.gnu.org
> Date: Sun, 19 Jan 2025 20:56:57 +0200
> 
> >> Actually... are you sure these changes fixed the problem?  I've
> >> installed one of them, and /proc/kallsyms is not read in its entirety.
> >> Which file did you try this with, and what is its real size?
> 
> I remember that both fixed the problem on /proc/kallsyms that is almost 18 MB.
> 
> > I've now installed a followup fix, please test.
> 
> I confirm that the installed change fixes the problem.

Thanks, closing.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.