Subject: Re: bug#75672: 31.0.50; scratch/igc memory usage/collection issues

From: Eli Zaretskii <[email-masked]>
Date: Tue, 21 Jan 2025 15:35:51 +0200

> Date: Mon, 20 Jan 2025 21:07:55 +0000
> From: Pip Cet <[email-masked]>
> Cc: [email-masked], [email-masked]
> 
> "Eli Zaretskii" <[email-masked]> writes:
> 
> >> >> committed                                    1       400138240  400138240             0
> >> >> pause-time                                 nil             0.1        nil           nil
> >> >> reserved                                     1       536875008  536875008             0
> >> >> spare                                      nil            0.75        nil           nil
> >> >> spare-committed                              1       217436160  217436160             0
> >> >
> >> > AFAIU, these 3 values mean that MPS uses about 1.15GB of memory
> >> > (unless "reserved" includes "committed").  If "reserved" means memory
> >>
> >> MPS "reserved" means "mmapped, not necessarily faulted in or used".
> >> MPS "committed" means "faulted in and possibly in use by MPS or the
> >> client program".
> >> MPS "spare-committed" means "in use by MPS, not the client program".
> >>
> >> So committed - spare-commited is how much usable memory MPS thinks has
> >> been allocated, about 183 MB.
> >
> > I agree, but did I say something different?
> 
> Not at all!  I was merely summarizing the (unusual) definitions MPS
> uses, and explain why they amount to 183MB of MPS memory in use by
> Emacs, not 1.15GB.

That depends on the definition of "in use".  The reserved memory
addresses cannot be used by any other process, so in some sense they
are "in use".

> > The question now becomes: how was the 1.5GB figure Alex reported
> > measured? did it include the reserved memory, or didn't it?
> 
> Even if it did, MPS accounts for at most 512MB: reserved is an upper
> limit on how much memory is in use.  So we'll have to look for other
> culprits, or for very strange MPS/igc bugs.

So you are saying "reserved" includes "committed"?

I'd still like to see the results of memory-info, since that will tell
us what glibc knowns about the memory.  AFAIU, MPS manages only part
of the memory of the process, right?

> >> It's virtual memory: MPS assumes virtual memory is very cheap.  That's
> >> not really true for processes that fork() a lot, so this assumption may
> >> have to be fine-tuned.
> >
> > AFAIK, fork'ed process uses copy-on-write, so we should be okay.
> 
> It does use copy-on-write, but on my machine, at least, Linux appears to
> copy the entire page table on every fork, which makes fork quite a bit
> more expensive in terms of CPU time than I thought it would be.  This is
> noticeable for large processes, as I've previously described.

What is the size of the page table in that case?