GNU bug report logs - #74805
30.0.92; Trying to build scratch/igc on Cygwin

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 74805 in the body.
You can then email your comments to 74805 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: bug-gnu-emacs <at> gnu.org
Subject: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Wed, 11 Dec 2024 17:50:40 -0500

I've made a first attempt to port mps to Cygwin.  If anyone else is
interested in seeing this or helping, you can find my work at

  https://github.com/kbrow1i/mps-cygwin

This is a fork of the mps git repo with two commits on top.

With the current attempt, I can build the scratch/igc branch, but there
are many test failures.  For example, when I run the process-tests, I
get 19 failures and 8 skipped tests.  But on the master branch I get no
failures and only 3 skipped.

This probably means that my current attempt to port mps is no good, and
I have to go back to the drawing board.  Before I do that, however, I'd
like to check and see if test failures on scratch/igc are to be
expected.  Are people testing other platforms seeing this too?

Thanks.

Ken

Message #8 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Wed, 11 Dec 2024 23:47:31 +0000

"Ken Brown" <kbrown <at> cornell.edu> writes:

> I've made a first attempt to port mps to Cygwin.  If anyone else is
> interested in seeing this or helping, you can find my work at
>
>    https://github.com/kbrow1i/mps-cygwin
>
> This is a fork of the mps git repo with two commits on top.
>
> With the current attempt, I can build the scratch/igc branch, but there
> are many test failures.  For example, when I run the process-tests, I

But the Emacs binary "works", to some extent? Is there anything in the
process-tests.log file which hints at what kind of problem it might be?

Does it work interactively, or crash right away?

> get 19 failures and 8 skipped tests.  But on the master branch I get no
> failures and only 3 skipped.

That sounds like a "minor" GC problem; a word size mismatch or alignment
problem would probably lead to a hard crash, so it's more likely to be a
problem in scanning the stack.

I'm a bit confused about the role of prmccyi6.c; it's included in the
cyi6gc.mk fragment, but it's not included by mps.c directly. How are you
building MPS?

I see that you have defined (in prmccyi6.c)

Addr MutatorContextSP(MutatorContext context)
{
  AVERT(MutatorContext, context);
  return (Addr)context->ucontext->uc_mcontext.rsp;   /* .sp */
}

using the MutatorContextStruct from prmcix.h, which uses an ucontext.
However, you also include prmcw3.c, which uses a different definition of
MutatorContextStruct. I don't think those two are compatible, so it's
possible that's the problem, depending on how you build mps.

> This probably means that my current attempt to port mps is no good, and
> I have to go back to the drawing board.  Before I do that, however, I'd
> like to check and see if test failures on scratch/igc are to be
> expected.  Are people testing other platforms seeing this too?

I see a few test failures on Windows, but nothing as catastrophic as
what you describe.

Do you have a usable debugger/gdb on your platform? That would
be one approach to diagnosing crashes (if the problem is indeed that
Emacs crashes).  Another approach would be to run the test suite
included in MPS.

Pip

Message #11 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Andrea Corallo <acorallo <at> gnu.org>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Wed, 11 Dec 2024 18:57:49 -0500

Ken Brown <kbrown <at> cornell.edu> writes:

> I've made a first attempt to port mps to Cygwin.  If anyone else is
> interested in seeing this or helping, you can find my work at
>
>   https://github.com/kbrow1i/mps-cygwin
>
> This is a fork of the mps git repo with two commits on top.
>
> With the current attempt, I can build the scratch/igc branch, but there
> are many test failures.  For example, when I run the process-tests, I
> get 19 failures and 8 skipped tests.  But on the master branch I get no
> failures and only 3 skipped.

Is the testsuite supposed to be clean on igc on supported platforms?
Last time I asked Gerd explained me it was still not the case but maybe
things evolved.

  Andrea

Message #14 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 08:09:07 +0200

> Date: Wed, 11 Dec 2024 17:50:40 -0500
> From: Ken Brown <kbrown <at> cornell.edu>
> 
> I've made a first attempt to port mps to Cygwin.  If anyone else is
> interested in seeing this or helping, you can find my work at
> 
>    https://github.com/kbrow1i/mps-cygwin
> 
> This is a fork of the mps git repo with two commits on top.

Thanks, good news!  Could you please post the patches here, so that
they are recorded in our bug tracker?

> With the current attempt, I can build the scratch/igc branch, but there
> are many test failures.  For example, when I run the process-tests, I
> get 19 failures and 8 skipped tests.  But on the master branch I get no
> failures and only 3 skipped.
> 
> This probably means that my current attempt to port mps is no good, and
> I have to go back to the drawing board.

Did you run the MPS test suite, and if you did, were there any
failures?  My suggestion is to make sure the MPS test suite passes
cleanly, including in several consecutive runs (when I ported MPS to
32-bit MinGW, I had intermittent failures which only happened once in
several runs, until I fixed that).

Message #17 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Ken Brown <kbrown <at> cornell.edu>, 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 03:05:58 -0800

Ken Brown <kbrown <at> cornell.edu> writes:

> I've made a first attempt to port mps to Cygwin.  If anyone else is
> interested in seeing this or helping, you can find my work at
>
>    https://github.com/kbrow1i/mps-cygwin
>
> This is a fork of the mps git repo with two commits on top.
>
> With the current attempt, I can build the scratch/igc branch, but there
> are many test failures.  For example, when I run the process-tests, I
> get 19 failures and 8 skipped tests.  But on the master branch I get no
> failures and only 3 skipped.
>
> This probably means that my current attempt to port mps is no good, and
> I have to go back to the drawing board.  Before I do that, however, I'd
> like to check and see if test failures on scratch/igc are to be
> expected.  Are people testing other platforms seeing this too?

Do we expect upstream to be amenable to eventually taking these patches?
I see that they officially only support "Windows Vista or later, on
IA-32 and x86-64, using Microsoft Visual C/C++".

Message #20 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Kangas <stefankangas <at> gmail.com>,
 Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: 74805 <at> debbugs.gnu.org, kbrown <at> cornell.edu
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 13:30:59 +0200

> From: Stefan Kangas <stefankangas <at> gmail.com>
> Date: Thu, 12 Dec 2024 03:05:58 -0800
> 
> Ken Brown <kbrown <at> cornell.edu> writes:
> 
> > I've made a first attempt to port mps to Cygwin.  If anyone else is
> > interested in seeing this or helping, you can find my work at
> >
> >    https://github.com/kbrow1i/mps-cygwin
> >
> > This is a fork of the mps git repo with two commits on top.
> >
> > With the current attempt, I can build the scratch/igc branch, but there
> > are many test failures.  For example, when I run the process-tests, I
> > get 19 failures and 8 skipped tests.  But on the master branch I get no
> > failures and only 3 skipped.
> >
> > This probably means that my current attempt to port mps is no good, and
> > I have to go back to the drawing board.  Before I do that, however, I'd
> > like to check and see if test failures on scratch/igc are to be
> > expected.  Are people testing other platforms seeing this too?
> 
> Do we expect upstream to be amenable to eventually taking these patches?

I don't think we've asked them.  Maybe Gerd knows?

> I see that they officially only support "Windows Vista or later, on
> IA-32 and x86-64, using Microsoft Visual C/C++".

Yes.

Message #23 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 74805 <at> debbugs.gnu.org, Stefan Kangas <stefankangas <at> gmail.com>,
 kbrown <at> cornell.edu
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 13:06:41 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

>> From: Stefan Kangas <stefankangas <at> gmail.com>
>> Date: Thu, 12 Dec 2024 03:05:58 -0800
>> 
>> Ken Brown <kbrown <at> cornell.edu> writes:
>> 
>> > I've made a first attempt to port mps to Cygwin.  If anyone else is
>> > interested in seeing this or helping, you can find my work at
>> >
>> >    https://github.com/kbrow1i/mps-cygwin
>> >
>> > This is a fork of the mps git repo with two commits on top.
>> >
>> > With the current attempt, I can build the scratch/igc branch, but there
>> > are many test failures.  For example, when I run the process-tests, I
>> > get 19 failures and 8 skipped tests.  But on the master branch I get no
>> > failures and only 3 skipped.
>> >
>> > This probably means that my current attempt to port mps is no good, and
>> > I have to go back to the drawing board.  Before I do that, however, I'd
>> > like to check and see if test failures on scratch/igc are to be
>> > expected.  Are people testing other platforms seeing this too?
>> 
>> Do we expect upstream to be amenable to eventually taking these patches?
>
> I don't think we've asked them.  Maybe Gerd knows?

I haven't done anything like that. I've only submitted some minor Github
issue when I was playing with MPS, and I've sent them a mail to let them
know what's going on when I made what I had public here on emacs-devel.

Message #26 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: 74805 <at> debbugs.gnu.org, stefankangas <at> gmail.com, kbrown <at> cornell.edu
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 16:38:07 +0200

> From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
> Cc: Stefan Kangas <stefankangas <at> gmail.com>,  kbrown <at> cornell.edu,
>   74805 <at> debbugs.gnu.org
> Date: Thu, 12 Dec 2024 13:06:41 +0100
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> >> From: Stefan Kangas <stefankangas <at> gmail.com>
> >> Date: Thu, 12 Dec 2024 03:05:58 -0800
> >> 
> >> Ken Brown <kbrown <at> cornell.edu> writes:
> >> 
> >> > I've made a first attempt to port mps to Cygwin.  If anyone else is
> >> > interested in seeing this or helping, you can find my work at
> >> >
> >> >    https://github.com/kbrow1i/mps-cygwin
> >> >
> >> > This is a fork of the mps git repo with two commits on top.
> >> >
> >> > With the current attempt, I can build the scratch/igc branch, but there
> >> > are many test failures.  For example, when I run the process-tests, I
> >> > get 19 failures and 8 skipped tests.  But on the master branch I get no
> >> > failures and only 3 skipped.
> >> >
> >> > This probably means that my current attempt to port mps is no good, and
> >> > I have to go back to the drawing board.  Before I do that, however, I'd
> >> > like to check and see if test failures on scratch/igc are to be
> >> > expected.  Are people testing other platforms seeing this too?
> >> 
> >> Do we expect upstream to be amenable to eventually taking these patches?
> >
> > I don't think we've asked them.  Maybe Gerd knows?
> 
> I haven't done anything like that. I've only submitted some minor Github
> issue when I was playing with MPS, and I've sent them a mail to let them
> know what's going on when I made what I had public here on emacs-devel.

So maybe we should ask them.  It could be a factor in our decision
whether we want to fork MPS and maintain our own fork.

Message #29 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 74805 <at> debbugs.gnu.org, stefankangas <at> gmail.com, kbrown <at> cornell.edu
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 16:01:03 +0100

Eli Zaretskii <eliz <at> gnu.org> writes:

>> I haven't done anything like that. I've only submitted some minor Github
>> issue when I was playing with MPS, and I've sent them a mail to let them
>> know what's going on when I made what I had public here on emacs-devel.
>
> So maybe we should ask them.  It could be a factor in our decision
> whether we want to fork MPS and maintain our own fork.

Yes, I think something "official" from the maintainers would be a good
thing. The email I used back then to inform them of what's
going on was info <at> ravenbrook.com. Couldn't find something better.

Message #32 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 13:12:46 -0500

[Message part 1 (text/plain, inline)]

On 12/12/2024 1:09 AM, Eli Zaretskii wrote:
>> Date: Wed, 11 Dec 2024 17:50:40 -0500
>> From: Ken Brown <kbrown <at> cornell.edu>
>>
>> I've made a first attempt to port mps to Cygwin.  If anyone else is
>> interested in seeing this or helping, you can find my work at
>>
>>     https://github.com/kbrow1i/mps-cygwin
>>
>> This is a fork of the mps git repo with two commits on top.
> 
> Thanks, good news!  Could you please post the patches here, so that
> they are recorded in our bug tracker?

Unfortunately, the approach I tried, which was suggested by one of the 
MPS developers, fails badly.  I already mentioned the test failures, but 
now I've tried to run Emacs interactively, and I can't even get a 
directory listing.  For the record, I'm attaching the patches I've 
accumulated so far, but I'll have to try a completely different 
approach.  This will probably take me a while.
> 
>> With the current attempt, I can build the scratch/igc branch, but there
>> are many test failures.  For example, when I run the process-tests, I
>> get 19 failures and 8 skipped tests.  But on the master branch I get no
>> failures and only 3 skipped.
>>
>> This probably means that my current attempt to port mps is no good, and
>> I have to go back to the drawing board.
> 
> Did you run the MPS test suite, and if you did, were there any
> failures?  My suggestion is to make sure the MPS test suite passes
> cleanly, including in several consecutive runs (when I ported MPS to
> 32-bit MinGW, I had intermittent failures which only happened once in
> several runs, until I fixed that).

There are lots of test failures.  Back to the drawing board.

Ken

[0001-Porting-to-Cygwin-first-attempt.patch (text/plain, attachment)]

[0002-Porting-to-Cygwin-second-attempt.patch (text/plain, attachment)]

[0003-Update-Cygwin-Makefile.patch (text/plain, attachment)]

[0004-Pacifying-testsuite-don-t-use-long-long.patch (text/plain, attachment)]

Message #35 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 13:14:16 -0500

On 12/11/2024 6:47 PM, Pip Cet wrote:
> "Ken Brown" <kbrown <at> cornell.edu> writes:
> 
>> I've made a first attempt to port mps to Cygwin.  If anyone else is
>> interested in seeing this or helping, you can find my work at
>>
>>     https://github.com/kbrow1i/mps-cygwin
>>
>> This is a fork of the mps git repo with two commits on top.
>>
>> With the current attempt, I can build the scratch/igc branch, but there
>> are many test failures.  For example, when I run the process-tests, I
> 
> But the Emacs binary "works", to some extent? Is there anything in the
> process-tests.log file which hints at what kind of problem it might be?
> 
> Does it work interactively, or crash right away?
> 
>> get 19 failures and 8 skipped tests.  But on the master branch I get no
>> failures and only 3 skipped.
> 
> That sounds like a "minor" GC problem; a word size mismatch or alignment
> problem would probably lead to a hard crash, so it's more likely to be a
> problem in scanning the stack.
> 
> I'm a bit confused about the role of prmccyi6.c; it's included in the
> cyi6gc.mk fragment, but it's not included by mps.c directly. How are you
> building MPS?
> 
> I see that you have defined (in prmccyi6.c)
> 
> Addr MutatorContextSP(MutatorContext context)
> {
>    AVERT(MutatorContext, context);
>    return (Addr)context->ucontext->uc_mcontext.rsp;   /* .sp */
> }
> 
> using the MutatorContextStruct from prmcix.h, which uses an ucontext.
> However, you also include prmcw3.c, which uses a different definition of
> MutatorContextStruct. I don't think those two are compatible, so it's
> possible that's the problem, depending on how you build mps.
> 
>> This probably means that my current attempt to port mps is no good, and
>> I have to go back to the drawing board.  Before I do that, however, I'd
>> like to check and see if test failures on scratch/igc are to be
>> expected.  Are people testing other platforms seeing this too?
> 
> I see a few test failures on Windows, but nothing as catastrophic as
> what you describe.
> 
> Do you have a usable debugger/gdb on your platform? That would
> be one approach to diagnosing crashes (if the problem is indeed that
> Emacs crashes).  Another approach would be to run the test suite
> included in MPS.
Thanks for the reply and suggestions.  See my response to Ely.

Ken

Message #38 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 21:25:47 +0200

> Date: Thu, 12 Dec 2024 13:12:46 -0500
> Cc: 74805 <at> debbugs.gnu.org
> From: Ken Brown <kbrown <at> cornell.edu>
> 
> >>     https://github.com/kbrow1i/mps-cygwin
> >>
> >> This is a fork of the mps git repo with two commits on top.
> > 
> > Thanks, good news!  Could you please post the patches here, so that
> > they are recorded in our bug tracker?
> 
> Unfortunately, the approach I tried, which was suggested by one of the 
> MPS developers, fails badly.  I already mentioned the test failures, but 
> now I've tried to run Emacs interactively, and I can't even get a 
> directory listing.  For the record, I'm attaching the patches I've 
> accumulated so far, but I'll have to try a completely different 
> approach.  This will probably take me a while.
> > 
> >> With the current attempt, I can build the scratch/igc branch, but there
> >> are many test failures.  For example, when I run the process-tests, I
> >> get 19 failures and 8 skipped tests.  But on the master branch I get no
> >> failures and only 3 skipped.
> >>
> >> This probably means that my current attempt to port mps is no good, and
> >> I have to go back to the drawing board.
> > 
> > Did you run the MPS test suite, and if you did, were there any
> > failures?  My suggestion is to make sure the MPS test suite passes
> > cleanly, including in several consecutive runs (when I ported MPS to
> > 32-bit MinGW, I had intermittent failures which only happened once in
> > several runs, until I fixed that).
> 
> There are lots of test failures.  Back to the drawing board.

OK, thanks.

FTR, I built the official released version of MPS, not their latest
code in the repository.  So maybe try that as well at some point,
perhaps the development version has new bugs.

What I did was to take the MSVC makefiles they provide and produce
equivalent MinGW makefiles (for GNU Make) instead, using the same
source files.  The rest was just to resolve any compilation warnings
or errors.  HTH.

Message #41 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Stefan Kangas <stefankangas <at> gmail.com>
To: Eli Zaretskii <eliz <at> gnu.org>, Ken Brown <kbrown <at> cornell.edu>
Cc: 74805 <at> debbugs.gnu.org
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 12 Dec 2024 15:17:03 -0800

Eli Zaretskii <eliz <at> gnu.org> writes:

> What I did was to take the MSVC makefiles they provide and produce
> equivalent MinGW makefiles (for GNU Make) instead, using the same
> source files.  The rest was just to resolve any compilation warnings
> or errors.  HTH.

Was any of that work upstreamed?  Maybe it doesn't make sense to do
that, but I thought I'd ask anyways.

Would it make sense to document which ports are supported in README-IGC?

Message #44 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Stefan Kangas <stefankangas <at> gmail.com>
Cc: 74805 <at> debbugs.gnu.org, kbrown <at> cornell.edu
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Fri, 13 Dec 2024 09:51:40 +0200

> From: Stefan Kangas <stefankangas <at> gmail.com>
> Date: Thu, 12 Dec 2024 15:17:03 -0800
> Cc: 74805 <at> debbugs.gnu.org
> 
> Eli Zaretskii <eliz <at> gnu.org> writes:
> 
> > What I did was to take the MSVC makefiles they provide and produce
> > equivalent MinGW makefiles (for GNU Make) instead, using the same
> > source files.  The rest was just to resolve any compilation warnings
> > or errors.  HTH.
> 
> Was any of that work upstreamed?  Maybe it doesn't make sense to do
> that, but I thought I'd ask anyways.

I didn't upstream, but then no one asked me for the patches, which for
me is a clear sign that the 32-bit port of MPS to Windows/MinGW is not
too popular, to put it mildly.

> Would it make sense to document which ports are supported in README-IGC?

It cannot do any harm, although I also don't see how it will help.  If
the instructions there already don't produce a working library, then
the instructions should be amended, and perhaps subdivided to
system-specific ones; otherwise saying on which systems the library
works can only make sense if you want to avert people from trying the
branch on systems where we never tried to compile the library.

Message #47 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Fri, 20 Dec 2024 18:48:37 -0500

On 12/12/2024 1:09 AM, Eli Zaretskii wrote:
>> Date: Wed, 11 Dec 2024 17:50:40 -0500
>> From: Ken Brown <kbrown <at> cornell.edu>
>>
>> I've made a first attempt to port mps to Cygwin.  If anyone else is
>> interested in seeing this or helping, you can find my work at
>>
>>     https://github.com/kbrow1i/mps-cygwin
>>
>> This is a fork of the mps git repo with two commits on top.
> 
> Thanks, good news!  Could you please post the patches here, so that
> they are recorded in our bug tracker?
> 
>> With the current attempt, I can build the scratch/igc branch, but there
>> are many test failures.  For example, when I run the process-tests, I
>> get 19 failures and 8 skipped tests.  But on the master branch I get no
>> failures and only 3 skipped.
>>
>> This probably means that my current attempt to port mps is no good, and
>> I have to go back to the drawing board.
> 
> Did you run the MPS test suite, and if you did, were there any
> failures?  My suggestion is to make sure the MPS test suite passes
> cleanly, including in several consecutive runs (when I ported MPS to
> 32-bit MinGW, I had intermittent failures which only happened once in
> several runs, until I fixed that).

This is an update on my attempts to port the MPS to Cygwin and build the 
scratch/igc branch.  The GitHub repo that I cited above now consists of 
4 branches:

1. The "master" branch is an MPS fork with README.Cygwin added.  This 
file explains in detail what I'm summarizing here.

2. The "w3" branch attempts to use the Windows interface on Cygwin.  For 
example, it uses direct calls to VirtualAlloc instead of Cygwin's mmap. 
This attempt fails because it's not compatible with Cygwin's fork.

3. The "mmap" branch is a straightforward port, mostly imitating the 
FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a 
limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in the 
works that removes that limitation.  With that patch, 37 of the 38 MPS 
tests pass.  I still need to debug the failing test.  I'm cautiously 
optimistic that I can get this approach to work.  Either way, I expect 
the Cygwin patch to soon be available in a test release of Cygwin 3.6.0 
so that other Cygwin users can try it.

4. The "generic" branch uses generic implementations of some of the MPS 
features.  For example, it uses malloc instead of mmap.  This passes all 
of the MPS tests, and Emacs seems to work well so far in limited 
testing.  I suspect that there may be some performance problems, but I 
haven't seen them yet.

Once I've determined whether or not the mmap approach will work, I'll 
post the MPS patches here either for the mmap branch or the generic branch.

Ken

Message #50 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: 74805 <at> debbugs.gnu.org, rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Sat, 21 Dec 2024 09:24:22 +0200

> Date: Fri, 20 Dec 2024 18:48:37 -0500
> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
> From: Ken Brown <kbrown <at> cornell.edu>
> 
> This is an update on my attempts to port the MPS to Cygwin and build the 
> scratch/igc branch.  The GitHub repo that I cited above now consists of 
> 4 branches:
> 
> 1. The "master" branch is an MPS fork with README.Cygwin added.  This 
> file explains in detail what I'm summarizing here.
> 
> 2. The "w3" branch attempts to use the Windows interface on Cygwin.  For 
> example, it uses direct calls to VirtualAlloc instead of Cygwin's mmap. 
> This attempt fails because it's not compatible with Cygwin's fork.
> 
> 3. The "mmap" branch is a straightforward port, mostly imitating the 
> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a 
> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in the 
> works that removes that limitation.  With that patch, 37 of the 38 MPS 
> tests pass.  I still need to debug the failing test.  I'm cautiously 
> optimistic that I can get this approach to work.  Either way, I expect 
> the Cygwin patch to soon be available in a test release of Cygwin 3.6.0 
> so that other Cygwin users can try it.
> 
> 4. The "generic" branch uses generic implementations of some of the MPS 
> features.  For example, it uses malloc instead of mmap.  This passes all 
> of the MPS tests, and Emacs seems to work well so far in limited 
> testing.  I suspect that there may be some performance problems, but I 
> haven't seen them yet.
> 
> Once I've determined whether or not the mmap approach will work, I'll 
> post the MPS patches here either for the mmap branch or the generic branch.

Thanks for the update, I think this is very good news.

Message #53 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Pip Cet <pipcet <at> protonmail.com>, rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Sat, 21 Dec 2024 11:56:30 -0500

[Message part 1 (text/plain, inline)]

On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>> From: Ken Brown <kbrown <at> cornell.edu>
>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in the
>> works that removes that limitation.  With that patch, 37 of the 38 MPS
>> tests pass.  I still need to debug the failing test.  I'm cautiously
>> optimistic that I can get this approach to work.  Either way, I expect
>> the Cygwin patch to soon be available in a test release of Cygwin 3.6.0
>> so that other Cygwin users can try it.
> 
> Thanks for the update, I think this is very good news.

I could use some help from the MPS experts in debugging the failing 
test, which is arenacv.  I ran the test under strace and didn't see any 
mmap or munmap failures.  I'm attaching the test log, which doesn't mean 
a thing to me.  I also built an unoptimized arenacv and can run it under 
gdb if someone tells me what to look for.

TIA.

Ken

[9-arenacv (text/plain, attachment)]

Message #56 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Gerd Möllmann <gerd.moellmann <at> gmail.com>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Pip Cet <pipcet <at> protonmail.com>, rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Sat, 21 Dec 2024 18:21:53 +0100

Ken Brown <kbrown <at> cornell.edu> writes:

> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>> From: Ken Brown <kbrown <at> cornell.edu>
>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in the
>>> works that removes that limitation.  With that patch, 37 of the 38 MPS
>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>> optimistic that I can get this approach to work.  Either way, I expect
>>> the Cygwin patch to soon be available in a test release of Cygwin 3.6.0
>>> so that other Cygwin users can try it.
>> Thanks for the update, I think this is very good news.
>
> I could use some help from the MPS experts in debugging the failing
> test, which is arenacv.  I ran the test under strace and didn't see
> any mmap or munmap failures.  I'm attaching the test log, which
> doesn't mean a thing to me.  I also built an unoptimized arenacv and
> can run it under gdb if someone tells me what to look for.
>
> TIA.
>
> Ken

Hi Ken.

I'm not an MPS internals expert, but AFAICT, this likely comes from here
in arenacv.c, but without being able to debug this, I find it hard to
tell what's going on in the allocator->alloc.

static void testAllocAndIterate(Arena arena, Pool pool,
                                Size pageSize, Count numPerPage,
                                AllocatorClass allocator)
{
  AllocInfoStruct offsetRegion, gapRegion, newRegion, topRegion;
  LocusPrefStruct pref;
  Count offset, gap, new;
  ZoneSet zone = (ZoneSet)2;
  int i;

  LocusPrefInit(&pref);

  /* Testing the behaviour with various sizes of gaps in the page table. */

  /* Assume the allocation strategy is first-fit.  The idea of the tests is */
  /* to allocate a region of memory, then deallocate a gap in the middle, */
  /* then allocate a new region that fits in the gap with various amounts */
  /* left over.  Like this: */
  /* |-offsetRegion-||----gapRegion----||-topRegion-| */
  /* |-offsetRegion-||-newRegion-|      |-topRegion-| */
  /* This is done with three different sizes of offsetRegion, in two */
  /* different zones to ensure that all page boundary cases are tested. */
  for(i = 0; i < 2; ++i) { /* zone loop */
    for(offset = 0; offset <= 2*numPerPage; offset += numPerPage) {
      if(offset != 0)
        die(allocator->alloc(&offsetRegion, &pref, offset * pageSize, pool),
            "offsetRegion");
      for(gap = numPerPage+1; gap <= 3 * (numPerPage+1);
          gap += (numPerPage+1)) {
        die(allocator->alloc(&gapRegion, &pref, gap * pageSize, pool),
            ^^^^^^^^^^^^^^^^
            "gapRegion");

Message #59 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Gerd Möllmann <gerd.moellmann <at> gmail.com>
Cc: 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Ken Brown <kbrown <at> cornell.edu>, rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Sat, 21 Dec 2024 17:34:30 +0000

Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:

> Ken Brown <kbrown <at> cornell.edu> writes:
>
>> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>>> From: Ken Brown <kbrown <at> cornell.edu>
>>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in the
>>>> works that removes that limitation.  With that patch, 37 of the 38 MPS
>>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>>> optimistic that I can get this approach to work.  Either way, I expect
>>>> the Cygwin patch to soon be available in a test release of Cygwin 3.6.0
>>>> so that other Cygwin users can try it.
>>> Thanks for the update, I think this is very good news.
>>
>> I could use some help from the MPS experts in debugging the failing
>> test, which is arenacv.  I ran the test under strace and didn't see
>> any mmap or munmap failures.  I'm attaching the test log, which

What about mprotect, or whatever the equivalent is?

>> doesn't mean a thing to me.  I also built an unoptimized arenacv and
>> can run it under gdb if someone tells me what to look for.
>>
>> TIA.
>>
>> Ken
>
> Hi Ken.
>
> I'm not an MPS internals expert, but AFAICT, this likely comes from here
> in arenacv.c, but without being able to debug this, I find it hard to
> tell what's going on in the allocator->alloc.
>
> static void testAllocAndIterate(Arena arena, Pool pool,
>                                 Size pageSize, Count numPerPage,
>                                 AllocatorClass allocator)

Can you check pageSize is correct?

Pip

Message #62 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Pip Cet <pipcet <at> protonmail.com>, Gerd Möllmann
 <gerd.moellmann <at> gmail.com>
Cc: 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>, rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Sat, 21 Dec 2024 15:25:45 -0500

On 12/21/2024 12:34 PM, Pip Cet wrote:
> Gerd Möllmann <gerd.moellmann <at> gmail.com> writes:
> 
>> Ken Brown <kbrown <at> cornell.edu> writes:
>>
>>> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>>>> From: Ken Brown <kbrown <at> cornell.edu>
>>>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in the
>>>>> works that removes that limitation.  With that patch, 37 of the 38 MPS
>>>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>>>> optimistic that I can get this approach to work.  Either way, I expect
>>>>> the Cygwin patch to soon be available in a test release of Cygwin 3.6.0
>>>>> so that other Cygwin users can try it.
>>>> Thanks for the update, I think this is very good news.
>>>
>>> I could use some help from the MPS experts in debugging the failing
>>> test, which is arenacv.  I ran the test under strace and didn't see
>>> any mmap or munmap failures.  I'm attaching the test log, which
> 
> What about mprotect, or whatever the equivalent is?

I can't find any calls to mprotect when this test is run.

>>> doesn't mean a thing to me.  I also built an unoptimized arenacv and
>>> can run it under gdb if someone tells me what to look for.
>>>
>>> TIA.
>>>
>>> Ken
>>
>> Hi Ken.
>>
>> I'm not an MPS internals expert, but AFAICT, this likely comes from here
>> in arenacv.c, but without being able to debug this, I find it hard to
>> tell what's going on in the allocator->alloc.
>>
>> static void testAllocAndIterate(Arena arena, Pool pool,
>>                                  Size pageSize, Count numPerPage,
>>                                  AllocatorClass allocator)
> 
> Can you check pageSize is correct?

pageSize is 65536, which is the Windows allocation granularity.  It 
should be OK.

Ken

Message #65 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Richard Brooksby <rb <at> ravenbrook.com>
To: Ken Brown <kbrown <at> cornell.edu>, Eli Zaretskii <eliz <at> gnu.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Pip Cet <pipcet <at> protonmail.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Sun, 22 Dec 2024 08:24:03 +0000

On 2024-12-21 16:56, Ken Brown wrote:
> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>> From: Ken Brown <kbrown <at> cornell.edu>
>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in the
>>> works that removes that limitation.  With that patch, 37 of the 38 MPS
>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>> optimistic that I can get this approach to work.  Either way, I expect
>>> the Cygwin patch to soon be available in a test release of Cygwin 3.6.0
>>> so that other Cygwin users can try it.
>>
>> Thanks for the update, I think this is very good news.
> 
> I could use some help from the MPS experts in debugging the failing 
> test, which is arenacv.  I ran the test under strace and didn't see any 
> mmap or munmap failures.  I'm attaching the test log, which doesn't mean 
> a thing to me.  I also built an unoptimized arenacv and can run it under 
> gdb if someone tells me what to look for.

Hello.  I can't offer much direct help just now, but here's where I'd start.

The test is exiting because the MPS is returning code ResRESOURCE from 
these lines 
https://github.com/Ravenbrook/mps/blob/9fd0577cf1231e61c9801c81499e5d16d0743806/code/arenacv.c#L356-L357 
while calling an allocator.

ResRESOURCE only occurs in a few places, and is almost certainly to do 
with a lack of available address space.  Given that you've had to patch 
mmap, that's a smoking gun, but it should be fairly easy to breakpoint 
all the lines that raise ResRESOURCE to figure out what's happening.

Sometimes it's difficult to breakpoint the MPS because it runs in weird 
signal contexts.  We've been known to crash debuggers.  I don't know if 
it's tricksy in Cygwin.  In that case, you can hack around with 
_mps_RES_ENUM in mps.h to make ResRESOURCE call a function.

$ grep --color=auto -nH --null -e ResRESOURCE *.c
arena.c:767:    return ResRESOURCE;
arena.c:1048:    return ResRESOURCE;
arena.c:1227:  return ResRESOURCE;
arenacv.c:461:      die((res == ResRESOURCE) ? ResOK : res, "right error 
code");
arenavm.c:722:    res = ResRESOURCE;
fotest.c:55:    return ResRESOURCE;
mpm.c:254:  return (res == ResMEMORY || res == ResRESOURCE || res == 
ResCOMMIT_LIMIT);
thw3.c:101:    return ResRESOURCE;
vman.c:52:    return ResRESOURCE;
vman.c:59:    return ResRESOURCE;
vmix.c:107:    return ResRESOURCE;
vmix.c:110:    return ResRESOURCE;
vmix.c:122:    return ResRESOURCE;
vmw3.c:117:    return ResRESOURCE;
vmw3.c:120:    return ResRESOURCE;
vmw3.c:130:    return ResRESOURCE;

It's very unlikely that you're actually running out of address space on 
a 64-bit system.

Message #68 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Richard Brooksby <rb <at> ravenbrook.com>
To: Ken Brown <kbrown <at> cornell.edu>, Eli Zaretskii <eliz <at> gnu.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Pip Cet <pipcet <at> protonmail.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Sun, 22 Dec 2024 08:32:22 +0000

On 2024-12-22 08:24, Richard Brooksby wrote:
> On 2024-12-21 16:56, Ken Brown wrote:
>> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>>> From: Ken Brown <kbrown <at> cornell.edu>
>>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin in 
>>>> the
>>>> works that removes that limitation.  With that patch, 37 of the 38 MPS
>>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>>> optimistic that I can get this approach to work.  Either way, I expect
>>>> the Cygwin patch to soon be available in a test release of Cygwin 3.6.0
>>>> so that other Cygwin users can try it.
>>>
>>> Thanks for the update, I think this is very good news.
>>
>> I could use some help from the MPS experts in debugging the failing 
>> test, which is arenacv.  I ran the test under strace and didn't see 
>> any mmap or munmap failures.  I'm attaching the test log, which 
>> doesn't mean a thing to me.  I also built an unoptimized arenacv and 
>> can run it under gdb if someone tells me what to look for.
> 
> Hello.  I can't offer much direct help just now, but here's where I'd 
> start.
...
> It's very unlikely that you're actually running out of address space 
on a 64-bit system.

I should add that arenacv is a coverage test that is *trying* to provoke 
the ResRESOURCE error paths in some circumstances 
https://github.com/Ravenbrook/mps/blob/9fd0577cf1231e61c9801c81499e5d16d0743806/code/arenacv.c#L461 
but note that this isn't where the test is failing.  However, it may be 
relevant, perhaps if your munmap doesn't successfully free address space.

Message #71 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Richard Brooksby <rb <at> ravenbrook.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Pip Cet <pipcet <at> protonmail.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Mon, 23 Dec 2024 18:32:35 -0500

On 12/22/2024 3:32 AM, Richard Brooksby wrote:
> On 2024-12-22 08:24, Richard Brooksby wrote:
>> On 2024-12-21 16:56, Ken Brown wrote:
>>> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>>>> From: Ken Brown <kbrown <at> cornell.edu>
>>>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin 
>>>>> in the
>>>>> works that removes that limitation.  With that patch, 37 of the 38 MPS
>>>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>>>> optimistic that I can get this approach to work.  Either way, I expect
>>>>> the Cygwin patch to soon be available in a test release of Cygwin 
>>>>> 3.6.0
>>>>> so that other Cygwin users can try it.
>>>>
>>>> Thanks for the update, I think this is very good news.
>>>
>>> I could use some help from the MPS experts in debugging the failing 
>>> test, which is arenacv.  I ran the test under strace and didn't see 
>>> any mmap or munmap failures.  I'm attaching the test log, which 
>>> doesn't mean a thing to me.  I also built an unoptimized arenacv and 
>>> can run it under gdb if someone tells me what to look for.
>>
>> Hello.  I can't offer much direct help just now, but here's where I'd 
>> start.
> ...
>  > It's very unlikely that you're actually running out of address space 
> on a 64-bit system.
> 
> I should add that arenacv is a coverage test that is *trying* to provoke 
> the ResRESOURCE error paths in some circumstances https://github.com/ 
> Ravenbrook/mps/blob/9fd0577cf1231e61c9801c81499e5d16d0743806/code/ 
> arenacv.c#L461 but note that this isn't where the test is failing.  
> However, it may be relevant, perhaps if your munmap doesn't successfully 
> free address space.
Thanks!  You've given me some good hints.

Ken

Message #74 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Richard Brooksby <rb <at> ravenbrook.com>, Eli Zaretskii <eliz <at> gnu.org>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Pip Cet <pipcet <at> protonmail.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Tue, 21 Jan 2025 15:07:06 -0500

On 12/23/2024 6:32 PM, Ken Brown wrote:
> On 12/22/2024 3:32 AM, Richard Brooksby wrote:
>> On 2024-12-22 08:24, Richard Brooksby wrote:
>>> On 2024-12-21 16:56, Ken Brown wrote:
>>>> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>>>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>>>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>>>>> From: Ken Brown <kbrown <at> cornell.edu>
>>>>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>>>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>>>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin 
>>>>>> in the
>>>>>> works that removes that limitation.  With that patch, 37 of the 38 
>>>>>> MPS
>>>>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>>>>> optimistic that I can get this approach to work.  Either way, I 
>>>>>> expect
>>>>>> the Cygwin patch to soon be available in a test release of Cygwin 
>>>>>> 3.6.0
>>>>>> so that other Cygwin users can try it.
>>>>>
>>>>> Thanks for the update, I think this is very good news.
>>>>
>>>> I could use some help from the MPS experts in debugging the failing 
>>>> test, which is arenacv.  I ran the test under strace and didn't see 
>>>> any mmap or munmap failures.  I'm attaching the test log, which 
>>>> doesn't mean a thing to me.  I also built an unoptimized arenacv and 
>>>> can run it under gdb if someone tells me what to look for.
>>>
>>> Hello.  I can't offer much direct help just now, but here's where I'd 
>>> start.
>> ...
>>  > It's very unlikely that you're actually running out of address 
>> space on a 64-bit system.
>>
>> I should add that arenacv is a coverage test that is *trying* to 
>> provoke the ResRESOURCE error paths in some circumstances https:// 
>> github.com/ Ravenbrook/mps/ 
>> blob/9fd0577cf1231e61c9801c81499e5d16d0743806/code/ arenacv.c#L461 but 
>> note that this isn't where the test is failing. However, it may be 
>> relevant, perhaps if your munmap doesn't successfully free address space.
> Thanks!  You've given me some good hints.
Here's the latest update on my attempts.  The good news is that I've 
been running a build from the igc branch with no noticeable problems. 
The bad news is that I am unable to debug the failing arenacv test.  I 
have no idea how critical this is from the point of view of the igc branch.

What follows is a very long description of my debugging efforts, in the 
hopes that someone familiar with the MPS code can help.

1. Of all the tests called in arenacv.c:main(), the only one that fails 
is this one:

  testPageTable((ArenaClass)mps_arena_class_cl(), TEST_ARENA_SIZE, 
block, FALSE)

at line 481.

2. Of the two calls to testAllocAndIterate in testPageTable, it's only 
the second one that fails, provided I apply the following patch:

--- a/code/arenacv.c
+++ b/code/arenacv.c
@@ -333,6 +333,7 @@ static void testAllocAndIterate(Arena arena, Pool pool,
   Count offset, gap, new;
   ZoneSet zone = (ZoneSet)2;
   int i;
+  mps_res_t res;

   LocusPrefInit(&pref);

@@ -353,8 +354,14 @@ static void testAllocAndIterate(Arena arena, Pool pool,
             "offsetRegion");
       for(gap = numPerPage+1; gap <= 3 * (numPerPage+1);
           gap += (numPerPage+1)) {
-        die(allocator->alloc(&gapRegion, &pref, gap * pageSize, pool),
-            "gapRegion");
+       res = allocator->alloc(&gapRegion, &pref, gap * pageSize, pool);
+       if (res != ResOK) {
+         fprintf(stdout, "res = %d, gap = %lu, offset = %lu, i = %d\n",
+                 res, gap, offset, i);
+         die(res, "gapRegion");
+       }
+        /* die(allocator->alloc(&gapRegion, &pref, gap * pageSize, 
pool), */
+        /*     "gapRegion"); */
         die(allocator->alloc(&topRegion, &pref, pageSize, pool),
             "topRegion");
         allocator->free(&gapRegion);
@@ -473,8 +480,8 @@ int main(int argc, char *argv[])

   testlib_init(argc, argv);

-  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0, 
TRUE);
-  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0, 
FALSE);
+  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 
0, TRUE); */
+  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 
0, FALSE); */

   block = malloc(TEST_ARENA_SIZE);
   cdie(block != NULL, "malloc");


If I don't apply that patch then the first call fails (and of course the 
second one doesn't get run).  I can't make sense out of that.

3. After I apply the patch, there are no calls to mmap or munmap 
(according to strace) except an initial call to mmap when the code calls 
malloc.  [This is an internal implementation detail, that Cygwin uses 
mmap in malloc.]  If I don't apply that patch, there are lots of calls 
to both, all of which succeed.  I don't understand why the code is not 
calling mmap/munmap when the patch is applied.  How else could it be 
trying to acquire address space?

4. When testAllocAndIterate fails, it's at the very first iteration 
through the loops.  In other words, i = 0, offset = 0, and gap = 
numPerPage = 2731.

5. The failure can be traced to PolicyAlloc() returning ResRESOURCE at 
policy.c:126.  Unfortunately, there are about 7 calls to 
ArenaFreeLandAlloc that have to *all* fail before we reach line 126.  I 
tried stepping through all 7 of them in the failure case, but, being 
completely unfamiliar with the code, I couldn't see what was happening.

6. Here's the backtrace when we reach policy.c:126 (in a run without the 
patch above):

#0  PolicyAlloc (tractReturn=0x7ffffc760, arena=0x6ffffbff0010,
    pref=0x7ffffc870, size=178978816, pool=0x6ffffc030248) at policy.c:126
#1  0x0000000100407852 in ArenaAlloc (baseReturn=0x7ffffc7f0, 
pref=0x7ffffc870,
    size=178978816, pool=0x6ffffc030248) at arena.c:1092
#2  0x000000010040135c in allocAsTract (aiReturn=0x7ffffc8d0, 
pref=0x7ffffc870,
    size=178978816, pool=0x6ffffc030248) at arenacv.c:164
#3  0x000000010040181c in testAllocAndIterate (arena=0x6ffffbff0010,
    pool=0x6ffffc030248, pageSize=65536, numPerPage=2730,
    allocator=0x10048e020 <allocatorTractStruct>) at arenacv.c:356
#4  0x0000000100401d46 in testPageTable (
    klass=0x1004b0180 <ClassStaticClientArena>, size=67108864,
    addr=0x6ffffbff0010, zoned=0) at arenacv.c:425
#5  0x0000000100401fc3 in main (argc=1, argv=0xa000017b0) at arenacv.c:481


Thanks in advance to anyone who has ideas about this.

Ken

Message #77 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: gerd.moellmann <at> gmail.com, 74805 <at> debbugs.gnu.org, pipcet <at> protonmail.com,
 rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Wed, 22 Jan 2025 14:43:21 +0200

> Date: Tue, 21 Jan 2025 15:07:06 -0500
> From: Ken Brown <kbrown <at> cornell.edu>
> Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
>  74805 <at> debbugs.gnu.org, Pip Cet <pipcet <at> protonmail.com>
> 
> Here's the latest update on my attempts.  The good news is that I've 
> been running a build from the igc branch with no noticeable problems. 

That _is_ good news, thanks.

> 2. Of the two calls to testAllocAndIterate in testPageTable, it's only 
> the second one that fails, provided I apply the following patch:
> 
> --- a/code/arenacv.c
> +++ b/code/arenacv.c
> @@ -333,6 +333,7 @@ static void testAllocAndIterate(Arena arena, Pool pool,
>     Count offset, gap, new;
>     ZoneSet zone = (ZoneSet)2;
>     int i;
> +  mps_res_t res;
> 
>     LocusPrefInit(&pref);
> 
> @@ -353,8 +354,14 @@ static void testAllocAndIterate(Arena arena, Pool pool,
>               "offsetRegion");
>         for(gap = numPerPage+1; gap <= 3 * (numPerPage+1);
>             gap += (numPerPage+1)) {
> -        die(allocator->alloc(&gapRegion, &pref, gap * pageSize, pool),
> -            "gapRegion");
> +       res = allocator->alloc(&gapRegion, &pref, gap * pageSize, pool);
> +       if (res != ResOK) {
> +         fprintf(stdout, "res = %d, gap = %lu, offset = %lu, i = %d\n",
> +                 res, gap, offset, i);
> +         die(res, "gapRegion");
> +       }
> +        /* die(allocator->alloc(&gapRegion, &pref, gap * pageSize, 
> pool), */
> +        /*     "gapRegion"); */
>           die(allocator->alloc(&topRegion, &pref, pageSize, pool),
>               "topRegion");
>           allocator->free(&gapRegion);
> @@ -473,8 +480,8 @@ int main(int argc, char *argv[])
> 
>     testlib_init(argc, argv);
> 
> -  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0, 
> TRUE);
> -  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0, 
> FALSE);
> +  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 
> 0, TRUE); */
> +  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 
> 0, FALSE); */
> 
>     block = malloc(TEST_ARENA_SIZE);
>     cdie(block != NULL, "malloc");
> 
> 
> If I don't apply that patch then the first call fails (and of course the 
> second one doesn't get run).  I can't make sense out of that.
> 
> 3. After I apply the patch, there are no calls to mmap or munmap 
> (according to strace) except an initial call to mmap when the code calls 
> malloc.  [This is an internal implementation detail, that Cygwin uses 
> mmap in malloc.]  If I don't apply that patch, there are lots of calls 
> to both, all of which succeed.  I don't understand why the code is not 
> calling mmap/munmap when the patch is applied.  How else could it be 
> trying to acquire address space?

A stab in the dark: could it be that the call to fprintf you added in
the patch causes malloc to be called from fprintf, and thus changes
the state of malloc and/or the arena in a way that causes the above
unexpected effects?  Can you try using a different print-out function
instead that is guaranteed not to allocate any memory?

Message #80 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Eli Zaretskii <eliz <at> gnu.org>
Cc: gerd.moellmann <at> gmail.com, 74805 <at> debbugs.gnu.org, pipcet <at> protonmail.com,
 rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Mon, 27 Jan 2025 11:20:24 -0500

On 1/22/2025 7:43 AM, Eli Zaretskii wrote:
>> Date: Tue, 21 Jan 2025 15:07:06 -0500
>> From: Ken Brown <kbrown <at> cornell.edu>
>> Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
>>   74805 <at> debbugs.gnu.org, Pip Cet <pipcet <at> protonmail.com>
>>
>> Here's the latest update on my attempts.  The good news is that I've
>> been running a build from the igc branch with no noticeable problems.
> 
> That _is_ good news, thanks.
> 
>> 2. Of the two calls to testAllocAndIterate in testPageTable, it's only
>> the second one that fails, provided I apply the following patch:
>>
>> --- a/code/arenacv.c
>> +++ b/code/arenacv.c
>> @@ -333,6 +333,7 @@ static void testAllocAndIterate(Arena arena, Pool pool,
>>      Count offset, gap, new;
>>      ZoneSet zone = (ZoneSet)2;
>>      int i;
>> +  mps_res_t res;
>>
>>      LocusPrefInit(&pref);
>>
>> @@ -353,8 +354,14 @@ static void testAllocAndIterate(Arena arena, Pool pool,
>>                "offsetRegion");
>>          for(gap = numPerPage+1; gap <= 3 * (numPerPage+1);
>>              gap += (numPerPage+1)) {
>> -        die(allocator->alloc(&gapRegion, &pref, gap * pageSize, pool),
>> -            "gapRegion");
>> +       res = allocator->alloc(&gapRegion, &pref, gap * pageSize, pool);
>> +       if (res != ResOK) {
>> +         fprintf(stdout, "res = %d, gap = %lu, offset = %lu, i = %d\n",
>> +                 res, gap, offset, i);
>> +         die(res, "gapRegion");
>> +       }
>> +        /* die(allocator->alloc(&gapRegion, &pref, gap * pageSize,
>> pool), */
>> +        /*     "gapRegion"); */
>>            die(allocator->alloc(&topRegion, &pref, pageSize, pool),
>>                "topRegion");
>>            allocator->free(&gapRegion);
>> @@ -473,8 +480,8 @@ int main(int argc, char *argv[])
>>
>>      testlib_init(argc, argv);
>>
>> -  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0,
>> TRUE);
>> -  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0,
>> FALSE);
>> +  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE,
>> 0, TRUE); */
>> +  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE,
>> 0, FALSE); */
>>
>>      block = malloc(TEST_ARENA_SIZE);
>>      cdie(block != NULL, "malloc");
>>
>>
>> If I don't apply that patch then the first call fails (and of course the
>> second one doesn't get run).  I can't make sense out of that.
>>
>> 3. After I apply the patch, there are no calls to mmap or munmap
>> (according to strace) except an initial call to mmap when the code calls
>> malloc.  [This is an internal implementation detail, that Cygwin uses
>> mmap in malloc.]  If I don't apply that patch, there are lots of calls
>> to both, all of which succeed.  I don't understand why the code is not
>> calling mmap/munmap when the patch is applied.  How else could it be
>> trying to acquire address space?
> 
> A stab in the dark: could it be that the call to fprintf you added in
> the patch causes malloc to be called from fprintf, and thus changes
> the state of malloc and/or the arena in a way that causes the above
> unexpected effects?  Can you try using a different print-out function
> instead that is guaranteed not to allocate any memory?

Thanks, but that doesn't explain it.  I tried simply removing the call 
to fprintf, but the strace output still shows no calls to mmap or 
munmap.  Very strange.

Ken

Message #83 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Eli Zaretskii <eliz <at> gnu.org>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: gerd.moellmann <at> gmail.com, 74805 <at> debbugs.gnu.org, pipcet <at> protonmail.com,
 rb <at> ravenbrook.com
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Mon, 27 Jan 2025 18:43:43 +0200

> Date: Mon, 27 Jan 2025 11:20:24 -0500
> Cc: rb <at> ravenbrook.com, gerd.moellmann <at> gmail.com, 74805 <at> debbugs.gnu.org,
>  pipcet <at> protonmail.com
> From: Ken Brown <kbrown <at> cornell.edu>
> 
> >> 3. After I apply the patch, there are no calls to mmap or munmap
> >> (according to strace) except an initial call to mmap when the code calls
> >> malloc.  [This is an internal implementation detail, that Cygwin uses
> >> mmap in malloc.]  If I don't apply that patch, there are lots of calls
> >> to both, all of which succeed.  I don't understand why the code is not
> >> calling mmap/munmap when the patch is applied.  How else could it be
> >> trying to acquire address space?
> > 
> > A stab in the dark: could it be that the call to fprintf you added in
> > the patch causes malloc to be called from fprintf, and thus changes
> > the state of malloc and/or the arena in a way that causes the above
> > unexpected effects?  Can you try using a different print-out function
> > instead that is guaranteed not to allocate any memory?
> 
> Thanks, but that doesn't explain it.  I tried simply removing the call 
> to fprintf, but the strace output still shows no calls to mmap or 
> munmap.  Very strange.

Hmm... I guess this is where you give up and ask the MPS developers to
help (if you haven't already)?

My gut feeling is that this is somehow related to the way Cygwin
implements mmap/munmap, but I have no evidence to support that, and
definitely cannot tell what is it in the Cygwin implementation that
violates the assumptions of this test.

Thanks for working on this.

Message #86 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Richard Brooksby <rb <at> ravenbrook.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Mon, 27 Jan 2025 17:40:30 +0000

"Ken Brown" <kbrown <at> cornell.edu> writes:

> On 12/23/2024 6:32 PM, Ken Brown wrote:
>> On 12/22/2024 3:32 AM, Richard Brooksby wrote:
>>> On 2024-12-22 08:24, Richard Brooksby wrote:
>>>> On 2024-12-21 16:56, Ken Brown wrote:
>>>>> On 12/21/2024 2:24 AM, Eli Zaretskii wrote:
>>>>>>> Date: Fri, 20 Dec 2024 18:48:37 -0500
>>>>>>> Cc: 74805 <at> debbugs.gnu.org, Richard Brooksby <rb <at> ravenbrook.com>
>>>>>>> From: Ken Brown <kbrown <at> cornell.edu>
>>>>>>> 3. The "mmap" branch is a straightforward port, mostly imitating the
>>>>>>> FreeBSD port.  It currently (with Cygwin 3.5.5) fails because of a
>>>>>>> limitation of Cygwin's mmap.  But I have a simple patch to Cygwin
>>>>>>> in the
>>>>>>> works that removes that limitation.  With that patch, 37 of the 38
>>>>>>> MPS
>>>>>>> tests pass.  I still need to debug the failing test.  I'm cautiously
>>>>>>> optimistic that I can get this approach to work.  Either way, I
>>>>>>> expect
>>>>>>> the Cygwin patch to soon be available in a test release of Cygwin
>>>>>>> 3.6.0
>>>>>>> so that other Cygwin users can try it.
>>>>>>
>>>>>> Thanks for the update, I think this is very good news.
>>>>>
>>>>> I could use some help from the MPS experts in debugging the failing
>>>>> test, which is arenacv.  I ran the test under strace and didn't see
>>>>> any mmap or munmap failures.  I'm attaching the test log, which
>>>>> doesn't mean a thing to me.  I also built an unoptimized arenacv and
>>>>> can run it under gdb if someone tells me what to look for.
>>>>
>>>> Hello.  I can't offer much direct help just now, but here's where I'd
>>>> start.
>>> ...
>>>  > It's very unlikely that you're actually running out of address
>>> space on a 64-bit system.
>>>
>>> I should add that arenacv is a coverage test that is *trying* to
>>> provoke the ResRESOURCE error paths in some circumstances https://
>>> github.com/ Ravenbrook/mps/
>>> blob/9fd0577cf1231e61c9801c81499e5d16d0743806/code/ arenacv.c#L461 but
>>> note that this isn't where the test is failing. However, it may be
>>> relevant, perhaps if your munmap doesn't successfully free address space.
>> Thanks!  You've given me some good hints.
> Here's the latest update on my attempts.  The good news is that I've

Sorry for responding to this only now.  I saw Eli's suggestion, thought
that it must have been the reason, and was hoping the problem was fixed.

> been running a build from the igc branch with no noticeable problems.

That is excellent.  Please report any issues!

> The bad news is that I am unable to debug the failing arenacv test.  I
> have no idea how critical this is from the point of view of the igc branch.

I think it would be quite important to find out why it's failing; if it
turns out to be harmless, that's okay.

After thinking about this for a bit, I think we should probably ignore
this test failure for now: there's some (most likely) fragmentation
issue on cygwin, it seems, which causes this unusually aggressive stress
test to fail eventually.  I assume Emacs isn't that aggressive.

> 2. Of the two calls to testAllocAndIterate in testPageTable, it's only
> the second one that fails, provided I apply the following patch:
>
> --- a/code/arenacv.c
> +++ b/code/arenacv.c
> @@ -333,6 +333,7 @@ static void testAllocAndIterate(Arena arena, Pool pool,
>     Count offset, gap, new;
>     ZoneSet zone = (ZoneSet)2;
>     int i;
> +  mps_res_t res;
>
>     LocusPrefInit(&pref);
>
> @@ -353,8 +354,14 @@ static void testAllocAndIterate(Arena arena, Pool pool,
>               "offsetRegion");
>         for(gap = numPerPage+1; gap <= 3 * (numPerPage+1);
>             gap += (numPerPage+1)) {
> -        die(allocator->alloc(&gapRegion, &pref, gap * pageSize, pool),
> -            "gapRegion");
> +       res = allocator->alloc(&gapRegion, &pref, gap * pageSize, pool);
> +       if (res != ResOK) {
> +         fprintf(stdout, "res = %d, gap = %lu, offset = %lu, i = %d\n",
> +                 res, gap, offset, i);
> +         die(res, "gapRegion");
> +       }
> +        /* die(allocator->alloc(&gapRegion, &pref, gap * pageSize,
> pool), */
> +        /*     "gapRegion"); */
>           die(allocator->alloc(&topRegion, &pref, pageSize, pool),
>               "topRegion");
>           allocator->free(&gapRegion);
> @@ -473,8 +480,8 @@ int main(int argc, char *argv[])
>
>     testlib_init(argc, argv);
>
> -  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0,
> TRUE);
> -  testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE, 0,
> FALSE);
> +  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE,
> 0, TRUE); */
> +  /* testPageTable((ArenaClass)mps_arena_class_vm(), TEST_ARENA_SIZE,
> 0, FALSE); */
>
>     block = malloc(TEST_ARENA_SIZE);
>     cdie(block != NULL, "malloc");
>
>
> If I don't apply that patch then the first call fails (and of course the
> second one doesn't get run).  I can't make sense out of that.

So the two calls to testPageTable that succeed but were commented out in
your patch must have left the VM system in a different state?  My theory
is the allocator is running out of VA space or fragmenting it so badly
that it eventually cannot make the (large) allocation.

> 3. After I apply the patch, there are no calls to mmap or munmap
> (according to strace) except an initial call to mmap when the code calls
> malloc.  [This is an internal implementation detail, that Cygwin uses

I *think* that's okay: we're working with the large malloc'd area.

> mmap in malloc.]  If I don't apply that patch, there are lots of calls
> to both, all of which succeed.  I don't understand why the code is not
> calling mmap/munmap when the patch is applied.  How else could it be
> trying to acquire address space?

sbrk, maybe, or it's in that inital malloc...

> 4. When testAllocAndIterate fails, it's at the very first iteration
> through the loops.  In other words, i = 0, offset = 0, and gap =
> numPerPage = 2731.

I don't see how gap can ever equal numPerPage.

> 5. The failure can be traced to PolicyAlloc() returning ResRESOURCE at
> policy.c:126.  Unfortunately, there are about 7 calls to
> ArenaFreeLandAlloc that have to *all* fail before we reach line 126.  I
> tried stepping through all 7 of them in the failure case, but, being
> completely unfamiliar with the code, I couldn't see what was happening.
>
> 6. Here's the backtrace when we reach policy.c:126 (in a run without the
> patch above):
>
> #0  PolicyAlloc (tractReturn=0x7ffffc760, arena=0x6ffffbff0010,
>      pref=0x7ffffc870, size=178978816, pool=0x6ffffc030248) at policy.c:126

This looks like we ran out of VA space.  Maybe we failed to return
free'd VA space to the system, eventually running out of it even on a
64-bit machine?  Can you print the return values in the allocator in any
way?  If you can't, can you make the allocator keep count of it and
inspect the counter in gdb on failure?

The most unusual thing seems to be that pageSize / ArenaGrainSize is so
large (64 KB).  Maybe that simply means that TEST_ARENA_SIZE needs to be
increased?

Pip

Message #89 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Richard Brooksby <rb <at> ravenbrook.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Mon, 27 Jan 2025 18:00:22 +0000

Pip Cet <pipcet <at> protonmail.com> writes:

> The most unusual thing seems to be that pageSize / ArenaGrainSize is so
> large (64 KB).  Maybe that simply means that TEST_ARENA_SIZE needs to be
> increased?

Changing ARENA_CLIENT_GRAIN_SIZE to 65536 makes the test fail here, too,
also when the counter reaches 2730 tracts per page in the page table.

Changing

#define TEST_ARENA_SIZE              ((Size)16<<22)

to

#define TEST_ARENA_SIZE              ((Size)16<<25)

fixes it again.  My best guess is that

#define TEST_ARENA_SIZE (ARENA_CLIENT_GRAIN_SIZE << 13)

would make the arena large enough not to fail the test on all systems.

If you can confirm, we should open an issue for this in the MPS
repository (still at GitHub, I assume).  If tehre isn't one already :-)

Pip

Message #92 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Richard Brooksby <rb <at> ravenbrook.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Mon, 27 Jan 2025 16:37:39 -0500

On 1/27/2025 1:00 PM, Pip Cet wrote:
> Pip Cet <pipcet <at> protonmail.com> writes:
> 
>> The most unusual thing seems to be that pageSize / ArenaGrainSize is so
>> large (64 KB).  Maybe that simply means that TEST_ARENA_SIZE needs to be
>> increased?
> 
> Changing ARENA_CLIENT_GRAIN_SIZE to 65536 makes the test fail here, too,
> also when the counter reaches 2730 tracts per page in the page table.
> 
> Changing
> 
> #define TEST_ARENA_SIZE              ((Size)16<<22)
> 
> to
> 
> #define TEST_ARENA_SIZE              ((Size)16<<25)
> 
> fixes it again.  My best guess is that
> 
> #define TEST_ARENA_SIZE (ARENA_CLIENT_GRAIN_SIZE << 13)
> 
> would make the arena large enough not to fail the test on all systems.
> 
> If you can confirm, we should open an issue for this in the MPS
> repository (still at GitHub, I assume).  If tehre isn't one already :-)
Thank you very much!!  I had to use 17 instead of 13 to get all the 
tests to pass consistently, but then they did in several consecutive 
runs.  (Note that there is a random element to the testsuite because of 
the use of randomize in testlib.c.)

I appreciate your help.  I guess the authors of the tests never imagined 
a platform with a 64K page size.  I hope it's simply the test suite 
that's affected and not the rest of the MPS code.

Ken

Ken

Message #95 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Pip Cet <pipcet <at> protonmail.com>
To: Ken Brown <kbrown <at> cornell.edu>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Richard Brooksby <rb <at> ravenbrook.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Mon, 27 Jan 2025 22:31:25 +0000

"Ken Brown" <kbrown <at> cornell.edu> writes:

> On 1/27/2025 1:00 PM, Pip Cet wrote:
>> Pip Cet <pipcet <at> protonmail.com> writes:
>>
>>> The most unusual thing seems to be that pageSize / ArenaGrainSize is so
>>> large (64 KB).  Maybe that simply means that TEST_ARENA_SIZE needs to be
>>> increased?
>>
>> Changing ARENA_CLIENT_GRAIN_SIZE to 65536 makes the test fail here, too,
>> also when the counter reaches 2730 tracts per page in the page table.
>>
>> Changing
>>
>> #define TEST_ARENA_SIZE              ((Size)16<<22)
>>
>> to
>>
>> #define TEST_ARENA_SIZE              ((Size)16<<25)
>>
>> fixes it again.  My best guess is that
>>
>> #define TEST_ARENA_SIZE (ARENA_CLIENT_GRAIN_SIZE << 13)
>>
>> would make the arena large enough not to fail the test on all systems.
>>
>> If you can confirm, we should open an issue for this in the MPS
>> repository (still at GitHub, I assume).  If tehre isn't one already :-)
> Thank you very much!!

> I had to use 17 instead of 13 to get all the tests to pass
> consistently, but then they did in several consecutive runs.  (Note
> that there is a random element to the testsuite because of the use of
> randomize in testlib.c.)

IIUC, that makes TEST_ARENA_SIZE 8 GiB.  That's a lot, and such a large
constant would probably not be acceptable in the test suite.  Maybe
there is a bug somewhere where we don't reuse VA addresses,...

> I appreciate your help.  I guess the authors of the tests never imagined
> a platform with a 64K page size.

Is that actually the physical page size on your system?

> I hope it's simply the test suite
> that's affected and not the rest of the MPS code.

But discovering such bugs in MPS is important, too! We won't find
unusual bugs if we all run the some configuration.

> Ken
>
> Ken

Thanks to both of you :-)

Pip

Message #98 received at 74805 <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 74805 <at> debbugs.gnu.org, Eli Zaretskii <eliz <at> gnu.org>,
 Richard Brooksby <rb <at> ravenbrook.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Tue, 28 Jan 2025 10:10:27 -0500

On 1/27/2025 5:31 PM, Pip Cet wrote:
> "Ken Brown" <kbrown <at> cornell.edu> writes:
>> I appreciate your help.  I guess the authors of the tests never imagined
>> a platform with a 64K page size.
> 
> Is that actually the physical page size on your system?
I'm not sure what "physical" means in this context, but 64K is the value 
returned by sysconf(_SC_PAGESIZE).  This value was chosen many years ago 
by the Cygwin developers because the allocation granularity on Windows 
is 64K, and it was difficult or impossible to maintain POSIX 
compatibility without using 64K as the page size.

Ken

Message #103 received at 74805-done <at> debbugs.gnu.org (full text, mbox):

From: Ken Brown <kbrown <at> cornell.edu>
To: Pip Cet <pipcet <at> protonmail.com>
Cc: Gerd Möllmann <gerd.moellmann <at> gmail.com>,
 Eli Zaretskii <eliz <at> gnu.org>, 74805-done <at> debbugs.gnu.org,
 Richard Brooksby <rb <at> ravenbrook.com>
Subject: Re: bug#74805: 30.0.92; Trying to build scratch/igc on Cygwin
Date: Thu, 6 Feb 2025 18:27:44 -0500

Closing.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.