Subject: bug#18505: closed (Re: bug#18505: 24.3.93; intermittent unexec failures when building on Mac OS X 10.10 beta, Xcode 6.0)

From: [email-masked] (GNU bug Tracking System)
Date: Sun, 21 Sep 2014 20:38:03 +0000

Your bug report

#18505: 24.3.93; intermittent unexec failures when building on Mac OS X 10.10 beta, Xcode 6.0

which was filed against the emacs package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
18505: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=18505
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#18505: 24.3.93; intermittent unexec failures when building on Mac OS X 10.10 beta, Xcode 6.0
From: Jan Djärv <[email-masked]>
To: David Caldwell <[email-masked]>
Cc: [email-masked]
Date: Sun, 21 Sep 2014 22:37:38 +0200
List-Id: <debbugs-submit.debbugs.gnu.org>

Hi.

21 sep 2014 kl. 20:07 skrev David Caldwell <[email-masked]>:

> On 9/21/14 2:15 AM, Jan Djärv wrote:
>> Hello.
>> 
>> 20 sep 2014 kl. 20:31 skrev David Caldwell <[email-masked]>:
>> 
>>> On 9/20/14 8:31 AM, Jan Djärv wrote:
>>>> Hello.
>>>> 
>>>> 19 sep 2014 kl. 06:13 skrev David Caldwell <[email-masked]>:
>>>> 
>>>>> Hello,
>>>>> 
>>>>> I tried to build the latest pretest on Mac OS X Yosemite Beta with the
>>>>> new Xcode 6.0 (GM) tools and ran into this error during the unexec step:
>>>>> 
>>>>> unexec: not enough room for load commands for new __DATA segments
>>>> 
>>>> Does it happen all the time or just some times?
>>> 
>>> It depends on 2 variables: the number of load commands that need to be
>>> added (num_unexec_regions) and text_seg_lowest_offset.
>>> 
>>> num_unexec_regions jumps around a lot, doing "make clean && make" over
>>> and over it'll be different every time. Somewhere between 12 and 34.
>> 
>> What makes it do that?  Some address randomization?  Some other unknown bug?
>> I would expect num_unexec_regions to be the same for every make.
> 
> I don't know. I would also expect num_unexec_regions to be the same. If
> it changes, it seems to mean the malloc behavior is different on every
> run. But yes, perhaps address space randomization could cause that to
> happen. I don't understand that part of the code well enough to
> speculate too much.

Me neither. 

> 
>> text_seg_lowest_offset could be address randomization, but if it stays somewhat constant, that can't be it.
> 
> I just figured that out (and smacked my head because it was obvious).
> That changed when I changed headerpad. I got curious and did a binary
> search to figure out exactly how -headerpad affects
> text_seg_lowest_offset in my setup (all number hex):
> 
> -headerpad      text_seg_lowest_offset
>   0 ->  740     17a0
> 741 -> 1740     27a0
> 1741 -> ???      37a0
> 
> So, text_seg_lowest_offset directly correlate with -headerpad and ld is
> doing some sort of alignment.

Okay.

> 
>> I've seen this failure before, but usually a new make works.
>> I'm trying to decide if this is emacs 24 or trunk material.
> 
> I think it should go in both. It's really quite a low-risk change: the
> -headerpad option is well documented in ld, and the amount my patch adds
> gives an extra 1.5K of headroom on a 6M binary (.02%).
> 
> I did a bunch of 'bzr log' searches to understand the nature of the
> -headerpad setting and it appears to not have been touched since 2006
> (in the 32 bit era). I believe that is why the comment in configure is
> incorrect: load commands may have been 56 bytes on 32 bit archs, but
> they are 78 bytes on my 64 bit computer (which is all current Macs going
> forward).

I checked it in in the 24-branch.

> 
>> Is there a way to dynamically react to these changes and adjust headerpad_extra dynamically at dump time?
> 
> Unfortunately the -headerpad is specified during link time so to change
> it dynamically would require re-linking after unexec-ing. That's a large
> Makefile change to get that all working correctly.

And possibly overkill.

Thanks,

	Jan D.

Subject: 24.3.93; intermittent unexec failures when building on Mac OS X 10.10 beta, Xcode 6.0
From: David Caldwell <[email-masked]>
To: [email-masked]
Date: Thu, 18 Sep 2014 21:13:41 -0700
List-Id: <debbugs-submit.debbugs.gnu.org>

Hello,

I tried to build the latest pretest on Mac OS X Yosemite Beta with the
new Xcode 6.0 (GM) tools and ran into this error during the unexec step:

unexec: not enough room for load commands for new __DATA segments

(Full unexec log is attached).

Poking around I discovered a couple discrepancies. The "headerpad_extra"
setting in configure.ac says entries are 56 bytes, but they are actually
78 bytes (probably because the comment was originally written in the 32
bit days). Also the number of load commands needed is variable now--it
depends on how many malloc segments were found and how many it could
merge (see find_emacs_zone_regions() and unexec_regions_merge() in
unexmacosx.c).

I printed num_unexec_regions in unexmacosx.c and observed anywhere
from about 14 to 34 regions needed. But given that it seems to be
different on every run and it's super annoying to debug when you run out
of space, I suggest bumping headerpad_extra up to 0x1000. This gives
about 1.5KB of headroom (or wasted space, depending on how you look at
it) and allows for 52 load commands, which should be future proof for
quite a while. I have attached a patch for configure.ac that does this
and changes the comment to reflect my understanding.

I've also provided a patch that makes the unexec error more helpful: it
computes the size needed for headerpad_extra given the number of load
commands that it needed for that particular unexec run.

I assume this affects the trunk code, too, though I didn't look at it.

-David
--- configure.ac.orig	2014-09-18 18:00:52.000000000 -0700
+++ configure.ac	2014-09-18 20:51:55.000000000 -0700
@@ -4856,17 +4856,19 @@
   darwin)
    ## The -headerpad option tells ld (see man page) to leave room at the
    ## end of the header for adding load commands.  Needed for dumping.
-   ## 0x690 is the total size of 30 segment load commands (at 56
-   ## each); under Cocoa 31 commands are required.
+   ## 0x1000 is enough for roughly 52 load commands on the x86_64
+   ## architecture (where they are 78 bytes each). The actual number of
+   ## load commands added is not consistent but normally ranges from
+   ## about 14 to about 34. Setting it high gets us plenty of slop and
+   ## only costs about 1.5K of wasted binary space.
+   headerpad_extra=1000
    if test "$HAVE_NS" = "yes"; then
      libs_nsgui="-framework AppKit"
      if test "$NS_IMPL_COCOA" = "yes"; then
         libs_nsgui="$libs_nsgui -framework IOKit"
      fi
-     headerpad_extra=6C8
    else
      libs_nsgui=
-     headerpad_extra=690
    fi
    LD_SWITCH_SYSTEM_TEMACS="-fno-pie -prebind $libs_nsgui -Xlinker -headerpad -Xlinker $headerpad_extra"
 
--- src/unexmacosx.c.orig	2014-09-18 20:53:21.000000000 -0700
+++ src/unexmacosx.c	2014-09-18 21:10:42.000000000 -0700
@@ -1302,7 +1306,9 @@
       }
 
   if (curr_header_offset > text_seg_lowest_offset)
-    unexec_error ("not enough room for load commands for new __DATA segments");
+    unexec_error ("not enough room for load commands for new __DATA segments"
+		  " (increase headerpad_extra in configure.in to at least %lX)",
+		  num_unexec_regions * sizeof (struct segment_command));
 
   printf ("%ld unused bytes follow Mach-O header\n",
 	  text_seg_lowest_offset - curr_header_offset);