From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 09 16:11:13 2011 Received: (at submit) by debbugs.gnu.org; 9 Aug 2011 20:11:13 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qqsds-0008A9-3p for submit@debbugs.gnu.org; Tue, 09 Aug 2011 16:11:12 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qqsdn-00089z-Ex for submit@debbugs.gnu.org; Tue, 09 Aug 2011 16:11:09 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qqsce-0007yl-TV for submit@debbugs.gnu.org; Tue, 09 Aug 2011 16:09:57 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-5.0 required=5.0 tests=BAYES_00,RCVD_IN_DNSWL_MED, RP_MATCHES_RCVD autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:59238) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qqsce-0007yh-S1 for submit@debbugs.gnu.org; Tue, 09 Aug 2011 16:09:56 -0400 Received: from eggs.gnu.org ([140.186.70.92]:45967) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qqscd-00066j-Ci for bug-gnu-emacs@gnu.org; Tue, 09 Aug 2011 16:09:56 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qqscb-0007yA-Po for bug-gnu-emacs@gnu.org; Tue, 09 Aug 2011 16:09:55 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141]:57630 helo=authusersmtp.mail.cornell.edu) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qqscb-0007xc-MU for bug-gnu-emacs@gnu.org; Tue, 09 Aug 2011 16:09:53 -0400 Received: from [128.84.234.236] (dhcp236.math.cornell.edu [128.84.234.236]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p79K9mMx003421 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT) for ; Tue, 9 Aug 2011 16:09:48 -0400 (EDT) Message-ID: <4E41940C.2010605@cornell.edu> Date: Tue, 09 Aug 2011 16:09:48 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.18) Gecko/20110616 Thunderbird/3.1.11 MIME-Version: 1.0 To: bug-gnu-emacs@gnu.org Subject: 23.3; malloc initialization should (sometimes) happen at runtime Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-detected-operating-system: by eggs.gnu.org: Solaris 9 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) The code in src/gmalloc.c makes assumptions about how a system maintains its memory that are not necessarily valid. In particular, they will not be valid on Cygwin starting with version 1.7.10 (which will almost certainly be released before emacs 24.1). The problem is that malloc initialization is done by temacs, and the results are dumped into emacs. This includes the setting __malloc_initialized = 1, so no malloc initialization is done when emacs is run. But the dumped value of _heapbase, while appropriate for temacs, may not point to the beginning of the runtime heap for emacs. This causes all code that uses the BLOCK and ADDRESS macros to be invalid. Here's what happens on Cygwin. temacs (on Cygwin) uses a static buffer as its heap and a function bss_sbrk that simulates sbrk. (See src/sheap.c.) The data in this buffer, including malloc information, are then dumped into emacs.exe as initialized data. But when the dumped emacs is run, it uses Cygwin's sbrk, which allocates memory on a heap that won't (as of Cygwin 1.7.10) be contiguous with the static heap. The saved value of _heapbase, which points into the static heap, is never changed, but it will mess up later calculations as soon as sbrk is called for the first time. All of this is described in detail on the Cygwin mailing list in the thread starting at http://cygwin.com/ml/cygwin/2011-08/msg00153.html See especially http://cygwin.com/ml/cygwin/2011-08/msg00193.html which contains a gdb session illustrating the problem. The context for that session is that, as a result of the problem I'm reporting, morecore_nolock went into an infinite loop. I attached gdb to that looping process. Maybe the solution is for emacs to do malloc initialization, including the assignment of _heapbase, every time it starts, at least on systems that use gmalloc.c. I made one naive attempt to do this, but it didn't work (and it was Cygwin specific). Namely, I made unexec (for Cygwin) set _malloc_initialized = 0 before dumping. The resulting emacs aborted as soon as it was started. I haven't figured out what went wrong, but I'm not sure that's the right answer anyway. Ken From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 09 20:25:48 2011 Received: (at 9273) by debbugs.gnu.org; 10 Aug 2011 00:25:49 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QqwcG-0007y9-1d for submit@debbugs.gnu.org; Tue, 09 Aug 2011 20:25:48 -0400 Received: from fencepost.gnu.org ([140.186.70.10]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QqwcD-0007y2-PK for 9273@debbugs.gnu.org; Tue, 09 Aug 2011 20:25:46 -0400 Received: from rms by fencepost.gnu.org with local (Exim 4.71) (envelope-from ) id 1Qqwb4-000581-Et; Tue, 09 Aug 2011 20:24:34 -0400 Date: Tue, 09 Aug 2011 20:24:34 -0400 Message-Id: Content-Type: text/plain; charset=ISO-8859-15 From: Richard Stallman To: Ken Brown In-reply-to: <4E41940C.2010605@cornell.edu> (message from Ken Brown on Tue, 09 Aug 2011 16:09:48 -0400) Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> X-Spam-Score: -6.6 (------) X-Debbugs-Envelope-To: 9273 Cc: 9273@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: rms@gnu.org List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -6.6 (------) If Cygwin does something very strange, there are limits to how far we would want to change Emacs to cope with that. -- Dr Richard Stallman President, Free Software Foundation 51 Franklin St Boston MA 02110 USA www.fsf.org www.gnu.org Skype: No way! That's nonfree (freedom-denying) software. Use free telephony http://directory.fsf.org/category/tel/ From debbugs-submit-bounces@debbugs.gnu.org Wed Aug 10 11:57:48 2011 Received: (at 9273) by debbugs.gnu.org; 10 Aug 2011 15:57:48 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrBAC-0007Qf-5X for submit@debbugs.gnu.org; Wed, 10 Aug 2011 11:57:48 -0400 Received: from mtaout22.012.net.il ([80.179.55.172]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrBAA-0007QX-2O for 9273@debbugs.gnu.org; Wed, 10 Aug 2011 11:57:47 -0400 Received: from conversion-daemon.a-mtaout22.012.net.il by a-mtaout22.012.net.il (HyperSendmail v2007.08) id <0LPP00100YVJ9G00@a-mtaout22.012.net.il> for 9273@debbugs.gnu.org; Wed, 10 Aug 2011 18:56:08 +0300 (IDT) Received: from HOME-C4E4A596F7 ([84.228.94.185]) by a-mtaout22.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LPP00LF3YXIFO61@a-mtaout22.012.net.il>; Wed, 10 Aug 2011 18:56:08 +0300 (IDT) Date: Wed, 10 Aug 2011 18:56:09 +0300 From: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime In-reply-to: <4E41940C.2010605@cornell.edu> X-012-Sender: halo1@inter.net.il To: Ken Brown Message-id: <83zkjhxnty.fsf@gnu.org> References: <4E41940C.2010605@cornell.edu> X-Spam-Score: -2.0 (--) X-Debbugs-Envelope-To: 9273 Cc: 9273@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.0 (--) > Date: Tue, 09 Aug 2011 16:09:48 -0400 > From: Ken Brown > > The code in src/gmalloc.c makes assumptions about how a system maintains > its memory that are not necessarily valid. In particular, they will not > be valid on Cygwin starting with version 1.7.10 (which will almost > certainly be released before emacs 24.1). The problem is that malloc > initialization is done by temacs, and the results are dumped into emacs. > This includes the setting __malloc_initialized = 1, so no malloc > initialization is done when emacs is run. But the dumped value of > _heapbase, while appropriate for temacs, may not point to the beginning > of the runtime heap for emacs. This causes all code that uses the BLOCK > and ADDRESS macros to be invalid. If Cygwin developers cannot or won't add to the Cygwin memory allocation enough features and knobs to cater to the special needs of Emacs dumping, then your only hope is to make Cygwin-specific changes to Emacs. You will see that 2 other ports that need to live with Windows memory allocation either have such knobs and features (DJGPP, used to build the DOS port; see the beginning of src/msdos.c) or use their own emulation of sbrk that upholds the contract expected by gmalloc.c and ralloc.c (w32heap.c, for the native Windows build). > But when the dumped emacs is run, it uses Cygwin's sbrk, which > allocates memory on a heap that won't (as of Cygwin 1.7.10) be > contiguous with the static heap. The saved value of _heapbase, > which points into the static heap, is never changed, but it will > mess up later calculations as soon as sbrk is called for the first > time. Are you sure this is all that's at work here? AFAIR, gmalloc does have code to cope with non-contiguous memory regions returned by sbrk. > All of this is described in detail on the Cygwin mailing list in the > thread starting at > > http://cygwin.com/ml/cygwin/2011-08/msg00153.html > > See especially > > http://cygwin.com/ml/cygwin/2011-08/msg00193.html I have read all the discussion there, but I'm sorry to say that I cannot figure out what you are talking about: there's too much Cygwin-isms in that thread that I couldn't penetrate. > Maybe the solution is for emacs to do malloc initialization, including > the assignment of _heapbase, every time it starts, at least on systems > that use gmalloc.c. Most supported systems don't need that. The native Windows build indeed does, see w32heap.c. Perhaps you could reuse some or even most of it for Cygwin. (What is so special about the Cygwin sbrk that is worth sticking to it?) > I made one naive attempt to do this, but it didn't work (and it was > Cygwin specific). Namely, I made unexec (for Cygwin) set > _malloc_initialized = 0 before dumping. The resulting emacs aborted > as soon as it was started. I haven't figured out what went wrong, > but I'm not sure that's the right answer anyway. One more evidence that something else is at work here. I would suggest to walk through the session that reinitializes the heap after unexec and see what goes wrong there. From debbugs-submit-bounces@debbugs.gnu.org Wed Aug 10 13:53:58 2011 Received: (at 9273) by debbugs.gnu.org; 10 Aug 2011 17:53:58 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrCyb-0001hb-CD for submit@debbugs.gnu.org; Wed, 10 Aug 2011 13:53:58 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrCyX-0001hQ-FO for 9273@debbugs.gnu.org; Wed, 10 Aug 2011 13:53:55 -0400 Received: from [128.84.234.236] (dhcp236.math.cornell.edu [128.84.234.236]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7AHqcIn009959 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); Wed, 10 Aug 2011 13:52:38 -0400 (EDT) Message-ID: <4E42C567.2000303@cornell.edu> Date: Wed, 10 Aug 2011 13:52:39 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.18) Gecko/20110616 Thunderbird/3.1.11 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> In-Reply-To: <83zkjhxnty.fsf@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) On 8/10/2011 11:56 AM, Eli Zaretskii wrote: >> Date: Tue, 09 Aug 2011 16:09:48 -0400 >> From: Ken Brown > >> But when the dumped emacs is run, it uses Cygwin's sbrk, which >> allocates memory on a heap that won't (as of Cygwin 1.7.10) be >> contiguous with the static heap. The saved value of _heapbase, >> which points into the static heap, is never changed, but it will >> mess up later calculations as soon as sbrk is called for the first >> time. > > Are you sure this is all that's at work here? AFAIR, gmalloc does > have code to cope with non-contiguous memory regions returned by sbrk. The issue isn't that sbrk returns non-contiguous regions. The issue is that two different of sbrk are used. One is used when temacs is running, and a different one is used when the dumped emacs.exe is running. This is controlled by the Cygwin-specific code in gmalloc.c, and it has to do with the way unexec works in the Cygwin build of emacs. >> All of this is described in detail on the Cygwin mailing list in the >> thread starting at >> >> http://cygwin.com/ml/cygwin/2011-08/msg00153.html >> >> See especially >> >> http://cygwin.com/ml/cygwin/2011-08/msg00193.html > > I have read all the discussion there, but I'm sorry to say that I > cannot figure out what you are talking about: there's too much > Cygwin-isms in that thread that I couldn't penetrate. Thanks for trying. >> Maybe the solution is for emacs to do malloc initialization, including >> the assignment of _heapbase, every time it starts, at least on systems >> that use gmalloc.c. > > Most supported systems don't need that. The native Windows build > indeed does, see w32heap.c. Perhaps you could reuse some or even most > of it for Cygwin. (What is so special about the Cygwin sbrk that is > worth sticking to it?) Thanks for the suggestions. Ken From debbugs-submit-bounces@debbugs.gnu.org Wed Aug 10 14:12:14 2011 Received: (at 9273) by debbugs.gnu.org; 10 Aug 2011 18:12:14 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrDGH-00028o-9g for submit@debbugs.gnu.org; Wed, 10 Aug 2011 14:12:14 -0400 Received: from mtaout20.012.net.il ([80.179.55.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrDGE-00028g-OC for 9273@debbugs.gnu.org; Wed, 10 Aug 2011 14:12:11 -0400 Received: from conversion-daemon.a-mtaout20.012.net.il by a-mtaout20.012.net.il (HyperSendmail v2007.08) id <0LPQ00C0053GQF00@a-mtaout20.012.net.il> for 9273@debbugs.gnu.org; Wed, 10 Aug 2011 21:10:35 +0300 (IDT) Received: from HOME-C4E4A596F7 ([84.228.94.185]) by a-mtaout20.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LPQ00ALV55L7F91@a-mtaout20.012.net.il>; Wed, 10 Aug 2011 21:10:34 +0300 (IDT) Date: Wed, 10 Aug 2011 21:10:36 +0300 From: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime In-reply-to: <4E42C567.2000303@cornell.edu> X-012-Sender: halo1@inter.net.il To: Ken Brown Message-id: <83k4alxhlv.fsf@gnu.org> References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E42C567.2000303@cornell.edu> X-Spam-Score: -2.0 (--) X-Debbugs-Envelope-To: 9273 Cc: 9273@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.0 (--) > Date: Wed, 10 Aug 2011 13:52:39 -0400 > From: Ken Brown > CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> > > On 8/10/2011 11:56 AM, Eli Zaretskii wrote: > >> Date: Tue, 09 Aug 2011 16:09:48 -0400 > >> From: Ken Brown > > > >> But when the dumped emacs is run, it uses Cygwin's sbrk, which > >> allocates memory on a heap that won't (as of Cygwin 1.7.10) be > >> contiguous with the static heap. The saved value of _heapbase, > >> which points into the static heap, is never changed, but it will > >> mess up later calculations as soon as sbrk is called for the first > >> time. > > > > Are you sure this is all that's at work here? AFAIR, gmalloc does > > have code to cope with non-contiguous memory regions returned by sbrk. > > The issue isn't that sbrk returns non-contiguous regions. The issue is > that two different of sbrk are used. One is used when temacs is > running, and a different one is used when the dumped emacs.exe is > running. I still don't see the problem: the memory sbrk'ed before dumping is frozen in the dumped Emacs, so I don't see how that could matter. Perhaps I'm missing something. From debbugs-submit-bounces@debbugs.gnu.org Wed Aug 10 14:51:11 2011 Received: (at 9273) by debbugs.gnu.org; 10 Aug 2011 18:51:11 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrDry-00036r-Nd for submit@debbugs.gnu.org; Wed, 10 Aug 2011 14:51:11 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QrDru-00036h-CU for 9273@debbugs.gnu.org; Wed, 10 Aug 2011 14:51:08 -0400 Received: from [128.84.234.236] (dhcp236.math.cornell.edu [128.84.234.236]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7AInowj019512 (version=TLSv1/SSLv3 cipher=AES256-SHA bits=256 verify=NOT); Wed, 10 Aug 2011 14:49:50 -0400 (EDT) Message-ID: <4E42D2CF.1080200@cornell.edu> Date: Wed, 10 Aug 2011 14:49:51 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.18) Gecko/20110616 Thunderbird/3.1.11 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E42C567.2000303@cornell.edu> <83k4alxhlv.fsf@gnu.org> In-Reply-To: <83k4alxhlv.fsf@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.8 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.8 (-----) On 8/10/2011 2:10 PM, Eli Zaretskii wrote: >> Date: Wed, 10 Aug 2011 13:52:39 -0400 >> From: Ken Brown >> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >> >> On 8/10/2011 11:56 AM, Eli Zaretskii wrote: >>>> Date: Tue, 09 Aug 2011 16:09:48 -0400 >>>> From: Ken Brown >>> >>>> But when the dumped emacs is run, it uses Cygwin's sbrk, which >>>> allocates memory on a heap that won't (as of Cygwin 1.7.10) be >>>> contiguous with the static heap. The saved value of _heapbase, >>>> which points into the static heap, is never changed, but it will >>>> mess up later calculations as soon as sbrk is called for the first >>>> time. >>> >>> Are you sure this is all that's at work here? AFAIR, gmalloc does >>> have code to cope with non-contiguous memory regions returned by sbrk. >> >> The issue isn't that sbrk returns non-contiguous regions. The issue is >> that two different of sbrk are used. One is used when temacs is >> running, and a different one is used when the dumped emacs.exe is >> running. > > I still don't see the problem: the memory sbrk'ed before dumping is > frozen in the dumped Emacs, so I don't see how that could matter. > Perhaps I'm missing something. The memory sbrk'ed before dumping is in the static heap, which is somewhere in relatively low memory. All the variables that malloc uses for keeping track of this involve these low addresses. But when the dumped emacs is run, Cygwin's sbrk is called, and it returns addresses starting at wherever Cygwin decides to put the heap (which will be either 0x20000000 or 0x80000000 in Cygwin 1.7.10, depending on whether or not large address awareness is enabled for emacs.exe and is supported by the underlying Windows system). The calculations done in gmalloc.c are based on the assumption that the heap starts in the same place in the dumped executable as it did before dumping. See especially the BLOCK and ADDRESS macros, which use the _heapbase variable. But _heapbase was set before dumping, and it points somewhere in the static heap; this is now much lower than the beginning of the runtime heap. The specific problem that led me to notice this was that under some circumstances emacs went into an infinite loop when executing the following (from morecore_nolock in gmalloc.c): newsize = heapsize; do newsize *= 2; while ((__malloc_size_t) BLOCK ((char *) result + size) > newsize); Here `result' is very large, and BLOCK returns a large number because it's using a small _heapbase. So the test is always true, newsize becomes 0 because of overflow, and the loop never terminates. Aside from the infinite loop, however, BLOCK and ADDRESS simply yield results that don't make sense when the heap starts in high memory but _heapbase points to low memory. Surprisingly, I haven't yet run into any problems when Cygwin's heap starts at 0x20000000. (It was 0x80000000 in the situation above.) I don't know if there's a good reason for this or if it's just luck. I think what happens is that malloc behaves as if it's allowed to allocate memory ranging all the way from the static heap to 0x20000000 and beyond. If for some reason it really is legal for malloc to use the memory between the static heap and 0x20000000, then I guess there's no harm done as long as the large addresses don't lead to overflow. Ken From debbugs-submit-bounces@debbugs.gnu.org Thu Aug 11 17:47:23 2011 Received: (at 9273) by debbugs.gnu.org; 11 Aug 2011 21:47:23 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrd62-0002Eg-SC for submit@debbugs.gnu.org; Thu, 11 Aug 2011 17:47:23 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrd60-0002ET-2I for 9273@debbugs.gnu.org; Thu, 11 Aug 2011 17:47:21 -0400 Received: from [192.168.1.4] (cpe-67-249-194-47.twcny.res.rr.com [67.249.194.47]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7BLjvTR013865 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Thu, 11 Aug 2011 17:45:58 -0400 (EDT) Message-ID: <4E444D85.4090907@cornell.edu> Date: Thu, 11 Aug 2011 17:45:41 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> In-Reply-To: <83zkjhxnty.fsf@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.7 (-----) On 8/10/2011 11:56 AM, Eli Zaretskii wrote: >> Date: Tue, 09 Aug 2011 16:09:48 -0400 >> From: Ken Brown >> I made one naive attempt to do this, but it didn't work (and it was >> Cygwin specific). Namely, I made unexec (for Cygwin) set >> _malloc_initialized = 0 before dumping. The resulting emacs aborted >> as soon as it was started. I haven't figured out what went wrong, >> but I'm not sure that's the right answer anyway. The problem was that realloc got called on memory that had been allocated prior to dumping, and the malloc information that was used then had disappeared. I think there's an obvious solution to this. At the time of reinitialization, we save the previous malloc state. Then if realloc is called on a pointer to something in the static heap, we temporarily restore the old state and let realloc proceed as it did in temacs prior to dumping. Unless I've (again) missed something obvious, it shouldn't be too hard to do this. I'm about to go on vacation, but I should have a patch ready within a couple weeks. Ken From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 12 02:55:51 2011 Received: (at 9273) by debbugs.gnu.org; 12 Aug 2011 06:55:51 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrlep-0005xr-0C for submit@debbugs.gnu.org; Fri, 12 Aug 2011 02:55:51 -0400 Received: from mtaout23.012.net.il ([80.179.55.175]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrlek-0005xg-CD for 9273@debbugs.gnu.org; Fri, 12 Aug 2011 02:55:47 -0400 Received: from conversion-daemon.a-mtaout23.012.net.il by a-mtaout23.012.net.il (HyperSendmail v2007.08) id <0LPS00F00Z0B1M00@a-mtaout23.012.net.il> for 9273@debbugs.gnu.org; Fri, 12 Aug 2011 09:54:12 +0300 (IDT) Received: from HOME-C4E4A596F7 ([84.228.94.185]) by a-mtaout23.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LPS00EZKZ6BKHA0@a-mtaout23.012.net.il>; Fri, 12 Aug 2011 09:54:12 +0300 (IDT) Date: Fri, 12 Aug 2011 09:54:14 +0300 From: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime In-reply-to: <4E444D85.4090907@cornell.edu> X-012-Sender: halo1@inter.net.il To: Ken Brown Message-id: <838vqzxgq1.fsf@gnu.org> References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> X-Spam-Score: -1.7 (-) X-Debbugs-Envelope-To: 9273 Cc: 9273@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -1.7 (-) > Date: Thu, 11 Aug 2011 17:45:41 -0400 > From: Ken Brown > CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> > > The problem was that realloc got called on memory that had been > allocated prior to dumping, and the malloc information that was used > then had disappeared. Can you show the code which called realloc on that memory? I'm surprised that Emacs does that, but perhaps I'm missing something. From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 12 06:12:19 2011 Received: (at 9273) by debbugs.gnu.org; 12 Aug 2011 10:12:19 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qroix-0001rT-5R for submit@debbugs.gnu.org; Fri, 12 Aug 2011 06:12:19 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qroiv-0001rL-16 for 9273@debbugs.gnu.org; Fri, 12 Aug 2011 06:12:18 -0400 Received: from [192.168.1.4] (cpe-67-249-194-47.twcny.res.rr.com [67.249.194.47]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7CAApOG022438 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 12 Aug 2011 06:10:52 -0400 (EDT) Message-ID: <4E44FC1B.9010207@cornell.edu> Date: Fri, 12 Aug 2011 06:10:35 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> In-Reply-To: <838vqzxgq1.fsf@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.7 (-----) On 8/12/2011 2:54 AM, Eli Zaretskii wrote: >> Date: Thu, 11 Aug 2011 17:45:41 -0400 >> From: Ken Brown >> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >> >> The problem was that realloc got called on memory that had been >> allocated prior to dumping, and the malloc information that was used >> then had disappeared. > > Can you show the code which called realloc on that memory? I'm > surprised that Emacs does that, but perhaps I'm missing something. Here's the code that I stumbled across (as a result of a SEGV). I haven't checked to see if there are other examples. From terminal.c: /* Deletes the bootstrap terminal device. Called through delete_terminal_hook. */ static void delete_initial_terminal (struct terminal *terminal) { if (terminal != initial_terminal) abort (); delete_terminal (terminal); initial_terminal = NULL; } From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 12 07:35:30 2011 Received: (at 9273) by debbugs.gnu.org; 12 Aug 2011 11:35:30 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrq1S-0004Wj-B6 for submit@debbugs.gnu.org; Fri, 12 Aug 2011 07:35:30 -0400 Received: from mtaout22.012.net.il ([80.179.55.172]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrq1Q-0004WX-1H for 9273@debbugs.gnu.org; Fri, 12 Aug 2011 07:35:29 -0400 Received: from conversion-daemon.a-mtaout22.012.net.il by a-mtaout22.012.net.il (HyperSendmail v2007.08) id <0LPT00400BZ3HV00@a-mtaout22.012.net.il> for 9273@debbugs.gnu.org; Fri, 12 Aug 2011 14:33:11 +0300 (IDT) Received: from HOME-C4E4A596F7 ([84.228.94.185]) by a-mtaout22.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LPT0040MC38BG30@a-mtaout22.012.net.il>; Fri, 12 Aug 2011 14:33:09 +0300 (IDT) Date: Fri, 12 Aug 2011 14:33:13 +0300 From: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime In-reply-to: <4E44FC1B.9010207@cornell.edu> X-012-Sender: halo1@inter.net.il To: Ken Brown Message-id: <831uwqyidi.fsf@gnu.org> References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> X-Spam-Score: -2.0 (--) X-Debbugs-Envelope-To: 9273 Cc: 9273@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.0 (--) > Date: Fri, 12 Aug 2011 06:10:35 -0400 > From: Ken Brown > CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> > > On 8/12/2011 2:54 AM, Eli Zaretskii wrote: > >> Date: Thu, 11 Aug 2011 17:45:41 -0400 > >> From: Ken Brown > >> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> > >> > >> The problem was that realloc got called on memory that had been > >> allocated prior to dumping, and the malloc information that was used > >> then had disappeared. > > > > Can you show the code which called realloc on that memory? I'm > > surprised that Emacs does that, but perhaps I'm missing something. > > Here's the code that I stumbled across (as a result of a SEGV). I > haven't checked to see if there are other examples. From terminal.c: > > /* Deletes the bootstrap terminal device. > Called through delete_terminal_hook. */ > > static void > delete_initial_terminal (struct terminal *terminal) > { > if (terminal != initial_terminal) > abort (); > > delete_terminal (terminal); > initial_terminal = NULL; > } delete_terminal doesn't call realloc, it just calls xfree. Do the problems with the Cygwin build go away if the call to delete_terminal is commented out? That is, does the infloop still happen or not? From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 12 08:20:25 2011 Received: (at 9273) by debbugs.gnu.org; 12 Aug 2011 12:20:25 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrqiu-0005fU-4S for submit@debbugs.gnu.org; Fri, 12 Aug 2011 08:20:25 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qrqir-0005fM-2o for 9273@debbugs.gnu.org; Fri, 12 Aug 2011 08:20:22 -0400 Received: from [192.168.1.4] (cpe-67-249-194-47.twcny.res.rr.com [67.249.194.47]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7CCIsqD009313 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 12 Aug 2011 08:18:55 -0400 (EDT) Message-ID: <4E451A1F.4060409@cornell.edu> Date: Fri, 12 Aug 2011 08:18:39 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> In-Reply-To: <831uwqyidi.fsf@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.7 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.7 (-----) On 8/12/2011 7:33 AM, Eli Zaretskii wrote: >> Date: Fri, 12 Aug 2011 06:10:35 -0400 >> From: Ken Brown >> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >> >> On 8/12/2011 2:54 AM, Eli Zaretskii wrote: >>>> Date: Thu, 11 Aug 2011 17:45:41 -0400 >>>> From: Ken Brown >>>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >>>> >>>> The problem was that realloc got called on memory that had been >>>> allocated prior to dumping, and the malloc information that was used >>>> then had disappeared. >>> >>> Can you show the code which called realloc on that memory? I'm >>> surprised that Emacs does that, but perhaps I'm missing something. >> >> Here's the code that I stumbled across (as a result of a SEGV). I >> haven't checked to see if there are other examples. From terminal.c: >> >> /* Deletes the bootstrap terminal device. >> Called through delete_terminal_hook. */ >> >> static void >> delete_initial_terminal (struct terminal *terminal) >> { >> if (terminal != initial_terminal) >> abort (); >> >> delete_terminal (terminal); >> initial_terminal = NULL; >> } > > delete_terminal doesn't call realloc, it just calls xfree. Maybe I mis-remembered where the call to realloc is. I'll reproduce it later and let you know. (I don't have time at the moment.) But I assure you that I did a backtrace showing that realloc was called on something related to terminals. > Do the problems with the Cygwin build go away if the call to > delete_terminal is commented out? No. At the very least, I have to force reinitialization of malloc. Otherwise the BLOCK macro yields wrong results that lead to infinite looping or crashing. After reinitialization, I have to be able to handle calls to free() on memory allocated prior to dumping. Probably it's OK to just ignore such calls. If I can also take care of calls to realloc too, everything will be OK. Ken From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 12 16:26:17 2011 Received: (at 9273) by debbugs.gnu.org; 12 Aug 2011 20:26:17 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QryJ7-0000YH-BE for submit@debbugs.gnu.org; Fri, 12 Aug 2011 16:26:17 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QryJ3-0000Y4-Pq for 9273@debbugs.gnu.org; Fri, 12 Aug 2011 16:26:15 -0400 Received: from [192.168.1.92] (c-76-118-2-147.hsd1.ma.comcast.net [76.118.2.147]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7CKOegg026434 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Fri, 12 Aug 2011 16:24:45 -0400 (EDT) Message-ID: <4E458BF4.9080108@cornell.edu> Date: Fri, 12 Aug 2011 16:24:20 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> In-Reply-To: <4E451A1F.4060409@cornell.edu> Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) On 8/12/2011 8:18 AM, Ken Brown wrote: > On 8/12/2011 7:33 AM, Eli Zaretskii wrote: >>> Date: Fri, 12 Aug 2011 06:10:35 -0400 >>> From: Ken Brown >>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >>> >>> On 8/12/2011 2:54 AM, Eli Zaretskii wrote: >>>>> Date: Thu, 11 Aug 2011 17:45:41 -0400 >>>>> From: Ken Brown >>>>> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >>>>> >>>>> The problem was that realloc got called on memory that had been >>>>> allocated prior to dumping, and the malloc information that was used >>>>> then had disappeared. >>>> >>>> Can you show the code which called realloc on that memory? I'm >>>> surprised that Emacs does that, but perhaps I'm missing something. >>> >>> Here's the code that I stumbled across (as a result of a SEGV). I >>> haven't checked to see if there are other examples. From terminal.c: >>> >>> /* Deletes the bootstrap terminal device. >>> Called through delete_terminal_hook. */ >>> >>> static void >>> delete_initial_terminal (struct terminal *terminal) >>> { >>> if (terminal != initial_terminal) >>> abort (); >>> >>> delete_terminal (terminal); >>> initial_terminal = NULL; >>> } >> >> delete_terminal doesn't call realloc, it just calls xfree. > > Maybe I mis-remembered where the call to realloc is. I'll reproduce it > later and let you know. (I don't have time at the moment.) But I > assure you that I did a backtrace showing that realloc was called on > something related to terminals. > >> Do the problems with the Cygwin build go away if the call to >> delete_terminal is commented out? > > No. At the very least, I have to force reinitialization of malloc. > Otherwise the BLOCK macro yields wrong results that lead to infinite > looping or crashing. After reinitialization, I have to be able to > handle calls to free() on memory allocated prior to dumping. Probably > it's OK to just ignore such calls. If I can also take care of calls to > realloc too, everything will be OK. OK, here's a backtrace showing realloc being called on memory in the static heap (at 0x897040). This is after applying the patch appended at the end of this message. (I think it's self-explanatory, but I'll be glad to explain further.) (gdb) r -Q Starting program: /home/kbrown/src/emacs/test/src/emacs.exe -Q [New Thread 4756.0x1144] warning: cYgFFFFFFFF 611857C0 [New Thread 4756.0xd80] warning: cYgstd 28ccf5 d 3 Program received signal SIGSEGV, Segmentation fault. 0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28) at gmalloc.c:1394 1394 type = _heapinfo[block].busy.type; (gdb) p block $1 = 4294838425 (gdb) bt #0 0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28) at gmalloc.c:1394 #1 0x00636bd7 in _realloc_internal (ptr=0x897040, size=28) at gmalloc.c:1499 #2 0x00636c42 in realloc (ptr=0x897040, size=28) at gmalloc.c:1516 #3 0x00596856 in xrealloc (block=0x897040, size=28) at alloc.c:711 #4 0x00589648 in regex_compile (pattern=0xa7ec60 "site-lisp", size=9, syntax=3408388, bufp=0x846258) at regex.c:3684 #5 0x0059556d in re_compile_pattern (pattern=0xa7ec60 "site-lisp", length=9, bufp=0x846258) at regex.c:6361 #6 0x005768d0 in compile_pattern_1 (cp=0x846248, pattern=9810241, translate=8930309, posix=0) at search.c:150 #7 0x00576b32 in compile_pattern (pattern=9810241, regp=0x8475d8, translate=8930309, posix=0, multibyte=0) at search.c:245 #8 0x005771b8 in string_match_1 (regexp=9810241, string=9810337, start=8968218, posix=0) at search.c:401 #9 0x005773ab in Fstring_match (regexp=9810241, string=9810337, start=8968218) at search.c:451 #10 0x005e4f91 in init_lread () at lread.c:4111 #11 0x0052866c in main (argc=2, argv=0x2001cc00) at emacs.c:1467 (gdb) p _heapbase $3 = 0x20000000 "" (gdb) p block $1 = 4294838425 The SEGV comes from the ridiculous value of block, which was calculated by the BLOCK macro. === modified file 'src/gmalloc.c' --- src/gmalloc.c 2011-08-04 17:04:39 +0000 +++ src/gmalloc.c 2011-08-12 19:47:21 +0000 @@ -584,6 +584,12 @@ mcheck (NULL); #endif +#ifdef CYGWIN + if (bss_sbrk_did_unexec) + /* we're reinitializing the dumped emacs. */ + memset (_fraghead, 0, BLOCKLOG * sizeof (struct list)); +#endif + if (__malloc_initialize_hook) (*__malloc_initialize_hook) (); @@ -1054,6 +1060,12 @@ if (ptr == NULL) return; +#ifdef CYGWIN + if (ptr < _heapbase) + /* we're being asked to free something in the static heap */ + return; +#endif + PROTECT_MALLOC_STATE (0); LOCK_ALIGNED_BLOCKS (); === modified file 'src/unexcw.c' --- src/unexcw.c 2011-03-17 20:18:59 +0000 +++ src/unexcw.c 2011-08-12 15:37:47 +0000 @@ -33,6 +33,8 @@ extern int bss_sbrk_did_unexec; +extern int __malloc_initialized; + /* emacs symbols that indicate where bss and data end for emacs internals */ extern char my_endbss[]; extern char my_edata[]; @@ -210,9 +212,12 @@ lseek (fd, (long) (exe_header->section_header[i].s_scnptr), SEEK_SET); assert (ret != -1); + /* force the dumped emacs to reinitialize malloc */ + __malloc_initialized = 0; ret = write (fd, (char *) start_address, my_endbss - (char *) start_address); + __malloc_initialized = 1; assert (ret == (my_endbss - (char *) start_address)); if (debug_unexcw) printf (" .bss, mem start 0x%08x mem length %d\n", From debbugs-submit-bounces@debbugs.gnu.org Fri Aug 12 19:53:23 2011 Received: (at submit) by debbugs.gnu.org; 12 Aug 2011 23:53:23 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qs1XX-0005LX-0d for submit@debbugs.gnu.org; Fri, 12 Aug 2011 19:53:23 -0400 Received: from eggs.gnu.org ([140.186.70.92]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qs1XV-0005LR-PN for submit@debbugs.gnu.org; Fri, 12 Aug 2011 19:53:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qs1W5-00065L-8Y for submit@debbugs.gnu.org; Fri, 12 Aug 2011 19:51:53 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.1 (2010-03-16) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-1.9 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_NONE, T_TO_NO_BRKTS_FREEMAIL autolearn=unavailable version=3.3.1 Received: from lists.gnu.org ([140.186.70.17]:60558) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qs1W5-00065H-78 for submit@debbugs.gnu.org; Fri, 12 Aug 2011 19:51:53 -0400 Received: from eggs.gnu.org ([140.186.70.92]:55028) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Qs1W4-0000kT-9u for bug-gnu-emacs@gnu.org; Fri, 12 Aug 2011 19:51:53 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Qs1W3-000651-7I for bug-gnu-emacs@gnu.org; Fri, 12 Aug 2011 19:51:52 -0400 Received: from mailout-de.gmx.net ([213.165.64.22]:46500) by eggs.gnu.org with smtp (Exim 4.71) (envelope-from ) id 1Qs1W2-00064v-OJ for bug-gnu-emacs@gnu.org; Fri, 12 Aug 2011 19:51:51 -0400 Received: (qmail invoked by alias); 12 Aug 2011 23:51:48 -0000 Received: from unknown (EHLO [10.65.63.233]) [82.113.106.41] by mail.gmx.net (mp048) with SMTP; 13 Aug 2011 01:51:48 +0200 X-Authenticated: #18588216 X-Provags-ID: V01U2FsdGVkX18TPe4ZgywTSd0/ngxYSZnVrv5vKZvbMmhArDPVje daJvIN8xDJmdHM Message-ID: <4E45BC8E.7050104@gmx.de> Date: Sat, 13 Aug 2011 01:51:42 +0200 From: grischka User-Agent: Thunderbird 2.0.0.24 (Windows/20100228) MIME-Version: 1.0 To: kbrown@cornell.edu Subject: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: 4E451A1F.4060409@cornell.edu Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Y-GMX-Trusted: 0 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6 (newer, 3) X-Received-From: 140.186.70.17 X-Spam-Score: -5.4 (-----) X-Debbugs-Envelope-To: submit Cc: bug-gnu-emacs@gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.4 (-----) > No. At the very least, I have to force reinitialization of malloc. > Otherwise the BLOCK macro yields wrong results that lead to infinite > looping or crashing. I don't think the results from the BLOCK macro are wrong. I think your analysis of the problem is wrong. ;) --- grischka From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 04:06:50 2011 Received: (at 9273) by debbugs.gnu.org; 13 Aug 2011 08:06:50 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qs9F4-0007ha-89 for submit@debbugs.gnu.org; Sat, 13 Aug 2011 04:06:50 -0400 Received: from mtaout21.012.net.il ([80.179.55.169]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1Qs9F1-0007hO-Ju for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 04:06:49 -0400 Received: from conversion-daemon.a-mtaout21.012.net.il by a-mtaout21.012.net.il (HyperSendmail v2007.08) id <0LPU00B00WPXNP00@a-mtaout21.012.net.il> for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 11:05:16 +0300 (IDT) Received: from HOME-C4E4A596F7 ([84.229.155.132]) by a-mtaout21.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LPU00BAFX4RMO40@a-mtaout21.012.net.il>; Sat, 13 Aug 2011 11:05:16 +0300 (IDT) Date: Sat, 13 Aug 2011 11:05:20 +0300 From: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime In-reply-to: <4E458BF4.9080108@cornell.edu> X-012-Sender: halo1@inter.net.il To: Ken Brown Message-id: <83fwl5wxbz.fsf@gnu.org> References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> X-Spam-Score: -2.0 (--) X-Debbugs-Envelope-To: 9273 Cc: 9273@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.0 (--) > Date: Fri, 12 Aug 2011 16:24:20 -0400 > From: Ken Brown > CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> > > Program received signal SIGSEGV, Segmentation fault. > 0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28) > at gmalloc.c:1394 > 1394 type = _heapinfo[block].busy.type; > (gdb) p block > $1 = 4294838425 I'm confused: since you patched unexecw.c to set __malloc_initialized to zero, the dumped Emacs should have called malloc_initialize_1, which should have allocated a new copy of _heapinfo, that was supposed to be consistent with the current heap. Why isn't that working? why `block' still gets a value that is relative to the "old" _heapinfo? From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 09:50:50 2011 Received: (at 9273) by debbugs.gnu.org; 13 Aug 2011 13:50:50 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsEbw-0007WJ-T4 for submit@debbugs.gnu.org; Sat, 13 Aug 2011 09:50:49 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsEbp-0007W6-QI for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 09:50:44 -0400 Received: from [192.168.1.96] (c-76-118-2-147.hsd1.ma.comcast.net [76.118.2.147]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7DDn8DA012864 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 13 Aug 2011 09:49:09 -0400 (EDT) Message-ID: <4E4680C4.6040605@cornell.edu> Date: Sat, 13 Aug 2011 09:48:52 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> In-Reply-To: <83fwl5wxbz.fsf@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) On 8/13/2011 4:05 AM, Eli Zaretskii wrote: >> Date: Fri, 12 Aug 2011 16:24:20 -0400 >> From: Ken Brown >> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >> >> Program received signal SIGSEGV, Segmentation fault. >> 0x006368f5 in _realloc_internal_nolock (ptr=0x897040, size=28) >> at gmalloc.c:1394 >> 1394 type = _heapinfo[block].busy.type; >> (gdb) p block >> $1 = 4294838425 > > I'm confused: since you patched unexecw.c to set __malloc_initialized > to zero, the dumped Emacs should have called malloc_initialize_1, > which should have allocated a new copy of _heapinfo, that was supposed > to be consistent with the current heap. Why isn't that working? why > `block' still gets a value that is relative to the "old" _heapinfo? _heapinfo is indeed consistent with the current heap. But the pointer that was passed to realloc points into the old heap. So applying BLOCK to that pointer yields an absurd result. I can easily catch such cases by testing for ptr < _heapbase, as in my patch to _free_internal_nolock, but I have to figure out the best way to handle them once I've caught them. I have work in progress that tries to keep track of both heaps, but I haven't got it working yet. An alternative would be to have realloc return NULL (or some other special value) in these cases, but then I would have to find all possible callers of realloc (with pointers to the old heap) and make sure they know how to deal with that return value. I'm guessing my first approach is safer and easier to implement. Ken From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 10:43:18 2011 Received: (at 9273) by debbugs.gnu.org; 13 Aug 2011 14:43:18 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsFQk-0000Ou-3c for submit@debbugs.gnu.org; Sat, 13 Aug 2011 10:43:18 -0400 Received: from mtaout20.012.net.il ([80.179.55.166]) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsFQg-0000Ok-5o for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 10:43:15 -0400 Received: from conversion-daemon.a-mtaout20.012.net.il by a-mtaout20.012.net.il (HyperSendmail v2007.08) id <0LPV00600FCN8Y00@a-mtaout20.012.net.il> for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 17:41:41 +0300 (IDT) Received: from HOME-C4E4A596F7 ([84.229.155.132]) by a-mtaout20.012.net.il (HyperSendmail v2007.08) with ESMTPA id <0LPV0047JFHGKTP0@a-mtaout20.012.net.il>; Sat, 13 Aug 2011 17:41:41 +0300 (IDT) Date: Sat, 13 Aug 2011 17:41:45 +0300 From: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime In-reply-to: <4E4680C4.6040605@cornell.edu> X-012-Sender: halo1@inter.net.il To: Ken Brown Message-id: <83d3g9weza.fsf@gnu.org> References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> <4E4680C4.6040605@cornell.edu> X-Spam-Score: -2.0 (--) X-Debbugs-Envelope-To: 9273 Cc: 9273@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list Reply-To: Eli Zaretskii List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.0 (--) > Date: Sat, 13 Aug 2011 09:48:52 -0400 > From: Ken Brown > CC: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> > > _heapinfo is indeed consistent with the current heap. But the pointer > that was passed to realloc points into the old heap. So applying BLOCK > to that pointer yields an absurd result. I can easily catch such cases > by testing for ptr < _heapbase, as in my patch to _free_internal_nolock, > but I have to figure out the best way to handle them once I've caught > them. malloc a buffer, then copy the contents of the old one to the new one. You will have to know the size of the old block, which means you will have to access the old copy of _heapinfo. From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 10:55:21 2011 Received: (at 9273) by debbugs.gnu.org; 13 Aug 2011 14:55:21 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsFcP-0000fQ-7a for submit@debbugs.gnu.org; Sat, 13 Aug 2011 10:55:21 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsFcM-0000fI-Vp for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 10:55:20 -0400 Received: from [192.168.1.98] (c-76-118-2-147.hsd1.ma.comcast.net [76.118.2.147]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7DErfF0021377 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 13 Aug 2011 10:53:47 -0400 (EDT) Message-ID: <4E468FE5.100@cornell.edu> Date: Sat, 13 Aug 2011 10:53:25 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> <4E4680C4.6040605@cornell.edu> <83d3g9weza.fsf@gnu.org> In-Reply-To: <83d3g9weza.fsf@gnu.org> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org> X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) On 8/13/2011 10:41 AM, Eli Zaretskii wrote: >> Date: Sat, 13 Aug 2011 09:48:52 -0400 >> From: Ken Brown >> CC: "9273@debbugs.gnu.org"<9273@debbugs.gnu.org> >> >> _heapinfo is indeed consistent with the current heap. But the pointer >> that was passed to realloc points into the old heap. So applying BLOCK >> to that pointer yields an absurd result. I can easily catch such cases >> by testing for ptr< _heapbase, as in my patch to _free_internal_nolock, >> but I have to figure out the best way to handle them once I've caught >> them. > > malloc a buffer, then copy the contents of the old one to the new > one. You will have to know the size of the old block, which means you > will have to access the old copy of _heapinfo. Thanks! That's much easier than what I was trying to do. Ken From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 11:09:22 2011 Received: (at 9273) by debbugs.gnu.org; 13 Aug 2011 15:09:22 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsFpy-0000y5-NT for submit@debbugs.gnu.org; Sat, 13 Aug 2011 11:09:22 -0400 Received: from ironport2-out.teksavvy.com ([206.248.154.183] helo=ironport2-out.pppoe.ca) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsFpx-0000xy-50 for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 11:09:21 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EALOSRk5FxKeo/2dsb2JhbABBDqdpeIFAAQEFViMQCw4mEhQYDSTBKIZHBJ9ug2FU X-IronPort-AV: E=Sophos;i="4.67,367,1309752000"; d="scan'208";a="131001511" Received: from 69-196-167-168.dsl.teksavvy.com (HELO ceviche.home) ([69.196.167.168]) by ironport2-out.pppoe.ca with ESMTP/TLS/ADH-AES256-SHA; 13 Aug 2011 11:07:48 -0400 Received: by ceviche.home (Postfix, from userid 20848) id 26540660CF; Sat, 13 Aug 2011 11:07:48 -0400 (EDT) From: Stefan Monnier To: Ken Brown Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime Message-ID: References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> <4E4680C4.6040605@cornell.edu> <83d3g9weza.fsf@gnu.org> <4E468FE5.100@cornell.edu> Date: Sat, 13 Aug 2011 11:07:48 -0400 In-Reply-To: <4E468FE5.100@cornell.edu> (Ken Brown's message of "Sat, 13 Aug 2011 10:53:25 -0400") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>, Eli Zaretskii X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.1 (--) >>> _heapinfo is indeed consistent with the current heap. But the pointer >>> that was passed to realloc points into the old heap. So applying BLOCK >>> to that pointer yields an absurd result. I can easily catch such cases >>> by testing for ptr< _heapbase, as in my patch to _free_internal_nolock, >>> but I have to figure out the best way to handle them once I've caught >>> them. >> malloc a buffer, then copy the contents of the old one to the new >> one. You will have to know the size of the old block, which means you >> will have to access the old copy of _heapinfo. > Thanks! That's much easier than what I was trying to do. Wouldn't it be easier to try and convince malloc to keep using the old _heapinfo? Stefan From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 11:35:56 2011 Received: (at 9273) by debbugs.gnu.org; 13 Aug 2011 15:35:56 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsGFg-0001Y3-1O for submit@debbugs.gnu.org; Sat, 13 Aug 2011 11:35:56 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsGFd-0001Xw-FY for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 11:35:54 -0400 Received: from [192.168.1.98] (c-76-118-2-147.hsd1.ma.comcast.net [76.118.2.147]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7DFYHDG026907 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 13 Aug 2011 11:34:18 -0400 (EDT) Message-ID: <4E469967.40201@cornell.edu> Date: Sat, 13 Aug 2011 11:33:59 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Stefan Monnier Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> <4E4680C4.6040605@cornell.edu> <83d3g9weza.fsf@gnu.org> <4E468FE5.100@cornell.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>, Eli Zaretskii X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) On 8/13/2011 11:07 AM, Stefan Monnier wrote: >>>> _heapinfo is indeed consistent with the current heap. But the pointer >>>> that was passed to realloc points into the old heap. So applying BLOCK >>>> to that pointer yields an absurd result. I can easily catch such cases >>>> by testing for ptr< _heapbase, as in my patch to _free_internal_nolock, >>>> but I have to figure out the best way to handle them once I've caught >>>> them. >>> malloc a buffer, then copy the contents of the old one to the new >>> one. You will have to know the size of the old block, which means you >>> will have to access the old copy of _heapinfo. >> Thanks! That's much easier than what I was trying to do. > > Wouldn't it be easier to try and convince malloc to keep using the old > _heapinfo? Probably. That would keep me from having to get involved in the details of how _heapinfo works, and it would probably be less error prone. So I would do something like the following: 1. Temporarily restore the pre-dump malloc state. 2. Call xrealloc, getting storage of the new size in the old heap. 3. Copy the contents into a temporary buffer, and then free the recently allocated storage in the old heap. 4. Restore the malloc state, call malloc, and then copy the contents of the temporary buffer into the just-acquired storage in the new heap. Is that what you had in mind? Ken From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 15:20:42 2011 Received: (at 9273) by debbugs.gnu.org; 13 Aug 2011 19:20:42 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsJlC-0006ZP-82 for submit@debbugs.gnu.org; Sat, 13 Aug 2011 15:20:42 -0400 Received: from ironport2-out.teksavvy.com ([206.248.154.181] helo=ironport2-out.pppoe.ca) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsJl8-0006ZH-MN for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 15:20:39 -0400 X-IronPort-Anti-Spam-Filtered: true X-IronPort-Anti-Spam-Result: Av0EAFXNRk5FxKeo/2dsb2JhbABBDqdpeIFAAQEEAVYjBQsLDiYSFBgNJIgDuDaGRwSfboNhVA X-IronPort-AV: E=Sophos;i="4.67,367,1309752000"; d="scan'208";a="131053309" Received: from 69-196-167-168.dsl.teksavvy.com (HELO ceviche.home) ([69.196.167.168]) by ironport2-out.pppoe.ca with ESMTP/TLS/ADH-AES256-SHA; 13 Aug 2011 15:19:05 -0400 Received: by ceviche.home (Postfix, from userid 20848) id 0A74F660CF; Sat, 13 Aug 2011 15:19:05 -0400 (EDT) From: Stefan Monnier To: Ken Brown Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime Message-ID: References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> <4E4680C4.6040605@cornell.edu> <83d3g9weza.fsf@gnu.org> <4E468FE5.100@cornell.edu> <4E469967.40201@cornell.edu> Date: Sat, 13 Aug 2011 15:19:05 -0400 In-Reply-To: <4E469967.40201@cornell.edu> (Ken Brown's message of "Sat, 13 Aug 2011 11:33:59 -0400") User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Score: -2.1 (--) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>, Eli Zaretskii X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -2.1 (--) > Probably. That would keep me from having to get involved in the details of > how _heapinfo works, and it would probably be less error prone. So I would > do something like the following: > 1. Temporarily restore the pre-dump malloc state. > 2. Call xrealloc, getting storage of the new size in the old heap. > 3. Copy the contents into a temporary buffer, and then free the recently > allocated storage in the old heap. > 4. Restore the malloc state, call malloc, and then copy the contents of the > temporary buffer into the just-acquired storage in the new heap. > Is that what you had in mind? No, I meant, during initialization of Emacs, make sure we get the old _heapinfo state, so that malloc/realloc/... just works. Stefan From debbugs-submit-bounces@debbugs.gnu.org Sat Aug 13 23:15:37 2011 Received: (at 9273) by debbugs.gnu.org; 14 Aug 2011 03:15:37 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsRAm-0000Cv-VE for submit@debbugs.gnu.org; Sat, 13 Aug 2011 23:15:37 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QsRAj-0000Cn-E5 for 9273@debbugs.gnu.org; Sat, 13 Aug 2011 23:15:35 -0400 Received: from [172.160.101.188] (fl-184-7-236-177.sta.embarqhsd.net [184.7.236.177]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7E3Dt2Y028850 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Sat, 13 Aug 2011 23:13:57 -0400 (EDT) Message-ID: <4E473D63.7040703@cornell.edu> Date: Sat, 13 Aug 2011 23:13:39 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Stefan Monnier Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> <4E4680C4.6040605@cornell.edu> <83d3g9weza.fsf@gnu.org> <4E468FE5.100@cornell.edu> <4E469967.40201@cornell.edu> In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: 9273 Cc: "9273@debbugs.gnu.org" <9273@debbugs.gnu.org>, Eli Zaretskii X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) On 8/13/2011 3:19 PM, Stefan Monnier wrote: >> Probably. That would keep me from having to get involved in the details of >> how _heapinfo works, and it would probably be less error prone. So I would >> do something like the following: > >> 1. Temporarily restore the pre-dump malloc state. >> 2. Call xrealloc, getting storage of the new size in the old heap. >> 3. Copy the contents into a temporary buffer, and then free the recently >> allocated storage in the old heap. >> 4. Restore the malloc state, call malloc, and then copy the contents of the >> temporary buffer into the just-acquired storage in the new heap. > >> Is that what you had in mind? > > No, I meant, during initialization of Emacs, make sure we get the old > _heapinfo state, so that malloc/realloc/... just works. No, that wouldn't be good because the old _heapinfo state uses the static heap, which is very small. It's big enough for temacs, but not for a usable emacs. In any case, Eli's suggestion turned out to be very easy to implement. My revised patch is appended below, and it seems to solve all the problems I encountered. I still want to test it further, but I think it's OK. Eli, thanks for your help. Ken === modified file 'src/gmalloc.c' --- src/gmalloc.c 2011-08-04 17:04:39 +0000 +++ src/gmalloc.c 2011-08-13 22:32:45 +0000 @@ -351,11 +351,19 @@ #endif #include -/* How to really get more memory. */ -#if defined(CYGWIN) +/* On Cygwin there are two heaps. temacs uses the static heap + (defined in sheap.c and managed with bss_sbrk), and the dumped + emacs uses the Cygwin heap (managed with sbrk). When emacs starts + on Cygwin, it reinitializes malloc, and we save the old info for + use by free and realloc if they're called with a pointer into the + static heap. */ +#ifdef CYGWIN extern __ptr_t bss_sbrk PP ((ptrdiff_t __size)); extern int bss_sbrk_did_unexec; +char *bss_sbrk_heapbase; +malloc_info *bss_sbrk_heapinfo; #endif + __ptr_t (*__morecore) PP ((__malloc_ptrdiff_t __size)) = __default_morecore; /* Debugging hook for `malloc'. */ @@ -584,6 +592,16 @@ mcheck (NULL); #endif +#ifdef CYGWIN + if (bss_sbrk_did_unexec) + /* we're reinitializing the dumped emacs */ + { + bss_sbrk_heapbase = _heapbase; + bss_sbrk_heapinfo = _heapinfo; + memset (_fraghead, 0, BLOCKLOG * sizeof (struct list)); + } +#endif + if (__malloc_initialize_hook) (*__malloc_initialize_hook) (); @@ -1054,6 +1072,12 @@ if (ptr == NULL) return; +#ifdef CYGWIN + if (ptr < _heapbase) + /* We're being asked to free something in the static heap. */ + return; +#endif + PROTECT_MALLOC_STATE (0); LOCK_ALIGNED_BLOCKS (); @@ -1346,9 +1370,33 @@ #include #endif - #define min(A, B) ((A) < (B) ? (A) : (B)) +/* On Cygwin the dumped emacs may try to realloc storage allocated in + the static heap. We just malloc space in the new heap and copy the + data. */ +#ifdef CYGWIN +__ptr_t +special_realloc (ptr, size) + __ptr_t ptr; + __malloc_size_t size; +{ + __ptr_t result; + int type; + __malloc_size_t block, oldsize; + + block = ((char *) ptr - bss_sbrk_heapbase) / BLOCKSIZE + 1; + type = bss_sbrk_heapinfo[block].busy.type; + oldsize = + type == 0 ? bss_sbrk_heapinfo[block].busy.info.size * BLOCKSIZE + : (__malloc_size_t) 1 << type; + result = _malloc_internal_nolock (size); + if (result != NULL) + memcpy (result, ptr, min (oldsize, size)); + return result; +} +#endif + /* Debugging hook for realloc. */ __ptr_t (*__realloc_hook) PP ((__ptr_t __ptr, __malloc_size_t __size)); @@ -1375,6 +1423,12 @@ else if (ptr == NULL) return _malloc_internal_nolock (size); +#ifdef CYGWIN + if (ptr < _heapbase) + /* ptr points into the static heap */ + return special_realloc (ptr, size); +#endif + block = BLOCK (ptr); PROTECT_MALLOC_STATE (0); === modified file 'src/unexcw.c' --- src/unexcw.c 2011-03-17 20:18:59 +0000 +++ src/unexcw.c 2011-08-12 20:10:03 +0000 @@ -33,6 +33,8 @@ extern int bss_sbrk_did_unexec; +extern int __malloc_initialized; + /* emacs symbols that indicate where bss and data end for emacs internals */ extern char my_endbss[]; extern char my_edata[]; @@ -210,9 +212,12 @@ lseek (fd, (long) (exe_header->section_header[i].s_scnptr), SEEK_SET); assert (ret != -1); + /* force the dumped emacs to reinitialize malloc */ + __malloc_initialized = 0; ret = write (fd, (char *) start_address, my_endbss - (char *) start_address); + __malloc_initialized = 1; assert (ret == (my_endbss - (char *) start_address)); if (debug_unexcw) printf (" .bss, mem start 0x%08x mem length %d\n", From debbugs-submit-bounces@debbugs.gnu.org Tue Aug 16 09:32:51 2011 Received: (at 9273-done) by debbugs.gnu.org; 16 Aug 2011 13:32:52 +0000 Received: from localhost ([127.0.0.1] helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QtJlA-0003Pi-VG for submit@debbugs.gnu.org; Tue, 16 Aug 2011 09:32:51 -0400 Received: from granite1.mail.cornell.edu ([128.253.83.141] helo=authusersmtp.mail.cornell.edu) by debbugs.gnu.org with esmtp (Exim 4.69) (envelope-from ) id 1QtJl1-0003PU-Rt for 9273-done@debbugs.gnu.org; Tue, 16 Aug 2011 09:32:44 -0400 Received: from [172.160.101.188] (fl-184-7-236-177.sta.embarqhsd.net [184.7.236.177]) (authenticated bits=0) by authusersmtp.mail.cornell.edu (8.14.4/8.12.10) with ESMTP id p7GDUonc006921 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=NOT); Tue, 16 Aug 2011 09:30:51 -0400 (EDT) Message-ID: <4E4A70F7.6060301@cornell.edu> Date: Tue, 16 Aug 2011 09:30:31 -0400 From: Ken Brown User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:5.0) Gecko/20110624 Thunderbird/5.0 MIME-Version: 1.0 To: Stefan Monnier , Eli Zaretskii Subject: Re: bug#9273: 23.3; malloc initialization should (sometimes) happen at runtime References: <4E41940C.2010605@cornell.edu> <83zkjhxnty.fsf@gnu.org> <4E444D85.4090907@cornell.edu> <838vqzxgq1.fsf@gnu.org> <4E44FC1B.9010207@cornell.edu> <831uwqyidi.fsf@gnu.org> <4E451A1F.4060409@cornell.edu> <4E458BF4.9080108@cornell.edu> <83fwl5wxbz.fsf@gnu.org> <4E4680C4.6040605@cornell.edu> <83d3g9weza.fsf@gnu.org> <4E468FE5.100@cornell.edu> <4E469967.40201@cornell.edu> <4E473D63.7040703@cornell.edu> In-Reply-To: <4E473D63.7040703@cornell.edu> Content-Type: text/plain; charset=ISO-8859-1; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -5.9 (-----) X-Debbugs-Envelope-To: 9273-done Cc: 9273-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.11 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -5.9 (-----) I've committed the changes and am closing the bug. Ken From unknown Thu Sep 11 11:26:37 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Wed, 14 Sep 2011 11:24:03 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator