GNU bug report logs - #12095
Protecting pointer on bytevector with guardian does not protect memory

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 12095 in the body.
You can then email your comments to 12095 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Patrick Bernaud <patrickb <at> chez.com>
To: bug-guile <at> gnu.org
Subject: Protecting pointer on bytevector with guardian does not protect memory
Date: Mon, 30 Jul 2012 18:01:09 +0200

[Message part 1 (text/plain, inline)]

The memory from a bytevector of which a pointer is taken (with
'bytevector->pointer') can be overwritten even if that pointer has
been put inside a guardian. 'make-c-struct' from (system foreign) is
using 'bytevector->pointer'.

With the test script attached:

$ guile -v | head -1
guile (GNU Guile) 2.0.6.8-cc26b9-dirty
$ guile --no-auto-compile -s test.scm
#<pointer 0x974648c>
#vu8(1 1 1 1 1 1 1 1 1 1)
#vu8(1 1 1 1 1 1 1 1 1 1)
#vu8(110 103 45 108 101 110 103 116 104 0)  <<<< memory overwrite with "ng-length\0" from module
#<pointer 0x974648c>
$

With auto compilation turned on, it looks like the problem can not be
reproduced.

-- 
Patrick Bernaud

[test.scm (application/octet-stream, attachment)]

Message #8 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: Ian Price <ianprice90 <at> googlemail.com>
To: Patrick Bernaud <patrickb <at> chez.com>
Cc: 12095 <at> debbugs.gnu.org
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Mon, 06 Aug 2012 17:37:17 +0100

Patrick Bernaud <patrickb <at> chez.com> writes:

> With auto compilation turned on, it looks like the problem can not be
> reproduced.

I cannot reproduce this on 32 bit fedora 16 with guile (GNU Guile)
2.0.6-dirty (commit 1321a36ed61deb9431b41768dc92cb7230c9afa1). However,
there was one caveat, as I didn't have html prag, I substituted for
various other libraries (ice-9 regex)/(ice-9 threads)/(sxml simple)/ and
my own (pfds queues).

Is this bug somehow particular to htmlprag, or can you confirm it with others?

-- 
Ian Price

"Programming is like pinball. The reward for doing it well is
the opportunity to do it again" - from "The Wizardy Compiled"

Message #11 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Hartwig <mandyke <at> gmail.com>
To: Ian Price <ianprice90 <at> googlemail.com>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095 <at> debbugs.gnu.org
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Tue, 7 Aug 2012 11:32:10 +0800

[Message part 1 (text/plain, inline)]

On 7 August 2012 00:37, Ian Price <ianprice90 <at> googlemail.com> wrote:
> Patrick Bernaud <patrickb <at> chez.com> writes:
>
>> With auto compilation turned on, it looks like the problem can not be
>> reproduced.

That is to say, when using --fresh-auto-compile.

>
> I cannot reproduce this on 32 bit fedora 16 with guile (GNU Guile)
> 2.0.6-dirty (commit 1321a36ed61deb9431b41768dc92cb7230c9afa1). However,
> there was one caveat, as I didn't have html prag, I substituted for
> various other libraries (ice-9 regex)/(ice-9 threads)/(sxml simple)/ and
> my own (pfds queues).
>
> Is this bug somehow particular to htmlprag, or can you confirm it with others?

On x86 Debian sid I reproduced the bug by loading (web server) with
the original test.scm from the mailing list.

The bug report copy differs from the original by replacing
parse-c-struct with pointer->bytevector.

I attach test1.scm, with a loop to load many modules one at a time.
Curiously, when using pointer->bytevector the contents change (for me)
only when freshly compiling:

$ guile --fresh-auto-compile test1.scm
;;; note: auto-compilation is enabled, set GUILE_AUTO_COMPILE=0
;;;       or pass the --no-auto-compile argument to disable.
;;; compiling /home/daniel/Downloads/test1.scm
;;; compiled /home/daniel/.cache/guile/ccache/2.0-LE-4-2.0/home/daniel/Downloads/test1.scm.go
#<pointer 0xa1cdb7c>
#vu8(1 1 1 1 1 1 1 1 1 1)
Contents differ after (web http)
#vu8(110 116 104 101 115 105 122 101 0 1)
#<pointer 0xa1cdb7c>
$ guile test1.scm
#<pointer 0x82c754c>
#vu8(1 1 1 1 1 1 1 1 1 1)
#<pointer 0x82c754c>

With the original (using parse-c-struct) it was the other way around.

[test1.scm (application/octet-stream, attachment)]

Message #14 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Daniel Hartwig <mandyke <at> gmail.com>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095 <at> debbugs.gnu.org,
	Ian Price <ianprice90 <at> googlemail.com>
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Sat, 06 Oct 2012 23:41:03 +0200

Hi,

Daniel Hartwig <mandyke <at> gmail.com> skribis:

> (define x (bytevector->pointer (make-bytevector len 1)))
> (define a (pointer-address x))
> (display x)(newline)
> (my-guardian x)
> ;(my-guardian (pointer->bytevector x len))
> (set! x #f)
>
> (define (dump-struct)
>   (write (pointer->bytevector (make-pointer a) len))(newline))

This is expected to fail: ‘bytevector->pointer’ creates a weak reference
from the returned pointer object to the given bytevector.  So when the
pointer object is reclaimed, the bytevector can be reclaimed too, hence
the problem you’re observing.  (And no, guardians don’t protect objects
from garbage collection.)

To put it differently, memory management is left to the user.  The weak
reference I mention is a convenience for simple cases, but for “real
world” situations, one has to take all steps necessary to ensure that
the lifetime of C objects and that of their Scheme counterparts is in sync.

Hope this helps,
Ludo’.

Message #17 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Hartwig <mandyke <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095 <at> debbugs.gnu.org,
	Ian Price <ianprice90 <at> googlemail.com>
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Sun, 7 Oct 2012 10:56:55 +0800

On 7 October 2012 05:41, Ludovic Courtès <ludo <at> gnu.org> wrote:
> Hi,
>
> Daniel Hartwig <mandyke <at> gmail.com> skribis:
>
>> (define x (bytevector->pointer (make-bytevector len 1)))
>> (define a (pointer-address x))
>> (display x)(newline)
>> (my-guardian x)
>> ;(my-guardian (pointer->bytevector x len))
>> (set! x #f)
>>
>> (define (dump-struct)
>>   (write (pointer->bytevector (make-pointer a) len))(newline))
>
> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
> from the returned pointer object to the given bytevector.  So when the
> pointer object is reclaimed, the bytevector can be reclaimed too, hence
> the problem you’re observing.  (And no, guardians don’t protect objects
> from garbage collection.)

If I understand correctly, there is never any non-weak reference to
the bv above and so it can be collected at any time.

It's a bit of an “of course!” moment to realise that the pointer is
only a weak reference.

Thanks

Message #20 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Daniel Hartwig <mandyke <at> gmail.com>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095 <at> debbugs.gnu.org,
	Ian Price <ianprice90 <at> googlemail.com>
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Sun, 07 Oct 2012 22:38:33 +0200

Hi,

Daniel Hartwig <mandyke <at> gmail.com> skribis:

> On 7 October 2012 05:41, Ludovic Courtès <ludo <at> gnu.org> wrote:
>> Hi,
>>
>> Daniel Hartwig <mandyke <at> gmail.com> skribis:
>>
>>> (define x (bytevector->pointer (make-bytevector len 1)))
>>> (define a (pointer-address x))
>>> (display x)(newline)
>>> (my-guardian x)
>>> ;(my-guardian (pointer->bytevector x len))
>>> (set! x #f)
>>>
>>> (define (dump-struct)
>>>   (write (pointer->bytevector (make-pointer a) len))(newline))
>>
>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>> from the returned pointer object to the given bytevector.  So when the
>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>> the problem you’re observing.  (And no, guardians don’t protect objects
>> from garbage collection.)
>
> If I understand correctly, there is never any non-weak reference to
> the bv above and so it can be collected at any time.

There’s a weak reference from the pointer object to the bytevector.

Once that pointer object has been collected (as in the example above),
the bytevector can be collected anytime.

Thanks,
Ludo’.

Message #23 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Hartwig <mandyke <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095 <at> debbugs.gnu.org,
	Ian Price <ianprice90 <at> googlemail.com>
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Mon, 8 Oct 2012 07:49:18 +0800

On 8 October 2012 04:38, Ludovic Courtès <ludo <at> gnu.org> wrote:
>>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>>> from the returned pointer object to the given bytevector.  So when the
>>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>>> the problem you’re observing.  (And no, guardians don’t protect objects
>>> from garbage collection.)
>>
>> If I understand correctly, there is never any non-weak reference to
>> the bv above and so it can be collected at any time.
>
> There’s a weak reference from the pointer object to the bytevector.
>
> Once that pointer object has been collected (as in the example above),
> the bytevector can be collected anytime.

Right.  But then the pointer is being collected even though it remains
inside the guardian, in the example it is never extracted from there.

I'm not sure I follow. :-/

Message #26 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Daniel Hartwig <mandyke <at> gmail.com>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095 <at> debbugs.gnu.org,
	Ian Price <ianprice90 <at> googlemail.com>
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Mon, 08 Oct 2012 15:44:49 +0200

Hi,

Daniel Hartwig <mandyke <at> gmail.com> skribis:

> On 8 October 2012 04:38, Ludovic Courtès <ludo <at> gnu.org> wrote:
>>>> This is expected to fail: ‘bytevector->pointer’ creates a weak reference
>>>> from the returned pointer object to the given bytevector.  So when the
>>>> pointer object is reclaimed, the bytevector can be reclaimed too, hence
>>>> the problem you’re observing.  (And no, guardians don’t protect objects
>>>> from garbage collection.)
>>>
>>> If I understand correctly, there is never any non-weak reference to
>>> the bv above and so it can be collected at any time.
>>
>> There’s a weak reference from the pointer object to the bytevector.
>>
>> Once that pointer object has been collected (as in the example above),
>> the bytevector can be collected anytime.
>
> Right.  But then the pointer is being collected even though it remains
> inside the guardian, in the example it is never extracted from there.

Well, when the object reaches the guardian’s zombie list, that’s because
it’s been finalized, so any weak references from that object can also be
nullified.

Anyway, guardians are not a mechanism to protect objects from being
GC’d.  To prevent the bytevector from being GC’d, you should either keep
the pointer object or the bytevector itself in non-GC’d memory, such as
a global variable or hash table.

How does it help?  Should we close the bug?  :-)

Thanks,
Ludo’.

Message #29 received at 12095 <at> debbugs.gnu.org (full text, mbox):

From: Daniel Hartwig <mandyke <at> gmail.com>
To: Ludovic Courtès <ludo <at> gnu.org>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095 <at> debbugs.gnu.org,
	Ian Price <ianprice90 <at> googlemail.com>
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Mon, 8 Oct 2012 22:10:54 +0800

On 8 October 2012 21:44, Ludovic Courtès <ludo <at> gnu.org> wrote:
>> On 8 October 2012 04:38, Ludovic Courtès <ludo <at> gnu.org> wrote:
>> Right.  But then the pointer is being collected even though it remains
>> inside the guardian, in the example it is never extracted from there.
>
> Well, when the object reaches the guardian’s zombie list, that’s because
> it’s been finalized, so any weak references from that object can also be
> nullified.

Ah.  So I thought that being in the zombie list prevented any
finalization, thus when the guardian returns an object it is still
fully functional and only after the reference is lost again does it
really get finalized.

I will have to reread the doc for make-guardian where it talks about
the weak links, perhaps with a healthy amount of source-code
inspection ;-)

Admittedly the test case here is quite contrived, and I agreed with
your next remark in the original (pre-bug-report) thread.

> Anyway, guardians are not a mechanism to protect objects from being
> GC’d.  To prevent the bytevector from being GC’d, you should either keep
> the pointer object or the bytevector itself in non-GC’d memory, such as
> a global variable or hash table.

Yes.  The intended use was obviously troubled, but it still seemed odd
about the guardian's lack of protection.

> How does it help?  Should we close the bug?  :-)

Sure.  At least you seem convinced and you has actually hacked on it :-)

Regards

Message #34 received at 12095-done <at> debbugs.gnu.org (full text, mbox):

From: ludo <at> gnu.org (Ludovic Courtès)
To: Daniel Hartwig <mandyke <at> gmail.com>
Cc: Patrick Bernaud <patrickb <at> chez.com>, 12095-done <at> debbugs.gnu.org,
	Ian Price <ianprice90 <at> googlemail.com>
Subject: Re: bug#12095: Protecting pointer on bytevector with guardian does
	not protect memory
Date: Mon, 08 Oct 2012 17:42:49 +0200

Daniel Hartwig <mandyke <at> gmail.com> skribis:

> On 8 October 2012 21:44, Ludovic Courtès <ludo <at> gnu.org> wrote:
>>> On 8 October 2012 04:38, Ludovic Courtès <ludo <at> gnu.org> wrote:
>>> Right.  But then the pointer is being collected even though it remains
>>> inside the guardian, in the example it is never extracted from there.
>>
>> Well, when the object reaches the guardian’s zombie list, that’s because
>> it’s been finalized, so any weak references from that object can also be
>> nullified.
>
> Ah.  So I thought that being in the zombie list prevented any
> finalization, thus when the guardian returns an object it is still
> fully functional and only after the reference is lost again does it
> really get finalized.

Well, the object is still usable when the guardian returns it, because
it’s been kept alive by the finalizer (‘finalize_guarded’ in
guardians.c).

Now, whether weak references from the object are subject to
“nullification” by the GC is clearly a gray area, but I’m not surprised
that it is.

>> How does it help?  Should we close the bug?  :-)
>
> Sure.  At least you seem convinced and you has actually hacked on it :-)

Good.  :-)

So closing it now, but feel free to reopen if you think something’s wrong.

Thanks,
Ludo’.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.