GNU bug report logs - #20079
Fwd: Memory leak from seek/ftell with files larger than 2GB

Previous Next

To add a comment to this bug, you must first unarchive it, by sending
a message to control AT debbugs.gnu.org, with unarchive 20079 in the body.
You can then email your comments to 20079 AT debbugs.gnu.org in the normal way.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Anand Mohanadoss <anand108 <at> gmail.com>
To: bug-guile <at> gnu.org
Subject: Fwd: Memory leak from seek/ftell with files larger than 2GB
Date: Wed, 11 Mar 2015 18:08:18 +0530

[Message part 1 (text/plain, inline)]

Hi,

I had sent the following to the user forum and did not receive any
comments.  I am reposting it in the bug forum with the hope that one of the
experts may be able to comment...

Thanks,
Anand

---------- Forwarded message ----------
From: Anand Mohanadoss <anand108 <at> gmail.com>
Date: Wed, Feb 25, 2015 at 9:35 PM
Subject: Memory leak from seek/ftell with files larger than 2GB
To: guile-user <at> gnu.org


Hi,

We are seeing an issue with seek and ftell leaking memory with files larger
than 2GB.

We are using 2.0.11 guile built as a 32-bit application with large file
support enabled (guile was built using gcc 4.4.0 for Linux with flags
_FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE).  The
issue also appears to happen with guile 2.2.

The memory leaks start only after the offset exceeds maximum positive value
for a 32-bit signed integer. ftell and seek do work as expected (given how
lseek should work with large file support).

Appended is a program that illustrates the problem.  The first seek simply
skips the part of the file where you won't see a memory leak. If you
comment out ftell and the second seek lines and un-comment the lines that
follow them, there is no memory leak.

Is this a bug in guile or should we be doing things differently?  If this
is a known issue, is there a recommended work around?

Thanks,
Anand

(define MAX_SIGNED_INT 2147483647)
(define BYTES_TO_READ 10)

(define file "/tmp/test.pcap")  ;sample file greater than 2.5GB

(define (traverse file)
 (let* ((port (open-input-file file #:binary #t))
        (file-sz (stat:size (stat port)))
        (ua (make-bytevector BYTES_TO_READ 0))
        (cur-offset 0))
   (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
   (while (< (ftell port) (- file-sz BYTES_TO_READ))
   ;(while (< cur-offset (- file-sz BYTES_TO_READ))
       (seek  port BYTES_TO_READ SEEK_CUR)
       ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
       (set! cur-offset (+ BYTES_TO_READ cur-offset)))
   (close-port port)))

(traverse file)

[Message part 2 (text/html, inline)]

Message #8 received at 20079 <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: Anand Mohanadoss <anand108 <at> gmail.com>
Cc: 20079 <at> debbugs.gnu.org
Subject: Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger
 than 2GB
Date: Thu, 23 Jun 2016 14:22:33 +0200

[Message part 1 (text/plain, inline)]

Very strange bug!  I can reproduce it with this file:

[foo.scm (text/plain, inline)]

(use-modules (rnrs bytevectors)
             (ice-9 binary-ports))

(define MAX_SIGNED_INT 2147483647)
(define BYTES_TO_READ 10)

(define (traverse port)
  (let* ((file-sz (stat:size (stat port)))
         (ua (make-bytevector BYTES_TO_READ 0))
         (cur-offset 0))
    (let lp ((cur-offset (seek port (- MAX_SIGNED_INT 1000) SEEK_CUR)))
      (when (< cur-offset (- file-sz BYTES_TO_READ))
        (lp (seek port BYTES_TO_READ SEEK_CUR))))
    (close-port port))) 

(define port (mkstemp! (string-copy "/tmp/big-file-XXXXXX")))
(truncate-file port #e2.5e9)
(traverse port)

[Message part 3 (text/plain, inline)]

I wonder what it could be!

Andy

On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108 <at> gmail.com> writes:

> Hi,
>
> I had sent the following to the user forum and did not receive any
> comments. I am reposting it in the bug forum with the hope that one of
> the experts may be able to comment...
>
> Thanks,
> Anand
>
> ---------- Forwarded message ----------
> From: Anand Mohanadoss <anand108 <at> gmail.com>
> Date: Wed, Feb 25, 2015 at 9:35 PM
> Subject: Memory leak from seek/ftell with files larger than 2GB
> To: guile-user <at> gnu.org
>
> Hi,
>
> We are seeing an issue with seek and ftell leaking memory with files
> larger than 2GB.
>
> We are using 2.0.11 guile built as a 32-bit application with large
> file support enabled (guile was built using gcc 4.4.0 for Linux with
> flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _
> LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.
>
> The memory leaks start only after the offset exceeds maximum positive
> value for a 32-bit signed integer. ftell and seek do work as expected
> (given how lseek should work with large file support).
>
> Appended is a program that illustrates the problem. The first seek
> simply skips the part of the file where you won't see a memory leak.
> If you comment out ftell and the second seek lines and un-comment the
> lines that follow them, there is no memory leak. 
>
> Is this a bug in guile or should we be doing things differently? If
> this is a known issue, is there a recommended work around?
>
> Thanks,
> Anand
>
> (define MAX_SIGNED_INT 2147483647)
> (define BYTES_TO_READ 10)
>
> (define file "/tmp/test.pcap") ;sample file greater than 2.5GB
>
> (define (traverse file)
> (let* ((port (open-input-file file #:binary #t))
> (file-sz (stat:size (stat port)))
> (ua (make-bytevector BYTES_TO_READ 0))
> (cur-offset 0))
> (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> (while (< (ftell port) (- file-sz BYTES_TO_READ))
> ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> (seek port BYTES_TO_READ SEEK_CUR)
> ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> (close-port port))) 
>
> (traverse file)

Message #13 received at 20079-done <at> debbugs.gnu.org (full text, mbox):

From: Andy Wingo <wingo <at> pobox.com>
To: Anand Mohanadoss <anand108 <at> gmail.com>
Cc: 20079-done <at> debbugs.gnu.org
Subject: Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger
 than 2GB
Date: Thu, 23 Jun 2016 15:01:57 +0200

Hi,

Thank you very much for this one!  Turns out we had an incredibly
embarrassing bug in which we forgot to attach finalizers for bignums
created by scm_from_{uint64,int64} on 32-bit platforms.  Fixed in master
and stable-2.0.

Cheers,

Andy

On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108 <at> gmail.com> writes:

> Hi,
>
> I had sent the following to the user forum and did not receive any
> comments. I am reposting it in the bug forum with the hope that one of
> the experts may be able to comment...
>
> Thanks,
> Anand
>
> ---------- Forwarded message ----------
> From: Anand Mohanadoss <anand108 <at> gmail.com>
> Date: Wed, Feb 25, 2015 at 9:35 PM
> Subject: Memory leak from seek/ftell with files larger than 2GB
> To: guile-user <at> gnu.org
>
> Hi,
>
> We are seeing an issue with seek and ftell leaking memory with files
> larger than 2GB.
>
> We are using 2.0.11 guile built as a 32-bit application with large
> file support enabled (guile was built using gcc 4.4.0 for Linux with
> flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _
> LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.
>
> The memory leaks start only after the offset exceeds maximum positive
> value for a 32-bit signed integer. ftell and seek do work as expected
> (given how lseek should work with large file support).
>
> Appended is a program that illustrates the problem. The first seek
> simply skips the part of the file where you won't see a memory leak.
> If you comment out ftell and the second seek lines and un-comment the
> lines that follow them, there is no memory leak. 
>
> Is this a bug in guile or should we be doing things differently? If
> this is a known issue, is there a recommended work around?
>
> Thanks,
> Anand
>
> (define MAX_SIGNED_INT 2147483647)
> (define BYTES_TO_READ 10)
>
> (define file "/tmp/test.pcap") ;sample file greater than 2.5GB
>
> (define (traverse file)
> (let* ((port (open-input-file file #:binary #t))
> (file-sz (stat:size (stat port)))
> (ua (make-bytevector BYTES_TO_READ 0))
> (cur-offset 0))
> (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> (while (< (ftell port) (- file-sz BYTES_TO_READ))
> ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> (seek port BYTES_TO_READ SEEK_CUR)
> ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> (close-port port))) 
>
> (traverse file)

Message #16 received at 20079-done <at> debbugs.gnu.org (full text, mbox):

From: Anand Mohanadoss <anand108 <at> gmail.com>
To: Andy Wingo <wingo <at> pobox.com>
Cc: 20079-done <at> debbugs.gnu.org
Subject: Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger
 than 2GB
Date: Thu, 23 Jun 2016 20:13:01 +0530

[Message part 1 (text/plain, inline)]

Hi Andy,

Thanks a lot for the fix!

Anand

On Thu, Jun 23, 2016 at 6:31 PM, Andy Wingo <wingo <at> pobox.com> wrote:

> Hi,
>
> Thank you very much for this one!  Turns out we had an incredibly
> embarrassing bug in which we forgot to attach finalizers for bignums
> created by scm_from_{uint64,int64} on 32-bit platforms.  Fixed in master
> and stable-2.0.
>
> Cheers,
>
> Andy
>
> On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108 <at> gmail.com> writes:
>
> > Hi,
> >
> > I had sent the following to the user forum and did not receive any
> > comments. I am reposting it in the bug forum with the hope that one of
> > the experts may be able to comment...
> >
> > Thanks,
> > Anand
> >
> > ---------- Forwarded message ----------
> > From: Anand Mohanadoss <anand108 <at> gmail.com>
> > Date: Wed, Feb 25, 2015 at 9:35 PM
> > Subject: Memory leak from seek/ftell with files larger than 2GB
> > To: guile-user <at> gnu.org
> >
> > Hi,
> >
> > We are seeing an issue with seek and ftell leaking memory with files
> > larger than 2GB.
> >
> > We are using 2.0.11 guile built as a 32-bit application with large
> > file support enabled (guile was built using gcc 4.4.0 for Linux with
> > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _
> > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.
> >
> > The memory leaks start only after the offset exceeds maximum positive
> > value for a 32-bit signed integer. ftell and seek do work as expected
> > (given how lseek should work with large file support).
> >
> > Appended is a program that illustrates the problem. The first seek
> > simply skips the part of the file where you won't see a memory leak.
> > If you comment out ftell and the second seek lines and un-comment the
> > lines that follow them, there is no memory leak.
> >
> > Is this a bug in guile or should we be doing things differently? If
> > this is a known issue, is there a recommended work around?
> >
> > Thanks,
> > Anand
> >
> > (define MAX_SIGNED_INT 2147483647)
> > (define BYTES_TO_READ 10)
> >
> > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB
> >
> > (define (traverse file)
> > (let* ((port (open-input-file file #:binary #t))
> > (file-sz (stat:size (stat port)))
> > (ua (make-bytevector BYTES_TO_READ 0))
> > (cur-offset 0))
> > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> > (while (< (ftell port) (- file-sz BYTES_TO_READ))
> > ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> > (seek port BYTES_TO_READ SEEK_CUR)
> > ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> > (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> > (close-port port)))
> >
> > (traverse file)
>

[Message part 2 (text/html, inline)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.