From debbugs-submit-bounces@debbugs.gnu.org Wed Mar 11 08:38:32 2015 Received: (at submit) by debbugs.gnu.org; 11 Mar 2015 12:38:32 +0000 Received: from localhost ([127.0.0.1]:42356 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVfuB-0005NT-N2 for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:32 -0400 Received: from eggs.gnu.org ([208.118.235.92]:37602) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVfu9-0005N9-1D for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVfu2-0007sx-9o for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:23 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: * X-Spam-Status: No, score=1.1 required=5.0 tests=BAYES_50, FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:56954) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVfu2-0007st-6B for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39319) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVfu0-0006Hc-Tk for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVftz-0007rD-KD for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:20 -0400 Received: from mail-ob0-x22f.google.com ([2607:f8b0:4003:c01::22f]:35398) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVftz-0007qM-AB for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:19 -0400 Received: by obcwo20 with SMTP id wo20so8496517obc.2 for ; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=+hjBN9N/jJuZ9BPtMndgDn02v5in5pXYkqJcFIrxp20=; b=I0iRrsmWqcRd8ibESmwiSRyNRm85xYJ959j3eiNjNL1RpCEnxm/oUVc0kCOmdADlC2 O/iVRuhi4+ZMoR99n6PNbL28Mv6xhhgl6GDoNzr5zDKYd5/ih/z3EG9qBaxiufxAN7BV NzD+kahOkPUoQByf/WpsVhEMui8ADoVrVFz36OMKuauouVmY29r7N1BUqMm07Kwqda9c 6QhIdVg93gi3YOPk4GPs8NYSgl2nB9f4r8cU6gtqr72VwcC0c5tdVXdpNUJo+FAfFCTz tH+gtJ+EXwP90LPbgzvaOknehCw3voPxMCeGUZ0am74zrGr6xjbJMBedjF15y2UAhcr+ xE0w== MIME-Version: 1.0 X-Received: by 10.182.65.71 with SMTP id v7mr29738780obs.42.1426077498152; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) Received: by 10.202.195.9 with HTTP; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) In-Reply-To: References: Date: Wed, 11 Mar 2015 18:08:18 +0530 Message-ID: Subject: Fwd: Memory leak from seek/ftell with files larger than 2GB From: Anand Mohanadoss To: bug-guile@gnu.org Content-Type: multipart/alternative; boundary=089e01537f62b320b20511028a18 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.8 (---) --089e01537f62b320b20511028a18 Content-Type: text/plain; charset=UTF-8 Hi, I had sent the following to the user forum and did not receive any comments. I am reposting it in the bug forum with the hope that one of the experts may be able to comment... Thanks, Anand ---------- Forwarded message ---------- From: Anand Mohanadoss Date: Wed, Feb 25, 2015 at 9:35 PM Subject: Memory leak from seek/ftell with files larger than 2GB To: guile-user@gnu.org Hi, We are seeing an issue with seek and ftell leaking memory with files larger than 2GB. We are using 2.0.11 guile built as a 32-bit application with large file support enabled (guile was built using gcc 4.4.0 for Linux with flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. The memory leaks start only after the offset exceeds maximum positive value for a 32-bit signed integer. ftell and seek do work as expected (given how lseek should work with large file support). Appended is a program that illustrates the problem. The first seek simply skips the part of the file where you won't see a memory leak. If you comment out ftell and the second seek lines and un-comment the lines that follow them, there is no memory leak. Is this a bug in guile or should we be doing things differently? If this is a known issue, is there a recommended work around? Thanks, Anand (define MAX_SIGNED_INT 2147483647) (define BYTES_TO_READ 10) (define file "/tmp/test.pcap") ;sample file greater than 2.5GB (define (traverse file) (let* ((port (open-input-file file #:binary #t)) (file-sz (stat:size (stat port))) (ua (make-bytevector BYTES_TO_READ 0)) (cur-offset 0)) (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) (while (< (ftell port) (- file-sz BYTES_TO_READ)) ;(while (< cur-offset (- file-sz BYTES_TO_READ)) (seek port BYTES_TO_READ SEEK_CUR) ;(get-bytevector-n! port ua 0 BYTES_TO_READ) (set! cur-offset (+ BYTES_TO_READ cur-offset))) (close-port port))) (traverse file) --089e01537f62b320b20511028a18 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

I had sent the following t= o the user forum and did not receive any comments.=C2=A0 I am reposting it = in the bug forum with the hope that one of the experts may be able to comme= nt...

Thanks,
Anand

---------- Forwarded message ----------
From: Anand Mohanadoss <anand108@gmail.com>
Date: W= ed, Feb 25, 2015 at 9:35 PM
Subject: Memory leak from seek/ftell with fi= les larger than 2GB
To: guile-user= @gnu.org


Hi,<= br>
We are seeing an issue with seek and ftell leaking memory with= files larger than 2GB.

We are using 2.0.11 guile built as a 3= 2-bit application with large file support enabled (guile was built using gc= c 4.4.0 for Linux with flags _FILE_OFFSET_BITS=3D64, _LARGEFILE_SOURCE and = _LARGEFILE64_SOURCE).=C2=A0 The issue also appears to happen with guile 2.2= .

The memory leaks start only after the offset exceeds ma= ximum positive value for a 32-bit signed integer. ftell and seek do work as= expected (given how lseek should work with large file support).

Appended is a program that illustrates the problem.=C2=A0 The first seek= simply skips the part of the file where you won't see a memory leak. I= f you comment out ftell and the second seek lines and un-comment the lines = that follow them, there is no memory leak.=C2=A0

Is this a bu= g in guile or should we be doing things differently?=C2=A0 If this is a kno= wn issue, is there a recommended work around?

Thanks,
Anand
=C2=A0
(define MAX_SIGNED_= INT 214= 7483647)
(define BYTES_TO_READ 10)

(define file "/tmp/te= st.pcap")=C2=A0 ;sample file greater than 2.5GB

(define (traver= se file)
=C2=A0(let* ((port (open-input-file file #:binary #t))
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (file-sz (stat:size (stat port)))=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (ua (make-bytevector BYTES_TO_= READ 0))
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (cur-offset 0))
= =C2=A0=C2=A0 (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
=C2=A0=C2=A0= (while (< (ftell port) (- file-sz BYTES_TO_READ))
=C2=A0=C2=A0 ;(whi= le (< cur-offset (- file-sz BYTES_TO_READ))
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 (seek=C2=A0 port BYTES_TO_READ SEEK_CUR)
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (set! cur-offset (+ BYTES_TO_READ cur-offset= )))
=C2=A0=C2=A0 (close-port port)))

(traverse file)

--089e01537f62b320b20511028a18-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 23 08:22:45 2016 Received: (at 20079) by debbugs.gnu.org; 23 Jun 2016 12:22:45 +0000 Received: from localhost ([127.0.0.1]:51897 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG3ee-0006oW-U7 for submit@debbugs.gnu.org; Thu, 23 Jun 2016 08:22:45 -0400 Received: from pb-sasl1.pobox.com ([64.147.108.66]:54641 helo=sasl.smtp.pobox.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG3ed-0006oP-30 for 20079@debbugs.gnu.org; Thu, 23 Jun 2016 08:22:43 -0400 Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id 7B30921A99; Thu, 23 Jun 2016 08:22:41 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=L7Nx0gZQ9xNCcnvQFgBotxJr4NY=; b=nD2Wmd /n15Zs5AUuUNqvuxSEzhotJsuhRY9/bL5RztWHsXfHQjf1XYP30GsoTwpFrvVZW7 6iTKW9phKssIdNRYY/8apyIjSaQUUYEyB1DnppC1Ko7zwRgeB0/WBtWYSRCQKV+4 7j9SXJNDiLYZK72JYbQaTliRmUPuZ5GuQSCn8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=Y6Sp0oGHRZwL9q9Fc7Knb89agJ/UHg6u Lapw7ubG/rPrMisCKFq/CUvXkJtO09wknUyJSV7wPbFZj0OQ2MGQvIM580TNFnmQ jaqOGLt1gSitR/46RQPDA5jB3KHQgTpQ+cEVKNb64PwfKFToQUccK9t+s+vVAm6a BcOQvYLi10Q= Received: from pb-sasl1.nyi.icgroup.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id 7247E21A98; Thu, 23 Jun 2016 08:22:41 -0400 (EDT) Received: from clucks (unknown [88.160.190.192]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by pb-sasl1.pobox.com (Postfix) with ESMTPSA id A70D121A97; Thu, 23 Jun 2016 08:22:40 -0400 (EDT) From: Andy Wingo To: Anand Mohanadoss Subject: Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB References: Date: Thu, 23 Jun 2016 14:22:33 +0200 In-Reply-To: (Anand Mohanadoss's message of "Wed, 11 Mar 2015 18:08:18 +0530") Message-ID: <87y45wnlh2.fsf@pobox.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Pobox-Relay-ID: 2DF36444-393D-11E6-96DB-C1836462E9F6-02397024!pb-sasl1.pobox.com X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: 20079 Cc: 20079@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.4 (-) --=-=-= Content-Type: text/plain Very strange bug! I can reproduce it with this file: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=foo.scm Content-Description: run as "guile foo.scm" (use-modules (rnrs bytevectors) (ice-9 binary-ports)) (define MAX_SIGNED_INT 2147483647) (define BYTES_TO_READ 10) (define (traverse port) (let* ((file-sz (stat:size (stat port))) (ua (make-bytevector BYTES_TO_READ 0)) (cur-offset 0)) (let lp ((cur-offset (seek port (- MAX_SIGNED_INT 1000) SEEK_CUR))) (when (< cur-offset (- file-sz BYTES_TO_READ)) (lp (seek port BYTES_TO_READ SEEK_CUR)))) (close-port port))) (define port (mkstemp! (string-copy "/tmp/big-file-XXXXXX"))) (truncate-file port #e2.5e9) (traverse port) --=-=-= Content-Type: text/plain I wonder what it could be! Andy On Wed 11 Mar 2015 13:38, Anand Mohanadoss writes: > Hi, > > I had sent the following to the user forum and did not receive any > comments. I am reposting it in the bug forum with the hope that one of > the experts may be able to comment... > > Thanks, > Anand > > ---------- Forwarded message ---------- > From: Anand Mohanadoss > Date: Wed, Feb 25, 2015 at 9:35 PM > Subject: Memory leak from seek/ftell with files larger than 2GB > To: guile-user@gnu.org > > Hi, > > We are seeing an issue with seek and ftell leaking memory with files > larger than 2GB. > > We are using 2.0.11 guile built as a 32-bit application with large > file support enabled (guile was built using gcc 4.4.0 for Linux with > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _ > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. > > The memory leaks start only after the offset exceeds maximum positive > value for a 32-bit signed integer. ftell and seek do work as expected > (given how lseek should work with large file support). > > Appended is a program that illustrates the problem. The first seek > simply skips the part of the file where you won't see a memory leak. > If you comment out ftell and the second seek lines and un-comment the > lines that follow them, there is no memory leak. > > Is this a bug in guile or should we be doing things differently? If > this is a known issue, is there a recommended work around? > > Thanks, > Anand > > (define MAX_SIGNED_INT 2147483647) > (define BYTES_TO_READ 10) > > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB > > (define (traverse file) > (let* ((port (open-input-file file #:binary #t)) > (file-sz (stat:size (stat port))) > (ua (make-bytevector BYTES_TO_READ 0)) > (cur-offset 0)) > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) > (while (< (ftell port) (- file-sz BYTES_TO_READ)) > ;(while (< cur-offset (- file-sz BYTES_TO_READ)) > (seek port BYTES_TO_READ SEEK_CUR) > ;(get-bytevector-n! port ua 0 BYTES_TO_READ) > (set! cur-offset (+ BYTES_TO_READ cur-offset))) > (close-port port))) > > (traverse file) --=-=-=-- From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 23 09:02:14 2016 Received: (at 20079-done) by debbugs.gnu.org; 23 Jun 2016 13:02:14 +0000 Received: from localhost ([127.0.0.1]:51915 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG4Go-0000vs-SE for submit@debbugs.gnu.org; Thu, 23 Jun 2016 09:02:14 -0400 Received: from pb-sasl1.pobox.com ([64.147.108.66]:65234 helo=sasl.smtp.pobox.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG4Gk-0000sR-3L for 20079-done@debbugs.gnu.org; Thu, 23 Jun 2016 09:02:09 -0400 Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id AC1F721EF8; Thu, 23 Jun 2016 09:02:05 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=+5dJZCKBitAFSrfAaK+mX2ZfDK8=; b=Q5wt51 l82FsGZYewehD1cynCL1FctkWDZTbh6URnzweLhgqY+gVgfqWDd/pI7E54ZKDpJo iNkVg2LGd3m/JdBSZXdpXWATsfM8UeL41lH79ZqP7N+E9LrnF0ApBrQjFTmKZPlP sbaM5qCEmJgGuwwWEqOqd7BkReCsfftpYEKX8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=kAIuhh29qw9HTyMf8oT8DzAxPrUuuFuG GxedX2XBdUlYp1w2bFQjN6n7FU57fuAF8MEehh502QJoeycPPnFi4Z8IrgWqFIan xhEDT8t8lgk17nXjywxQCh/v2j7KwArPXE453ayOeQcPIcWZpHmxwRXWqDufTYt8 BUlraAci3dY= Received: from pb-sasl1.nyi.icgroup.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id A212A21EF6; Thu, 23 Jun 2016 09:02:05 -0400 (EDT) Received: from clucks (unknown [88.160.190.192]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by pb-sasl1.pobox.com (Postfix) with ESMTPSA id BF45C21EF5; Thu, 23 Jun 2016 09:02:04 -0400 (EDT) From: Andy Wingo To: Anand Mohanadoss Subject: Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB References: Date: Thu, 23 Jun 2016 15:01:57 +0200 In-Reply-To: (Anand Mohanadoss's message of "Wed, 11 Mar 2015 18:08:18 +0530") Message-ID: <87shw4njne.fsf@pobox.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: AF269310-3942-11E6-9342-C1836462E9F6-02397024!pb-sasl1.pobox.com X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: 20079-done Cc: 20079-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.4 (-) Hi, Thank you very much for this one! Turns out we had an incredibly embarrassing bug in which we forgot to attach finalizers for bignums created by scm_from_{uint64,int64} on 32-bit platforms. Fixed in master and stable-2.0. Cheers, Andy On Wed 11 Mar 2015 13:38, Anand Mohanadoss writes: > Hi, > > I had sent the following to the user forum and did not receive any > comments. I am reposting it in the bug forum with the hope that one of > the experts may be able to comment... > > Thanks, > Anand > > ---------- Forwarded message ---------- > From: Anand Mohanadoss > Date: Wed, Feb 25, 2015 at 9:35 PM > Subject: Memory leak from seek/ftell with files larger than 2GB > To: guile-user@gnu.org > > Hi, > > We are seeing an issue with seek and ftell leaking memory with files > larger than 2GB. > > We are using 2.0.11 guile built as a 32-bit application with large > file support enabled (guile was built using gcc 4.4.0 for Linux with > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _ > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. > > The memory leaks start only after the offset exceeds maximum positive > value for a 32-bit signed integer. ftell and seek do work as expected > (given how lseek should work with large file support). > > Appended is a program that illustrates the problem. The first seek > simply skips the part of the file where you won't see a memory leak. > If you comment out ftell and the second seek lines and un-comment the > lines that follow them, there is no memory leak. > > Is this a bug in guile or should we be doing things differently? If > this is a known issue, is there a recommended work around? > > Thanks, > Anand > > (define MAX_SIGNED_INT 2147483647) > (define BYTES_TO_READ 10) > > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB > > (define (traverse file) > (let* ((port (open-input-file file #:binary #t)) > (file-sz (stat:size (stat port))) > (ua (make-bytevector BYTES_TO_READ 0)) > (cur-offset 0)) > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) > (while (< (ftell port) (- file-sz BYTES_TO_READ)) > ;(while (< cur-offset (- file-sz BYTES_TO_READ)) > (seek port BYTES_TO_READ SEEK_CUR) > ;(get-bytevector-n! port ua 0 BYTES_TO_READ) > (set! cur-offset (+ BYTES_TO_READ cur-offset))) > (close-port port))) > > (traverse file) From debbugs-submit-bounces@debbugs.gnu.org Thu Jun 23 10:43:10 2016 Received: (at 20079-done) by debbugs.gnu.org; 23 Jun 2016 14:43:10 +0000 Received: from localhost ([127.0.0.1]:52786 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG5qX-0003kH-P5 for submit@debbugs.gnu.org; Thu, 23 Jun 2016 10:43:10 -0400 Received: from mail-io0-f171.google.com ([209.85.223.171]:36533) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG5qW-0003jz-8n for 20079-done@debbugs.gnu.org; Thu, 23 Jun 2016 10:43:08 -0400 Received: by mail-io0-f171.google.com with SMTP id s63so69479567ioi.3 for <20079-done@debbugs.gnu.org>; Thu, 23 Jun 2016 07:43:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=g51otMSx0L/nsA0Cr3TVG0dwf50IlvpGRlfSgS70IYM=; b=ZrQZBua3Bt3kAmvJec9DRPgoDrk53pp4VtoTQBfAaH8l9IOMSybDma4CDuL90gIjyz Bv8Vj9v2vbjNrchGSnv7tittXZWvLwZuKVU4y40+vRlCsR0E3bFEFeFYIdiWqIk/Ylv4 gHKkIPWTrCjNslSkmx8GwXgc6KTaXX/PdkmnyAbyVzGHgGtZVmGIrOrV9VFskfpp9JHH VTe9CtK/0SJOTZMESEoK3HPE1eMtaq5JVGzaerkv9Ji8WgK6UNb6UE5aItkVWP/WnEbE wbogmPoKkCqKwTk+aSPsLaBxaJ4u5rMhgNyOPkAY8oxcGplR07vR8PaBnU0btH1bt98p n5pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=g51otMSx0L/nsA0Cr3TVG0dwf50IlvpGRlfSgS70IYM=; b=irRPuzDUfNYs4x7bQheaUtt7owzQZl6YxLh1pN001YdCeZwGLvHxz6riGYKVuwhuoG X6+7OVhQw1W7c0YX5sAZIHNuQPvIP3gT6ilDRilAQ5b6y6tMwNB9HIqD5kmh0ZI1EnWv oOCzvzf/yxwwcMtWQHbRyRkoBhqhmLwS6URQ9Zll6yT0i953s+mBHRFpKa0YwtPqaLsN xdTZ5IZn9KB0gQwI6NgMjNvqlp1OQLfPPBaGUe1oOrVCOt7DL2MhSzTG+A3dnnRfAWmp rwBk9Q3QF7bTSmU8OH1n1nOntlKjyimjoyBOV1ogqMSnEiT6t6WiBeYroL/Rsi7bmekj 4Uhw== X-Gm-Message-State: ALyK8tJ7QQA7c3DeWnGNsQQ35rc8mM9OEKWzi6nulPivib+31/M3Qyf2kDTL33G3sdg2JNbhMZnBhUsGh4vK5w== X-Received: by 10.107.170.170 with SMTP id g42mr1343556ioj.44.1466692982300; Thu, 23 Jun 2016 07:43:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.154.68 with HTTP; Thu, 23 Jun 2016 07:43:01 -0700 (PDT) In-Reply-To: <87shw4njne.fsf@pobox.com> References: <87shw4njne.fsf@pobox.com> From: Anand Mohanadoss Date: Thu, 23 Jun 2016 20:13:01 +0530 Message-ID: Subject: Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB To: Andy Wingo Content-Type: multipart/alternative; boundary=001a114260ba346e8c0535f312d0 X-Spam-Score: -0.4 (/) X-Debbugs-Envelope-To: 20079-done Cc: 20079-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.4 (/) --001a114260ba346e8c0535f312d0 Content-Type: text/plain; charset=UTF-8 Hi Andy, Thanks a lot for the fix! Anand On Thu, Jun 23, 2016 at 6:31 PM, Andy Wingo wrote: > Hi, > > Thank you very much for this one! Turns out we had an incredibly > embarrassing bug in which we forgot to attach finalizers for bignums > created by scm_from_{uint64,int64} on 32-bit platforms. Fixed in master > and stable-2.0. > > Cheers, > > Andy > > On Wed 11 Mar 2015 13:38, Anand Mohanadoss writes: > > > Hi, > > > > I had sent the following to the user forum and did not receive any > > comments. I am reposting it in the bug forum with the hope that one of > > the experts may be able to comment... > > > > Thanks, > > Anand > > > > ---------- Forwarded message ---------- > > From: Anand Mohanadoss > > Date: Wed, Feb 25, 2015 at 9:35 PM > > Subject: Memory leak from seek/ftell with files larger than 2GB > > To: guile-user@gnu.org > > > > Hi, > > > > We are seeing an issue with seek and ftell leaking memory with files > > larger than 2GB. > > > > We are using 2.0.11 guile built as a 32-bit application with large > > file support enabled (guile was built using gcc 4.4.0 for Linux with > > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _ > > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. > > > > The memory leaks start only after the offset exceeds maximum positive > > value for a 32-bit signed integer. ftell and seek do work as expected > > (given how lseek should work with large file support). > > > > Appended is a program that illustrates the problem. The first seek > > simply skips the part of the file where you won't see a memory leak. > > If you comment out ftell and the second seek lines and un-comment the > > lines that follow them, there is no memory leak. > > > > Is this a bug in guile or should we be doing things differently? If > > this is a known issue, is there a recommended work around? > > > > Thanks, > > Anand > > > > (define MAX_SIGNED_INT 2147483647) > > (define BYTES_TO_READ 10) > > > > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB > > > > (define (traverse file) > > (let* ((port (open-input-file file #:binary #t)) > > (file-sz (stat:size (stat port))) > > (ua (make-bytevector BYTES_TO_READ 0)) > > (cur-offset 0)) > > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) > > (while (< (ftell port) (- file-sz BYTES_TO_READ)) > > ;(while (< cur-offset (- file-sz BYTES_TO_READ)) > > (seek port BYTES_TO_READ SEEK_CUR) > > ;(get-bytevector-n! port ua 0 BYTES_TO_READ) > > (set! cur-offset (+ BYTES_TO_READ cur-offset))) > > (close-port port))) > > > > (traverse file) > --001a114260ba346e8c0535f312d0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Andy,

Thanks a lot for the fix! =

Anand

On Thu, Jun 23, 2016 at 6:31 PM, Andy Wingo &= lt;wingo@pobox.com= > wrote:
Hi,

Thank you very much for this one!=C2=A0 Turns out we had an incredibly
embarrassing bug in which we forgot to attach finalizers for bignums
created by scm_from_{uint64,int64} on 32-bit platforms.=C2=A0 Fixed in mast= er
and stable-2.0.

Cheers,

Andy

On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108@gmail.com> writes:

> Hi,
>
> I had sent the following to the user forum and did not receive any
> comments. I am reposting it in the bug forum with the hope that one of=
> the experts may be able to comment...
>
> Thanks,
> Anand
>
> ---------- Forwarded message ----------
> From: Anand Mohanadoss <anand= 108@gmail.com>
> Date: Wed, Feb 25, 2015 at 9:35 PM
> Subject: Memory leak from seek/ftell with files larger than 2GB
> To: guile-user@gnu.org
>
> Hi,
>
> We are seeing an issue with seek and ftell leaking memory with files > larger than 2GB.
>
> We are using 2.0.11 guile built as a 32-bit application with large
> file support enabled (guile was built using gcc 4.4.0 for Linux with > flags _FILE_OFFSET_BITS=3D64, _LARGEFILE_SOURCE and _
> LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.<= br> >
> The memory leaks start only after the offset exceeds maximum positive<= br> > value for a 32-bit signed integer. ftell and seek do work as expected<= br> > (given how lseek should work with large file support).
>
> Appended is a program that illustrates the problem. The first seek
> simply skips the part of the file where you won't see a memory lea= k.
> If you comment out ftell and the second seek lines and un-comment the<= br> > lines that follow them, there is no memory leak.
>
> Is this a bug in guile or should we be doing things differently? If > this is a known issue, is there a recommended work around?
>
> Thanks,
> Anand
>
> (define MAX_SIGNED_INT 2147483647)
> (define BYTES_TO_READ 10)
>
> (define file "/tmp/test.pcap") ;sample file greater than 2.5= GB
>
> (define (traverse file)
> (let* ((port (open-input-file file #:binary #t))
> (file-sz (stat:size (stat port)))
> (ua (make-bytevector BYTES_TO_READ 0))
> (cur-offset 0))
> (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> (while (< (ftell port) (- file-sz BYTES_TO_READ))
> ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> (seek port BYTES_TO_READ SEEK_CUR)
> ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> (close-port port)))
>
> (traverse file)

--001a114260ba346e8c0535f312d0-- From unknown Mon Jun 23 02:21:13 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 22 Jul 2016 11:24:05 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator