From unknown Mon Jun 23 02:23:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB Resent-From: Anand Mohanadoss Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Wed, 11 Mar 2015 12:39:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: report 20079 X-GNU-PR-Package: guile X-GNU-PR-Keywords: To: 20079@debbugs.gnu.org X-Debbugs-Original-To: bug-guile@gnu.org Received: via spool by submit@debbugs.gnu.org id=B.142607751220680 (code B ref -1); Wed, 11 Mar 2015 12:39:02 +0000 Received: (at submit) by debbugs.gnu.org; 11 Mar 2015 12:38:32 +0000 Received: from localhost ([127.0.0.1]:42356 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVfuB-0005NT-N2 for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:32 -0400 Received: from eggs.gnu.org ([208.118.235.92]:37602) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVfu9-0005N9-1D for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVfu2-0007sx-9o for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:23 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: * X-Spam-Status: No, score=1.1 required=5.0 tests=BAYES_50, FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:56954) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVfu2-0007st-6B for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39319) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVfu0-0006Hc-Tk for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVftz-0007rD-KD for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:20 -0400 Received: from mail-ob0-x22f.google.com ([2607:f8b0:4003:c01::22f]:35398) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVftz-0007qM-AB for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:19 -0400 Received: by obcwo20 with SMTP id wo20so8496517obc.2 for ; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=+hjBN9N/jJuZ9BPtMndgDn02v5in5pXYkqJcFIrxp20=; b=I0iRrsmWqcRd8ibESmwiSRyNRm85xYJ959j3eiNjNL1RpCEnxm/oUVc0kCOmdADlC2 O/iVRuhi4+ZMoR99n6PNbL28Mv6xhhgl6GDoNzr5zDKYd5/ih/z3EG9qBaxiufxAN7BV NzD+kahOkPUoQByf/WpsVhEMui8ADoVrVFz36OMKuauouVmY29r7N1BUqMm07Kwqda9c 6QhIdVg93gi3YOPk4GPs8NYSgl2nB9f4r8cU6gtqr72VwcC0c5tdVXdpNUJo+FAfFCTz tH+gtJ+EXwP90LPbgzvaOknehCw3voPxMCeGUZ0am74zrGr6xjbJMBedjF15y2UAhcr+ xE0w== MIME-Version: 1.0 X-Received: by 10.182.65.71 with SMTP id v7mr29738780obs.42.1426077498152; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) Received: by 10.202.195.9 with HTTP; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) In-Reply-To: References: Date: Wed, 11 Mar 2015 18:08:18 +0530 Message-ID: From: Anand Mohanadoss Content-Type: multipart/alternative; boundary=089e01537f62b320b20511028a18 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -3.8 (---) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.8 (---) --089e01537f62b320b20511028a18 Content-Type: text/plain; charset=UTF-8 Hi, I had sent the following to the user forum and did not receive any comments. I am reposting it in the bug forum with the hope that one of the experts may be able to comment... Thanks, Anand ---------- Forwarded message ---------- From: Anand Mohanadoss Date: Wed, Feb 25, 2015 at 9:35 PM Subject: Memory leak from seek/ftell with files larger than 2GB To: guile-user@gnu.org Hi, We are seeing an issue with seek and ftell leaking memory with files larger than 2GB. We are using 2.0.11 guile built as a 32-bit application with large file support enabled (guile was built using gcc 4.4.0 for Linux with flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. The memory leaks start only after the offset exceeds maximum positive value for a 32-bit signed integer. ftell and seek do work as expected (given how lseek should work with large file support). Appended is a program that illustrates the problem. The first seek simply skips the part of the file where you won't see a memory leak. If you comment out ftell and the second seek lines and un-comment the lines that follow them, there is no memory leak. Is this a bug in guile or should we be doing things differently? If this is a known issue, is there a recommended work around? Thanks, Anand (define MAX_SIGNED_INT 2147483647) (define BYTES_TO_READ 10) (define file "/tmp/test.pcap") ;sample file greater than 2.5GB (define (traverse file) (let* ((port (open-input-file file #:binary #t)) (file-sz (stat:size (stat port))) (ua (make-bytevector BYTES_TO_READ 0)) (cur-offset 0)) (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) (while (< (ftell port) (- file-sz BYTES_TO_READ)) ;(while (< cur-offset (- file-sz BYTES_TO_READ)) (seek port BYTES_TO_READ SEEK_CUR) ;(get-bytevector-n! port ua 0 BYTES_TO_READ) (set! cur-offset (+ BYTES_TO_READ cur-offset))) (close-port port))) (traverse file) --089e01537f62b320b20511028a18 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

I had sent the following t= o the user forum and did not receive any comments.=C2=A0 I am reposting it = in the bug forum with the hope that one of the experts may be able to comme= nt...

Thanks,
Anand

---------- Forwarded message ----------
From: Anand Mohanadoss <anand108@gmail.com>
Date: W= ed, Feb 25, 2015 at 9:35 PM
Subject: Memory leak from seek/ftell with fi= les larger than 2GB
To: guile-user= @gnu.org


Hi,<= br>
We are seeing an issue with seek and ftell leaking memory with= files larger than 2GB.

We are using 2.0.11 guile built as a 3= 2-bit application with large file support enabled (guile was built using gc= c 4.4.0 for Linux with flags _FILE_OFFSET_BITS=3D64, _LARGEFILE_SOURCE and = _LARGEFILE64_SOURCE).=C2=A0 The issue also appears to happen with guile 2.2= .

The memory leaks start only after the offset exceeds ma= ximum positive value for a 32-bit signed integer. ftell and seek do work as= expected (given how lseek should work with large file support).

Appended is a program that illustrates the problem.=C2=A0 The first seek= simply skips the part of the file where you won't see a memory leak. I= f you comment out ftell and the second seek lines and un-comment the lines = that follow them, there is no memory leak.=C2=A0

Is this a bu= g in guile or should we be doing things differently?=C2=A0 If this is a kno= wn issue, is there a recommended work around?

Thanks,
Anand
=C2=A0
(define MAX_SIGNED_= INT 214= 7483647)
(define BYTES_TO_READ 10)

(define file "/tmp/te= st.pcap")=C2=A0 ;sample file greater than 2.5GB

(define (traver= se file)
=C2=A0(let* ((port (open-input-file file #:binary #t))
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (file-sz (stat:size (stat port)))=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (ua (make-bytevector BYTES_TO_= READ 0))
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (cur-offset 0))
= =C2=A0=C2=A0 (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
=C2=A0=C2=A0= (while (< (ftell port) (- file-sz BYTES_TO_READ))
=C2=A0=C2=A0 ;(whi= le (< cur-offset (- file-sz BYTES_TO_READ))
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 (seek=C2=A0 port BYTES_TO_READ SEEK_CUR)
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (set! cur-offset (+ BYTES_TO_READ cur-offset= )))
=C2=A0=C2=A0 (close-port port)))

(traverse file)

--089e01537f62b320b20511028a18-- From unknown Mon Jun 23 02:23:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB Resent-From: Andy Wingo Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Thu, 23 Jun 2016 12:23:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 20079 X-GNU-PR-Package: guile X-GNU-PR-Keywords: To: Anand Mohanadoss Cc: 20079@debbugs.gnu.org Received: via spool by 20079-submit@debbugs.gnu.org id=B20079.146668456526201 (code B ref 20079); Thu, 23 Jun 2016 12:23:02 +0000 Received: (at 20079) by debbugs.gnu.org; 23 Jun 2016 12:22:45 +0000 Received: from localhost ([127.0.0.1]:51897 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG3ee-0006oW-U7 for submit@debbugs.gnu.org; Thu, 23 Jun 2016 08:22:45 -0400 Received: from pb-sasl1.pobox.com ([64.147.108.66]:54641 helo=sasl.smtp.pobox.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG3ed-0006oP-30 for 20079@debbugs.gnu.org; Thu, 23 Jun 2016 08:22:43 -0400 Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id 7B30921A99; Thu, 23 Jun 2016 08:22:41 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=L7Nx0gZQ9xNCcnvQFgBotxJr4NY=; b=nD2Wmd /n15Zs5AUuUNqvuxSEzhotJsuhRY9/bL5RztWHsXfHQjf1XYP30GsoTwpFrvVZW7 6iTKW9phKssIdNRYY/8apyIjSaQUUYEyB1DnppC1Ko7zwRgeB0/WBtWYSRCQKV+4 7j9SXJNDiLYZK72JYbQaTliRmUPuZ5GuQSCn8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=Y6Sp0oGHRZwL9q9Fc7Knb89agJ/UHg6u Lapw7ubG/rPrMisCKFq/CUvXkJtO09wknUyJSV7wPbFZj0OQ2MGQvIM580TNFnmQ jaqOGLt1gSitR/46RQPDA5jB3KHQgTpQ+cEVKNb64PwfKFToQUccK9t+s+vVAm6a BcOQvYLi10Q= Received: from pb-sasl1.nyi.icgroup.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id 7247E21A98; Thu, 23 Jun 2016 08:22:41 -0400 (EDT) Received: from clucks (unknown [88.160.190.192]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by pb-sasl1.pobox.com (Postfix) with ESMTPSA id A70D121A97; Thu, 23 Jun 2016 08:22:40 -0400 (EDT) From: Andy Wingo References: Date: Thu, 23 Jun 2016 14:22:33 +0200 In-Reply-To: (Anand Mohanadoss's message of "Wed, 11 Mar 2015 18:08:18 +0530") Message-ID: <87y45wnlh2.fsf@pobox.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: multipart/mixed; boundary="=-=-=" X-Pobox-Relay-ID: 2DF36444-393D-11E6-96DB-C1836462E9F6-02397024!pb-sasl1.pobox.com X-Spam-Score: -1.4 (-) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.4 (-) --=-=-= Content-Type: text/plain Very strange bug! I can reproduce it with this file: --=-=-= Content-Type: text/plain Content-Disposition: inline; filename=foo.scm Content-Description: run as "guile foo.scm" (use-modules (rnrs bytevectors) (ice-9 binary-ports)) (define MAX_SIGNED_INT 2147483647) (define BYTES_TO_READ 10) (define (traverse port) (let* ((file-sz (stat:size (stat port))) (ua (make-bytevector BYTES_TO_READ 0)) (cur-offset 0)) (let lp ((cur-offset (seek port (- MAX_SIGNED_INT 1000) SEEK_CUR))) (when (< cur-offset (- file-sz BYTES_TO_READ)) (lp (seek port BYTES_TO_READ SEEK_CUR)))) (close-port port))) (define port (mkstemp! (string-copy "/tmp/big-file-XXXXXX"))) (truncate-file port #e2.5e9) (traverse port) --=-=-= Content-Type: text/plain I wonder what it could be! Andy On Wed 11 Mar 2015 13:38, Anand Mohanadoss writes: > Hi, > > I had sent the following to the user forum and did not receive any > comments. I am reposting it in the bug forum with the hope that one of > the experts may be able to comment... > > Thanks, > Anand > > ---------- Forwarded message ---------- > From: Anand Mohanadoss > Date: Wed, Feb 25, 2015 at 9:35 PM > Subject: Memory leak from seek/ftell with files larger than 2GB > To: guile-user@gnu.org > > Hi, > > We are seeing an issue with seek and ftell leaking memory with files > larger than 2GB. > > We are using 2.0.11 guile built as a 32-bit application with large > file support enabled (guile was built using gcc 4.4.0 for Linux with > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _ > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. > > The memory leaks start only after the offset exceeds maximum positive > value for a 32-bit signed integer. ftell and seek do work as expected > (given how lseek should work with large file support). > > Appended is a program that illustrates the problem. The first seek > simply skips the part of the file where you won't see a memory leak. > If you comment out ftell and the second seek lines and un-comment the > lines that follow them, there is no memory leak. > > Is this a bug in guile or should we be doing things differently? If > this is a known issue, is there a recommended work around? > > Thanks, > Anand > > (define MAX_SIGNED_INT 2147483647) > (define BYTES_TO_READ 10) > > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB > > (define (traverse file) > (let* ((port (open-input-file file #:binary #t)) > (file-sz (stat:size (stat port))) > (ua (make-bytevector BYTES_TO_READ 0)) > (cur-offset 0)) > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) > (while (< (ftell port) (- file-sz BYTES_TO_READ)) > ;(while (< cur-offset (- file-sz BYTES_TO_READ)) > (seek port BYTES_TO_READ SEEK_CUR) > ;(get-bytevector-n! port ua 0 BYTES_TO_READ) > (set! cur-offset (+ BYTES_TO_READ cur-offset))) > (close-port port))) > > (traverse file) --=-=-=-- From unknown Mon Jun 23 02:23:33 2025 MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) X-Loop: help-debbugs@gnu.org From: help-debbugs@gnu.org (GNU bug Tracking System) To: Anand Mohanadoss Subject: bug#20079: closed (Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB) Message-ID: References: <87shw4njne.fsf@pobox.com> X-Gnu-PR-Message: they-closed 20079 X-Gnu-PR-Package: guile Reply-To: 20079@debbugs.gnu.org Date: Thu, 23 Jun 2016 13:03:02 +0000 Content-Type: multipart/mixed; boundary="----------=_1466686982-3650-1" This is a multi-part message in MIME format... ------------=_1466686982-3650-1 Content-Disposition: inline Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="utf-8" Your bug report #20079: Fwd: Memory leak from seek/ftell with files larger than 2GB which was filed against the guile package, has been closed. The explanation is attached below, along with your original report. If you require more details, please reply to 20079@debbugs.gnu.org. --=20 20079: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D20079 GNU Bug Tracking System Contact help-debbugs@gnu.org with problems ------------=_1466686982-3650-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at 20079-done) by debbugs.gnu.org; 23 Jun 2016 13:02:14 +0000 Received: from localhost ([127.0.0.1]:51915 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG4Go-0000vs-SE for submit@debbugs.gnu.org; Thu, 23 Jun 2016 09:02:14 -0400 Received: from pb-sasl1.pobox.com ([64.147.108.66]:65234 helo=sasl.smtp.pobox.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG4Gk-0000sR-3L for 20079-done@debbugs.gnu.org; Thu, 23 Jun 2016 09:02:09 -0400 Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id AC1F721EF8; Thu, 23 Jun 2016 09:02:05 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=+5dJZCKBitAFSrfAaK+mX2ZfDK8=; b=Q5wt51 l82FsGZYewehD1cynCL1FctkWDZTbh6URnzweLhgqY+gVgfqWDd/pI7E54ZKDpJo iNkVg2LGd3m/JdBSZXdpXWATsfM8UeL41lH79ZqP7N+E9LrnF0ApBrQjFTmKZPlP sbaM5qCEmJgGuwwWEqOqd7BkReCsfftpYEKX8= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=kAIuhh29qw9HTyMf8oT8DzAxPrUuuFuG GxedX2XBdUlYp1w2bFQjN6n7FU57fuAF8MEehh502QJoeycPPnFi4Z8IrgWqFIan xhEDT8t8lgk17nXjywxQCh/v2j7KwArPXE453ayOeQcPIcWZpHmxwRXWqDufTYt8 BUlraAci3dY= Received: from pb-sasl1.nyi.icgroup.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id A212A21EF6; Thu, 23 Jun 2016 09:02:05 -0400 (EDT) Received: from clucks (unknown [88.160.190.192]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by pb-sasl1.pobox.com (Postfix) with ESMTPSA id BF45C21EF5; Thu, 23 Jun 2016 09:02:04 -0400 (EDT) From: Andy Wingo To: Anand Mohanadoss Subject: Re: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB References: Date: Thu, 23 Jun 2016 15:01:57 +0200 In-Reply-To: (Anand Mohanadoss's message of "Wed, 11 Mar 2015 18:08:18 +0530") Message-ID: <87shw4njne.fsf@pobox.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: AF269310-3942-11E6-9342-C1836462E9F6-02397024!pb-sasl1.pobox.com X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: 20079-done Cc: 20079-done@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -1.4 (-) Hi, Thank you very much for this one! Turns out we had an incredibly embarrassing bug in which we forgot to attach finalizers for bignums created by scm_from_{uint64,int64} on 32-bit platforms. Fixed in master and stable-2.0. Cheers, Andy On Wed 11 Mar 2015 13:38, Anand Mohanadoss writes: > Hi, > > I had sent the following to the user forum and did not receive any > comments. I am reposting it in the bug forum with the hope that one of > the experts may be able to comment... > > Thanks, > Anand > > ---------- Forwarded message ---------- > From: Anand Mohanadoss > Date: Wed, Feb 25, 2015 at 9:35 PM > Subject: Memory leak from seek/ftell with files larger than 2GB > To: guile-user@gnu.org > > Hi, > > We are seeing an issue with seek and ftell leaking memory with files > larger than 2GB. > > We are using 2.0.11 guile built as a 32-bit application with large > file support enabled (guile was built using gcc 4.4.0 for Linux with > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _ > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. > > The memory leaks start only after the offset exceeds maximum positive > value for a 32-bit signed integer. ftell and seek do work as expected > (given how lseek should work with large file support). > > Appended is a program that illustrates the problem. The first seek > simply skips the part of the file where you won't see a memory leak. > If you comment out ftell and the second seek lines and un-comment the > lines that follow them, there is no memory leak. > > Is this a bug in guile or should we be doing things differently? If > this is a known issue, is there a recommended work around? > > Thanks, > Anand > > (define MAX_SIGNED_INT 2147483647) > (define BYTES_TO_READ 10) > > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB > > (define (traverse file) > (let* ((port (open-input-file file #:binary #t)) > (file-sz (stat:size (stat port))) > (ua (make-bytevector BYTES_TO_READ 0)) > (cur-offset 0)) > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) > (while (< (ftell port) (- file-sz BYTES_TO_READ)) > ;(while (< cur-offset (- file-sz BYTES_TO_READ)) > (seek port BYTES_TO_READ SEEK_CUR) > ;(get-bytevector-n! port ua 0 BYTES_TO_READ) > (set! cur-offset (+ BYTES_TO_READ cur-offset))) > (close-port port))) > > (traverse file) ------------=_1466686982-3650-1 Content-Type: message/rfc822 Content-Disposition: inline Content-Transfer-Encoding: 7bit Received: (at submit) by debbugs.gnu.org; 11 Mar 2015 12:38:32 +0000 Received: from localhost ([127.0.0.1]:42356 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVfuB-0005NT-N2 for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:32 -0400 Received: from eggs.gnu.org ([208.118.235.92]:37602) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from ) id 1YVfu9-0005N9-1D for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:29 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVfu2-0007sx-9o for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:23 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: * X-Spam-Status: No, score=1.1 required=5.0 tests=BAYES_50, FREEMAIL_ENVFROM_END_DIGIT,FREEMAIL_FROM,HTML_MESSAGE,T_DKIM_INVALID autolearn=disabled version=3.3.2 Received: from lists.gnu.org ([2001:4830:134:3::11]:56954) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVfu2-0007st-6B for submit@debbugs.gnu.org; Wed, 11 Mar 2015 08:38:22 -0400 Received: from eggs.gnu.org ([2001:4830:134:3::10]:39319) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVfu0-0006Hc-Tk for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:22 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1YVftz-0007rD-KD for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:20 -0400 Received: from mail-ob0-x22f.google.com ([2607:f8b0:4003:c01::22f]:35398) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1YVftz-0007qM-AB for bug-guile@gnu.org; Wed, 11 Mar 2015 08:38:19 -0400 Received: by obcwo20 with SMTP id wo20so8496517obc.2 for ; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=+hjBN9N/jJuZ9BPtMndgDn02v5in5pXYkqJcFIrxp20=; b=I0iRrsmWqcRd8ibESmwiSRyNRm85xYJ959j3eiNjNL1RpCEnxm/oUVc0kCOmdADlC2 O/iVRuhi4+ZMoR99n6PNbL28Mv6xhhgl6GDoNzr5zDKYd5/ih/z3EG9qBaxiufxAN7BV NzD+kahOkPUoQByf/WpsVhEMui8ADoVrVFz36OMKuauouVmY29r7N1BUqMm07Kwqda9c 6QhIdVg93gi3YOPk4GPs8NYSgl2nB9f4r8cU6gtqr72VwcC0c5tdVXdpNUJo+FAfFCTz tH+gtJ+EXwP90LPbgzvaOknehCw3voPxMCeGUZ0am74zrGr6xjbJMBedjF15y2UAhcr+ xE0w== MIME-Version: 1.0 X-Received: by 10.182.65.71 with SMTP id v7mr29738780obs.42.1426077498152; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) Received: by 10.202.195.9 with HTTP; Wed, 11 Mar 2015 05:38:18 -0700 (PDT) In-Reply-To: References: Date: Wed, 11 Mar 2015 18:08:18 +0530 Message-ID: Subject: Fwd: Memory leak from seek/ftell with files larger than 2GB From: Anand Mohanadoss To: bug-guile@gnu.org Content-Type: multipart/alternative; boundary=089e01537f62b320b20511028a18 X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-detected-operating-system: by eggs.gnu.org: Error: Malformed IPv6 address (bad octet value). X-Received-From: 2001:4830:134:3::11 X-Spam-Score: -3.8 (---) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.8 (---) --089e01537f62b320b20511028a18 Content-Type: text/plain; charset=UTF-8 Hi, I had sent the following to the user forum and did not receive any comments. I am reposting it in the bug forum with the hope that one of the experts may be able to comment... Thanks, Anand ---------- Forwarded message ---------- From: Anand Mohanadoss Date: Wed, Feb 25, 2015 at 9:35 PM Subject: Memory leak from seek/ftell with files larger than 2GB To: guile-user@gnu.org Hi, We are seeing an issue with seek and ftell leaking memory with files larger than 2GB. We are using 2.0.11 guile built as a 32-bit application with large file support enabled (guile was built using gcc 4.4.0 for Linux with flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. The memory leaks start only after the offset exceeds maximum positive value for a 32-bit signed integer. ftell and seek do work as expected (given how lseek should work with large file support). Appended is a program that illustrates the problem. The first seek simply skips the part of the file where you won't see a memory leak. If you comment out ftell and the second seek lines and un-comment the lines that follow them, there is no memory leak. Is this a bug in guile or should we be doing things differently? If this is a known issue, is there a recommended work around? Thanks, Anand (define MAX_SIGNED_INT 2147483647) (define BYTES_TO_READ 10) (define file "/tmp/test.pcap") ;sample file greater than 2.5GB (define (traverse file) (let* ((port (open-input-file file #:binary #t)) (file-sz (stat:size (stat port))) (ua (make-bytevector BYTES_TO_READ 0)) (cur-offset 0)) (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) (while (< (ftell port) (- file-sz BYTES_TO_READ)) ;(while (< cur-offset (- file-sz BYTES_TO_READ)) (seek port BYTES_TO_READ SEEK_CUR) ;(get-bytevector-n! port ua 0 BYTES_TO_READ) (set! cur-offset (+ BYTES_TO_READ cur-offset))) (close-port port))) (traverse file) --089e01537f62b320b20511028a18 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi,

I had sent the following t= o the user forum and did not receive any comments.=C2=A0 I am reposting it = in the bug forum with the hope that one of the experts may be able to comme= nt...

Thanks,
Anand

---------- Forwarded message ----------
From: Anand Mohanadoss <anand108@gmail.com>
Date: W= ed, Feb 25, 2015 at 9:35 PM
Subject: Memory leak from seek/ftell with fi= les larger than 2GB
To: guile-user= @gnu.org


Hi,<= br>
We are seeing an issue with seek and ftell leaking memory with= files larger than 2GB.

We are using 2.0.11 guile built as a 3= 2-bit application with large file support enabled (guile was built using gc= c 4.4.0 for Linux with flags _FILE_OFFSET_BITS=3D64, _LARGEFILE_SOURCE and = _LARGEFILE64_SOURCE).=C2=A0 The issue also appears to happen with guile 2.2= .

The memory leaks start only after the offset exceeds ma= ximum positive value for a 32-bit signed integer. ftell and seek do work as= expected (given how lseek should work with large file support).

Appended is a program that illustrates the problem.=C2=A0 The first seek= simply skips the part of the file where you won't see a memory leak. I= f you comment out ftell and the second seek lines and un-comment the lines = that follow them, there is no memory leak.=C2=A0

Is this a bu= g in guile or should we be doing things differently?=C2=A0 If this is a kno= wn issue, is there a recommended work around?

Thanks,
Anand
=C2=A0
(define MAX_SIGNED_= INT 214= 7483647)
(define BYTES_TO_READ 10)

(define file "/tmp/te= st.pcap")=C2=A0 ;sample file greater than 2.5GB

(define (traver= se file)
=C2=A0(let* ((port (open-input-file file #:binary #t))
=C2= =A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (file-sz (stat:size (stat port)))=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (ua (make-bytevector BYTES_TO_= READ 0))
=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (cur-offset 0))
= =C2=A0=C2=A0 (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
=C2=A0=C2=A0= (while (< (ftell port) (- file-sz BYTES_TO_READ))
=C2=A0=C2=A0 ;(whi= le (< cur-offset (- file-sz BYTES_TO_READ))
=C2=A0=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0 (seek=C2=A0 port BYTES_TO_READ SEEK_CUR)
=C2=A0=C2=A0=C2=A0= =C2=A0=C2=A0=C2=A0 ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
=C2=A0= =C2=A0=C2=A0=C2=A0=C2=A0=C2=A0 (set! cur-offset (+ BYTES_TO_READ cur-offset= )))
=C2=A0=C2=A0 (close-port port)))

(traverse file)

--089e01537f62b320b20511028a18-- ------------=_1466686982-3650-1-- From unknown Mon Jun 23 02:23:33 2025 X-Loop: help-debbugs@gnu.org Subject: bug#20079: Fwd: Memory leak from seek/ftell with files larger than 2GB Resent-From: Anand Mohanadoss Original-Sender: "Debbugs-submit" Resent-CC: bug-guile@gnu.org Resent-Date: Thu, 23 Jun 2016 14:44:02 +0000 Resent-Message-ID: Resent-Sender: help-debbugs@gnu.org X-GNU-PR-Message: followup 20079 X-GNU-PR-Package: guile X-GNU-PR-Keywords: To: Andy Wingo Cc: 20079-done@debbugs.gnu.org Received: via spool by 20079-done@debbugs.gnu.org id=D20079.146669299014405 (code D ref 20079); Thu, 23 Jun 2016 14:44:02 +0000 Received: (at 20079-done) by debbugs.gnu.org; 23 Jun 2016 14:43:10 +0000 Received: from localhost ([127.0.0.1]:52786 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG5qX-0003kH-P5 for submit@debbugs.gnu.org; Thu, 23 Jun 2016 10:43:10 -0400 Received: from mail-io0-f171.google.com ([209.85.223.171]:36533) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1bG5qW-0003jz-8n for 20079-done@debbugs.gnu.org; Thu, 23 Jun 2016 10:43:08 -0400 Received: by mail-io0-f171.google.com with SMTP id s63so69479567ioi.3 for <20079-done@debbugs.gnu.org>; Thu, 23 Jun 2016 07:43:08 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:from:date:message-id:subject:to :cc; bh=g51otMSx0L/nsA0Cr3TVG0dwf50IlvpGRlfSgS70IYM=; b=ZrQZBua3Bt3kAmvJec9DRPgoDrk53pp4VtoTQBfAaH8l9IOMSybDma4CDuL90gIjyz Bv8Vj9v2vbjNrchGSnv7tittXZWvLwZuKVU4y40+vRlCsR0E3bFEFeFYIdiWqIk/Ylv4 gHKkIPWTrCjNslSkmx8GwXgc6KTaXX/PdkmnyAbyVzGHgGtZVmGIrOrV9VFskfpp9JHH VTe9CtK/0SJOTZMESEoK3HPE1eMtaq5JVGzaerkv9Ji8WgK6UNb6UE5aItkVWP/WnEbE wbogmPoKkCqKwTk+aSPsLaBxaJ4u5rMhgNyOPkAY8oxcGplR07vR8PaBnU0btH1bt98p n5pQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:from:date :message-id:subject:to:cc; bh=g51otMSx0L/nsA0Cr3TVG0dwf50IlvpGRlfSgS70IYM=; b=irRPuzDUfNYs4x7bQheaUtt7owzQZl6YxLh1pN001YdCeZwGLvHxz6riGYKVuwhuoG X6+7OVhQw1W7c0YX5sAZIHNuQPvIP3gT6ilDRilAQ5b6y6tMwNB9HIqD5kmh0ZI1EnWv oOCzvzf/yxwwcMtWQHbRyRkoBhqhmLwS6URQ9Zll6yT0i953s+mBHRFpKa0YwtPqaLsN xdTZ5IZn9KB0gQwI6NgMjNvqlp1OQLfPPBaGUe1oOrVCOt7DL2MhSzTG+A3dnnRfAWmp rwBk9Q3QF7bTSmU8OH1n1nOntlKjyimjoyBOV1ogqMSnEiT6t6WiBeYroL/Rsi7bmekj 4Uhw== X-Gm-Message-State: ALyK8tJ7QQA7c3DeWnGNsQQ35rc8mM9OEKWzi6nulPivib+31/M3Qyf2kDTL33G3sdg2JNbhMZnBhUsGh4vK5w== X-Received: by 10.107.170.170 with SMTP id g42mr1343556ioj.44.1466692982300; Thu, 23 Jun 2016 07:43:02 -0700 (PDT) MIME-Version: 1.0 Received: by 10.107.154.68 with HTTP; Thu, 23 Jun 2016 07:43:01 -0700 (PDT) In-Reply-To: <87shw4njne.fsf@pobox.com> References: <87shw4njne.fsf@pobox.com> From: Anand Mohanadoss Date: Thu, 23 Jun 2016 20:13:01 +0530 Message-ID: Content-Type: multipart/alternative; boundary=001a114260ba346e8c0535f312d0 X-Spam-Score: -0.4 (/) X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -0.4 (/) --001a114260ba346e8c0535f312d0 Content-Type: text/plain; charset=UTF-8 Hi Andy, Thanks a lot for the fix! Anand On Thu, Jun 23, 2016 at 6:31 PM, Andy Wingo wrote: > Hi, > > Thank you very much for this one! Turns out we had an incredibly > embarrassing bug in which we forgot to attach finalizers for bignums > created by scm_from_{uint64,int64} on 32-bit platforms. Fixed in master > and stable-2.0. > > Cheers, > > Andy > > On Wed 11 Mar 2015 13:38, Anand Mohanadoss writes: > > > Hi, > > > > I had sent the following to the user forum and did not receive any > > comments. I am reposting it in the bug forum with the hope that one of > > the experts may be able to comment... > > > > Thanks, > > Anand > > > > ---------- Forwarded message ---------- > > From: Anand Mohanadoss > > Date: Wed, Feb 25, 2015 at 9:35 PM > > Subject: Memory leak from seek/ftell with files larger than 2GB > > To: guile-user@gnu.org > > > > Hi, > > > > We are seeing an issue with seek and ftell leaking memory with files > > larger than 2GB. > > > > We are using 2.0.11 guile built as a 32-bit application with large > > file support enabled (guile was built using gcc 4.4.0 for Linux with > > flags _FILE_OFFSET_BITS=64, _LARGEFILE_SOURCE and _ > > LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2. > > > > The memory leaks start only after the offset exceeds maximum positive > > value for a 32-bit signed integer. ftell and seek do work as expected > > (given how lseek should work with large file support). > > > > Appended is a program that illustrates the problem. The first seek > > simply skips the part of the file where you won't see a memory leak. > > If you comment out ftell and the second seek lines and un-comment the > > lines that follow them, there is no memory leak. > > > > Is this a bug in guile or should we be doing things differently? If > > this is a known issue, is there a recommended work around? > > > > Thanks, > > Anand > > > > (define MAX_SIGNED_INT 2147483647) > > (define BYTES_TO_READ 10) > > > > (define file "/tmp/test.pcap") ;sample file greater than 2.5GB > > > > (define (traverse file) > > (let* ((port (open-input-file file #:binary #t)) > > (file-sz (stat:size (stat port))) > > (ua (make-bytevector BYTES_TO_READ 0)) > > (cur-offset 0)) > > (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR) > > (while (< (ftell port) (- file-sz BYTES_TO_READ)) > > ;(while (< cur-offset (- file-sz BYTES_TO_READ)) > > (seek port BYTES_TO_READ SEEK_CUR) > > ;(get-bytevector-n! port ua 0 BYTES_TO_READ) > > (set! cur-offset (+ BYTES_TO_READ cur-offset))) > > (close-port port))) > > > > (traverse file) > --001a114260ba346e8c0535f312d0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
Hi Andy,

Thanks a lot for the fix! =

Anand

On Thu, Jun 23, 2016 at 6:31 PM, Andy Wingo &= lt;wingo@pobox.com= > wrote:
Hi,

Thank you very much for this one!=C2=A0 Turns out we had an incredibly
embarrassing bug in which we forgot to attach finalizers for bignums
created by scm_from_{uint64,int64} on 32-bit platforms.=C2=A0 Fixed in mast= er
and stable-2.0.

Cheers,

Andy

On Wed 11 Mar 2015 13:38, Anand Mohanadoss <anand108@gmail.com> writes:

> Hi,
>
> I had sent the following to the user forum and did not receive any
> comments. I am reposting it in the bug forum with the hope that one of=
> the experts may be able to comment...
>
> Thanks,
> Anand
>
> ---------- Forwarded message ----------
> From: Anand Mohanadoss <anand= 108@gmail.com>
> Date: Wed, Feb 25, 2015 at 9:35 PM
> Subject: Memory leak from seek/ftell with files larger than 2GB
> To: guile-user@gnu.org
>
> Hi,
>
> We are seeing an issue with seek and ftell leaking memory with files > larger than 2GB.
>
> We are using 2.0.11 guile built as a 32-bit application with large
> file support enabled (guile was built using gcc 4.4.0 for Linux with > flags _FILE_OFFSET_BITS=3D64, _LARGEFILE_SOURCE and _
> LARGEFILE64_SOURCE). The issue also appears to happen with guile 2.2.<= br> >
> The memory leaks start only after the offset exceeds maximum positive<= br> > value for a 32-bit signed integer. ftell and seek do work as expected<= br> > (given how lseek should work with large file support).
>
> Appended is a program that illustrates the problem. The first seek
> simply skips the part of the file where you won't see a memory lea= k.
> If you comment out ftell and the second seek lines and un-comment the<= br> > lines that follow them, there is no memory leak.
>
> Is this a bug in guile or should we be doing things differently? If > this is a known issue, is there a recommended work around?
>
> Thanks,
> Anand
>
> (define MAX_SIGNED_INT 2147483647)
> (define BYTES_TO_READ 10)
>
> (define file "/tmp/test.pcap") ;sample file greater than 2.5= GB
>
> (define (traverse file)
> (let* ((port (open-input-file file #:binary #t))
> (file-sz (stat:size (stat port)))
> (ua (make-bytevector BYTES_TO_READ 0))
> (cur-offset 0))
> (seek port (- MAX_UNSIGNED_INT 1000) SEEK_CUR)
> (while (< (ftell port) (- file-sz BYTES_TO_READ))
> ;(while (< cur-offset (- file-sz BYTES_TO_READ))
> (seek port BYTES_TO_READ SEEK_CUR)
> ;(get-bytevector-n! port ua 0 BYTES_TO_READ)
> (set! cur-offset (+ BYTES_TO_READ cur-offset)))
> (close-port port)))
>
> (traverse file)

--001a114260ba346e8c0535f312d0--