From debbugs-submit-bounces@debbugs.gnu.org Mon Dec 03 22:21:45 2012 Received: (at submit) by debbugs.gnu.org; 4 Dec 2012 03:21:45 +0000 Received: from localhost ([127.0.0.1]:52271 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Tfj4p-000349-94 for submit@debbugs.gnu.org; Mon, 03 Dec 2012 22:21:44 -0500 Received: from eggs.gnu.org ([208.118.235.92]:39106) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Tfj4n-000342-4q for submit@debbugs.gnu.org; Mon, 03 Dec 2012 22:21:41 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Tfj2Q-0004Kg-Es for submit@debbugs.gnu.org; Mon, 03 Dec 2012 22:19:15 -0500 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-102.6 required=5.0 tests=BAYES_00,FREEMAIL_FROM, RCVD_IN_DNSWL_LOW, T_DKIM_INVALID, USER_IN_WHITELIST autolearn=unavailable version=3.3.2 Received: from lists.gnu.org ([208.118.235.17]:55268) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tfj2Q-0004Kc-AL for submit@debbugs.gnu.org; Mon, 03 Dec 2012 22:19:14 -0500 Received: from eggs.gnu.org ([208.118.235.92]:37934) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tfj2M-0001iO-6i for bug-coreutils@gnu.org; Mon, 03 Dec 2012 22:19:14 -0500 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1Tfj2G-0004K2-Gi for bug-coreutils@gnu.org; Mon, 03 Dec 2012 22:19:10 -0500 Received: from mail-pb0-f41.google.com ([209.85.160.41]:34185) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1Tfj2G-0004Jr-AG for bug-coreutils@gnu.org; Mon, 03 Dec 2012 22:19:04 -0500 Received: by mail-pb0-f41.google.com with SMTP id xa7so2411512pbc.0 for ; Mon, 03 Dec 2012 19:19:03 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:date:message-id:subject:from:to:content-type; bh=8ImAO5c2b51ZUEdQL7i+GeXaT/y9bngw6QLYPVILDtE=; b=MyvyC+iy8fy9QsCCWp+UbVSALXIeHQp6odtKURn9OSBtmiGbzKG/xpuQCpaddGNpy3 THmsVgD1RWk8djYcHmMwQCc7BEWisUkzOb69DpcWZ4Xo709xIz22ibVWsTrB2I+R+VGf 8FU+fdkfnYoCNlHkJe/6Ibl7Wz6ZTjwNDmPu5j2UUBKFcj7oVr3d06TxwTAbyPhCUjGZ 0hnQf1Va3jHQBXkUUrb4Cwh/i7xGh6mJve62qLQVoqt1U73OT2kxWjLDJx3m0trA029K phSrXtgY7WO0ZNDDyBhzkeYVCJJaXHs6G+kxSeGvhzioNB86NuNC/Mx7tY2IzKp2X2lP UPzA== MIME-Version: 1.0 Received: by 10.68.230.10 with SMTP id su10mr34516918pbc.47.1354591142914; Mon, 03 Dec 2012 19:19:02 -0800 (PST) Received: by 10.66.191.67 with HTTP; Mon, 3 Dec 2012 19:19:02 -0800 (PST) Date: Mon, 3 Dec 2012 21:19:02 -0600 Message-ID: Subject: fifo unlimited buffer size? From: Peng Yu To: bug-coreutils@gnu.org Content-Type: text/plain; charset=ISO-8859-1 X-detected-operating-system: by eggs.gnu.org: GNU/Linux 3.x [fuzzy] X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x X-Received-From: 208.118.235.17 X-Spam-Score: -3.4 (---) X-Debbugs-Envelope-To: submit X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -3.4 (---) Hi, I have the following script. When the number to the right of 'seq' is large (as 100000 in the example), the script will hang. But when the number is small (say 1000), the script can be finished correctly. I suspect that the problem is that there is a limit on the buffer size for fifo. Is it so? Is there a way to make the following script work no matter how large the number is? Thanks! ~/linux/test/gnu/gnu/coreutils/mkfifo/tee$ cat main2.sh #!/usr/bin/env bash rm -rf a b c mkfifo a b c seq 100000 | tee a > b & sort -k 1,1n a > c & join -j 1 <(awk 'BEGIN{OFS="\t"; FS="\t"} {print $1, $1+10}' < c) <(awk 'BEGIN{OFS="\t"; FS="\t"}{print $1, $1+20}' < b) -- Regards, Peng From debbugs-submit-bounces@debbugs.gnu.org Tue Dec 04 07:27:11 2012 Received: (at 13075) by debbugs.gnu.org; 4 Dec 2012 12:27:11 +0000 Received: from localhost ([127.0.0.1]:52529 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Tfrah-0007I3-4H for submit@debbugs.gnu.org; Tue, 04 Dec 2012 07:27:11 -0500 Received: from mx1.redhat.com ([209.132.183.28]:47825) by debbugs.gnu.org with esmtp (Exim 4.72) (envelope-from ) id 1Tfrae-0007Hv-GT for 13075@debbugs.gnu.org; Tue, 04 Dec 2012 07:27:10 -0500 Received: from int-mx09.intmail.prod.int.phx2.redhat.com (int-mx09.intmail.prod.int.phx2.redhat.com [10.5.11.22]) by mx1.redhat.com (8.14.4/8.14.4) with ESMTP id qB4COddx029364 (version=TLSv1/SSLv3 cipher=DHE-RSA-AES256-SHA bits=256 verify=OK); Tue, 4 Dec 2012 07:24:39 -0500 Received: from [10.36.116.70] (ovpn-116-70.ams2.redhat.com [10.36.116.70]) by int-mx09.intmail.prod.int.phx2.redhat.com (8.14.4/8.14.4) with ESMTP id qB4COZMe015933 (version=TLSv1/SSLv3 cipher=DHE-RSA-CAMELLIA256-SHA bits=256 verify=NO); Tue, 4 Dec 2012 07:24:38 -0500 Message-ID: <50BDEB83.3040405@draigBrady.com> Date: Tue, 04 Dec 2012 12:24:35 +0000 From: =?ISO-8859-1?Q?P=E1draig_Brady?= User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20120615 Thunderbird/13.0.1 MIME-Version: 1.0 To: Peng Yu Subject: Re: bug#13075: fifo unlimited buffer size? References: In-Reply-To: Content-Type: text/plain; charset=ISO-8859-1; format=flowed X-Scanned-By: MIMEDefang 2.68 on 10.5.11.22 Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by mx1.redhat.com id qB4COddx029364 X-Spam-Score: -4.2 (----) X-Debbugs-Envelope-To: 13075 Cc: 13075@debbugs.gnu.org X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.13 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: debbugs-submit-bounces@debbugs.gnu.org Errors-To: debbugs-submit-bounces@debbugs.gnu.org X-Spam-Score: -4.2 (----) tag 13075 + notabug close 13075 thanks On 12/04/2012 03:19 AM, Peng Yu wrote: > Hi, > > I have the following script. When the number to the right of 'seq' is > large (as 100000 in the example), the script will hang. But when the > number is small (say 1000), the script can be finished correctly. I > suspect that the problem is that there is a limit on the buffer size > for fifo. Is it so? Is there a way to make the following script work > no matter how large the number is? Thanks! > > ~/linux/test/gnu/gnu/coreutils/mkfifo/tee$ cat main2.sh > #!/usr/bin/env bash > > rm -rf a b c > mkfifo a b c > seq 100000 | tee a > b & > sort -k 1,1n a > c & > join -j 1 <(awk 'BEGIN{OFS=3D"\t"; FS=3D"\t"} {print $1, $1+10}' < c) > <(awk 'BEGIN{OFS=3D"\t"; FS=3D"\t"}{print $1, $1+20}' < b) So this is problematic due to `sort`. That's special as it needs to consume all its input before producing any output. Therefore unless the buffers connecting the other commands in || can consume the data, there will be a deadlock. This version doesn't block for example as the input is being generated asynchronously for the sort command. #!/usr/bin/env bash rm -rf a b c mkfifo a b c join -j 1 <(awk 'BEGIN{OFS=3D"\t"; FS=3D"\t"} {print $1, $1+10}' < c) \ <(awk 'BEGIN{OFS=3D"\t"; FS=3D"\t"}{print $1, $1+20}' < b) & seq 100000 | sort -k 1,1n > c & seq 100000 > b wait Obviously, if your input is expensive to generate, then you'd be best copying to another file and sorting that. thanks, P=E1draig. From debbugs-submit-bounces@debbugs.gnu.org Thu Oct 18 18:58:36 2018 Received: (at control) by debbugs.gnu.org; 18 Oct 2018 22:58:36 +0000 Received: from localhost ([127.0.0.1]:57514 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gDHFU-0001ak-Gz for submit@debbugs.gnu.org; Thu, 18 Oct 2018 18:58:36 -0400 Received: from mail-it1-f172.google.com ([209.85.166.172]:36128) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1gDHFS-0001aR-Ud for control@debbugs.gnu.org; Thu, 18 Oct 2018 18:58:35 -0400 Received: by mail-it1-f172.google.com with SMTP id c85-v6so2476363itd.1 for ; Thu, 18 Oct 2018 15:58:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=to:from:message-id:date:user-agent:mime-version:content-language :content-transfer-encoding; bh=3NxHo71kl0XWIgBOVCa5CXkQFPyI94koGPDOM+L4P90=; b=Z8ahyi1QyYvXg3d88ymOSnO9SNENVYg5apoYzHGjLGaNf0WmIWUl2oqIzAr4lqvsCQ ESmH3p8loT3ECeJlLCjvBQh63wJs+c2frWBkHm9w1X95TrCS7EGU57dqHYk2fbwShV1R r5hXtUvgF0BuiRU+ZRbox/bK6cPkrgOsv73ChRmhMHC9nR6lo0hCMSacfyGulvX0Mkeb KjNdD0Lx3QU5A9E2dXU7Qt0u5NrbRCAmidSHKQokQ+mP/+Cax3vhAMdoFBLlFBw2D/NV n3SYmw16YfnlXsgNC8sCHnpupXf3+zEto/gcYohJkMzhLG8W963pEvkYFpd3lhCBPFxa Z7vw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:to:from:message-id:date:user-agent:mime-version :content-language:content-transfer-encoding; bh=3NxHo71kl0XWIgBOVCa5CXkQFPyI94koGPDOM+L4P90=; b=dxMcEp1H+7aXhcFv+Iu/b/BJzO9BUiu0qTQbQg7/0RbjP2ba/bk1RUV3Ni/MpMz/ok mVzrWoRPBXd5UN8iwn4VgVcX+AhnsxyDc2gDlL0BUOhcnzoOLqfocGkLirlhUo5K+BOn PVS2K2HaB7Rw01w4fvqcvt5uxtdNpFimNysBUrKo4ODvJVS78Q0kuven2QUBAKcBiFGu kzD9pix26GDnKjCTZUH3WAfZkCM7xpi2s7/i0YNtW9fUJL0lft7ZeII2J29vn931j7R5 V7GyK+bOWLwiJeRnTexzbgb+Xmnol4Nn0sCh1lkq/kpbWpQI72NmKsOtpFDxkIRPZLxJ l9SQ== X-Gm-Message-State: ABuFfoi8Bj2B7aJC3laqean9fsJwCrcwp26zNbYHWMU1VfEy1wfPagkh nJK2EoNYIE4hMphEw/Ej/R29+Id5 X-Google-Smtp-Source: ACcGV63Ae8GE85qQBquK/Bi/mcxE2FZfAQk85d9YSZ4q426BcUzMbCo82FV2Eze5Fc4Ubbdtr27orA== X-Received: by 2002:a24:728e:: with SMTP id x136-v6mr1548634itc.40.1539903508726; Thu, 18 Oct 2018 15:58:28 -0700 (PDT) Received: from tomato.housegordon.com (moose.housegordon.com. [184.68.105.38]) by smtp.googlemail.com with ESMTPSA id q123-v6sm8135838iod.23.2018.10.18.15.58.27 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Thu, 18 Oct 2018 15:58:27 -0700 (PDT) To: control@debbugs.gnu.org From: Assaf Gordon Message-ID: <146d7d36-1cdc-2c6f-9bdd-0c17e870cb13@gmail.com> Date: Thu, 18 Oct 2018 16:58:26 -0600 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.9.1 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: 2.0 (++) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: tags 13075 notabug close 13075 [...] Content analysis details: (2.0 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 FREEMAIL_FROM Sender email is commonly abused enduser mail provider (assafgordon[at]gmail.com) -0.0 SPF_PASS SPF: sender matches SPF record -0.0 RCVD_IN_MSPIKE_H2 RBL: Average reputation (+2) [209.85.166.172 listed in wl.mailspike.net] -0.0 RCVD_IN_DNSWL_NONE RBL: Sender listed at http://www.dnswl.org/, no trust [209.85.166.172 listed in list.dnswl.org] 1.8 MISSING_SUBJECT Missing Subject: header 0.2 NO_SUBJECT Extra score for no subject X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: 1.0 (+) tags 13075 notabug close 13075 From unknown Fri Aug 15 19:35:05 2025 Received: (at fakecontrol) by fakecontrolmessage; To: internal_control@debbugs.gnu.org From: Debbugs Internal Request Subject: Internal Control Message-Id: bug archived. Date: Fri, 16 Nov 2018 12:24:08 +0000 User-Agent: Fakemail v42.6.9 # This is a fake control message. # # The action: # bug archived. thanks # This fakemail brought to you by your local debbugs # administrator