From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 10:12:55 2021 Received: (at submit) by debbugs.gnu.org; 15 Jun 2021 14:12:55 +0000 Received: from localhost ([127.0.0.1]:50371 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lt9oF-0005Bf-9D for submit@debbugs.gnu.org; Tue, 15 Jun 2021 10:12:55 -0400 Received: from lists.gnu.org ([209.51.188.17]:42852) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1lt7he-0003Bd-Cl for submit@debbugs.gnu.org; Tue, 15 Jun 2021 07:57:58 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:35802) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1lt7he-0006xc-3L for bug-diffutils@gnu.org; Tue, 15 Jun 2021 07:57:58 -0400 Received: from mail-lf1-x12f.google.com ([2a00:1450:4864:20::12f]:38588) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.90_1) (envelope-from ) id 1lt7hb-0006oh-3z for bug-diffutils@gnu.org; Tue, 15 Jun 2021 07:57:57 -0400 Received: by mail-lf1-x12f.google.com with SMTP id r5so26587698lfr.5 for ; Tue, 15 Jun 2021 04:57:53 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:subject:to:message-id:date:user-agent:mime-version :content-language:content-transfer-encoding; bh=PRS/OkyXz5ab8aCjPi7kbKa9W5dgDIBjnemV2H4B2IE=; b=Wkm0szfwt+exjziM30Q2F6CJbuEqLpV/IXADson/NdnumzLmhvmOoMd2tL4vNNHxXr EhlgA1RcVslRCkOQK/WL0JL2KWsenlgfGDEbOgmnFluApcNhH792IYDAkKPLoom7dhai nfrqr48p8mCdP8cFT3kws7SAS29L9Inmclbx8oVlG7RQr56EjquqJ2uNF7vDdwFcmSWw 7H8RsaPdTowUDCmsM2Qo0pSkvtZ2gHgvRcxNB+tU1xJwiezzIVkUbNi6Tt1+A5mqpAuh H1xllzCfFO0e8UNpIeGFDfLuhphJ5SgeBsBECN2yUu3reHxd6RxDK2R/lWfUUbVknuvM Hg1g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:subject:to:message-id:date:user-agent :mime-version:content-language:content-transfer-encoding; bh=PRS/OkyXz5ab8aCjPi7kbKa9W5dgDIBjnemV2H4B2IE=; b=Zw8D6RO2olP6l3dvWFA5m52gJgHhVO7yBs7uoJQ7H+gJnjYIcxAFa9wsVJYVMAI/Ef LJi4hXgtwvG80ypqFfJO4bZUWinJlvQZJvgg/SzcEFWAOnkDZ1gNBSBA3HT7qRVRPwNd R4LmFRVQwXg31lBpD029rK4uWMy4/Qgy2EqzSaKeqyXtZt6cFXqciu+yDwqyO3bU4Kjv 4eU0jtrZ+FwYAJ0+4iaA12IEu2wjDJrQaOpowL+fVaTSHo4ob4tBLe7bN90hYe6w15tB I9ymAEQtNaELAfFO264KZdmxmdHO0OsfgEEixtgSCPDVrxookGfAxBBAFBAG5/gqW6YQ Uo7w== X-Gm-Message-State: AOAM531PhfBJL+uzpnddio0fg/YdDKpT/mp/3e+aABZkmhMiB+hNZ6gr oDWg0M62h656JO88Qsu58dULSVhHwJjYwQ== X-Google-Smtp-Source: ABdhPJzawbXLIR2G4YC3f4kboNoLMTJcJZOzniF1RUAl70sew5kLkODi5RO1qynmpIdZC1BV1BM/Uw== X-Received: by 2002:a05:6512:3047:: with SMTP id b7mr9066484lfb.49.1623758271514; Tue, 15 Jun 2021 04:57:51 -0700 (PDT) Received: from [10.0.0.10] (ti0040q160-3065.bb.online.no. [62.16.223.30]) by smtp.gmail.com with ESMTPSA id i29sm1781639lfv.163.2021.06.15.04.57.50 for (version=TLS1_2 cipher=ECDHE-ECDSA-AES128-GCM-SHA256 bits=128/128); Tue, 15 Jun 2021 04:57:51 -0700 (PDT) From: Gisle Vanem Subject: Speeding up diff on Windows To: bug-diffutils@gnu.org Message-ID: <0432d4d2-b221-7e63-fdd9-5cecb9ae9887@gmail.com> Date: Tue, 15 Jun 2021 13:57:49 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-GB Content-Transfer-Encoding: 7bit Received-SPF: pass client-ip=2a00:1450:4864:20::12f; envelope-from=gisle.vanem@gmail.com; helo=mail-lf1-x12f.google.com X-Spam_score_int: -20 X-Spam_score: -2.1 X-Spam_bar: -- X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.3 (-) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Tue, 15 Jun 2021 10:12:51 -0400 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -2.3 (--) Hello list. I feel a 'diff --recursive ..' takes way too much time to diff 2 trees of a few 1000 files each. Reading the MSDN docs for '_open()', the under-laying cache-behaviour and the 'O_SEQUENTIAL' flag, it seems sensible that the Windows "Cache Manager" [1] works faster with the 'O_SEQUENTIAL' flag. So with this little pseudo-patch of 'diff.c': +#ifndef O_SEQUENTIAL +#define O_SEQUENTIAL 0 +#endif /* Open the files and record their descriptors. */ int oflags = O_RDONLY | (binary ? O_BINARY : 0); + oflags |= O_SEQUENTIAL; + if (cmp.file[0].desc == UNOPENED) if ((cmp.file[0].desc = open (cmp.file[0].name, oflags, 0)) < 0) I seem to get the job done quicker. I cannot give any numbers on the speed-increase. But others may have some comments? [1] https://docs.microsoft.com/en-us/windows/win32/fileio/file-caching -- --gv From debbugs-submit-bounces@debbugs.gnu.org Tue Jun 15 15:51:25 2021 Received: (at 49038) by debbugs.gnu.org; 15 Jun 2021 19:51:25 +0000 Received: from localhost ([127.0.0.1]:50892 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ltF5p-0001e2-Ju for submit@debbugs.gnu.org; Tue, 15 Jun 2021 15:51:25 -0400 Received: from zimbra.cs.ucla.edu ([131.179.128.68]:36804) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from ) id 1ltF5o-0001dn-8i for 49038@debbugs.gnu.org; Tue, 15 Jun 2021 15:51:25 -0400 Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id EBE9D16005E; Tue, 15 Jun 2021 12:51:17 -0700 (PDT) Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10032) with ESMTP id Lwt48SMDv5yR; Tue, 15 Jun 2021 12:51:17 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by zimbra.cs.ucla.edu (Postfix) with ESMTP id 1E3671600AD; Tue, 15 Jun 2021 12:51:17 -0700 (PDT) X-Virus-Scanned: amavisd-new at zimbra.cs.ucla.edu Received: from zimbra.cs.ucla.edu ([127.0.0.1]) by localhost (zimbra.cs.ucla.edu [127.0.0.1]) (amavisd-new, port 10026) with ESMTP id KpZfJ4Af7vKd; Tue, 15 Jun 2021 12:51:17 -0700 (PDT) Received: from [192.168.1.9] (cpe-172-91-119-151.socal.res.rr.com [172.91.119.151]) by zimbra.cs.ucla.edu (Postfix) with ESMTPSA id F026216005E; Tue, 15 Jun 2021 12:51:16 -0700 (PDT) Subject: Re: [bug-diffutils] bug#49038: Speeding up diff on Windows To: Gisle Vanem , 49038@debbugs.gnu.org References: <0432d4d2-b221-7e63-fdd9-5cecb9ae9887@gmail.com> From: Paul Eggert Organization: UCLA Computer Science Department Message-ID: <40e57b23-e31e-e915-b067-4061af6674a9@cs.ucla.edu> Date: Tue, 15 Jun 2021 12:51:16 -0700 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:78.0) Gecko/20100101 Thunderbird/78.8.1 MIME-Version: 1.0 In-Reply-To: <0432d4d2-b221-7e63-fdd9-5cecb9ae9887@gmail.com> Content-Type: text/plain; charset=utf-8; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Spam-Score: -2.4 (--) X-Debbugs-Envelope-To: 49038 X-BeenThere: debbugs-submit@debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: debbugs-submit-bounces@debbugs.gnu.org Sender: "Debbugs-submit" X-Spam-Score: -3.4 (---) On 6/15/21 4:57 AM, Gisle Vanem wrote: > I seem to get the job done quicker. I cannot give any numbers on > the speed-increase. Although GNU/Linux doesn't have O_SEQUENTIAL, it has somewhat similar flags (e.g., madvise with MADV_SEQUENTIAL). I've tried using these flags in the past and came up empty: they complicate the code and sometimes make things faster and sometimes slower and it wasn't worth the maintenance trouble to tease things out. I expect that Microsoft's O_SEQUENTIAL flag is similar. Besides, MS-Windows is relatively low priority for us.