GNU bug report logs - #20511
split : does not account for --numeric-suffixes=FROM in calculation of suffix length?

Previous Next

Full log

View this message in rfc822 format

From: help-debbugs <at> gnu.org (GNU bug Tracking System)
To: Pádraig Brady <P <at> draigBrady.com>
Cc: tracker <at> debbugs.gnu.org
Subject: bug#20511: closed (split : does not account for --numeric-suffixes=FROM
 in calculation of suffix length?)
Date: Wed, 13 May 2015 01:22:02 +0000

[Message part 1 (text/plain, inline)]

Your message dated Wed, 13 May 2015 02:20:27 +0100
with message-id <5552A6DB.2000602 <at> draigBrady.com>
and subject line Re: bug#20511: split : does not account for --numeric-suffixes=FROM in calculation of suffix length?
has caused the debbugs.gnu.org bug report #20511,
regarding split : does not account for --numeric-suffixes=FROM in calculation of suffix length?
to be marked as done.

(If you believe you have received this mail in error, please contact
help-debbugs <at> gnu.org.)


-- 
20511: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=20511
GNU Bug Tracking System
Contact help-debbugs <at> gnu.org with problems

[Message part 2 (message/rfc822, inline)]

From: Ben Rusholme <rusholme <at> caltech.edu>
To: bug-coreutils <at> gnu.org
Subject: split : does not account for --numeric-suffixes=FROM in calculation
 of suffix length?
Date: Tue, 5 May 2015 13:42:12 -0700

[Message part 3 (text/plain, inline)]

Hi,

“split” (in the current GNU coreutils 8.23 release) does not account for the optional start index (“split --numeric-suffixes=FROM”) when calculating suffix length.

I couldn’t find any prior reference to this problem in either the bug tracker or mailing list archive.

Thanks, Ben



$ seq 100 >& input.txt
$ split --numeric-suffixes --number=l/100 input.txt
$ ls
input.txt  x06  x13  x20  x27  x34  x41  x48  x55  x62  x69  x76  x83  x90  x97
x00        x07  x14  x21  x28  x35  x42  x49  x56  x63  x70  x77  x84  x91  x98
x01        x08  x15  x22  x29  x36  x43  x50  x57  x64  x71  x78  x85  x92  x99
x02        x09  x16  x23  x30  x37  x44  x51  x58  x65  x72  x79  x86  x93
x03        x10  x17  x24  x31  x38  x45  x52  x59  x66  x73  x80  x87  x94
x04        x11  x18  x25  x32  x39  x46  x53  x60  x67  x74  x81  x88  x95
x05        x12  x19  x26  x33  x40  x47  x54  x61  x68  x75  x82  x89  x96


$ rm x*
$ split --numeric-suffixes=1 --number=l/100 input.txt
split: output file suffixes exhausted
$ ls
input.txt  x07  x14  x21  x28  x35  x42  x49  x56  x63  x70  x77  x84  x91  x98
x01        x08  x15  x22  x29  x36  x43  x50  x57  x64  x71  x78  x85  x92  x99
x02        x09  x16  x23  x30  x37  x44  x51  x58  x65  x72  x79  x86  x93
x03        x10  x17  x24  x31  x38  x45  x52  x59  x66  x73  x80  x87  x94
x04        x11  x18  x25  x32  x39  x46  x53  x60  x67  x74  x81  x88  x95
x05        x12  x19  x26  x33  x40  x47  x54  x61  x68  x75  x82  x89  x96
x06        x13  x20  x27  x34  x41  x48  x55  x62  x69  x76  x83  x90  x97
$ # Should run from x001 to x100!


$ rm x*
$ split --numeric-suffixes=1 --number=l/101 input.txt
$ ls
input.txt  x008  x016  x024  x032  x040  x048  x056  x064  x072  x080  x088  x096
x001       x009  x017  x025  x033  x041  x049  x057  x065  x073  x081  x089  x097
x002       x010  x018  x026  x034  x042  x050  x058  x066  x074  x082  x090  x098
x003       x011  x019  x027  x035  x043  x051  x059  x067  x075  x083  x091  x099
x004       x012  x020  x028  x036  x044  x052  x060  x068  x076  x084  x092  x100
x005       x013  x021  x029  x037  x045  x053  x061  x069  x077  x085  x093  x101
x006       x014  x022  x030  x038  x046  x054  x062  x070  x078  x086  x094
x007       x015  x023  x031  x039  x047  x055  x063  x071  x079  x087  x095

[Message part 4 (text/html, inline)]

[Message part 5 (message/rfc822, inline)]

From: Pádraig Brady <P <at> draigBrady.com>
To: Ben Rusholme <rusholme <at> caltech.edu>, 20511-done <at> debbugs.gnu.org
Subject: Re: bug#20511: split : does not account for --numeric-suffixes=FROM
 in calculation of suffix length?
Date: Wed, 13 May 2015 02:20:27 +0100

[Message part 6 (text/plain, inline)]

On 06/05/15 11:53, Pádraig Brady wrote:
> On 06/05/15 05:29, Ben Rusholme wrote:
>> As you say, this can always be fixed by the "--suffix-length" argument, but it’s only required for certain combinations of FROM and CHUNK, (and “split” already has all the information it needs).
>>
>>> Now you could bump the suffix length based on the start number,
>>> though I don't think we should as that would impact on future
>>> processing (ordering) of the resultant files.  I.E. specifying
>>> a FROM value to --numeric-suffixes should only impact the
>>> start value, rather than the width.
>>
>> Could you clarify this for me? Doesn’t the zero-padding ensure correct processing order?
> 
> There are two use cases supported by specifying FROM.
> 1. Setting the start for a single run (FROM is usually 1 in this case)
> 2. Setting the offset for multiple independent split runs.
> In the second case we can't infer the size of the total set
> in any particular run, and thus require that --suffix-length is specified appropriately.
> I.E. for multiple independent runs, the suffix length needs to be
> fixed width across the entire set for the total ordering to be correct.
> 
> 
> Things we could change are...
> 
> 1. Special case FROM=1 to assume a single run and thus
> enable auto suffix expansion or appropriately sized suffix with CHUNK.
> This would be a backwards incompat change and also not
> guaranteed a single run, so I'm reluctant to do that.
> 
> 2. Give an early error with specified FROM and CHUNK
> that would overflow the suffix size for CHUNK.
> This would save some processing, though doesn't add
> any protections against latent issues. I.E. you still get
> the error which is dependent on the parameters rather than the input data size.
> Therefore it's probably not worth the complication.
> 
> 3. Leave suffix length at 2 when both FROM and CHUNK are specified.
> In retrospect, this would probably have been the best option
> to avoid ambiguities like this. However now we'd be breaking
> compat with scripts with FROM=1 and CHUNK=200 etc.
> While CHUNK values > 100 would be unusual
> 
> 4. Auto set the suffix len based on FROM + CHUNK.
> That would support use case 1 (single run),
> but _silently_ break subsequent processing order
> of outputs from multiple split runs
> (as FROM is increased in multiples of CHUNK size).
> We could mitigate the _silent_ breakage though
> by limiting this change to when FROM < CHUNK.
> 
> 5. Document in man page and with more detail in info docs
> that -a is recommended when specifying FROM
> 
> So I'll do 4 and 5 I think.

Attached.

cheers,
Pádraig

[split-from-width.patch (text/x-patch, attachment)]

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.