GNU bug report logs - #20745
sort: add options to sort by IPv4/IPv6

Previous Next

To reply to this bug, email your comments to 20745 AT debbugs.gnu.org.

Toggle the display of automated, internal messages from the tracker.

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: "Silverman, Jeffrey X. -ND" <Jeffrey.X.Silverman.-ND <at> disney.com>
To: "bug-coreutils <at> gnu.org" <bug-coreutils <at> gnu.org>
Subject: I would like to make a request for the sort command
Date: Fri, 5 Jun 2015 17:13:21 +0000

[Message part 1 (text/plain, inline)]

People,

I realize that the sort command must be on the order of 40 years old and is "mature" software, but I would like to make a request for enhancement anyway.  I would like the capability of adding 2 data types that can be sorted: IPv4 addresses and IPv6 addresses.  To do that, the addresses should be converted to integers of length 32 bits and 128 bits and then sorted. I'm not sure that 128 bit integers is portable, but converting to a 32 bit string with no missing zeros would suffice.


If this is something that interests you, then I think I could write a preprocessor in python as a proof of concept.


Sincerely yours,


Jeff Silverman
You may also reach me at jeffsilverm <at> gmail.com

[Message part 2 (text/html, inline)]

Message #8 received at 20745 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: "Silverman, Jeffrey X. -ND" <Jeffrey.X.Silverman.-ND <at> disney.com>,
 20745 <at> debbugs.gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Fri, 05 Jun 2015 18:58:39 +0100

On 05/06/15 18:13, Silverman, Jeffrey X. -ND wrote:
> People,
> 
> I realize that the sort command must be on the order of 40 years old and is “mature” software, but I would like to make a request for enhancement anyway.  I would like the capability of adding 2 data types that can be sorted: IPv4 addresses and IPv6 addresses.  To do that, the addresses should be converted to integers of length 32 bits and 128 bits and then sorted. I’m not sure that 128 bit integers is portable, but converting to a 32 bit string with no missing zeros would suffice.
> 
> 
> If this is something that interests you, then I think I could write a preprocessor in python as a proof of concept.

This was previously discussed, and while has merit
at the time it was thought not important enough to add:

  http://www.gnu.org/software/coreutils/rejected_requests.html
    http://lists.gnu.org/archive/html/coreutils/2011-06/msg00082.html

thanks,
Pádraig.

Message #11 received at 20745 <at> debbugs.gnu.org (full text, mbox):

From: "Silverman, Jeffrey X. -ND" <Jeffrey.X.Silverman.-ND <at> disney.com>
To: Pádraig Brady <P <at> draigBrady.com>,
 "20745 <at> debbugs.gnu.org" <20745 <at> debbugs.gnu.org>
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Fri, 5 Jun 2015 19:35:30 +0000

On 6/5/15, 10:58 AM, "Pádraig Brady" <P <at> draigBrady.com> wrote:

>On 05/06/15 18:13, Silverman, Jeffrey X. -ND wrote:
>> People,
>> 
>> I realize that the sort command must be on the order of 40 years old
>>and is ³mature² software, but I would like to make a request for
>>enhancement anyway.  I would like the capability of adding 2 data types
>>that can be sorted: IPv4 addresses and IPv6 addresses.  To do that, the
>>addresses should be converted to integers of length 32 bits and 128 bits
>>and then sorted. I¹m not sure that 128 bit integers is portable, but
>>converting to a 32 bit string with no missing zeros would suffice.
>> 
>> 
>> If this is something that interests you, then I think I could write a
>>preprocessor in python as a proof of concept.
>
>This was previously discussed, and while has merit
>at the time it was thought not important enough to add:
>
>  http://www.gnu.org/software/coreutils/rejected_requests.html
>    http://lists.gnu.org/archive/html/coreutils/2011-06/msg00082.html
>
>thanks,
>Pádraig.

Pádraig,


I would like to join the debate.  Would you entertain that, or is the
issue settled.

If I wrote the code, would you include it?


Jeff

>

Message #14 received at 20745 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: "Silverman, Jeffrey X. -ND" <Jeffrey.X.Silverman.-ND <at> disney.com>,
 Pádraig Brady <P <at> draigBrady.com>,
 "20745 <at> debbugs.gnu.org" <20745 <at> debbugs.gnu.org>
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Fri, 05 Jun 2015 13:57:33 -0600

[Message part 1 (text/plain, inline)]

On 06/05/2015 01:35 PM, Silverman, Jeffrey X. -ND wrote:
>> This was previously discussed, and while has merit
>> at the time it was thought not important enough to add:
>>
>>  http://www.gnu.org/software/coreutils/rejected_requests.html
>>    http://lists.gnu.org/archive/html/coreutils/2011-06/msg00082.html
>>

> 
> I would like to join the debate.  Would you entertain that, or is the
> issue settled.
> 
> If I wrote the code, would you include it?

Patches certainly speak louder than words.  Last time it was discussed
it appears the biggest category of response could probably be
categorized as "I don't need sorted IP addresses, so I won't spend time
writing the patch", rather than "it's a lousy idea that no one should
implement".  So yes, feel free to propose it as a patch, along with
justification on why people want sorted IP addresses.

You'll need to have copyright assignment on file with the FSF before we
can take such a patch, though, as it would probably be non-trivial.
Also, make sure that the patch includes documentation and unit tests.

When comparing IP addresses to be sorted, would you declare that all
IPv4 addresses sort earlier than IPv6?  Would you make the code sort
hostnames based on what IP address a DNS resolution produces for that
name, or would you stick to pure numeric formats?  Make sure the sorting
is consistent based on network ordering (the same address pairing should
sort the same way on both big and little endian machines).  Also, more
than one string can resolve to the same address (for example, ::, ::0,
and even ::0.0.0.0 are the same IPv6 address), so you'll need to
consider how to do tie-breaking sorts of strings that map to the same
address.

-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #17 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Erik Auerswald <auerswal <at> unix-ag.uni-kl.de>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 8 Jun 2015 11:16:37 +0200

Hi,

On Fri, Jun 05, 2015 at 01:57:33PM -0600, Eric Blake wrote:
> On 06/05/2015 01:35 PM, Silverman, Jeffrey X. -ND wrote:
> 
> >> This was previously discussed, and while has merit
> >> at the time it was thought not important enough to add:
> >>
> >>  http://www.gnu.org/software/coreutils/rejected_requests.html
> >>    http://lists.gnu.org/archive/html/coreutils/2011-06/msg00082.html
> > 
> > I would like to join the debate.  Would you entertain that, or is the
> > issue settled.
> > 
> > If I wrote the code, would you include it?
> 
> [...]
> justification on why people want sorted IP addresses.

FWIW I use 'sort' to sort IPv4 addresses in my ping_scan[1] script.

The info documentation for sort provides another example, log files
sorted by IP address and time stamp. That specific example even needs
two runs of sort, because sort lacks built-in support for IP addresses.

While IPv4 addresses are readily sorted by "sort -s -t '.' -k 1,1n -k
2,2n -k 3,3n -k 4,4n", this is not the case for IPv6 addresses. Having
an option for sorting IP addresses that supports both IPv4 and IPv6
seems like a useful addition to me.

Thanks,
Erik

[1] https://www.unix-ag.uni-kl.de/~auerswal/ping_scan/
-- 
Be water, my friend.
                        -- Bruce Lee

Message #20 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephane Chazelas <stephane.chazelas <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 8 Jun 2015 10:51:59 +0100

2015-06-08 11:16:37 +0200, Erik Auerswald:
[...]
> FWIW I use 'sort' to sort IPv4 addresses in my ping_scan[1] script.
> 
> The info documentation for sort provides another example, log files
> sorted by IP address and time stamp. That specific example even needs
> two runs of sort, because sort lacks built-in support for IP addresses.
> 
> While IPv4 addresses are readily sorted by "sort -s -t '.' -k 1,1n -k
> 2,2n -k 3,3n -k 4,4n", this is not the case for IPv6 addresses. Having
> an option for sorting IP addresses that supports both IPv4 and IPv6
> seems like a useful addition to me.
[...]

In the spirit of tools doing one thing and doing it well, it
would make more sense to have a tool that converts an IP address
to something sortable and use that instead in combination with
sort.

I'm not even sure having a tool just for that specific task
would make sense though. Here, it sounds more like a job for a
high level language like perl/python... (what if I want to sort
on roman numerals now, week day names, astrological signs...)

for instance, here using yash syntax (you can use named pipes or
possibly coprocs with some other shells):

ip2hex() {
  perl -MSocket=:all -nle '
    print unpack "(H2)*", inet_pton(/:/?AF_INET6:AF_INET, $_)'
}

mysort() {
  (
    exec 3>>|4
    tee /dev/fd/3 |
      cut -f1 3>&- | ip2hex 3>&- |
      paste - /dev/fd/4 3>&-
  ) | sort | cut -f2-
}

mysort << EOF
127.0.0.1       blah
6.6.6.6         foo
::1             bar
EOF

That's still quite awkward. A shame that piping capabilities in
shells don't extend to  more  complex scenarii where the output
of some command can be piped to two others the output of which
can be merged back easily.

named pipes can be used for that, but cleaning up and
restricting access to them makes their usage quite messy.

Of course, the whole thing can be done with perl.

-- 
Stephane

Message #23 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Erik Auerswald <auerswal <at> unix-ag.uni-kl.de>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 8 Jun 2015 12:43:32 +0200

Hi,

On Mon, Jun 08, 2015 at 10:51:59AM +0100, Stephane Chazelas wrote:
> 2015-06-08 11:16:37 +0200, Erik Auerswald:
> [...]
> > FWIW I use 'sort' to sort IPv4 addresses in my ping_scan[1] script.
> > 
> > The info documentation for sort provides another example, log files
> > sorted by IP address and time stamp. That specific example even needs
> > two runs of sort, because sort lacks built-in support for IP addresses.
> > 
> > While IPv4 addresses are readily sorted by "sort -s -t '.' -k 1,1n -k
> > 2,2n -k 3,3n -k 4,4n", this is not the case for IPv6 addresses. Having
> > an option for sorting IP addresses that supports both IPv4 and IPv6
> > seems like a useful addition to me.
> [...]
> 
> I'm not even sure having a tool just for that specific task
> would make sense though. Here, it sounds more like a job for a
> high level language like perl/python... (what if I want to sort
> on roman numerals now, week day names, astrological signs...)

Well, IP addresses are often encountered on Internet connected computers.
;-)

> for instance, here using yash syntax (you can use named pipes or
> possibly coprocs with some other shells):
> 
> ip2hex() {
>   perl -MSocket=:all -nle '
>     print unpack "(H2)*", inet_pton(/:/?AF_INET6:AF_INET, $_)'
> }
> 
> mysort() {
>   (
>     exec 3>>|4
>     tee /dev/fd/3 |
>       cut -f1 3>&- | ip2hex 3>&- |
>       paste - /dev/fd/4 3>&-
>   ) | sort | cut -f2-
> }
> 
> mysort << EOF
> 127.0.0.1       blah
> 6.6.6.6         foo
> ::1             bar
> EOF
> 
> That's still quite awkward. A shame that piping capabilities in
> shells don't extend to  more  complex scenarii where the output
> of some command can be piped to two others the output of which
> can be merged back easily.
> 
> named pipes can be used for that, but cleaning up and
> restricting access to them makes their usage quite messy.
> 
> Of course, the whole thing can be done with perl.

I'd say the above is a very good reason for implementing the asked for
feature in sort.

Thanks,
Erik
-- 
Design your product to please the users.
                        -- Paul Graham

Message #26 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephane Chazelas <stephane.chazelas <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 8 Jun 2015 12:14:07 +0100

2015-06-08 12:43:32 +0200, Erik Auerswald:
[...]
> > I'm not even sure having a tool just for that specific task
> > would make sense though. Here, it sounds more like a job for a
> > high level language like perl/python... (what if I want to sort
> > on roman numerals now, week day names, astrological signs...)
> 
> Well, IP addresses are often encountered on Internet connected computers.
> ;-)

And roman numerals are common in bibliography processing.

The "sort" command has nothing to do with "IP addresses".

OK, there's already a -M for month sorting (locale dependant)
useful in log sorting. There's already a -h and -V.

Those are still quite generic. But IPv4 and IPv6 addresses are
very specific (and the right sort order not necessarily obvious
(IPv4 vs v6, link-local addresses, different IPv4 notations...))

Maybe there's a way to allow that without having to implement
the specifics in sort.

Like sort key flags to invoke commands:

   sort '-k1,1|ip2hex' '-k2,2n|roman2int' '-k3,3|iconv -t us//TRANSLIT'

For sort to invoke those ip2hex/roman2int/iconv shell
command lines to pre-process the sort keys before sorting.

-- 
Stephane

Message #29 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephane Chazelas <stephane.chazelas <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 8 Jun 2015 13:13:54 +0100

2015-06-08 11:16:37 +0200, Erik Auerswald:
[...]
> FWIW I use 'sort' to sort IPv4 addresses in my ping_scan[1] script.
> 
> The info documentation for sort provides another example, log files
> sorted by IP address and time stamp. That specific example even needs
> two runs of sort, because sort lacks built-in support for IP addresses.
> 
> While IPv4 addresses are readily sorted by "sort -s -t '.' -k 1,1n -k
> 2,2n -k 3,3n -k 4,4n", this is not the case for IPv6 addresses.
[...]
> [1] https://www.unix-ag.uni-kl.de/~auerswal/ping_scan/
[...]

Note that IPv4 address in quad-decimal notation can be sorted
with sort -V. Not IPv6 ones.

$ printf '%s\n' 1.2.3.4 1.12.3.4 a:2:b a:1a:b | sort -V
1.2.3.4
1.12.3.4
a:1a:b
a:2:b

IPv6 addresses sort lexically when fully expanded (as in
0000:0000:0000:0000:0000:0000:0000:0001 instead of ::1)

and IPv4 addresses sort lexically as well when in hex notation
(0x7f000001) instead of quad-decimal, or when using the
127.000.000.001 notation (though that one conflicts with the
traditional parsing (inet_addr/gethostbyname) as octal
(010.000.000.001 is traditionaly 8.0.0.1)).

So one can always pre-process the data to convert the IP
addresses in those alternative formats.


-- 
Stephane

Message #32 received at 20745 <at> debbugs.gnu.org (full text, mbox):

From: Eric Blake <eblake <at> redhat.com>
To: Stephane Chazelas <stephane.chazelas <at> gmail.com>, 20745 <at> debbugs.gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 08 Jun 2015 08:14:20 -0600

[Message part 1 (text/plain, inline)]

On 06/08/2015 05:14 AM, Stephane Chazelas wrote:

> Maybe there's a way to allow that without having to implement
> the specifics in sort.
> 
> Like sort key flags to invoke commands:
> 
>    sort '-k1,1|ip2hex' '-k2,2n|roman2int' '-k3,3|iconv -t us//TRANSLIT'

That would result in a LOT of process overhead (one command spawned per
sort key per line).  True, it would be more generic, but the performance
would suffer compared to having a direct way to do the sorting within
the current process.  It's going to be more efficient to do a single
pre-process pass, then sort, then post-process, with a complicated
script language doing the right processing per field in a single pass,
than to do one conversion per key per row.


-- 
Eric Blake   eblake redhat com    +1-919-301-3266
Libvirt virtualization library http://libvirt.org

[signature.asc (application/pgp-signature, attachment)]

Message #35 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Stephane Chazelas <stephane.chazelas <at> gmail.com>
To: bug-coreutils <at> gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 8 Jun 2015 15:56:44 +0100

2015-06-08 08:14:20 -0600, Eric Blake:
> On 06/08/2015 05:14 AM, Stephane Chazelas wrote:
> 
> > Maybe there's a way to allow that without having to implement
> > the specifics in sort.
> > 
> > Like sort key flags to invoke commands:
> > 
> >    sort '-k1,1|ip2hex' '-k2,2n|roman2int' '-k3,3|iconv -t us//TRANSLIT'
> 
> That would result in a LOT of process overhead (one command spawned per
> sort key per line).  True, it would be more generic, but the performance
> would suffer compared to having a direct way to do the sorting within
> the current process.  It's going to be more efficient to do a single
> pre-process pass, then sort, then post-process, with a complicated
> script language doing the right processing per field in a single pass,
> than to do one conversion per key per row.
[...]

I'm not suggesting one process per line. Those are meant to be
filters processing the whole input, so one ip2hex process, one
roman2int... That would have to be done carefully to avoid
deadlocks, but should be efficient if done properly (with proper
buffering).

The idea being to implement the cumbersome plumbing in my
previous yash approach inside sort.

-- 
Stephane

Message #38 received at 20745 <at> debbugs.gnu.org (full text, mbox):

From: Pádraig Brady <P <at> draigBrady.com>
To: Stephane Chazelas <stephane.chazelas <at> gmail.com>, 
 20745 <at> debbugs.gnu.org
Subject: Re: bug#20745: I would like to make a request for the sort command
Date: Mon, 08 Jun 2015 16:38:42 +0100

On 08/06/15 10:51, Stephane Chazelas wrote:
> 2015-06-08 11:16:37 +0200, Erik Auerswald:
> [...]
>> FWIW I use 'sort' to sort IPv4 addresses in my ping_scan[1] script.
>>
>> The info documentation for sort provides another example, log files
>> sorted by IP address and time stamp. That specific example even needs
>> two runs of sort, because sort lacks built-in support for IP addresses.
>>
>> While IPv4 addresses are readily sorted by "sort -s -t '.' -k 1,1n -k
>> 2,2n -k 3,3n -k 4,4n", this is not the case for IPv6 addresses. Having
>> an option for sorting IP addresses that supports both IPv4 and IPv6
>> seems like a useful addition to me.
> [...]
> 
> In the spirit of tools doing one thing and doing it well, it
> would make more sense to have a tool that converts an IP address
> to something sortable and use that instead in combination with
> sort.
> 
> I'm not even sure having a tool just for that specific task
> would make sense though. Here, it sounds more like a job for a
> high level language like perl/python... (what if I want to sort
> on roman numerals now, week day names, astrological signs...)
> 
> for instance, here using yash syntax (you can use named pipes or
> possibly coprocs with some other shells):
> 
> ip2hex() {
>   perl -MSocket=:all -nle '
>     print unpack "(H2)*", inet_pton(/:/?AF_INET6:AF_INET, $_)'
> }
> 
> mysort() {
>   (
>     exec 3>>|4
>     tee /dev/fd/3 |
>       cut -f1 3>&- | ip2hex 3>&- |
>       paste - /dev/fd/4 3>&-
>   ) | sort | cut -f2-
> }
> 
> mysort << EOF
> 127.0.0.1       blah
> 6.6.6.6         foo
> ::1             bar
> EOF
> 
> That's still quite awkward. A shame that piping capabilities in
> shells don't extend to  more  complex scenarii where the output
> of some command can be piped to two others the output of which
> can be merged back easily.
> 
> named pipes can be used for that, but cleaning up and
> restricting access to them makes their usage quite messy.
> 
> Of course, the whole thing can be done with perl.

This is a useful example.
We're essentially talking about generalizing the
Decorate Sort Undecorate pattern here, which can be broken down to:

  Decorate
    parse
    process
    insert
  Sort
    parse
    sort
  Undecorate
    parse
    remove

Your example above, generalizes the Decorate and Undecorate
parts using shell constructs, and you had a further suggestion
for pulling those internally to sort like:

  sort '-k1,1|ip2hex' '-k2,2n|roman2int' '-k3,3|iconv -t us//TRANSLIT'

Note we have similar kind of sub processing support in split for example:

  seq 10 20 | split -nr/$(nproc) --filter='rev'

If doing within sort(1), we'd have to read as well as write to the pipe,
and also for performance these filters would be used to process the input
before passing to the standard sort consumer functions.
Now having this within sort only provides for conciseness rather
than providing a functional advantage.

Another alternative would be to generalise the Decorate portion
of the process, which would be simpler as write only, and also
inherently distributed on multicore with a separate "decorate" process.
Doing this would also be of more general use than just for sort.
So we might have:

  decorate '-k1,1|ip2hex' '-k2,2n|roman2int' '-k3,3|iconv -t us//TRANSLIT'

Generally with "decorate", you would add a column rather than replacing,
though that would be controllable with options.

You could also do this whole decorate processing with something like
http://www.pixelbeat.org/scripts/funcpy or your perl -nle method.
and that would also support correlated operations like filtering.
Though that would require users to know the python/perl or whatever,
so there is some merit I think to having something like the
above decorate command.

thanks,
Pádraig

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.