GNU bug report logs - #10877
[sort] too eager to use temp files

Previous Next

Full log

View this message in rfc822 format

From: Rogier Wolff <R.E.Wolff <at> BitWizard.nl>
To: Paul Eggert <eggert <at> cs.ucla.edu>
Cc: 10877 <at> debbugs.gnu.org, Rogier Wolff <R.E.Wolff <at> BitWizard.nl>
Subject: bug#10877: Wimpy external files.
Date: Sun, 26 Feb 2012 08:14:51 +0100

On Sat, Feb 25, 2012 at 10:41:05AM -0800, Paul Eggert wrote:
> A problem with that idea is, suppose we have many independent 'sort'
> invocations running at at the same time, as part of a shell pipeline
> say?  If they each grab 1/8 of physical RAM, merely because they
> want to sort piped data of a few bytes, they may exhaust swap space.

Two things. 

First: Many modern operating systems do "lazy" allocation. With 8G RAM
+ 8G SWAP you can have hundreds of apps each allocating a gigabyte, as
long as they don't all use that gigabyte. 

Second: If you're afraid for a lone operating system that DOES show
this problem, then a slight change in the codebase might be in order
for "unknown sort size". Then the "1/8th of memory" becomes an upper
bound and the buffer should be dynamically expanded. If you do not
have to "move" the data from the old buffer to a new buffer you can
just allocate a 2x larger buffer each time you reach the limit (but
are below the upper bound). If you DO have the expense of having to 
move all the data up to that point, I would suggest allocating a
4x larger buffer up to the upper bound. 

	Roger. 

-- 
** R.E.Wolff <at> BitWizard.nl ** http://www.BitWizard.nl/ ** +31-15-2600998 **
**    Delftechpark 26 2628 XH  Delft, The Netherlands. KVK: 27239233    **
*-- BitWizard writes Linux device drivers for any device you may have! --*
The plan was simple, like my brother-in-law Phil. But unlike
Phil, this plan just might work.

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.