GNU bug report logs - #22108
diff wrapper script for very large files, low memory

Previous Next

Full log

Message #5 received at submit <at> debbugs.gnu.org (full text, mbox):

From: Taco van Dijk <taco <at> waag.org>
To: bug-diffutils <at> gnu.org
Subject: diff wrapper script for very large files, low memory
Date: Mon, 7 Dec 2015 12:45:50 +0100 (CET)

Hi,

For our current project we faced the following problem;
When trying to compare two large files (2* 4+ Gb) exceeding the RAM of the machine, 
the machine would become unresponsive.

To solve this problem we have found a solution that might be worthwhile sharing, based around xxhash.

For anyone interested, you can find it here.

https://github.com/waagsociety/hashed-diff

Kind regards,

Taco van Dijk & Lodewijk Loos
Waag Society

-- 
PGP: 82EDF574 

Previous Next

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997,2003 nCipher Corporation Ltd, 1994-97 Ian Jackson.