Subject: bug#46048: closed (Re: bug#46048: split -n K/N loses data, sum of output files is smaller than input file.)

From: [email-masked] (GNU bug Tracking System)
Date: Mon, 25 Jan 2021 14:22:02 +0000

Your bug report

#46048: split -n K/N loses data, sum of output files is smaller than input file.

which was filed against the coreutils package, has been closed.

The explanation is attached below, along with your original report.
If you require more details, please reply to [email-masked].

-- 
46048: http://debbugs.gnu.org/cgi/bugreport.cgi?bug=46048
GNU Bug Tracking System
Contact [email-masked] with problems

Subject: Re: bug#46048: split -n K/N loses data, sum of output files is smaller than input file.
From: Pádraig Brady <[email-masked]>
To: Paul Eggert <[email-masked]>
Cc: Paul Hirst <[email-masked]>, [email-masked]
Date: Mon, 25 Jan 2021 14:21:35 +0000
List-Id: <debbugs-submit.debbugs.gnu.org>

On 24/01/2021 19:55, Paul Eggert wrote:
> On 1/24/21 8:52 AM, Pádraig Brady wrote:
>> -      if (lseek (STDIN_FILENO, start, SEEK_CUR) < 0)
>> +      if (lseek (STDIN_FILENO, start, SEEK_SET) < 0)
> 
> Dumb question: will this handle the case where you're splitting from
> stdin and stdin is a seekable file and its initial file offset is nonzero?

Right. Following on the logic from input_file_size(),
I'm going with the attached, which I'll push later.
Marking this as done.

thanks,
Pádraig
>From 8741d726327bddce3271de23af4aae4cfc185774 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?P=C3=A1draig=20Brady?= <[email-masked]>
Date: Mon, 25 Jan 2021 14:12:48 +0000
Subject: [PATCH] split: fix --number=K/N to output correct part of file

This functionality regressed with the adjustments
in commit v8.25-4-g62e7af032

* src/split.c (bytes_chunk_extract): Account for already read data
when seeking into the file.
* tests/split/b-chunk.sh: Use the hidden ---io-blksize option,
to test this functionality.
* NEWS: Mention the bug fix.
Fixes https://bugs.gnu.org/46048
---
 NEWS                   |  4 ++++
 src/split.c            |  2 +-
 tests/split/b-chunk.sh | 45 ++++++++++++++++++++++++------------------
 3 files changed, 31 insertions(+), 20 deletions(-)

diff --git a/NEWS b/NEWS
index c2474fee3..e7fbde8ed 100644
--- a/NEWS
+++ b/NEWS
@@ -27,6 +27,10 @@ GNU coreutils NEWS                                    -*- outline -*-
   rm no longer skips an extra file when the removal of an empty directory fails.
   [bug introduced by the rewrite to use fts in coreutils-8.0]
 
+  split --number=K/N will again correctly split chunk K of N to stdout.
+  Previously a chunk starting after 128KiB, output the wrong part of the file.
+  [bug introduced in coreutils-8.26]
+
   tr no longer crashes when using --complement with certain
   invalid combinations of case character classes.
   [bug introduced in coreutils-8.6]
diff --git a/src/split.c b/src/split.c
index 0660da13f..59c234c12 100644
--- a/src/split.c
+++ b/src/split.c
@@ -1001,7 +1001,7 @@ bytes_chunk_extract (uintmax_t k, uintmax_t n, char *buf, size_t bufsize,
     }
   else
     {
-      if (lseek (STDIN_FILENO, start, SEEK_CUR) < 0)
+      if (lseek (STDIN_FILENO, start - initial_read, SEEK_CUR) < 0)
         die (EXIT_FAILURE, errno, "%s", quotef (infile));
       initial_read = SIZE_MAX;
     }
diff --git a/tests/split/b-chunk.sh b/tests/split/b-chunk.sh
index 8238dcb6d..dbed681f7 100755
--- a/tests/split/b-chunk.sh
+++ b/tests/split/b-chunk.sh
@@ -35,32 +35,39 @@ split -e -n 10 /dev/null || fail=1
 returns_ 1 stat x?? 2>/dev/null || fail=1
 
 printf '1\n2\n3\n4\n5\n' > input || framework_failure_
+printf '1\n2' > exp-1 || framework_failure_
+printf '\n3\n' > exp-2 || framework_failure_
+printf '4\n5\n' > exp-3 || framework_failure_
 
 for file in input /proc/version /sys/kernel/profiling; do
   test -f $file || continue
 
-  split -n 3 $file > out || fail=1
-  split -n 1/3 $file > b1 || fail=1
-  split -n 2/3 $file > b2 || fail=1
-  split -n 3/3 $file > b3 || fail=1
+  for blksize in 1 2 4096; do
+    if ! test "$file" = 'input'; then
+      # For /proc like files we must be able to read all
+      # into the internal buffer to be able to determine size.
+      test "$blksize" = 4096 || continue
+    fi
 
-  case $file in
-    input)
-      printf '1\n2' > exp-1
-      printf '\n3\n' > exp-2
-      printf '4\n5\n' > exp-3
+    split -n 3 ---io-blksize=$blksize $file > out || fail=1
+    split -n 1/3 ---io-blksize=$blksize $file > b1 || fail=1
+    split -n 2/3 ---io-blksize=$blksize $file > b2 || fail=1
+    split -n 3/3 ---io-blksize=$blksize $file > b3 || fail=1
 
-      compare exp-1 xaa || fail=1
-      compare exp-2 xab || fail=1
-      compare exp-3 xac || fail=1
-      ;;
-  esac
+    case $file in
+      input)
+        compare exp-1 xaa || fail=1
+        compare exp-2 xab || fail=1
+        compare exp-3 xac || fail=1
+        ;;
+    esac
 
-  compare xaa b1 || fail=1
-  compare xab b2 || fail=1
-  compare xac b3 || fail=1
-  cat xaa xab xac | compare - $file || fail=1
-  test -f xad && fail=1
+    compare xaa b1 || fail=1
+    compare xab b2 || fail=1
+    compare xac b3 || fail=1
+    cat xaa xab xac | compare - $file || fail=1
+    test -f xad && fail=1
+  done
 done
 
 Exit $fail
-- 
2.26.2

Subject: split -n K/N loses data, sum of output files is smaller than input file.
From: Paul Hirst <[email-masked]>
To: [email-masked]
Date: Fri, 22 Jan 2021 18:58:03 -1000
List-Id: <debbugs-submit.debbugs.gnu.org>

split --number K/N appears to lose data in, with the sum of the sizes of
the output files being smaller than the original input file by 131072 bytes.

$ split --version
split (GNU coreutils) 8.30
...

$ head -c 1000000 < /dev/urandom > test.dat
$ split --number=1/4 test.dat > t1
$ split --number=2/4 test.dat > t2
$ split --number=3/4 test.dat > t3
$ split --number=4/4 test.dat > t4

$ ls -l
-rw-r--r-- 1 user user  250000 Jan 22 18:36 t1
-rw-r--r-- 1 user user  250000 Jan 22 18:36 t2
-rw-r--r-- 1 user user  250000 Jan 22 18:36 t3
-rw-r--r-- 1 user user  118928 Jan 22 18:36 t4
-rw-r--r-- 1 user user 1000000 Jan 22 18:33 test.dat

Surely this should not be the case?

Paul