Hello,
I am using split to split up some large, paired fastq files (nearly 4 billion lines each). I am using the -l flag to split into files of 10 million reads (40 million lines) each and though the fastq files have matched and sorted reads, split is creating different numbers of split files for the two paired fastq files, and the pairing becomes off at some point. The jobs finished without exceeding memory and with an exit status 0, and I noticed the help file said to email this address if there were bugs, so I thought I would mention it.

This is the line I am using to call split on my zipped fastq files:
zcat MH1_R1.fastq.gz | split - -l 40000000 DHT_R1_

zcat MH1_R2.fastq.gz | split - -l 40000000 DHT_R2_

This creates 96 chunks for the R1 and 95 chunks for R2, even though the orignal fastq files have the same number of reads.
Do you have any suggestions for how to proceed? Perhaps zcatting and piping the files is not the best way to call split?

Thanks,

~ Heather

Heather Wick
PhD Candidate, Human Genetics
Labs of Sarah Wheelan and Vasan Yegnasubramanian
Institute of Genetic Medicine
Johns Hopkins University School of Medicine

hwick1@jhmi.edu