Subject: Re: bug#51787: GC takes more than 9 hours on berlin

From: Ricardo Wurmus <[email-masked]>
Date: Fri, 17 Dec 2021 15:08:23 +0100

Ricardo Wurmus <[email-masked]> writes:

> I’ll run this on a few of the build nodes to get some more comparisons.

I ran “guix deploy” for hydra-guix-107, a node that was bought at the
same time as the one that is now the head node of ci.guix.gnu.org.  I
copied over a current Guix and installed “fio”
(/gnu/store/qs9cyy5s95n2fbjmxs48iccqvsvj6wxr-fio-3.28/bin/fio) there.

Here’s the output:

--8<---------------cut here---------------start------------->8---
root@hydra-guix-107 ~# fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test --filename=test --bs=4k --iodepth=64 --size=4G --readwrite=randrw --rwmixread=75
test: (g=0): rw=randrw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64
fio-3.28
Starting 1 process
test: Laying out IO file (1 file / 4096MiB)
Jobs: 1 (f=1): [m(1)][100.0%][r=172MiB/s,w=56.9MiB/s][r=44.2k,w=14.6k IOPS][eta 00m:00s]
test: (groupid=0, jobs=1): err= 0: pid=45365: Fri Dec 17 14:50:41 2021
  read: IOPS=42.9k, BW=167MiB/s (176MB/s)(3070MiB/18331msec)
   bw (  KiB/s): min=21208, max=199928, per=100.00%, avg=171621.56, stdev=46624.05, samples=36
   iops        : min= 5302, max=49982, avg=42905.39, stdev=11656.01, samples=36
  write: IOPS=14.3k, BW=56.0MiB/s (58.7MB/s)(1026MiB/18331msec); 0 zone resets
   bw (  KiB/s): min= 7424, max=66720, per=100.00%, avg=57364.22, stdev=15612.42, samples=36
   iops        : min= 1856, max=16680, avg=14341.06, stdev=3903.10, samples=36
  cpu          : usr=6.27%, sys=24.78%, ctx=121626, majf=0, minf=11
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0%
     issued rwts: total=785920,262656,0,0 short=0,0,0,0 dropped=0,0,0,0
     latency   : target=0, window=0, percentile=100.00%, depth=64

Run status group 0 (all jobs):
   READ: bw=167MiB/s (176MB/s), 167MiB/s-167MiB/s (176MB/s-176MB/s), io=3070MiB (3219MB), run=18331-18331msec
  WRITE: bw=56.0MiB/s (58.7MB/s), 56.0MiB/s-56.0MiB/s (58.7MB/s-58.7MB/s), io=1026MiB (1076MB), run=18331-18331msec

Disk stats (read/write):
  sda: ios=779469/260494, merge=0/6, ticks=932265/195191, in_queue=1127456, util=99.50%
--8<---------------cut here---------------end--------------->8---

Here sda is RAID of two SSDs:

--8<---------------cut here---------------start------------->8---
racadm>>storage get pdisks -o
Disk.Bay.0:Enclosure.Internal.0-1:RAID.Integrated.1-1
   Status                           = Ok                                       
   DeviceDescription                = Disk 0 in Backplane 1 of Integrated RAID Controller 1
   RollupStatus                     = Ok                                       
   Name                             = Solid State Disk 0:1:0                   
   State                            = Online                                   
   OperationState                   = Not Applicable                           
   PowerStatus                      = On                                       
   Size                             = 223.001 GB                               
   FailurePredicted                 = NO                                       
   RemainingRatedWriteEndurance     = 99 %                                     
   SecurityStatus                   = Not Capable                              
   BusProtocol                      = SATA                                     
   MediaType                        = SSD                                      
   UsedRaidDiskSpace                = 223.001 GB                               
   AvailableRaidDiskSpace           = 0.001 GB                                 
   Hotspare                         = NO                                       
   Manufacturer                     = INTEL                                    
   ProductId                        = SSDSC2KG240G8R                           
   Revision                         = XCV1DL67                                 
   SerialNumber                     = BTYG91520CHD240AGN                       
   PartNumber                       = CN0T1WH8PESIT95302LTA01                  
   NegotiatedSpeed                  = 6.0 Gb/s                                 
   ManufacturedDay                  = 0                                        
   ManufacturedWeek                 = 0                                        
   ManufacturedYear                 = 0                                        
   ForeignKeyIdentifier             = null                                     
   SasAddress                       = 0x4433221104000000                       
   WWN                              = 0x4433221104000000                       
   FormFactor                       = 2.5 Inch                                 
   RaidNominalMediumRotationRate    = 1                                        
   T10PICapability                  = Not Capable                              
   BlockSizeInBytes                 = 512                                      
   MaxCapableSpeed                  = 6 Gb/s                                   
   RaidType                         = Unknown                                  
   SystemEraseCapability            = CryptographicErasePD                     
   SelfEncryptingDriveCapability    = Not Capable                              
   EncryptionCapability             = Not Capable                              
   CryptographicEraseCapability     = Capable                                  
   Certified                        = Yes                                      
   NonRAIDDiskCachePolicy           = Not Applicable                           
   EncryptionProtocol               = None                                     
Disk.Bay.1:Enclosure.Internal.0-1:RAID.Integrated.1-1
   Status                           = Ok                                       
   DeviceDescription                = Disk 1 in Backplane 1 of Integrated RAID Controller 1
   RollupStatus                     = Ok                                       
   Name                             = Solid State Disk 0:1:1                   
   State                            = Online                                   
   OperationState                   = Not Applicable                           
   PowerStatus                      = On                                       
   Size                             = 223.001 GB                               
   FailurePredicted                 = NO                                       
   RemainingRatedWriteEndurance     = 99 %                                     
   SecurityStatus                   = Not Capable                              
   BusProtocol                      = SATA                                     
   MediaType                        = SSD                                      
   UsedRaidDiskSpace                = 223.001 GB                               
   AvailableRaidDiskSpace           = 0.001 GB                                 
   Hotspare                         = NO                                       
   Manufacturer                     = INTEL                                    
   ProductId                        = SSDSC2KG240G8R                           
   Revision                         = XCV1DL67                                 
   SerialNumber                     = BTYG915502D9240AGN                       
   PartNumber                       = CN0T1WH8PESIT95303BSA01                  
   NegotiatedSpeed                  = 6.0 Gb/s                                 
   ManufacturedDay                  = 0                                        
   ManufacturedWeek                 = 0                                        
   ManufacturedYear                 = 0                                        
   ForeignKeyIdentifier             = null                                     
   SasAddress                       = 0x4433221100000000                       
   WWN                              = 0x4433221100000000                       
   FormFactor                       = 2.5 Inch                                 
   RaidNominalMediumRotationRate    = 1                                        
   T10PICapability                  = Not Capable                              
   BlockSizeInBytes                 = 512                                      
   MaxCapableSpeed                  = 6 Gb/s                                   
   RaidType                         = Unknown                                  
   SystemEraseCapability            = CryptographicErasePD                     
   SelfEncryptingDriveCapability    = Not Capable                              
   EncryptionCapability             = Not Capable                              
   CryptographicEraseCapability     = Capable                                  
   Certified                        = Yes                                      
   NonRAIDDiskCachePolicy           = Not Applicable                           
   EncryptionProtocol               = None                                     
--8<---------------cut here---------------end--------------->8---

-- 
Ricardo