Subject: Re: Shepherd: flaky self tests

From: Simon Josefsson <[email-masked]>
Date: Sun, 06 Jul 2025 22:10:33 +0200

Ludovic Courtès <[email-masked]> writes:

> Also, I wonder if the problem could be a race condition, which would be
> fixed like this:
>
> diff --git a/tests/services/log-rotation.sh b/tests/services/log-rotation.sh
> index 375f563..178bb49 100644
> --- a/tests/services/log-rotation.sh
> +++ b/tests/services/log-rotation.sh
> @@ -182,6 +182,6 @@ $herd trigger log-rotation
>  until test -f "$service_log3.2.gz"; do sleep 0.2; done
>  $herd status three
>  $herd stop three
> -test $(file_descriptor_count) -le $initial_fd_count
> +until test $(file_descriptor_count) -le $initial_fd_count; do sleep 0.3; done
>  
>  $herd stop log-rotation
>
>
> Could you also give that a try?

That just infloops and times out after 3 minutes...

/Simon
FAIL: tests/services/log-rotation
=================================

+ shepherd --version
shepherd (GNU Shepherd) 1.0.5
Copyright (C) 2025 the Shepherd authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ herd --version
herd (GNU Shepherd) 1.0.5
Copyright (C) 2025 the Shepherd authors
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
+ socket=t-socket-147839
+ conf=t-conf-147839
+ log=t-log-147839
+ pid=t-pid-147839
+ service_log1=/build/shepherd-1.0.5/t-service-log1-147839
+ service_log2=/build/shepherd-1.0.5/t-service-log2-147839
+ service_log3=/build/shepherd-1.0.5/t-service-log3-147839
+ external_log=/build/shepherd-1.0.5/t-service-extlog-147839
+ invalid_external_log=/build/shepherd-1.0.5/t-service-invalid-extlog-147839
+ herd='herd -s t-socket-147839'
+ trap 'zcat t-log-147839.* || true; cat t-log-147839 || true;
      rm -f t-socket-147839 t-conf-147839 t-log-147839* /build/shepherd-1.0.5/t-service-log1-147839* /build/shepherd-1.0.5/t-service-log2-147839* /build/shepherd-1.0.5/t-service-log3-147839* /build/shepherd-1.0.5/t-service-extlog-147839*;
      rmdir /build/shepherd-1.0.5/t-service-invalid-extlog-147839;
      test -f t-pid-147839 && kill `cat t-pid-147839` || true; rm -f t-pid-147839' EXIT
+ cat
+ rm -f t-pid-147839
+ test -f t-pid-147839
+ sleep 0.3
+ shepherd -I -s t-socket-147839 -c t-conf-147839 -l t-log-147839 --pid=t-pid-147839
GNU Shepherd 1.0.5 (Guile 3.0.10, x86_64-pc-linux-gnu)
Starting service root...
Service root started.
Service root running with value #<<process> id: 148599 command: #f>.
Service root has been started.
Configuration successfully loaded from 't-conf-147839'.
+ test -f t-pid-147839
++ cat t-pid-147839
+ shepherd_pid=148599
+ herd -s t-socket-147839 start one
Starting service one...
Service one has been started.
+ herd -s t-socket-147839 start two
Starting service two...
Service two has been started.
+ herd -s t-socket-147839 start log-rotation
Starting service log-rotation...
Service log-rotation started.
Service log-rotation running with value #<timer #<<calendar-event> seconds: (0) minutes: (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59) hours: (0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23) days-of-month: (1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31) months: (6) days-of-week: (0 1 2 3 4 5 6)> #<procedure rotation ()> 7fb8e5934dc0>.
Service log-rotation has been started.
+ sleep 0.5
+ for file in "$service_log1" "$service_log2" "$external_log" "$invalid_external_log" "$log"
+ herd -s t-socket-147839 files log-rotation
+ grep /build/shepherd-1.0.5/t-service-log1-147839
/build/shepherd-1.0.5/t-service-log1-147839	one
+ for file in "$service_log1" "$service_log2" "$external_log" "$invalid_external_log" "$log"
+ herd -s t-socket-147839 files log-rotation
+ grep /build/shepherd-1.0.5/t-service-log2-147839
/build/shepherd-1.0.5/t-service-log2-147839	two
+ for file in "$service_log1" "$service_log2" "$external_log" "$invalid_external_log" "$log"
+ herd -s t-socket-147839 files log-rotation
+ grep /build/shepherd-1.0.5/t-service-extlog-147839
/build/shepherd-1.0.5/t-service-extlog-147839	(external)
+ for file in "$service_log1" "$service_log2" "$external_log" "$invalid_external_log" "$log"
+ grep /build/shepherd-1.0.5/t-service-invalid-extlog-147839
+ herd -s t-socket-147839 files log-rotation
/build/shepherd-1.0.5/t-service-invalid-extlog-147839	(external)
+ for file in "$service_log1" "$service_log2" "$external_log" "$invalid_external_log" "$log"
+ herd -s t-socket-147839 files log-rotation
+ grep t-log-147839
/build/shepherd-1.0.5/t-log-147839	root
+ test -f /build/shepherd-1.0.5/t-service-log1-147839
+ test -f /build/shepherd-1.0.5/t-service-log2-147839
+ echo 'This is an external log file.'
+ mkdir /build/shepherd-1.0.5/t-service-invalid-extlog-147839
+ herd -s t-socket-147839 trigger log-rotation
Triggering timer.
+ grep 'Rotated ' t-log-147839
+ sleep 0.5
Rotating '/build/shepherd-1.0.5/t-log-147839' to '/build/shepherd-1.0.5/t-log-147839.1'.
+ grep 'Rotated ' t-log-147839
2025-07-06 20:06:31 Rotated '/build/shepherd-1.0.5/t-log-147839'.
2025-07-06 20:06:31 Rotated '/build/shepherd-1.0.5/t-service-extlog-147839'.
+ test -f /build/shepherd-1.0.5/t-service-log1-147839
+ test -f /build/shepherd-1.0.5/t-service-log2-147839
+ test -f t-log-147839
+ test -f /build/shepherd-1.0.5/t-service-log1-147839.1.gz
+ test -f t-log-147839.1.gz
+ test -f /build/shepherd-1.0.5/t-service-log2-147839.1.gz
+ test -f /build/shepherd-1.0.5/t-service-extlog-147839.1.gz
+ gunzip
+ grep 'external log file'
This is an external log file.
+ test -f /build/shepherd-1.0.5/t-service-extlog-147839
+ guile -c '(exit (zero? (stat:size (stat "/build/shepherd-1.0.5/t-service-extlog-147839"))))'
+ zgrep 'Not rotating .*/build/shepherd-1.0.5/t-service-invalid-extlog-147839' t-log-147839 t-log-147839.1.gz
t-log-147839:2025-07-06 20:06:31 Not rotating '/build/shepherd-1.0.5/t-service-invalid-extlog-147839', which is not a regular file.
+ herd -s t-socket-147839 trigger log-rotation
Triggering timer.
+ test -f /build/shepherd-1.0.5/t-service-log1-147839.2.gz
+ test -f /build/shepherd-1.0.5/t-service-log1-147839.1.gz
+ sleep 0.5
Rotating '/build/shepherd-1.0.5/t-log-147839' to '/build/shepherd-1.0.5/t-log-147839.1'.
+ test -f /build/shepherd-1.0.5/t-service-log1-147839.1.gz
+ test -f t-log-147839.2.gz
+ test -f /build/shepherd-1.0.5/t-service-log1-147839
+ test -f /build/shepherd-1.0.5/t-service-log2-147839.1.gz
+ touch -d 2017-10-01 /build/shepherd-1.0.5/t-service-log1-147839.2.gz
+ test -s /build/shepherd-1.0.5/t-service-log1-147839
+ herd -s t-socket-147839 trigger log-rotation
Triggering timer.
+ zgrep 'Deleting .*/build/shepherd-1.0.5/t-service-log1-147839.2.gz' t-log-147839 t-log-147839.1.gz t-log-147839.2.gz
t-log-147839:2025-07-06 20:06:32 Deleting old log file '/build/shepherd-1.0.5/t-service-log1-147839.2.gz'.
Rotating '/build/shepherd-1.0.5/t-log-147839' to '/build/shepherd-1.0.5/t-log-147839.1'.
+ test -f /build/shepherd-1.0.5/t-service-log1-147839.2.gz
+ test -f /build/shepherd-1.0.5/t-service-log1-147839.1.gz
+ test -f /build/shepherd-1.0.5/t-service-log1-147839.3.gz
++ file_descriptor_count
++ wc -l
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/15 /proc/148599/fd/16 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
+ initial_fd_count=19
+ herd -s t-socket-147839 start three
Service three has been started.
+ herd -s t-socket-147839 trigger three
Triggering timer.
+ test -f /build/shepherd-1.0.5/t-service-log3-147839
+ herd -s t-socket-147839 status three
+ grep 'Child process'
+ herd -s t-socket-147839 status three
+ grep 'Timer triggered from /build/shepherd-1.0.5'
  2025-07-06 20:06:32 Timer triggered from /build/shepherd-1.0.5.
+ herd -s t-socket-147839 files log-rotation
+ grep /build/shepherd-1.0.5/t-service-log3-147839
/build/shepherd-1.0.5/t-service-log3-147839	three
+ herd -s t-socket-147839 trigger log-rotation
Rotating '/build/shepherd-1.0.5/t-service-log3-147839' to '/build/shepherd-1.0.5/t-service-log3-147839.1'.
Triggering timer.
+ test -f /build/shepherd-1.0.5/t-service-log3-147839.1.gz
+ sleep 0.2
Rotating '/build/shepherd-1.0.5/t-log-147839' to '/build/shepherd-1.0.5/t-log-147839.1'.
+ test -f /build/shepherd-1.0.5/t-service-log3-147839.1.gz
+ herd -s t-socket-147839 status three
Status of three:
  It is running since 20:06:32 (0 seconds ago).
  Timed service.
  Periodically running: sh -c "echo Timer triggered from $PWD."
  It is enabled.
  Provides: three
  Custom action: trigger
  Will not be respawned.
  Log file: /build/shepherd-1.0.5/t-service-log3-147839

Recent runs:
  2025-07-06 20:06:32 Process exited successfully after 0 seconds.

Recent messages (use '-n' to view more or less):
  2025-07-06 20:06:32 Timer triggered from /build/shepherd-1.0.5.

Upcoming timer alarms:
  Tue Jan  6 20:07:00 2026 (in 184 days)
  Tue Jan  6 20:08:00 2026 (in 184 days)
  Tue Jan  6 20:09:00 2026 (in 184 days)
  Tue Jan  6 20:10:00 2026 (in 184 days)
  Tue Jan  6 20:11:00 2026 (in 184 days)
+ herd -s t-socket-147839 status three
+ grep 'Timer triggered from /build/shepherd-1.0.5'
  2025-07-06 20:06:32 Timer triggered from /build/shepherd-1.0.5.
+ herd -s t-socket-147839 trigger three
Triggering timer.
+ herd -s t-socket-147839 trigger log-rotation
Rotating '/build/shepherd-1.0.5/t-service-log3-147839' to '/build/shepherd-1.0.5/t-service-log3-147839.1'.
Triggering timer.
+ test -f /build/shepherd-1.0.5/t-service-log3-147839.2.gz
+ herd -s t-socket-147839 status three
Rotating '/build/shepherd-1.0.5/t-log-147839' to '/build/shepherd-1.0.5/t-log-147839.1'.
Status of three:
  It is running since 20:06:32 (1 second ago).
  Timed service.
  Periodically running: sh -c "echo Timer triggered from $PWD."
  It is enabled.
  Provides: three
  Custom action: trigger
  Will not be respawned.
  Log file: /build/shepherd-1.0.5/t-service-log3-147839

Recent runs:
  2025-07-06 20:06:32 Process exited successfully after 0 seconds.
  2025-07-06 20:06:33 Process exited successfully after 0 seconds.

Recent messages (use '-n' to view more or less):
  2025-07-06 20:06:33 Timer triggered from /build/shepherd-1.0.5.

Upcoming timer alarms:
  Tue Jan  6 20:07:00 2026 (in 184 days)
  Tue Jan  6 20:08:00 2026 (in 184 days)
  Tue Jan  6 20:09:00 2026 (in 184 days)
  Tue Jan  6 20:10:00 2026 (in 184 days)
  Tue Jan  6 20:11:00 2026 (in 184 days)
+ herd -s t-socket-147839 stop three
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ wc -l
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ wc -l
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ wc -l
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
++ file_descriptor_count
++ ls -l /proc/148599/fd/0 /proc/148599/fd/1 /proc/148599/fd/10 /proc/148599/fd/11 /proc/148599/fd/12 /proc/148599/fd/13 /proc/148599/fd/14 /proc/148599/fd/16 /proc/148599/fd/17 /proc/148599/fd/18 /proc/148599/fd/19 /proc/148599/fd/2 /proc/148599/fd/20 /proc/148599/fd/21 /proc/148599/fd/22 /proc/148599/fd/3 /proc/148599/fd/4 /proc/148599/fd/5 /proc/148599/fd/6 /proc/148599/fd/7 /proc/148599/fd/8 /proc/148599/fd/9
++ wc -l
+ test 22 -le 19
+ sleep 0.3
...