| Age | Commit message (Collapse) | Author |
|---|
|
* src/base64.c (main): Use the higher level xnumtoumax()
rather than xstrtoumax(), which is simpler and improves
validation of input. Also pass the _empty_ rather than NULL
string as the suffixes parameter so that invalid trailing
characters are not allowed. For example -w08 is now
flagged as an error, rather than being interpreted as 0.
A subsequent commit will further improve verification
of numbers with leading zeros by dropping backwards compatibility
wrt auto parsing oct and hex numbers.
* tests/misc/base64.pl: Add tests for invalid wrap values.
|
|
Suggested in https://bugzilla.redhat.com/1250113
* AUTHORS: Add base32.
* THANKS.in: Add suggester.
* README: Reference the new program.
* NEWS: Mention the new program.
* src/.gitignore: Ignore the new binary.
* bootstrap.conf: Reference the gnulib base32 module.
* build-aux/gen-lists-of-programs.sh: Add base32.
* man/base32.x: A new template.
* man/.gitignore: Ignore the new man page.
* man/local.mk: Reference the new man page.
* doc/coreutils.texi (base32 invocation): Document the new command.
* src/local.mk: Adjust to build base32 based on base64.c.
* src/base64.c: Parameterize to use the correct headers,
functions and buffer sizes, depending on which binary
is being built.
* tests/misc/base64.pl: Adjust to test both base32 and base64.
* tests/misc/tty-eof.pl: Add base32 as a program that
accepts input on stdin without any options specified.
* scripts/git-hooks/commit-msg: Add base32 to the template.
|
|
Since glibc-2.22, specifically commit [0], the opendir() implementation
implicitly makes an additional stat call thus leading to a FP.
Seen on openSUSE:Tumbleweed since snapshot 20150821.
[0]
https://sourceware.org/git/?p=glibc.git;a=commitdiff;h=46f894d8c60a
* tests/ls/stat-free-color.sh: Change the test to verify that ls(1)
needs the same number of stat-like calls for a single, empty directory
argument as for one with a few directory entries (sub-directory,
regular file, symlink, etc.).
|
|
This was detected in about 25% of runs with gcc -fsanitize=address
ERROR: AddressSanitizer: global-buffer-overflow on address ...
READ of size 4 at 0x000000416628 thread T0
#0 0x40479f in genpattern src/shred.c:782
#1 0x4050d9 in do_wipefd src/shred.c:921
#2 0x406203 in wipefile src/shred.c:1175
#3 0x406b84 in main src/shred.c:1316
#4 0x7f3454a1ef9f in __libc_start_main (/lib64/libc.so.6+0x1ff9f)
#5 0x4025d8 (/tmp/coreutils-8.23/src/shred+0x4025d8)
0x000000416628 is located 56 bytes to the left of
global variable '*.LC49' from 'src/shred.c' (0x416660) of size 17
0x000000416628 is located 12 bytes to the right of
global variable 'patterns' from 'src/shred.c' (0x416540) of size 220
SUMMARY: AddressSanitizer: global-buffer-overflow src/shred.c:782
* src/shred.c (gen_patterns): Restrict pattern selection
to the K available, which regressed due to v5.92-1462-g65533e1.
* tests/misc/shred-passes.sh: Add a deterministic test case.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/20998
|
|
* gnulib: Update to latest, with fixes to tests edge cases.
* tests/init.sh: Update from gnulib.
|
|
Since non interactive shells don't generally set $SHELL,
its value is propagated through the tests and may cause issues;
for example if $SHELL implicitly adjusts $PATH when run.
Instead we set $SHELL to that determined by the posix-shell module,
and use that consistently for all test sub scripts,
including those created thorugh the `split --filter` command.
* tests/local.mk: Explicitly set $SHELL to $(PREFERABLY_POSIX_SHELL)
which defaults to $CONFIG_SHELL and thus usually /bin/sh.
* tests/envvar-check: Remove bash environment variables with
side effects, in case /bin/bash was selected for $SHELL.
* tests/misc/help-version.sh: Remove redundant initialization of $SHELL.
* tests/install/strip-program.sh: Use $SHELL for sub script.
* tests/misc/sort-compress-hang.sh: Likewise.
* tests/misc/sort-compress-proc.sh: Likewise.
* tests/misc/sort-compress.sh: Likewise.
* tests/misc/timeout-group.sh: Likewise.
* tests/rm/fail-eperm.xpl: Remove redundant elision of bash env vars.
* tests/misc/pwd-long.sh: Likewise.
|
|
* tests/du/threshold.sh: Homogenize getopt error messages.
* tests/misc/numfmt.pl: Likewise.
* tests/mv/i-3.sh: Skip on *BSD not just FreeBSD.
|
|
* tests/misc/stty.sh: FreeBSD returns ENOTTY for
the TIOCEXT ioctl, so just avoid this option for now.
|
|
* tests/dd/stats.sh: Wait 20s for dd to write 250MB through a fifo,
rather than 10s for 500MB. The failure was seen often on
a lightly loaded SPARC-Enterprise-T5220 running Solaris 10.
|
|
* tests/misc/sync.sh: Ensure dir is unreadable before
including the permission check.
|
|
The LD_PRELOAD checks by -fsanitize=address are overly strict:
https://groups.google.com/forum/#!topic/address-sanitizer/jEvOJgkDqQk
A workaround is to first export LD_PRELOAD=libasan.so.2
The tests below are adjusted so that workaround is not discarded.
* tests/cp/no-ctx.sh: Append to $LD_PRELOAD.
* tests/df/no-mtab-status.sh: Likewise.
* tests/df/skip-duplicates.sh: Likewise.
* tests/ls/getxattr-speedup.sh: Likewise.
* tests/rm/r-root.sh: Likewise.
* tests/cp/nfs-removal-race.sh: Likewise. Also check that
LD_PRELOAD is effective to aid future maintainability
and avoid false failure if libasan.so.2 is not preloaded.
|
|
* src/factor.c (n_out): A new global variable to track
how much data has been written to stdout.
(print_factors_single): Use n_out to determine whether
to flush the current (and previous) lines.
* tests/misc/factor-parallel.sh: Add a new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
|
|
src/seq.c (scan_arg): Set precision and width _after_ exponentiation.
For example, this will make '1.1e1 12' and '11 1.2e1' equivalent.
One can still set the precision by specifying extra precision on
the start value, or more naturally with a precision on a step value.
* tests/misc/seq-precision.sh: Add new cases.
|
|
* src/seq.c (scan_arg): Set precision to 0 for hex constants
(while avoiding hex floats). This will use then use the
fast path for these arguments. Note we also set the precision
of inf to 0 here, which ensures we use consistent precision
on output where possible.
* tests/misc/seq-precision.sh: Add corresponding test cases.
|
|
* src/seq.c (main): Call seq_fast for infinite last value.
This implicitly avoids format conversion on the
999999 -> 1000000 transition.
* src/seq.c (seq_fast): Generalize the buffer handling,
and adjust to handle the "inf" last value specifics.
* tests/misc/seq-precision.sh: A new test.
* tests/local.mk: Reference the new test.
|
|
* src/numfmt.c (MAX_UNSCALED_DIGITS): Set this to LDBL_DIG
rather than hardcoding at 18 for better portability.
* tests/misc/numfmt.pl: Restrict limit tests to supported platforms.
|
|
* src/numfmt.c (simple_strtod_int): Don't count leading zeros
as significant digits. Also have leading zeros as optional
for floating point numbers.
* tests/misc/numfmt.pl: Add test cases.
* NEWS: Mention the fix.
|
|
Due to existing limits this is usually triggered
with an increased precision. We also add further
restrictions to the output of increased precision numbers.
* src/numfmt.c (simple_round): Avoid intmax_t overflow.
(simple_strtod_int): Count digits consistently
for precision loss and overflow detection.
(prepare_padded_number): Include the precision
when excluding numbers to output, since the precision
determines the ultimate values used in the rounding scheme
in double_to_human().
* tests/misc/numfmt.pl: Add previously failing test cases.
* NEWS: Mention the fix.
|
|
* src/numfmt.c (usage): Update the --format description
to indicate precision is allowed.
(parse_format_string): Parse a precision specification
like the standard printf does.
(double_to_human): Honor the precision in --to mode.
* tests/misc/numfmt.pl: New tests.
* doc/coreutils.texi (numfmt invocation): Mention the new feature.
* NEWS: Likewise.
|
|
* src/numfmt.c: Replace field handling code with logic that understands
field range specifiers. Instead of processing a single field and
printing line prefix/suffix around it, process each field in the line
checking whether it has been included for conversion. If so convert and
print, otherwise just print the unaltered field.
(extract_fields): Removed.
(skip_fields): Removed.
(process_line): Gutted and heavily reworked.
(process_suffixed_number): FIELD is now passed as an arg instead of
using a global.
(parse_field_arg): New function that parses field range specifiers.
(next_field): New function that returns pointers to the next field in
a line.
(process_field): New function that wraps the field conversion logic
(include_field): New function that checks whether a field should be
converted
(compare_field): New function used for field value comparisons in a
gl_list.
(free_field): New function used for freeing field values in a gl_list.
Global variable FIELD removed.
New global variable all_fields indicates whether all fields should be
processed.
New global variable all_fields_after stores the first field of a N-
style range.
New global variable all_fields_before stores the last field of a -M
style range.
New global variable field_list stores explicitly specified fields to
process (N N,M or N-M style specifiers).
(usage): Document newly supported field range specifiers.
* bootstrap.conf: Include xlist and linked-list modules. numfmt now
uses the gl_linked_list implementation to store the field ranges.
* tests/misc/numfmt.pl: Add tests for 'cut style' field ranges.
Adjust existing tests as partial output can occur before an error
Remove test for the 'invalid' field -5.. this is now a valid range.
* gnulib: update to avoid compiler warnings in linked-list.
* NEWS: Mention the new feature.
|
|
* src/numfmt.c (unit_to_umax): Support SI (power of 10) suffixes
with the --from-unit and --to-unit options. Treat suffixes like
is done with --from=auto, which for example will change the meaning
of --to-unit=G to that of --to-unit=Gi. The suffix support was
previously undocumented and it's better to avoid the traditional
coreutils suffix handling in numfmt by default.
* doc/coreutils.texi: Document the new behavior. Also fix a typo
mentioning {from,to}=units=.
* tests/misc/numfmt.pl: Adjust accordingly.
* NEWS: Mention the change in behavior.
|
|
* src/tail.c (recheck): Display diagnostices for replaced files
even with reused inodes which is a common case.
* tests/tail-2/F-vs-missing.sh: Use correct diagnostic in comment.
* tests/tail-2/F-vs-rename.sh: Likewise.
|
|
* src/tail.c (tail_forever_inotify): Use the fspec pointer to
distinguish previously output files, rather than a descriptor
from the inotify event. That event descriptor was that of
the parent directory when files were created or renamed etc.
(check_fspec): Adjust for the new comparison. Also show the
header when the file is truncated, since we show data
in this case also.
* tests/tail-2/F-headers.sh: A new test case.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug fix.
|
|
* configure.ac: Comment on why we link rather than run the test,
and remove the moot __ELF__ check since we never ran it anyway,
and the new CFLAGS and LDFLAGS are a more direct test of support.
* tests/misc/wc-parallel.sh: Fix a syntax error in the previous change.
* tests/misc/md5sum-parallel.sh: Use better error checking, consistent
with that used in wc-parallel.sh.
|
|
Problems reported by Michael Felt, and and part of this fix taken
from code suggested by Pádraig Brady in:
http://bugs.gnu.org/20733#112
* configure.ac (stdbuf_supported): Check for warnings, and
for -fPIC and -shared, for AIX.
* src/stat.c (STRUCT_STATVFS): Define to struct statvfs64 if
STATFS is statvfs64.
* src/sync.c (sync_arg) [_AIX]: Open in write mode,
since AIX fsync doesn't work on read-only file descriptors.
* tests/misc/wc-parallel.sh: Skip test if xargs -P does not work.
|
|
* tests/tail-2/wait.sh: Without inotify, skip a portion of the test
that checks that -F never outputs from a tailed descriptor
after the followed name is recreated, because tail_forever()
doesn't guarantee that.
Noticed at http://hydra.nixos.org/build/22766288
|
|
* tests/misc/uniq-perf.sh: Use our standard 10s timeout,
which is sufficient to trigger the failure and also
avoids a false failure on slow/loaded systems.
Noticed at http://hydra.nixos.org/build/22766288
|
|
* tests/ln/hard-to-sym.sh: Only call framework_failure_ when ln
returns success.
|
|
* src/copy.c (CAN_HARDLINK_SYMLINKS): Don't enable use of linkat()
on Darwin 14, as the gnulib fallback emulation there doesn't
preserve ownership and timestamps etc. This fixes a test failure
in tests/cp/link-symlink.sh
* tests/cp/link-deref.sh: Adjust accordingly.
|
|
When the parent directory exists and has a different
default context to the final directory, the context
was incorrectly left as that of the parent directory.
* src/mkdir.c (process_dir): Because defaultcon() is called for
existing ancestors (as it must be to avoid races), then we must
unconditionally call restorecon() on the last component due to
the already documented caveat with make_dir_parents().
Alternatively you could temp disable o->set_security_context
around make_dir_parents(), but that would be subject to races.
* tests (tests/mkdir/restorecon.sh): Add a TODO for improvement.
Reference mknod and mkfifo with print_ver_.
* NEWS: Mention the bug fix.
Fixes http://bugs.gnu.org/20616
|
|
* tests/df/df-output.sh: Allow for multiple spaces in the header line
of 'df', resulting from alignment with disk sizes >= 10TB.
|
|
* tests/cp/no-ctx.sh: Scope of `var=val func` is inconsistent
across shells, so avoid that construct with functions.
* tests/df/no-mtab-status.sh: Likewise.
* tests/tail-2/inotify-race.sh: `read` needs an argument.
* tests/tail-2/inotify-race2.sh: Likewise.
|
|
* cfg.mk (sc_case_insensitive_file_names): A new syntax-check rule.
* tests/tail-2/descriptor-vs-rename.sh: Rename from
tests/tail-2/f-vs-rename.sh
* tests/local.mk: Reference the renamed test.
Reported by Jim Meyering.
|
|
* tests/tail-2/inotify-rotate-resources.sh: s/(is_local_dir)/\1_/
|
|
* tests/dd/sparse.sh: Sync files before checking allocations,
which may be done asynchronously on NFS and BTRFS at least.
Also mark this test as very expensive on remote file systems.
* tests/du/2g.sh: Likewise, also use fallocate if available
to efficiently allocate the large file, otherwise skip
on remote file systems.
* tests/tail-2/inotify-rotate-resources.sh: Use the more
standard is_local_dir_() to check remoteness.
* tests/cp/fiemap-empty.sh: Comment on the sync issue
for this currerntly unused test.
Fixes http://bugs.gnu.org/20570
|
|
Supporting `split --numeric-suffixes=1 -n100` for example.
* doc/coreutils.texi (split invocation): Mention the two
use cases for the FROM parameter, and the consequences on
the suffix length determination.
* src/split.c (set_suffix_length): Use the --numeric-suffixes
FROM parameter in the suffix width calculation, when it's
less than the number of files specified in --number.
* tests/split/suffix-auto-length.sh: Add test cases.
Fixes http://bugs.gnu.org/20511
|
|
Generally if logs are truncated, they're truncated to 0 length,
so output all existing data when our heuristic determines truncation.
Note with inotify, truncate() and write() are often determined
independently and so all data would be written if that was the case.
* src/tail.c (check_fspec): Reset file offset to 0 upon truncation.
(tail_forever): Likewise.
(recheck): Add a FIXME for the related issue where tail may lose
data due to tail discounting older log files too early.
* tests/tail-2/truncate.sh: A new test.
* tests/local.mk: Reference the new test.
* NEWS: Mention the fix.
|
|
The previous fixes to races in the various tail tests,
identified actual races in the tail inotify implementation.
With --follow=descriptor, if the tailed file was replaced before
the inotify watch was added, then any subsequent changes were ignored.
Similarly in --follow=name mode, all changes to a new name were
effectively ignored if that name was created after the original open()
but before the inotify_add_watch().
* src/tail.c (tail_forever_inotify): Fix 3 cases.
1. With -f, don't stop tailing when file removed before watch.
2. With -f, watch right file when file replaced before watch.
3. With -F, inspect correct file when replaced before watch.
Existing tests identify these when tail compiled with TAIL_TEST_SLEEP.
* tests/tail-2/inotify-rotate-resources.sh:
This test also identifies the issue with --follow=name
when TAIL_TEST_SLEEP is used. Adjust so the test is immune
to such races, and also fail quicker on remote file systems.
* tests/tail-2/inotify-race2.sh: A new test using GDB,
based on inotify-race.sh, which tests the -F race
without needed recompilation with sleeps.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug.
|
|
Reap background processes so that:
- Stray processes aren't left on the system
- Files aren't held open causing deletion issues on NFS
- Partitions used to run the tests from can be unmounted
* tests/tail-2/F-vs-missing.sh: Add the `kill && wait` of the
background $pid(s) to cleanup_().
* tests/tail-2/F-vs-rename.sh: Likewise.
* tests/tail-2/f-vs-rename.sh: Likewise.
* tests/tail-2/append-only.sh: Likewise.
* tests/tail-2/assert-2.sh: Likewise.
* tests/tail-2/assert.sh: Likewise.
* tests/tail-2/flush-initial.sh: Likewise.
* tests/tail-2/inotify-hash-abuse.sh: Likewise.
* tests/tail-2/inotify-hash-abuse2.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/tail-2/inotify-rotate-resources.sh: Likewise.
* tests/tail-2/inotify-rotate.sh: Likewise.
* tests/tail-2/pid.sh: Likewise.
* tests/tail-2/pipe-f2.sh: Likewise.
* tests/tail-2/retry.sh: Likewise.
* tests/tail-2/symlink.sh: Likewise.
* tests/tail-2/tail-n0f.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
* tests/cp/existing-perm-race.sh: Likewise.
* tests/cp/file-perm-race.sh: Likewise.
* tests/cp/parent-perm-race.sh: Likewise.
* tests/cp/sparse-to-pipe.sh: Likewise.
* tests/dd/stats.sh: Likewise.
* tests/du/move-dir-while-traversing.sh: Likewise.
* tests/misc/cat-buf.sh: Likewise.
* tests/misc/help-version.sh: Likewise.
* tests/misc/printf-surprise.sh: Likewise.
* tests/misc/sort-compress-proc.sh: Likewise.
* tests/misc/sort-spinlock-abuse.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/misc/tac-continue.sh: Likewise.
* tests/misc/timeout-group.sh: Likewise.
* tests/mv/i-3.sh: Likewise.
* tests/rm/dangling-symlink.sh: Likewise.
* tests/rm/isatty.sh: Likewise.
* cfg.mk (sc_prohibit_test_background_without_cleanup_):
A new syntax-check to ensure cleanup_() is defined
when background tasks are created in a test.
|
|
* tests/tail-2/inotify-race.sh: Add a `wait` to ensure that
we reap all background gdb and tail processes. That resulted
in the test hanging intermittently and upon investigation was
due to gdb intermittently failing to terminate the child process
due to receiving a SIGCONT signal. Therefore we avoid using
timeout(1) which sends that signal, and instead rely on tail's
inbuilt --pid monitoring on a background sleep process.
Given this new implementation, the VERY_EXPENSIVE guard was removed.
Related issues with this test hanging were previously discussed at:
https://lists.gnu.org/archive/html/bug-coreutils/2009-12/msg00025.html
|
|
* tests/tail-2/F-vs-missing.sh: Use standard "fastpoll" options
(-s.1 --max-unchanged-stats=1) to speedup the non-inotify case.
Add the non-inotify case to the test. `wait` on the background
tail process to terminate which should avoid the need for the
non standard `retry_delay_ cleanup ...` on NFS.
* tests/tail-2/F-vs-rename.sh: Remove 'out' at the start of the loop,
to avoid a race in checking its contents. Also ensure 'a' & 'b'
files are present before the tail process starts. Use the standard
"fastpoll" options as above.
* tests/tail-2/f-vs-rename.sh: Likewise.
* tests/tail-2/append-only.sh: Use more standard variable names.
* tests/tail-2/flush-initial.sh: Use "fastpoll" options for
non-inotify platforms. Also `wait` on the background tail to avoid
stray processes and file cleanup issues on NFS.
* tests/tail-2/inotify-hash-abuse.sh: Always run non-inotify case.
Use "fastpoll" options. Use a more standard retry_delay_ instead
of a hardcoded sleep loop. Add a `wait` on the background tail.
* tests/tail-2/inotify-hash-abuse2.sh: Likewise.
* tests/tail-2/inotify-rotate-resources.sh: Wait just on the
specific tail $pid needed.
* tests/tail-2/inotify-rotate.sh: Use "fastpoll" options.
* tests/tail-2/pid.sh: Use standard variable names.
Add a `wait` on the background tails.
* tests/tail-2/pipe-f2.sh: Likewise.
* tests/tail-2/tail-n0f.sh: Likewise.
* tests/tail-2/retry.sh: Use "fastpoll" options.
* tests/tail-2/symlink.sh: Likewise.
* tests/tail-2/wait.sh: Likewise. Speedup by using sub second
parameters to timeout(1). Improve the part ensuring that
-F never follows a renamed file.
* tests/tail-2/infloop-1.sh: Remove invalid test. tail(1) was not
being passed the --pid=$yes_pid option, retry_delay_ wasn't used
to avoid races, and yes could write huge files before being killed.
* tests/local.mk: Remove the invalid test reference.
* tests/tail-2/assert-2.sh: Rewrite using retry_delay_(). Since
no longer hardcoding large delays, remove the VERY_EXPENSIVE tag.
* tests/tail-2/assert.sh: Likewise.
|
|
* tests/df/no-mtab-status.sh: getmntent is no longer called
when /proc/self/mountinfo is present, thus causing the test
to be skipped. Therefore wrap fopen() to ignore mountinfo,
and use the test genmntent table instead.
* tests/df/skip-duplicates.sh: Likewise.
|
|
* tests/ls/no-cap.sh: Ensure the test isn't skipped even if
capability coloring is disabled in the current $LS_COLORS.
Also just enable/disable capability coloring to avoid the
dircolors(1) overhead.
|
|
* src/df.c (filter_mount_list): With -l, avoid stating remote mounts.
* init.cfg: Avoid test hangs with inaccessible remote mounts.
* tests/df/no-mtab-status.sh: Skip with inaccessible remote mounts.
* tests/df/skip-rootfs.sh: Likewise.
* tests/df/total-verify.sh: Likewise.
* NEWS: Mention the bug fix.
Reported at http://bugzilla.redhat.com/1199679
|
|
* src/tail.c (tail_forever_inotify): Only monitor write()s and
truncate()s to files in --follow=descriptor mode, thus avoiding
the bug where we removed the watch on renamed files.
Also adjust the inotify event processing code that is
now significant only in --follow=name mode.
* tests/tail-2/F-vs-rename.sh: Improve this existing test by running
in both polling and inotify modes.
* tests/tail-2/f-vs-rename.sh: A new test based on the existing one.
* tests/local.mk: Reference the new test.
* NEWS: Mention the bug.
Fixes http://bugs.gnu.org/19760
|
|
* tests/fs/skip-duplicates.sh: On this platform .mnt_opts is significant
so define to empty to avoid a NULL deref in read_file_system_list().
Fixes http://bugs.gnu.org/20210
|
|
* src/yes.c (main): Even when the internal buffer isn't large enough,
output what we've buffered already, and interate over the rest.
This improves the performance in the edge case where there are
many small arguments that overflow the buffer.
* tests/misc/yes.sh: Add a test case for the many small arguments case.
|
|
yes(1) may be used to generate repeating patterns of text
for test inputs etc., so adjust to be more efficient.
Profiling the case where yes(1) is outputting small items
through stdio (which was the default case), shows the overhead
of continuously processing small items in main() and in stdio:
$ yes >/dev/null & perf top -p $!
31.02% yes [.] main
27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5
14.51% libc-2.20.so [.] fputs_unlocked
13.50% libc-2.20.so [.] strlen
10.66% libc-2.20.so [.] __GI___mempcpy
1.98% yes [.] fputs_unlocked@plta
Sending more data per stdio call improves the situation,
but still, there is significant stdio overhead due to memory copies,
and the repeated string length checking:
$ yes "`echo {1..1000}`" >/dev/null & perf top -p $!
42.26% libc-2.20.so [.] __GI___mempcpy
17.38% libc-2.20.so [.] strlen
5.21% [kernel] [k] __srcu_read_lock
4.58% [kernel] [k] __srcu_read_unlock
4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5
2.50% libc-2.20.so [.] __GI___libc_write
2.45% [kernel] [k] system_call
2.40% [kernel] [k] system_call_after_swapgs
2.27% [kernel] [k] vfs_write
2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5
2.01% [kernel] [k] fsnotify
1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5
1.44% yes [.] main
We can avoid all stdio overhead by building up the buffer
_once_ and outputting that, and the profile below shows
the bottleneck moved to the kernel:
$ src/yes >/dev/null & perf top -p $!
15.42% [kernel] [k] __srcu_read_lock
12.98% [kernel] [k] __srcu_read_unlock
9.41% libc-2.20.so [.] __GI___libc_write
9.11% [kernel] [k] vfs_write
8.35% [kernel] [k] fsnotify
8.02% [kernel] [k] system_call
5.84% [kernel] [k] system_call_after_swapgs
4.54% [kernel] [k] __fget_light
3.98% [kernel] [k] sys_write
3.65% [kernel] [k] selinux_file_permission
3.44% [kernel] [k] rw_verify_area
2.94% [kernel] [k] __fsnotify_parent
2.76% [kernel] [k] security_file_permission
2.39% yes [.] main
2.17% [kernel] [k] __fdget_pos
2.13% [kernel] [k] sysret_check
0.81% [kernel] [k] write_null
0.36% yes [.] write@plt
Note this change also ensures that yes(1) will only write
complete lines for lines shorter than BUFSIZ.
* src/yes.c (main): Build up a BUFSIZ buffer of lines,
and output that, rather than having stdio process each item.
* tests/misc/yes.sh: Add a new test for various buffer sizes.
* tests/local.mk: Reference the new test.
Fixes http://bugs.gnu.org/20029
|
|
Adjust commit v8.23-140-gfdd6ebf to add the --output-error option
instead of --write-error, and treat open() errors like write() errors.
* doc/coreutils.texi (tee invocation): s/write-error/output-error/.
* src/tee.c (main): Exit on open() error if appropriate.
* tests/misc/tee.sh: Add a case to test open() errors.
* NEWS: Adjust for the more general output error behavior.
Suggested by Bernhard Voelker.
|
|
tee is very often used with pipes and this gives better control
when writing to them. There are 3 classes of file descriptors
that tee can write to: files(1), pipes(2), and early close pipes(3).
Handling write errors to 1 & 2 is supported at present with the caveat
that failure writing to any pipe will terminate tee immediately.
Handling write errors to type 3 is not currently supported.
To improve the supported combinations we add these options:
--write-error=warn
Warn if error writing any output including pipes.
Allows continued writing to still open files/pipes.
Exit status is failure if any output had error.
--write-error=warn-nopipe, -p
Warn if error writing any output except pipes.
Allows continued writing to still open files/pipes.
Exit status is failure if any non pipe output had error.
--write-error=exit
Exit if error writing any output including pipes.
--write-error=exit-nopipe
Exit if error writing any output except pipes.
Use the "nopipe" variants when files are of types 1 and 3, otherwise
use the standard variants with types 1 and 2. A caveat with the above
scheme is that a combination of pipe types (2 & 3) is not supported
robustly. I.e. if you use the "nopipe" variants when using both type
2 and 3 pipes, then any "real" errors on type 2 pipes will not be
diagnosed.
Note also a general issue with type 3 pipes that are not on tee's
stdout, is that shell constructs don't allow to distinguish early
close from real failures. For example `tee >(head -n1) | grep -m1 ..`
can't distinguish between an error or an early close in "head" pipe,
while the fail on the grep part of the pipe is distinguished
independently from the resulting pipe errors. This is a general
issue with the >() construct, rather than with tee itself.
* NEWS: Mention the new feature.
* doc/coreutils.texi (tee invocation): Describe the new option.
* src/tee.c (usage): Likewise.
(main): With --write-error ignore SIGPIPE, and handle
the various exit, diagnostics combinations.
* tests/misc/tee.sh: Tess all the new options.
Fixes http://bugs.gnu.org/11540
|
