summaryrefslogtreecommitdiff
path: root/tests/misc
AgeCommit message (Collapse)Author
2015-11-04paste: avoid confusing backslash quoting in diagnosticPádraig Brady
* src/paste.c (main): Setting the quoting style to "escape" went against the intent of the comment about presenting doubled backslashes to the user. Instead use "c-maybe" which is the only mode which avoids doubled backslashes, and provides protection against arbitrary control characters. * tests/misc/paste.pl: Adjust accordingly.
2015-11-04printf: support the %q format to quote for shellPádraig Brady
* src/printf.c (usage): Mention the new format. (print_formatted): Handle the quoting by calling out to the quotearg module with "shell-escape" mode. * doc/coreutils.texi (printf invocation): Document %q. * tests/misc/printf-quote.sh: New test. * tests/local.mk: Reference new test. * NEWS: Mention the new feature.
2015-11-04ls: avoid redundant processing when already escapingPádraig Brady
This is mainly noticeable when the multi-byte code within ls.c is triggered by multi-byte quotes. $ seq 200000 | xargs touch $ time ls-old -U --quoting=locale --hide-control-chars >/dev/null real 0m0.483s $ time ls-new -U --quoting=locale --hide-control-chars >/dev/null real 0m0.430s * src/ls.c (quote_name): Avoid rescanning the output looking for unprintable chars when we know the quoting mode already escapes them. * tests/misc/ls-misc.pl: Add tests for all quoting modes, with and without -q, to verify this assumption.
2015-11-04ls: document and test new shell-escape quotingPádraig Brady
* doc/coreutils.texi (ls invocation): Describe the new 'shell-escape' and 'shell-escape-always' quoting options. * src/ls.c (usage): Mention the new quoting options. * tests/misc/ls-misc.pl: Add a test for 'shell-escape'
2015-11-04all: avoid quoting file names when possiblePádraig Brady
Quote file names using the "shell-escape" or "shell-escape-always" methods, which quote as appropriate for most shells, and better support copy and paste of presented names. The "always" variant is used when the file name is embedded in an error message with surrounding spaces. * cfg.mk (sc_error_shell_quotes): A new syntax check rule to suggest quotef() where appropriate. (sc_error_shell_always_quotes): Likewise for quoteaf(). * src/system.h (quotef): A new define to apply shell quoting when needed. I.E. when shell character or ':' is present. (quoteaf): Likewise, but always quote. * src/*.c: Use quotef() and quoteaf() rather than quote() where appropriate. * tests/: Adjust accordingly.
2015-11-04md5sum: ensure a single status line per filePádraig Brady
* src/md5sum.c: Use the same file name escaping method used when generating and checking checksums. I.E. ensure a single line per file by starting the line with '\' for any file name containing '\n' and replacing those with "\\n". * NEWS: Move the item from changes in behavior to improvements, since this is no longer a backwards incompat change when processing stdout status messages. * tests/misc/md5sum.pl: Remove quotes from expected status output. * tests/misc/sha1sum.pl: Likewise.
2015-11-04all: replace most uses of quotearg_colon() with quote()Pádraig Brady
Related to commit v8.24-61-g6796698 this provides more consistent quoting, as quotearg_colon() defaults to "literal" quoting by default, while quote() provides appropriate quoting for diagnostics by default. * gl/modules/randread: Depend on quote module rather than quotearg. * gl/lib/randread.c: Used quote() not quotearg_colon(). * src/: Likewise. * src/shred.c: Likewise. Also avoid unnecessary quoting introducing overhead when wiping names. * cfg.mk: Relax the matching expression to allow "qname" variables as used in shred.c to satisfy the check. * tests/: Adjust accordingly.
2015-10-27date: use extended format timezone for --iso-8601Pádraig Brady
* src/date.c (main): Use %:z rather than %z with --iso-8601 as the standard states to consistently use extended format. Note either format can be parsed by date. * tests/misc/date.pl: Adjust accordingly. * doc/coreutils.texi (du invocation): Clarify that "iso" time styles are only similar to ISO-8601. (ls invocation): Likewise. (date invocation): Adjust the comment stating that only --rfc-3339 output can be parsed by date(1). * NEWS: Mention the change in behavior. Reported at http://bugs.debian.org/799479
2015-10-27all: quote string arguments in error messagesPádraig Brady
These strings are often file names or other user specified parameters, which can give confusing errors in the presence of unexpected characters for example. * cfg.mk (sc_error_quotes): A new syntax check rule. * src/*.c: Wrap error() string arguments with quote(). * tests/: Adjust accordingly. * NEWS: Mention the improvement.
2015-10-27md5sum: quote all printed file namesPádraig Brady
This is especially significant when using --check with files generated on a windows system, where the \r characters produce corrupted and confusing error messages. This also ensures status messages are output on a single line. * src/md5sum.c: Use quote() for printed file names. * tests/misc/md5sum.pl: Adjust accordingly. * NEWS: Mention the change in behavior. Fixes http://bugs.gnu.org/21757
2015-10-19tests: adjust recent change to csplit VM limitPádraig Brady
* tests/misc/csplit-heap.sh: A little more memory is required for the full run case. Noticed with --enable-single-binary.
2015-10-16dircolors: support globbing of TERM entriesPádraig Brady
* src/dircolors.c (dc_parse_stream): Support globbing of TERM entries, to allow entries like "TERM *256color*" for example. * src/dircolors.hin: Reduce the internal list with globbing. * tests/misc/dircolors.pl: New test cases. * NEWS: Mention the improvement.
2015-10-10tests: make a long running test responsive to Ctrl-CPádraig Brady
* tests/misc/sort-compress-hang.sh: Use --foreground with the timeout(1) command (noting the caveats), to run the sort command in the foreground program group, and thus be responsive to Ctrl-C. This very_expensive_ test takes over a minute on a i3-2310M, with RAM backed /tmp.
2015-10-08tests: adjust recent changes to virtual memory limitsPádraig Brady
* tests/dd/no-allocate.sh: Account for timeout(1) when determining the required mem, as timeout has additional shared libs. This avoids the need for the hardcoded 4M addition to the limit. * tests/misc/head-c.sh: Increase the base limit, to account for the fact that head(1) will allocate some additional mem in this case. * tests/misc/cut-huge-range.sh: Remove mention of specific limits. * tests/misc/printf-surprise.sh: Likewise. Reported by Dmitry V. Levin
2015-10-06csplit: remove erroneous mention of -m in --helpPádraig Brady
* src/csplit.c (usage): -m is not accepted, only --suppress-matched. * tests/misc/csplit-suppress-matched.pl: Spelling fix. Reported by Ondrej Oprala
2015-09-22maint: use adaptive approach for `ulimit -v` based testsBernhard Voelker
When configured with either 'symlinks' or 'shebangs' as value for the --enable-single-binary option, tests based on `ulimit -v` are skipped. The reason is that the multicall 'coreutils' binary requires much more memory due to shared libraries being loaded, and the size of the 'date' binary (~290KiB) compared to the multicall binary (~5MiB), of course. Finally, in the case of 'shebangs', the starting shell requires more memory, too Instead of using hard-coded values for the memory limit, use an adaptive approach: first determine the amount of memory for a similar, yet more trivial invocation of the command, and then do the real test run using that limit (plus some buffer in some cases). * init.cfg (require_ulimit_v_): Remove function. (get_min_ulimit_v_): Add function to determine the minimum memory limit required for a given command in an adaptive way. * cfg.mk (sc_prohibit_test_ulimit_without_require_): Change the name of the above function in the syntax-check rule. * tests/cp/link-heap.sh: Use the above function to determine the minimum memory required to run a command simpler than in the real test run. Use that limit plus a buffer there. While at it, change to list of commands in the subshell to fail also if the beginning `ulimit -v` fails. * tests/dd/no-allocate.sh: Likewise. * tests/misc/csplit-heap.sh: Likewise. * tests/misc/cut-huge-range.sh: Likewise. * tests/misc/head-c.sh: Likewise. * tests/misc/printf-surprise.sh: Likewise. * tests/split/line-bytes.sh: Likewise. * tests/rm/many-dir-entries-vs-OOM.sh: Likewise - doing it separately for each program under test.
2015-09-22sort,numfmt: with --debug, diagnose failure to set localePádraig Brady
* src/sort.c (main): With --debug, warn upon setlocale() failure, which can happen due to incorrectly specified environment variables, or due to memory exhaustion (simulated with ulimit -v), etc. * tests/misc/sort-debug-warn.sh: Add a test case. See also http://savannah.gnu.org/bugs/11004
2015-09-12numfmt: use new set-fields module to parse --fieldAssaf Gordon
numfmt --field=LIST can accept the same options as cut. * bootstrap.conf: remove xlist, linked-list * src/local.mk: link numfmt with set-fields * src/numfmt.c: use set-fields.c instead of custom field parsing code. (include_field): adapt to new code. * tests/misc/numfmt.pl: add new tests, adapt current tests to new error message wording from set-fields.c
2015-09-12cut: refactor into set-fields moduleAssaf Gordon
Extract the functionality of parsing --field=LIST into a separate module, to be used by other programs. * src/cut.c: move field parsing code from here ... * src/set-fields.{c,h}: ... to here. (set_fields): generalize by supporting multiple parsing/reporting options. (struct range_pair): rename to field_range_pair. * src/local.mk: link cut with set-field. * po/POTFILES.in: add set-field.c * tests/misc/cut.pl: update wording of error messages
2015-09-03tests: test numfmt stdin behaviorPádraig Brady
* tests/misc/tty-eof.pl: Add numfmt to the list of programs that accept input on stdin.
2015-09-03base64: no longer support hex or oct --wrap paramsPádraig Brady
* src/base64.c (main): Support decimal numbers with leading zeros, by disabling the auto detection of octal and hex. It's not envisaged that base conversion is needed for --wrap parameters, and in the edge case it is, $((0x0)) shell constructs can be used. * tests/misc/base64.pl: Adjust accordingly. * NEWS: Mention the change in behavior.
2015-09-03base64: use stricter validation on wrap columnPádraig Brady
* src/base64.c (main): Use the higher level xnumtoumax() rather than xstrtoumax(), which is simpler and improves validation of input. Also pass the _empty_ rather than NULL string as the suffixes parameter so that invalid trailing characters are not allowed. For example -w08 is now flagged as an error, rather than being interpreted as 0. A subsequent commit will further improve verification of numbers with leading zeros by dropping backwards compatibility wrt auto parsing oct and hex numbers. * tests/misc/base64.pl: Add tests for invalid wrap values.
2015-09-03base32: A new program similar to base64Pádraig Brady
Suggested in https://bugzilla.redhat.com/1250113 * AUTHORS: Add base32. * THANKS.in: Add suggester. * README: Reference the new program. * NEWS: Mention the new program. * src/.gitignore: Ignore the new binary. * bootstrap.conf: Reference the gnulib base32 module. * build-aux/gen-lists-of-programs.sh: Add base32. * man/base32.x: A new template. * man/.gitignore: Ignore the new man page. * man/local.mk: Reference the new man page. * doc/coreutils.texi (base32 invocation): Document the new command. * src/local.mk: Adjust to build base32 based on base64.c. * src/base64.c: Parameterize to use the correct headers, functions and buffer sizes, depending on which binary is being built. * tests/misc/base64.pl: Adjust to test both base32 and base64. * tests/misc/tty-eof.pl: Add base32 as a program that accepts input on stdin without any options specified. * scripts/git-hooks/commit-msg: Add base32 to the template.
2015-07-07shred: fix pattern selection for certain iteration countsPádraig Brady
This was detected in about 25% of runs with gcc -fsanitize=address ERROR: AddressSanitizer: global-buffer-overflow on address ... READ of size 4 at 0x000000416628 thread T0 #0 0x40479f in genpattern src/shred.c:782 #1 0x4050d9 in do_wipefd src/shred.c:921 #2 0x406203 in wipefile src/shred.c:1175 #3 0x406b84 in main src/shred.c:1316 #4 0x7f3454a1ef9f in __libc_start_main (/lib64/libc.so.6+0x1ff9f) #5 0x4025d8 (/tmp/coreutils-8.23/src/shred+0x4025d8) 0x000000416628 is located 56 bytes to the left of global variable '*.LC49' from 'src/shred.c' (0x416660) of size 17 0x000000416628 is located 12 bytes to the right of global variable 'patterns' from 'src/shred.c' (0x416540) of size 220 SUMMARY: AddressSanitizer: global-buffer-overflow src/shred.c:782 * src/shred.c (gen_patterns): Restrict pattern selection to the K available, which regressed due to v5.92-1462-g65533e1. * tests/misc/shred-passes.sh: Add a deterministic test case. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/20998
2015-07-03tests: avoid side effects of $SHELL environment variablePádraig Brady
Since non interactive shells don't generally set $SHELL, its value is propagated through the tests and may cause issues; for example if $SHELL implicitly adjusts $PATH when run. Instead we set $SHELL to that determined by the posix-shell module, and use that consistently for all test sub scripts, including those created thorugh the `split --filter` command. * tests/local.mk: Explicitly set $SHELL to $(PREFERABLY_POSIX_SHELL) which defaults to $CONFIG_SHELL and thus usually /bin/sh. * tests/envvar-check: Remove bash environment variables with side effects, in case /bin/bash was selected for $SHELL. * tests/misc/help-version.sh: Remove redundant initialization of $SHELL. * tests/install/strip-program.sh: Use $SHELL for sub script. * tests/misc/sort-compress-hang.sh: Likewise. * tests/misc/sort-compress-proc.sh: Likewise. * tests/misc/sort-compress.sh: Likewise. * tests/misc/timeout-group.sh: Likewise. * tests/rm/fail-eperm.xpl: Remove redundant elision of bash env vars. * tests/misc/pwd-long.sh: Likewise.
2015-07-02tests: avoid false failures on OpenBSD 5.7Pádraig Brady
* tests/du/threshold.sh: Homogenize getopt error messages. * tests/misc/numfmt.pl: Likewise. * tests/mv/i-3.sh: Skip on *BSD not just FreeBSD.
2015-07-01tests: avoid false failure on FreeBSD systemsPádraig Brady
* tests/misc/stty.sh: FreeBSD returns ENOTTY for the TIOCEXT ioctl, so just avoid this option for now.
2015-06-25tests: avoid false failure when running as rootPádraig Brady
* tests/misc/sync.sh: Ensure dir is unreadable before including the permission check.
2015-06-24factor: avoid interspersed lines for parallel runsPádraig Brady
* src/factor.c (n_out): A new global variable to track how much data has been written to stdout. (print_factors_single): Use n_out to determine whether to flush the current (and previous) lines. * tests/misc/factor-parallel.sh: Add a new test. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix.
2015-06-24seq: handle exponents more consistentlyPádraig Brady
src/seq.c (scan_arg): Set precision and width _after_ exponentiation. For example, this will make '1.1e1 12' and '11 1.2e1' equivalent. One can still set the precision by specifying extra precision on the start value, or more naturally with a precision on a step value. * tests/misc/seq-precision.sh: Add new cases.
2015-06-24seq: use consistent output format with hex integersPádraig Brady
* src/seq.c (scan_arg): Set precision to 0 for hex constants (while avoiding hex floats). This will use then use the fast path for these arguments. Note we also set the precision of inf to 0 here, which ensures we use consistent precision on output where possible. * tests/misc/seq-precision.sh: Add corresponding test cases.
2015-06-24seq: support inf last item more generally/efficientlyPádraig Brady
* src/seq.c (main): Call seq_fast for infinite last value. This implicitly avoids format conversion on the 999999 -> 1000000 transition. * src/seq.c (seq_fast): Generalize the buffer handling, and adjust to handle the "inf" last value specifics. * tests/misc/seq-precision.sh: A new test. * tests/local.mk: Reference the new test.
2015-06-22numfmt: don't hardcode floating point limitsPádraig Brady
* src/numfmt.c (MAX_UNSCALED_DIGITS): Set this to LDBL_DIG rather than hardcoding at 18 for better portability. * tests/misc/numfmt.pl: Restrict limit tests to supported platforms.
2015-06-22numfmt: handle leading zeros correctlyPádraig Brady
* src/numfmt.c (simple_strtod_int): Don't count leading zeros as significant digits. Also have leading zeros as optional for floating point numbers. * tests/misc/numfmt.pl: Add test cases. * NEWS: Mention the fix.
2015-06-22numfmt: avoid integer overflow when roundingPádraig Brady
Due to existing limits this is usually triggered with an increased precision. We also add further restrictions to the output of increased precision numbers. * src/numfmt.c (simple_round): Avoid intmax_t overflow. (simple_strtod_int): Count digits consistently for precision loss and overflow detection. (prepare_padded_number): Include the precision when excluding numbers to output, since the precision determines the ultimate values used in the rounding scheme in double_to_human(). * tests/misc/numfmt.pl: Add previously failing test cases. * NEWS: Mention the fix.
2015-06-21numfmt: support user specified output precisionPádraig Brady
* src/numfmt.c (usage): Update the --format description to indicate precision is allowed. (parse_format_string): Parse a precision specification like the standard printf does. (double_to_human): Honor the precision in --to mode. * tests/misc/numfmt.pl: New tests. * doc/coreutils.texi (numfmt invocation): Mention the new feature. * NEWS: Likewise.
2015-06-19numfmt: implement support for field rangesDylan Cali
* src/numfmt.c: Replace field handling code with logic that understands field range specifiers. Instead of processing a single field and printing line prefix/suffix around it, process each field in the line checking whether it has been included for conversion. If so convert and print, otherwise just print the unaltered field. (extract_fields): Removed. (skip_fields): Removed. (process_line): Gutted and heavily reworked. (process_suffixed_number): FIELD is now passed as an arg instead of using a global. (parse_field_arg): New function that parses field range specifiers. (next_field): New function that returns pointers to the next field in a line. (process_field): New function that wraps the field conversion logic (include_field): New function that checks whether a field should be converted (compare_field): New function used for field value comparisons in a gl_list. (free_field): New function used for freeing field values in a gl_list. Global variable FIELD removed. New global variable all_fields indicates whether all fields should be processed. New global variable all_fields_after stores the first field of a N- style range. New global variable all_fields_before stores the last field of a -M style range. New global variable field_list stores explicitly specified fields to process (N N,M or N-M style specifiers). (usage): Document newly supported field range specifiers. * bootstrap.conf: Include xlist and linked-list modules. numfmt now uses the gl_linked_list implementation to store the field ranges. * tests/misc/numfmt.pl: Add tests for 'cut style' field ranges. Adjust existing tests as partial output can occur before an error Remove test for the 'invalid' field -5.. this is now a valid range. * gnulib: update to avoid compiler warnings in linked-list. * NEWS: Mention the new feature.
2015-06-19numfmt: handle suffixes consistently with --{from,to}-unitPádraig Brady
* src/numfmt.c (unit_to_umax): Support SI (power of 10) suffixes with the --from-unit and --to-unit options. Treat suffixes like is done with --from=auto, which for example will change the meaning of --to-unit=G to that of --to-unit=Gi. The suffix support was previously undocumented and it's better to avoid the traditional coreutils suffix handling in numfmt by default. * doc/coreutils.texi: Document the new behavior. Also fix a typo mentioning {from,to}=units=. * tests/misc/numfmt.pl: Adjust accordingly. * NEWS: Mention the change in behavior.
2015-06-06tests: fix false failure in recent test adjustmentPádraig Brady
* configure.ac: Comment on why we link rather than run the test, and remove the moot __ELF__ check since we never ran it anyway, and the new CFLAGS and LDFLAGS are a more direct test of support. * tests/misc/wc-parallel.sh: Fix a syntax error in the previous change. * tests/misc/md5sum-parallel.sh: Use better error checking, consistent with that used in wc-parallel.sh.
2015-06-06build: port to AIXPaul Eggert
Problems reported by Michael Felt, and and part of this fix taken from code suggested by Pádraig Brady in: http://bugs.gnu.org/20733#112 * configure.ac (stdbuf_supported): Check for warnings, and for -fPIC and -shared, for AIX. * src/stat.c (STRUCT_STATVFS): Define to struct statvfs64 if STATFS is statvfs64. * src/sync.c (sync_arg) [_AIX]: Open in write mode, since AIX fsync doesn't work on read-only file descriptors. * tests/misc/wc-parallel.sh: Skip test if xargs -P does not work.
2015-06-01tests: fix false failure on loaded systemsPádraig Brady
* tests/misc/uniq-perf.sh: Use our standard 10s timeout, which is sufficient to trigger the failure and also avoids a false failure on slow/loaded systems. Noticed at http://hydra.nixos.org/build/22766288
2015-05-11tests: cleanup background processes upon interruptionPádraig Brady
Reap background processes so that: - Stray processes aren't left on the system - Files aren't held open causing deletion issues on NFS - Partitions used to run the tests from can be unmounted * tests/tail-2/F-vs-missing.sh: Add the `kill && wait` of the background $pid(s) to cleanup_(). * tests/tail-2/F-vs-rename.sh: Likewise. * tests/tail-2/f-vs-rename.sh: Likewise. * tests/tail-2/append-only.sh: Likewise. * tests/tail-2/assert-2.sh: Likewise. * tests/tail-2/assert.sh: Likewise. * tests/tail-2/flush-initial.sh: Likewise. * tests/tail-2/inotify-hash-abuse.sh: Likewise. * tests/tail-2/inotify-hash-abuse2.sh: Likewise. * tests/tail-2/inotify-race.sh: Likewise. * tests/tail-2/inotify-rotate-resources.sh: Likewise. * tests/tail-2/inotify-rotate.sh: Likewise. * tests/tail-2/pid.sh: Likewise. * tests/tail-2/pipe-f2.sh: Likewise. * tests/tail-2/retry.sh: Likewise. * tests/tail-2/symlink.sh: Likewise. * tests/tail-2/tail-n0f.sh: Likewise. * tests/tail-2/wait.sh: Likewise. * tests/cp/existing-perm-race.sh: Likewise. * tests/cp/file-perm-race.sh: Likewise. * tests/cp/parent-perm-race.sh: Likewise. * tests/cp/sparse-to-pipe.sh: Likewise. * tests/dd/stats.sh: Likewise. * tests/du/move-dir-while-traversing.sh: Likewise. * tests/misc/cat-buf.sh: Likewise. * tests/misc/help-version.sh: Likewise. * tests/misc/printf-surprise.sh: Likewise. * tests/misc/sort-compress-proc.sh: Likewise. * tests/misc/sort-spinlock-abuse.sh: Likewise. * tests/misc/stdbuf.sh: Likewise. * tests/misc/tac-continue.sh: Likewise. * tests/misc/timeout-group.sh: Likewise. * tests/mv/i-3.sh: Likewise. * tests/rm/dangling-symlink.sh: Likewise. * tests/rm/isatty.sh: Likewise. * cfg.mk (sc_prohibit_test_background_without_cleanup_): A new syntax-check to ensure cleanup_() is defined when background tasks are created in a test.
2015-03-10yes: improve efficiency when all args aren't bufferedGiuseppe Scrivano
* src/yes.c (main): Even when the internal buffer isn't large enough, output what we've buffered already, and interate over the rest. This improves the performance in the edge case where there are many small arguments that overflow the buffer. * tests/misc/yes.sh: Add a test case for the many small arguments case.
2015-03-10yes: output data more efficientlyPádraig Brady
yes(1) may be used to generate repeating patterns of text for test inputs etc., so adjust to be more efficient. Profiling the case where yes(1) is outputting small items through stdio (which was the default case), shows the overhead of continuously processing small items in main() and in stdio: $ yes >/dev/null & perf top -p $! 31.02% yes [.] main 27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 14.51% libc-2.20.so [.] fputs_unlocked 13.50% libc-2.20.so [.] strlen 10.66% libc-2.20.so [.] __GI___mempcpy 1.98% yes [.] fputs_unlocked@plta Sending more data per stdio call improves the situation, but still, there is significant stdio overhead due to memory copies, and the repeated string length checking: $ yes "`echo {1..1000}`" >/dev/null & perf top -p $! 42.26% libc-2.20.so [.] __GI___mempcpy 17.38% libc-2.20.so [.] strlen 5.21% [kernel] [k] __srcu_read_lock 4.58% [kernel] [k] __srcu_read_unlock 4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 2.50% libc-2.20.so [.] __GI___libc_write 2.45% [kernel] [k] system_call 2.40% [kernel] [k] system_call_after_swapgs 2.27% [kernel] [k] vfs_write 2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5 2.01% [kernel] [k] fsnotify 1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5 1.44% yes [.] main We can avoid all stdio overhead by building up the buffer _once_ and outputting that, and the profile below shows the bottleneck moved to the kernel: $ src/yes >/dev/null & perf top -p $! 15.42% [kernel] [k] __srcu_read_lock 12.98% [kernel] [k] __srcu_read_unlock 9.41% libc-2.20.so [.] __GI___libc_write 9.11% [kernel] [k] vfs_write 8.35% [kernel] [k] fsnotify 8.02% [kernel] [k] system_call 5.84% [kernel] [k] system_call_after_swapgs 4.54% [kernel] [k] __fget_light 3.98% [kernel] [k] sys_write 3.65% [kernel] [k] selinux_file_permission 3.44% [kernel] [k] rw_verify_area 2.94% [kernel] [k] __fsnotify_parent 2.76% [kernel] [k] security_file_permission 2.39% yes [.] main 2.17% [kernel] [k] __fdget_pos 2.13% [kernel] [k] sysret_check 0.81% [kernel] [k] write_null 0.36% yes [.] write@plt Note this change also ensures that yes(1) will only write complete lines for lines shorter than BUFSIZ. * src/yes.c (main): Build up a BUFSIZ buffer of lines, and output that, rather than having stdio process each item. * tests/misc/yes.sh: Add a new test for various buffer sizes. * tests/local.mk: Reference the new test. Fixes http://bugs.gnu.org/20029
2015-03-04tee: generalize the --write-error option to --output-errorPádraig Brady
Adjust commit v8.23-140-gfdd6ebf to add the --output-error option instead of --write-error, and treat open() errors like write() errors. * doc/coreutils.texi (tee invocation): s/write-error/output-error/. * src/tee.c (main): Exit on open() error if appropriate. * tests/misc/tee.sh: Add a case to test open() errors. * NEWS: Adjust for the more general output error behavior. Suggested by Bernhard Voelker.
2015-02-24tee: add --write-error to control handling of closed pipesPádraig Brady
tee is very often used with pipes and this gives better control when writing to them. There are 3 classes of file descriptors that tee can write to: files(1), pipes(2), and early close pipes(3). Handling write errors to 1 & 2 is supported at present with the caveat that failure writing to any pipe will terminate tee immediately. Handling write errors to type 3 is not currently supported. To improve the supported combinations we add these options: --write-error=warn Warn if error writing any output including pipes. Allows continued writing to still open files/pipes. Exit status is failure if any output had error. --write-error=warn-nopipe, -p Warn if error writing any output except pipes. Allows continued writing to still open files/pipes. Exit status is failure if any non pipe output had error. --write-error=exit Exit if error writing any output including pipes. --write-error=exit-nopipe Exit if error writing any output except pipes. Use the "nopipe" variants when files are of types 1 and 3, otherwise use the standard variants with types 1 and 2. A caveat with the above scheme is that a combination of pipe types (2 & 3) is not supported robustly. I.e. if you use the "nopipe" variants when using both type 2 and 3 pipes, then any "real" errors on type 2 pipes will not be diagnosed. Note also a general issue with type 3 pipes that are not on tee's stdout, is that shell constructs don't allow to distinguish early close from real failures. For example `tee >(head -n1) | grep -m1 ..` can't distinguish between an error or an early close in "head" pipe, while the fail on the grep part of the pipe is distinguished independently from the resulting pipe errors. This is a general issue with the >() construct, rather than with tee itself. * NEWS: Mention the new feature. * doc/coreutils.texi (tee invocation): Describe the new option. * src/tee.c (usage): Likewise. (main): With --write-error ignore SIGPIPE, and handle the various exit, diagnostics combinations. * tests/misc/tee.sh: Tess all the new options. Fixes http://bugs.gnu.org/11540
2015-02-20tee: treat '-' operand as file name as mandated by POSIXBernhard Voelker
Since v5.2.1-1247-g8dafbe5, tee(1) treated '-' as stdout while POSIX explicitly requires to treat this as a file name. Revert this change, as the interleaved output - due to sending another copy of input to stdout - is not considered to be useful. Discussed in http://lists.gnu.org/archive/html/coreutils/2015-02/msg00085.html * src/tee.c (tee_files): Remove the special handling for "-" operands. (usage): Remove the corresponding sentence. * doc/coreutils.texi (common options): Remove the "tee -" example. (tee invocation): Document that tee(1) now treats "-" as a file name. * tests/misc/tee.sh: Add a test case for "tee -". While at it, re-indent the above multi-argument processing case and extend that to 13 operands, as POSIX mandates that, too. * tests/misc/tee-dash.sh: Remove now-obsolete test. * tests/local.mk (all_tests): Remove the above test. * NEWS (Changes in behavior): Mention the change.
2015-02-18tee: exit early if no more writable outputsPádraig Brady
* src/tee.c (main): Don't continue reading if we can't output anywhere. * tests/misc/tee.sh: Ensure we exit when no more outputs. * NEWS: Mention the change in behavior.
2015-02-03tests: avoid a FP on aarch64 when comparing env/printenv outputBernhard Voelker
The following test fails on aarch64 on openSUSE's OpenBuildService due to glibc's execvp reversing the pointers of 'environ', i.e., the output of "env|tac" equals "env env" on that platform. * tests/misc/printenv.sh: Use 'env env' to work around the behavior on that platform. While at it, fix the grep pattern which suppressed all environment variables starting with an underscore "_" instead of "$_" (and "$LD_PRELOAD") only.
2015-01-31doc,maint: fix use of "i.e." in documentation and commentsBernhard Voelker
To align with all other places (and correct grammar), change all upper-case "I.E." to "I.e.". Furthermore, ensure that "i.e." is followed by a comma. Finally, ensure to use a double-space before "I.e.," at the beginning of a sentence. The following was used to change all offending uses (apart from old ChangeLog files): $ git grep -liF 'i.e.' \ | xargs sed -i \ -e 's/I\.E\./I.e./g' \ -e 's/\. \(I\.e\.\)/. \1/g' \ -e 's/\([Ii]\.e\.\)\( \)/\1,\2/g' \ -e 's/\([Ii]\.e\.\)$/\1,/g' * cfg.mk (sc_prohibit_uppercase_id_est): Add new rule. (sc_ensure_double_space_after_dot_before_id_est): Likewise. (sc_ensure_comma_after_id_est): Likewise. (old_NEWS_hash): Refresh hash via "make update-NEWS-hash". * NEWS: Change use of "id est" abbreviation via the above command. * README: Likewise. * README-prereq: Likewise. * doc/coreutils.texi: Likewise. * gl/lib/rand-isaac.c: Likewise. * gl/lib/tempname.c.diff: Likewise. * man/stdbuf.x: Likewise. * src/cat.c: Likewise. * src/copy.c: Likewise. * src/copy.h: Likewise. * src/cp.c: Likewise. * src/cut.c: Likewise. * src/dd.c: Likewise. * src/df.c: Likewise. * src/fiemap.h: Likewise. * src/longlong.h: Likewise. * src/ls.c: Likewise. * src/numfmt.c: Likewise. * src/pr.c: Likewise. * src/shred.c: Likewise. * src/shuf.c: Likewise. * src/split.c: Likewise. * tests/Coreutils.pm: Likewise. * tests/df/df-symlink.sh: Likewise. * tests/df/skip-rootfs.sh: Likewise. * tests/init.sh: Likewise. * tests/ls/color-norm.sh: Likewise. * tests/misc/basename.pl: Likewise. * tests/misc/ls-misc.pl: Likewise. * tests/misc/md5sum-bsd.sh: Likewise. * tests/misc/shred-exact.sh: Likewise. * tests/misc/sort.pl: Likewise. * tests/misc/stdbuf.sh: Likewise. * tests/misc/tac-continue.sh: Likewise. * tests/rm/r-root.sh: Likewise. * tests/tail-2/symlink.sh: Likewise.