summaryrefslogtreecommitdiff
path: root/src
AgeCommit message (Collapse)Author
2015-06-24maint: avoid undefined behavior in qsort callPádraig Brady
GCC 5.1.1 -fsanitize=undefined with glibc 2.21 is returning: "runtime error: null pointer passed as argument 1, which is declared to never be null" * src/ptx.c (sort_found_occurs): Avoid the call with no entries.
2015-06-24factor: avoid interspersed lines for parallel runsPádraig Brady
* src/factor.c (n_out): A new global variable to track how much data has been written to stdout. (print_factors_single): Use n_out to determine whether to flush the current (and previous) lines. * tests/misc/factor-parallel.sh: Add a new test. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix.
2015-06-24seq: handle exponents more consistentlyPádraig Brady
src/seq.c (scan_arg): Set precision and width _after_ exponentiation. For example, this will make '1.1e1 12' and '11 1.2e1' equivalent. One can still set the precision by specifying extra precision on the start value, or more naturally with a precision on a step value. * tests/misc/seq-precision.sh: Add new cases.
2015-06-24seq: use consistent output format with hex integersPádraig Brady
* src/seq.c (scan_arg): Set precision to 0 for hex constants (while avoiding hex floats). This will use then use the fast path for these arguments. Note we also set the precision of inf to 0 here, which ensures we use consistent precision on output where possible. * tests/misc/seq-precision.sh: Add corresponding test cases.
2015-06-24seq: support inf last item more generally/efficientlyPádraig Brady
* src/seq.c (main): Call seq_fast for infinite last value. This implicitly avoids format conversion on the 999999 -> 1000000 transition. * src/seq.c (seq_fast): Generalize the buffer handling, and adjust to handle the "inf" last value specifics. * tests/misc/seq-precision.sh: A new test. * tests/local.mk: Reference the new test.
2015-06-22numfmt: don't hardcode floating point limitsPádraig Brady
* src/numfmt.c (MAX_UNSCALED_DIGITS): Set this to LDBL_DIG rather than hardcoding at 18 for better portability. * tests/misc/numfmt.pl: Restrict limit tests to supported platforms.
2015-06-22numfmt: handle leading zeros correctlyPádraig Brady
* src/numfmt.c (simple_strtod_int): Don't count leading zeros as significant digits. Also have leading zeros as optional for floating point numbers. * tests/misc/numfmt.pl: Add test cases. * NEWS: Mention the fix.
2015-06-22numfmt: avoid integer overflow when roundingPádraig Brady
Due to existing limits this is usually triggered with an increased precision. We also add further restrictions to the output of increased precision numbers. * src/numfmt.c (simple_round): Avoid intmax_t overflow. (simple_strtod_int): Count digits consistently for precision loss and overflow detection. (prepare_padded_number): Include the precision when excluding numbers to output, since the precision determines the ultimate values used in the rounding scheme in double_to_human(). * tests/misc/numfmt.pl: Add previously failing test cases. * NEWS: Mention the fix.
2015-06-21numfmt: support user specified output precisionPádraig Brady
* src/numfmt.c (usage): Update the --format description to indicate precision is allowed. (parse_format_string): Parse a precision specification like the standard printf does. (double_to_human): Honor the precision in --to mode. * tests/misc/numfmt.pl: New tests. * doc/coreutils.texi (numfmt invocation): Mention the new feature. * NEWS: Likewise.
2015-06-19numfmt: implement support for field rangesDylan Cali
* src/numfmt.c: Replace field handling code with logic that understands field range specifiers. Instead of processing a single field and printing line prefix/suffix around it, process each field in the line checking whether it has been included for conversion. If so convert and print, otherwise just print the unaltered field. (extract_fields): Removed. (skip_fields): Removed. (process_line): Gutted and heavily reworked. (process_suffixed_number): FIELD is now passed as an arg instead of using a global. (parse_field_arg): New function that parses field range specifiers. (next_field): New function that returns pointers to the next field in a line. (process_field): New function that wraps the field conversion logic (include_field): New function that checks whether a field should be converted (compare_field): New function used for field value comparisons in a gl_list. (free_field): New function used for freeing field values in a gl_list. Global variable FIELD removed. New global variable all_fields indicates whether all fields should be processed. New global variable all_fields_after stores the first field of a N- style range. New global variable all_fields_before stores the last field of a -M style range. New global variable field_list stores explicitly specified fields to process (N N,M or N-M style specifiers). (usage): Document newly supported field range specifiers. * bootstrap.conf: Include xlist and linked-list modules. numfmt now uses the gl_linked_list implementation to store the field ranges. * tests/misc/numfmt.pl: Add tests for 'cut style' field ranges. Adjust existing tests as partial output can occur before an error Remove test for the 'invalid' field -5.. this is now a valid range. * gnulib: update to avoid compiler warnings in linked-list. * NEWS: Mention the new feature.
2015-06-19doc: use correct units in df | numfmt examplePádraig Brady
* src/numfmt.c (usage): Don't scale output from df so that numfmt outputs the correct values.
2015-06-19numfmt: handle suffixes consistently with --{from,to}-unitPádraig Brady
* src/numfmt.c (unit_to_umax): Support SI (power of 10) suffixes with the --from-unit and --to-unit options. Treat suffixes like is done with --from=auto, which for example will change the meaning of --to-unit=G to that of --to-unit=Gi. The suffix support was previously undocumented and it's better to avoid the traditional coreutils suffix handling in numfmt by default. * doc/coreutils.texi: Document the new behavior. Also fix a typo mentioning {from,to}=units=. * tests/misc/numfmt.pl: Adjust accordingly. * NEWS: Mention the change in behavior.
2015-06-09tail: display consistent diagnostics upon file replacementPádraig Brady
* src/tail.c (recheck): Display diagnostices for replaced files even with reused inodes which is a common case. * tests/tail-2/F-vs-missing.sh: Use correct diagnostic in comment. * tests/tail-2/F-vs-rename.sh: Likewise.
2015-06-09tail: display file headers correctly with inotifyPádraig Brady
* src/tail.c (tail_forever_inotify): Use the fspec pointer to distinguish previously output files, rather than a descriptor from the inotify event. That event descriptor was that of the parent directory when files were created or renamed etc. (check_fspec): Adjust for the new comparison. Also show the header when the file is truncated, since we show data in this case also. * tests/tail-2/F-headers.sh: A new test case. * tests/local.mk: Reference the new test. * NEWS: Mention the bug fix.
2015-06-06build: port to AIXPaul Eggert
Problems reported by Michael Felt, and and part of this fix taken from code suggested by Pádraig Brady in: http://bugs.gnu.org/20733#112 * configure.ac (stdbuf_supported): Check for warnings, and for -fPIC and -shared, for AIX. * src/stat.c (STRUCT_STATVFS): Define to struct statvfs64 if STATFS is statvfs64. * src/sync.c (sync_arg) [_AIX]: Open in write mode, since AIX fsync doesn't work on read-only file descriptors. * tests/misc/wc-parallel.sh: Skip test if xargs -P does not work.
2015-06-04build: port single_binary_prog to POSIX shellPaul Eggert
Problem reported privately by Michael Felt. * Makefile.am (install-exec-hook): * src/local.mk (src/coreutils_symlinks, src/coreutils_shebangs) (clean-local): Port to POSIX shell, which doesn't allow 'for i in ; do ...'.
2015-05-30maint: avoid new coverity warningsPádraig Brady
* src/sync.c (sync_arg): Initialise variable to avoid unitialized access if assert is disabled. * src/head.c (elide_tail_bytes_file): Support this function with ---presume-input-pipe and larger files, which regressed with commit v8.23-47-g2662702. (elide_tail_lines_file): Likewise. * src/dd.c (dd_copy): Explicitly don't try to ftruncate() upon failure to lseek() (the existing check against st_size was already protecting that). * src/factor.c (factor_using_squfof): Assert (only when linting due to performance) to avoid the implication of divide by zero. * src/od.c (read_block): Remove dead code. * src/tac.c (tac_seekable): Likewise. * src/ls.c (gobble_file): Likewise.
2015-05-28copy: prefer our hardlink to symlink emulation on OS X 10.10Pádraig Brady
* src/copy.c (CAN_HARDLINK_SYMLINKS): Don't enable use of linkat() on Darwin 14, as the gnulib fallback emulation there doesn't preserve ownership and timestamps etc. This fixes a test failure in tests/cp/link-symlink.sh * tests/cp/link-deref.sh: Adjust accordingly.
2015-05-26maint: remove sys/types.h include order checkPádraig Brady
* src/system.h: This was inadvertently ineffective due to a typo in commit v8.9-10-ge1aaf89 (Jan 2011), but has not caused any issues, so remove.
2015-05-23mkdir: fix -pZ with existing parent directoriesPádraig Brady
When the parent directory exists and has a different default context to the final directory, the context was incorrectly left as that of the parent directory. * src/mkdir.c (process_dir): Because defaultcon() is called for existing ancestors (as it must be to avoid races), then we must unconditionally call restorecon() on the last component due to the already documented caveat with make_dir_parents(). Alternatively you could temp disable o->set_security_context around make_dir_parents(), but that would be subject to races. * tests (tests/mkdir/restorecon.sh): Add a TODO for improvement. Reference mknod and mkfifo with print_ver_. * NEWS: Mention the bug fix. Fixes http://bugs.gnu.org/20616
2015-05-13timeout: with --foreground don't send SIGCONTPádraig Brady
* src/timeout.c (cleanup): Don't send SIGCONT to the monitored program when --foreground is specified, as it's generally not needed for foreground programs, and can cause intermittent signal delivery issues with monitors like GDB for example. * doc/coreutils.texi (timeout invocation): Mention that SIGCONT is not sent with --foreground. * NEWS: Mention the behavior change.
2015-05-13split: auto set suffix len for --numeric-suffixes=<N --number=NPádraig Brady
Supporting `split --numeric-suffixes=1 -n100` for example. * doc/coreutils.texi (split invocation): Mention the two use cases for the FROM parameter, and the consequences on the suffix length determination. * src/split.c (set_suffix_length): Use the --numeric-suffixes FROM parameter in the suffix width calculation, when it's less than the number of files specified in --number. * tests/split/suffix-auto-length.sh: Add test cases. Fixes http://bugs.gnu.org/20511
2015-05-13doc: clarify the operation of wc -LAssaf Gordon
* src/wc.c (usage): State that it calculates display width. * doc/coreutils.texi (wc invocation): Detail the distinct items used to determine the display width.
2015-05-11tail: consistently output all data for truncated filesPádraig Brady
Generally if logs are truncated, they're truncated to 0 length, so output all existing data when our heuristic determines truncation. Note with inotify, truncate() and write() are often determined independently and so all data would be written if that was the case. * src/tail.c (check_fspec): Reset file offset to 0 upon truncation. (tail_forever): Likewise. (recheck): Add a FIXME for the related issue where tail may lose data due to tail discounting older log files too early. * tests/tail-2/truncate.sh: A new test. * tests/local.mk: Reference the new test. * NEWS: Mention the fix.
2015-05-11tail: fix inotify startup racesPádraig Brady
The previous fixes to races in the various tail tests, identified actual races in the tail inotify implementation. With --follow=descriptor, if the tailed file was replaced before the inotify watch was added, then any subsequent changes were ignored. Similarly in --follow=name mode, all changes to a new name were effectively ignored if that name was created after the original open() but before the inotify_add_watch(). * src/tail.c (tail_forever_inotify): Fix 3 cases. 1. With -f, don't stop tailing when file removed before watch. 2. With -f, watch right file when file replaced before watch. 3. With -F, inspect correct file when replaced before watch. Existing tests identify these when tail compiled with TAIL_TEST_SLEEP. * tests/tail-2/inotify-rotate-resources.sh: This test also identifies the issue with --follow=name when TAIL_TEST_SLEEP is used. Adjust so the test is immune to such races, and also fail quicker on remote file systems. * tests/tail-2/inotify-race2.sh: A new test using GDB, based on inotify-race.sh, which tests the -F race without needed recompilation with sleeps. * tests/local.mk: Reference the new test. * NEWS: Mention the bug.
2015-05-10build: rewrite is_ENOTSUP without an #if directivePaul Eggert
* src/system.h (is_ENOTSUP): Avoid in-function #if directive.
2015-05-09build: avoid a warning form gcc's new -Wlogical-opJim Meyering
Without this change, very recent gcc (e.g., version 6.0.0 20150509) would print the following when configured with --enable-gcc-warnings: src/copy.c:165:30: error: logical 'or' of equal expressions \ [-Werror=logical-op] && (errno == EOPNOTSUPP || errno == ENOTSUP || errno == ENOSYS)) ^ * src/system.h (is_ENOTSUP): New function. * src/copy.c (punch_hole): Use it. * src/ls.c (errno_unsupported, gobble_file): Use it.
2015-04-30doc: standardize messages about the '-' stdin FILEPádraig Brady
* src/system.h (emit_stdin_note): A new function, refactoring the usage note about the '-' FILE implying stdin. * src/base64.c (usage): Use the new function to emit the note in a standard location and with standard separation. * src/cat.c (usage): Likewise. * src/csplit.c (usage): Likewise. * src/cut.c (usage): Likewise. * src/expand.c (usage): Likewise. * src/fmt.c (usage): Likewise. * src/head.c (usage): Likewise. * src/md5sum.c (usage): Likewise. * src/nl.c (usage): Likewise. * src/od.c (usage): Likewise. * src/paste.c (usage): Likewise. * src/pr.c (usage): Likewise. * src/ptx.c (usage): Likewise. * src/shred.c (usage): Likewise. * src/shuf.c (usage): Likewise. * src/sort.c (usage): Likewise. * src/sum.c (usage): Likewise. * src/tac.c (usage): Likewise. * src/tail.c (usage): Likewise. * src/tsort.c (usage): Likewise. * src/unexpand.c (usage): Likewise. * src/wc.c (usage): Likewise. * src/join.c (usage): Adjust the separation used for the message referring to FILE1 or FILE2 as stdin. * src/comm.c (usage): Add a message using the same wording (translation) as used in join. * src/split.c (usage): Reword to using FILE rather than INPUT, allowing use of emit_stdin_note(). Also remove the mention of "fixed-size" pieces as this isn't now always the case. Fixes http://pad.lv/1450179
2015-04-28build: fix SINGLE_BINARY build when printf is a macroYunlian Jiang
* src/coreutils.c (usage): include coreutils.h outside the printf call, because if it's a macro you will get the error: embedding a #include directive within macro arguments is not supported
2015-04-28maint: avoid a new -Werror=return-type warning in yes.cPádraig Brady
* src/yes.c (main): Simplify the logic so that the compiler can see this function always returns a value. This was seen with GCC 5.0 in SINGLE_BINARY mode.
2015-04-27maint: fix printf format for signed integersPádraig Brady
With GCC 5 and the newly added warnings from gnulib, ensure the correct signed integer is passed for the printf format, to avoid -Werror=format= failures.
2015-04-26ls: on GNU/Linux, remove dependency on libaclPaul Eggert
* src/local.mk (src_ls_LDADD): Change from LIB_ACL to LIB_HAS_ACL.
2015-04-22build: fix potential factor build failure on arm and mipsPádraig Brady
* src/longlong.h: Sync with the latest longlong.h from libgmp to: - Use __builtin_c[lt]zl on arm64. - Fix sparc64 vis3 build failure due to missing __clz_tab. - Avoid a clang build issue on mips. - Support thumb2 arm 32 bit system. * src/cfg.mk (sc_ensure_comma_after_id_est): Exclude longlong.h to ease merges.
2015-04-22maint: avoid -Werror=strict-overflow warnings with GCC 5Pádraig Brady
All warnings were of the form: "assuming signed overflow does not occur when simplifying conditional to constant [-Werror=strict-overflow]" * src/dd.c (cache_round): Use an appropriately sized unsigned type, to avoid possibility of undefined signed overflow. * src/mknod.c (main): Likewise. * src/pr.c (pad_down): Likewise. * src/wc.c (main): Likewise. * src/tail.c (main): Assert that argc >= 0 thus allowing the compiler to assume without implication that argc - optind is positive.
2015-04-17dircolors: add 'MISSING' to the default databasePádraig Brady
* src/dircolors.hin: Add the MISSING entry, to indicate this as a possibility in new templates output from dircolors, and also to ease comparison with existing databases that generally do define a MISSING entry.
2015-04-13df: fix --local hanging with inaccessible remote mountsPádraig Brady
* src/df.c (filter_mount_list): With -l, avoid stating remote mounts. * init.cfg: Avoid test hangs with inaccessible remote mounts. * tests/df/no-mtab-status.sh: Skip with inaccessible remote mounts. * tests/df/skip-rootfs.sh: Likewise. * tests/df/total-verify.sh: Likewise. * NEWS: Mention the bug fix. Reported at http://bugzilla.redhat.com/1199679
2015-04-03df: fix use of uninitialized variable reported by valgrindPádraig Brady
Conditional jump or move depends on uninitialised value(s) at 0x40380C: get_field_values (df.c:840) by 0x403E16: get_dev (df.c:994) by 0x404D65: get_all_entries (df.c:1364) by 0x405926: main (df.c:1714) * src/df.c (get_dev): Initialize the fsu.fsu_bavail_top_bit_set member, when adding placeholder entries. (main): Avoid a "definitely lost" memory leak warning from valgrind, reported by Bernhard Voelker.
2015-04-01doc: clarify that ls --sort=time is newest firstPádraig Brady
* src/ls.c (usage): Add punctuation to avoid ambiguity in the description of the --time option. Mention that both the -u and --sort=time default order is newest first.
2015-03-31tail: fix -f to follow changes after a renameStephane Chazelas
* src/tail.c (tail_forever_inotify): Only monitor write()s and truncate()s to files in --follow=descriptor mode, thus avoiding the bug where we removed the watch on renamed files. Also adjust the inotify event processing code that is now significant only in --follow=name mode. * tests/tail-2/F-vs-rename.sh: Improve this existing test by running in both polling and inotify modes. * tests/tail-2/f-vs-rename.sh: A new test based on the existing one. * tests/local.mk: Reference the new test. * NEWS: Mention the bug. Fixes http://bugs.gnu.org/19760
2015-03-27nohup: clarify stdin redirectionPaul Eggert
Problem reported by Isaac Schwabacher in: http://bugs.gnu.org/20214 * doc/coreutils.texi (nohup invocation): Clarify that when nohup's stdin gets redirected, it's unreadable. * doc/coreutils.texi (nohup invocation): * src/nohup.c (usage): Don't promise /dev/null.
2015-03-26doc: disambiguate the ls --color descriptionChristoph Anton Mitterer
* src/ls.c (usage): Avoid the implication that the default ls behavior is to --color=always. Reported in http://bugs.debian.org/781208
2015-03-26doc: clarify the date standard output formatsPádraig Brady
* src/date.c (usage): Use FMT rather than TIMESPEC as the parameter, since it's simpler to understand and can be better aligned. Give an example for the --iso-8601 output format. Adjust the example used for the 3 standard formats to be unambiguous with respect to day/mon ordering and use of leading zeros in the time. Reorder the options descriptions slightly, so that the 3 standards options are together. Indent the multi-line descriptions so that grouping is obvious. Remove a redundant description of the --rfc-3339 format, which is obvious in the existing example. Separate these 3 standards options to their own translatable string to simplify translation. Change 'date and time' to 'date/time' in the --iso-8601 description to be consistent with --rfc-3339 and to help avoid the implication that the time is always output or even output by default. Fixes http://bugs.gnu.org/20203
2015-03-24doc: clarify the uniq -D man page descriptionPádraig Brady
* src/uniq.c (usage): The description was very confusing in the man page due to the stripped newlines. Add punctuation for clarification.
2015-03-24wc: use a more adaptive wc -l implementationPádraig Brady
* src/wc.c (wc): Allow any block to select the count implementation, rather than just using the first 10 lines. This also simplifies the code from 3 loops to 2.
2015-03-23doc: clarify default order for ls --sort=sizeDan Jacobson
* src/ls.c (usage): Mention that default order is largest first. Fixes http://bugs.gnu.org/20172
2015-03-20wc: speedup counting of short linesKristoffer Brånemyr
Using a test file generated with: yes | head -n100M > 2x100M.txt before> time wc -l 2x100M.txt real 0.842s user 0.810s sys 0.033s after> time wc -l 2x100M.txt real 0.142s user 0.111s sys 0.031s * src/wc.c (wc): Split the loop that deals with -l into 3. The first is used at the start of the input to determine if the average line length is < 15, and if so the second loop is used to look for '\n' internally to wc. For longer lines, memchr is used as before to take advantage of system specific optimizations which any outweigh function call overhead. Note the first 2 loops could be combined, though in testing, GCC 4.9.2 at least, wasn't sophisticated enough to separate the loops based on the "check_len" invariant. Note also __builtin_memchr() isn't significant here as GCC currently only applies constant folding with that. * NEWS: Mention the improvement.
2015-03-10yes: improve efficiency when all args aren't bufferedGiuseppe Scrivano
* src/yes.c (main): Even when the internal buffer isn't large enough, output what we've buffered already, and interate over the rest. This improves the performance in the edge case where there are many small arguments that overflow the buffer. * tests/misc/yes.sh: Add a test case for the many small arguments case.
2015-03-10yes: output data more efficientlyPádraig Brady
yes(1) may be used to generate repeating patterns of text for test inputs etc., so adjust to be more efficient. Profiling the case where yes(1) is outputting small items through stdio (which was the default case), shows the overhead of continuously processing small items in main() and in stdio: $ yes >/dev/null & perf top -p $! 31.02% yes [.] main 27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 14.51% libc-2.20.so [.] fputs_unlocked 13.50% libc-2.20.so [.] strlen 10.66% libc-2.20.so [.] __GI___mempcpy 1.98% yes [.] fputs_unlocked@plta Sending more data per stdio call improves the situation, but still, there is significant stdio overhead due to memory copies, and the repeated string length checking: $ yes "`echo {1..1000}`" >/dev/null & perf top -p $! 42.26% libc-2.20.so [.] __GI___mempcpy 17.38% libc-2.20.so [.] strlen 5.21% [kernel] [k] __srcu_read_lock 4.58% [kernel] [k] __srcu_read_unlock 4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5 2.50% libc-2.20.so [.] __GI___libc_write 2.45% [kernel] [k] system_call 2.40% [kernel] [k] system_call_after_swapgs 2.27% [kernel] [k] vfs_write 2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5 2.01% [kernel] [k] fsnotify 1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5 1.44% yes [.] main We can avoid all stdio overhead by building up the buffer _once_ and outputting that, and the profile below shows the bottleneck moved to the kernel: $ src/yes >/dev/null & perf top -p $! 15.42% [kernel] [k] __srcu_read_lock 12.98% [kernel] [k] __srcu_read_unlock 9.41% libc-2.20.so [.] __GI___libc_write 9.11% [kernel] [k] vfs_write 8.35% [kernel] [k] fsnotify 8.02% [kernel] [k] system_call 5.84% [kernel] [k] system_call_after_swapgs 4.54% [kernel] [k] __fget_light 3.98% [kernel] [k] sys_write 3.65% [kernel] [k] selinux_file_permission 3.44% [kernel] [k] rw_verify_area 2.94% [kernel] [k] __fsnotify_parent 2.76% [kernel] [k] security_file_permission 2.39% yes [.] main 2.17% [kernel] [k] __fdget_pos 2.13% [kernel] [k] sysret_check 0.81% [kernel] [k] write_null 0.36% yes [.] write@plt Note this change also ensures that yes(1) will only write complete lines for lines shorter than BUFSIZ. * src/yes.c (main): Build up a BUFSIZ buffer of lines, and output that, rather than having stdio process each item. * tests/misc/yes.sh: Add a new test for various buffer sizes. * tests/local.mk: Reference the new test. Fixes http://bugs.gnu.org/20029
2015-03-04tee: generalize the --write-error option to --output-errorPádraig Brady
Adjust commit v8.23-140-gfdd6ebf to add the --output-error option instead of --write-error, and treat open() errors like write() errors. * doc/coreutils.texi (tee invocation): s/write-error/output-error/. * src/tee.c (main): Exit on open() error if appropriate. * tests/misc/tee.sh: Add a case to test open() errors. * NEWS: Adjust for the more general output error behavior. Suggested by Bernhard Voelker.
2015-03-04maint: update stale comment about ls color sequencesJarosław Gruca
* src/ls.c (color_indicator[C_END]): Comment with the correct sequence, which was used since commit v6.10-61-g483297d Fixes http://bugs.gnu.org/19992