| Age | Commit message (Collapse) | Author |
|---|
|
Problems reported by Michael Felt, and and part of this fix taken
from code suggested by Pádraig Brady in:
http://bugs.gnu.org/20733#112
* configure.ac (stdbuf_supported): Check for warnings, and
for -fPIC and -shared, for AIX.
* src/stat.c (STRUCT_STATVFS): Define to struct statvfs64 if
STATFS is statvfs64.
* src/sync.c (sync_arg) [_AIX]: Open in write mode,
since AIX fsync doesn't work on read-only file descriptors.
* tests/misc/wc-parallel.sh: Skip test if xargs -P does not work.
|
|
* tests/misc/uniq-perf.sh: Use our standard 10s timeout,
which is sufficient to trigger the failure and also
avoids a false failure on slow/loaded systems.
Noticed at http://hydra.nixos.org/build/22766288
|
|
Reap background processes so that:
- Stray processes aren't left on the system
- Files aren't held open causing deletion issues on NFS
- Partitions used to run the tests from can be unmounted
* tests/tail-2/F-vs-missing.sh: Add the `kill && wait` of the
background $pid(s) to cleanup_().
* tests/tail-2/F-vs-rename.sh: Likewise.
* tests/tail-2/f-vs-rename.sh: Likewise.
* tests/tail-2/append-only.sh: Likewise.
* tests/tail-2/assert-2.sh: Likewise.
* tests/tail-2/assert.sh: Likewise.
* tests/tail-2/flush-initial.sh: Likewise.
* tests/tail-2/inotify-hash-abuse.sh: Likewise.
* tests/tail-2/inotify-hash-abuse2.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/tail-2/inotify-rotate-resources.sh: Likewise.
* tests/tail-2/inotify-rotate.sh: Likewise.
* tests/tail-2/pid.sh: Likewise.
* tests/tail-2/pipe-f2.sh: Likewise.
* tests/tail-2/retry.sh: Likewise.
* tests/tail-2/symlink.sh: Likewise.
* tests/tail-2/tail-n0f.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
* tests/cp/existing-perm-race.sh: Likewise.
* tests/cp/file-perm-race.sh: Likewise.
* tests/cp/parent-perm-race.sh: Likewise.
* tests/cp/sparse-to-pipe.sh: Likewise.
* tests/dd/stats.sh: Likewise.
* tests/du/move-dir-while-traversing.sh: Likewise.
* tests/misc/cat-buf.sh: Likewise.
* tests/misc/help-version.sh: Likewise.
* tests/misc/printf-surprise.sh: Likewise.
* tests/misc/sort-compress-proc.sh: Likewise.
* tests/misc/sort-spinlock-abuse.sh: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/misc/tac-continue.sh: Likewise.
* tests/misc/timeout-group.sh: Likewise.
* tests/mv/i-3.sh: Likewise.
* tests/rm/dangling-symlink.sh: Likewise.
* tests/rm/isatty.sh: Likewise.
* cfg.mk (sc_prohibit_test_background_without_cleanup_):
A new syntax-check to ensure cleanup_() is defined
when background tasks are created in a test.
|
|
* src/yes.c (main): Even when the internal buffer isn't large enough,
output what we've buffered already, and interate over the rest.
This improves the performance in the edge case where there are
many small arguments that overflow the buffer.
* tests/misc/yes.sh: Add a test case for the many small arguments case.
|
|
yes(1) may be used to generate repeating patterns of text
for test inputs etc., so adjust to be more efficient.
Profiling the case where yes(1) is outputting small items
through stdio (which was the default case), shows the overhead
of continuously processing small items in main() and in stdio:
$ yes >/dev/null & perf top -p $!
31.02% yes [.] main
27.36% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5
14.51% libc-2.20.so [.] fputs_unlocked
13.50% libc-2.20.so [.] strlen
10.66% libc-2.20.so [.] __GI___mempcpy
1.98% yes [.] fputs_unlocked@plta
Sending more data per stdio call improves the situation,
but still, there is significant stdio overhead due to memory copies,
and the repeated string length checking:
$ yes "`echo {1..1000}`" >/dev/null & perf top -p $!
42.26% libc-2.20.so [.] __GI___mempcpy
17.38% libc-2.20.so [.] strlen
5.21% [kernel] [k] __srcu_read_lock
4.58% [kernel] [k] __srcu_read_unlock
4.27% libc-2.20.so [.] _IO_file_xsputn@@GLIBC_2.2.5
2.50% libc-2.20.so [.] __GI___libc_write
2.45% [kernel] [k] system_call
2.40% [kernel] [k] system_call_after_swapgs
2.27% [kernel] [k] vfs_write
2.09% libc-2.20.so [.] _IO_do_write@@GLIBC_2.2.5
2.01% [kernel] [k] fsnotify
1.95% libc-2.20.so [.] _IO_file_write@@GLIBC_2.2.5
1.44% yes [.] main
We can avoid all stdio overhead by building up the buffer
_once_ and outputting that, and the profile below shows
the bottleneck moved to the kernel:
$ src/yes >/dev/null & perf top -p $!
15.42% [kernel] [k] __srcu_read_lock
12.98% [kernel] [k] __srcu_read_unlock
9.41% libc-2.20.so [.] __GI___libc_write
9.11% [kernel] [k] vfs_write
8.35% [kernel] [k] fsnotify
8.02% [kernel] [k] system_call
5.84% [kernel] [k] system_call_after_swapgs
4.54% [kernel] [k] __fget_light
3.98% [kernel] [k] sys_write
3.65% [kernel] [k] selinux_file_permission
3.44% [kernel] [k] rw_verify_area
2.94% [kernel] [k] __fsnotify_parent
2.76% [kernel] [k] security_file_permission
2.39% yes [.] main
2.17% [kernel] [k] __fdget_pos
2.13% [kernel] [k] sysret_check
0.81% [kernel] [k] write_null
0.36% yes [.] write@plt
Note this change also ensures that yes(1) will only write
complete lines for lines shorter than BUFSIZ.
* src/yes.c (main): Build up a BUFSIZ buffer of lines,
and output that, rather than having stdio process each item.
* tests/misc/yes.sh: Add a new test for various buffer sizes.
* tests/local.mk: Reference the new test.
Fixes http://bugs.gnu.org/20029
|
|
Adjust commit v8.23-140-gfdd6ebf to add the --output-error option
instead of --write-error, and treat open() errors like write() errors.
* doc/coreutils.texi (tee invocation): s/write-error/output-error/.
* src/tee.c (main): Exit on open() error if appropriate.
* tests/misc/tee.sh: Add a case to test open() errors.
* NEWS: Adjust for the more general output error behavior.
Suggested by Bernhard Voelker.
|
|
tee is very often used with pipes and this gives better control
when writing to them. There are 3 classes of file descriptors
that tee can write to: files(1), pipes(2), and early close pipes(3).
Handling write errors to 1 & 2 is supported at present with the caveat
that failure writing to any pipe will terminate tee immediately.
Handling write errors to type 3 is not currently supported.
To improve the supported combinations we add these options:
--write-error=warn
Warn if error writing any output including pipes.
Allows continued writing to still open files/pipes.
Exit status is failure if any output had error.
--write-error=warn-nopipe, -p
Warn if error writing any output except pipes.
Allows continued writing to still open files/pipes.
Exit status is failure if any non pipe output had error.
--write-error=exit
Exit if error writing any output including pipes.
--write-error=exit-nopipe
Exit if error writing any output except pipes.
Use the "nopipe" variants when files are of types 1 and 3, otherwise
use the standard variants with types 1 and 2. A caveat with the above
scheme is that a combination of pipe types (2 & 3) is not supported
robustly. I.e. if you use the "nopipe" variants when using both type
2 and 3 pipes, then any "real" errors on type 2 pipes will not be
diagnosed.
Note also a general issue with type 3 pipes that are not on tee's
stdout, is that shell constructs don't allow to distinguish early
close from real failures. For example `tee >(head -n1) | grep -m1 ..`
can't distinguish between an error or an early close in "head" pipe,
while the fail on the grep part of the pipe is distinguished
independently from the resulting pipe errors. This is a general
issue with the >() construct, rather than with tee itself.
* NEWS: Mention the new feature.
* doc/coreutils.texi (tee invocation): Describe the new option.
* src/tee.c (usage): Likewise.
(main): With --write-error ignore SIGPIPE, and handle
the various exit, diagnostics combinations.
* tests/misc/tee.sh: Tess all the new options.
Fixes http://bugs.gnu.org/11540
|
|
Since v5.2.1-1247-g8dafbe5, tee(1) treated '-' as stdout while POSIX
explicitly requires to treat this as a file name. Revert this change,
as the interleaved output - due to sending another copy of input to
stdout - is not considered to be useful. Discussed in
http://lists.gnu.org/archive/html/coreutils/2015-02/msg00085.html
* src/tee.c (tee_files): Remove the special handling for "-" operands.
(usage): Remove the corresponding sentence.
* doc/coreutils.texi (common options): Remove the "tee -" example.
(tee invocation): Document that tee(1) now treats "-" as a file name.
* tests/misc/tee.sh: Add a test case for "tee -".
While at it, re-indent the above multi-argument processing case and
extend that to 13 operands, as POSIX mandates that, too.
* tests/misc/tee-dash.sh: Remove now-obsolete test.
* tests/local.mk (all_tests): Remove the above test.
* NEWS (Changes in behavior): Mention the change.
|
|
* src/tee.c (main): Don't continue reading if we can't
output anywhere.
* tests/misc/tee.sh: Ensure we exit when no more outputs.
* NEWS: Mention the change in behavior.
|
|
The following test fails on aarch64 on openSUSE's OpenBuildService
due to glibc's execvp reversing the pointers of 'environ', i.e.,
the output of "env|tac" equals "env env" on that platform.
* tests/misc/printenv.sh: Use 'env env' to work around the behavior
on that platform.
While at it, fix the grep pattern which suppressed all environment
variables starting with an underscore "_" instead of "$_" (and
"$LD_PRELOAD") only.
|
|
To align with all other places (and correct grammar), change all
upper-case "I.E." to "I.e.". Furthermore, ensure that "i.e." is
followed by a comma. Finally, ensure to use a double-space before
"I.e.," at the beginning of a sentence.
The following was used to change all offending uses (apart from
old ChangeLog files):
$ git grep -liF 'i.e.' \
| xargs sed -i \
-e 's/I\.E\./I.e./g' \
-e 's/\. \(I\.e\.\)/. \1/g' \
-e 's/\([Ii]\.e\.\)\( \)/\1,\2/g' \
-e 's/\([Ii]\.e\.\)$/\1,/g'
* cfg.mk (sc_prohibit_uppercase_id_est): Add new rule.
(sc_ensure_double_space_after_dot_before_id_est): Likewise.
(sc_ensure_comma_after_id_est): Likewise.
(old_NEWS_hash): Refresh hash via "make update-NEWS-hash".
* NEWS: Change use of "id est" abbreviation via the above command.
* README: Likewise.
* README-prereq: Likewise.
* doc/coreutils.texi: Likewise.
* gl/lib/rand-isaac.c: Likewise.
* gl/lib/tempname.c.diff: Likewise.
* man/stdbuf.x: Likewise.
* src/cat.c: Likewise.
* src/copy.c: Likewise.
* src/copy.h: Likewise.
* src/cp.c: Likewise.
* src/cut.c: Likewise.
* src/dd.c: Likewise.
* src/df.c: Likewise.
* src/fiemap.h: Likewise.
* src/longlong.h: Likewise.
* src/ls.c: Likewise.
* src/numfmt.c: Likewise.
* src/pr.c: Likewise.
* src/shred.c: Likewise.
* src/shuf.c: Likewise.
* src/split.c: Likewise.
* tests/Coreutils.pm: Likewise.
* tests/df/df-symlink.sh: Likewise.
* tests/df/skip-rootfs.sh: Likewise.
* tests/init.sh: Likewise.
* tests/ls/color-norm.sh: Likewise.
* tests/misc/basename.pl: Likewise.
* tests/misc/ls-misc.pl: Likewise.
* tests/misc/md5sum-bsd.sh: Likewise.
* tests/misc/shred-exact.sh: Likewise.
* tests/misc/sort.pl: Likewise.
* tests/misc/stdbuf.sh: Likewise.
* tests/misc/tac-continue.sh: Likewise.
* tests/rm/r-root.sh: Likewise.
* tests/tail-2/symlink.sh: Likewise.
|
|
* m4/jm-macros.m4 (coreutils_MACROS): Check for syncfs().
* man/sync.x: Add references to syncfs, fsync and fdatasync.
* doc/coreutils.texi (sync invocation): Document the new feature.
* src/sync.c: Include "quote.h".
(AUTHORS): Include myself.
(MODE_FILE, MODE_DATA, MODE_FILE_SYSTEM, MODE_SYNC): New enum values.
(long_options): Define.
(sync_arg): New function.
(usage): Describe that arguments are now accepted.
(main): Add arguments parsing and add support for fsync(2),
fdatasync(2) and syncfs(2).
* tests/misc/sync.sh: New (and only) test for sync.
* tests/local.mk: Reference the new test.
* AUTHORS: Add myself to sync's authors.
* NEWS: Mention the new feature.
|
|
When some program produces unexpected output, that use of
compare-vs-/dev/null will ensure that the surprising output is
printed in the test's output. With "test -s err" only, one
would have to instrument and rerun in order to see the offending
output.
* cfg.mk (sc_prohibit_and_fail_1): Exempt 'compare' from this check.
* tests/dd/misc.sh: Change "tests -s ... || fail=1" to
"compare /dev/null ... && fail=1".
* tests/misc/nice.sh: Likewise.
* tests/rm/read-only.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/touch/no-dereference.sh: Likewise.
Suggested by Jim Meyering in
http://lists.gnu.org/archive/html/coreutils/2015-01/msg00042.html
|
|
Many tests use `program ... && fail=1` to ensure expected
error situations are indicated. However that would mask
an unexpected exit (like a crash). Therefore explicitly
check the expected exit code.
Note where error messages are also verified, the extra
protection is not added.
* tests/init.sh (returns_): A new helper function to
check the return code of a command, and used
throughout the tests.
* cfg.mk (sc_prohibit_and_fail_1): Add a syntax check
to avoid new instances of this issue.
|
|
* tests/cp/cp-a-selinux.sh: Comment why unused variables are assigned.
Fix misspellings noticed while adjusting.
* tests/cp/fiemap-perf.sh: Fix quoting.
* tests/misc/shuf.sh: Avoid useless use of cat.
* tests/misc/printf-surprise.sh: Likewise.
|
|
* tests/misc/shuf.sh: Improve the test so it detects
crashes in more cases.
* NEWS: Mention the previous fix.
|
|
Problem reported by Daiki Ueno in: http://bugs.gnu.org/19520
* src/shuf.c (main): Avoid core dump if !input_range.
* tests/misc/shuf.sh: Test for this bug.
|
|
Run "make update-copyright" and then...
* tests/sample-test: Adjust to use the single most recent year.
* tests/du/bind-mount-dir-cycle-v2.sh: Fix case in copyright message,
so that year is updated automatically in future.
|
|
http://hydra.nixos.org/build/18129583 identified (on OS X)
an incorrect test assumption in the previous commit.
* gl/lib/xdectoint.c (__xnumtoint): Suppress the EINVAL
error message as it's redundant in this context.
* tests/misc/tail.pl: Suppress _optionally_ appended
strerror messages.
* tests/fmt/base.pl: Likewise.
* tests/pr/pr-tests.pl: Likewise.
* tests/split/l-chunk.sh: Likewise.
|
|
Following on from commit v8.23-82-gaddae94, consistently diagnose
numbers that are too large, so as to distinguish from other errors,
and make the limits obvious.
* gl/modules/xdectoint: A new module implementing xdecto[iu]max(),
which handles the common case of parsing a bounded integer and
exiting with a diagnostic on error.
* gl/lib/xdectoimax.c: The signed variant.
* gl/lib/xdectoint.c: The parameterized implementation.
* gl/lib/xdectoint.h: The interface.
* gl/lib/xdectoumax.c: The unsigned variant.
* bootstrap.conf: Reference the new module.
* cfg.mk (exclude_file_name_regexp--sc_require_config_h_first):
Exclude the parameterized templates.
* src/csplit.c: Output EOVERFLOW or ERANGE errors if appropriate.
* src/fmt.c: Likewise.
* src/fold.c: Likewise.
* src/head.c: Likewise.
* src/ls.c: Likewise.
* src/nl.c: Likewise.
* src/nproc.c: Likewise.
* src/shred.c: Likewise.
* src/shuf.c: Likewise.
* src/stdbuf.c: Likewise.
* src/stty.c: Likewise.
* src/tail.c: Likewise.
* src/truncate.c: Likewise.
* src/split.c: Likewise.
* src/pr.c: Likewise.
* tests/pr/pr-tests.pl: Adjust to avoid matching errno diagnostic.
* tests/fmt/base.pl: Likewise.
* tests/split/l-chunk.sh: Likewise.
* tests/misc/shred-negative.sh: Likewise.
* tests/misc/tail.pl: Likewise. Also remove the redundant
existing ERR_SUBST from test err-6.
* tests/ls/hex-option.sh: Check HEX/OCT options.
* tests/misc/shred-size.sh: Likewise.
* tests/misc/stty-row-col.sh: Likewise.
|
|
glibc <= 2.5 would crash when passed invalid long double values,
therefore internal gnulib routines were used, essentially only by od,
to output such invalid values. Later glibc versions don't crash,
as per https://sourceware.org/bugzilla/show_bug.cgi?id=4586
and subsequently od was adjusted to use the system printf routines
through the use of the ftoastr module with commit v8.7-22-ga71c22f.
Consequently our testing of this feature was moot, and use of
the gnulib printf replacement for printf(1), od(1) and error(3) etc.
was redundant.
* configure.ac (gl_printf_safe): Unset so that we don't check that
"nan" is output for these long double values.
* tests/misc/od-float.sh: Adjust all existing checks to fail if od
exits with failure status (like crashing for example). Add a new case
for one of the problematic invalid long double values for x86_64.
We only check that od exits successfully at present, which may change
if https://sourceware.org/bugzilla/show_bug.cgi?id=17661 is resolved.
|
|
file_t is now mapped to unlabeled_t as per:
http://danwalsh.livejournal.com/68189.html
Therefore use the latter to ensure we match correctly.
This is needed on >= Fedora 21 for example,
while it also works on earlier releases.
|
|
* tests/tail-2/F-vs-missing.sh: Comment with the correct total delay.
* tests/tail-2/F-vs-rename.shi: Likewise.
* tests/tail-2/flush-initial.sh: Likewise.
* tests/tail-2/inotify-hash-abuse.sh: Likewise.
* tests/tail-2/pipe-f2.sh: Likewise.
* tests/misc/chroot-fail.sh: Initialize can_chroot_root in all cases.
|
|
At least the MHz number in /proc/cpuinfo may change, thus leading to
a false positive failure when comparing the expected against the
actual output file. Use an invariant file instead: /proc/version.
* tests/misc/head-c.sh: s/cpuinfo/version/
|
|
* src/chroot.c (is_root): Adjust to compare canonicalized paths
rather than inodes, to handle (return false in) the case where
we have a tree that is constructed by first bind mounting "/"
(thus having the same inode).
(main): Unconditionally call chroot() because it's safer
and of minimal performance benefit to avoid in this case.
This will cause inconsistency with some platforms
not allowing `chroot / true` for non root users.
* tests/misc/chroot-fail.sh: Adjust appropriately.
* NEWS: Mention the bug fixes.
Fixes http://bugs.gnu.org/18736
|
|
* tests/misc/od-j.sh: Non standard comparison order.
* tests/split/b-chunk.sh: Confusing input file name.
* tests/tail-2/tail-c.sh: Redundant require ulimit.
|
|
Fix similar problems in head, od, split, tac, and tail.
Reported by George Shuklin in: http://bugs.gnu.org/18621
* NEWS: Document this.
* src/head.c (elseek): Move up.
(elide_tail_bytes_pipe, elide_tail_lines_pipe): New arg
CURRENT_POS. All uses changed.
(elide_tail_bytes_file, elide_tail_lines_file):
New arg ST and remove arg SIZE. All uses changed.
* src/head.c (elide_tail_bytes_file):
* src/od.c (skip): Avoid optimization for /sys files, where
st_size is bogus and st_size == st_blksize.
Don't report error at EOF when not optimizing.
* src/head.c, src/od.c, src/tail.c: Include "stat-size.h".
* src/split.c (input_file_size): New function.
(bytes_split, lines_chunk_split, bytes_chunk_extract): New arg
INITIAL_READ. All uses changed. Use it to double-check st_size.
* src/tac.c (tac_seekable): New arg FILE_POS. All uses changed.
(copy_to_temp): Return size of temp file. All uses changed.
* src/tac.c (tac_seekable):
* src/tail.c (tail_bytes):
* src/wc.c (wc):
Don't trust st_size; double-check by reading.
* src/wc.c (wc): New arg CURRENT_POS. All uses changed.
* tests/local.mk (all_tests): Add tests/misc/wc-proc.sh,
tests/misc/od-j.sh, tests/tail-2/tail-c.sh.
* tests/misc/head-c.sh:
* tests/misc/tac-2-nonseekable.sh:
* tests/split/b-chunk.sh:
Add tests for problems with /proc and /sys files.
* tests/misc/od-j.sh, tests/misc/wc-proc.sh, tests/tail-2/tail-c.sh:
New files.
|
|
Problem reported by Göran Uddeborg in: http://bugs.gnu.org/18540
* tests/misc/sort.pl: New test 23.
|
|
Problem reported by Vincent Lefevre in: http://bugs.gnu.org/18449
* src/cat.c (main): Allow copying an empty file to itself.
* tests/misc/cat-self.sh: New test.
* tests/local.mk (all_tests): Add it.
|
|
Since commit v8.22-94-g99960ee, chroot(1) skips the chroot(2) syscall
for "/" arguments (and synonyms). The problem is that it also skips
the following chdir("/") call in that case. The latter breaks existing
scripts which expect "/" to be the working directory inside the chroot.
While the first part of the change - i.e., skipping chroot("/") - is
okay for consistency with systems where it might succeed for a non-root
user, the second part might be malicious, e.g.
cd /home/user && chroot '/' bin/foo
In the "best" case, chroot(1) could not execute 'bin/foo' with ENOENT,
but in the worst case, chroot(1) would execute '/home/user/bin/foo' in
the case that exists - instead of '/bin/foo'.
Revert that second part of the patch, i.e., perform the chdir("/)
in the common case again - unless the new --skip-chdir option is
specified. Restrict this new option to the case of "/" arguments.
* src/chroot.c (SKIP_CHDIR): Add enum.
(long_opts): Add entry for the new --skip-chdir option.
(usage): Add --skip-chdir option, and while at it, move the other
to options into alphabetical order.
(main): Accept the above new option, allowing it only in the case
when NEWROOT is the old "/".
Move down the chdir() call after the if-clause to ensure it is
run in any case - unless --skip-chdir is specified.
Add a 'newroot' variable for the new root directory as it is used
in a couple of places now.
* tests/misc/chroot-fail.sh: Invert the last tests which check the
working directory of the execvp()ed program when a "/"-like
argument was passed: now expect it to be "/" - unless --skip-chdir
is given.
* doc/coreutils.texi (chroot invocation): Document the new option.
Document that chroot(1) usually calls chdir("/") unless the new
--skip-chdir option is specified. Sort options.
* NEWS (Changes in behavior): Mention the fix.
(New features): Mention the new option.
* init.cfg (nonroot_has_perm_): Add chroot's new --skip-chdir option.
* tests/cp/preserve-gid.sh (t1): Likewise.
* tests/cp/special-bits.sh: Likewise.
* tests/id/setgid.sh: Likewise.
* tests/misc/truncate-owned-by-other.sh: Likewise.
* tests/mv/sticky-to-xpart.sh: Likewise.
* tests/rm/fail-2eperm.sh: Likewise.
* tests/rm/no-give-up.sh: Likewise.
* tests/touch/now-owned-by-other.sh: Likewise.
Reported by Andreas Schwab in http://bugs.gnu.org/18062
|
|
* tests/misc/env.sh: Skip if we can't execute the generated shebang,
which would be the case if there are spaces in the directory hierarchy.
This is triggered by `make distcheck`
|
|
* tests/misc/od-endian.sh: '\n' is not generally supported
in the replacement, so use the more portable "\\$NL".
|
|
Add the --enable-single-binary option to the configure file.
When enabled, this option builds a single binary file containing
the selected tools. Which tool gets executed depends on the value
of argv[0] which can be set implicitly through symlinks to the
single program.
This setup reduces significantly the size of a complete coreutils
install, since code from lib/libcoreutils.a is not duplicated in
every one of the more than 100 binaries. Runtime overhead is
increased due to more dynamic libraries being loaded, and extra
initialization being performed for all utils. Also initially
a larger binary is loaded from storage, though this is usually
alleviated due to caching and lazy mmaping of unused blocks,
and in fact the single binary should have better caching
characteristics.
Comparing the size of the individual versus single binary on x86_64:
$ cd src
$ size coreutils
$ size -t $(../build-aux/gen-lists-of-programs.sh --list-progs |
grep -Ev '(coreutils|libstdbuf)') | tail -n1
text data bss dec hex filename
1097416 5388 88432 1191236 122d44 src/coreutils
4901010 124964 163768 5189742 4f306e (TOTALS)
Storage requirements are reduced similarly:
$ cd src
$ du -h coreutils
$ du -ch $(../build-aux/gen-lists-of-programs.sh --list-progs |
grep -Ev '(coreutils|libstdbuf)') | tail -n1
1.2M coreutils
5.3M total
When installing, the makefile will create either symlinks or
shebangs based on the --enable-single-binary setting, for
each configured tool. In this way, all the tools are still
callable individually, but they are all implemented by the same
"coreutils" binary installed on the same directory.
* .gitignore: Add new generated files.
* Makefile.am: New rules to generate build-aux/gen-single-binary.sh
and install symlinks.
* NEWS: Mention the new feature.
* README: Add "coreutils" to the list of utils.
* bootstrap.conf: Regenerate src/single-binary.mk
* build-aux/gen-lists-of-programs.sh: New --list-progs option.
* build-aux/gen-single-binary.sh: Regenerate
* configure.ac: New --enable-single-binary option and other variables.
Disallow --enable-single-binary=symlinks with --program-prefix et. al.
* man/coreutils.x: Manpage hook.
* man/local.mk: Add manpage hook and fix dependencies.
* src/coreutils.c: Multicall implementation.
* src/local.mk: New rules for the single binary option.
* tests/local.mk: Add $single_binary_progs to support
require_built_() from init.cfg
* tests/misc/env.sh: Avoid the use of symlink to echo.
* tests/misc/help-version.sh: Add exception for coreutils.
* tests/install/basic-1.sh: Really avoid using ginstall strip
functionality if there is an issue with the independent strip command.
* src/kill.c: Changes to call exit() in main.
* src/readlink.c: Likewise.
* src/shuf.c: Likewise.
* src/timeout.c: Likewise.
* src/truncate.c: Likewise.
|
|
Avoid complicated and error-prone parsing of df's output via
sed(1), cut(1), etc., and instead use df's more modern --output
option.
* src/ioblksize.h (in a comment): Simplify the extraction of the
device name of the mounted file system from df's output.
* tests/dd/skip-seek-past-dev.sh: Likewise.
* tests/du/2g.sh: Likewise for the 'avail' column here.
Also avoid the deprecated use of "tail -NUM".
* tests/misc/stat-mount.sh: While at it, remove the determination
of the mount point of "." via df(1) plus sed(1) as it is unused
since commit v8.5-159-gf57cb37 anyway. Instead, improve this test
by verifying that the output of "stat -c%m ." at least starts with
a slash '/'.
|
|
Revert commit v8.22-131-g3e89d5b as even though POSIX
states that the default mode should be -L,
common practice for stand-alone pwd implementations
is to default to -P.
* src/pwd.c (usage): Retain mention of the default mode of operation.
Suggested by Bob Proulx
|
|
This works for any program that might also be a shell built-in.
It is more concise. In addition, it makes output more reproducible:
some diagnostics include argv[0], which will now be just the program
name, rather than the full absolute name of the executable.
* tests/misc/sort-compress.sh: Use env kill, rather than absolute name.
* tests/install/trap.sh: Likewise.
* tests/misc/timeout.sh: Likewise.
* tests/touch/no-dereference.sh: Do the same for two uses of "test".
* tests/touch/no-create-missing.sh: Likewise.
|
|
* tests/misc/pwd-option.sh (base): Initialize with -P,
now that -L is the default, to accommodate an initial
working directory with a symlink component.
* tests/misc/readlink-fp-loop.sh: Use $(env pwd -P) to get the
absolute working directory. Using "env" ensures we do not invoke
any shell built-in, and PATH ensures we invoke the one from coreutils.
* tests/readlink/can-e.sh: Likewise.
* tests/readlink/can-f.sh: Likewise.
* tests/readlink/can-m.sh: Likewise.
|
|
* src/pwd.c (main): Adjust default mode to be "logical"
and independent of the POSIXLY_CORRECT env var.
(usage): Mention the default mode of operation.
* doc/coreutils.texi (pwd invocation): Adjust accordingly.
* tests/misc/pwd-option.sh: Likewise.
* NEWS: Mention the change in behavior.
|
|
* HACKING: GID is more useful in tests than group name, so rename
input param from NON_ROOT_GROUP to NON_ROOT_GID to make it obvious
that only a group ID is now acceptable, thus allowing GID lookups
to be avoided throughout the tests.
* init.cfg (require_root_): Likewise.
* tests/misc/truncate-owned-by-other.sh: Avoid looking up the GID.
* tests/touch/now-owned-by-other.sh: Likewise.
* tests/misc/chroot-credentials.sh: Likewise. Also fix an instance
of comparison against NON_ROOT_GROUP which would have given a false
failure if a non numeric value was passed in.
* tests/id/setgid.sh: Use previously looked up gid as a more
accurate base for the subsequent adjustment, and move
the uid lookup within chroot, rather than having the overhead
of a separate `id` invocation.
|
|
* src/seq.c (main): Avoid seq_fast() with a start or end of -0.
* tests/misc/seq.pl: Add test cases.
* NEWS: Mention the fix.
Fixes http://bugs.gnu.org/17800
|
|
* src/true.c (main): Add a comment about the possibility
of true returning EXIT_FAILURE due to write failure.
* tests/misc/false-status.sh: Fix so we're testing
the tool and not the shell builtin. Add a case for true(1).
* tests/misc/help-version.sh: Skip /dev/full test
for true as well as false since the exit status is tested separately.
Also remove the iterations for different LC_MESSAGES, as this was only
applied for false(1). Translations are not honored in the test dir
and so would need separate handling in any case.
|
|
commits v8.20-98-g51ce0bf and v8.20-99-gd302aed changed cut(1)
to process each line independently and thus promptly output
each line without buffering. As part of those changes we removed
the special handling of --delimiter=$'\n' --fields=... which
could be used to select arbitrary (ranges of) lines, so as to
simplify and optimize the implementation while also matching the
behavior of different cut(1) implementations.
However that GNU behavior was in place for a long time, and
could be useful in certain cases like making a separated list like
`seq 10 | cut -f1- -d$'\n' --output-delimiter=,` although other tools
like head(1) and paste(1) are more suited to this operation.
This patch reinstates that functionality but restricts the
"line behind" buffering behavior to only the -d$'\n' case.
We also fix the following related edge case to be more consistent:
before> printf "\n" | cut -s -d$'\n' -f1- | wc -l
2
before> printf "\n" | cut -d$'\n' -f1- | wc -l
1
after > printf "\n" | cut -s -d$'\n' -f1- | wc -l
1
after > printf "\n" | cut -d$'\n' -f1- | wc -l
1
* src/cut.c (cut_fields): Adjust as discussed above.
* tests/misc/cut.pl: Likewise.
* NEWS: Mention the change in behavior both for v8.21
and this effective revert.
* cfg.mk (old_NEWS_hash): Adjust for originally omitted v8.21 entry.
* src/paste.c: s/delimeter/delimiter/ comment typo fix.
|
|
* init.cfg (require_root_): Adjust to use chroot, and make
`require_built_ chroot` implicit when chroot used in the test.
* po/POTFILES.in: Remove reference to setuidgid tool.
* src/.gitignore: Likewise.
* src/local.mk: Likewise.
* src/setuidgid.c: Remove.
* tests/cp/preserve-gid.sh: s/setuidgid/chroot --user/.
* tests/cp/special-bits.sh: Likewise.
* tests/id/setgid.sh: Likewise.
* tests/misc/truncate-owned-by-other.sh
* tests/mv/sticky-to-xpart.sh: Likewise.
* tests/rm/fail-2eperm.sh: Likewise.
* tests/rm/no-give-up.sh: Likewise.
* tests/touch/now-owned-by-other.sh: Likewise.
* tests/misc/chroot-fail.sh: Skip if chroot not built.
|
|
* src/chroot.c (is_root): A new helper function to
determine if the passed argument is the root directory
based on inode comparison.
(main): Use the new helper rather than comparing strings.
* tests/misc/chroot-fail.sh: Add cases for alternative root paths.
|
|
It's dangerous and confusing to leave root's supplemental
groups in place when specifying other users with --userspec.
In the edge case that that is desired one can explicitly
specify --groups.
Also we implicitly set the system defined supplemental groups
for a user. The existing mechanism where supplemental groups
needed to be explicitly specified is confusing and not general
when the lookup needs to be done within the chroot.
Also we extend the --groups syntax slightly to allow clearing
the set of supplementary groups using --groups=''.
* src/chroot.c (setgroups): On systems without supplemental groups,
clearing then is a noop and so should return success.
(main): Lookup the primary GID with getpwuid() when just a numeric
uid is specified, and also infer the USERNAME from this call,
needed when we're later looking up the supplemental groups for a user.
Support clearing supplemental groups, either implicitly for
unknown users, or explicitly when --groups='' is specified.
* tests/misc/chroot-credentials.sh: Various new test cases
* doc/coreutils.texi (chroot invocation): Adjust for the new behavior.
* NEWS: Mention the change in behavior.
|
|
This allows chroot to be used as a light weight tool
to change user identification for a command,
while not changing the current working directory.
It also makes `chroot / true` consistently succeed on
all platforms for non root users.
* src/chroot.c (main): If the same root is specified. i.e. '/'
then don't change the current working directory, and avoid the
overhead of the other redundant calls.
* tests/misc/chroot-fail.sh: Remove failure guard previously
needed on some systems. Also add an explicit case to ensure
we don't change directory.
* NEWS: Mention the change in behavior.
|
|
* src/shred.c (main): With the preceding change, shred -s-2 FILE
would write 64KB blocks forever -- or until disk full. This change
makes shred reject a negative size.
* tests/misc/shred-negative.sh: New file.
* tests/local.mk (all_tests): Add it.
|
|
* tests/chmod/c-option.sh: Use `compare /dev/null ... || fail=1`
rather than `test -s ... && fail=1`, so that the file contents
are output, thus improving diagnostics for failing tests.
* tests/cp/acl.sh: Likewise.
* tests/cp/cp-a-selinux.sh: Likewise.
* tests/cp/cp-mv-enotsup-xattr.sh: Likewise.
* tests/cp/reflink-perm.sh: Likewise.
* tests/dd/misc.sh: Likewise.
* tests/misc/env-null.sh: Likewise.
* tests/misc/env.sh: Likewise.
* tests/misc/nice.sh: Likewise.
* tests/misc/nohup.sh: Likewise.
* tests/misc/printenv.sh: Likewise.
* tests/misc/xattr.sh: Likewise.
* tests/mv/update.sh: Likewise.
* tests/rm/deep-2.sh: Likewise.
* tests/rm/read-only.sh: Likewise.
* tests/split/r-chunk.sh: Likewise.
* tests/tail-2/follow-stdin.sh: Likewise.
* tests/tail-2/inotify-race.sh: Likewise.
* tests/tail-2/wait.sh: Likewise.
* tests/touch/no-dereference.sh: Likewise.
* cfg.mk (sc_prohibit_test_empty:): New syntax-check.
* tests/cp/proc-zero-len.sh: Adjust to avoid false syntax-check failure.
* tests/cp/proc-zero-len.sh: Likewise.
* tests/mv/part-symlink.sh: Likewise.
* tests/tail-2/infloop-1.sh: Likewise.
|
|
* src/numfmt.c (setup_padding_buffer): Simplify the code by not
explicitly dealing with heap exhaustion.
(parse_format_string): Likewise. Handle multiple grouping
modifiers as does the standard printf. Handle the new leading
zero --format modifier.
(double_to_human): Use more defensive coding against overwriting
stack buffers. Honor the leading zeros width.
(usage): Mention the leading zero --format modifier.
(main): Allow --padding in combo with a --format (width),
as the number of leading zeros are useful independent of
the main field width.
* doc/coreutils.texi (numfmt invocation): Likewise.
* tests/misc/numfmt.pl: Add new test cases.
* NEWS: Mention the improvement.
|
|
This issue was identified by running the test suite with
http://code.google.com/p/address-sanitizer/
which is included in GCC 4.8 and enabled with -fsanitize=address
This was checked on Fedora 20 with GCC 4.8 as follows:
$ yum install libasan # http://bugzilla.redhat.com/991003
$ rm -f src/ptx.o
$ make check AM_CFLAGS='-fsanitize=address' SUBDIRS=. VERBOSE=yes
$ failure identified in tests/test-suite.log
To see this particular failure triggered with multiple files:
$ src/ptx <(echo a) <(echo a) 2>&1 | asan_symbolize.py -d
=================================================================
==32178==ERROR: AddressSanitizer: heap-buffer-overflow on address
0x60200000e74f at pc 0x435442 bp 0x7fffe8a1b290 sp 0x7fffe8a1b288
READ of size 1 at 0x60200000e74f thread T0
#0 0x435441 in define_all_fields coreutils/src/ptx.c:1425
#1 0x7fa206d31d64 in __libc_start_main ??:?
#2 0x42f77c in _start ??:?
0x60200000e74f is located 1 bytes to the left of 3-byte region
[0x60200000e750,0x60200000e753) allocated by thread T0 here:
#0 0x421809 in realloc ??:?
#1 0x439b4e in fread_file coreutils/lib/read-file.c:97
Shadow bytes around the buggy address:
0x0c047fff9c90: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9ca0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9cb0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9cc0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
0x0c047fff9cd0: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fd fd
=>0x0c047fff9ce0: fa fa 03 fa fa fa fd fd fa[fa]03 fa fa fa 00 00
0x0c047fff9cf0: fa fa 04 fa fa fa 04 fa fa fa fd fa fa fa fd fa
0x0c047fff9d00: fa fa 00 fa fa fa fd fa fa fa 00 fa fa fa 00 fa
0x0c047fff9d10: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
0x0c047fff9d20: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fa
0x0c047fff9d30: fa fa fd fa fa fa 00 fa fa fa 00 fa fa fa 00 fa
Shadow byte legend (one shadow byte represents 8 application bytes):
Addressable: 00
Partially addressable: 01 02 03 04 05 06 07
Heap left redzone: fa
Heap right redzone: fb
Freed heap region: fd
Stack left redzone: f1
Stack mid redzone: f2
Stack right redzone: f3
Stack partial redzone: f4
Stack after return: f5
Stack use after scope: f8
Global redzone: f9
Global init order: f6
Poisoned by user: f7
ASan internal: fe
==32178==ABORTING
The initial report and high level analysis were from Jim Meyering...
"The underlying problem is that swallow_file_in_memory()
is setting the contents of the global text_buffer for the first file,
then updating it (clobbering old value) for the second file.
Yet, some pointers to the initial buffer have been squirreled away
and later, one of them (keyafter) is presumed to point into
the new "text_buffer", which it does not. The subsequent
SKIP_WHITE_BACKWARDS use backs up "cursor" and goes out of bounds."
* src/ptx.c (text_buffers): Maintain references for the limits of each
buffer corresponding to each file, rather than just the last processed.
(struct OCCURS): Add a member to map back to the corresponding file.
Note normally this could be computed from the "reference" member
rather than needing the extra storage, however this is not possible
when in --references mode.
(find_occurs_in_text): Reference the array rather than a single entry.
(define_all_fields): Likewise. Also avoid computing the file index
since this is now stored directly.
(main): Update text_buffers[] array rather than a single text_buffer.
* tests/misc/ptx-overrun.sh: Even though this issue is already triggered
with AddressSanitizer, add a new case to demonstrate the whitespace
trimming issue, and to trigger without AddressSanitizer.
Fixes https://bugs.gnu.org/16171
|
