summaryrefslogtreecommitdiff
path: root/tests/misc/cut-huge-range.sh
AgeCommit message (Collapse)Author
2016-04-21tests: avoid potential overflows with `expr`Pádraig Brady
* cfg.mk (sc_prohibit_expr_unsigned): expr uses intmax_t internally when GMP is not available, so flag any uses of unsigned limits. * tests/misc/timeout-parameters.sh: Remove the overflow related to UINT_MAX as it's handled by the following case. Change the following case from 99... to $TIME_T_OFLOW to be more expressive in what it's actually testing. Directly check that commands succeed, rather than inspecting $? afterwards. * tests/dd/skip-seek-past-dev.sh: Guard against large device sizes. * tests/id/setgid.sh: Protect: Guard against large group IDs. * tests/misc/cut-huge-range.sh: Tweak comment to avoid syntax check. It's tempting to simplify to just skip the test if expr fails, but SIZE_MAX > INTMAX_MAX is the common case.
2016-01-01maint: update all copyright year number rangesPádraig Brady
Run "make update-copyright" and then... * gnulib: Update to latest with copyright year adjusted. * tests/init.sh: Sync with gnulib to pick up copyright year. * bootstrap: Likewise. * tests/sample-test: Adjust to use the single most recent year.
2015-12-17tests: avoid intermittent ulimit -v failuresPádraig Brady
* init.cfg (get_min_ulimit_v_): Increase the determined virtual memory limit for the command by a page size to avoid alignment issues causing false failures for subsequent runs. * tests/misc/cut-huge-range.sh: Be defensive and match the cut invocations under test, more closely with the form used to determine the VM memory limit. This was noticed on i686 linux at: http://hydra.nixos.org/build/28990456
2015-10-08tests: adjust recent changes to virtual memory limitsPádraig Brady
* tests/dd/no-allocate.sh: Account for timeout(1) when determining the required mem, as timeout has additional shared libs. This avoids the need for the hardcoded 4M addition to the limit. * tests/misc/head-c.sh: Increase the base limit, to account for the fact that head(1) will allocate some additional mem in this case. * tests/misc/cut-huge-range.sh: Remove mention of specific limits. * tests/misc/printf-surprise.sh: Likewise. Reported by Dmitry V. Levin
2015-09-22maint: use adaptive approach for `ulimit -v` based testsBernhard Voelker
When configured with either 'symlinks' or 'shebangs' as value for the --enable-single-binary option, tests based on `ulimit -v` are skipped. The reason is that the multicall 'coreutils' binary requires much more memory due to shared libraries being loaded, and the size of the 'date' binary (~290KiB) compared to the multicall binary (~5MiB), of course. Finally, in the case of 'shebangs', the starting shell requires more memory, too Instead of using hard-coded values for the memory limit, use an adaptive approach: first determine the amount of memory for a similar, yet more trivial invocation of the command, and then do the real test run using that limit (plus some buffer in some cases). * init.cfg (require_ulimit_v_): Remove function. (get_min_ulimit_v_): Add function to determine the minimum memory limit required for a given command in an adaptive way. * cfg.mk (sc_prohibit_test_ulimit_without_require_): Change the name of the above function in the syntax-check rule. * tests/cp/link-heap.sh: Use the above function to determine the minimum memory required to run a command simpler than in the real test run. Use that limit plus a buffer there. While at it, change to list of commands in the subshell to fail also if the beginning `ulimit -v` fails. * tests/dd/no-allocate.sh: Likewise. * tests/misc/csplit-heap.sh: Likewise. * tests/misc/cut-huge-range.sh: Likewise. * tests/misc/head-c.sh: Likewise. * tests/misc/printf-surprise.sh: Likewise. * tests/split/line-bytes.sh: Likewise. * tests/rm/many-dir-entries-vs-OOM.sh: Likewise - doing it separately for each program under test.
2015-01-14tests: add extra protection against unexpected exitsPádraig Brady
Many tests use `program ... && fail=1` to ensure expected error situations are indicated. However that would mask an unexpected exit (like a crash). Therefore explicitly check the expected exit code. Note where error messages are also verified, the extra protection is not added. * tests/init.sh (returns_): A new helper function to check the return code of a command, and used throughout the tests. * cfg.mk (sc_prohibit_and_fail_1): Add a syntax check to avoid new instances of this issue.
2015-01-01maint: update all copyright year number rangesPádraig Brady
Run "make update-copyright" and then... * tests/sample-test: Adjust to use the single most recent year. * tests/du/bind-mount-dir-cycle-v2.sh: Fix case in copyright message, so that year is updated automatically in future.
2014-01-02maint: update all copyright year number rangesBernhard Voelker
Run "make update-copyright", but then also run this, perl -pi -e 's/2\d\d\d-//' tests/sample-test to make that one script use the single most recent year number.
2013-05-18tests: don't assume expr was built with GMPPaul Eggert
* tests/misc/cut-huge-range.sh (subtract_one): New string. (CUT_MAX): Don't pass a too-large integer to 'expr'.
2013-05-08cut: improve performance, especially with --output-delimiterCojocaru Alexandru
Use a sentinel value that's checked implicitly, rather than a bit array, to determine if an item should be output. Benchmark results for this change are: $ yes abcdfeg | head -n1MB > big-file $ for c in orig sentinel; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 big-file > /dev/null done == orig == real 0m0.049s user 0m0.044s sys 0m0.005s == sentinel == real 0m0.035s user 0m0.032s sys 0m0.002s ## Again with --output-delimiter ## $ for c in orig sentinel; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 --output-delimiter=: big-file > /dev/null done == orig == real 0m0.106s user 0m0.103s sys 0m0.002s == sentinel == real 0m0.055s user 0m0.052s sys 0m0.003s eol_range_start: Removed. 'n-' is no longer treated specially, and instead SIZE_MAX is set for the 'hi' limit, and tested implicitly. complement_rp: Used to complement 'rp' when '--complement' is specified. ADD_RANGE_PAIR: Macro renamed to 'add_range_pair' function. * tests/misc/cut-huge-range.sh: Adjust to the SENTINEL value. Also remove the overlapping range test as this is no longer dependent on large ranges and also is already handled with the EOL-subsumed-3 test in cut.pl.
2013-05-07cut: fix handling of overlapping rangesCojocaru Alexandru
This issue was introduced in commit v8.21-43-g3e466ad * src/cut.c (set_fields): Process all range pairs when merging. * tests/misc/cut-huge-range.sh: Add a test for this edge case. Also fix an issue where we could miss reported errors due to truncation of the 'err' file.
2013-04-29cut: reduce CPU overhead in determining item to outputCojocaru Alexandru
print_kth() is the central function of cut used to determine if an item is to be output or not, so simplify it by moving some logic outside. Benchmark results for this change are: $ yes abcdfeg | head -n1MB > big-file $ for c in orig split; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 big-file > /dev/null done == orig == real 0m0.111s user 0m0.108s sys 0m0.002s == split == real 0m0.088s user 0m0.081s sys 0m0.007s * src/cut.c (print_kth): Refactor a branch to outside the function. Related to http://bugs.gnu.org/13127
2013-04-29cut: make memory allocation independent of range widthCojocaru Alexandru
The current implementation of cut, uses a bit array, an array of `struct range_pair's, and (when --output-delimiter is specified) a hash_table. The new implementation will use only an array of `struct range_pair's. The old implementation is memory inefficient because: 1. When -b with a big num is specified, it allocates a lot of memory for `printable_field'. 2. When --output-delimiter is specified, it will allocate 31 buckets. Even if only a few ranges are specified. Note CPU overhead is increased to determine if an item is to be printed, as shown by: $ yes abcdfeg | head -n1MB > big-file $ for c in with-bitarray without-bitarray; do src/cut-$c 2>/dev/null echo -ne "\n== $c ==" time src/cut-$c -b1,3 big-file > /dev/null done == with-bitarray == real 0m0.084s user 0m0.078s sys 0m0.006s == without-bitarray == real 0m0.111s user 0m0.108s sys 0m0.002s Subsequent patches will reduce this overhead. * src/cut.c (set_fields): Set and initialize RP instead of printable_field. * src/cut.c (is_range_start_index): Use CURRENT_RP rather than a hash. * tests/misc/cut.pl: Check if `eol_range_start' is set correctly. * tests/misc/cut-huge-range.sh: Rename from cut-huge-to-eol-range.sh, and add a test to verify large amounts of mem aren't allocated. Fixes http://bugs.gnu.org/13127