summaryrefslogtreecommitdiff
path: root/tests/misc/sort-NaN-infloop
AgeCommit message (Collapse)Author
2011-09-27sort: avoid a NaN-induced infloopJim Meyering
These commands would fail to terminate: yes -- -nan | head -156903 | sort -g > /dev/null echo nan > F; sort -m -g F F That can happen with any strtold implementation that includes uninitialized data in its return value. The problem arises in the mergefps function when bubble-sorting the two or more lines, each from one of the input streams being merged: compare(a,b) returns 64, yet compare(b,a) also returns a positive value. With a broken comparison function like that, the bubble sort never terminates. Why do the long-double bit strings corresponding to two identical "nan" strings not compare equal? Because some parts of the result are uninitialized and thus depend on the state of the stack. For more details, see http://bugs.gnu.org/9612. * src/sort.c (nan_compare): New function. (general_numcompare): Use it rather than bare memcmp. Reported by Aaron Denney in http://bugs.debian.org/642557. * NEWS (Bug fixes): Mention it. * tests/misc/sort-NaN-infloop: New file. * tests/Makefile.am (TESTS): Add it.