[ast-developers] ast-ksh.2013-09-24 uniq(1) hang in command substitution... / was: Re: Shift state/|mbinit()|/etc. fixes for GB18030 vs. \u[hex]/\w[hex] ...

[Roland Mainz]

Attached (as "astksh20130913_gb18030_mbinit_fixes001.diff.txt") is a
patch which fixes some issues related to shift state handling which
were found during testing with \u[hex] and \w[hex] ...
- The Euro symbol (Unicode codepoint 0x20ac) may or may not return
|true| for |iswalpha(wchar_euro)| ... this was the issue where the
"wchar.sh" test was tripping over with "zh_CN.GB18030" on Solaris
11/B145

The following testcase "hangs" on Solaris 11/B145/AMD64/64bit ... with
LC_ALL="en_US.UTF-8" as local
-- snip --
# test 2a: Edward Pilatowicz <edward.pilatowicz at sun.com>'s
Solaris-specific testcase
# prechecks
[[ -f "/etc/termcap" ]] || print -u2 "/etc/termcap missing"

# common functions/variables
function isvalidpid
{
kill -n 0 ${1} 2>/dev/null && return 0
return 1
}

tmpfile='mytmp.txt'

${SHELL} -o errexit -c \
'builtin uniq ; set -- `cat /etc/termcap | sort | uniq` ; true'
2>"${tmpfile}" >/dev/null &
(( childpid=$! ))
sleep 15
sync
if isvalidpid ${childpid} ; then
print -u2 "test2a: child (pid=${childpid}) still busy."
pstack ${childpid}
kill -s KILL ${childpid} 2>/dev/null
fi

wait
(( exitcode=$? ))

if (( exitcode != 0 )) ; then
# wait for child (and/or avoid zombies/slime)
print -u2 "test2a: Child returned non-zero exit code ${exitcode}."
fi
-- snip --
... the output looks like this:
-- snip --
$ (export SHELL=~/bin/ksh ; $SHELL test.sh )
test2a: child (pid=11755) still busy.
11755: /home/test001/bin/ksh -o errexit -c builtin uniq ; set -- `cat /etc/te
fffffd7fff24b60a write (1, fffffd7fff036c40, 43)
00000000005d5397 sfwr () + 5d7
00000000005b4756 _sfflsbuf () + 526
00000000005d5d4e sfwrite () + 71e
0000000000637d06 uniq () + 786
000000000063853e b_uniq () + 57e
00000000004c836d sh_exec () + 284d
00000000004ca92f sh_exec () + 4e0f
00000000004cb7e9 sh_exec () + 5cc9
00000000004be63c sh_subshell () + c3c
0000000000485967 comsubst () + 1157
000000000047cbe3 copyto () + d73
000000000047a9c7 sh_macexpand () + 4c7
000000000051cc15 arg_expand () + 1a5
000000000051c152 sh_argbuild () + 172
00000000004c60ad sh_exec () + 58d
00000000004cbba7 sh_exec () + 6087
00000000004434f7 exfile () + 1147
0000000000442372 sh_main () + 1802
0000000000440a22 main () + 92
000000000044083c ???????? ()
-- snip --

A more detailed stack trace looks like this:
-- snip --
Attached to process 11876
stopped in __write at 0xfffffd7fff24b60a
0xfffffd7fff24b60a: __write+0x000a: jae __write+0x16 [
0xfffffd7fff24b616, .+0xc ]
Current function is sfwr
209 if((w = syswritef(f->file,buf,n)) > 0)
[1] __write(0x1, 0xfffffd7fff036c40, 0x43, 0x0, 0x6d28c0, 0x43), at
0xfffffd7fff24b60a
[2] write(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff23109f
=>[3] sfwr(f = 0x6d28c0, buf = 0xfffffd7fff036c40, n = 67U, disc =
0xfffffd7fff050eb0), line 209 in "sfwr.c"
[4] _sfflsbuf(f = 0x6d28c0, c = -1), line 97 in "sfflsbuf.c"
[5] sfwrite(f = 0x6d28c0, buf = 0xfffffd7fff036c40, n = 0), line 149
in "sfwrite.c"
[6] uniq(fdin = 0x6d2810, fdout = 0x6d28c0, fields = 0, chars = 0,
width = -1, mode = 0, all = (nil), compare = 0x43f6c0), line 209 in
"uniq.c"
[7] b_uniq(argc = 1, argv = 0xfffffd7fff04cca8, context = 0x6cefd0),
line 338 in "uniq.c"
[8] sh_exec(shp = 0x6ceae8, t = 0xfffffd7fff04cc40, flags = 4), line
1379 in "xec.c"
[9] sh_exec(shp = 0x6ceae8, t = 0xfffffd7fff04ccb0, flags = 5), line
1984 in "xec.c"
[10] sh_exec(shp = 0x6ceae8, t = 0xfffffd7fff04ccb0, flags = 5),
line 2203 in "xec.c"
[11] sh_subshell(shp = 0x6ceae8, t = 0xfffffd7fff04ccf0, flags = 5,
comsub = 1), line 603 in "subshell.c"
[12] comsubst(mp = 0xfffffd7fff024820, t = 0xfffffd7fff04ccf0, type
= 1), line 2164 in "macro.c"
[13] copyto(mp = 0xfffffd7fff024820, endch = 0, newquote = 0), line
601 in "macro.c"
[14] sh_macexpand(shp = 0x6ceae8, argp = 0xfffffd7fff04c940, arghead
= 0xfffffd7fffdff358, flag = 0), line 245 in "macro.c"
[15] arg_expand(shp = 0x6ceae8, argp = 0xfffffd7fff04c940, argchain
= 0xfffffd7fffdff358, flag = 0), line 1112 in "args.c"
[16] sh_argbuild(shp = 0x6ceae8, nargs = 0xfffffd7fffdff4ec, comptr
= 0xfffffd7fff04c8e0, flag = 0), line 950 in "args.c"
[17] sh_exec(shp = 0x6ceae8, t = 0xfffffd7fff04c8e0, flags = 4),
line 992 in "xec.c"
[18] sh_exec(shp = 0x6ceae8, t = 0xfffffd7fff04ca10, flags = 5),
line 2252 in "xec.c"
[19] exfile(shp = 0x6ceae8, iop = 0xfffffd7fff0224d0, fno = -1),
line 610 in "main.c"
[20] sh_main(ac = 5, av = 0xfffffd7fffdffa48, userinit = (nil)),
line 382 in "main.c"
[21] main(argc = 5, argv = 0xfffffd7fffdffa48), line 45 in "pmain.c"
detaching from process 11876
-- snip -

----

Bye,
Roland

[Roland Mainz]

Post by Roland Mainz

Attached (as "astksh20130913_gb18030_mbinit_fixes001.diff.txt") is a
patch which fixes some issues related to shift state handling which
were found during testing with \u[hex] and \w[hex] ...
- The Euro symbol (Unicode codepoint 0x20ac) may or may not return
|true| for |iswalpha(wchar_euro)| ... this was the issue where the
"wchar.sh" test was tripping over with "zh_CN.GB18030" on Solaris
11/B145

The following testcase "hangs" on Solaris 11/B145/AMD64/64bit ... with
LC_ALL="en_US.UTF-8" as local
-- snip --
# test 2a: Edward Pilatowicz <edward.pilatowicz at sun.com>'s
Solaris-specific testcase
# prechecks
[[ -f "/etc/termcap" ]] || print -u2 "/etc/termcap missing"
# common functions/variables
function isvalidpid
{
kill -n 0 ${1} 2>/dev/null && return 0
return 1
}
tmpfile='mytmp.txt'
${SHELL} -o errexit -c \
'builtin uniq ; set -- `cat /etc/termcap | sort | uniq` ; true'
2>"${tmpfile}" >/dev/null &
(( childpid=$! ))
sleep 15
sync
if isvalidpid ${childpid} ; then
print -u2 "test2a: child (pid=${childpid}) still busy."
pstack ${childpid}
kill -s KILL ${childpid} 2>/dev/null
fi
wait
(( exitcode=$? ))
if (( exitcode != 0 )) ; then
# wait for child (and/or avoid zombies/slime)
print -u2 "test2a: Child returned non-zero exit code ${exitcode}."
fi
-- snip --
-- snip --

[snip]

Post by Roland Mainz
-- snip --
-- snip --
Attached to process 11876
stopped in __write at 0xfffffd7fff24b60a
0xfffffd7fff24b60a: __write+0x000a: jae __write+0x16 [
0xfffffd7fff24b616, .+0xc ]
Current function is sfwr
209 if((w = syswritef(f->file,buf,n)) > 0)
[1] __write(0x1, 0xfffffd7fff036c40, 0x43, 0x0, 0x6d28c0, 0x43), at
0xfffffd7fff24b60a
[2] write(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xfffffd7fff23109f
=>[3] sfwr(f = 0x6d28c0, buf = 0xfffffd7fff036c40, n = 67U, disc =
0xfffffd7fff050eb0), line 209 in "sfwr.c"
[4] _sfflsbuf(f = 0x6d28c0, c = -1), line 97 in "sfflsbuf.c"
[5] sfwrite(f = 0x6d28c0, buf = 0xfffffd7fff036c40, n = 0), line 149
in "sfwrite.c"
[6] uniq(fdin = 0x6d2810, fdout = 0x6d28c0, fields = 0, chars = 0,
width = -1, mode = 0, all = (nil), compare = 0x43f6c0), line 209 in
"uniq.c"
[7] b_uniq(argc = 1, argv = 0xfffffd7fff04cca8, context = 0x6cefd0),
line 338 in "uniq.c"

[snip]

Post by Roland Mainz
-- snip -

SuSE 12.3/AMD64/64bit Linux hangs for the same testcase in the same place:
-- snip --
(gdb) #0 0x00007f15f775a9c0 in __write_nocancel () at
../sysdeps/unix/syscall-template.S:81
#1 0x000000000051bf0a in sfwr (f=0x809060 <_Sfstdout>,
buf=0x7f15f8247d00, n=53, disc=0x7f15f8261df0)
at /home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/lib/libast/sfio/sfwr.c:209
#2 0x0000000000504b7a in _sfflsbuf (f=0x809060 <_Sfstdout>, c=-1) at
/home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/lib/libast/sfio/sfflsbuf.c:97
#3 0x000000000051c661 in sfwrite (f=0x809060 <_Sfstdout>,
buf=0x7f15f8247d00, n=0)
at /home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/lib/libast/sfio/sfwrite.c:149
#4 0x000000000055dc3e in uniq (fdin=0x808fa0 <_Sfstdin>,
fdout=0x809060 <_Sfstdout>, fields=0, chars=0, width=-1, mode=0,
all=0x0, compare=0x7f15f77c9060 <__memcmp_sse4_1>)
at /home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/lib/libcmd/uniq.c:209
#5 0x000000000055e1e9 in b_uniq (argc=1, argv=0x7f15f825dc68,
context=0x80b890 <sh+1392>) at
/home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/lib/libcmd/uniq.c:338
#6 0x000000000046998c in sh_exec (shp=0x80b320 <sh>,
t=0x7f15f825dc00, flags=4) at
/home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/xec.c:1379
#7 0x000000000046ba67 in sh_exec (shp=0x80b320 <sh>,
t=0x7f15f825dc70, flags=5) at
/home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/xec.c:1984
#8 0x000000000046c758 in sh_exec (shp=0x80b320 <sh>,
t=0x7f15f825dc70, flags=5) at
/home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/xec.c:2203
#9 0x000000000046284d in sh_subshell (shp=0x80b320 <sh>,
t=0x7f15f825dcb0, flags=5, comsub=1)
at /home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/subshell.c:603
#10 0x000000000043cbef in comsubst (mp=0x7f15f82358e0,
t=0x7f15f825dcb0, type=1) at
/home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/macro.c:2164
#11 0x0000000000436cdc in copyto (mp=0x7f15f82358e0, endch=0,
newquote=0) at /home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/macro.c:601
#12 0x00000000004356c0 in sh_macexpand (shp=0x80b320 <sh>,
argp=0x7f15f825d900, arghead=0x7ffff8892580, flag=0)
at /home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/macro.c:245
#13 0x00000000004a411e in arg_expand (shp=0x80b320 <sh>,
argp=0xfffffffffffffe00, argchain=0x7ffff8892580, flag=0)
at /home/test001/work/ast_ksh_20130924/build_i386_64bit_debug/src/cmd/ksh93/sh/args.c:1112
[snip]
-- snip --

----

Bye,
Roland