Discussion:
[ast-developers] Strings in backticks cannot be longer than 65536 characters
Dr. Werner Fink
2014-07-16 09:01:58 UTC
Permalink
Hi,

with the following lines the ksh93u upto ksh93v- 2014-06-25 do hang:

dd if=/dev/zero of=testfile.txt bs=1 count=65537
x=`cat testfile.txt | cat -`

... it works with count=65536 and/or

x=$(cat testfile.txt | cat -)

but AFAIK the backticks are very common and also supported.

Btw: Even a Ctrl-C does *not* interrupt the running ksh:

noether:~ # strace -p 31289
Process 31289 attached - interrupt to quit
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)
--- {si_signo=SIGINT, si_code=SI_KERNEL, si_value={int=1060230178, ptr=0x7f943f31d422}} (Interrupt) ---
rt_sigreturn(0x7d6ef8) = -1 EINTR (Interrupted system call)
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)



Werner
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140716/450f15b5/attachment.sig>
Dr. Werner Fink
2014-07-18 14:05:32 UTC
Permalink
Post by Dr. Werner Fink
Hi,
dd if=/dev/zero of=testfile.txt bs=1 count=65537
x=`cat testfile.txt | cat -`
... it works with count=65536 and/or
x=$(cat testfile.txt | cat -)
but AFAIK the backticks are very common and also supported.
noether:~ # strace -p 31289
Process 31289 attached - interrupt to quit
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)
--- {si_signo=SIGINT, si_code=SI_KERNEL, si_value={int=1060230178, ptr=0x7f943f31d422}} (Interrupt) ---
rt_sigreturn(0x7d6ef8) = -1 EINTR (Interrupted system call)
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)
After debugging I found that the following change

---------------------------* snip *---------------------------------------
--- ksh93/src/cmd/ksh93/sh/macro.c
+++ ksh93/src/cmd/ksh93/sh/macro.c 2014-07-18 13:50:47.590235743 +0000
@@ -2198,7 +2198,7 @@ static void comsubst(Mac_t *mp,register
mp->shp->inlineno = error_info.line+mp->shp->st.firstline;
t = (Shnode_t*)sh_parse(mp->shp, sp,SH_EOF|SH_NL);
mp->shp->inlineno = c;
- type = 1;
+ type = 3;
}
#if KSHELL
if(t)
---------------------------* snap *---------------------------------------

would fix the problem. The question rises if this will cause other
problems? AFAICS this causes that the subshell is using SH_PIPEFAIL
and a smaller buffer for stdout as well as does not wait on the job.

Werner
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140718/ff59cc60/attachment.sig>
Dr. Werner Fink
2014-07-18 14:22:47 UTC
Permalink
Post by Dr. Werner Fink
Post by Dr. Werner Fink
Hi,
dd if=/dev/zero of=testfile.txt bs=1 count=65537
x=`cat testfile.txt | cat -`
... it works with count=65536 and/or
x=$(cat testfile.txt | cat -)
but AFAIK the backticks are very common and also supported.
noether:~ # strace -p 31289
Process 31289 attached - interrupt to quit
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)
--- {si_signo=SIGINT, si_code=SI_KERNEL, si_value={int=1060230178, ptr=0x7f943f31d422}} (Interrupt) ---
rt_sigreturn(0x7d6ef8) = -1 EINTR (Interrupted system call)
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)
After debugging I found that the following change
---------------------------* snip *---------------------------------------
--- ksh93/src/cmd/ksh93/sh/macro.c
+++ ksh93/src/cmd/ksh93/sh/macro.c 2014-07-18 13:50:47.590235743 +0000
@@ -2198,7 +2198,7 @@ static void comsubst(Mac_t *mp,register
mp->shp->inlineno = error_info.line+mp->shp->st.firstline;
t = (Shnode_t*)sh_parse(mp->shp, sp,SH_EOF|SH_NL);
mp->shp->inlineno = c;
- type = 1;
+ type = 3;
}
#if KSHELL
if(t)
---------------------------* snap *---------------------------------------
would fix the problem. The question rises if this will cause other
problems? AFAICS this causes that the subshell is using SH_PIPEFAIL
and a smaller buffer for stdout as well as does not wait on the job.
Indeed:

test basic begins at 2014-07-18+14:20:30
basic.sh[447]: ``command substitution background process output error -- got 'fore', expected 'foreback'
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140718/361c47f1/attachment.sig>
Dr. Werner Fink
2014-07-18 14:34:18 UTC
Permalink
Post by Dr. Werner Fink
Post by Dr. Werner Fink
Hi,
dd if=/dev/zero of=testfile.txt bs=1 count=65537
x=`cat testfile.txt | cat -`
... it works with count=65536 and/or
x=$(cat testfile.txt | cat -)
but AFAIK the backticks are very common and also supported.
noether:~ # strace -p 31289
Process 31289 attached - interrupt to quit
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)
--- {si_signo=SIGINT, si_code=SI_KERNEL, si_value={int=1060230178, ptr=0x7f943f31d422}} (Interrupt) ---
rt_sigreturn(0x7d6ef8) = -1 EINTR (Interrupted system call)
write(1, "\0", 1) = ? ERESTARTSYS (To be restarted)
After debugging I found that the following change
---------------------------* snip *---------------------------------------
--- ksh93/src/cmd/ksh93/sh/macro.c
+++ ksh93/src/cmd/ksh93/sh/macro.c 2014-07-18 13:50:47.590235743 +0000
@@ -2198,7 +2198,7 @@ static void comsubst(Mac_t *mp,register
mp->shp->inlineno = error_info.line+mp->shp->st.firstline;
t = (Shnode_t*)sh_parse(mp->shp, sp,SH_EOF|SH_NL);
mp->shp->inlineno = c;
- type = 1;
+ type = 3;
}
#if KSHELL
if(t)
---------------------------* snap *---------------------------------------
but this causes trouble:

test basic begins at 2014-07-18+14:20:30
basic.sh[447]: ``command substitution background process output error -- got 'fore', expected 'foreback'
test basic failed at 2014-07-18+14:21:11 with exit code 1 [ 110 tests 1 error ]
test basic(C.UTF-8) begins at 2014-07-18+14:21:11
basic.sh[447]: ``command substitution background process output error -- got 'fore', expected 'foreback'
test basic(C.UTF-8) failed at 2014-07-18+14:21:52 with exit code 1 [ 110 tests 1 error ]
test basic(shcomp) begins at 2014-07-18+14:21:52
shcomp-basic.ksh[447]: ``command substitution background process output error -- got 'fore', expected 'foreback'
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140718/b7a57ea4/attachment.sig>
David Korn
2014-07-24 15:35:01 UTC
Permalink
I finally got a chance to look at this.

The problem is that the standard is ambiguous about when a command
substitution ends.

The bourne shell used a pipe for command substitution and command
substitution terminated when the pipe closed. Thus, if you had a command
the wrote on standard output, forked and exited and the child wrote on that
pipe, the output from both processes was part of the command substitution.

ksh considered the command substitution done when the command completed and
used memory for files less that 64K(IOBSIZE) and this created a tmp file
if it was larger.

I tried to restore the old bourne shell semantics for `` so that the
foreback test would work. However, this caused the example you have to
hang. Here is a change which should fix this by using pipes for all `` not
just large ones. Here is the one line patch:

--- old/sh/subshell.c Wed Apr 2 09:57:00 2014
+++new/sh/subshell.c Thu Jul 24 11:19:43 2014
@@ -593,7 +574,7 @@
sp->tmpfd = -1;
sp->pipefd = -1;
/* use sftmp() file for standard output */
- if(!(iop = sftmp(comsub==1?PIPE_BUF:IOBSIZE)))
+ if(!(iop = sftmp(comsub==1?0:IOBSIZE)))
{
sfswap(sp->saveout,sfstdout);

errormsg(SH_DICT,ERROR_system(1),e_tmpcreate);



Let me know.
Post by Dr. Werner Fink
Post by Dr. Werner Fink
Post by Dr. Werner Fink
Hi,
dd if=/dev/zero of=testfile.txt bs=1 count=65537
x=`cat testfile.txt | cat -`
... it works with count=65536 and/or
x=$(cat testfile.txt | cat -)
but AFAIK the backticks are very common and also supported.
noether:~ # strace -p 31289
Process 31289 attached - interrupt to quit
write(1, "\0", 1) = ? ERESTARTSYS (To be
restarted)
Post by Dr. Werner Fink
Post by Dr. Werner Fink
--- {si_signo=SIGINT, si_code=SI_KERNEL, si_value={int=1060230178,
ptr=0x7f943f31d422}} (Interrupt) ---
Post by Dr. Werner Fink
Post by Dr. Werner Fink
rt_sigreturn(0x7d6ef8) = -1 EINTR (Interrupted
system call)
Post by Dr. Werner Fink
Post by Dr. Werner Fink
write(1, "\0", 1) = ? ERESTARTSYS (To be
restarted)
Post by Dr. Werner Fink
After debugging I found that the following change
---------------------------* snip
*---------------------------------------
Post by Dr. Werner Fink
--- ksh93/src/cmd/ksh93/sh/macro.c
+++ ksh93/src/cmd/ksh93/sh/macro.c 2014-07-18 13:50:47.590235743 +0000
@@ -2198,7 +2198,7 @@ static void comsubst(Mac_t *mp,register
mp->shp->inlineno = error_info.line+mp->shp->st.firstline;
t = (Shnode_t*)sh_parse(mp->shp, sp,SH_EOF|SH_NL);
mp->shp->inlineno = c;
- type = 1;
+ type = 3;
}
#if KSHELL
if(t)
---------------------------* snap
*---------------------------------------
test basic begins at 2014-07-18+14:20:30
basic.sh[447]: ``command substitution background process output
error -- got 'fore', expected 'foreback'
test basic failed at 2014-07-18+14:21:11 with exit code 1 [ 110 tests 1 error ]
test basic(C.UTF-8) begins at 2014-07-18+14:21:11
basic.sh[447]: ``command substitution background process output
error -- got 'fore', expected 'foreback'
test basic(C.UTF-8) failed at 2014-07-18+14:21:52 with exit code 1 [ 110 tests 1 error ]
test basic(shcomp) begins at 2014-07-18+14:21:52
shcomp-basic.ksh[447]: ``command substitution background process
output error -- got 'fore', expected 'foreback'
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140724/21269557/attachment.html>
Dr. Werner Fink
2014-07-25 08:04:00 UTC
Permalink
Post by David Korn
I finally got a chance to look at this.
The problem is that the standard is ambiguous about when a command
substitution ends.
The bourne shell used a pipe for command substitution and command
substitution terminated when the pipe closed. Thus, if you had a command
the wrote on standard output, forked and exited and the child wrote on that
pipe, the output from both processes was part of the command substitution.
ksh considered the command substitution done when the command completed and
used memory for files less that 64K(IOBSIZE) and this created a tmp file
if it was larger.
I tried to restore the old bourne shell semantics for `` so that the
foreback test would work. However, this caused the example you have to
hang. Here is a change which should fix this by using pipes for all `` not
--- old/sh/subshell.c Wed Apr 2 09:57:00 2014
+++new/sh/subshell.c Thu Jul 24 11:19:43 2014
@@ -593,7 +574,7 @@
sp->tmpfd = -1;
sp->pipefd = -1;
/* use sftmp() file for standard output */
- if(!(iop = sftmp(comsub==1?PIPE_BUF:IOBSIZE)))
+ if(!(iop = sftmp(comsub==1?0:IOBSIZE)))
{
sfswap(sp->saveout,sfstdout);
errormsg(SH_DICT,ERROR_system(1),e_tmpcreate);
Let me know.
This works for the specific code but break other tests,
from ksh test suite

[ 1162s] test subshell begins at 2014-07-25+07:39:05
[ 1175s] foo
[ 1175s] subshell.sh[642]: nested command substitution with 2>&1 not working
[ 1178s] test subshell failed at 2014-07-25+07:39:21 with exit code 1 [ 70 tests 1 error ]
[ 1178s] test subshell(C.UTF-8) begins at 2014-07-25+07:39:21
[ 1194s] foo
[ 1194s] subshell.sh[642]: nested command substitution with 2>&1 not working
[ 1197s] test subshell(C.UTF-8) failed at 2014-07-25+07:39:40 with exit code 1 [ 70 tests 1 error ]
[ 1197s] test subshell(shcomp) begins at 2014-07-25+07:39:40
[ 1211s] foo
[ 1211s] shcomp-subshell.ksh[642]: nested command substitution with 2>&1 not working
[ 1214s] test subshell(shcomp) failed at 2014-07-25+07:39:57 with exit code 1 [ 70 tests 1 error ]

from my own test suite

[ 1233s] ++ /tmp/ksh-build.D5gJUl/bin/ksh -k -c 'd=`/bin/echo x y=z`; echo $d x y=z'
[ 1233s] + result='x
[ 1233s] x'
[ 1233s] + test 'x
[ 1233s] x' = 'x x'
[ 1233s] + exit 1

also further tests do fail

result=$(/tmp/ksh-build.D5gJUl/bin/ksh -c 'echo | echo "x`/bin/echo y`"')
echo "$result"
x
y
result=$(/tmp/ksh-build.D5gJUl/bin/ksh -c 'echo | echo "x$(/bin/echo y)"')
echo "$result"
xy

now it seems that newlines will cause trouble.

Also the question rise why there is this difference between `...` and $(...) or
why not map `...` to $(...) ?

Werner
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140725/a8a5a5fb/attachment.sig>
David Korn
2014-08-07 01:03:22 UTC
Permalink
I have attached a different fix that doesn't have the problem the last one
does. However, if the last command in the pipeline is a shell built-in, it
could hang for large output.

Let me know if this works for you.
Post by Dr. Werner Fink
Post by David Korn
I finally got a chance to look at this.
The problem is that the standard is ambiguous about when a command
substitution ends.
The bourne shell used a pipe for command substitution and command
substitution terminated when the pipe closed. Thus, if you had a
command
Post by David Korn
the wrote on standard output, forked and exited and the child wrote on
that
Post by David Korn
pipe, the output from both processes was part of the command
substitution.
Post by David Korn
ksh considered the command substitution done when the command completed
and
Post by David Korn
used memory for files less that 64K(IOBSIZE) and this created a tmp file
if it was larger.
I tried to restore the old bourne shell semantics for `` so that the
foreback test would work. However, this caused the example you have to
hang. Here is a change which should fix this by using pipes for all ``
not
Post by David Korn
--- old/sh/subshell.c Wed Apr 2 09:57:00 2014
+++new/sh/subshell.c Thu Jul 24 11:19:43 2014
@@ -593,7 +574,7 @@
sp->tmpfd = -1;
sp->pipefd = -1;
/* use sftmp() file for standard output */
- if(!(iop = sftmp(comsub==1?PIPE_BUF:IOBSIZE)))
+ if(!(iop = sftmp(comsub==1?0:IOBSIZE)))
{
sfswap(sp->saveout,sfstdout);
errormsg(SH_DICT,ERROR_system(1),e_tmpcreate);
Let me know.
This works for the specific code but break other tests,
from ksh test suite
[ 1162s] test subshell begins at 2014-07-25+07:39:05
[ 1175s] foo
[ 1175s] subshell.sh[642]: nested command substitution with 2>&1 not working
[ 1178s] test subshell failed at 2014-07-25+07:39:21 with exit code 1 [ 70 tests 1 error ]
[ 1178s] test subshell(C.UTF-8) begins at 2014-07-25+07:39:21
[ 1194s] foo
[ 1194s] subshell.sh[642]: nested command substitution with 2>&1 not working
[ 1197s] test subshell(C.UTF-8) failed at 2014-07-25+07:39:40 with exit
code 1 [ 70 tests 1 error ]
[ 1197s] test subshell(shcomp) begins at 2014-07-25+07:39:40
[ 1211s] foo
[ 1211s] shcomp-subshell.ksh[642]: nested command substitution
with 2>&1 not working
[ 1214s] test subshell(shcomp) failed at 2014-07-25+07:39:57 with exit
code 1 [ 70 tests 1 error ]
from my own test suite
[ 1233s] ++ /tmp/ksh-build.D5gJUl/bin/ksh -k -c 'd=`/bin/echo x y=z`; echo $d x y=z'
[ 1233s] + result='x
[ 1233s] x'
[ 1233s] + test 'x
[ 1233s] x' = 'x x'
[ 1233s] + exit 1
also further tests do fail
result=$(/tmp/ksh-build.D5gJUl/bin/ksh -c 'echo | echo "x`/bin/echo y`"')
echo "$result"
x
y
This doesn't happen with the new version.
Post by Dr. Werner Fink
result=$(/tmp/ksh-build.D5gJUl/bin/ksh -c 'echo | echo "x$(/bin/echo y)"')
echo "$result"
xy
now it seems that newlines will cause trouble.
Also the question rise why there is this difference between `...` and $(...) or
why not map `...` to $(...) ?
The reason is backward compatibility. With the Bourne shell, `...` used
pipes and waited for the pipe to close to decide when `` is complete.
$(...) uses temp file and if a process in `` forks and the parent exits,
$(...) won't get the output from the parent like `` will.
Post by Dr. Werner Fink
Werner
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140806/928519de/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: subhang.patch
Type: application/octet-stream
Size: 952 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140806/928519de/attachment.obj>
Dr. Werner Fink
2014-08-18 13:18:59 UTC
Permalink
Post by David Korn
I have attached a different fix that doesn't have the problem the last one
does. However, if the last command in the pipeline is a shell built-in, it
could hang for large output.
Let me know if this works for you.
It hangs as `cat' is a shell built-in in normal case:

/usr/src/packages/BUILD/ksh93> ./arch/linux.i386-64/src/cmd/ksh93/ksh
/usr/src/packages/BUILD/ksh93> x=`cat ~/testfile.txt | cat -`
Killed

whereas this one

/usr/src/packages/BUILD/ksh93> x=`cat ~/testfile.txt | /bin/cat -`
/usr/src/packages/BUILD/ksh93>

does work ... IMHO both should work. The test suite does survive with
this patch.

Werner
Post by David Korn
Post by David Korn
I finally got a chance to look at this.
The problem is that the standard is ambiguous about when a command
substitution ends.
The bourne shell used a pipe for command substitution and command
substitution terminated when the pipe closed. Thus, if you had a command
the wrote on standard output, forked and exited and the child wrote on that
pipe, the output from both processes was part of the command substitution.
ksh considered the command substitution done when the command completed and
used memory for files less that 64K(IOBSIZE) and this created a tmp file
if it was larger.
I tried to restore the old bourne shell semantics for `` so that the
foreback test would work. However, this caused the example you have to
hang. Here is a change which should fix this by using pipes for all `` not
--- old/sh/subshell.c Wed Apr 2 09:57:00 2014
+++new/sh/subshell.c Thu Jul 24 11:19:43 2014
@@ -593,7 +574,7 @@
sp->tmpfd = -1;
sp->pipefd = -1;
/* use sftmp() file for standard output */
- if(!(iop = sftmp(comsub==1?PIPE_BUF:IOBSIZE)))
+ if(!(iop = sftmp(comsub==1?0:IOBSIZE)))
{
sfswap(sp->saveout,sfstdout);
errormsg(SH_DICT,ERROR_system(1),e_tmpcreate);
Let me know.
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140818/d6705d7d/attachment.sig>
David Korn
2014-08-19 00:50:07 UTC
Permalink
I should be able to detect that cat is a builtin version of /bin/cat and
implicitly treat it as /bin/cat. I'll put this on my list.
Post by Dr. Werner Fink
Post by David Korn
I have attached a different fix that doesn't have the problem the last
one
Post by David Korn
does. However, if the last command in the pipeline is a shell built-in,
it
Post by David Korn
could hang for large output.
Let me know if this works for you.
/usr/src/packages/BUILD/ksh93> ./arch/linux.i386-64/src/cmd/ksh93/ksh
/usr/src/packages/BUILD/ksh93> x=`cat ~/testfile.txt | cat -`
Killed
whereas this one
/usr/src/packages/BUILD/ksh93> x=`cat ~/testfile.txt | /bin/cat -`
/usr/src/packages/BUILD/ksh93>
does work ... IMHO both should work. The test suite does survive with
this patch.
Werner
Post by David Korn
Post by David Korn
I finally got a chance to look at this.
The problem is that the standard is ambiguous about when a command
substitution ends.
The bourne shell used a pipe for command substitution and command
substitution terminated when the pipe closed. Thus, if you had a
command
on that
Post by David Korn
Post by David Korn
pipe, the output from both processes was part of the command
substitution.
Post by David Korn
Post by David Korn
ksh considered the command substitution done when the command
completed and
Post by David Korn
Post by David Korn
used memory for files less that 64K(IOBSIZE) and this created a tmp
file
Post by David Korn
Post by David Korn
if it was larger.
I tried to restore the old bourne shell semantics for `` so that the
foreback test would work. However, this caused the example you have
to
Post by David Korn
Post by David Korn
hang. Here is a change which should fix this by using pipes for all
`` not
Post by David Korn
Post by David Korn
--- old/sh/subshell.c Wed Apr 2 09:57:00 2014
+++new/sh/subshell.c Thu Jul 24 11:19:43 2014
@@ -593,7 +574,7 @@
sp->tmpfd = -1;
sp->pipefd = -1;
/* use sftmp() file for standard output */
- if(!(iop =
sftmp(comsub==1?PIPE_BUF:IOBSIZE)))
Post by David Korn
Post by David Korn
+ if(!(iop = sftmp(comsub==1?0:IOBSIZE)))
{
sfswap(sp->saveout,sfstdout);
errormsg(SH_DICT,ERROR_system(1),e_tmpcreate);
Let me know.
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140818/22d80e12/attachment.html>
David Korn
2014-08-19 01:34:56 UTC
Permalink
Try this patch. It should ignore that cat is a built-in of /bin/cat and
runs /bin/cat instead when it is the last command of ``. Let me know if
this fixes the problem.

--- old/sh/xec.c Mon Aug 18 17:53:43 2014
+++ new/sh/xec.c Mon Aug 18 18:23:58 2014
@@ -1253,9 +1253,14 @@
}
if(np && pipejob==2)
{
- job_unlock();
- nlock--;
- pipejob = 1;
+ if(shp->comsub==1 && np &&
is_abuiltin(np) && *np->nvname=='/')
+ np = 0;
+ else
+ {
+ job_unlock();
+ nlock--;
+ pipejob = 1;
+ }
}
/* check for builtins */
if(np && is_abuiltin(np))
Post by David Korn
I should be able to detect that cat is a builtin version of /bin/cat and
implicitly treat it as /bin/cat. I'll put this on my list.
Post by David Korn
Post by David Korn
I have attached a different fix that doesn't have the problem the last
one
Post by David Korn
does. However, if the last command in the pipeline is a shell
built-in, it
Post by David Korn
could hang for large output.
Let me know if this works for you.
/usr/src/packages/BUILD/ksh93> ./arch/linux.i386-64/src/cmd/ksh93/ksh
/usr/src/packages/BUILD/ksh93> x=`cat ~/testfile.txt | cat -`
Killed
whereas this one
/usr/src/packages/BUILD/ksh93> x=`cat ~/testfile.txt | /bin/cat -`
/usr/src/packages/BUILD/ksh93>
does work ... IMHO both should work. The test suite does survive with
this patch.
Werner
Post by David Korn
On Fri, Jul 25, 2014 at 4:04 AM, Dr. Werner Fink <werner at suse.de>
Post by David Korn
I finally got a chance to look at this.
The problem is that the standard is ambiguous about when a command
substitution ends.
The bourne shell used a pipe for command substitution and command
substitution terminated when the pipe closed. Thus, if you had a
command
on that
Post by David Korn
Post by David Korn
pipe, the output from both processes was part of the command
substitution.
Post by David Korn
Post by David Korn
ksh considered the command substitution done when the command
completed and
Post by David Korn
Post by David Korn
used memory for files less that 64K(IOBSIZE) and this created a
tmp file
Post by David Korn
Post by David Korn
if it was larger.
I tried to restore the old bourne shell semantics for `` so that the
foreback test would work. However, this caused the example you
have to
Post by David Korn
Post by David Korn
hang. Here is a change which should fix this by using pipes for
all `` not
Post by David Korn
Post by David Korn
--- old/sh/subshell.c Wed Apr 2 09:57:00 2014
+++new/sh/subshell.c Thu Jul 24 11:19:43 2014
@@ -593,7 +574,7 @@
sp->tmpfd = -1;
sp->pipefd = -1;
/* use sftmp() file for standard output */
- if(!(iop =
sftmp(comsub==1?PIPE_BUF:IOBSIZE)))
Post by David Korn
Post by David Korn
+ if(!(iop = sftmp(comsub==1?0:IOBSIZE)))
{
sfswap(sp->saveout,sfstdout);
errormsg(SH_DICT,ERROR_system(1),e_tmpcreate);
Let me know.
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140818/21ec451b/attachment.html>
Dr. Werner Fink
2014-08-19 12:44:29 UTC
Permalink
Post by David Korn
Try this patch. It should ignore that cat is a built-in of /bin/cat and
runs /bin/cat instead when it is the last command of ``. Let me know if
this fixes the problem.
--- old/sh/xec.c Mon Aug 18 17:53:43 2014
+++ new/sh/xec.c Mon Aug 18 18:23:58 2014
@@ -1253,9 +1253,14 @@
}
if(np && pipejob==2)
{
- job_unlock();
- nlock--;
- pipejob = 1;
+ if(shp->comsub==1 && np &&
is_abuiltin(np) && *np->nvname=='/')
+ np = 0;
+ else
+ {
+ job_unlock();
+ nlock--;
+ pipejob = 1;
+ }
}
/* check for builtins */
if(np && is_abuiltin(np))
This change works for the cat builtin but the test suite now shows:

[ 907s] test subshell begins at 2014-08-19+12:40:15
[ 922s] subshell.sh[682]: non-existant last command in pipeline causes `` to fail
[ 922s] test subshell failed at 2014-08-19+12:40:30 with exit code 1 [ 70 tests 1 error ]
[ 922s] test subshell(C.UTF-8) begins at 2014-08-19+12:40:30
[ 936s] subshell.sh[682]: non-existant last command in pipeline causes `` to fail
[ 936s] test subshell(C.UTF-8) failed at 2014-08-19+12:40:44 with exit code 1 [ 70 tests 1 error ]
[ 936s] test subshell(shcomp) begins at 2014-08-19+12:40:44
[ 951s] shcomp-subshell.ksh[682]: non-existant last command in pipeline causes `` to fail
[ 951s] test subshell(shcomp) failed at 2014-08-19+12:40:59 with exit code 1 [ 70 tests 1 error ]
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140819/89d73127/attachment.sig>
Dr. Werner Fink
2014-08-28 13:20:04 UTC
Permalink
Post by Dr. Werner Fink
Post by David Korn
Try this patch. It should ignore that cat is a built-in of /bin/cat and
runs /bin/cat instead when it is the last command of ``. Let me know if
this fixes the problem.
--- old/sh/xec.c Mon Aug 18 17:53:43 2014
+++ new/sh/xec.c Mon Aug 18 18:23:58 2014
@@ -1253,9 +1253,14 @@
}
if(np && pipejob==2)
{
- job_unlock();
- nlock--;
- pipejob = 1;
+ if(shp->comsub==1 && np &&
is_abuiltin(np) && *np->nvname=='/')
+ np = 0;
+ else
+ {
+ job_unlock();
+ nlock--;
+ pipejob = 1;
+ }
}
/* check for builtins */
if(np && is_abuiltin(np))
[ 907s] test subshell begins at 2014-08-19+12:40:15
[ 922s] subshell.sh[682]: non-existant last command in pipeline causes `` to fail
[ 922s] test subshell failed at 2014-08-19+12:40:30 with exit code 1 [ 70 tests 1 error ]
[ 922s] test subshell(C.UTF-8) begins at 2014-08-19+12:40:30
[ 936s] subshell.sh[682]: non-existant last command in pipeline causes `` to fail
[ 936s] test subshell(C.UTF-8) failed at 2014-08-19+12:40:44 with exit code 1 [ 70 tests 1 error ]
[ 936s] test subshell(shcomp) begins at 2014-08-19+12:40:44
[ 951s] shcomp-subshell.ksh[682]: non-existant last command in pipeline causes `` to fail
[ 951s] test subshell(shcomp) failed at 2014-08-19+12:40:59 with exit code 1 [ 70 tests 1 error ]
Just seen this during debugging:

BUILD/ksh93> ./arch/linux.i386-64/src/cmd/ksh93/ksh -c 'while((SECONDS<3)); do test -z `/bin/false | /bin/false | /bin/doesnotexist`;done;:' 2> /dev/null
Segmentation fault (core dumped)
BUILD/ksh93> gdb ./arch/linux.i386-64/src/cmd/ksh93/ksh core
GNU gdb (GDB) SUSE (7.5.1-0.7.29)
Copyright (C) 2012 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
[...]
Core was generated by `./arch/linux.i386-64/src/cmd/ksh93/ksh -c while((SECONDS<3)); do test -z `/bin/'.
Program terminated with signal 11, Segmentation fault.
#0 bestlistreclaim (listp=<optimized out>, pack=<optimized out>, vm=<optimized out>) at /usr/src/packages/BUILD/ksh93/src/lib/libast/vmalloc/vmbest.c:471
471 SIZE(fp) |= MARK;
(gdb) print fp
$1 = (Block_t *) 0x400000

Werner
--
"Having a smoking section in a restaurant is like having
a peeing section in a swimming pool." -- Edward Burr
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 836 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20140828/068685e1/attachment.sig>
Loading...