[ast-developers] 'printf "foo" >bar; head -1 bar' - ksh93 builtin vs OSX portability?

[Irek Szczesniak]

The question below came up on IRC today. Is this a bug in ksh93u+?

[17:17:08] <jperkin> should it be considered a bug that ksh's built-in
head doesn't handle lines with newlines? compare 'printf "foo" >bar;
head -1 bar' with /usr/bin/head
[17:19:10] <jperkin> whatever the ksh93 variant is in OSX doesn't have
the same behaviour

Irek

[FELLIN, JEFFREY K (JEFF)]

Irek,
When I test printf "foo" >bar there is no newline in the file:
$ printf "foo" >bar
$ od -bc bar
0000000 146 157 157
f o o
0000003

So if head -1 is suppose to print the first line, there is no end of line, hence to no line. Given the definition of head prints lines, wouldn't a file containing no newline chars, not have any output from head?

Jeff Fellin

-----Original Message-----
From: ast-users-bounces at lists.research.att.com [mailto:ast-users-bounces at lists.research.att.com] On Behalf Of Irek Szczesniak
Sent: Thursday, September 19, 2013 14:38
To: ast-users; ast-developers at research.att.com
Subject: [ast-users] 'printf "foo" >bar; head -1 bar' - ksh93 builtin vs OSX portability?

The question below came up on IRC today. Is this a bug in ksh93u+?

[17:17:08] <jperkin> should it be considered a bug that ksh's built-in head doesn't handle lines with newlines? compare 'printf "foo" >bar; head -1 bar' with /usr/bin/head [17:19:10] <jperkin> whatever the ksh93 variant is in OSX doesn't have the same behaviour

Irek

[Roland Mainz]

On Thu, Sep 19, 2013 at 8:53 PM, FELLIN, JEFFREY K (JEFF)

Post by FELLIN, JEFFREY K (JEFF)
$ printf "foo" >bar
$ od -bc bar
0000000 146 157 157
f o o
0000003
So if head -1 is suppose to print the first line, there is no end of line, hence to no line. Given the definition of head prints lines, wouldn't a file containing no newline chars, not have any output from head?

One counterargument may be that we're talking about the "last line",
which may/should be always printed, regardless whether it has a
newline or not.

The other issue is that I can't find a head(1) implementation which
behaves like AST head(1) with Irek's example:
On Solaris 10 (/usr/bin/head is the natiev SystemV implementation, not
the head(1) from AST) I get this:
-- snip --
$ bash -c 'printf "foo" >bar ; /usr/bin/head -1 bar ; printf '\n''
foo
$ bash -c 'printf "foo" >bar ; /usr/gnu/bin/head -1 bar ; printf '\n''
foo
$ bash -c 'printf "foo" >bar ; /usr/bsd/bin/head -1 bar ; printf '\n''
foo
-- snip --

On SuSE/Linux I get this for GNU coreutils and busybox head(1):
-- snip --
$ bash -c 'printf "foo" >bar ; /usr/bin/head -1 bar ; printf '\n''
foo
$ printf "foo" >bar ; busybox head -1 bar ; printf '\n'
foo
-- snip --

Erm... this looks pretty much uniform... ;-/

----

Bye,
Roland

[Glenn Fowler]

this was an interesting diversion
a text file with size>0 and newline!=last-char has an incomplete line
the head and tail descriptions do not mention incomplete lines
but head has some text that is equivalent to "incomplete line counts as a line":

When a file contains less than number lines, it shall be copied
to standard output in its entirety. This shall not be an error.

for consistency tail should treat incomplete lines the same way head does

given that, both the current ast head and ast tail are inconsistent w.r.t. incomplete lines

head was easy to fix by implementing the "entirety" text above
tail was a bit more interesting because it if revealed a bug in sfmove(from,to,size,record_sep)
when record_sep is '\n' -- it loses the data for an incomplete line
fixing tail involved checking if sfmove() hit an incomplete line and
then calling sfmove(...,-1) to move all remaining *and* fixing sfmove(...,-1)
to recognize that an incomplete line was hit

regression tests were added to src/cmd/builtin and now head and tail align with gnu

from what I can tell the standard conformance tests completely miss this issue

this will be in the next alpha
and that will be early next week

Post by Roland Mainz
On Thu, Sep 19, 2013 at 8:53 PM, FELLIN, JEFFREY K (JEFF)

Post by FELLIN, JEFFREY K (JEFF)
$ printf "foo" >bar
$ od -bc bar
0000000 146 157 157
f o o
0000003
So if head -1 is suppose to print the first line, there is no end of line, hence to no line. Given the definition of head prints lines, wouldn't a file containing no newline chars, not have any output from head?

One counterargument may be that we're talking about the "last line",
which may/should be always printed, regardless whether it has a
newline or not.
The other issue is that I can't find a head(1) implementation which
On Solaris 10 (/usr/bin/head is the natiev SystemV implementation, not
-- snip --
$ bash -c 'printf "foo" >bar ; /usr/bin/head -1 bar ; printf '\n''
foo
$ bash -c 'printf "foo" >bar ; /usr/gnu/bin/head -1 bar ; printf '\n''
foo
$ bash -c 'printf "foo" >bar ; /usr/bsd/bin/head -1 bar ; printf '\n''
foo
-- snip --
-- snip --
$ bash -c 'printf "foo" >bar ; /usr/bin/head -1 bar ; printf '\n''
foo
$ printf "foo" >bar ; busybox head -1 bar ; printf '\n'
foo
-- snip --
Erm... this looks pretty much uniform... ;-/
----
Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL [phone]
(;O/ \/ \O;)

[Cedric Blancher]

Post by Glenn Fowler
this was an interesting diversion
a text file with size>0 and newline!=last-char has an incomplete line
the head and tail descriptions do not mention incomplete lines
When a file contains less than number lines, it shall be copied
to standard output in its entirety. This shall not be an error.
for consistency tail should treat incomplete lines the same way head does
given that, both the current ast head and ast tail are inconsistent w.r.t. incomplete lines
head was easy to fix by implementing the "entirety" text above
tail was a bit more interesting because it if revealed a bug in sfmove(from,to,size,record_sep)
when record_sep is '\n' -- it loses the data for an incomplete line
fixing tail involved checking if sfmove() hit an incomplete line and
then calling sfmove(...,-1) to move all remaining *and* fixing sfmove(...,-1)
to recognize that an incomplete line was hit
regression tests were added to src/cmd/builtin and now head and tail align with gnu
from what I can tell the standard conformance tests completely miss this issue
this will be in the next alpha
and that will be early next week

Glenn, could you send the patch to the list so that we can test it, please?

Ced