[ast-developers] ksh93v- beta, and must have bug fixes

[ольга крыжановская]

IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.

Submissions for the worst of worst bugs please to the list now.

Olga

---------- Forwarded message ----------
From: Glenn Fowler <gsf at research.att.com>
Date: Thu, Aug 29, 2013 at 7:45 PM
Subject: [ast-developers] AT&T Software Technology ast alpha software
download update
To: ast-developers at research.att.com



the AT&T Software Technology ast alpha 2013-08-29 source release
has been posted to the download site
http://www.research.att.com/sw/download/alpha/
the package names and md5 checksums are
INIT 132e0403af573fa1cb1e202267fedeb8
ast-open 334615fb3a652575106194c281d27b5c
ast-ksh ebcc56d9ab673aaafbb163d6eee1a93c
the md5 sums should match the ones listed on the download page

this is still a work in progress, but we are getting closer to a beta
we are interested in verification that new features work
and at this point not interested in quibbles about naming
once the underlying features work then naming discussions can commence

a big chunk of time was spent porting to the various combinations of
architectures X os X compilers X build options
a lot of it due to differing levels of support for the *at() syscalls
which then kicked in different parts of the -last emulation code

with adoption of *at() file descriptor leaks pop up all over the place
to see how bad *at() coverage is on different os/release combinations compare
$INSTALLROOT/src/lib/libast/FEATURE/atdev
produced by
$PACKAGEROOT/src/lib/libast/features/atdev.c
on different os/release combinations

one file descriptor leak in 3d+ksh brought down the entire build system
it took a day to figure everything out and recover, including code checkins
that sometimes did the opposite (keep old drop new)
so once again there is no such thing as a simple patch

don't use this alpha on linux.ia64 -- looks like *at() problems are
lurking there

look for ast runtime debugging tips in a separate message

changes since 2013-08-14

:::::::: bgp ::::::::

13-08-28 mrt.h: MRT_ATTR_AIGP_20120425 => MRT_ATTR_AIGP
13-08-28 bgp.c: add BGP_aigp, "aigp_t aigp { values_t aigp }"
13-08-28 bgp-mrt.c: parse MRT_ATTR_AIGP subtype 1

:::::::: ip_t ::::::::

13-08-28 bgp.h: add BGP_aigp
13-08-28 ip_t.c: add "values_t" uint32_t vector type

:::::::: ksh93 ::::::::

13-08-27 --- Release ksh93v- ---
13-08-27 +enum constants can have .DIG, .MAX, or .MIN sub-variables.
13-08-26 +Floating variables can be following by DIG, MAX, MIN, EPSILON,
INT_MAX, INT_MIN, or MAX_10_EXP to give values correspong to
the floating type of the variable.
13-08-26 +Integer variables can be followed by .DIG, .MAX, or .MIN in
arithmetic expression so give the values corresponding to that
integer type.
13-08-26 +Added 12 math constants such as E, PI, and SQRT2.
13-08-23 Documented __ in sh.1 man page.
13-08-22 Fixed __ to work in nested types.
13-08-22 A bug in which print -v .sh.stats did not work within a namespace
has been fixed.
13-08-21 enumeration subvariables introduce in 13-07-30 can now be used
with enum arrays and reference to enum array elements.
13-08-20 A bug in arithemtic with variables used in an arithmetic expression
inside a namespace has been fixed.
13-08-19 A bug in which compound variables created in a namespace did not
display correctly outside of the namespace has been fixed.
13-08-19 +The variable .sh.pwdfd which expands to the file descriptor number
corresponding to $PWD has been added.
13-08-15 A bug in which copying the .sh.stats and .sh.siginfo variables
gave values that could vary has been fixed.
13-08-14 A bug in which unknow starup options, for example ksh --foobarbla
would cause the shell to core dump has been fied.
13-08-14 A bug in which {n}<&0 did not work after <<< string hs been fixed.

:::::::: tw ::::::::

13-08-26 expr.c: fix null pointer deref in matchval() -- thanks Philippe

:::::::: libast ::::::::

13-08-26 features/float: fix (FLT|DBL|LDBL)_*_(MIN|MAX) algorithm to
not use exact powers of 2!!
13-08-25 path/pathcanon.c: finalize //@// extended/hidden attribute
namespace *for testing*
13-08-24 include/fts.h,misc/fts.c: make preparations for
FTS.fts_dirfd, fdopendir(), fgetcwd(), openat(), fstatat()
13-08-23 path/pathcanon.c: add PATH_DROP_LEAD_SLASH2 and
PATH_DROP_TAIL_SLASH to unify ast c14n
13-08-23 features/float: add (FLT|DBL|LDBL)_EPSILON logic
13-08-22 features/lib,include/ast_std.h,comp/qsort.c,comp/qsort_r.c:
add { qsort qsort_r } from Bentley/McIlroy code
13-08-22 misc/getcwd.c: now a wrapper call to fgetcwd(AT_FDCWD)
13-08-21 regex/regdecomp.c: fix REG_SHELL decompilation bug
13-08-21 regex/regcomp.c: fix REG_SHELL ~(A)(...) parse bug that
didn't treat second ')' as closing paren
13-08-20 misc/fgetcwd.c: add, inspired by olga
13-08-17 path/pathcanon.c: /dev/file/flag[,flag...]:[/]path for
relative and absolute paths
13-08-17 port/intercept.c: move pathdev() intercepts to ast_*at() and
make ast_*[!at]() call ast_*at() counterpart
13-08-15 features/lib,features/dirent,port/intercept.c: add
ast_opendir() and fdopendir() tests
13-08-14 features/atdev: additional support for comp/at.c emulation on
frankenstein systems

_______________________________________________
ast-developers mailing list
ast-developers at lists.research.att.com
http://lists.research.att.com/mailman/listinfo/ast-developers

[Lionel Cons]

Post by ольга крыжановская
IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.
Submissions for the worst of worst bugs please to the list now.

Is there still time to submit issues?
* Our top entries - based on engineering survey of 20130902 would be:
- sfpoll2()
- stable and reliable interface to SIGRTMIN-SIGRTMAX signals, including .sh.sig
- sync(1) with support for fsync() and syncfs()

* Medium priorities:
- singlebyte locales like en_GB.iso885915 must work
- support for \u and \w, as done with Roland Mainz's patch, as this
has proved useful for my staff during today's preliminary testing. The
patch would allow them to remove a lot of weired ( LC_ALL=en_US.utf8;
printf '...' | iconv -f UTF-8; ) lines with a plain print -f
'\u[hexvalue]'. This implies that \u and \w work on singlebyte locales
too
- grep builtin performance improvements for large files. As of
ast-ksh.2013-08-14 AST grep only reaches 85% of the performance of GNU
grep
- bigger mmap() windows for sfio file IO. The break even we calculated
is somewhere between 108-140MB window size on an Intel Sandy Bridge
four way server and 144-157 for an AMD Jaguar prototype machine
(X1150). This is likely to 'fix' most of the grep performance issue.
Also forwarding a clarifying comment: " explain them that this is NOT
akin to buffer bloat. It only the defines the maximum size of the
address space window a process can have for this file. The kernel
itself is in charge to select an appropriate number of pages it can
donate to pass data through this window"
- /dev/file/@direct and /dev/file/@directory to open files with
O_DIRECT (no kernel buffering, but maybe zero copy data passing) or
O_DIRECTORY (self explanatory). I leave the exact spelling of
/dev/@file at options@@@/@path@ to Glenn's discretion :)
- no //@//. Self explanatory. A lot of people here went to great
lengths to clean their scripts and filter extra slashes from path
names out of the shear fear that //@// may blow up their scripts. I'd
always welcome cleanup, but this fear-induced craziness was real waste
of engineering time

* Low priority:
- namespace feature
- public, web-based issue tracker
- more space-efficient associative arrays in ksh93. As of
ast-ksh.2013-08-14 they consume a lot more memory than indexed arrays
- return of the fixed-sized integer arrays, i.e. that integer -a
array[5000000] preallocate memory for an indexed array with 5000000
slots. This may improve memory usage by allocating the array once
instead of growing it entry by entry and thrashing the heap on the way
- the ksh93 test suite should pass without warnings or crashes on the
major Linux distributions (Ubuntu/Debian, Suse, Redhat/Fedora)
- cd -@ to enter and leave attribute directories on platforms which
support open(2) O_XATTR
- open directory file descriptor to a attribute directory on platforms
which support open(2) O_XATTR
- mathematical constants like M_PI and MIN/MAX values for all integer types
- test ksh93 on ARM64

Lionel

[Lionel Cons]

Post by Lionel Cons

Post by ольга крыжановская
IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.
Submissions for the worst of worst bugs please to the list now.

Is there still time to submit issues?
- sfpoll2()
- stable and reliable interface to SIGRTMIN-SIGRTMAX signals, including .sh.sig
- sync(1) with support for fsync() and syncfs()
- singlebyte locales like en_GB.iso885915 must work
- support for \u and \w, as done with Roland Mainz's patch, as this
has proved useful for my staff during today's preliminary testing. The
patch would allow them to remove a lot of weired ( LC_ALL=en_US.utf8;
printf '...' | iconv -f UTF-8; ) lines with a plain print -f
'\u[hexvalue]'. This implies that \u and \w work on singlebyte locales
too
- grep builtin performance improvements for large files. As of
ast-ksh.2013-08-14 AST grep only reaches 85% of the performance of GNU
grep
- bigger mmap() windows for sfio file IO. The break even we calculated
is somewhere between 108-140MB window size on an Intel Sandy Bridge
four way server and 144-157 for an AMD Jaguar prototype machine
(X1150). This is likely to 'fix' most of the grep performance issue.
Also forwarding a clarifying comment: " explain them that this is NOT
akin to buffer bloat. It only the defines the maximum size of the
address space window a process can have for this file. The kernel
itself is in charge to select an appropriate number of pages it can
donate to pass data through this window"
O_DIRECT (no kernel buffering, but maybe zero copy data passing) or
O_DIRECTORY (self explanatory). I leave the exact spelling of
lengths to clean their scripts and filter extra slashes from path
always welcome cleanup, but this fear-induced craziness was real waste
of engineering time
- namespace feature
- public, web-based issue tracker
- more space-efficient associative arrays in ksh93. As of
ast-ksh.2013-08-14 they consume a lot more memory than indexed arrays
- return of the fixed-sized integer arrays, i.e. that integer -a
array[5000000] preallocate memory for an indexed array with 5000000
slots. This may improve memory usage by allocating the array once
instead of growing it entry by entry and thrashing the heap on the way
- the ksh93 test suite should pass without warnings or crashes on the
major Linux distributions (Ubuntu/Debian, Suse, Redhat/Fedora)
support open(2) O_XATTR
- open directory file descriptor to a attribute directory on platforms
which support open(2) O_XATTR
- mathematical constants like M_PI and MIN/MAX values for all integer types
- test ksh93 on ARM64

* High priority:
- Make libast, libshell and libcmd header files C++ compatible

Lionel

[Glenn Fowler]

Post by Lionel Cons

Post by Lionel Cons

Post by ольга крыжановская
IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.
Submissions for the worst of worst bugs please to the list now.

Is there still time to submit issues?
- sfpoll2()
- stable and reliable interface to SIGRTMIN-SIGRTMAX signals, including .sh.sig
- sync(1) with support for fsync() and syncfs()
- singlebyte locales like en_GB.iso885915 must work
- support for \u and \w, as done with Roland Mainz's patch, as this
has proved useful for my staff during today's preliminary testing. The
patch would allow them to remove a lot of weired ( LC_ALL=en_US.utf8;
printf '...' | iconv -f UTF-8; ) lines with a plain print -f
'\u[hexvalue]'. This implies that \u and \w work on singlebyte locales
too
- grep builtin performance improvements for large files. As of
ast-ksh.2013-08-14 AST grep only reaches 85% of the performance of GNU
grep
- bigger mmap() windows for sfio file IO. The break even we calculated
is somewhere between 108-140MB window size on an Intel Sandy Bridge
four way server and 144-157 for an AMD Jaguar prototype machine
(X1150). This is likely to 'fix' most of the grep performance issue.
Also forwarding a clarifying comment: " explain them that this is NOT
akin to buffer bloat. It only the defines the maximum size of the
address space window a process can have for this file. The kernel
itself is in charge to select an appropriate number of pages it can
donate to pass data through this window"
O_DIRECT (no kernel buffering, but maybe zero copy data passing) or
O_DIRECTORY (self explanatory). I leave the exact spelling of
lengths to clean their scripts and filter extra slashes from path
always welcome cleanup, but this fear-induced craziness was real waste
of engineering time
- namespace feature
- public, web-based issue tracker
- more space-efficient associative arrays in ksh93. As of
ast-ksh.2013-08-14 they consume a lot more memory than indexed arrays
- return of the fixed-sized integer arrays, i.e. that integer -a
array[5000000] preallocate memory for an indexed array with 5000000
slots. This may improve memory usage by allocating the array once
instead of growing it entry by entry and thrashing the heap on the way
- the ksh93 test suite should pass without warnings or crashes on the
major Linux distributions (Ubuntu/Debian, Suse, Redhat/Fedora)
support open(2) O_XATTR
- open directory file descriptor to a attribute directory on platforms
which support open(2) O_XATTR
- mathematical constants like M_PI and MIN/MAX values for all integer types
- test ksh93 on ARM64

- Make libast, libshell and libcmd header files C++ compatible

can you point out where they are not C++ compatible

all of the headers in $INSTALLROOT/include/ast are generated by filtering
through the ast proto(1) command which should handle C <=> C++ linkage

well, all of the headers that contain this comment were installed that way:
/* : : generated by proto : : */

[Irek Szczesniak]

On Fri, Aug 30, 2013 at 9:05 PM, ????? ????????????

Post by ольга крыжановская
IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.
Submissions for the worst of worst bugs please to the list now.

Here is our list, sorted by priority/importance:
- poll(1) builtin (AFAIK nothing to do here, it just works)
- Fix job management. The current *design* is broken as described in
lists.research.att.com/pipermail/ast-developers/2013q3/003033.html
- Make typeset -c (copy variables) public
- Fix typeset -c for any bugs you can find
- /dev/file/direct@ should work to open a filesystem node with
O_DIRECT (for performance reasons)
- Fix ar=() to clear an array, currently 'compound -a c=( [4]=(
hello=1) ); c=() ; typeset -p c' prints 'typeset -a c' which is wrong
(missing -C attribute). Its even worse for arrays of types.
- readlink builtin in libcmd for compatibility with existing software
- Test /proc/$pid/fd emulation on Solaris and fix any bugs you find
- Test directory fds and fix any bugs you find
- cd -@ should just work
- /dev/file/xattr@ should work to open a filesystem node with O_XATTR
- It should be possible to *nest* /dev/file/options at .../dev/file/options at ...
- It should be possible to combine /dev/file/options@ with /dev/fd$fd
and /proc/$$/fd/$fd, even on Solaris!
- New UWIN alpha

I'm thinking about putting Roland's patch for \u[wchar_val] and
\w[wchar_val] on that list. We don't have much use product-wise for it
now but I see it fixes single byte locales _and_ improves the
situation for non-English languages in ksh93 significantly (French!).

Irek

[Tina Harriott]

Post by ольга крыжановская
IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.
Submissions for the worst of worst bugs please to the list now.

ksh -c 'typeset -s -E x=4 ; print $(( x=nextafter(x,5) ))' should use
C nextafterf(),
ksh -c 'typeset -E x=4 ; print $(( x=nextafter(x,5) ))' should use
C nextafter() and
ksh -c 'typeset -l -E x=4 ; print $(( x=nextafter(x,5) ))' should use
C nextafterl().

ksh arithmetic nexttoward() should call C nexttowardf(), C
nexttoward() or nexttowardl() depending on the right ksh floating
point type, like in the example above.

Tina

[Simon Toedt]

On Fri, Aug 30, 2013 at 9:05 PM, ????? ????????????

Post by ольга крыжановская
IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.
Submissions for the worst of worst bugs please to the list now.

- UWIN: Ability to enter german umlauts, as requested in my posting to

Post by ольга крыжановская
How can I enter German umlaut characters in the uwin console window? I
tried under Windows XP Professional and got only question marks
instead of umlaut characters.
Does uwin support non-ascii characters on WinXP? Reading
http://en.wikipedia.org/wiki/Win32_console#Windows_NT_and_CE_based the
Windows console window supports UTF-8 as code page 65001 if you use
the Unicode apis from Windows for I/O.

- poll builtin in UWIN

- Bimonthly UWIN alpha releases

- More people at AT&T working on UWIN and responding to questions

- Public issue database for UWIN

- Windows 8 support for UWIN

- Fix WINE that UWIN works there

- POSIX open_memstream() in libast
[http://pubs.opengroup.org/onlinepubs/9699919799/functions/open_memstream.html]

Simon

[Cedric Blancher]

Post by ольга крыжановская
IMO before you move ksh93v- to "beta" status, we should collect bugs
and changes which should be done and finalized before we call it beta.
There are very, very bad issues in ast-ksh.20130829, and they should
be addressed. I am going - due lack of Bugzilla, a list on our svn.
Submissions for the worst of worst bugs please to the list now.

Personal (non job) item. Just two:
1. fix AST mailx that it can work with Googlemail
2. item one implies that mailx needs IMAP/SSL support

Ced