[ast-developers] AT&T Software Technology ast alpha software download update

Discussion:

Glenn Fowler

2013-06-28 13:39:53 UTC

the AT&T Software Technology ast alpha 2013-06-28 source release
has been posted to the download site
http://www.research.att.com/sw/download/alpha/
the package names and md5 checksums are
INIT eddbf89d061348519d86f2618b708a94
ast-base a745a7d4ce6f53c2e4134af4cc835ff7
ast-open fdb74839ff041e34c800c333188a050e
ast-ksh 8f22428cf30af7146bd210664c2fd166
the md5 sums should match the ones listed on the download page

NOTE NOTE NOTE NOTE NOTE NOTE

(*) *at() emulations for systems that need it
(*) O_CLOEXEC F_DUPFD_CLOEXEC SOCK_CLOEXEC
(*) accept4 pipe2
(*) syscall restart on EINTR controlled by astserial(AST_SERIAL_RESTART,op)
(*) REALLY IMPORTANT: works on modern linux, some problems on solaris
REASON: one of the ast team will not be logging on for 16 days and
didn't have time left to figure out the solaris problems
most likely in one of these
src/lib/libast/comp/at.c
src/lib/libast/port/intercept.c
figuring that out left as a summer homework assignment
(*) thanks to roland and olga for patches that resulted in this release,
in spirit in some places more than not byte for byte
(*) we know there may be some controversy on intercepting some calls
give it a chance to soak (and work with solaris etc. patches before judgement
the current setup does minimal damage to ast code
especially code that can be used as standalone commands and ksh builtins
(*) we have tried in the past to manage releases better so close to vacation
but still haven't figured it out -- inspriation hits with time constraints
I guess

changes since 2013-06-13

:::::::: INIT ::::::::

13-06-24 add -lsocket => -lsocket -lnsl for systems that don't have it in -lc
13-06-17 package.sh: fix host type logic to handle -m32/-m64 almost cross-compile

:::::::: ksh93 ::::::::

13-06-25 --- Release ksh93v- ---
13-06-25 ksh93 uses the new ast system call intercepts to restart interrupted
signals.
13-06-21 A -f nn option has been added to cd to change to a directory
relative to a file descriptor of an open directory. cd -f nn
is equivalent to cd ~{nn}.
13-06-20 A bug in which ${IFS+abc} did not expand to abc when IFS is
unset has been fixed.
13-06-18 A bug on some systems with arithmentic expressions containing
x**y has been fixed.
13-06-13 Changes made for reliable delivery of queued signals.

:::::::: pax ::::::::

13-06-24 pax.h,pax.c,file.c: change chmod members to modekeep and restoremode

:::::::: libast ::::::::

13-06-28 intercept.c: add AST_SERIAL_always, in particular for AST_SERIAL_RESTART
13-06-25 intercept.c: add syscall intercepts to unify { *at() restart pathopen() }
13-06-25 pathopen.c: low level physical/logical/special path open using pathdev()
13-06-24 pathcanon.c: add pathdev() for { /dev/... /proc/... } parse
13-06-19 vmbest.c: fix strict-aliasing warnings
13-06-19 debug.h,debug.c: add debug_printf(fd,fmt,...), debug_vprintf(fd,fmt,ap), debug_snprintf(buf,siz,fmt,ap)
13-06-18 features/lib: fix lib *64 tests to reference related headers
13-06-18 malloc.c: add VMALLOC_OPTIONS=debug for debug trace vs verbose trace
13-06-18 reg[nr]exec.c: fix file size < min boyer-moore match uninitialized data ref
13-06-17 vmalloc: harden code to handle signal storm -- almost working save one workaround

:::::::: libjcl ::::::::

13-06-27 path.c,find.c: add ${N:s%old%new%:...} edit expressions to prefix file

Cedric Blancher

2013-06-28 22:37:11 UTC

Permalink

Post by Glenn Fowler
the AT&T Software Technology ast alpha 2013-06-28 source release
has been posted to the download site
http://www.research.att.com/sw/download/alpha/
the package names and md5 checksums are
INIT eddbf89d061348519d86f2618b708a94
ast-base a745a7d4ce6f53c2e4134af4cc835ff7
ast-open fdb74839ff041e34c800c333188a050e
ast-ksh 8f22428cf30af7146bd210664c2fd166
the md5 sums should match the ones listed on the download page

The release is unusable. The new "API" - if it can be called like that
- added wrappers to all syscalls via #define, which breaks down on
OpenBSD or other platforms which already use #defines for security
wrappers. It's also undebuggable by adding yet another layer of hidden
complexity. I wouldn't mind if if the code would call _ast_open() and
friends directly but hiding it via #define open _ast_open collides
with too many other things, including system libraries and the ability
of normal minds to grok it.

So this won't fly.

Ced

--
Cedric Blancher <cedric.blancher at googlemail.com>
Institute Pasteur

Cedric Blancher

2013-06-29 00:35:40 UTC

Permalink

Post by Cedric Blancher

cd is broken, too.

ksh -c 'rmdir a1 >/dev/null; mkdir a1 ; chmod a-r a1 ; cd a1 ; :'
/home/ced/bin/ksh: cd: a1: [Permission denied]

The problem is that ksh93 stopped to pass O_PATH or O_SEARCH to
openat(), despite use of a -DO_SEARCH=O_PATH.

I really wish AST would stop the #undef-#define-#redefine massacre -
it renders the code unmaintainable from outside AT&T. Another side
effect is that it's no longer possible to build 3rd-party applications
which do their own restart handling by just putting -I/usr/include/ast
-last in the configure options. perl and some other stuff is broken
thanks to this change.

Ced

--
Cedric Blancher <cedric.blancher at googlemail.com>
Institute Pasteur

Irek Szczesniak

2013-06-29 19:24:24 UTC

Permalink

On Sat, Jun 29, 2013 at 12:37 AM, Cedric Blancher

Post by Cedric Blancher

Well, I hope this is a one time experiment which is better not going
to be repeated. The whole concept of the intercepts is utterly broken
by design and can't even work in real life. I still having hiccups
from this code:
#define RESTART(r,f) \
do { \
uint32_t serial; \
do \
{ \
serial = asoget32(&GLOBAL(restart)); \
} while ((r = f) == -1 && errno == EINTR && (serial ==
AST_SERIAL_always || serial != asoget32(&GLOBAL(restart)))); \
} while (0)

Why? This is mixing process and thread semantics with signal restart
functionality even for syscalls which never return EINTR. The code is
prone to integer overflows, too.

Another two 'gems':
int
ast_eaccess(const char* path, int mode)
{
int r;

RESTART(r, faccessat(LOCAL(pwd), path, mode, AT_EACCESS));
return r;
}
int
ast_faccessat(int cwd, const char* path, mode_t mode, int flags)
{
int r;

RESTART(r, faccessat(cwd, path, mode, flags));
return r;
}

The first form (ast_eaccess) is redundant since faccessat() has become
available and should be mapped via a macro to ast_faccessat().
Otherwise you're producing code bloat.

Finally, cd(1) is broken for error cases and returns garbage.

Irek

Glenn Fowler

2013-07-17 16:49:40 UTC

Permalink

Post by Irek Szczesniak
On Sat, Jun 29, 2013 at 12:37 AM, Cedric Blancher

Post by Cedric Blancher

granted the aso calls may be overkill but for *alpha* testing I wanted to make
sure that even in non-threaded apps a signal didn't catch the code mid-increment

I wish statements like "broken", "prone to" would be accompanied by code
in particular, where is this code prone to integer overflow?

Post by Irek Szczesniak
int
ast_eaccess(const char* path, int mode)
{
int r;
RESTART(r, faccessat(LOCAL(pwd), path, mode, AT_EACCESS));
return r;
}
int
ast_faccessat(int cwd, const char* path, mode_t mode, int flags)
{
int r;
RESTART(r, faccessat(cwd, path, mode, flags));
return r;
}
The first form (ast_eaccess) is redundant since faccessat() has become
available and should be mapped via a macro to ast_faccessat().
Otherwise you're producing code bloat.

ast provides eaccess() in its portability layer and will continue to do so
*at() are still not universal so we chose to uniformly intercept calls
if eaccess() becomes a problem in the future we can deal with it then

one of the goals of this release was to leverage a lot of debugging hooks
with minimal source code change -- it is most likely not in official final
release form

Post by Irek Szczesniak
Finally, cd(1) is broken for error cases and returns garbage.

this is probably due to missteps in the first release of *at() emulation
roland finds some of those problems in subsequent posts

Glenn Fowler

2013-07-17 16:40:22 UTC

Permalink

Post by Cedric Blancher

thanks for the feedback

unfortunately we don't have acces to bsd machines anymore
bsd *never* did headers right
e.g., if posix says
#include <foo.h>
bsd takes it on itself to demand
#include <sys/hack.h>
#include <sys/hackier.h>
#include <foo.h>
so I'm not surprised that we hit macro clashes
send me offlist the files named by
bin/package results path
and if you did more than one build
bin/package results path old

we knew the intercepts would be controversial, especially the varargs ioctl()
but this is the best way we could think of to flesh out EINTR problems
that arose from the recent signal/queue storm tests
as far as we can tell few system calls in all ast libraries and commands are
immune from EINTR error returns, including surprising ones like close() and stat()

there is no way we could do the edit to wrap syscalls with restart logic,
possibly just for debugging purposes, in a timely manner
so we did it by default for all ast code via macro black magic
knowing that we may run afoul of others doing similar black magic
as the problems arise we'll address them
for now the default is to always intercept
but there is a way to build with intercepts disabled
-D_AST_INTERCEPT=0
*but don't do this for ast code*
in the future the default could change

the intercept approach addresses many issues
* adding restart logic, macros or not, to every piece of ast code would be fugly
and I would not like editing, debugging or maintaining that code on a daily basis
so please don't submit patches to restartify ast code
* whos to say some other issues like EINTR won't arise tomorrow -- with intercepts
we may have a much easier pathway to address those issues
* any user code that expects to be used in ksh builtins or ast plugins must
do the restart logic -- its much easier to instruct builtin developers to
"#include <ast.h>" than to "wrap each syscall with foomacro() barmacro()"
and the latter would have to be bullet proof -- not that easy to say months later
"oops, we should have said foomacro(special arg) barmacro(another special arg)";
if the user code doesn't do syscall macro intercepts it should go smoothly,
otherwise the users will have to "-D_AST_INTERCEPT=0" and examine the user
or 3rd party code for EINTR restartedness
* although syscall restart on interrupt is part of posix, no 2 unix implementations
apply restart in the same way on the same set of syscalls -- e.g., the intesection
between the ast intercepts and any unix implementation != ast intercepts
* intercepts can now portably handle O_* and F_* extensions like O_SEARCH and
F_DUPFD_CLOEXEC using standard macro names (with the help of iffe) and emulate those
features not supported in any particular implementation
* we now have a mechanism to trace and track down all EINTR probs and address them
as the signals are handled, including the possibility of adding interrupt handler
logic that can determine if a particular call should be restarted or is really hung

the crystal ball shows that roland will submit patches to handle some of the
build issues, so hopefully this will be a non-issue and the alpha debugging and testing
can proceed

I have to say that I was surprised at how hard it is to get non-threaded code to work
correctly in the face of signal storms, especially code like vmalloc that maintains
complex internal state -- in many cases its much more difficult to make signal-safe code
than thread-safe code

Irek Szczesniak

2013-06-29 21:22:55 UTC

Permalink

On Solaris 10 the attributes.sh test fails and prints this line in an
endless loop while consuming more and more memory (10MB/sec):
./src/cmd/ksh93/tests/attributes.sh: line 133: /dev/null: cannot open
[Permission denied]

Irek

Irek Szczesniak

2013-06-29 21:25:35 UTC

Permalink

Post by Irek Szczesniak

On Solaris 10 the attributes.sh test fails and prints this line in an
./src/cmd/ksh93/tests/attributes.sh: line 133: /dev/null: cannot open
[Permission denied]

shcomp test runs are broken for all tests:
test arith(shcomp) begins at 2013-06-29+23:23:19
./src/cmd/ksh93/tests/arith.sh: I/O error [Bad file number]
test arith(shcomp) failed to compile at 2013-06-29+23:23:19 with exit
code 1 [ 1 test 1 error ]

Irek