Discussion:
[ast-developers] cd(1) still broken on Solaris/AIX/etc. ...
Roland Mainz
2013-08-10 02:55:17 UTC
Permalink
Hi!

----

Attached (as "astksh20130807_solaris_cd_fixes001.diff.txt") is a patch
(technically the 4th or 5th attempt to submit this patch... ;-( ) to
get cd(1) fixed on platforms like Solaris/AIX/etc.

* Issues fixed:
- cd -@ shoud use correct error handling
- |rehash()| is a libc function on some systems and causes issues on
such platforms... the patch renames |rehash()| to
|rehash_relpathbindings()|
- Don't use |fstat()| to test for directory if we have |O_DIRECTORY|.
Fixes Solaris's samfs
- src/cmd/ksh93/include/shell.h - add a comment for |shp->pwdfd| to
explain which functions should be used to obtain this file descriptor

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
-------------- next part --------------
diff -r -u original/src/cmd/ksh93/bltins/cd_pwd.c build_cdfix/src/cmd/ksh93/bltins/cd_pwd.c
--- src/cmd/ksh93/bltins/cd_pwd.c 2013-07-22 20:59:06.000000000 +0200
+++ src/cmd/ksh93/bltins/cd_pwd.c 2013-08-10 04:34:50.715023380 +0200
@@ -41,7 +41,7 @@
/*
* Invalidate path name bindings to relative paths
*/
-static void rehash(register Namval_t *np,void *data)
+static void rehash_relpathbindings(register Namval_t *np,void *data)
{
Pathcomp_t *pp = (Pathcomp_t*)np->nvalue.cp;
NOT_USED(data);
@@ -49,6 +49,10 @@
_nv_unset(np,0);
}

+#if _ast_O_LOCAL && O_DIRECTORY >= _ast_O_LOCAL
+#define O_DIRECTORY_EMULATED 1
+#endif
+
/*
* Obtain a file handle to the directory "path" relative to directory
* "dir", or open a NFSv4 xattr directory handle for file dir/path.
@@ -56,8 +60,10 @@
int sh_diropenat(Shell_t *shp, int dir, const char *path, bool xattr)
{
int fd,shfd;
- int savederrno=errno;
+ int savederrno;
+#ifdef O_DIRECTORY_EMULATED
struct stat fs;
+#endif
#ifndef O_XATTR
NOT_USED(xattr);
#endif
@@ -70,7 +76,9 @@
if((apfd = openat(dir, path, O_RDONLY|O_NONBLOCK|O_CLOEXEC))>=0)
{
fd = openat(apfd, e_dot, O_XATTR|O_CLOEXEC);
+ savederrno=errno
close(apfd);
+ errno=savederrno;
}
}
else
@@ -79,12 +87,15 @@

if(fd < 0)
return fd;
+/* Only stat the fd if we don't have a native O_DIRECTORY */
+#ifdef O_DIRECTORY_EMULATED
if (!fstat(fd, &fs) && !S_ISDIR(fs.st_mode))
{
close(fd);
errno = ENOTDIR;
return -1;
}
+#endif

/* Move fd to a number > 10 and register the fd number with the shell */
shfd = sh_fcntl(fd, F_DUPFD_CLOEXEC, 10);
@@ -418,7 +429,7 @@
nv_putval(pwdnod,dir,NV_RDONLY);
nv_onattr(pwdnod,NV_NOFREE|NV_EXPORT);
shp->pwd = pwdnod->nvalue.cp;
- nv_scan(shp->track_tree,rehash,(void*)0,NV_TAGGED,NV_TAGGED);
+ nv_scan(shp->track_tree,rehash_relpathbindings,(void*)0,NV_TAGGED,NV_TAGGED);
path_newdir(shp,shp->pathlist);
path_newdir(shp,shp->cdpathlist);
if(oldpwd)
diff -r -u original/src/cmd/ksh93/include/shell.h build_cdfix/src/cmd/ksh93/include/shell.h
--- src/cmd/ksh93/include/shell.h 2013-07-23 03:12:23.000000000 +0200
+++ src/cmd/ksh93/include/shell.h 2013-08-10 04:34:50.716023502 +0200
@@ -155,7 +155,7 @@
char shcomp; /* set when runing shcomp */
short subshell; /* set for virtual subshell */
Stk_t *stk; /* stack poiter */
- int pwdfd; /* file descriptor for pwd */
+ int pwdfd; /* file descriptor for pwd (must be from sh_diropenat()/sh_fcntl()!) */
#ifdef _SH_PRIVATE
_SH_PRIVATE
#endif /* _SH_PRIVATE */
Glenn Fowler
2013-08-10 13:36:19 UTC
Permalink
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
or is it a more general problem that should be handled in a more general way

if any other ast command/library/user could trip up over the same rigmarole
then it belongs in a common place

and it still hasn't be explained how these fs extensions ignore
fs semantics and paradigms that until this point have been working fine since 1970

in particular, just how many commands in the wild need -@ args now?
is this an opportunity to do something like
/dev/rigmarole at .../...
and have it magically work for all ast commands/libs/plugins
--089e0149c390636d2604e38f0513
Content-Type: text/plain; charset=ISO-8859-1
Hi!
----
Attached (as "astksh20130807_solaris_cd_fixes001.diff.txt") is a patch
(technically the 4th or 5th attempt to submit this patch... ;-( ) to
get cd(1) fixed on platforms like Solaris/AIX/etc.
- |rehash()| is a libc function on some systems and causes issues on
such platforms... the patch renames |rehash()| to
|rehash_relpathbindings()|
- Don't use |fstat()| to test for directory if we have |O_DIRECTORY|.
Fixes Solaris's samfs
- src/cmd/ksh93/include/shell.h - add a comment for |shp->pwdfd| to
explain which functions should be used to obtain this file descriptor
----
Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
--089e0149c390636d2604e38f0513
Content-Type: text/plain; charset=US-ASCII;
name="astksh20130807_solaris_cd_fixes001.diff.txt"
Content-Disposition: attachment;
filename="astksh20130807_solaris_cd_fixes001.diff.txt"
Content-Transfer-Encoding: base64
X-Attachment-Id: f_hk6860240
ZGlmZiAtciAtdSBvcmlnaW5hbC9zcmMvY21kL2tzaDkzL2JsdGlucy9jZF9wd2QuYyBidWlsZF9j
ZGZpeC9zcmMvY21kL2tzaDkzL2JsdGlucy9jZF9wd2QuYwotLS0gc3JjL2NtZC9rc2g5My9ibHRp
bnMvY2RfcHdkLmMJMjAxMy0wNy0yMiAyMDo1OTowNi4wMDAwMDAwMDAgKzAyMDAKKysrIHNyYy9j
bWQva3NoOTMvYmx0aW5zL2NkX3B3ZC5jCTIwMTMtMDgtMTAgMDQ6MzQ6NTAuNzE1MDIzMzgwICsw
MjAwCkBAIC00MSw3ICs0MSw3IEBACiAvKgogICogSW52YWxpZGF0ZSBwYXRoIG5hbWUgYmluZGlu
Z3MgdG8gcmVsYXRpdmUgcGF0aHMKICAqLwotc3RhdGljIHZvaWQgcmVoYXNoKHJlZ2lzdGVyIE5h
bXZhbF90ICpucCx2b2lkICpkYXRhKQorc3RhdGljIHZvaWQgcmVoYXNoX3JlbHBhdGhiaW5kaW5n
cyhyZWdpc3RlciBOYW12YWxfdCAqbnAsdm9pZCAqZGF0YSkKIHsKIAlQYXRoY29tcF90ICpwcCA9
IChQYXRoY29tcF90KilucC0+bnZhbHVlLmNwOwogCU5PVF9VU0VEKGRhdGEpOwpAQCAtNDksNiAr
NDksMTAgQEAKIAkJX252X3Vuc2V0KG5wLDApOwogfQogCisjaWYgX2FzdF9PX0xPQ0FMICYmIE9f
RElSRUNUT1JZID49IF9hc3RfT19MT0NBTAorI2RlZmluZSBPX0RJUkVDVE9SWV9FTVVMQVRFRCAx
CisjZW5kaWYKKwogLyoKICAqIE9idGFpbiBhIGZpbGUgaGFuZGxlIHRvIHRoZSBkaXJlY3Rvcnkg
InBhdGgiIHJlbGF0aXZlIHRvIGRpcmVjdG9yeQogICogImRpciIsIG9yIG9wZW4gYSBORlN2NCB4
YXR0ciBkaXJlY3RvcnkgaGFuZGxlIGZvciBmaWxlIGRpci9wYXRoLgpAQCAtNTYsOCArNjAsMTAg
QEAKIGludCBzaF9kaXJvcGVuYXQoU2hlbGxfdCAqc2hwLCBpbnQgZGlyLCBjb25zdCBjaGFyICpw
YXRoLCBib29sIHhhdHRyKQogewogCWludCBmZCxzaGZkOwotCWludCBzYXZlZGVycm5vPWVycm5v
OworCWludCBzYXZlZGVycm5vOworI2lmZGVmIE9fRElSRUNUT1JZX0VNVUxBVEVECiAJc3RydWN0
IHN0YXQgZnM7CisjZW5kaWYKICNpZm5kZWYgT19YQVRUUgogCU5PVF9VU0VEKHhhdHRyKTsKICNl
bmRpZgpAQCAtNzAsNyArNzYsOSBAQAogCQlpZigoYXBmZCA9IG9wZW5hdChkaXIsIHBhdGgsIE9f
UkRPTkxZfE9fTk9OQkxPQ0t8T19DTE9FWEVDKSk+PTApCiAJCXsKIAkJCWZkID0gb3BlbmF0KGFw
ZmQsIGVfZG90LCBPX1hBVFRSfE9fQ0xPRVhFQyk7CisJCQlzYXZlZGVycm5vPWVycm5vCiAJCQlj
bG9zZShhcGZkKTsKKwkJCWVycm5vPXNhdmVkZXJybm87CiAJCX0KIAl9CiAJZWxzZQpAQCAtNzks
MTIgKzg3LDE1IEBACiAKIAlpZihmZCA8IDApCiAJCXJldHVybiBmZDsKKy8qIE9ubHkgc3RhdCB0
aGUgZmQgaWYgd2UgZG9uJ3QgaGF2ZSBhIG5hdGl2ZSBPX0RJUkVDVE9SWSAqLworI2lmZGVmIE9f
RElSRUNUT1JZX0VNVUxBVEVECiAJaWYgKCFmc3RhdChmZCwgJmZzKSAmJiAhU19JU0RJUihmcy5z
dF9tb2RlKSkKIAl7CiAJCWNsb3NlKGZkKTsKIAkJZXJybm8gPSBFTk9URElSOwogCQlyZXR1cm4g
LTE7CiAJfQorI2VuZGlmCiAKIAkvKiBNb3ZlIGZkIHRvIGEgbnVtYmVyID4gMTAgYW5kIHJlZ2lz
dGVyIHRoZSBmZCBudW1iZXIgd2l0aCB0aGUgc2hlbGwgKi8KIAlzaGZkID0gc2hfZmNudGwoZmQs
IEZfRFVQRkRfQ0xPRVhFQywgMTApOwpAQCAtNDE4LDcgKzQyOSw3IEBACiAJbnZfcHV0dmFsKHB3
ZG5vZCxkaXIsTlZfUkRPTkxZKTsKIAludl9vbmF0dHIocHdkbm9kLE5WX05PRlJFRXxOVl9FWFBP
UlQpOwogCXNocC0+cHdkID0gcHdkbm9kLT5udmFsdWUuY3A7Ci0JbnZfc2NhbihzaHAtPnRyYWNr
X3RyZWUscmVoYXNoLCh2b2lkKikwLE5WX1RBR0dFRCxOVl9UQUdHRUQpOworCW52X3NjYW4oc2hw
LT50cmFja190cmVlLHJlaGFzaF9yZWxwYXRoYmluZGluZ3MsKHZvaWQqKTAsTlZfVEFHR0VELE5W
X1RBR0dFRCk7CiAJcGF0aF9uZXdkaXIoc2hwLHNocC0+cGF0aGxpc3QpOwogCXBhdGhfbmV3ZGly
KHNocCxzaHAtPmNkcGF0aGxpc3QpOwogCWlmKG9sZHB3ZCkKZGlmZiAtciAtdSBvcmlnaW5hbC9z
cmMvY21kL2tzaDkzL2luY2x1ZGUvc2hlbGwuaCBidWlsZF9jZGZpeC9zcmMvY21kL2tzaDkzL2lu
Y2x1ZGUvc2hlbGwuaAotLS0gc3JjL2NtZC9rc2g5My9pbmNsdWRlL3NoZWxsLmgJMjAxMy0wNy0y
MyAwMzoxMjoyMy4wMDAwMDAwMDAgKzAyMDAKKysrIHNyYy9jbWQva3NoOTMvaW5jbHVkZS9zaGVs
bC5oCTIwMTMtMDgtMTAgMDQ6MzQ6NTAuNzE2MDIzNTAyICswMjAwCkBAIC0xNTUsNyArMTU1LDcg
QEAKIAljaGFyCQlzaGNvbXA7CQkvKiBzZXQgd2hlbiBydW5pbmcgc2hjb21wICovCiAJc2hvcnQJ
CXN1YnNoZWxsOwkvKiBzZXQgZm9yIHZpcnR1YWwgc3Vic2hlbGwgKi8KIAlTdGtfdAkJKnN0azsJ
CS8qIHN0YWNrIHBvaXRlciAqLwotCWludAkJcHdkZmQ7CQkvKiBmaWxlIGRlc2NyaXB0b3IgZm9y
IHB3ZCAqLworCWludAkJcHdkZmQ7CQkvKiBmaWxlIGRlc2NyaXB0b3IgZm9yIHB3ZCAobXVzdCBi
ZSBmcm9tIHNoX2Rpcm9wZW5hdCgpL3NoX2ZjbnRsKCkhKSAqLwogI2lmZGVmIF9TSF9QUklWQVRF
CiAJX1NIX1BSSVZBVEUKICNlbmRpZiAvKiBfU0hfUFJJVkFURSAqLwo=
--089e0149c390636d2604e38f0513--
Irek Szczesniak
2013-08-10 15:19:29 UTC
Permalink
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
Post by Glenn Fowler
or is it a more general problem that should be handled in a more general way
No, I don't think so. The XATTR directory is a solution to implement
filesystem forks (not fork(), file resource forks as
http://en.wikipedia.org/wiki/Fork_%28file_system%29 describes).
Post by Glenn Fowler
if any other ast command/library/user could trip up over the same rigmarole
then it belongs in a common place
and it still hasn't be explained how these fs extensions ignore
fs semantics and paradigms that until this point have been working fine since 1970
Go and ask Apple, Novell, Microsoft why they did it? Sun only provided
support for this so their servers can implement this feature to host
files with these semantics.
cd(1), maybe ls(1). Roland may have a better overview.
Post by Glenn Fowler
is this an opportunity to do something like
/dev/rigmarole at .../...
and have it magically work for all ast commands/libs/plugins
Hell NO.
1. In my opinion you're trying to overdesign this
2. You're breaking established features (first O_SEARCH, now cd -@).
IMO it would be beneficial to get something working and then optimize
it. Right now both feature are broken and that severely interrupts
testing, development and operations on our side and cripples the
ability to give feedback.

Irek
Glenn Fowler
2013-08-10 15:34:54 UTC
Permalink
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Irek Szczesniak
2013-08-10 15:38:19 UTC
Permalink
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
You can cd -@ into all filesystem objects and not only directories.
Links included. And they all have their own resource forks/XATTR.

Irek
Glenn Fowler
2013-08-10 15:44:29 UTC
Permalink
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
maybe I want to pax resource forks and everything
or, don't tell me, read(2)/write(2) don't work on resource forks
Irek Szczesniak
2013-08-10 15:54:56 UTC
Permalink
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
Because it's a rare and rarely used ability and feature to deal with
resource forks/XATTR files? I do understand why Roland did it and
Cedric and my people own like it. We're both Apple shops in our labs
and the admins need a way to access file files in emergencies. But I'm
not convinced that putting resource fork/XATTR support into every
utility is a good idea.
Post by Glenn Fowler
maybe I want to pax resource forks and everything
or, don't tell me, read(2)/write(2) don't work on resource forks
open(), read(), write(), mmap() all works for resource forks/XATTR.
They just don't have a name relative to / because neither Apple,
Microsoft or Novell came up with a naming scheme, likely because forks
are 'private' to the parent file they are a child node of.

Solaris /usr/bin/tar can backup resource forks/XATTR files, but I
don't know how to do that. Roland likely knows.

Irek
Glenn Fowler
2013-08-10 16:09:25 UTC
Permalink
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
Because it's a rare and rarely used ability and feature to deal with
resource forks/XATTR files? I do understand why Roland did it and
Cedric and my people own like it. We're both Apple shops in our labs
and the admins need a way to access file files in emergencies. But I'm
not convinced that putting resource fork/XATTR support into every
utility is a good idea.
but isn't "putting resource fork/XATTR support into every utility" exactly what cd -@ does?
after all, 'cd -@ barf; ls' will affect ls -- will ls need to know its inside a fork?

I'm balking at this because I know (but can't enumerate) that -@ infects more than cd(1)
and the patches for cd -@ are fragile (not the patcher's fault, its weirdnix)
when chunks of code like this are so fragile, at least I want to understand what's going on
including and especially the scope

suppose we finally get cd -@ working to everone's satisfaction
what is the inevitable "add -@ to foo" patch going to look like?
the same fragility, the *same* code?
Irek Szczesniak
2013-08-10 16:20:08 UTC
Permalink
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
Because it's a rare and rarely used ability and feature to deal with
resource forks/XATTR files? I do understand why Roland did it and
Cedric and my people own like it. We're both Apple shops in our labs
and the admins need a way to access file files in emergencies. But I'm
not convinced that putting resource fork/XATTR support into every
utility is a good idea.
No. I don't think so.
Weirdnix here is that you cd(1) into a file. But we've seen this
before with reiser4, smbfs and samfs. That's why
fstat()/S_ISDIR(fs.st_mode) is a bad idea.
Post by Glenn Fowler
when chunks of code like this are so fragile, at least I want to understand what's going on
including and especially the scope
What's fragile in this case? As long as you don't ask the pwd(1)
question in a XATTR directory 'all is well'.
Post by Glenn Fowler
the same fragility, the *same* code?
With ls(1) as the exception I don't remember or imagine another
consumer of - at . pax(1) doesn't count because its a filesystem in a
bottle.

Irek
Glenn Fowler
2013-08-10 17:55:22 UTC
Permalink
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
Because it's a rare and rarely used ability and feature to deal with
resource forks/XATTR files? I do understand why Roland did it and
Cedric and my people own like it. We're both Apple shops in our labs
and the admins need a way to access file files in emergencies. But I'm
not convinced that putting resource fork/XATTR support into every
utility is a good idea.
I have access to a stock
Darwin * 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
can you point me to anything that may have a fork/XATTR
or provide a script that creates a { file directory symlink } each with a fork
Roland Mainz
2013-08-10 20:46:32 UTC
Permalink
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
Because it's a rare and rarely used ability and feature to deal with
resource forks/XATTR files? I do understand why Roland did it and
Cedric and my people own like it. We're both Apple shops in our labs
and the admins need a way to access file files in emergencies. But I'm
not convinced that putting resource fork/XATTR support into every
utility is a good idea.
I have access to a stock
Darwin * 12.4.0 Darwin Kernel Version 12.4.0: Wed May 1 17:57:12 PDT 2013; root:xnu-2050.24.15~1/RELEASE_X86_64 x86_64
can you point me to anything that may have a fork/XATTR
or provide a script that creates a { file directory symlink } each with a fork
See http://lists.research.att.com/pipermail/ast-developers/2013q3/003064.html
... on OSX there are currently some custom APIs to access resource
forks. But that's what
http://lists.research.att.com/pipermail/ast-developers/2013q3/003064.html
explains...

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Roland Mainz
2013-08-10 20:32:44 UTC
Permalink
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
maybe I want to pax resource forks and everything
That may be very interesting... but it quickly gets very painfull
because each damn OS (WinNT, MacOS X) all have their custom resource
fork APIs. AFAIK none of the people implementing pax/tar have _ever_
tried that.

And... please... before we do that we should look at ACLs first.
That's AFAIK more interesting and rewarding.
Post by Glenn Fowler
or, don't tell me, read(2)/write(2) don't work on resource forks
Heh. No no... beyond being hidden behind |O_XATTR| they are normal
files. symlinks/hardlinks/etc. all work (but hardlinks are restricted
to that directory level... so forget funny things like $ ln
myxattrfile /etc/profile # (NFSv4 doesn't forbit it... but the
underlying filesystems do)).

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Cedric Blancher
2013-08-11 15:59:16 UTC
Permalink
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
The word "overkill" comes in my mind. File system alternate streams
AKA resource forks AKA NFSv4 XATTR are quite specialised. Usually the
OS uses it to store things like icon data (small picture of the large
image), mime data, http cookies or in our case a cache of precomputed
data to speed processing up. These are all things which are PRIVATE to
the application doing the processing.

Ced
--
Cedric Blancher <cedric.blancher at gmail.com>
Institute Pasteur
Glenn Fowler
2013-08-12 16:25:38 UTC
Permalink
Post by Cedric Blancher
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
Post by Irek Szczesniak
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
IMO yes, because entering a NFSv4 xattr directory is unique for a
shell. Otherwise the only way to do it is to run /usr/bin/runat <obj>
<prog> which is cumbersome at best and useless if you have builtins.
how is the chdir()/fchdir() done by cd(1) different from the chdir()/fchdir()
done by find(1) or tw(1) or any of the -R commands?
Links included. And they all have their own resource forks/XATTR.
ok, but why should/shouldn't the other commands mentioned above be able to the same?
The word "overkill" comes in my mind. File system alternate streams
AKA resource forks AKA NFSv4 XATTR are quite specialised. Usually the
OS uses it to store things like icon data (small picture of the large
image), mime data, http cookies or in our case a cache of precomputed
data to speed processing up. These are all things which are PRIVATE to
the application doing the processing.
sorry for all the back and forth on this
but you have to look at the view dgk and I see
a storm of patches, sometimes dozens in a week, some related, some not
some dealing with features we have never used and possibly have a hard time reproducing
I'm at the point of needing to stand back to understand just where these patches are leading
in particular I'm concerned about trickles of patches leading ast to a weird place
that might have a better abstraction when taken as a whole

my concern for "cd -@" was to make sure it doesn't lead to overkill and a stream of -@'ify patches
Roland Mainz
2013-08-10 20:27:23 UTC
Permalink
Post by Glenn Fowler
Post by Roland Mainz
Attached (as "astksh20130807_solaris_cd_fixes001.diff.txt") is a patch
(technically the 4th or 5th attempt to submit this patch... ;-( ) to
get cd(1) fixed on platforms like Solaris/AIX/etc.
- |rehash()| is a libc function on some systems and causes issues on
such platforms... the patch renames |rehash()| to
|rehash_relpathbindings()|
- Don't use |fstat()| to test for directory if we have |O_DIRECTORY|.
Fixes Solaris's samfs
- src/cmd/ksh93/include/shell.h - add a comment for |shp->pwdfd| to
explain which functions should be used to obtain this file descriptor
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
or is it a more general problem that should be handled in a more general way
I'm not sure... for now I've implemented cd -@ based the
furious/frustrated stories of Sun's PIT (=PreIntegration Testing)
people in Ireland who had no way to look at the XATTR files in any
usefull way.

For now I would keep it at this level (e.g. cd -@, ls -@, find -xattr
and "pax" support) until we have more feedback from the users what
they intend to do with it. And we only get that if $ cd -@ myfile #
works properly...
Post by Glenn Fowler
if any other ast command/library/user could trip up over the same rigmarole
then it belongs in a common place
Agreed... but please please not this time. Not for the next couple of alphas.
The concept of |O_XATTR| is fairly new and is making it only slowly
into the various OSes and filesystems. |O_XATTR| is a solution of
combining&&unifying ([1]) the concepts of the so-called "resource
forks" on Apple (pre-OSX), Amiga's INFO files, WindowsNT's "Alternate
Data Streams" and almost all other file metadata concepts one API.
The basic idea is: You can have metadata of unlimited size (not
limited-sized string APIs), store it "near" the file object it belongs
to and they are removed when the file gets removed (anyone recalling
AmigaOS... if you remove files from the shell you could've left-over
INFO files).

[1]=For example Apple OS (pre-OSX and OSX) doesn't have traditional
filesystem APIs to access resource forks. However a NFSv4 filesystem
client module will translate all the custom resource fork APIs into
|O_XATTR| calls to the NFSv4 server and the server will understand
what's meant.
The same applies to "Alternate Data Streams" ... Windows uses custom
APIs for this but an smbfs client translates these calls again into
|O_XATTR| file accesses...

Note that this means you should not try to put that into libast or
UWIN and emulate |O_XATTR| ... that's even more painfull than the work
for ACLs.
The good side is that |O_XATTR| slowly filtering into the OSes because
both smbfs and NFSv4 need it and the existing limited-size string APIs
are horrible...
Post by Glenn Fowler
and it still hasn't be explained how these fs extensions ignore
fs semantics and paradigms that until this point have been working fine since 1970
Grumpf... the issue is that there is no POSIX-conforming way to do it
unless you restrict the namespace available for filenames. The other
issue is that the XATTR files should be removed with the "parent file
object" causing even more semantic pain. Finally Sun's enginners
decided to avoid the issue completely for now and just don't decide on
a naming scheme at all.
Post by Glenn Fowler
is this an opportunity to do something like
/dev/rigmarole at .../...
and have it magically work for all ast commands/libs/plugins
Not really... I already thought about /dev/xattr@ and decided against
it because it raises more dead issues from the grave than it solves.
Right now the |O_XATTR| directories are "flat", e.g. no subdirectories
are allowed in the Sun _implementation_... but the code in NFSv4
explicitly allows subdirs... which makes it hard to come-up with a
naming scheme for the same reason Sun didn't come-up with a naming
scheme to represent such files in the filesystem itself (and I don't
want to start with { \t, \v etc. } ... Sun's engineers tried that as
experiment and promptly broke ar(1) with that).

Thinking about it... IMO the best thing is really really to just fix
cd(1) and see what users do with it...

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Glenn Fowler
2013-08-10 20:53:11 UTC
Permalink
Post by Roland Mainz
Post by Glenn Fowler
Post by Roland Mainz
Attached (as "astksh20130807_solaris_cd_fixes001.diff.txt") is a patch
(technically the 4th or 5th attempt to submit this patch... ;-( ) to
get cd(1) fixed on platforms like Solaris/AIX/etc.
- |rehash()| is a libc function on some systems and causes issues on
such platforms... the patch renames |rehash()| to
|rehash_relpathbindings()|
- Don't use |fstat()| to test for directory if we have |O_DIRECTORY|.
Fixes Solaris's samfs
- src/cmd/ksh93/include/shell.h - add a comment for |shp->pwdfd| to
explain which functions should be used to obtain this file descriptor
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
or is it a more general problem that should be handled in a more general way
furious/frustrated stories of Sun's PIT (=PreIntegration Testing)
people in Ireland who had no way to look at the XATTR files in any
usefull way.
and "pax" support) until we have more feedback from the users what
works properly...
Post by Glenn Fowler
if any other ast command/library/user could trip up over the same rigmarole
then it belongs in a common place
Agreed... but please please not this time. Not for the next couple of alphas.
The concept of |O_XATTR| is fairly new and is making it only slowly
into the various OSes and filesystems. |O_XATTR| is a solution of
combining&&unifying ([1]) the concepts of the so-called "resource
forks" on Apple (pre-OSX), Amiga's INFO files, WindowsNT's "Alternate
Data Streams" and almost all other file metadata concepts one API.
The basic idea is: You can have metadata of unlimited size (not
limited-sized string APIs), store it "near" the file object it belongs
to and they are removed when the file gets removed (anyone recalling
AmigaOS... if you remove files from the shell you could've left-over
INFO files).
[1]=For example Apple OS (pre-OSX and OSX) doesn't have traditional
filesystem APIs to access resource forks. However a NFSv4 filesystem
client module will translate all the custom resource fork APIs into
|O_XATTR| calls to the NFSv4 server and the server will understand
what's meant.
The same applies to "Alternate Data Streams" ... Windows uses custom
APIs for this but an smbfs client translates these calls again into
|O_XATTR| file accesses...
Note that this means you should not try to put that into libast or
UWIN and emulate |O_XATTR| ... that's even more painfull than the work
for ACLs.
The good side is that |O_XATTR| slowly filtering into the OSes because
both smbfs and NFSv4 need it and the existing limited-size string APIs
are horrible...
Post by Glenn Fowler
and it still hasn't be explained how these fs extensions ignore
fs semantics and paradigms that until this point have been working fine since 1970
Grumpf... the issue is that there is no POSIX-conforming way to do it
unless you restrict the namespace available for filenames. The other
issue is that the XATTR files should be removed with the "parent file
object" causing even more semantic pain. Finally Sun's enginners
decided to avoid the issue completely for now and just don't decide on
a naming scheme at all.
Post by Glenn Fowler
is this an opportunity to do something like
/dev/rigmarole at .../...
and have it magically work for all ast commands/libs/plugins
it because it raises more dead issues from the grave than it solves.
Right now the |O_XATTR| directories are "flat", e.g. no subdirectories
are allowed in the Sun _implementation_... but the code in NFSv4
explicitly allows subdirs... which makes it hard to come-up with a
naming scheme for the same reason Sun didn't come-up with a naming
scheme to represent such files in the filesystem itself (and I don't
want to start with { \t, \v etc. } ... Sun's engineers tried that as
experiment and promptly broke ar(1) with that).
Thinking about it... IMO the best thing is really really to just fix
cd(1) and see what users do with it...
thanks for the background

we did something similar with version files in 3d fs
we encroached on the namespace by taking "..."
if you want the latest version of file foo name it
"foo"
if you want to see all versions of foo and other resources name it
"foo/..."
if you want to see version bar of file foo
"foo/.../bar"

we didn't use it much beyond proof of concept
so e.g. readdir() did not return "..." files
Roland Mainz
2013-08-10 21:04:16 UTC
Permalink
Post by Glenn Fowler
Post by Roland Mainz
Post by Glenn Fowler
Post by Roland Mainz
Attached (as "astksh20130807_solaris_cd_fixes001.diff.txt") is a patch
(technically the 4th or 5th attempt to submit this patch... ;-( ) to
get cd(1) fixed on platforms like Solaris/AIX/etc.
- |rehash()| is a libc function on some systems and causes issues on
such platforms... the patch renames |rehash()| to
|rehash_relpathbindings()|
- Don't use |fstat()| to test for directory if we have |O_DIRECTORY|.
Fixes Solaris's samfs
- src/cmd/ksh93/include/shell.h - add a comment for |shp->pwdfd| to
explain which functions should be used to obtain this file descriptor
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
or is it a more general problem that should be handled in a more general way
furious/frustrated stories of Sun's PIT (=PreIntegration Testing)
people in Ireland who had no way to look at the XATTR files in any
usefull way.
and "pax" support) until we have more feedback from the users what
works properly...
Post by Glenn Fowler
if any other ast command/library/user could trip up over the same rigmarole
then it belongs in a common place
Agreed... but please please not this time. Not for the next couple of alphas.
The concept of |O_XATTR| is fairly new and is making it only slowly
into the various OSes and filesystems. |O_XATTR| is a solution of
combining&&unifying ([1]) the concepts of the so-called "resource
forks" on Apple (pre-OSX), Amiga's INFO files, WindowsNT's "Alternate
Data Streams" and almost all other file metadata concepts one API.
The basic idea is: You can have metadata of unlimited size (not
limited-sized string APIs), store it "near" the file object it belongs
to and they are removed when the file gets removed (anyone recalling
AmigaOS... if you remove files from the shell you could've left-over
INFO files).
[1]=For example Apple OS (pre-OSX and OSX) doesn't have traditional
filesystem APIs to access resource forks. However a NFSv4 filesystem
client module will translate all the custom resource fork APIs into
|O_XATTR| calls to the NFSv4 server and the server will understand
what's meant.
The same applies to "Alternate Data Streams" ... Windows uses custom
APIs for this but an smbfs client translates these calls again into
|O_XATTR| file accesses...
Note that this means you should not try to put that into libast or
UWIN and emulate |O_XATTR| ... that's even more painfull than the work
for ACLs.
The good side is that |O_XATTR| slowly filtering into the OSes because
both smbfs and NFSv4 need it and the existing limited-size string APIs
are horrible...
Post by Glenn Fowler
and it still hasn't be explained how these fs extensions ignore
fs semantics and paradigms that until this point have been working fine since 1970
Grumpf... the issue is that there is no POSIX-conforming way to do it
unless you restrict the namespace available for filenames. The other
issue is that the XATTR files should be removed with the "parent file
object" causing even more semantic pain. Finally Sun's enginners
decided to avoid the issue completely for now and just don't decide on
a naming scheme at all.
Post by Glenn Fowler
is this an opportunity to do something like
/dev/rigmarole at .../...
and have it magically work for all ast commands/libs/plugins
it because it raises more dead issues from the grave than it solves.
Right now the |O_XATTR| directories are "flat", e.g. no subdirectories
are allowed in the Sun _implementation_... but the code in NFSv4
explicitly allows subdirs... which makes it hard to come-up with a
naming scheme for the same reason Sun didn't come-up with a naming
scheme to represent such files in the filesystem itself (and I don't
want to start with { \t, \v etc. } ... Sun's engineers tried that as
experiment and promptly broke ar(1) with that).
Thinking about it... IMO the best thing is really really to just fix
cd(1) and see what users do with it...
thanks for the background
we did something similar with version files in 3d fs
we encroached on the namespace by taking "..."
if you want the latest version of file foo name it
"foo"
if you want to see all versions of foo and other resources name it
"foo/..."
if you want to see version bar of file foo
"foo/.../bar"
we didn't use it much beyond proof of concept
so e.g. readdir() did not return "..." files
I know... "..." works... but looking at my parents Windows and OS X
machine I think every other sane/insane naming scheme is already in
use by Windows applications. And the users... the users... make it
worse. You'd have files like "foobar[2].backup at 4.doc" and the users
turn that into "foobar[2].backup at 4.doc -->{fuer morgen, geht um
$$$$!!}<--" (short... they may put real-world language (comments,
notes, curses etc.) with ASCII art into filenames... including the use
of '&', '*', '^', '|' etc.).

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Glenn Fowler
2013-08-11 14:33:43 UTC
Permalink
Post by Glenn Fowler
thanks for the background
we did something similar with version files in 3d fs
we encroached on the namespace by taking "..."
if you want the latest version of file foo name it
"foo"
if you want to see all versions of foo and other resources name it
"foo/..."
if you want to see version bar of file foo
"foo/.../bar"
we didn't use it much beyond proof of concept
so e.g. readdir() did not return "..." files
I typed more than thought yesterday
3d does use "..." but not for versioning
we only versioned regular files
a versioned filed accessed as
"foo"
to see all the versions acces the file as a directory
"foo/"
pysically the latest version was a hard link to a file like
"foo/1.2.3.4"
had we versioned directories we would have had to encroach the namespace
Cedric Blancher
2013-08-11 15:48:11 UTC
Permalink
Post by Glenn Fowler
Post by Glenn Fowler
thanks for the background
we did something similar with version files in 3d fs
we encroached on the namespace by taking "..."
if you want the latest version of file foo name it
"foo"
if you want to see all versions of foo and other resources name it
"foo/..."
if you want to see version bar of file foo
"foo/.../bar"
we didn't use it much beyond proof of concept
so e.g. readdir() did not return "..." files
I typed more than thought yesterday
3d does use "..." but not for versioning
we only versioned regular files
a versioned filed accessed as
"foo"
to see all the versions acces the file as a directory
"foo/"
pysically the latest version was a hard link to a file like
"foo/1.2.3.4"
had we versioned directories we would have had to encroach the namespace
Why? The standard way to add version numbers to files is
"filename;version". See ISO9660.
Post by Glenn Fowler
_______________________________________________
ast-developers mailing list
ast-developers at lists.research.att.com
http://lists.research.att.com/mailman/listinfo/ast-developers
Ced
--
Cedric Blancher <cedric.blancher at gmail.com>
Institute Pasteur
Glenn Fowler
2013-08-12 16:10:27 UTC
Permalink
Post by Cedric Blancher
Post by Glenn Fowler
Post by Glenn Fowler
thanks for the background
we did something similar with version files in 3d fs
we encroached on the namespace by taking "..."
if you want the latest version of file foo name it
"foo"
if you want to see all versions of foo and other resources name it
"foo/..."
if you want to see version bar of file foo
"foo/.../bar"
we didn't use it much beyond proof of concept
so e.g. readdir() did not return "..." files
I typed more than thought yesterday
3d does use "..." but not for versioning
we only versioned regular files
a versioned filed accessed as
"foo"
to see all the versions acces the file as a directory
"foo/"
pysically the latest version was a hard link to a file like
"foo/1.2.3.4"
had we versioned directories we would have had to encroach the namespace
Why? The standard way to add version numbers to files is
"filename;version". See ISO9660.
* because it did not require a special char like ';'
* keeps versions and any other attributes neatly tucked in a subdir
* determining the latest version is trivial: foo.c/latest
Cedric Blancher
2013-08-11 15:54:41 UTC
Permalink
Post by Glenn Fowler
it still has not been explained why special fs treatment like this
goes into ksh and not src/lib/libast/something
is this really something *only* ksh will trip over
or is it a more general problem that should be handled in a more general way
if any other ast command/library/user could trip up over the same rigmarole
then it belongs in a common place
and it still hasn't be explained how these fs extensions ignore
fs semantics and paradigms that until this point have been working fine since 1970
is this an opportunity to do something like
/dev/rigmarole at .../...
and have it magically work for all ast commands/libs/plugins
Is the discussion now finished? I don't like to join the mayhem, but
as a humble user I'd like to politely ask if could you please just
*fix* cd -@ pathname for the next release? We use it in production
scripts, mostly for tagging or adding parsed/precompiled versions of
the original files, to speed BLAST (blastp, psi-blast, phi-blast)
processing up.

Ced
--
Cedric Blancher <cedric.blancher at gmail.com>
Institute Pasteur
Glenn Fowler
2013-08-15 16:54:13 UTC
Permalink
some questions / observations / rambling on cd -@

on a system that has O_XATTR is there an errno for
open(valid_fd, O_XATTR|O_RDONLY)
failing because valid_fd has no extended attributes?

reason: I believe that ksh cd should always accept -@ and return
the errno from above, even on systems that have no O_XATTR

I'm on a 'SunOS hostname 5.11 11.0 i86pc i386 i86pc' and
wanted to compare /bin/sh and /bin/bash handling of cd -@ vs ksh
and neither support cd -@ -- nice -- I wonder what jumbo patch has those
don't answer that, I'm not the solaris maintenance loop

also, and this goes to my complaints that there is no path notation
for xattr dirs, when any shell /bin/bash is running from an -@ dir
it chokes because it can't determine pwd (other shells, ksh too, have
problems with that too, more on that below)

some other weirdness from an interactive session,
again because there is no path notation
--
$ pwd
/home/gsf/tst/ksh
$ mkdir tmp
$ cd -@ tmp
$ /bin/bash -c pwd
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
$ /bin/sh -c pwd
pwd: cannot access parent directories [No such file or directory]
$ /bin/pwd
pwd: cannot determine current directory!
$ ksh -c pwd
pwd: cannot access parent directories [No such file or directory]
$ # ouch ksh bit by this too -- does olga's fgetcwd() handle this right?
$ pwd
/home/gsf/tst/ksh/tmp
$ ls
SUNWattr_ro SUNWattr_rw
$ cd -@ SUNWattr_ro
/home/gsf/arch/sol11.i386/bin/ksh: cd: SUNWattr_ro: [Not a directory]
$ # is that the right errno?
$ cd -
/home/gsf/tst/ksh
$ pwd
/home/gsf/tst/ksh
$ cd -
/home/gsf/tst/ksh/tmp
$ pwd
/home/gsf/tst/ksh/tmp
$ ls
$ # oops -- not in the -@ dir -- but we are in ~- # oh right, xattrs dirs don't have pathnames
--

so a lot of stuff going on there, most of it because -@ dirs are half-baked
the worst I guess is that when you are in an -@ dir you can't tell
(or can you -- does stat(2) or some other new syscall?)
you get a hint when the /bin/pwd algorithm breaks, but it can break for other
reasons that may give similar behavior

one could argue that once you 'cd -@' you better know what you are doing
but I bet the /bin/bash and /bin/pwd authors thought they knew what they were doing

I belive most of these oddities, and even the need to patch /bin/sh and /bin/bash and ksh,
could have been avoided if xattrs simply stole *something* from the path namespace
even a (pseudo) mount point or /dev entry would work

/dev/xattr/./tmp # . relative path
/dev/xattr/home/gsf/tst/ksh/tmp # absolute path

this might be able to be emulated if there were an is_this_an_xattr_dirat(AT_FDCWD, ".") call

is it possible to determine if "." is an extended attr dir?
maybe ls gives a hint?

$ /bin/ls -l .
total 2
-r--r--r-- 1 root root 156 Aug 15 11:15 SUNWattr_ro
-rw-r--r-- 1 root root 472 Aug 15 11:15 SUNWattr_rw
$ /bin/ls -ld .
drwxrwxrwt 4 gsf gsf 4 Aug 15 11:15 .
$ /bin/ls - at ld .
drwxrwxrwt 4 gsf gsf 4 Aug 15 11:15 .
$ /bin/df .
df: cannot canonicalize .: No such file or directory

hmm, something else: surely my xattr dir is not world writeable, is it?
I can create and delete files in there
and the sticky bit prevents me from modifying the root files

just checked man fsattr and saw the these ast-related commands had to be
- at -ified or -xattr-ified:

cp du find ls mv pax (and they forgot cd)

$ cd -@ .
/home/gsf/arch/sol11.i386/bin/ksh: cd: .: [Invalid argument]

AHA this might be the hook:

$ cd /home/gsf/tst/ksh/tmp
$ pwd
$ ls
$ cd -@ .
/home/gsf/arch/sol11.i386/bin/ksh: cd: .: [Invalid argument]

so it looks like openat(xattrfd, O_XATTR|O_RDONLY) fails with EINVAL
for an open xattr fd
Glenn Fowler
2013-08-15 17:03:47 UTC
Permalink
I should have mentioned that the interactive session was done with the latest ksh
Post by Glenn Fowler
$ cd /home/gsf/tst/ksh/tmp
$ pwd
$ ls
/home/gsf/arch/sol11.i386/bin/ksh: cd: .: [Invalid argument]
I messed up this sequence
partly due to 'cd -' + 'pwd' making me think I was in a
physical dir when I was really in an xattr dir
moral: when playing with 'cd -@' don't rely on pwd (builtin or /bin/pwd) for context

trying again:

$ cd /home/gsf/tst/ksh/tmp
$ pwd
/home/gsf/tst/ksh/tmp
$ ls
$ cd -@ .
$ ls
SUNWattr_ro SUNWattr_rw
$ cd -@ .
/home/gsf/arch/sol11.i386/bin/ksh: cd: .: [Invalid argument]
Post by Glenn Fowler
so it looks like openat(xattrfd, O_XATTR|O_RDONLY) fails with EINVAL
for an open xattr fd
Cedric Blancher
2013-08-16 08:06:10 UTC
Permalink
Post by Glenn Fowler
on a system that has O_XATTR is there an errno for
open(valid_fd, O_XATTR|O_RDONLY)
failing because valid_fd has no extended attributes?
If a filesystem supports NFSv4 extended attributes then all elements
in this filesystem, including files, dirs, pipes, fifios etc support
them.
If a filesystem doesn't support O_XATTR it will return EINVAL, try it
with /devices (devfs, device file system):
ksh -c 'cd -@ /devices/pseudo'
/home/ced/bin/ksh: cd: /devices/pseudo: [Invalid argument]
Post by Glenn Fowler
the errno from above, even on systems that have no O_XATTR
You better return ENOTSUP then
Post by Glenn Fowler
I'm on a 'SunOS hostname 5.11 11.0 i86pc i386 i86pc' and
don't answer that, I'm not the solaris maintenance loop
also, and this goes to my complaints that there is no path notation
it chokes because it can't determine pwd (other shells, ksh too, have
problems with that too, more on that below)
What about returning the shp->pwdfd path, i.e.
/proc/<pid>/fd/${shp.pwdfd}/ as physical path? Works great here:
ksh -c 'cd -@ . ; redirect {n}<. ; cd -f $n ; pwd ; :'
/proc/14894/fd/12
Post by Glenn Fowler
some other weirdness from an interactive session,
again because there is no path notation
--
$ pwd
/home/gsf/tst/ksh
$ mkdir tmp
$ /bin/bash -c pwd
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
$ /bin/sh -c pwd
pwd: cannot access parent directories [No such file or directory]
$ /bin/pwd
pwd: cannot determine current directory!
$ ksh -c pwd
pwd: cannot access parent directories [No such file or directory]
$ # ouch ksh bit by this too -- does olga's fgetcwd() handle this right?
$ pwd
/home/gsf/tst/ksh/tmp
$ ls
SUNWattr_ro SUNWattr_rw
/home/gsf/arch/sol11.i386/bin/ksh: cd: SUNWattr_ro: [Not a directory]
$ # is that the right errno?
$ cd -
/home/gsf/tst/ksh
$ pwd
/home/gsf/tst/ksh
$ cd -
/home/gsf/tst/ksh/tmp
$ pwd
/home/gsf/tst/ksh/tmp
$ ls
--
I don't think O_XATTR dirs are half-baked. They just don't have an
anchor which is connected to / directly. It's even worse with Windows
or OSX forks, you know? :)
Post by Glenn Fowler
(or can you -- does stat(2) or some other new syscall?)
you get a hint when the /bin/pwd algorithm breaks, but it can break for other
reasons that may give similar behavior
but I bet the /bin/bash and /bin/pwd authors thought they knew what they were doing
If think the problem is that getcwd() should return a pointer to
/proc/22214/cwd and /bin/pwd should do the same in case they are in a
O_XATTR dir.
All shells, which are a system interface, surely needs support -@, or
are boned in case they hit a directory which isn't connected to /. NFS
can put you into the same trouble if you unlink a directory (for
NFSv2/v3) or have it offlined under your feet (Solaris QFS, SAMFS and
NFSv4 can do that to you).
Post by Glenn Fowler
I belive most of these oddities, and even the need to patch /bin/sh and /bin/bash and ksh,
could have been avoided if xattrs simply stole *something* from the path namespace
even a (pseudo) mount point or /dev entry would work
/dev/xattr/./tmp # . relative path
/dev/xattr/home/gsf/tst/ksh/tmp # absolute path
What will happen if O_XATTR directories allow sub directories?
Post by Glenn Fowler
this might be able to be emulated if there were an is_this_an_xattr_dirat(AT_FDCWD, ".") call
is it possible to determine if "." is an extended attr dir?
maybe ls gives a hint?
$ /bin/ls -l .
total 2
-r--r--r-- 1 root root 156 Aug 15 11:15 SUNWattr_ro
-rw-r--r-- 1 root root 472 Aug 15 11:15 SUNWattr_rw
$ /bin/ls -ld .
drwxrwxrwt 4 gsf gsf 4 Aug 15 11:15 .
$ /bin/ls - at ld .
drwxrwxrwt 4 gsf gsf 4 Aug 15 11:15 .
1. -l overrides -@
2. -@ will only report @ if there are xattrs in the xattr dir which do
not belong to the filesystem itself:
ksh -c 'cd -@ . ; touch x ; ls - at ad . ..'
drwxr-xr-x 4 ced ced 177 Aug 16 10:02 .
drwxr-xr-x@ 2 ced ced 117 Aug 14 23:38 .

Ced
--
Cedric Blancher <cedric.blancher at gmail.com>
Institute Pasteur
Glenn Fowler
2013-08-16 13:27:12 UTC
Permalink
thanks Ced for the details
I'll comment on them later
Post by Cedric Blancher
I don't think O_XATTR dirs are half-baked. They just don't have an
anchor which is connected to / directly. It's even worse with Windows
or OSX forks, you know? :)
the idea of the extended attribute dirs a a physical entity is great
the half-baked part is how the standard utilities deal with them (or not)
Glenn Fowler
2013-08-16 21:01:47 UTC
Permalink
Post by Cedric Blancher
Post by Glenn Fowler
on a system that has O_XATTR is there an errno for
open(valid_fd, O_XATTR|O_RDONLY)
failing because valid_fd has no extended attributes?
If a filesystem supports NFSv4 extended attributes then all elements
in this filesystem, including files, dirs, pipes, fifios etc support
them.
If a filesystem doesn't support O_XATTR it will return EINVAL, try it
/home/ced/bin/ksh: cd: /devices/pseudo: [Invalid argument]
thanks
Post by Cedric Blancher
Post by Glenn Fowler
the errno from above, even on systems that have no O_XATTR
You better return ENOTSUP then
right
Post by Cedric Blancher
Post by Glenn Fowler
I'm on a 'SunOS hostname 5.11 11.0 i86pc i386 i86pc' and
don't answer that, I'm not the solaris maintenance loop
also, and this goes to my complaints that there is no path notation
it chokes because it can't determine pwd (other shells, ksh too, have
problems with that too, more on that below)
What about returning the shp->pwdfd path, i.e.
/proc/14894/fd/12
not bad but ephemeral
Post by Cedric Blancher
Post by Glenn Fowler
some other weirdness from an interactive session,
again because there is no path notation
--
$ pwd
/home/gsf/tst/ksh
$ mkdir tmp
$ /bin/bash -c pwd
shell-init: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
pwd: error retrieving current directory: getcwd: cannot access parent directories: No such file or directory
$ /bin/sh -c pwd
pwd: cannot access parent directories [No such file or directory]
$ /bin/pwd
pwd: cannot determine current directory!
$ ksh -c pwd
pwd: cannot access parent directories [No such file or directory]
$ # ouch ksh bit by this too -- does olga's fgetcwd() handle this right?
$ pwd
/home/gsf/tst/ksh/tmp
$ ls
SUNWattr_ro SUNWattr_rw
/home/gsf/arch/sol11.i386/bin/ksh: cd: SUNWattr_ro: [Not a directory]
$ # is that the right errno?
$ cd -
/home/gsf/tst/ksh
$ pwd
/home/gsf/tst/ksh
$ cd -
/home/gsf/tst/ksh/tmp
$ pwd
/home/gsf/tst/ksh/tmp
$ ls
--
I don't think O_XATTR dirs are half-baked. They just don't have an
anchor which is connected to / directly. It's even worse with Windows
or OSX forks, you know? :)
mentioned in prev message, the api is half-baked because it involves re-coding
and that coding is not trivial
Post by Cedric Blancher
Post by Glenn Fowler
(or can you -- does stat(2) or some other new syscall?)
you get a hint when the /bin/pwd algorithm breaks, but it can break for other
reasons that may give similar behavior
but I bet the /bin/bash and /bin/pwd authors thought they knew what they were doing
If think the problem is that getcwd() should return a pointer to
/proc/22214/cwd and /bin/pwd should do the same in case they are in a
O_XATTR dir.
are boned in case they hit a directory which isn't connected to /. NFS
can put you into the same trouble if you unlink a directory (for
NFSv2/v3) or have it offlined under your feet (Solaris QFS, SAMFS and
NFSv4 can do that to you).
the issue is knowing that you are in an O_XATTR dir in the first place
just discovered that O_XATTR dir inode == 0 -- this is for a default
O_XATTR dir -- once a non-default entry is added it gets a real inode

in an O_XATTR dir ".." is a hard link to the parent file
strange to see ".." listed as a regular file, but it makes sense
but its not easy retrieving the path given the inode number of a regular file
Post by Cedric Blancher
Post by Glenn Fowler
I belive most of these oddities, and even the need to patch /bin/sh and /bin/bash and ksh,
could have been avoided if xattrs simply stole *something* from the path namespace
even a (pseudo) mount point or /dev entry would work
/dev/xattr/./tmp # . relative path
/dev/xattr/home/gsf/tst/ksh/tmp # absolute path
What will happen if O_XATTR directories allow sub directories?
good question -- any path notation would have to handle that
Post by Cedric Blancher
Post by Glenn Fowler
this might be able to be emulated if there were an is_this_an_xattr_dirat(AT_FDCWD, ".") call
is it possible to determine if "." is an extended attr dir?
maybe ls gives a hint?
$ /bin/ls -l .
total 2
-r--r--r-- 1 root root 156 Aug 15 11:15 SUNWattr_ro
-rw-r--r-- 1 root root 472 Aug 15 11:15 SUNWattr_rw
$ /bin/ls -ld .
drwxrwxrwt 4 gsf gsf 4 Aug 15 11:15 .
$ /bin/ls - at ld .
drwxrwxrwt 4 gsf gsf 4 Aug 15 11:15 .
drwxr-xr-x 4 ced ced 177 Aug 16 10:02 .
so the sole purpose of -@ is, in conjunction with -l, to display an @
after the permissions if a file has non-default extended attributes

thanks

Loading...