Discussion:
[ast-developers] /dev/file/flags@xattr@/dev/fd/$n no longer works in ast-ksh.2013-09-26 ...
Roland Mainz
2013-09-27 01:50:24 UTC
Permalink
Hi!

----

The following code from one of my test scripts no longer work in
ast-ksh.2013-09-26 on Solaris 11/AMD64/64bit:
-- snip --
$ ksh -c 'redirect {n}<. ; redirect {m}</dev/file/flags at xattr@/dev/fd/$n '
/home/test001/bin/ksh: /dev/file/flags at xattr@/dev/fd/11: cannot open
[Invalid argument]
-- snip --

This fails because of the following code tries to open the parent dir
of an XATTR dir via:
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --

... erm.. I don't understand this... what is this code trying to do ?

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Roland Mainz
2013-09-27 02:17:31 UTC
Permalink
Post by Roland Mainz
Hi!
----
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
This patch fixes the problem:
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@

/* preserve open() semantics if possible */

- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);

/* see if the filesystem groks
.../[<pid>]/<fd>/... paths */
-- snip --

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Roland Mainz
2013-09-27 02:40:06 UTC
Permalink
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
[snip]
Post by Roland Mainz
-- snip --
Attached (as "astksh20130926_devflagsxattr_open_fix001.diff.txt") is
the same fix in attachment form (to avoid the Gmail issues with
whitespaces getting "eaten") ...

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
-------------- next part --------------
diff -r -u original/src/lib/libast/path/pathopen.c build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@

/* preserve open() semantics if possible */

- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".", oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".", O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".", oflags|O_INTERCEPT, mode);

/* see if the filesystem groks .../[<pid>]/<fd>/... paths */
Glenn Fowler
2013-09-27 04:17:19 UTC
Permalink
Post by Roland Mainz
Post by Roland Mainz
Hi!
----
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
this patch is wrong -- don't apply it
see my previous post
Roland Mainz
2013-09-27 04:48:33 UTC
Permalink
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
this patch is wrong -- don't apply it
see my previous post
Why is it wrong ? The |fd = openat(f, "..", oflags|O_INTERCEPT,
mode);| in the original code could've never worked in any possible way
because ".." in an xattr directory is doomed to fail anyway...

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Roland Mainz
2013-09-27 05:24:31 UTC
Permalink
Post by Roland Mainz
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
this patch is wrong -- don't apply it
see my previous post
Why is it wrong ? The |fd = openat(f, "..", oflags|O_INTERCEPT,
mode);| in the original code could've never worked in any possible way
because ".." in an xattr directory is doomed to fail anyway...
Actually I'm wrong... it should work... but the performance
implications are horrible because if the filesystem doesn't have the
xattr _data_ yet it may create them from the "default set"... as
result we trigger a write operation on the first time of access (of
course this doesn't happen if the XATTR or SATTR data are already in
use.. for example on UFS SATTR data may be used to store the ACL data
etc.) ... |O_XATTR| wasn't designed with this usage in mind...

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Roland Mainz
2013-09-27 05:53:58 UTC
Permalink
Post by Roland Mainz
Post by Roland Mainz
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
this patch is wrong -- don't apply it
see my previous post
Why is it wrong ? The |fd = openat(f, "..", oflags|O_INTERCEPT,
mode);| in the original code could've never worked in any possible way
because ".." in an xattr directory is doomed to fail anyway...
Actually I'm wrong... it should work... but the performance
implications are horrible because if the filesystem doesn't have the
xattr _data_ yet it may create them from the "default set"... as
result we trigger a write operation on the first time of access (of
course this doesn't happen if the XATTR or SATTR data are already in
use.. for example on UFS SATTR data may be used to store the ACL data
etc.) ... |O_XATTR| wasn't designed with this usage in mind...
Looking back at the original failure:
$ ksh -c 'redirect {n}<. ; redirect
{m}</dev/file/flags at xattr@/dev/fd/$n ; cd -f $m ; true' # creates this
truss output:
-- snip --
[snip]
open(".", O_RDONLY) = 3
fstat(3, 0xFFFFFD7FFFDFE810) = 0
fcntl(3, F_SETFD, 0x00000001) = 0
fcntl(3, F_DUPFD, 0x0000000A) = 11
close(3) = 0
ioctl(11, TCGETS, 0xFFFFFD7FFFDFF0EC) Err#25 ENOTTY
lseek(11, 0, SEEK_CUR) = 0
fstat(11, 0xFFFFFD7FFFDFF160) = 0
fcntl(11, F_SETFD, 0x00000001) = 0
fcntl(11, F_GETFL) = 8192
openat(11, ".", O_RDONLY|O_XATTR) = 3
fstat(3, 0xFFFFFD7FFFDFE810) = 0
openat(3, "..", O_RDONLY|O_XATTR) Err#22 EINVAL
close(3) = 0
open("/usr/lib/locale/en_US.UTF-8/LC_MESSAGES/SUNW_OST_OSLIB.mo",
O_RDONLY) Err#2 ENOENT
./arch/sol11.i386-64/bin/ksh: /dev/file/flags at xattr@/dev/fd/11: cannot
open [Invalid argument]
[snip]
-- snip --

The |openat(3, "..", O_RDONLY|O_XATTR)| ruines the whole thing.

However if I filter out the |O_XATTR| flag the whole "io.sh" testsuite
triggers mass failures... and if I disable the whole codeblock within
|#if O_XATTR ... #endif| I can't enter the XATTR dir anymore.

... and the following code doesn't work because it always passes the
whole path to |openat()| instead of just the /dev/fd/-portion:
-- snip --
236 /* see if the filesystem groks
.../[<pid>]/<fd>/... paths */
237
238 if ((!(oflags & O_CREAT) || !access(b,
F_OK)) && (fd = openat(AT_FDCWD, b, oflags|O_INTERCEPT, mode)) >= 0)
239 return fd;
-- snip --

That should work... but... when I tried it with this patch...
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 07:41:55.847283054 +0200
@@ -224,18 +224,11 @@

if (oflags & (O_DIRECTORY|O_SEARCH))
return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
-#endif

/* see if the filesystem groks
.../[<pid>]/<fd>/... paths */

- if ((!(oflags & O_CREAT) || !access(b, F_OK))
&& (fd = openat(AT_FDCWD, b, oflags|O_INTERCEPT, mode)) >= 0)
+ const char *b_devfd = strstr(b, "/dev/fd/");
+ if ((!(oflags & O_CREAT) || !access(b, F_OK))
&& (fd = openat(AT_FDCWD, b_devfd, oflags|O_INTERCEPT, mode)) >= 0)
return fd;

/* stuck with dup semantics -- the best we can
do at this point */
-- snip --
... I found out that the resulting |open("/dev/fd/11",
O_RDONLY|O_XATTR)| will NOT give me a XATTR directory (which is AFAIK
a bug...).

Short: xx@@@!!!!<censored>@@@!!!!<censored>@@@!!!!<censored>!!xx!!

BTW: Using /proc/self/fd/ seems to work:
-- snip --
$ ./arch/sol11.i386\-64/bin/ksh -c 'redirect {n}<. ; redirect
{m}</dev/file/flags at xattr@/proc/self/fd/$n ; cd -f $m ; ls;true'
SUNWattr_ro SUNWattr_rw
-- snip --

So basically we're back the point where the whole code within |#if
O_XATTR| doesn't work... and shouldn't be used because the
implementations will perform horribly performance-wise.

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Roland Mainz
2013-09-27 06:12:22 UTC
Permalink
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
[snip]
Post by Roland Mainz
So basically we're back the point where the whole code within |#if
O_XATTR| doesn't work... and shouldn't be used because the
implementations will perform horribly performance-wise.
Erm... question:
What is wrong with the following patch to fix the original problem
that $ ksh -c 'redirect {n}<. ; redirect
{m}</dev/file/flags at xattr@/dev/fd/$n ; cd -f $m ; ls;true' # doesn't
work:
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 08:03:03.149006810 +0200
@@ -222,7 +222,7 @@

/* preserve open() semantics if possible */

- if (oflags & (O_DIRECTORY|O_SEARCH))
+ if (oflags & (O_DIRECTORY|O_SEARCH|O_XATTR))
return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
#if O_XATTR
if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
-- snip --

Looking at the output of "truss" it seems to do exactly what's
requested at shell code level...

----

Bye,
Roland
--
__ . . __
(o.\ \/ /.o) roland.mainz at nrubsig.org
\__\/\/__/ MPEG specialist, C&&JAVA&&Sun&&Unix programmer
/O /==\ O\ TEL +49 641 3992797
(;O/ \/ \O;)
Glenn Fowler
2013-09-27 08:24:07 UTC
Permalink
Post by Roland Mainz
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
[snip]
Post by Roland Mainz
So basically we're back the point where the whole code within |#if
O_XATTR| doesn't work... and shouldn't be used because the
implementations will perform horribly performance-wise.
What is wrong with the following patch to fix the original problem
that $ ksh -c 'redirect {n}<. ; redirect
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 08:03:03.149006810 +0200
@@ -222,7 +222,7 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
+ if (oflags & (O_DIRECTORY|O_SEARCH|O_XATTR))
return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
#if O_XATTR
if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
-- snip --
that looks good but will require this at the top of the file

#ifndef O_XATTR
#define O_XATTR 0
#endif
Post by Roland Mainz
Looking at the output of "truss" it seems to do exactly what's
requested at shell code level...
there's path corruption somewhere -- more on that later today
Cedric Blancher
2013-09-27 09:39:19 UTC
Permalink
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
[snip]
Post by Roland Mainz
So basically we're back the point where the whole code within |#if
O_XATTR| doesn't work... and shouldn't be used because the
implementations will perform horribly performance-wise.
What is wrong with the following patch to fix the original problem
that $ ksh -c 'redirect {n}<. ; redirect
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 08:03:03.149006810 +0200
@@ -222,7 +222,7 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
+ if (oflags & (O_DIRECTORY|O_SEARCH|O_XATTR))
return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
#if O_XATTR
if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
-- snip --
that looks good but will require this at the top of the file
#ifndef O_XATTR
#define O_XATTR 0
#endif
Post by Roland Mainz
Looking at the output of "truss" it seems to do exactly what's
requested at shell code level...
there's path corruption somewhere -- more on that later today
Both Glenn and Roland: Do me a favour and don't break cd -@, do you?
We depend on that feature on production systems, and if you break it I
get nailed by the board of directors.

Ced
--
Cedric Blancher <cedric.blancher at gmail.com>
Institute Pasteur
Glenn Fowler
2013-09-27 10:04:34 UTC
Permalink
Post by Cedric Blancher
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
[snip]
Post by Roland Mainz
So basically we're back the point where the whole code within |#if
O_XATTR| doesn't work... and shouldn't be used because the
implementations will perform horribly performance-wise.
What is wrong with the following patch to fix the original problem
that $ ksh -c 'redirect {n}<. ; redirect
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 08:03:03.149006810 +0200
@@ -222,7 +222,7 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
+ if (oflags & (O_DIRECTORY|O_SEARCH|O_XATTR))
return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
#if O_XATTR
if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
-- snip --
that looks good but will require this at the top of the file
#ifndef O_XATTR
#define O_XATTR 0
#endif
Post by Roland Mainz
Looking at the output of "truss" it seems to do exactly what's
requested at shell code level...
there's path corruption somewhere -- more on that later today
We depend on that feature on production systems, and if you break it I
get nailed by the board of directors.
the stream of alphas have been a journey from a collection of sometimes incoherent patches
(each patch may have been fine by itself but incoherent in the union)
to a readable maintainable implementation with #ifdefs buried as deep in the
ast libraries as possible

we're not breaking it on purpose
although going through the process of molding raw patches into good abstractions
I expect a few bugs to crop up -- but I also expect them to be locaized to the abstractions

we're debugging the implementation as the patches melt into the mainstream code
this latest back-and-forth showed what I was hoping for with O_XATTR:
when bugs show up with O_XATTR almost all of them will originate in and be fixed in
libast/path/pathcanon.c and libast/path/pathopen.c, which, along with the iffe file
libast/features/fcntl.c happen to be the only ast code that references O_XATTR

one of my fears of the patch model is that there's sometimes no time to think
if we really want to move ast in the right direction we need some breathing room to think
and that's why *my* pace of rolling in patches is slower than the list would sometimes like
if there's a nice solution in the big picture I rather wait until that gels
Glenn Fowler
2013-09-27 09:47:03 UTC
Permalink
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
[snip]
Post by Roland Mainz
So basically we're back the point where the whole code within |#if
O_XATTR| doesn't work... and shouldn't be used because the
implementations will perform horribly performance-wise.
What is wrong with the following patch to fix the original problem
that $ ksh -c 'redirect {n}<. ; redirect
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 08:03:03.149006810 +0200
@@ -222,7 +222,7 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
+ if (oflags & (O_DIRECTORY|O_SEARCH|O_XATTR))
return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
#if O_XATTR
if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
-- snip --
that looks good but will require this at the top of the file
#ifndef O_XATTR
#define O_XATTR 0
#endif
Post by Roland Mainz
Looking at the output of "truss" it seems to do exactly what's
requested at shell code level...
there's path corruption somewhere -- more on that later today
now I can go to bed ...
roland, here is a context diff you'll have to do manually because my line number are off
the corruption happened in fgetcwd()
I tried to use the buffer space on the right but I forgot fgetcwd() used it for scratch too

--- .../path/pathcanon.c Fri Sep 27 02:15:03 2013
+++ path/pathcanon.c Fri Sep 27 05:17:46 2013
@@ -560,8 +560,8 @@
{
z = x || (flags & PATH_PHYSICAL) && dev->fd >= 0 ? s : (char*)path;
n = strlen(z) + 1;
- a = (char*)path + size - n;
- memmove(a, z, n);
+ a = fmtbuf(n);
+ memcpy(a, z, n);
}
else
{
@@ -575,10 +575,7 @@
v += strlen(v);
if ((v - z) > 1)
*v++ = '/';
- if (inplace)
- memmove(v, a, n);
- else
- memcpy(v, a, n);
+ memcpy(v, a, n);
s = v = r;
}
else if (*s != '/')
--- .../path/pathopen.c Thu Aug 29 01:17:52 2013
+++ path/pathopen.c Fri Sep 27 05:17:46 2013
@@ -19,6 +19,10 @@
#include <netinet/in.h>
#endif

+#ifndef O_XATTR
+#define O_XATTR 0
+#endif
+
#if !_lib_getaddrinfo

#if !defined(htons) && !_lib_htons
@@ -201,7 +205,7 @@

/* preserve open() semantics if possible */

- if (oflags & (O_DIRECTORY|O_SEARCH))
+ if (oflags & (O_DIRECTORY|O_SEARCH|O_XATTR))
return openat(dev.fd, ".", oflags|O_INTERCEPT, mode);
#if O_XATTR
if ((f = openat(dev.fd, ".", O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)

now here are 3 modified tests that show the problem is with
cd -f <FD> vs $PWD : somehow $PWD doesn't track cd -f changes
both scripts take -@ as an arg to use cd -@
they should work with/without the -@ arg
they use ast ls to list inodes after cd to verift the cd worked

first this one that shows O_XATTR working ok
--
unset CDPATH

redirect {topfd}<"."
print -u2 test#1 top $topfd $PWD $(ls --format='%(ino)d' --directory)

mkdir -p sub
cd $1 sub
redirect {subfd}<"."
print -u2 test#2 sub $subfd $PWD $(ls --format='%(ino)d' --directory)

cd ~{topfd} ; print -u2 test#3 top $topfd $PWD $(ls --format='%(ino)d' --directory)

( cd ~{subfd} ; print -u2 test#4 sub $subfd $PWD $(ls --format='%(ino)d' --directory) )

print -u2 test#5 top $topfd $PWD $(ls --format='%(ino)d' --directory)

( cd $1 sub ; print -u2 test#6 sub $subfd $PWD $(ls --format='%(ino)d' --directory) )

print -u2 test#7 top $topfd $PWD $(ls --format='%(ino)d' --directory)
--

and this one shows that cd -f and $PWD get out of sync with the physical pwd
this happend betweem test#5 and test#6
running without -@ shows that it has nothing to do with O_XATTR
--
unset CDPATH

redirect {topfd}<"."
print -u2 test#1 top $topfd $PWD $(ls --format='%(ino)d' --directory)

mkdir -p sub
cd $1 sub
redirect {subfd}<"."
print -u2 test#2 sub $subfd $PWD $(ls --format='%(ino)d' --directory)

cd -f ${topfd} ; print -u2 test#3 top $topfd $PWD $(ls --format='%(ino)d' --directory)

( cd -f ${subfd} ; print -u2 test#4 sub $subfd $PWD $(ls --format='%(ino)d' --directory) )

print -u2 test#5 top $topfd $PWD $(ls --format='%(ino)d' --directory)

( cd $1 sub ; print -u2 test#6 sub $subfd $PWD $(ls --format='%(ino)d' --directory) )

print -u2 test#7 top $topfd $PWD $(ls --format='%(ino)d' --directory)
--

this last test shows that the out-of-sync problem is with cd -f and (...) subshells
--
unset CDPATH

redirect {topfd}<"."
print -u2 test#1 top $topfd $PWD $(ls --format='%(ino)d' --directory)

mkdir -p sub
cd $1 sub
redirect {subfd}<"."
print -u2 test#2 sub $subfd $PWD $(ls --format='%(ino)d' --directory)

cd -f ${topfd} ; print -u2 test#3 top $topfd $PWD $(ls --format='%(ino)d' --directory)

cd -f ${subfd} ; print -u2 test#4 sub $subfd $PWD $(ls --format='%(ino)d' --directory) ; cd ~-

print -u2 test#5 top $topfd $PWD $(ls --format='%(ino)d' --directory)

cd $1 sub ; print -u2 test#6 sub $subfd $PWD $(ls --format='%(ino)d' --directory) ; cd ~-

print -u2 test#7 top $topfd $PWD $(ls --format='%(ino)d' --directory)
--
Glenn Fowler
2013-09-27 07:55:52 UTC
Permalink
Post by Roland Mainz
Post by Roland Mainz
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
this patch is wrong -- don't apply it
see my previous post
Why is it wrong ? The |fd = openat(f, "..", oflags|O_INTERCEPT,
mode);| in the original code could've never worked in any possible way
because ".." in an xattr directory is doomed to fail anyway...
Actually I'm wrong... it should work... but the performance
implications are horrible because if the filesystem doesn't have the
xattr _data_ yet it may create them from the "default set"... as
result we trigger a write operation on the first time of access (of
course this doesn't happen if the XATTR or SATTR data are already in
use.. for example on UFS SATTR data may be used to store the ACL data
etc.) ... |O_XATTR| wasn't designed with this usage in mind...
it does work
xattr gave the rope and I used it
and its a beautiful application of an unintened consequence

so at this point its a performance hit to get open() semantics
or no performance hit with dup() semantics
you want to debug all the cases that will fail with dup() semantics?

and are you sure readonly access to a synthesized default xattr causes a write?
now *that* would be a performance hit
find / of a small program that did openat(open(file,0),O_XATTR)
would really fill up the nameless space with physical default xattrs?
and I could do that for files I could read but don't own?

if one were to set up an experiment, what program lists xattr usage,
du -@? df -@? fsck -@ ...?
Glenn Fowler
2013-09-27 07:45:13 UTC
Permalink
Post by Roland Mainz
Post by Glenn Fowler
Post by Roland Mainz
Post by Roland Mainz
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
-- snip --
diff -r -u original/src/lib/libast/path/pathopen.c
build_i386_64bit_debug/src/lib/libast/path/pathopen.c
--- src/lib/libast/path/pathopen.c 2013-08-29 07:17:52.000000000 +0200
+++ src/lib/libast/path/pathopen.c 2013-09-27 04:06:38.362136396 +0200
@@ -222,16 +222,12 @@
/* preserve open() semantics if possible */
- if (oflags & (O_DIRECTORY|O_SEARCH))
- return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
-#if O_XATTR
- if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
- {
- fd = openat(f, "..", oflags|O_INTERCEPT, mode);
- close(f);
- return fd;
- }
+ if (oflags & (O_DIRECTORY|O_SEARCH
+#ifdef O_XATTR
+ |O_XATTR
#endif
+ ))
+ return openat(dev.fd, ".",
oflags|O_INTERCEPT, mode);
this patch is wrong -- don't apply it
see my previous post
Why is it wrong ? The |fd = openat(f, "..", oflags|O_INTERCEPT,
mode);| in the original code could've never worked in any possible way
because ".." in an xattr directory is doomed to fail anyway...
.. in an xattr dir points to the original object
if it doesn't then why is there any .. entry at all
it make sense that the nameless xattr space must have a way to get to the named objects
and it makes sense that .. is the place to hang that reference
and ls confirms it

$ ls -ail /dev/file/xattr at test79.sh//@// test79.sh
27772 -rw-r--r-- 1 gsf gsf 148 Sep 27 02:16 test79.sh

/dev/file/xattr at test79.sh//@//:
total 4
0 drwxrwxrwt 4 gsf gsf 4 Sep 27 02:16 .
27772 -rw-r--r-- 1 gsf gsf 148 Sep 27 02:16 ..
1 -r--r--r-- 1 root root 156 Sep 27 02:16 SUNWattr_ro
2 -rw-r--r-- 1 root root 472 Sep 27 02:16 SUNWattr_rw
Glenn Fowler
2013-09-27 04:14:10 UTC
Permalink
Post by Roland Mainz
Hi!
----
The following code from one of my test scripts no longer work in
-- snip --
[Invalid argument]
-- snip --
This fails because of the following code tries to open the parent dir
-- snip --
227 #if O_XATTR
228 if ((f = openat(dev.fd, ".",
O_INTERCEPT|O_RDONLY|O_XATTR)) >= 0)
229 {
230 fd = openat(f, "..",
oflags|O_INTERCEPT, mode);
231 close(f);
232 return fd;
233 }
234 #endif
-- snip --
... erm.. I don't understand this... what is this code trying to do ?
well someone finally read the code
(which btw has nothing to do with your core dump tickled by a spelling error)

this was a cute discovery

recall the main problem of emulating /dev/fd/<FD> (or /proc/self/fd/<FD>)
is that when its done in the kernel

open("/dev/fd/<FD>",...)

gives you a new file table entry with an independent seek offset and fcntl flags
emulations have had to settle with dup(<FD>) and cross fingers that
the caller would not notice the shared seek offset and fcntl flags

O_XATTR lets us emulate open semantics just like the kernel;
for any file that supports O_XATTR this sequence

xfd = openat(fd, ".", O_XATTR);
ffd = openat(xfd, ...);
close(xfd);

gives an independent descriptor ffd to the same file as descriptor fd

for directories its easier

ffd = openat(fd, ".", ...);

I actually added a comment

/* preserve open() semantics if possible */

its just a bit of a homework assignment to figure out why it does
Loading...