[ast-developers] building ksh alpha on debian testing

[Giovanni Rapagnani]

Hello,

I have been trying to build on Debian testing the various ksh alpha
version released during the last 2 months and unfortunately I could not
achieve to successfully compile any of them.

I would like to help but my coding skills are too limited to myself find
out what is wrong. Actually I could not even find the compilation
starting point inside the bin/package script :-(

This is how I tried to compile ast-ksh.2013-09-26:
<verify md5 sums>
mkdir build
tar -xf INIT.2013-09-26.tgz -C build
mkdir -p build/lib/package/tgz
cp ast-ksh.2013-09-26.tgz build/lib/package/tgz/
cd build && bin/package read && bin/package make

In attachment the output of: bin/package results

Currently popcon [1] reports that there are 4500 Debian systems on which
ksh is installed. It would be a pitty that those systems do not benefit
from the next stable version of ksh.

I'll be glad to help to find out what is wrong, so if I can do anything
to help, please tell me.

Best regards.

[1] Debian package popularity contest for ksh:
http://qa.debian.org/popcon.php?package=ksh
--
Giovanni Rapagnani

-------------- next part --------------
A non-text attachment was scrubbed...
Name: results.log
Type: text/x-log
Size: 79806 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20130929/4d052fe4/attachment-0001.bin>

[Cedric Blancher]

Post by Giovanni Rapagnani
Hello,
I have been trying to build on Debian testing the various ksh alpha
version released during the last 2 months and unfortunately I could not
achieve to successfully compile any of them.
I would like to help but my coding skills are too limited to myself find
out what is wrong. Actually I could not even find the compilation
starting point inside the bin/package script :-(
<verify md5 sums>
mkdir build
tar -xf INIT.2013-09-26.tgz -C build
mkdir -p build/lib/package/tgz
cp ast-ksh.2013-09-26.tgz build/lib/package/tgz/
cd build && bin/package read && bin/package make
In attachment the output of: bin/package results
Currently popcon [1] reports that there are 4500 Debian systems on which
ksh is installed. It would be a pitty that those systems do not benefit
from the next stable version of ksh.
I'll be glad to help to find out what is wrong, so if I can do anything
to help, please tell me.

If I look at your log I see that essential networking includes do not work:
In file included from /usr/include/netdb.h:27:0,
from
/home/gio/test/sources/build/src/lib/libast/path/pathopen.c:37:
/usr/include/netinet/in.h:226:5: error: expected
specifier-qualifier-list before '__SOCKADDR_COMMON'
/usr/include/netinet/in.h:240:5: error: expected
specifier-qualifier-list before '__SOCKADDR_COMMON'
/usr/include/netinet/in.h:292:29: error: field 'gr_group' has incomplete type
/usr/include/netinet/in.h:301:29: error: field 'gsr_group' has incomplete type
/usr/include/netinet/in.h:304:29: error: field 'gsr_source' has incomplete type
/usr/include/netinet/in.h:336:29: error: field 'gf_group' has incomplete type
/usr/include/netinet/in.h:344:29: error: array type has incomplete element type
In file included from
/home/gio/test/sources/build/src/lib/libast/path/pathopen.c:37:0:
/usr/include/netdb.h:467:7: error: unknown type name 'sa_family_t'
/usr/include/netdb.h:495:8: error: unknown type name 'sa_family_t'
/usr/include/netdb.h:517:10: error: unknown type name 'sa_family_t'
/usr/include/netdb.h:541:4: error: unknown type name 'sa_family_t'
/usr/include/netdb.h:560:41: error: unknown type name 'sa_family_t'
/home/gio/test/sources/build/src/lib/libast/path/pathopen.c: In
function 'pathopen':
/home/gio/test/sources/build/src/lib/libast/path/pathopen.c:261:21:
error: 'PF_UNSPEC' undeclared (first use in this function)
/home/gio/test/sources/build/src/lib/libast/path/pathopen.c:261:21:
note: each undeclared identifier is reported only once for each
function it appears in
/home/gio/test/sources/build/src/lib/libast/path/pathopen.c:272:24:
error: 'SOCK_STREAM' undeclared (first use in this function)
/home/gio/test/sources/build/src/lib/libast/path/pathopen.c:290:24:
error: 'SOCK_DGRAM' undeclared (first use in this function)
mamake [lib/libast]: *** exit code 1 making pathopen.o
/home/gio/test/sources/build/src/lib/libast/misc/procopen.c: In
function 'procopen':
/home/gio/test/sources/build/src/lib/libast/misc/procopen.c:492:19:
error: 'AF_UNIX' undeclared (first use in this function)
/home/gio/test/sources/build/src/lib/libast/misc/procopen.c:492:19:
note: each undeclared identifier is reported only once for each
function it appears in
/home/gio/test/sources/build/src/lib/libast/misc/procopen.c:492:28:
error: 'SOCK_STREAM' undeclared (first use in this function)
mamake [lib/libast]: *** exit code 1 making procopen.o

Are you sure you have all devel packages installed? SOCK_STREAM should
be defined.

BTW: Which Debian version do you use?

Ced

[Giovanni Rapagnani]

Post by Cedric Blancher
Are you sure you have all devel packages installed? SOCK_STREAM should
be defined.

If I search under /usr/include for files which defines some of the
undeclared variables (for instance SOCK_STREAM, SOCK_DGRAM or AF_UNIX),
I get:

$ find -name "*.h"|xargs grep -E "(SOCK_STREAM|SOCK_DGRAM|AF_UNIX)"
...
./x86_64-linux-gnu/bits/socket_type.h: SOCK_STREAM = 1, /* Sequenced,
reliable, connection-based
./x86_64-linux-gnu/bits/socket_type.h:#define SOCK_STREAM SOCK_STREAM
./x86_64-linux-gnu/bits/socket_type.h: SOCK_DGRAM = 2, /*
Connectionless, unreliable datagrams
./x86_64-linux-gnu/bits/socket_type.h:#define SOCK_DGRAM SOCK_DGRAM
./x86_64-linux-gnu/bits/socket.h:#define AF_UNIX PF_UNIX
..

Moreover, the current stable release (12-08-01 --- Release ksh93u+
---), which also uses the 3 variables above, does compile without problem:

gio at testing:~/test/ksh-93u+20120801/arch/linux.i386-64/bin$ ./ksh
$ echo $KSH_VERSION
Version AJM 93u+ 2012-08-01

Post by Cedric Blancher
BTW: Which Debian version do you use?

I am trying to compile it on the testing distribution of Debian.

Thank you to help to figure out what is wrong.

[ольга крыжановская]

Giovanni, can you download
http://svn.nrubsig.org/svn/people/gisburn/scripts/buildksh93.sh and
build ksh93 like this:

gunzip -c <../INIT.2013-09-26.tgz | tar -xf -
gunzip -c <../ast-ksh.2013-09-26.tgz | tar -xf -
time nice ksh ../buildksh93.sh build.debug.linux.i386.64bit.gcc.c1x
2>&1 | tee -a buildlog.log

That should work, and enable some important-but-usually-disabled
builtins, similar to busybox.

Olga

Post by Giovanni Rapagnani

Post by Cedric Blancher
Are you sure you have all devel packages installed? SOCK_STREAM should
be defined.

If I search under /usr/include for files which defines some of the
undeclared variables (for instance SOCK_STREAM, SOCK_DGRAM or AF_UNIX),
$ find -name "*.h"|xargs grep -E "(SOCK_STREAM|SOCK_DGRAM|AF_UNIX)"
...
./x86_64-linux-gnu/bits/socket_type.h: SOCK_STREAM = 1, /* Sequenced,
reliable, connection-based
./x86_64-linux-gnu/bits/socket_type.h:#define SOCK_STREAM SOCK_STREAM
./x86_64-linux-gnu/bits/socket_type.h: SOCK_DGRAM = 2, /*
Connectionless, unreliable datagrams
./x86_64-linux-gnu/bits/socket_type.h:#define SOCK_DGRAM SOCK_DGRAM
./x86_64-linux-gnu/bits/socket.h:#define AF_UNIX PF_UNIX
..
Moreover, the current stable release (12-08-01 --- Release ksh93u+
gio at testing:~/test/ksh-93u+20120801/arch/linux.i386-64/bin$ ./ksh
$ echo $KSH_VERSION
Version AJM 93u+ 2012-08-01

Post by Cedric Blancher
BTW: Which Debian version do you use?

I am trying to compile it on the testing distribution of Debian.
Thank you to help to figure out what is wrong.
--
Giovanni Rapagnani
_______________________________________________
ast-developers mailing list
ast-developers at lists.research.att.com
http://lists.research.att.com/mailman/listinfo/ast-developers

[Giovanni Rapagnani]

Post by ольга крыжановская
Giovanni, can you download
http://svn.nrubsig.org/svn/people/gisburn/scripts/buildksh93.sh and
gunzip -c <../INIT.2013-09-26.tgz | tar -xf -
gunzip -c <../ast-ksh.2013-09-26.tgz | tar -xf -
time nice ksh ../buildksh93.sh build.debug.linux.i386.64bit.gcc.c1x
2>&1 | tee -a buildlog.log
That should work, and enable some important-but-usually-disabled
builtins, similar to busybox.

Olga, unfortunately it did not work. I still get several errors.
buildlog.log in attachment.

[ольга крыжановская]

OK. This is the same kind of error, which you had with your first
attempt. Now the question is, where does this come from?

Which kind of Debian version do you use? What does uname -a print?
What does gcc --version print?

Olga

Post by Giovanni Rapagnani

Post by ольга крыжановская
Giovanni, can you download
http://svn.nrubsig.org/svn/people/gisburn/scripts/buildksh93.sh and
gunzip -c <../INIT.2013-09-26.tgz | tar -xf -
gunzip -c <../ast-ksh.2013-09-26.tgz | tar -xf -
time nice ksh ../buildksh93.sh build.debug.linux.i386.64bit.gcc.c1x
2>&1 | tee -a buildlog.log
That should work, and enable some important-but-usually-disabled
builtins, similar to busybox.

Olga, unfortunately it did not work. I still get several errors.
buildlog.log in attachment.
--
Giovanni Rapagnani

[Giovanni Rapagnani]

Here the info:
$ lsb_release -a
Distributor ID: Debian
Description: Debian GNU/Linux testing (jessie)
Release: testing
Codename: jessie
$ uname -a
Linux testing 3.10-3-amd64 #1 SMP Debian 3.10.11-1 (2013-09-10) x86_64
GNU/Linux
$ gcc --version
gcc (Debian 4.8.1-10) 4.8.1

Post by ольга крыжановская
OK. This is the same kind of error, which you had with your first
attempt. Now the question is, where does this come from?
Which kind of Debian version do you use? What does uname -a print?
What does gcc --version print?
Olga

Post by Giovanni Rapagnani

Post by ольга крыжановская
Giovanni, can you download
http://svn.nrubsig.org/svn/people/gisburn/scripts/buildksh93.sh and
gunzip -c <../INIT.2013-09-26.tgz | tar -xf -
gunzip -c <../ast-ksh.2013-09-26.tgz | tar -xf -
time nice ksh ../buildksh93.sh build.debug.linux.i386.64bit.gcc.c1x
2>&1 | tee -a buildlog.log
That should work, and enable some important-but-usually-disabled
builtins, similar to busybox.

Olga, unfortunately it did not work. I still get several errors.
buildlog.log in attachment.
--
Giovanni Rapagnani

[Giovanni Rapagnani]

Hello,
Actually, the socket.h file is found under
/usr/include/x86_64-linux-gnu/sys instead of /usr/include/sys . I then
created the symbolic link /usr/include/sys ->
/usr/include/x86_64-linux-gnu/sys and the compilation succeeded.

I think /usr/include/x86_64-linux-gnu is related to Debian multiarch
support.

Now the question is: How to compile ksh without creating the symbolic
link above? Should it be the responsibility of ksh sources to find the
right header directory? Or should an option be passed to bin/package make?

Best regards.

[Irek Szczesniak]

Post by Giovanni Rapagnani
Hello,
Actually, the socket.h file is found under
/usr/include/x86_64-linux-gnu/sys instead of /usr/include/sys . I then
created the symbolic link /usr/include/sys ->
/usr/include/x86_64-linux-gnu/sys and the compilation succeeded.
I think /usr/include/x86_64-linux-gnu is related to Debian multiarch
support.
Now the question is: How to compile ksh without creating the symbolic
link above? Should it be the responsibility of ksh sources to find the
right header directory? Or should an option be passed to bin/package make?

Does bash or any other software which does TCP compile in such a
situation? I have the distinct feeling that your build machine, or at
least the layout of your include files, is broken.

Does any *other* Debian machine behave that way?

Irek

[Giovanni Rapagnani]

Post by Irek Szczesniak
Does bash or any other software which does TCP compile in such a
situation? I have the distinct feeling that your build machine, or at
least the layout of your include files, is broken.

Yes, I tested bash compilation and it succeeded to compile. I started
the created bash command and run 'cat </dev/tcp/time.nist.gov/13' which
gave me:
56567 13-10-02 18:13:07 33 0 0 799.1 UTC(NIST) *

I also did a debootstrap [1] of a clean debian testing [2] system and
the same layout was found. ksh building would not succeed unless
creating the symlink sys -> x86_64-linux-gnu/sys.

Post by Irek Szczesniak
Does any *other* Debian machine behave that way?

Yes, I tried to compile ksh on Debian 7.0 [3] where there is the same
layout of include files, and the compilation failed. After creating the
symlink /usr/include/sys -> /usr/include/x86_64-linux-gnu/sys , the
compilation succeeded.

For instance, on a third debian system I have, on ARM architecture, the
sys/socket.h file is found at
/usr/include/arm-linux-gnueabihf/sys/socket.h .

Giovanni

[1] debootsrap is a command that retrieves a clean debian system from
the packages repository. The system is installed inside a directory
where you can do a chroot to get a clean debian system to work with.
[2] Debian testing, which is the system I am working with, is the
development version of Debian and will become at some point the new
stable version.
[3] Debian 7.0 is the current stable version of Debian.

[Cedric Blancher]

Post by Giovanni Rapagnani

Post by Irek Szczesniak
Does bash or any other software which does TCP compile in such a
situation? I have the distinct feeling that your build machine, or at
least the layout of your include files, is broken.

Yes, I tested bash compilation and it succeeded to compile. I started
the created bash command and run 'cat </dev/tcp/time.nist.gov/13' which
56567 13-10-02 18:13:07 33 0 0 799.1 UTC(NIST) *
I also did a debootstrap [1] of a clean debian testing [2] system and
the same layout was found. ksh building would not succeed unless
creating the symlink sys -> x86_64-linux-gnu/sys.

Post by Irek Szczesniak
Does any *other* Debian machine behave that way?

Yes, I tried to compile ksh on Debian 7.0 [3] where there is the same
layout of include files, and the compilation failed. After creating the
symlink /usr/include/sys -> /usr/include/x86_64-linux-gnu/sys , the
compilation succeeded.
For instance, on a third debian system I have, on ARM architecture, the
sys/socket.h file is found at
/usr/include/arm-linux-gnueabihf/sys/socket.h .
Giovanni
[1] debootsrap is a command that retrieves a clean debian system from
the packages repository. The system is installed inside a directory
where you can do a chroot to get a clean debian system to work with.
[2] Debian testing, which is the system I am working with, is the
development version of Debian and will become at some point the new
stable version.
[3] Debian 7.0 is the current stable version of Debian.

This is weird. I haven't see any other Linux system (Suse, Ubuntu,
Fedora) behave that way. System includes are supposed to be in
/usr/include and not some weird location only findable using grep ...
/

Guessing: Do you want to create a 32bit binary on a 64bit system? Does
clang (the compiler) have the same trouble?

Ced

[Giovanni Rapagnani]

Post by Cedric Blancher
This is weird. I haven't see any other Linux system (Suse, Ubuntu,
Fedora) behave that way. System includes are supposed to be in
/usr/include and not some weird location only findable using grep ...
/

I checked against an Ubuntu 12.04, the /usr/include/sys directory does
not exist. socket.h is also found inside
/usr/include/x86_64-linux-gnu/sys. The compilation fails unless creating
the symlink.

Actually, it seems this kind of setup has at least been around for 2
years. Have a look at this mail which was about implementing multiarch
support in gcc:
http://gcc.gnu.org/ml/gcc-patches/2011-08/msg01649.html

Post by Cedric Blancher
Guessing: Do you want to create a 32bit binary on a 64bit system? Does

No, I want to create a 64 bit executable on a 64 bit system.

Post by Cedric Blancher
clang (the compiler) have the same trouble?

After all those tests on different O/S and version of the same O/S, I
don't think it is a compiler problem. But if it really can be useful to
solve the issue I can try to compile with clang.

I still believe that either I need to pass an argument to 'bin/package
make' to enable searching headers inside /usr/include/x86_64-linux-gnu,
either ksh sources are missing something to be compiled under current
version of Debian, Ubuntu and probably all derivatives.

I may try to post a message on debian-devel ml and see what they think
about this.

Thanks.

[Cedric Blancher]

Post by Giovanni Rapagnani

Post by Cedric Blancher
This is weird. I haven't see any other Linux system (Suse, Ubuntu,
Fedora) behave that way. System includes are supposed to be in
/usr/include and not some weird location only findable using grep ...
/

I checked against an Ubuntu 12.04, the /usr/include/sys directory does
not exist. socket.h is also found inside
/usr/include/x86_64-linux-gnu/sys. The compilation fails unless creating
the symlink.
Actually, it seems this kind of setup has at least been around for 2
years. Have a look at this mail which was about implementing multiarch
http://gcc.gnu.org/ml/gcc-patches/2011-08/msg01649.html

Post by Cedric Blancher
Guessing: Do you want to create a 32bit binary on a 64bit system? Does

No, I want to create a 64 bit executable on a 64 bit system.

Post by Cedric Blancher
clang (the compiler) have the same trouble?

After all those tests on different O/S and version of the same O/S, I
don't think it is a compiler problem. But if it really can be useful to
solve the issue I can try to compile with clang.
I still believe that either I need to pass an argument to 'bin/package
make' to enable searching headers inside /usr/include/x86_64-linux-gnu,

Is there anything in /usr/include (not subdirs) which selects
/usr/include/x86_64-linux-gnu if certain flags are set? It strikes me
as dubious that no other Linux distribution or platform has this
issue.

Post by Giovanni Rapagnani
either ksh sources are missing something to be compiled under current
version of Debian, Ubuntu and probably all derivatives.

The ast-ksh and ast-open 2013-09-26 packages work just fine on Suse
9.2, 10.0, 12.3; Fedora, Centos; Solaris 9, Solaris 10, Solaris 11 and
Openindiana/Illumos. Something is amiss specific to Debian I think

Post by Giovanni Rapagnani
I may try to post a message on debian-devel ml and see what they think
about this.

Please post the archive url of the posting so we can follow the discussion.

Ced

[Giovanni Rapagnani]

Post by Cedric Blancher

Post by Giovanni Rapagnani
I may try to post a message on debian-devel ml and see what they think
about this.

Please post the archive url of the posting so we can follow the discussion.

I submitted today my question to debian-devel ml. Since 2 october I have
been trying to make iffe/mamake find the header files in the correct
location but without success.

The archive url is here:
http://lists.debian.org/debian-devel/2013/10/thrd2.html

look for the thread with subject:

AST ksh alpha: compilation failure related to multiarch

Kind regards.

[ольга крыжановская]

Giovanni, I can reproduce your problem, with gcc 4.8 and clang as the
compilers. As more than one compiler is affected, I think this is a
problem in the system includes, in Debian specifically, as no other
Linux has this trouble.

Do you know where the string 'x86_64-linux-gnu' comes from? Who
declares it? Is it created? What values do other platforms use?

The only reference I could find is this one:
$ grep -r x86_64-linux-gnu /usr/include/
/usr/include/llvm-3.2/llvm/Config/config.h:#define C_INCLUDE_DIRS
"/usr/include/x86_64-linux-gnu:/usr/include/x86_64-linux-gnu:/usr/include"
/usr/include/llvm-3.2/llvm/Config/config.h:#define LTDL_SYSSEARCHPATH
"/lib:/usr/lib:/usr/local/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu"

Olga

Post by Giovanni Rapagnani

Post by Cedric Blancher

Post by Giovanni Rapagnani
I may try to post a message on debian-devel ml and see what they think
about this.

Please post the archive url of the posting so we can follow the discussion.

I submitted today my question to debian-devel ml. Since 2 october I have
been trying to make iffe/mamake find the header files in the correct
location but without success.
http://lists.debian.org/debian-devel/2013/10/thrd2.html
AST ksh alpha: compilation failure related to multiarch
Kind regards.
--
Giovanni Rapagnani
Email secured by Check Point
_______________________________________________
ast-developers mailing list
ast-developers at lists.research.att.com
http://lists.research.att.com/mailman/listinfo/ast-developers

[Irek Szczesniak]

On Mon, Oct 28, 2013 at 12:55 AM, ????? ????????????

Post by ольга крыжановская
Giovanni, I can reproduce your problem, with gcc 4.8 and clang as the
compilers. As more than one compiler is affected, I think this is a
problem in the system includes, in Debian specifically, as no other
Linux has this trouble.
Do you know where the string 'x86_64-linux-gnu' comes from? Who
declares it? Is it created? What values do other platforms use?
$ grep -r x86_64-linux-gnu /usr/include/
/usr/include/llvm-3.2/llvm/Config/config.h:#define C_INCLUDE_DIRS
"/usr/include/x86_64-linux-gnu:/usr/include/x86_64-linux-gnu:/usr/include"
/usr/include/llvm-3.2/llvm/Config/config.h:#define LTDL_SYSSEARCHPATH
"/lib:/usr/lib:/usr/local/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu"

Well, this is a bug in debian. ast-ksh uses plain /usr/bin/cc and
Roland Mainz's buildksh93.sh uses either gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1, which per POSIX should give a
working compilation environment. Debian, and not ast-ksh, breaks this.

Giovanni, if you would kindly file a bug against Debian's system
includes which clearly states that specifying gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 does not result in a working
compilation environment where the X/OPEN and GNU extensions are
available.

This ball is now clearly in Debian's court. And it stinks like rotten flesh.

Irek

Email secured by Check Point

[Irek Szczesniak]

Post by Irek Szczesniak
On Mon, Oct 28, 2013 at 12:55 AM, ????? ????????????

Post by ольга крыжановская
Giovanni, I can reproduce your problem, with gcc 4.8 and clang as the
compilers. As more than one compiler is affected, I think this is a
problem in the system includes, in Debian specifically, as no other
Linux has this trouble.
Do you know where the string 'x86_64-linux-gnu' comes from? Who
declares it? Is it created? What values do other platforms use?
$ grep -r x86_64-linux-gnu /usr/include/
/usr/include/llvm-3.2/llvm/Config/config.h:#define C_INCLUDE_DIRS
"/usr/include/x86_64-linux-gnu:/usr/include/x86_64-linux-gnu:/usr/include"
/usr/include/llvm-3.2/llvm/Config/config.h:#define LTDL_SYSSEARCHPATH
"/lib:/usr/lib:/usr/local/lib:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu"

Well, this is a bug in debian. ast-ksh uses plain /usr/bin/cc and
Roland Mainz's buildksh93.sh uses either gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1, which per POSIX should give a
working compilation environment. Debian, and not ast-ksh, breaks this.
Giovanni, if you would kindly file a bug against Debian's system
includes which clearly states that specifying gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 does not result in a working
compilation environment where the X/OPEN and GNU extensions are
available.
This ball is now clearly in Debian's court. And it stinks like rotten flesh.

... or I should slap myself. On a coworkers machine the installation
of the 'libc6-dev-i386' package fixed the problem.

Now the question for Oliver and Olga is to figure out what cross
platform superclass package 'libc6-dev-i386' translates to so make
that a build dependency on all platforms.

Irek

Email secured by Check Point

[Giovanni Rapagnani]

Post by Irek Szczesniak

Post by Irek Szczesniak
Well, this is a bug in debian. ast-ksh uses plain /usr/bin/cc and
Roland Mainz's buildksh93.sh uses either gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1, which per POSIX should give a
working compilation environment. Debian, and not ast-ksh, breaks this.
Giovanni, if you would kindly file a bug against Debian's system
includes which clearly states that specifying gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 does not result in a working
compilation environment where the X/OPEN and GNU extensions are
available.
This ball is now clearly in Debian's court. And it stinks like rotten flesh.

I do not agree with this conclusion for several reasons (stated below)

Let me just remind that at the very beginning of the issue, Glenn told
me that it was probably an issue with the impossibility to find the
socket.h file. This is why I created the symlink (/usr/include/sys ->
/usr/include/x86_64-linux-gnu/sys) which allowed me to successfully
build ksh. I can now tell that the socket.h is the only file causing the
problem because when I create the directory /usr/include/sys and I only
put the file socket.h in it, the build is successful. (file copied from
/usr/include/x86_64-linux-gnu/sys).


Reason 1:
If I run 'echo | gcc -v -x c -E -', I can see in the output:

#include <...> search starts here:
/usr/lib/gcc/x86_64-linux-gnu/4.7/include
/usr/local/include
/usr/lib/gcc/x86_64-linux-gnu/4.7/include-fixed
/usr/include/x86_64-linux-gnu
/usr/include

which means that gcc has the ability to find the sys/socket.h in the
right place (i.e: /usr/include/x86_64-linux-gnu). Note that if I run the
same command on a ARM installation, I get:

#include <...> search starts here:
/usr/lib/gcc/arm-linux-gnueabihf/4.6/include
/usr/local/include
/usr/lib/gcc/arm-linux-gnueabihf/4.6/include-fixed
/usr/include/arm-linux-gnueabihf
/usr/include

which is also fine.


Reason 2:
If I compare a failed build log and a successful one, the first
meaningfull difference I encounter is:

build_ok.log:
iffe: test: is sys/socket an include path for the native header ... yes

build_fail.log:
iffe: test: is sys/socket an include path for the native header ... no

If I understand well, iffe is run before gcc.


Reason 3:
Moreover, when I do a comparison between all the files of a successful
build and the files of a failed one, I notice a difference in the file
arch/linux.i386-64/include/ast/ast_socket.h. The 2 lines below are found
in the ast_socket.h from the successful build and missing in the failed one:

#define _nxt_sys_socket <../include/sys/socket.h> /* include path for
the native <sys/socket.h> */
#define _nxt_sys_socket_str "../include/sys/socket.h" /* include
string for the native <sys/socket.h> */

Here also, this header file is created before gcc is invoked. Isn't it?


So I have the feeling that the culprit is one of the tools that is
invoked before gcc, probably the tool that creates the
arch/linux.i386-64/include/ast/ast_socket.h file. This tool probably has
no knowledge of the multiarch implementation in Debian.


In attachment you will find the following files:
* build_ok.log.gz: the build log recorded during a successful build
(with the symlink to /usr/include/x86_64-linux-gnu/sys)
* build_fail.log.gz: the build log recorded during a failed build
(without the symlink)
* build_file_diff.log.gz: the difference between each file of a
successful build and a failed one.

Post by Irek Szczesniak
... or I should slap myself. On a coworkers machine the installation
of the 'libc6-dev-i386' package fixed the problem.

libc6-dev-i386 is a package meant only to x86_64 system and which
install 32 bits development files (lib and header). This package
installs, among other, the file /usr/include/sys/socket.h. Of course
installing this package on my system is not what I want to do because I
am trying to build a 64 bit version of ksh on my 64 bit system. Thus
there is no point in installing 32 bit version of the header file.

Post by Irek Szczesniak
Now the question for Oliver and Olga is to figure out what cross
platform superclass package 'libc6-dev-i386' translates to so make
that a build dependency on all platforms.

If someone could have a look at the log files and clearly tells if the
issue is in debian or in ksh build chain, that would be great.

Kind regards.
--
Giovanni Rapagnani
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_fail.log.gz
Type: application/x-gzip
Size: 36187 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131029/55f54a4c/attachment-0003.gz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_file_diff.log.gz
Type: application/x-gzip
Size: 18254 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131029/55f54a4c/attachment-0004.gz>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build_ok.log.gz
Type: application/x-gzip
Size: 35157 bytes
Desc: not available
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131029/55f54a4c/attachment-0005.gz>

[Glenn Fowler]

what is the definition of _nxt_sys_socket in
$INSTALLROOT/src/lib/libast/ast_socket.h

Post by Irek Szczesniak

Post by Irek Szczesniak

Post by Irek Szczesniak
Well, this is a bug in debian. ast-ksh uses plain /usr/bin/cc and
Roland Mainz's buildksh93.sh uses either gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1, which per POSIX should give a
working compilation environment. Debian, and not ast-ksh, breaks this.
Giovanni, if you would kindly file a bug against Debian's system
includes which clearly states that specifying gcc -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 or clang -std=gnu99
-D_XOPEN_SOURCE=600 -D__EXTENSIONS__=1 does not result in a working
compilation environment where the X/OPEN and GNU extensions are
available.
This ball is now clearly in Debian's court. And it stinks like rotten

flesh.
I do not agree with this conclusion for several reasons (stated below)
Let me just remind that at the very beginning of the issue, Glenn told
me that it was probably an issue with the impossibility to find the
socket.h file. This is why I created the symlink (/usr/include/sys ->
/usr/include/x86_64-linux-gnu/sys) which allowed me to successfully
build ksh. I can now tell that the socket.h is the only file causing the
problem because when I create the directory /usr/include/sys and I only
put the file socket.h in it, the build is successful. (file copied from
/usr/include/x86_64-linux-gnu/sys).
/usr/lib/gcc/x86_64-linux-gnu/4.7/include
/usr/local/include
/usr/lib/gcc/x86_64-linux-gnu/4.7/include-fixed
/usr/include/x86_64-linux-gnu
/usr/include
which means that gcc has the ability to find the sys/socket.h in the
right place (i.e: /usr/include/x86_64-linux-gnu). Note that if I run the
/usr/lib/gcc/arm-linux-gnueabihf/4.6/include
/usr/local/include
/usr/lib/gcc/arm-linux-gnueabihf/4.6/include-fixed
/usr/include/arm-linux-gnueabihf
/usr/include
which is also fine.
If I compare a failed build log and a successful one, the first
iffe: test: is sys/socket an include path for the native header ... yes
iffe: test: is sys/socket an include path for the native header ... no
If I understand well, iffe is run before gcc.
Moreover, when I do a comparison between all the files of a successful
build and the files of a failed one, I notice a difference in the file
arch/linux.i386-64/include/ast/ast_socket.h. The 2 lines below are found
#define _nxt_sys_socket <../include/sys/socket.h> /* include path for
the native <sys/socket.h> */
#define _nxt_sys_socket_str "../include/sys/socket.h" /* include
string for the native <sys/socket.h> */
Here also, this header file is created before gcc is invoked. Isn't it?
So I have the feeling that the culprit is one of the tools that is
invoked before gcc, probably the tool that creates the
arch/linux.i386-64/include/ast/ast_socket.h file. This tool probably has
no knowledge of the multiarch implementation in Debian.
* build_ok.log.gz: the build log recorded during a successful build
(with the symlink to /usr/include/x86_64-linux-gnu/sys)
* build_fail.log.gz: the build log recorded during a failed build
(without the symlink)
* build_file_diff.log.gz: the difference between each file of a
successful build and a failed one.

Post by Irek Szczesniak
... or I should slap myself. On a coworkers machine the installation
of the 'libc6-dev-i386' package fixed the problem.

libc6-dev-i386 is a package meant only to x86_64 system and which
install 32 bits development files (lib and header). This package
installs, among other, the file /usr/include/sys/socket.h. Of course
installing this package on my system is not what I want to do because I
am trying to build a 64 bit version of ksh on my 64 bit system. Thus
there is no point in installing 32 bit version of the header file.

Post by Irek Szczesniak
Now the question for Oliver and Olga is to figure out what cross
platform superclass package 'libc6-dev-i386' translates to so make
that a build dependency on all platforms.

If someone could have a look at the log files and clearly tells if the
issue is in debian or in ksh build chain, that would be great.
Kind regards.
--
Giovanni Rapagnani
_______________________________________________
ast-developers mailing list
ast-developers at lists.research.att.com
http://lists.research.att.com/mailman/listinfo/ast-developers

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131030/3eee2787/attachment.html>

[Giovanni Rapagnani]

Post by Glenn Fowler
what is the definition of _nxt_sys_socket in
$INSTALLROOT/src/lib/libast/ast_socket.h

In the build that failed,_nxt_sys_socket is not defined. In the build
that succeeded, it is defined like this:

#define _nxt_sys_socket <../include/sys/socket.h> /* include path for
the native <sys/socket.h> */

[Glenn Fowler]

thanks
the problem is in "nxt" op in iffe which deals with header intercepts
among other things the intercepts enforce consistent EINTR behavior across
all intercepted calls

grab INIT.2013-10-30.tgz from gsf.cococlyde.org/download/

Post by Giovanni Rapagnani

Post by Glenn Fowler
what is the definition of _nxt_sys_socket in
$INSTALLROOT/src/lib/libast/ast_socket.h

In the build that failed,_nxt_sys_socket is not defined. In the build
#define _nxt_sys_socket <../include/sys/socket.h> /* include path for
the native <sys/socket.h> */
--
Giovanni Rapagnani

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131030/b087d6e1/attachment.html>

[Giovanni Rapagnani]

Hello,
the build fails, there is this error which appears only one time:
/ast_socket.h:12:55: fatal error: x86_64-1-gnu/sys/socket.h: No such
file or directory

The complete build log is in attachment.
Thank you for spending time on this.

Post by Glenn Fowler
thanks
the problem is in "nxt" op in iffe which deals with header intercepts
among other things the intercepts enforce consistent EINTR behavior
across all intercepted calls
grab INIT.2013-10-30.tgz from gsf.cococlyde.org/download/
<http://gsf.cococlyde.org/download/>
On Wed, Oct 30, 2013 at 4:11 AM, Giovanni Rapagnani <giovanni at ideanet.be

Post by Glenn Fowler
what is the definition of _nxt_sys_socket in
$INSTALLROOT/src/lib/libast/ast_socket.h

In the build that failed,_nxt_sys_socket is not defined. In the build
#define _nxt_sys_socket <../include/sys/socket.h> /* include path for
the native <sys/socket.h> */

[Glenn Fowler]

right, I just got a note from dgk about this on ubuntu
the iffe generated file has
#define foo <bar-linux-huh>
but gcc also has
#define linux 1
so foo expands to <bar-1-huh>
I reposted the INIT tarball with iffe fixed to generate "bar-1-huh" instead

Post by Giovanni Rapagnani
Hello,
/ast_socket.h:12:55: fatal error: x86_64-1-gnu/sys/socket.h: No such
file or directory
The complete build log is in attachment.
Thank you for spending time on this.

Post by Glenn Fowler
thanks
the problem is in "nxt" op in iffe which deals with header intercepts
among other things the intercepts enforce consistent EINTR behavior
across all intercepted calls
grab INIT.2013-10-30.tgz from gsf.cococlyde.org/download/
<http://gsf.cococlyde.org/download/>
On Wed, Oct 30, 2013 at 4:11 AM, Giovanni Rapagnani <giovanni at ideanet.be

Post by Glenn Fowler
what is the definition of _nxt_sys_socket in
$INSTALLROOT/src/lib/libast/ast_socket.h

In the build that failed,_nxt_sys_socket is not defined. In the build
#define _nxt_sys_socket <../include/sys/socket.h> /* include path for
the native <sys/socket.h> */

--
Giovanni Rapagnani

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131030/c74a9d7e/attachment.html>

[Giovanni Rapagnani]

I tested the build process on Debian stable 7.0 (x86_64), Debian
testing (x86_64), Ubuntu 12.04 (x86_64) and Raspbian Wheezy (arm 32bit).
It succeeded on all of them. (all of them use the debian multiarch
implementation)

Thank you.

Post by Glenn Fowler
right, I just got a note from dgk about this on ubuntu
the iffe generated file has
#define foo <bar-linux-huh>
but gcc also has
#define linux 1
so foo expands to <bar-1-huh>
I reposted the INIT tarball with iffe fixed to generate "bar-1-huh" instead
On Wed, Oct 30, 2013 at 3:55 PM, Giovanni Rapagnani <giovanni at ideanet.be
Hello,
/ast_socket.h:12:55: fatal error: x86_64-1-gnu/sys/socket.h: No such
file or directory
The complete build log is in attachment.
Thank you for spending time on this.

Post by Glenn Fowler
thanks
the problem is in "nxt" op in iffe which deals with header intercepts
among other things the intercepts enforce consistent EINTR behavior
across all intercepted calls
grab INIT.2013-10-30.tgz from gsf.cococlyde.org/download/

<http://gsf.cococlyde.org/download/>

Post by Glenn Fowler
<http://gsf.cococlyde.org/download/>
On Wed, Oct 30, 2013 at 4:11 AM, Giovanni Rapagnani

<giovanni at ideanet.be <mailto:giovanni at ideanet.be>

Post by Glenn Fowler

Post by Glenn Fowler
what is the definition of _nxt_sys_socket in
$INSTALLROOT/src/lib/libast/ast_socket.h

In the build that failed,_nxt_sys_socket is not defined. In

the build

Post by Glenn Fowler
#define _nxt_sys_socket <../include/sys/socket.h> /* include

path for

Post by Glenn Fowler
the native <sys/socket.h> */

[Glenn Fowler]

thanks for reporting back

dgk has a brand new ubuntu installation that is having runtime symbol
clashes
the known one is standalone libcmd grep referencing the ast error() but
getting bound to the glibc error()
its as if the runtime resolves glibc symbols before libast symbols
assuming it *can't really be doing that* its a mystery why it seems that way

Post by Giovanni Rapagnani
I tested the build process on Debian stable 7.0 (x86_64), Debian
testing (x86_64), Ubuntu 12.04 (x86_64) and Raspbian Wheezy (arm 32bit).
It succeeded on all of them. (all of them use the debian multiarch
implementation)
Thank you.

Post by Glenn Fowler
right, I just got a note from dgk about this on ubuntu
the iffe generated file has
#define foo <bar-linux-huh>
but gcc also has
#define linux 1
so foo expands to <bar-1-huh>
I reposted the INIT tarball with iffe fixed to generate "bar-1-huh"

instead

Post by Glenn Fowler
On Wed, Oct 30, 2013 at 3:55 PM, Giovanni Rapagnani <giovanni at ideanet.be
Hello,
/ast_socket.h:12:55: fatal error: x86_64-1-gnu/sys/socket.h: No such
file or directory
The complete build log is in attachment.
Thank you for spending time on this.

Post by Glenn Fowler
thanks
the problem is in "nxt" op in iffe which deals with header

intercepts

Post by Glenn Fowler

Post by Glenn Fowler
among other things the intercepts enforce consistent EINTR behavior
across all intercepted calls
grab INIT.2013-10-30.tgz from gsf.cococlyde.org/download/

<http://gsf.cococlyde.org/download/>

Post by Glenn Fowler
<http://gsf.cococlyde.org/download/>
On Wed, Oct 30, 2013 at 4:11 AM, Giovanni Rapagnani

<giovanni at ideanet.be <mailto:giovanni at ideanet.be>

Post by Glenn Fowler

Post by Glenn Fowler
what is the definition of _nxt_sys_socket in
$INSTALLROOT/src/lib/libast/ast_socket.h

In the build that failed,_nxt_sys_socket is not defined. In

the build

Post by Glenn Fowler
#define _nxt_sys_socket <../include/sys/socket.h> /* include

path for

Post by Glenn Fowler
the native <sys/socket.h> */

--
Giovanni Rapagnani

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131101/b37291ed/attachment.html>

[Cedric Blancher]

Post by Glenn Fowler
thanks for reporting back
dgk has a brand new ubuntu installation that is having runtime symbol
clashes
the known one is standalone libcmd grep referencing the ast error() but
getting bound to the glibc error()
its as if the runtime resolves glibc symbols before libast symbols
assuming it *can't really be doing that* its a mystery why it seems that way

The Linux runtime linker is different from the Unix runtime linker. If
two libraries have the same symbol name the Linux runtime linker will
first search all loaded libraries for a symbol in the order in which
they were loaded. If libc was loaded before libcmd the symbols will
override anything with the same symbol name coming from libcmd.

Use the maplibc ast build option to avoid this.
Or use a real Unix with ELF conforming symbol lookup.

Ced

[Glenn Fowler]

I don't buy the linux vs unix argument
redhat and suse don't have this behavior -- it was only noticed on a fresh
ubuntu

anyway I'm looking into making map-libc the default to avoid hitting this
on any variant
the only problem is a one time backwards binary compatibility hit that will
be hard to avoid


On Thu, Nov 14, 2013 at 6:58 PM, Cedric Blancher

Post by Cedric Blancher

Post by Glenn Fowler
thanks for reporting back
dgk has a brand new ubuntu installation that is having runtime symbol
clashes
the known one is standalone libcmd grep referencing the ast error() but
getting bound to the glibc error()
its as if the runtime resolves glibc symbols before libast symbols
assuming it *can't really be doing that* its a mystery why it seems that

way
The Linux runtime linker is different from the Unix runtime linker. If
two libraries have the same symbol name the Linux runtime linker will
first search all loaded libraries for a symbol in the order in which
they were loaded. If libc was loaded before libcmd the symbols will
override anything with the same symbol name coming from libcmd.
Use the maplibc ast build option to avoid this.
Or use a real Unix with ELF conforming symbol lookup.
Ced
--
Cedric Blancher <cedric.blancher at gmail.com>
Institute Pasteur

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.research.att.com/pipermail/ast-developers/attachments/20131115/33ac747b/attachment.html>

[Giovanni Rapagnani]

Hi Olga,

Post by ольга крыжановская
Giovanni, I can reproduce your problem, with gcc 4.8 and clang as the
compilers. As more than one compiler is affected, I think this is a
problem in the system includes, in Debian specifically, as no other
Linux has this trouble.

Note that I can reproduce the same issue on ubuntu 12.04 .

Debian has adopted their own implementation of multiarch. If I
understand well, Debian Multiarch allows to install libraries and header
files of whatever architecture you want on a given system. For instance
you can install arm 32 bits and mips 64 bits lib files and header files
on you running system which is based on x86 32 bits. This will allow,
for instance, to cross-compile or to run binaries of another
architecture via an emulator. The files will always be installed in the
same location, for instance, x86 64bit specific header files for a
system running a freebsd kernel will always be installed under
/usr/{lib,include}/x86_64-kfreebsd-gnu whereas the files for arm 32 bit
for a linux kernel will be in /usr/{lib,include}/arm-linux-gnueabi. It
is hard for me to better explain multiarch. If I can suggest some
selected-and-fast-to-read reading about Debian multiarch:

o "About" section of https://wiki.debian.org/Multiarch
o "Introduction" section, then "Proposed general solution" section of
https://wiki.debian.org/Multiarch/TheCaseForMultiarch
o List of gnu triplet (x86_64-kfreebsd-gnu, arm-linux-gnueabi are called
gnu triplet) https://wiki.debian.org/Multiarch/Tuples

Post by ольга крыжановская
Do you know where the string 'x86_64-linux-gnu' comes from? Who
declares it? Is it created? What values do other platforms use?

If I run gcc -v on my x86_64 sytem, I can see in the output:
Target: x86_64-linux-gnu

And on a ARM virtual machine, I have:
Target: arm-linux-gnueabihf

See the link above to the list of gnu triplet for other values.

I am going to give other details about my issue in response to the mail
of Irek.