[ast-developers] building ksh93 on macOS using clang/LLVM

[Kurtis Rader]

Has anyone else worked on making it possible to build ksh93 on macOS using
clang/LLVM rather than gcc? After many hours of investigation and
experimentation I have a working solution. But it involves hacking the
`./src/cmd/nmake/ppcc.sh` script to special-case clang and remove the `-I-`
flag from the list of arguments. Doing that, plus eliminating all the AST
items not needed to build ksh93, reduces the build time from 12 minutes to
8 minutes on my server. And that is without optimizing all the `iffe`
invocations.

The current build toolchain hasn't been updated in a very long time. A lot
of the logic involves dealing with ancient history such as C compilers that
only recognize K&R syntax, not ANSI C syntax. It is also incredibly
inefficient. For example, man source modules are compiled twice
(e.g., src/lib/libast/disc/sfstrtmp.c). Not to mention redundant `iffe`
invocations to detect various features of the platform.

There is an open issue, https://github.com/att/ast/issues/42, discussing
replacing the current build system with something based on autoconf or
Cmake. Given what I've observed in the past few days trying to get ksh93 to
build on macOS with clang I feel quite strongly the current build system
needs to be replaced. What do you think?

Note that I'm approaching this from the perspective that no one cares about
anything in the AST project other than ksh93. Which is certainly true for
myself. It also appears to be true for everyone else who has commented on
the code for this project hosted on Github: https://github.com/att/ast/.

[Eleftherios Koutsofios]

hi,

if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is supposed to
be all of AST.
I don't know if anyone still uses the whole thing but I'd rather keep it
whole.

Glenn and Dave used to package up ksh93 and just the libs needed to
build it,
I can put that up as /att/ksh93 if there's interest. it's from 2013,
like the master branch of AST.

the biggest issue I have with building AST isn't so much the legacy
code, but the header file incompatibilities, and these arise from AST
trying to replace system libraries with better versions.
at some point I had built AST with clang on linux but not on OSX and it
had to do
with the different headers in OSX.

thanks

lefteris

Post by Kurtis Rader
Has anyone else worked on making it possible to build ksh93 on macOS
using clang/LLVM rather than gcc? After many hours of investigation
and experimentation I have a working solution. But it involves hacking
the `./src/cmd/nmake/ppcc.sh` script to special-case clang and remove
the `-I-` flag from the list of arguments. Doing that, plus
eliminating all the AST items not needed to build ksh93, reduces the
build time from 12 minutes to 8 minutes on my server. And that is
without optimizing all the `iffe` invocations.
The current build toolchain hasn't been updated in a very long time. A
lot of the logic involves dealing with ancient history such as C
compilers that only recognize K&R syntax, not ANSI C syntax. It is
also incredibly inefficient. For example, man source modules are
compiled twice (e.g., src/lib/libast/disc/sfstrtmp.c). Not to mention
redundant `iffe` invocations to detect various features of the platform.
There is an open issue, https://github.com/att/ast/issues/42
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_att_ast_issues_42&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=w8fB702vkC3W-R3CrCrZ-z27uWBR8v2uNvO18rq1ia0&m=9ZDeGj9J8zJsJZtwaJZIlwXqG3_aTtX1uOQRouY9sVM&s=8uVh7UbGTIEmgiHrwImr8e0qpbHGiM7qXCCkqsEyHKI&e=>,
discussing replacing the current build system with something based on
autoconf or Cmake. Given what I've observed in the past few days
trying to get ksh93 to build on macOS with clang I feel quite strongly
the current build system needs to be replaced. What do you think?
Note that I'm approaching this from the perspective that no one cares
about anything in the AST project other than ksh93. Which is certainly
true for myself. It also appears to be true for everyone else who has
https://github.com/att/ast/
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_att_ast_&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=w8fB702vkC3W-R3CrCrZ-z27uWBR8v2uNvO18rq1ia0&m=9ZDeGj9J8zJsJZtwaJZIlwXqG3_aTtX1uOQRouY9sVM&s=CotRuyK0MbUsjGmjg-G5TJtSDZGoa4z9f4-ZYPib7Xk&e=>.
--
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank
_______________________________________________
ast-developers mailing list
http://lists.research.att.com/mailman/listinfo/ast-developers

[Siteshwar Vashisht]

----- Original Message -----

Sent: Thursday, November 2, 2017 3:47:52 PM
Subject: Re: [ast-developers] building ksh93 on macOS using clang/LLVM
hi,
if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is supposed to be
all of AST.
I don't know if anyone still uses the whole thing but I'd rather keep it
whole.

What is the reason for it ? I would not want to keep the code copied over to several places. As a reference to full code, you can branch out or tag your last commit in beta branch. But I am not sure how many people will be willing to maintain it.

Glenn and Dave used to package up ksh93 and just the libs needed to build it,
I can put that up as /att/ksh93 if there's interest. it's from 2013, like the
master branch of AST.
the biggest issue I have with building AST isn't so much the legacy code, but
the header file incompatibilities, and these arise from AST trying to
replace system libraries with better versions.
at some point I had built AST with clang on linux but not on OSX and it had
to do
with the different headers in OSX.
thanks
lefteris
--
E. Koutsofios
AT&T Labs - Research
_______________________________________________
ast-developers mailing list
http://lists.research.att.com/mailman/listinfo/ast-developers

[Siteshwar Vashisht]

----- Original Message -----

Sent: Thursday, November 2, 2017 3:47:52 PM
Subject: Re: [ast-developers] building ksh93 on macOS using clang/LLVM
hi,
if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is supposed to be
all of AST.
I don't know if anyone still uses the whole thing but I'd rather keep it
whole.
Glenn and Dave used to package up ksh93 and just the libs needed to build it,
I can put that up as /att/ksh93 if there's interest. it's from 2013, like the
master branch of AST.

Going back to the question of making a new repository, most of the discussion in current repository's issues is around ksh93 only, so it makes sense to continue with it. Making a new one will just create confusion in the community.

the biggest issue I have with building AST isn't so much the legacy code, but
the header file incompatibilities, and these arise from AST trying to
replace system libraries with better versions.
at some point I had built AST with clang on linux but not on OSX and it had
to do
with the different headers in OSX.
thanks
lefteris
--
E. Koutsofios
AT&T Labs - Research
_______________________________________________
ast-developers mailing list
http://lists.research.att.com/mailman/listinfo/ast-developers

[Kurtis Rader]

Post by Eleftherios Koutsofios
if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is supposed to
be all of AST.
I don't know if anyone still uses the whole thing but I'd rather keep it
whole.

I was leaning towards cloning just the components needed to build ksh93
into a new subtree. The existing directories would be left as is so that
anyone who wanted to build the other AST commands could continue to do so.
As well as build the ksh93 as it existed at the time the project was open
sourced. The reason to clone the ksh93 related files is to make it easier
to focus on only those pieces that would need to be modified to fix bugs,
lint, code style, the build tool chain, add new features, etc.

Post by Eleftherios Koutsofios
Glenn and Dave used to package up ksh93 and just the libs needed to build
it,
I can put that up as /att/ksh93 if there's interest. it's from 2013, like
the master branch of AST.

Thanks, but that's probably not needed. I've already spent a few hours to
identify the bare minimum of the existing directories needed to build
ksh93. See https://github.com/att/ast/issues/42#issuecomment-339186060

Post by Eleftherios Koutsofios
the biggest issue I have with building AST isn't so much the legacy code,
but the header file incompatibilities, and these arise from AST trying to
replace system libraries with better versions.

Yep, that has been the source of problems getting it to build on BSD
flavors (including macOS). With the help of a couple of other people we
recently merged a change that seems to fix those issues. But it would be
much better if we could eliminate things like the AST wchar.h header that
wraps the system provided header.

[Eleftherios Koutsofios]

that sounds good.
it seems there's at least a couple of people that are interested in
other parts of AST besides KSH,
so focusing on KSH but keeping everything else too seems like a good
compromise.

thanks

lefteris

On Thu, Nov 2, 2017 at 7:47 AM, Eleftherios Koutsofios
if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is
supposed to be all of AST.
I don't know if anyone still uses the whole thing but I'd rather
keep it whole.
I was leaning towards cloning just the components needed to build
ksh93 into a new subtree. The existing directories would be left as is
so that anyone who wanted to build the other AST commands could
continue to do so. As well as build the ksh93 as it existed at the
time the project was open sourced. The reason to clone the ksh93
related files is to make it easier to focus on only those pieces that
would need to be modified to fix bugs, lint, code style, the build
tool chain, add new features, etc.
Glenn and Dave used to package up ksh93 and just the libs needed
to build it,
I can put that up as /att/ksh93 if there's interest. it's from
2013, like the master branch of AST.
Thanks, but that's probably not needed. I've already spent a few hours
to identify the bare minimum of the existing directories needed to
build ksh93. See
https://github.com/att/ast/issues/42#issuecomment-339186060
<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_att_ast_issues_42-23issuecomment-2D339186060&d=DwMFaQ&c=LFYZ-o9_HUMeMTSQicvjIg&r=w8fB702vkC3W-R3CrCrZ-z27uWBR8v2uNvO18rq1ia0&m=4g6BlPj_-xNH2F9qRa_jylqrCbjWTOpOuqKNsVpVArw&s=MZMC3eKhD-jJLd7bBP5qqeqffs26WUjzlVCyvCbKGpo&e=>
the biggest issue I have with building AST isn't so much the
legacy code, but the header file incompatibilities, and these
arise from AST trying to replace system libraries with better
versions.
Yep, that has been the source of problems getting it to build on BSD
flavors (including macOS). With the help of a couple of other people
we recently merged a change that seems to fix those issues. But it
would be much better if we could eliminate things like the AST wchar.h
header that wraps the system provided header.
--
Kurtis Rader
Caretaker of the exceptional canines Junior and Hank

[Jeff Frontz]

Post by Eleftherios Koutsofios
if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is supposed to
be all of AST.
I don't know if anyone still uses the whole thing but I'd rather keep it
whole.

I resonate with Elefteris's sentiment -- in addition to ksh, I still use
the AST version of nmake (and whatever other AST tools/libraries nmake
implicitly uses).

Jeff

[EDGAR, ADAM]

I don’t know why a second build process that only builds what is needed for ksh can not be added to the existing source tree. I don’t think checking out unneeded code is as much of a burden as having multiple places to bug patch is. I’m already in the process of patching a few bugs I’ve found in codex/vcodex and realized there is duplicate code there which I need to merge. Having multiple branches will just make keeping in sync that much worse.

ASE

From: <ast-developers-***@lists.research.att.com> on behalf of Jeff Frontz <***@gmail.com>
Date: Thursday, November 2, 2017 at 10:44 AM
To: "ast-***@lists.research.att.com" <ast-***@lists.research.att.com>
Subject: Re: [ast-developers] building ksh93 on macOS using clang/LLVM




From: Eleftherios Koutsofios <***@research.att.com<mailto:***@research.att.com>>

if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is supposed to
be all of AST.
I don't know if anyone still uses the whole thing but I'd rather keep it
whole.

I resonate with Elefteris's sentiment -- in addition to ksh, I still use the AST version of nmake (and whatever other AST tools/libraries nmake implicitly uses).

Jeff

[Jeff Frontz]

Post by EDGAR, ADAM
I don’t know why a second build process that only builds what is needed
for ksh can not be added to the existing source tree.

I agree, that sounds fine -- but what I thought I'd been seeing was an
effort to strip things out of ksh or otherwise cause a divergence from what
would compile in the stock AST environment. Maybe I'm
misreading/misunderstanding all of the github traffic/comments?

Jeff

[Jeff Frontz]

I don't have any contact with David K; if his Google group is working on
ksh.next (under another name) it might not be worth the modernization
effort.

That is the $64K question, right? If he or Glenn F are just not that into
it any more (and never will be), then it's probably time for me (and the
projects I work on) to give in and welcome our bash (and non-nmake)
overlords (and I can lament the loss of ksh/nmake the way the old-timers
back-in-the-day lamented the loss of Multics ;-)


Jeff

[Siteshwar Vashisht]

----- Original Message -----

Sent: Thursday, November 2, 2017 5:09:15 PM
Subject: Re: [ast-developers] building ksh93 on macOS using clang/LLVM
I don’t know why a second build process that only builds what is needed for
ksh can not be added to the existing source tree.
I agree, that sounds fine -- but what I thought I'd been seeing was an effort
to strip things out of ksh or otherwise cause a divergence from what would
compile in the stock AST environment. Maybe I'm misreading/misunderstanding
all of the github traffic/comments?

You have been reading the comments correctly and it's unlikely it would be done without breaking the current build process.

Jeff
_______________________________________________
ast-developers mailing list
http://lists.research.att.com/mailman/listinfo/ast-developers

[Jeff Frontz]

reorganizing to leverage the new infrastructure shouldn't be resisted ;>

Again, agreed -- but can the new infrastructure be leveraged to bring the
entire AST build process along (or otherwise leave the existing process
still operable)?

[Siteshwar Vashisht]

----- Original Message -----

Sent: Thursday, November 2, 2017 5:14:43 PM
Subject: Re: [ast-developers] building ksh93 on macOS using clang/LLVM
reorganizing to leverage the new infrastructure shouldn't be resisted ;>
Again, agreed -- but can the new infrastructure be leveraged to bring the
entire AST build process along (or otherwise leave the existing process
still operable)?

I would suggest making a new branch with current build process and keep the branch under low maintenance.

_______________________________________________
ast-developers mailing list
http://lists.research.att.com/mailman/listinfo/ast-developers

[Kurtis Rader]

Post by Siteshwar Vashisht
I would suggest making a new branch with current build process and keep
the branch under low maintenance.

That would also be fine and perhaps a better option than cloning just the
ksh93 related directories into a new source subtree. It sort of depends on
how much maintenance to the existing, non-ksh93, AST code is anticipated.
From what I can tell it looks like there will be very few changes outside
what I'd like to see happen to the ksh93 related code.

[Kurtis Rader]

Post by Jeff Frontz

reorganizing to leverage the new infrastructure shouldn't be resisted ;>

Again, agreed -- but can the new infrastructure be leveraged to bring the
entire AST build process along (or otherwise leave the existing process
still operable)?

As I mentioned in my other reply my intent was to clone just the
directories needed to build ksh93 into a separate subtree and only do
backward incompatible changes (e.g., changing the build tool chain) in that
subtree. That would leave the existing AST build process still operable.
This does mean that the existing src/ tree is unlikely to see many changes.
Certainly I wasn't planning on backporting changes from the new ksh93 only
subtree to the existing src/ tree. That's because I expect the two code
bases to be radically different in the near future due to doing things like
running the code through clang-format to normalize the style and oclint to
identify code that is either outright broken or merely confusing.

We could simply fork a new project that was ksh93 only but it seems
preferable to me to keep the modernized ksh93 source under the umbrella of
the existing AST project.

[Siteshwar Vashisht]

----- Original Message -----

Sent: Thursday, November 2, 2017 4:49:28 PM
Subject: Re: [ast-developers] building ksh93 on macOS using clang/LLVM
I don’t know why a second build process that only builds what is needed for
ksh can not be added to the existing source tree. I don’t think checking out

There has been discussion around build tools[1] on GitHub. We would like to replace the current build toolchain (iffe, nmake etc.) that would also lead to changes in code. So it would break the old build process.

unneeded code is as much of a burden as having multiple places to bug patch
is. I’m already in the process of patching a few bugs I’ve found in
codex/vcodex and realized there is duplicate code there which I need to
merge. Having multiple branches will just make keeping in sync that much
worse.
ASE
Date: Thursday, November 2, 2017 at 10:44 AM
Subject: Re: [ast-developers] building ksh93 on macOS using clang/LLVM
if the plan is to reduce this repo to just enough to build ksh93,
I think it should be moved to a separate repo, since this is supposed to
be all of AST.
I don't know if anyone still uses the whole thing but I'd rather keep it
whole.
I resonate with E lefteris's sentiment -- in addition to ksh, I still use the
AST version of nmake (and whatever other AST tools/libraries nmake
implicitly uses).
Jeff
_______________________________________________
ast-developers mailing list
http://lists.research.att.com/mailman/listinfo/ast-developers

[1] https://github.com/att/ast/issues/42