pnacl's newlib code contains function that can/should not be called

The newlib code contains quite few functions that cannot be linked into a program because they have in turn unmet dependencies.

Cleaning those up is desirably because this allows us to start experimenting with PNaCl's shared infrastructure while working on the glibc port. 

This CL describes the problematic modules in newlib by showing 
how to edit various "Makefile.in"s.
This is NOT the right way of doing this, e.g. the real change would have to happen in Makefile.am, etc.

So this CL is also a request for proposals how to go about this.

Below is an overview of the problematic modules:
* libc/signal: should be completely excluded
* libc/posix:  exec stuff and globbing should be excluded 
* libc/reent: signal, file manipulation, etc should be excluded
* libc/search: hash table code (uses tmp files and page tables manip) should be excluded
* libc/stdio: file manipulation, tmp file stuff should be excluded
* libc/stdlib: system() should be excluded
* libm/math: this is a special case: wf_sqrt.c w_sqrt.c should be excluded as they are redefined/overloaded  in libm/machine/pnacl/
    

BUG= http://code.google.com/p/nativeclient/issues/detail?id=1754

[robertm, 2011-05-04 21:50:52]

PTAL

[pdox, 2011-05-04 22:06:22]

robert,

This is probably not the right way to do this. There must be a way to override
the compilation of sqrt, sqrtf, etc.  on a per-machine basis.

[robertm, 2011-05-04 22:16:59]

Pdox,

The existing mechanism seems to work but it is a hack of sorts as well.

I have a few more CLs for newlib in flight, let's wait for those, to find  the
best solution is.



On 2011/05/04 22:06:22, pdox wrote:
> robert,
> 
> This is probably not the right way to do this. There must be a way to override
> the compilation of sqrt, sqrtf, etc.  on a per-machine basis.

[robertm, 2011-05-05 16:26:46]

I just uploaded the bigger change to newlib that contains hopefully
all the problematic modules.

PTAL


On 2011/05/04 22:16:59, robertm wrote:
> Pdox,
> 
> The existing mechanism seems to work but it is a hack of sorts as well.
> 
> I have a few more CLs for newlib in flight, let's wait for those, to find  the
> best solution is.
> 
> 
> 
> On 2011/05/04 22:06:22, pdox wrote:
> > robert,
> > 
> > This is probably not the right way to do this. There must be a way to
override
> > the compilation of sqrt, sqrtf, etc.  on a per-machine basis.

[pdox, 2011-05-05 16:32:59]

Robert,

This is still not the right way to do this. This will be a pain to fix if it
ever breaks during merge, or if we need to merge with the nacl toolchain.

Can you please find the right way to enable custom functions on a per-machine
basis?

[robertm, 2011-05-05 16:48:15]

Pdox,

I am well aware of this (c.f. the new description).
I was hoping to get some ideas from you how to go about this.

Robert


On 2011/05/05 16:32:59, pdox wrote:
> Robert,
> 
> This is still not the right way to do this. This will be a pain to fix if it
> ever breaks during merge, or if we need to merge with the nacl toolchain.
> 
> Can you please find the right way to enable custom functions on a per-machine
> basis?

[jvoung - send to chromium..., 2011-05-05 18:46:09]

Robert,

So the issue with the others (non-libm/machine) is that the unmet dependencies
aren't normally required in with the static lib, but is in this case? Perhaps
this means that the llvm-link (or whichever tool) needs to be closer to a normal
link, in that respect?

For the libm vs libm/machine stuff, it looks like you already noticed it (you
mentioned a "hack of sorts"), but it looks like it really depends on "ar x"
generating individual .o files:

(1) archive of .o is built in libm/math
(2) archive of .o is built in libm/common
(3) archive of .o is built in libm/machine

In the same order, the files are extracted. Because of that ordering, the files
may overwrite each other:

for i in math/lib.a common/lib.a machine/lib.a; do \
           
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
x ../$i; \
        done; \
       
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
rc ../libm.a *.o



On 2011/05/05 16:48:15, robertm wrote:
> Pdox,
> 
> I am well aware of this (c.f. the new description).
> I was hoping to get some ideas from you how to go about this.
> 
> Robert
> 
> 
> On 2011/05/05 16:32:59, pdox wrote:
> > Robert,
> > 
> > This is still not the right way to do this. This will be a pain to fix if it
> > ever breaks during merge, or if we need to merge with the nacl toolchain.
> > 
> > Can you please find the right way to enable custom functions on a
per-machine
> > basis?

[robertm, 2011-05-05 19:01:25]

On 2011/05/05 18:46:09, jvoung wrote:
> Robert,
> 
> So the issue with the others (non-libm/machine) is that the unmet dependencies
> aren't normally required in with the static lib, but is in this case? Perhaps
> this means that the llvm-link (or whichever tool) needs to be closer to a
normal
> link, in that respect?

If we want to emulate a shared library environment we do not get to pick and
choose: you have to use the entire archive.

While this CL is aimed at getting something to play with until we
have a shared version of glibc working, my gut feeling is that we should have
pruned newlib of these functions anyway a long time ago.

> For the libm vs libm/machine stuff, it looks like you already noticed it (you
> mentioned a "hack of sorts"), but it looks like it really depends on "ar x"
> generating individual .o files:
> 
> (1) archive of .o is built in libm/math
> (2) archive of .o is built in libm/common
> (3) archive of .o is built in libm/machine
> 
> In the same order, the files are extracted. Because of that ordering, the
files
> may overwrite each other:

Right. I view the sqrt functions as the smaller problem. If we come up with a
way to eliminate posix/*exec* and friends we could just adopt that approach to
the sqrt functions as well and the newlib build a little more predictable.

> for i in math/lib.a common/lib.a machine/lib.a; do \
>            
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
> x ../$i; \
>         done; \
>        
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
> rc ../libm.a *.o
> 
> 
> 
> On 2011/05/05 16:48:15, robertm wrote:
> > Pdox,
> > 
> > I am well aware of this (c.f. the new description).
> > I was hoping to get some ideas from you how to go about this.
> > 
> > Robert
> > 
> > 
> > On 2011/05/05 16:32:59, pdox wrote:
> > > Robert,
> > > 
> > > This is still not the right way to do this. This will be a pain to fix if
it
> > > ever breaks during merge, or if we need to merge with the nacl toolchain.
> > > 
> > > Can you please find the right way to enable custom functions on a
> per-machine
> > > basis?

[robertm, 2011-05-06 22:11:25]

I made my changes to the actual Makefile.am files and then 
regenerated the Makefile.in files using 

autoconf

Both file families are included here since we do not run autoconf
in utman.sh.


I could not find any way to make this more elegant but I do not think
these changes are terrible invasive either.

I will resume testing on Monday. Please let me know if the general direction is
OK.


On 2011/05/05 19:01:25, robertm wrote:
> On 2011/05/05 18:46:09, jvoung wrote:
> > Robert,
> > 
> > So the issue with the others (non-libm/machine) is that the unmet
dependencies
> > aren't normally required in with the static lib, but is in this case?
Perhaps
> > this means that the llvm-link (or whichever tool) needs to be closer to a
> normal
> > link, in that respect?
> 
> If we want to emulate a shared library environment we do not get to pick and
> choose: you have to use the entire archive.
> 
> While this CL is aimed at getting something to play with until we
> have a shared version of glibc working, my gut feeling is that we should have
> pruned newlib of these functions anyway a long time ago.
> 
> > For the libm vs libm/machine stuff, it looks like you already noticed it
(you
> > mentioned a "hack of sorts"), but it looks like it really depends on "ar x"
> > generating individual .o files:
> > 
> > (1) archive of .o is built in libm/math
> > (2) archive of .o is built in libm/common
> > (3) archive of .o is built in libm/machine
> > 
> > In the same order, the files are extracted. Because of that ordering, the
> files
> > may overwrite each other:
> 
> Right. I view the sqrt functions as the smaller problem. If we come up with a
> way to eliminate posix/*exec* and friends we could just adopt that approach to
> the sqrt functions as well and the newlib build a little more predictable.
> 
> > for i in math/lib.a common/lib.a machine/lib.a; do \
> >            
> >
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
> > x ../$i; \
> >         done; \
> >        
> >
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
> > rc ../libm.a *.o
> > 
> > 
> > 
> > On 2011/05/05 16:48:15, robertm wrote:
> > > Pdox,
> > > 
> > > I am well aware of this (c.f. the new description).
> > > I was hoping to get some ideas from you how to go about this.
> > > 
> > > Robert
> > > 
> > > 
> > > On 2011/05/05 16:32:59, pdox wrote:
> > > > Robert,
> > > > 
> > > > This is still not the right way to do this. This will be a pain to fix
if
> it
> > > > ever breaks during merge, or if we need to merge with the nacl
toolchain.
> > > > 
> > > > Can you please find the right way to enable custom functions on a
> > per-machine
> > > > basis?

[Use chromium.org instead, 2011-05-06 22:17:46]

I imagine you actually ran "autoreconf" or "automake", since automake is
what reads Makefile.am files.
"autoreconf" is the recommended way to run one or more or autoconf,
automake, and other magic bits.

On Fri, May 6, 2011 at 3:11 PM, <robertm@google.com> wrote:

> I made my changes to the actual Makefile.am files and then
> regenerated the Makefile.in files using
>
> autoconf
>
> Both file families are included here since we do not run autoconf
> in utman.sh.
>
>
> I could not find any way to make this more elegant but I do not think
> these changes are terrible invasive either.
>
> I will resume testing on Monday. Please let me know if the general
> direction is
> OK.
>
>
>
> On 2011/05/05 19:01:25, robertm wrote:
>
>> On 2011/05/05 18:46:09, jvoung wrote:
>> > Robert,
>> >
>> > So the issue with the others (non-libm/machine) is that the unmet
>>
> dependencies
>
>> > aren't normally required in with the static lib, but is in this case?
>>
> Perhaps
>
>> > this means that the llvm-link (or whichever tool) needs to be closer to
>> a
>> normal
>> > link, in that respect?
>>
>
>  If we want to emulate a shared library environment we do not get to pick
>> and
>> choose: you have to use the entire archive.
>>
>
>  While this CL is aimed at getting something to play with until we
>> have a shared version of glibc working, my gut feeling is that we should
>> have
>> pruned newlib of these functions anyway a long time ago.
>>
>
>  > For the libm vs libm/machine stuff, it looks like you already noticed it
>>
> (you
>
>> > mentioned a "hack of sorts"), but it looks like it really depends on "ar
>> x"
>> > generating individual .o files:
>> >
>> > (1) archive of .o is built in libm/math
>> > (2) archive of .o is built in libm/common
>> > (3) archive of .o is built in libm/machine
>> >
>> > In the same order, the files are extracted. Because of that ordering,
>> the
>> files
>> > may overwrite each other:
>>
>
>  Right. I view the sqrt functions as the smaller problem. If we come up
>> with a
>> way to eliminate posix/*exec* and friends we could just adopt that
>> approach to
>> the sqrt functions as well and the newlib build a little more predictable.
>>
>
>  > for i in math/lib.a common/lib.a machine/lib.a; do \
>> >
>> >
>>
>
>
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
>
>> > x ../$i; \
>> >         done; \
>> >
>> >
>>
>
>
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
>
>> > rc ../libm.a *.o
>> >
>> >
>> >
>> > On 2011/05/05 16:48:15, robertm wrote:
>> > > Pdox,
>> > >
>> > > I am well aware of this (c.f. the new description).
>> > > I was hoping to get some ideas from you how to go about this.
>> > >
>> > > Robert
>> > >
>> > >
>> > > On 2011/05/05 16:32:59, pdox wrote:
>> > > > Robert,
>> > > >
>> > > > This is still not the right way to do this. This will be a pain to
>> fix
>>
> if
>
>> it
>> > > > ever breaks during merge, or if we need to merge with the nacl
>>
> toolchain.
>
>> > > >
>> > > > Can you please find the right way to enable custom functions on a
>> > per-machine
>> > > > basis?
>>
>
>
>
> http://codereview.chromium.org/6929022/
>
> --
> You received this message because you are subscribed to the Google Groups
> "Native-Client-Reviews" group.
> To post to this group, send email to
> native-client-reviews@googlegroups.com.
> To unsubscribe from this group, send email to
> native-client-reviews+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/native-client-reviews?hl=en.
>
>

-- 
You received this message because you are subscribed to the Google Groups
"Native-Client-Reviews" group.
To post to this group, send email to native-client-reviews@googlegroups.com.
To unsubscribe from this group, send email to
native-client-reviews+unsubscribe@googlegroups.com.
For more options, visit this group at
http://groups.google.com/group/native-client-reviews?hl=en.

[robertm, 2011-05-10 18:00:27]

I have not heard from anybody in two day, so I assume this change is fine in
principle and will put the final polish on this CL.



On 2011/05/06 22:17:46, mcgrathr wrote:
> I imagine you actually ran "autoreconf" or "automake", since automake is
> what reads Makefile.am files.
> "autoreconf" is the recommended way to run one or more or autoconf,
> automake, and other magic bits.
> 
> On Fri, May 6, 2011 at 3:11 PM, <mailto:robertm@google.com> wrote:
> 
> > I made my changes to the actual Makefile.am files and then
> > regenerated the Makefile.in files using
> >
> > autoconf
> >
> > Both file families are included here since we do not run autoconf
> > in utman.sh.
> >
> >
> > I could not find any way to make this more elegant but I do not think
> > these changes are terrible invasive either.
> >
> > I will resume testing on Monday. Please let me know if the general
> > direction is
> > OK.
> >
> >
> >
> > On 2011/05/05 19:01:25, robertm wrote:
> >
> >> On 2011/05/05 18:46:09, jvoung wrote:
> >> > Robert,
> >> >
> >> > So the issue with the others (non-libm/machine) is that the unmet
> >>
> > dependencies
> >
> >> > aren't normally required in with the static lib, but is in this case?
> >>
> > Perhaps
> >
> >> > this means that the llvm-link (or whichever tool) needs to be closer to
> >> a
> >> normal
> >> > link, in that respect?
> >>
> >
> >  If we want to emulate a shared library environment we do not get to pick
> >> and
> >> choose: you have to use the entire archive.
> >>
> >
> >  While this CL is aimed at getting something to play with until we
> >> have a shared version of glibc working, my gut feeling is that we should
> >> have
> >> pruned newlib of these functions anyway a long time ago.
> >>
> >
> >  > For the libm vs libm/machine stuff, it looks like you already noticed it
> >>
> > (you
> >
> >> > mentioned a "hack of sorts"), but it looks like it really depends on "ar
> >> x"
> >> > generating individual .o files:
> >> >
> >> > (1) archive of .o is built in libm/math
> >> > (2) archive of .o is built in libm/common
> >> > (3) archive of .o is built in libm/machine
> >> >
> >> > In the same order, the files are extracted. Because of that ordering,
> >> the
> >> files
> >> > may overwrite each other:
> >>
> >
> >  Right. I view the sqrt functions as the smaller problem. If we come up
> >> with a
> >> way to eliminate posix/*exec* and friends we could just adopt that
> >> approach to
> >> the sqrt functions as well and the newlib build a little more predictable.
> >>
> >
> >  > for i in math/lib.a common/lib.a machine/lib.a; do \
> >> >
> >> >
> >>
> >
> >
> >
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
> >
> >> > x ../$i; \
> >> >         done; \
> >> >
> >> >
> >>
> >
> >
> >
>
/usr/local/google/home/nacl4/native_client/toolchain/pnacl_linux_x86_64/bin/pnacl-ar
> >
> >> > rc ../libm.a *.o
> >> >
> >> >
> >> >
> >> > On 2011/05/05 16:48:15, robertm wrote:
> >> > > Pdox,
> >> > >
> >> > > I am well aware of this (c.f. the new description).
> >> > > I was hoping to get some ideas from you how to go about this.
> >> > >
> >> > > Robert
> >> > >
> >> > >
> >> > > On 2011/05/05 16:32:59, pdox wrote:
> >> > > > Robert,
> >> > > >
> >> > > > This is still not the right way to do this. This will be a pain to
> >> fix
> >>
> > if
> >
> >> it
> >> > > > ever breaks during merge, or if we need to merge with the nacl
> >>
> > toolchain.
> >
> >> > > >
> >> > > > Can you please find the right way to enable custom functions on a
> >> > per-machine
> >> > > > basis?
> >>
> >
> >
> >
> > http://codereview.chromium.org/6929022/
> >
> > --
> > You received this message because you are subscribed to the Google Groups
> > "Native-Client-Reviews" group.
> > To post to this group, send email to
> > mailto:native-client-reviews@googlegroups.com.
> > To unsubscribe from this group, send email to
> > mailto:native-client-reviews+unsubscribe@googlegroups.com.
> > For more options, visit this group at
> > http://groups.google.com/group/native-client-reviews?hl=en.
> >
> >
> 
> -- 
> You received this message because you are subscribed to the Google Groups
> "Native-Client-Reviews" group.
> To post to this group, send email to
mailto:native-client-reviews@googlegroups.com.
> To unsubscribe from this group, send email to
> mailto:native-client-reviews+unsubscribe@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/native-client-reviews?hl=en.
>

[jvoung - send to chromium..., 2011-05-10 21:25:36]

I guess it doesn't make much sense to have these unusable functions...

Some thoughts about approach: how do these relate to libnosys and the libnosys
approach? I guess that was mostly done by setting "sys_dir=nacl"... but that
controls an entire directory and not files within many different directories as
you are doing here...

Some possible badness of libnosys-style:
 - It allows the link to happen, and wait for a runtime failure instead of
stopping at link time...

Some possible goodness of libnosys-style:
 - A highly-curious user has a way of listing the unimplemented functions by
looking at the symbols in libnosys.
 - Less invasive changes to newlib.

http://codereview.chromium.org/6929022/diff/10001/newlib-trunk/newlib/libm/ma...
File newlib-trunk/newlib/libm/math/Makefile.am (right):

http://codereview.chromium.org/6929022/diff/10001/newlib-trunk/newlib/libm/ma...
newlib-trunk/newlib/libm/math/Makefile.am:31: src_pnacl_eliminated = w_sqrt.c
About the name of these, changes in this file are not so much "eliminated", but
supplied by the machine dir. Could use some documentation.

[pdox, 2011-05-11 15:40:27]

robertm,

I really don't like this CL. I don't understand what you are trying to
accomplish here. We are not going to be turning archives into .so's. The way to
create bitcode shared objects will be exactly like creating normal shared
objects... using pnacl-ld -shared, not pnacl-ar.

Also, why bother? As far as I know, we are not planning on turning newlib into a
shared library. We'll be using glibc for our libc.so. newlib will only ever be
used for static linking, and thus always in .a form.

[robertm, 2011-05-11 16:13:22]

On 2011/05/11 15:40:27, pdox wrote:
> robertm,
> 
> I really don't like this CL. I don't understand what you are trying to
> accomplish here. We are not going to be turning archives into .so's. The way
to
> create bitcode shared objects will be exactly like creating normal shared
> objects... using pnacl-ld -shared, not pnacl-ar.

Pdox,

Leaving your concerns about the linking issues aside.
This CL is pruning stuff from newlib that cannot be called anyway
without incurring link-time errors.
I think this alone is a good enough justification.
Maintaining these changes should be rather trivial as well.

What concretely is it that you do not like about this CL?

Robert


> 
> Also, why bother? As far as I know, we are not planning on turning newlib into
a
> shared library. We'll be using glibc for our libc.so. newlib will only ever be
> used for static linking, and thus always in .a form.

[sehr (please use chromium), 2011-05-11 16:45:01]

On 2011/05/11 16:13:22, robertm wrote:
> On 2011/05/11 15:40:27, pdox wrote:
> > robertm,
> > 
> > I really don't like this CL. I don't understand what you are trying to
> > accomplish here. We are not going to be turning archives into .so's. The way
> to
> > create bitcode shared objects will be exactly like creating normal shared
> > objects... using pnacl-ld -shared, not pnacl-ar.
> 
> Pdox,
> 
> Leaving your concerns about the linking issues aside.
> This CL is pruning stuff from newlib that cannot be called anyway
> without incurring link-time errors.
> I think this alone is a good enough justification.
> Maintaining these changes should be rather trivial as well.
> 
> What concretely is it that you do not like about this CL?
> 
> Robert
> 
> 
> > 
> > Also, why bother? As far as I know, we are not planning on turning newlib
into
> a
> > shared library. We'll be using glibc for our libc.so. newlib will only ever
be
> > used for static linking, and thus always in .a form.

While I have some other concerns I have expressed regarding the use of archivers
rather than linkers for producing .sos, this change seems unrelated and
generally acceptable.  LGTM.