[Proposal] jlink plugin which replaces dynamic shared libraries

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[Proposal] jlink plugin which replaces dynamic shared libraries

Severin Gehwolf
Hi,

Apologies if this isn't the right list. Please let me know where jlink
related issues are being discussed.

The Problem:
-----------------------------------------------
This issue originally surfaced in our packaged OpenJDK 11 builds for
Fedora. If a user created a custom image from the packaged JDK, the
resulting custom image would be huge in size because of non-stripped
debug symbols in the packaged binaries (~500MB vs. ~50MB).

Native libraries and executables in the packaged JDK image would be
properly stripped of debug symbols, but copies of those files in the
jmod files would not be. That has to do with a) How we build for the
Fedora distribution: --with-native-debug-symbols=internal configure
switch. b) When the stripping happens. In the Fedora case, b) happens
external to the OpenJDK build by the RPM build system *after* the
OpenJDK build actually completes. This means copies of native libraries
and executables with full debug symbols would already be "zipped up" in
jmod files when the distribution build's stripping process starts.
Hence, the distribution build system doesn't "see" DSO/EXE copies in
jmod files in the JDK images directory. They are archived in jmod
files. Because of this we end up with properly stripped libraries and
executables in the default, extracted JDK image, but DSOs/EXEs in the
'jmods' directory would still have full debug symbols internal to the
binaries. Yet, when jlink is being run, jmod files in directory 'jmods'
are being used for creating custom modular JDK images.

The original downstream issue is:
https://bugzilla.redhat.com/show_bug.cgi?id=1652177


Proposed Solution:
-----------------------------------------------
The solution we came up with is a new jlink plugin which - at runtime -
replaces matching DSOs from jmod files with versions from the JDK image
(or a JDK image of the same version pointed to by a property). As a
result, properly stripped DSOs are being used for the custom JDK image
and the size issue goes away. We are aware that this is a rather
distribution specific use-case, but it might be useful for other
downstream consumers (Debian?) too. That's why I'm proposing this here.
Would there be interest in having this jlink plugin upstream? If so,
I'd create a bug and will propose a formal RFR.

webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/jlink-native-replace-plugin/webrev/

Thoughts?

Thanks,
Severin

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Alan Bateman
On 04/12/2018 12:47, Severin Gehwolf wrote:

> :
>
>
> Proposed Solution:
> -----------------------------------------------
> The solution we came up with is a new jlink plugin which - at runtime -
> replaces matching DSOs from jmod files with versions from the JDK image
> (or a JDK image of the same version pointed to by a property). As a
> result, properly stripped DSOs are being used for the custom JDK image
> and the size issue goes away. We are aware that this is a rather
> distribution specific use-case, but it might be useful for other
> downstream consumers (Debian?) too. That's why I'm proposing this here.
> Would there be interest in having this jlink plugin upstream? If so,
> I'd create a bug and will propose a formal RFR.
>
> webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/jlink-native-replace-plugin/webrev/
>
> Thoughts?
>
If I read this correctly then it makes assumption that jlink is run on a
JDK build that matches exactly the build of the packaged modules on the
module path specified to the tool - is that right? If so then it will
work for the scenario you described but it won't work in general as the
builds may not match.  Also it could never work when cross targeting,
e.g. using jlink on linux-x64 to create a run-time image for an embedded
linux-arm.

You mail makes me wonder if it would be feasible to have a plugin that
does the stripping, maybe you've looked into that already?

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Severin Gehwolf
Hi Alan,

On Tue, 2018-12-04 at 13:19 +0000, Alan Bateman wrote:

> On 04/12/2018 12:47, Severin Gehwolf wrote:
> > :
> >
> >
> > Proposed Solution:
> > -----------------------------------------------
> > The solution we came up with is a new jlink plugin which - at runtime -
> > replaces matching DSOs from jmod files with versions from the JDK image
> > (or a JDK image of the same version pointed to by a property). As a
> > result, properly stripped DSOs are being used for the custom JDK image
> > and the size issue goes away. We are aware that this is a rather
> > distribution specific use-case, but it might be useful for other
> > downstream consumers (Debian?) too. That's why I'm proposing this here.
> > Would there be interest in having this jlink plugin upstream? If so,
> > I'd create a bug and will propose a formal RFR.
> >
> > webrev: http://cr.openjdk.java.net/~sgehwolf/webrevs/jlink-native-replace-plugin/webrev/
> >
> > Thoughts?
> >
>
> If I read this correctly then it makes assumption that jlink is run on a
> JDK build that matches exactly the build of the packaged modules on the
> module path specified to the tool - is that right?

The inherent assumption is that jlink is run on a JDK build that has a
module set at least equal to the modules specified to the tool. Other
than that, yes.

> If so then it will
> work for the scenario you described but it won't work in general as the
> builds may not match.  Also it could never work when cross targeting,
> e.g. using jlink on linux-x64 to create a run-time image for an embedded
> linux-arm.

Yes, understood.

> You mail makes me wonder if it would be feasible to have a plugin that
> does the stripping, maybe you've looked into that already?

We've considered it, but I haven't looked into it.

Thanks,
Severin

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Alan Bateman
On 04/12/2018 13:52, Severin Gehwolf wrote:
> :
> The inherent assumption is that jlink is run on a JDK build that has a
> module set at least equal to the modules specified to the tool. Other
> than that, yes.
The approach would need integrity as it would be too easy to end up with
a mismatch between the native libraries and the other resources in the
module.


> :
>
>> You mail makes me wonder if it would be feasible to have a plugin that
>> does the stripping, maybe you've looked into that already?
> We've considered it, but I haven't looked into it.
I think it's worth exploring, even if it means requiring the `strip`
tool on the system.

-Alan.
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Severin Gehwolf
On Tue, 2018-12-04 at 15:15 +0000, Alan Bateman wrote:
> > > You mail makes me wonder if it would be feasible to have a plugin that
> > > does the stripping, maybe you've looked into that already?
> >
> > We've considered it, but I haven't looked into it.
>
> I think it's worth exploring, even if it means requiring the `strip`
> tool on the system.

If that's an option I'll look into it:
https://bugs.openjdk.java.net/browse/JDK-8214796

Thanks,
Severin

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Severin Gehwolf
Hi Alan,

On Tue, 2018-12-04 at 16:55 +0100, Severin Gehwolf wrote:

> On Tue, 2018-12-04 at 15:15 +0000, Alan Bateman wrote:
> > > > You mail makes me wonder if it would be feasible to have a plugin that
> > > > does the stripping, maybe you've looked into that already?
> > >
> > > We've considered it, but I haven't looked into it.
> >
> > I think it's worth exploring, even if it means requiring the `strip`
> > tool on the system.
>
> If that's an option I'll look into it:
> https://bugs.openjdk.java.net/browse/JDK-8214796

I've added a proof-of-concept for a strip-native-debug-symbols plugin
to the bug. It requires objcopy to be on the system on which jlink is
being run. Comments welcome.

Thanks,
Severin

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Alan Bateman
On 18/12/2018 14:40, Severin Gehwolf wrote:
> :
> I've added a proof-of-concept for a strip-native-debug-symbols plugin
> to the bug. It requires objcopy to be on the system on which jlink is
> being run. Comments welcome.
>
I haven't had time to study this closely yet but I think having a plugin
to strip the native debug symbols would be good to have. I suspect that
jlink will need a bit of support for integrating platform specific
plugins into the build as tools such as objdump and options such as
--add-gun-debuglink aren't going to make sense everywhere. From the
examples, I also wonder about unwieldily command lines usages and
whether having a platform specific plugin would make this a bit easier.

-Alan.




Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Severin Gehwolf
On Fri, 2018-12-21 at 12:07 +0000, Alan Bateman wrote:
> On 18/12/2018 14:40, Severin Gehwolf wrote:
> > :
> > I've added a proof-of-concept for a strip-native-debug-symbols plugin
> > to the bug. It requires objcopy to be on the system on which jlink is
> > being run. Comments welcome.
> >
>
> I haven't had time to study this closely yet but I think having a plugin
> to strip the native debug symbols would be good to have.

Great.

> I suspect that
> jlink will need a bit of support for integrating platform specific
> plugins into the build as tools such as objdump and options such as
> --add-gun-debuglink aren't going to make sense everywhere.

Yes. It whould make sense to include this plugin only for Linux and,
perhaps, Solaris. If there was a way to include plugins based on the
build platform this would help.

> From the
> examples, I also wonder about unwieldily command lines usages and
> whether having a platform specific plugin would make this a bit easier.

Sure. One specific issue I've run into was that there seems to be no
way to specify for a plugin to take >= 0 arguments. Once a plugin
specifies, hasArgument() == true, it seems to require at least one.

Thanks,
Severin

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Mandy Chung
I also like this, stripping the native debug symbols, as a jlink plugin.

Currently jlink does not support platform-specific plugin.   The Plugin
API will need
to be extended to specific the supported platforms, for example
Plugin::isSupported
and default to return true.   jlink will need to filter the plugins when
it finds the plugin
providers.   You can update PluginRepository to do the filtering and see
if that covers
all cases (I suspect so).

Mandy

On 12/21/18 4:23 AM, Severin Gehwolf wrote:

>> I suspect that
>> jlink will need a bit of support for integrating platform specific
>> plugins into the build as tools such as objdump and options such as
>> --add-gun-debuglink aren't going to make sense everywhere.
> Yes. It whould make sense to include this plugin only for Linux and,
> perhaps, Solaris. If there was a way to include plugins based on the
> build platform this would help.
>
>>  From the
>> examples, I also wonder about unwieldily command lines usages and
>> whether having a platform specific plugin would make this a bit easier.
> Sure. One specific issue I've run into was that there seems to be no
> way to specify for a plugin to take >= 0 arguments. Once a plugin
> specifies, hasArgument() == true, it seems to require at least one.
>
> Thanks,
> Severin
>

Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Alan Bateman
On 21/12/2018 18:21, Mandy Chung wrote:

> I also like this, stripping the native debug symbols, as a jlink plugin.
>
> Currently jlink does not support platform-specific plugin.   The
> Plugin API will need
> to be extended to specific the supported platforms, for example
> Plugin::isSupported
> and default to return true.   jlink will need to filter the plugins
> when it finds the plugin
> providers.   You can update PluginRepository to do the filtering and
> see if that covers
> all cases (I suspect so).
It might be simpler to move the plugin to src/jdk.jlink/linux/classes
and create a module-info.java.extra to augment the module-info.java file
in the build. If that works then it would reduce the build work to
augmenting plugin.properties at build time with platform specific resources.

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: [Proposal] jlink plugin which replaces dynamic shared libraries

Severin Gehwolf
On Mon, 2018-12-31 at 07:33 +0000, Alan Bateman wrote:

> On 21/12/2018 18:21, Mandy Chung wrote:
> > I also like this, stripping the native debug symbols, as a jlink plugin.
> >
> > Currently jlink does not support platform-specific plugin.   The
> > Plugin API will need
> > to be extended to specific the supported platforms, for example
> > Plugin::isSupported
> > and default to return true.   jlink will need to filter the plugins
> > when it finds the plugin
> > providers.   You can update PluginRepository to do the filtering and
> > see if that covers
> > all cases (I suspect so).
>
> It might be simpler to move the plugin to src/jdk.jlink/linux/classes
> and create a module-info.java.extra to augment the module-info.java file
> in the build. If that works then it would reduce the build work to
> augmenting plugin.properties at build time with platform specific resources.

Thanks, Mandy and Alan for the input. I'll have a look and get back to
you.

Cheers,
Severin