Jigsaw and containers

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Jigsaw and containers

Daniel Latrémolière
I have a problem to understand new layout of files in JRE/JDK images for
the containers use-cases.

Containers (Docker, Flatpak) have a notion of single inheritance, where
an union FS merge the images of the container with parent containers.
This give much smaller images of containers when most of the files are
untouched and inherited from a common parent container.

The layout of JRE/JDK images, with one big file "/lib/modules" taking
most of the size, seems to be completely anti-modular and particularly
bad for fast redeployment. Isn't it more modular and space-efficient to
have not a file "/lib/modules" but a directory "/lib/modules/" where all
available files, like "0", "1", etc. would be loaded.

By example, an application suite can have a parent container with the
Java standard library and common libraries of the application suite
(creating a file "lib/modules/0" with the set of common modules). Each
application would be a container inheriting common code from parent and
with a file "lib/modules/1" with the own set of modules for the application.

Daniel.


Reply | Threaded
Open this post in threaded view
|

Re: Jigsaw and containers

Alan Bateman
On 31/08/2017 18:58, Daniel Latrémolière wrote:

> I have a problem to understand new layout of files in JRE/JDK images
> for the containers use-cases.
>
> Containers (Docker, Flatpak) have a notion of single inheritance,
> where an union FS merge the images of the container with parent
> containers. This give much smaller images of containers when most of
> the files are untouched and inherited from a common parent container.
>
> The layout of JRE/JDK images, with one big file "/lib/modules" taking
> most of the size, seems to be completely anti-modular and particularly
> bad for fast redeployment. Isn't it more modular and space-efficient
> to have not a file "/lib/modules" but a directory "/lib/modules/"
> where all available files, like "0", "1", etc. would be loaded.
>
> By example, an application suite can have a parent container with the
> Java standard library and common libraries of the application suite
> (creating a file "lib/modules/0" with the set of common modules). Each
> application would be a container inheriting common code from parent
> and with a file "lib/modules/1" with the own set of modules for the
> application.
>
Are you using `jlink` to create a run-time image for each application?
(just trying to establish if you are actually running into an issue or not).

-Alan

Reply | Threaded
Open this post in threaded view
|

Re: Jigsaw and containers

Daniel Latrémolière

> Are you using `jlink` to create a run-time image for each application?
> (just trying to establish if you are actually running into an issue or
> not).
With jlink per application, all will be working without problems, but
each application will have his own copy of JRE and common libraries (big
size).

With jlink only for jre and common libraries, these parts can be shared
between applications. But specific parts of each application will be
only in JAR files (and not be modularized).

These two choices works as reasonable targets, but no one is really
completely good (big size or no full modularization).

Daniel.
Reply | Threaded
Open this post in threaded view
|

Re: Jigsaw and containers

Remi Forax
Hi Daniel,
several points first,
the JRE doesn't exist anymore apart for compatibility (the jre is just a bunch of modules) and a jar can be a modular jar, just have a module-info.class in the jar, so applications that uses jars can be modular, they do not have to use jlink.

So you can have a jlink image as small as just java.base and all your applications and their dependencies as modular jars, the modules will be shared by the different applications, or a jlink all applications and have no sharing at all and all the states in between.

Rémi

----- Mail original -----
> De: "Daniel Latrémolière" <[hidden email]>
> À: "Alan Bateman" <[hidden email]>, "jigsaw-dev" <[hidden email]>
> Envoyé: Jeudi 31 Août 2017 20:47:35
> Objet: Re: Jigsaw and containers

>> Are you using `jlink` to create a run-time image for each application?
>> (just trying to establish if you are actually running into an issue or
>> not).
> With jlink per application, all will be working without problems, but
> each application will have his own copy of JRE and common libraries (big
> size).
>
> With jlink only for jre and common libraries, these parts can be shared
> between applications. But specific parts of each application will be
> only in JAR files (and not be modularized).
>
> These two choices works as reasonable targets, but no one is really
> completely good (big size or no full modularization).
>
> Daniel.
Reply | Threaded
Open this post in threaded view
|

Re: Jigsaw and containers

Alan Bateman
In reply to this post by Daniel Latrémolière
On 31/08/2017 19:47, Daniel Latrémolière wrote:
>
>> Are you using `jlink` to create a run-time image for each
>> application? (just trying to establish if you are actually running
>> into an issue or not).
> With jlink per application, all will be working without problems, but
> each application will have his own copy of JRE and common libraries
> (big size).
If you are using `jlink` then the resulting run-time image will only
contain the modules that you specify to the tool and the modules that
they recursively require. You should find that it's significantly
smaller that the JDK and JRE downloads.

>
> With jlink only for jre and common libraries, these parts can be
> shared between applications. But specific parts of each application
> will be only in JAR files (and not be modularized).
Modules can be packaged as JAR files too. You can even use most existing
JAR files are modules. It does forfeit some of the benefits of creating
your own run-time image but maybe that is the trade off that you are
looking for.

-Alan.
Reply | Threaded
Open this post in threaded view
|

Re: Jigsaw and containers

Daniel Latrémolière
In reply to this post by Remi Forax
Hi Rémi, Alan

I use already jlink to keep only useful (for me) parts of JRE/JDK, and avoid most big thinks (like Java UI, CORBA, RMI, etc).
Modular JAR are officially for migration [1] and are not used by modules from OpenJDK, even when they are pure Java. If they are a definitive target, why isn't it in OpenJDK a tool to transform a JMOD in a modular jar (at least when it is pure Java bytecode)?
As I understand jlink as an optimizer tool [2], I would try to use it for all modules and expect it to be able to do so. If I can not do it good, because they resulting image format is too much restricted, I will mitigate to have a less bad solution.

My problem is that nothing in Jigsaw requires technically to store all modules of an image in a big single file, but new image format has made this choice. Having multiple files, each containing a set of modules would allow much more efficient trade-offs. This is particularly true for containers, and probably also in pure application suite (by making links to a jlink-optimized file containing a shared subset of modules).
This is easy to be seen in containers, where the file system is usually the merge of multiples layers, like described in this image (from Docker [3]):
https://docs.docker.com/engine/userguide/storagedriver/images/overlay_constructs.jpg
Then having multiple files each with a specified set of modules allow some files to be shared between multiples containers. When we have all modules only in a single file, this file will never be the same between all containers and can never be shared. Then sharing common parts requires to convert non-shared modules in non-optimized modular JAR (blurring target of Jigsaw).

I understand that it is too late for Java 9 to allow multiple files for modules in jlink-build images, each file storing a subset of the modules of the image (but I think it would be better if the current single file is already named "modules/0" and not "modules"). I would hope that it is not a definitive limitation of Java.

Daniel.

[1]: http://openjdk.java.net/projects/jigsaw/doc/topics/modularjar.html
[2]: http://openjdk.java.net/jeps/282
[3]: https://docs.docker.com/engine/userguide/storagedriver/overlayfs-driver/

Le 31 août 2017 21:14:24 GMT+02:00, Remi Forax <[hidden email]> a écrit :

>Hi Daniel,
>several points first,
>the JRE doesn't exist anymore apart for compatibility (the jre is just
>a bunch of modules) and a jar can be a modular jar, just have a
>module-info.class in the jar, so applications that uses jars can be
>modular, they do not have to use jlink.
>
>So you can have a jlink image as small as just java.base and all your
>applications and their dependencies as modular jars, the modules will
>be shared by the different applications, or a jlink all applications
>and have no sharing at all and all the states in between.
>
>Rémi
>
>----------
>Modules can be packaged as JAR files too. You can even use most
>existing
>JAR files are modules. It does forfeit some of the benefits of creating
>your own run-time image but maybe that is the trade off that you are
>looking for.
>
>-Alan.
Reply | Threaded
Open this post in threaded view
|

Re: Jigsaw and containers

Dalibor Topic


On 31.08.2017 23:05, Daniel Latrémolière wrote:
> This is easy to be seen in containers, where the file system is usually the merge of multiples layers, like described in this image (from Docker [3]):
> https://docs.docker.com/engine/userguide/storagedriver/images/overlay_constructs.jpg
> Then having multiple files each with a specified set of modules allow some files to be shared between multiples containers. When we have all modules only in a single file, this file will never be the same between all containers and can never be shared. Then sharing common parts requires to convert non-shared modules in non-optimized modular JAR (blurring target of Jigsaw).

I believe that this behavior can differ from storage driver to storage
driver. OverlayFS works on the file level, while the storage drivers for
ZFS, btrfs, devicemapper etc. seem to work on the block level.

cheers,
dalibor topic

--
<http://www.oracle.com> Dalibor Topic | Principal Product Manager
Phone: +494089091214 <tel:+494089091214> | Mobile: +491737185961
<tel:+491737185961>

ORACLE Deutschland B.V. & Co. KG | Kühnehöfe 5 | 22761 Hamburg

ORACLE Deutschland B.V. & Co. KG
Hauptverwaltung: Riesstr. 25, D-80992 München
Registergericht: Amtsgericht München, HRA 95603

Komplementärin: ORACLE Deutschland Verwaltung B.V.
Hertogswetering 163/167, 3543 AS Utrecht, Niederlande
Handelsregister der Handelskammer Midden-Niederlande, Nr. 30143697
Geschäftsführer: Alexander van der Ven, Jan Schultheiss, Val Maher

<http://www.oracle.com/commitment> Oracle is committed to developing
practices and products that help protect the environment
Reply | Threaded
Open this post in threaded view
|

Re: Jigsaw and containers

Alan Bateman
In reply to this post by Daniel Latrémolière
On 31/08/2017 22:05, Daniel Latrémolière wrote:
> Hi Rémi, Alan
>
> I use already jlink to keep only useful (for me) parts of JRE/JDK, and avoid most big thinks (like Java UI, CORBA, RMI, etc).
> Modular JAR are officially for migration [1] and are not used by modules from OpenJDK, even when they are pure Java. If they are a definitive target, why isn't it in OpenJDK a tool to transform a JMOD in a modular jar (at least when it is pure Java bytecode)?
> As I understand jlink as an optimizer tool [2], I would try to use it for all modules and expect it to be able to do so. If I can not do it good, because they resulting image format is too much restricted, I will mitigate to have a less bad solution.
>
Many of the core modules in the JDK have native libraries, launchers,
configuration files, .. and so lend themselves to the JMOD format with
its defined sections for each type of resource. There are some modules
in the JDK that only contain class files and a valid alternative would
be for the OpenJDK build to package these as modular JARs. So don't read
anything into the JDK using the JMOD format, the bulk of modules in the
wider world will be packaged as modular JARs.

BTW:  The page you reference about modular JARs being "officially for
migration" is an obsolete page from the exploratory phase of Project
Jigsaw. I'm not sure how you got to that but it's a reminder that there
are old pages that needs to be labeled as obsolete.

As regards overlaying one run-time image over another then this brings
too many hazards and complications. If you want libraries of modules
(which I think is what this thread is really about) then put the modular
JARs in a directory. If you organize these into layers, base image, a
directory for common libraries, directories for domain specific
libraries, ... then they should work well when you union the file
systems. The resolution that is done at startup will ensure that there
aren't any missing modules and will catch other issues in the
configuration too.

-Alan