Loading an automatic module from an exploded directory

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Loading an automatic module from an exploded directory

Eirik Bjørsnøs
The current implementation of automatic modules seems to assume that an
automatic module is always packaged as a jar file.

I'm working on a module runtime where this is not always the case, and the
limitation has become a bit of a challenge.

I want to package applications (modules + runtime) at build time into a
single jar for distribution.

The runtime loads modules from directories within its own jar.  This means
that ModuleFinder.of(Path..) receives module locations in the form:

jar:file:///path/to/single.jar!/META-INF/modules/module-1.0/

This works fine as long as modules are explicit. (With the unrelated
limitation that the multi-release feature also seem to assume jar files)

Loading automatic modules this way is not as easy.

First, it requires duplicating non-trivial amounts of private, slightly
intricate code from ModulePath.deriveModuleDescriptor and friends.

Second, since deriveModuleDescriptor assumes jars on disk, it uses JarFile
scanning to find all packages to add to the automatic module. So the
scanning code must be rewritten to scan for class files (and
META-INF/services/ files) within the sub-directory in the main jar instead.

I have also identify an additional use case which is to allow hot-deploying
automatic modules during development from target/classes using a Maven
plugin.

These use cases are probably on the fringes of what the module system was
designed to support, but it did made me wonder if some small changes maybe
could make this easier.

Availability of some public API to create ModuleFinder for an automatic
module loaded in non-standard ways would be helpful.

The 'non-standardness' I have been able to identity in my case is:

1: The fallback name for the automatic module. In my case, there's no jar
file name to investigate, so instead I use the Maven artifactId directly
(after cleaning it)

2: The set of paths to use as input for identifying packages in the module
and the set of service files to use as input for defining 'provides'
attributes.

Here's a strawman API which I think would support my use case :

ModuleFinder auto = ModuleFinder.automatic(Path location, String
defaultName, Set<String> packages, Set<String> serviceNames)

Any feedback, ideas or similar experiences would be appreciated.

Cheers,
Eirik.
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Alan Bateman
On 09/04/2020 16:42, Eirik Bjørsnøs wrote:

> The current implementation of automatic modules seems to assume that an
> automatic module is always packaged as a jar file.
>
> I'm working on a module runtime where this is not always the case, and the
> limitation has become a bit of a challenge.
>
> I want to package applications (modules + runtime) at build time into a
> single jar for distribution.
>
> The runtime loads modules from directories within its own jar.  This means
> that ModuleFinder.of(Path..) receives module locations in the form:
>
> jar:file:///path/to/single.jar!/META-INF/modules/module-1.0/
>
> This works fine as long as modules are explicit. (With the unrelated
> limitation that the multi-release feature also seem to assume jar files)
The only packaging format for automatic modules that Java SE defines is
JAR files. The "Multi-release JAR files" feature is also JAR file only.

If I read your mail correctly, you are creating "multi-module JAR files"
where the modules are "exploded" under /META-INF/modules in
${NAME}-${VERSION} directories. It shouldn't be too hard to create your
own ModuleFinder that finds modules under META-INF/modules. This would
mean implementing ModuleFinder rather trying to use
ModuleFinder.of(Path...). I assume you've found ModuleDescriptor.read to
read/parse the module-info.class of explicit modules. You are right that
it would require code to scan directory trees, at least the equivalent
of automatic modules, maybe for explicit modules too. However, it
shouldn't be too hard. Have you tried the zip file system provider? That
would allow you to open the JAR file as a file system so you can use the
file system API.

> :
>
> I have also identify an additional use case which is to allow hot-deploying
> automatic modules during development from target/classes using a Maven
> plugin.
>
I'm not sure how to interpret this but just to say that the unit of
replacement is the module layer, you can't replace modules in a layer
and/or dynamically change the set of packages in a loaded module.

-Alan

Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Eirik Bjørsnøs
Alan,


> If I read your mail correctly, you are creating "multi-module JAR files"
> where the modules are "exploded" under /META-INF/modules in
> ${NAME}-${VERSION} directories.


Correct.


> It shouldn't be too hard to create your
> own ModuleFinder that finds modules under META-INF/modules. This would
> mean implementing ModuleFinder rather trying to use
> ModuleFinder.of(Path...).


This is what my implementation currently does.

To achieve this, I had to copy / replicate quite a bit of code from the
ModulePath class which is not accessible outside java.base.

It wasn't all that hard, I just wasn't happy with the amount of classes /
methods I had to copy from java.base only to slightly modify them

This includes:

Code to create / build the ModuleDescriptor (stolen from
deriveModuleDescriptor)
Code to clean the artifactId used to produce the automatic name (Stolen
from cleanModuleName, Patterns and Checks)
Duplication of ExplodedModuleReader which also dragged in Resources.

I assume you've found ModuleDescriptor.read to
> read/parse the module-info.class of explicit modules.


I don't actually need to read the explicit modules. I just check that they
have module-info.class and use let ModuleFinder.of do the rest of the job.


> You are right that
> it would require code to scan directory trees, at least the equivalent
> of automatic modules, maybe for explicit modules too. However, it
> shouldn't be too hard. Have you tried the zip file system provider? That
> would allow you to open the JAR file as a file system so you can use the
> file system API.
>
>
Yes, I'm opening a ZIP filesystem on the single-jar and using the file
system API to navigate it.


> > I have also identify an additional use case which is to allow
> hot-deploying
> > automatic modules during development from target/classes using a Maven
> > plugin.
> >
> I'm not sure how to interpret this but just to say that the unit of
> replacement is the module layer, you can't replace modules in a layer
> and/or dynamically change the set of packages in a loaded module.
>

The runtime loads a graph of module layers, each containing one or more
modules. Layers may depend on other layers, forming a DAG with
corresponding class loader delegation.

When the runtime detects that a module needs to be redeployed, the
transitive closure of depending layers is removed from the graph and new
module layers are created and added to the graph.

Services are bound using dependency injection, so there's also a DAG of
service dependencies. This allows restarting services which are affected by
the redeploy, either because they are provided from updated modules or
because they consume services from updated modules.

So not really hot-deploying in the JVM sense, but still pretty fast and
developer-friendly.

My demo has a module which provides a JAX-RS resource which is consumed by
a Jersey module which provides an HttpServlet which is again consumed by a
Jetty module which deploys the servlet in a servlet context.

Redeploying the module containing the JAX-RS resource takes something like
50ms IIRC.

Cheers,
Eirik.
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Eirik Bjørsnøs
In reply to this post by Alan Bateman
On Thu, Apr 9, 2020 at 7:37 PM Alan Bateman <[hidden email]> wrote:

The only packaging format for automatic modules that Java SE defines is
> JAR files. The "Multi-release JAR files" feature is also JAR file only.
>

My ModuleFinder for exploded-within-jar automatic modules needed a
ModuleReader implementation.

I decided to make that ModuleReader multi-release-aware. Like JarFile, it
inspects the manifest for the Multi-Release attribute and scans
META-INF/version directories accordingly.

Is it safe to assume that all access to content with a module happens
through the ModuleReader?

Eirik.
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Eirik Bjørsnøs
In reply to this post by Eirik Bjørsnøs
On Thu, Apr 9, 2020 at 5:42 PM Eirik Bjørsnøs <[hidden email]> wrote:

Here's a strawman API which I think would support my use case :
>
> ModuleFinder auto = ModuleFinder.automatic(Path location, String
> defaultName, Set<String> packages, Set<String> serviceNames)
>

Another thought...

Since ModuleReader already has the list() method, code in java.lang.module
could use that to scan for packages and service names it needs to derive
automatic ModuleDescriptors.

Clients would now only need to pass the custom module reader and a default
automatic name:

ModuleFinder moduleFinder = ModuleFinder.of(ModuleReader moduleReader,
String defaultAutomaticModuleName);

I would still have to create my custom module reader, but I would no longer
need to know or care about the details of how automatic modules work.

Something to consider?

Cheers,
Eirik.
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Alan Bateman
In reply to this post by Eirik Bjørsnøs
On 09/04/2020 19:23, Eirik Bjørsnøs wrote:

>
> Alan,
>
>     If I read your mail correctly, you are creating "multi-module JAR
>     files"
>     where the modules are "exploded" under /META-INF/modules in
>     ${NAME}-${VERSION} directories.
>
>
> Correct.
Do they need to be "exploded"? If the dependences are automatic modules
then I assume putting the JAR files (without unpacking) into
META-INF/modules should just work without needing a new ModuleFinder
implementation or implementing the exploded equivalent of multi-module
JAR files (or modular multi-release JAR files).

(The reason it should work is that the built-in implementation can work
with paths to locations in virtual file systems. It handles this by
copying the JAR files from the virtual file system to the default file
system so that it can be opened with the JarFile API. It does this in
order to support signed JARs as the zip file system provider doesn't
have any support for validating signatures. It's possible this approach
isn't very efficient, also we haven't looked into interactions with the
cron jobs that clean tmpfs so there may be a bit more work needed here).

>
> To achieve this, I had to copy / replicate quite a bit of code from
> the ModulePath class which is not accessible outside java.base.
>
> It wasn't all that hard, I just wasn't happy with the amount of
> classes / methods I had to copy from java.base only to slightly modify
> them
>
> This includes:
>
> Code to create / build the ModuleDescriptor (stolen from
> deriveModuleDescriptor)
> Code to clean the artifactId used to produce the automatic name
> (Stolen from cleanModuleName, Patterns and Checks)
> Duplication of ExplodedModuleReader which also dragged in Resources.
Understood although you probably don't need a deep copy of everything as
there is a lot in exploded module reader related to potential security
and traversal on the default file system that is less interesting in JAR
files.


> :
>
> The runtime loads a graph of module layers, each containing one or
> more modules. Layers may depend on other layers, forming a DAG with
> corresponding class loader delegation.
>
> When the runtime detects that a module needs to be redeployed, the
> transitive closure of depending layers is removed from the graph and
> new module layers are created and added to the graph.
>
> Services are bound using dependency injection, so there's also a DAG
> of service dependencies. This allows restarting services which are
> affected by the redeploy, either because they are provided from
> updated modules or because they consume services from updated modules.
>
> So not really hot-deploying in the JVM sense, but still pretty fast
> and developer-friendly.
>
> My demo has a module which provides a JAX-RS resource which is
> consumed by a Jersey module which provides an HttpServlet which is
> again consumed by a Jetty module which deploys the servlet in a
> servlet context.
>
> Redeploying the module containing the JAX-RS resource takes something
> like 50ms IIRC.
>
This sounds very interesting. I'm sure there are several people here
that would be interesting in seeing a write-up or demo of this. I'm
curious if the service wiring make use of the existing uses/provides or
something else. I'm also curious if you've run into any issues with
multi-parent configurations as that is an area that might need attention
some day.

-Alan



Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Alan Bateman
In reply to this post by Eirik Bjørsnøs
On 10/04/2020 10:53, Eirik Bjørsnøs wrote:
> :
>
> Is it safe to assume that all access to content with a module happens
> through the ModuleReader?
>
Yes, the ModuleReader is the only way to access the contents of a module
so you'll see all class loading, finding resources, etc. all go through
the reader. We can't stop nosey code opening the packaged modules
directly of course but none of the standard APIs bypass the module reader.

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Alan Bateman
In reply to this post by Eirik Bjørsnøs
On 10/04/2020 11:14, Eirik Bjørsnøs wrote:

> :
> Since ModuleReader already has the list() method, code in java.lang.module
> could use that to scan for packages and service names it needs to derive
> automatic ModuleDescriptors.
>
> Clients would now only need to pass the custom module reader and a default
> automatic name:
>
> ModuleFinder moduleFinder = ModuleFinder.of(ModuleReader moduleReader,
> String defaultAutomaticModuleName);
>
> I would still have to create my custom module reader, but I would no longer
> need to know or care about the details of how automatic modules work.
>
> Something to consider?
>
This would require the API specify how automatic modules are created
from the contents of "something" that already have a ModuleReader.
Specification could be crafted of course but I have concerns that it
goes beyond the original intention of automatic modules. That is, the
original intention was to allow yesterday's JAR files be used as today
modules. More sophisticated applications have the APIs to create
automatic modules for other cases where needed. So I think it needs a
bit more thought on whether additional APIs should be exposed.

-Alan.


Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Eirik Bjørsnøs
In reply to this post by Alan Bateman
>
> Do they need to be "exploded"? If the dependences are automatic modules
> then I assume putting the JAR files (without unpacking) into
> META-INF/modules should just work without needing a new ModuleFinder
> implementation or implementing the exploded equivalent of multi-module JAR
> files (or modular multi-release JAR files).
>

I once had the pleasure of debugging a system which after running in
production for a few days would suddenly start returning 404 errors.

Turned out someone (me) had forgotten to specify where Jetty should unpack
its WAR files. So they ended up on shared temp space where a weekly Centos
cron job cleaned them up.

After this experience I pledged to never unpack code to temp again :-)

Also, the performance implications seems not very appealing.


> It does this in order to support signed JARs as the zip file system
> provider doesn't have any support for validating signatures.
>

Hmm.. I guess exploded jars won't support signature verification.

Too bad that the implementation is so tightly coupled to jar files. It
could instead have defined behaviour in terms of abstract paths within a
package, in which signatures would 'just work'. Similar situation for
automatic modules. That could also have been specified in terms of abstract
paths without mandating a specific packaging format.


> Understood although you probably don't need a deep copy of everything as
> there is a lot in exploded module reader related to potential security and
> traversal on the default file system that is less interesting in JAR files.
>

Thanks, this code did seem security sensitive, so I conservatively did a
deep copy. I have pruned some of the code now.

Redeploying the module containing the JAX-RS resource takes something like
> 50ms IIRC.
>
> This sounds very interesting. I'm sure there are several people here that
> would be interesting in seeing a write-up or demo of this. I
>

I'll see if I can get a demo together at some point.


> I'm curious if the service wiring make use of the existing uses/provides
> or something else.
>

Modules make use of provides to expose binding classes to the runtime.
Binding classes contain methods which consume and produce services, thereby
forming a dependency graph.


> I'm also curious if you've run into any issues with multi-parent
> configurations as that is an area that might need attention some day.
>

The Jersey layer in the demo depends on both the servlet layer and the
jax.rs layers. However, these two layers do not share any modules, so I
didn't get into trouble.

One could imagine a layer depending on two parent layers who define
separate "instances" of the same module, which I think JPMS would flag as
an invalid configuration.

My Maven plugin already excludes modules from a layer when the dependency
is already visible up the ancestor chain, so it should be straightforward
to add a check for detecting same-module-in-multiple-parents and flag that
to the user at build time.

Eirik.
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Eirik Bjørsnøs
In reply to this post by Alan Bateman
On Fri, Apr 10, 2020 at 5:00 PM Alan Bateman <[hidden email]>
wrote:

>
> This would require the API specify how automatic modules are created
> from the contents of "something" that already have a ModuleReader.
> Specification could be crafted of course but I have concerns that it
> goes beyond the original intention of automatic modules. That is, the
> original intention was to allow yesterday's JAR files be used as today
> modules. More sophisticated applications have the APIs to create
> automatic modules for other cases where needed. So I think it needs a
> bit more thought on whether additional APIs should be exposed.
>

Fair enough. The use case and discussion is noted here for posterity if
similar needs should appear in the future.

Thanks for your thought and input, Alan.

Eirik.
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Alan Bateman
In reply to this post by Eirik Bjørsnøs
On 10/04/2020 17:48, Eirik Bjørsnøs wrote:

>
> I once had the pleasure of debugging a system which after running in
> production for a few days would suddenly start returning 404 errors.
>
> Turned out someone (me) had forgotten to specify where Jetty should
> unpack its WAR files. So they ended up on shared temp space where a
> weekly Centos cron job cleaned them up.
>
> After this experience I pledged to never unpack code to temp again :-)
>
> Also, the performance implications seems not very appealing.
If MultiModuleJARs [1] is re-visited then it would need to explore this
topic a bit more. For now, the only case where the JDK implementation
unpacks is when ModuleFinder.of is created with a file path to JAR file
in a custom file system or to a directory in a custom file system that
contains JAR files. These JAR files might be signed so this is why
JarFile (with the JAR verifier support) is used. It would be possible to
do the verification without support for the JarFile infrastructure, it
just wasn't a priority in JDK 9.

-Alan

[1] https://openjdk.java.net/projects/jigsaw/spec/issues/#MultiModuleJARs

Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Eirik Bjørsnøs
Alan

On Sat, Apr 18, 2020 at 12:24 PM Alan Bateman <[hidden email]>
wrote:

> If MultiModuleJARs [1] is re-visited then it would need to explore this
> topic a bit more.
>

Could we update ModulePath could to detect "automatic module directories"?.
Any directory path name ending with ".jar" could be loaded as an automatic
directory module, with the same semantics as if the directory was zipped
into a JAR.

This would require a refactoring of deriveModuleDescriptor to allow it
being called both from readJar and readExplodedModule.
Then readExplodedModule would need an update to detect the lack of
module-info.class, and call deriveModuleDescriptor just like readJar does.
scan(Path) would need an update to make it skip directory scanning on
directories ending with ".jar".

As far as I can tell, this would be (or allow) an implementation of
MultiModuleJARs.

Here's a patch which implements this:

diff --git
a/src/java.base/share/classes/jdk/internal/module/ModulePath.java
b/src/java.base/share/classes/jdk/internal/module/ModulePath.java
index 9971387258..5922858faf 100644
--- a/src/java.base/share/classes/jdk/internal/module/ModulePath.java
+++ b/src/java.base/share/classes/jdk/internal/module/ModulePath.java
@@ -51,6 +51,7 @@ import java.util.Map;
 import java.util.Objects;
 import java.util.Optional;
 import java.util.Set;
+import java.util.function.Function;
 import java.util.jar.Attributes;
 import java.util.jar.JarEntry;
 import java.util.jar.JarFile;
@@ -225,7 +226,7 @@ public class ModulePath implements ModuleFinder {

         try {

-            if (attrs.isDirectory()) {
+            if (attrs.isDirectory() &&
!entry.getFileName().toString().endsWith(".jar")) {
                 Path mi = entry.resolve(MODULE_INFO);
                 if (!Files.exists(mi)) {
                     // assume a directory of modules
@@ -449,18 +450,17 @@ public class ModulePath implements ModuleFinder {
      *
      * 1. The value of the Automatic-Module-Name attribute is the module
name
      * 2. The version, and the module name when the  Automatic-Module-Name
-     *    attribute is not present, is derived from the file ame of the
JAR file
+     *    attribute is not present, is derived from the file name of the
JAR file
      * 3. All packages are derived from the .class files in the JAR file
      * 4. The contents of any META-INF/services configuration files are
mapped
      *    to "provides" declarations
      * 5. The Main-Class attribute in the main attributes of the JAR
manifest
      *    is mapped to the module descriptor mainClass if possible
      */
-    private ModuleDescriptor deriveModuleDescriptor(JarFile jf)
+    private ModuleDescriptor deriveModuleDescriptor(Manifest man, String
fn, Set<String> entryNames, Function<String, InputStream>
serviceConfigurationFile)
         throws IOException
     {
         // Read Automatic-Module-Name attribute if present
-        Manifest man = jf.getManifest();
         Attributes attrs = null;
         String moduleName = null;
         if (man != null) {
@@ -470,8 +470,7 @@ public class ModulePath implements ModuleFinder {
             }
         }

-        // Derive the version, and the module name if needed, from JAR
file name
-        String fn = jf.getName();
+        // Derive the version, and the module name if needed, from JAR
file (or .jar directory) name
         int i = fn.lastIndexOf(File.separator);
         if (i != -1)
             fn = fn.substring(i + 1);
@@ -512,10 +511,8 @@ public class ModulePath implements ModuleFinder {
         if (vs != null)
             builder.version(vs);

-        // scan the names of the entries in the JAR file
-        Map<Boolean, Set<String>> map = jf.versionedStream()
-                .filter(e -> !e.isDirectory())
-                .map(JarEntry::getName)
+        // scan the names of the entries in the JAR file or .jar directory
+        Map<Boolean, Set<String>> map = entryNames.stream()
                 .filter(e -> (e.endsWith(".class") ^
e.startsWith(SERVICES_PREFIX)))
                 .collect(Collectors.partitioningBy(e ->
e.startsWith(SERVICES_PREFIX),
                                                    Collectors.toSet()));
@@ -541,9 +538,8 @@ public class ModulePath implements ModuleFinder {

         // parse each service configuration file
         for (String sn : serviceNames) {
-            JarEntry entry = jf.getJarEntry(SERVICES_PREFIX + sn);
             List<String> providerClasses = new ArrayList<>();
-            try (InputStream in = jf.getInputStream(entry)) {
+            try (InputStream in =
serviceConfigurationFile.apply(SERVICES_PREFIX +sn)) {
                 BufferedReader reader
                     = new BufferedReader(new InputStreamReader(in,
UTF_8.INSTANCE));
                 String cn;
@@ -641,7 +637,18 @@ public class ModulePath implements ModuleFinder {

                 // no module-info.class so treat it as automatic module
                 try {
-                    ModuleDescriptor md = deriveModuleDescriptor(jf);
+                    Set<String> entryNames = jf.versionedStream()
+                            .filter(e -> !e.isDirectory())
+                            .map(JarEntry::getName)
+                            .collect(Collectors.toSet());
+
+                    Function<String, InputStream> serviceCatalogProvider =
(sn) -> getInputStream(jf, sn);
+
+                    ModuleDescriptor md =
deriveModuleDescriptor(jf.getManifest(),
+                            jf.getName(),
+                            entryNames,
+                            serviceCatalogProvider);
+
                     attrs = new ModuleInfo.Attributes(md, null, null,
null);
                 } catch (RuntimeException e) {
                     throw new FindException("Unable to derive module
descriptor for "
@@ -659,6 +666,22 @@ public class ModulePath implements ModuleFinder {
         }
     }

+    private InputStream getInputStream(JarFile jf, String sn){
+        try {
+            return jf.getInputStream(jf.getJarEntry(sn));
+        } catch (IOException e) {
+            throw new UncheckedIOException(e);
+        }
+    }
+
+    private InputStream getInputStream(Path dir, String sn){
+        try {
+            return Files.newInputStream(dir.resolve(sn));
+        } catch (IOException e) {
+            throw new UncheckedIOException(e);
+        }
+    }
+

     // -- exploded directories --

@@ -666,7 +689,7 @@ public class ModulePath implements ModuleFinder {
         try {
             return Files.find(dir, Integer.MAX_VALUE,
                     ((path, attrs) -> attrs.isRegularFile() &&
!isHidden(path)))
-                    .map(path -> dir.relativize(path))
+                    .map(dir::relativize)
                     .map(this::toPackageName)
                     .flatMap(Optional::stream)
                     .collect(Collectors.toSet());
@@ -675,6 +698,18 @@ public class ModulePath implements ModuleFinder {
         }
     }

+    private Set<String> explodedPaths(Path dir) {
+        try {
+            return Files.find(dir, Integer.MAX_VALUE,
+                    ((path, attrs) -> attrs.isRegularFile() &&
!isHidden(path)))
+                    .map(dir::relativize)
+                    .map(Path::toString)
+                    .collect(Collectors.toSet());
+        } catch (IOException x) {
+            throw new UncheckedIOException(x);
+        }
+    }
+
     /**
      * Returns a {@code ModuleReference} to an exploded module on the file
      * system or {@code null} if {@code module-info.class} not found.
@@ -685,12 +720,32 @@ public class ModulePath implements ModuleFinder {
     private ModuleReference readExplodedModule(Path dir) throws
IOException {
         Path mi = dir.resolve(MODULE_INFO);
         ModuleInfo.Attributes attrs;
-        try (InputStream in = Files.newInputStream(mi)) {
-            attrs = ModuleInfo.read(new BufferedInputStream(in),
-                                    () -> explodedPackages(dir));
-        } catch (NoSuchFileException e) {
-            // for now
-            return null;
+        if(!Files.exists(mi)) {
+            Manifest man = null;
+            Path mf = dir.resolve("META-INF/MANIFEST.MF");
+            if(Files.exists(mf)) {
+                try(InputStream in = Files.newInputStream(mf)) {
+                    man = new Manifest(in);
+                }
+            }
+            Set<String> entryNameSupplier = explodedPaths(dir);
+            Function<String, InputStream> serviceCatalogProvider = (sn) ->
getInputStream(dir, sn);
+
+            ModuleDescriptor md = deriveModuleDescriptor(man,
+                    dir.getFileName().toString(),
+                    entryNameSupplier,
+                    serviceCatalogProvider);
+
+            attrs = new ModuleInfo.Attributes(md, null, null, null);
+
+        } else {
+            try (InputStream in = Files.newInputStream(mi)) {
+                attrs = ModuleInfo.read(new BufferedInputStream(in),
+                        () -> explodedPackages(dir));
+            } catch (NoSuchFileException e) {
+                // for now
+                return null;
+            }
         }
         return ModuleReferences.newExplodedModule(attrs, patcher, dir);
     }
Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Alan Bateman


On 18/04/2020 23:04, Eirik Bjørsnøs wrote:

> :
>
> Could we update ModulePath could to detect "automatic module
> directories"?. Any directory path name ending with ".jar" could be
> loaded as an automatic directory module, with the same semantics as if
> the directory was zipped into a JAR.
>
> This would require a refactoring of deriveModuleDescriptor to allow it
> being called both from readJar and readExplodedModule.
> Then readExplodedModule would need an update to detect the lack of
> module-info.class, and call deriveModuleDescriptor just like readJar
> does. scan(Path) would need an update to make it skip directory
> scanning on directories ending with ".jar".
>
> As far as I can tell, this would be (or allow) an implementation of
> MultiModuleJARs.
>
Directories with a ".jar" suffix is clever but I don't think it gets
away from the substantive issue that Java SE only specifies how
automatic modules come into being when they are packaged as JAR files. I
could also imagine tools and libraries that scan tripping up on
directories named ".jar". There are also many libraries that peek into
the JAR manifest with the java.util.jar APIs so they will be surprised too.

Just on MultiModuleJARs (and its buddy #MultiModuleExecutableJARs). This
would probably be significant enough to need a JEP. It would need to
explore whether the modules in the JAR are observable when the JAR file
is on the module path, this amounts to deciding if they are resolved and
loaded into the boot layer or child later. If observable on the module 
path then it mean javac would need to support it. Automatic modules
would bring many discussion points to the table. One of their important
roles of automatic modules is to bridge to the class path so what does
not mean when everything is packaged into the same JAR file, does the
JAR file have a directory with the "class path" too? There are
implications for jlink and several other tools (jar) aswell. So in
summary, and at least in my view, there is a lot more exploration needed
before doing just implementation changes to support "exploded automatic
modules" in JAR files.

-Alan

Reply | Threaded
Open this post in threaded view
|

Re: Loading an automatic module from an exploded directory

Eirik Bjørsnøs
>
> Directories with a ".jar" suffix is clever but I don't think it gets away
> from the substantive issue that Java SE only specifies how automatic
> modules come into being when they are packaged as JAR files. I could also
> imagine tools and libraries that scan tripping up on directories named
> ".jar". There are also many libraries that peek into the JAR manifest with
> the java.util.jar APIs so they will be surprised too.
>

I'm sure we could agree on some other convention to stop consumers from
tripping on ".jar" in a directory name. (They should check
ZipEntry.isDirectory, anyways)

Just on MultiModuleJARs (and its buddy #MultiModuleExecutableJARs). This

> would probably be significant enough to need a JEP. It would need to
> explore whether the modules in the JAR are observable when the JAR file is
> on the module path, this amounts to deciding if they are resolved and
> loaded into the boot layer or child later. If observable on the module
> path then it mean javac would need to support it. Automatic modules would
> bring many discussion points to the table. One of their important roles of
> automatic modules is to bridge to the class path so what does not mean when
> everything is packaged into the same JAR file, does the JAR file have a
> directory with the "class path" too? There are implications for jlink and
> several other tools (jar) aswell. So in summary, and at least in my view,
> there is a lot more exploration needed before doing just implementation
> changes to support "exploded automatic modules" in JAR files.
>

Ok, so #MultiModuleJARs obviously has a much broader scope than what I
originally thought. I'm not suggesting that the semantics of this JAR file
I produce should change. The JDK should remain blissfully unaware that I
packed modules somewhere deep into /META-INF/. I don't expect javac, jlink,
the java command line (or java -jar) to find these modules, nor know about
their existence. Loading these bundled modules would require rolling your
own custom main method using ModuleLayer and friends to set this up. I'm
not suggesting we standardize how this loading should happen.

All (I think) I'm suggesting is a local fix in ModulePath to allow reading
automatic modules from ZIP filesystem paths given that some specified
convention signals that this path represents an unpacked dusty jar. This
convention would need to be documented (in ModulePath?), but that should
represent a small javadoc change. The semantics of what an automatic module
is would not change.

ModulePath already supports reading proper modules from such ZIP paths, so
one can claim that not supporting automatic modules represents a bit of an
asymmetry.

I think this change unlocks potentially useful new distribution models for
modules, and if #MultiModuleJARs was to be revisited, something like this
would probably be required as a building block anyway.

Cheers,
Eirik.