Scanning modules?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Scanning modules?

Greg Wilkins
All,
another question on scanning, but more jigsaw related than my other
question.

App containers have to scan jars deployed in EARs, WARs etc. for
annotations, fragments, resources etc.  However, many containers also offer
features to be able to optionally scan jars that are on the system
classpath.

For example, Jetty deploys the JSTL jar on the system classpath, but adds
it to the annotation scanning for every webapp deployed.  Similarly when
running in embedded mode, where all jars are on the system class path, we
have been able to find and scan those that matched a pattern.

However, with java9 modules, we have two problems if a jar like jstl.jar is
deployed on at runtime.

Firstly, the system loader is no longer a URLClassLoader, so we cannot scan
to list of provided jar files looking for potential matches of known jars
that we wish to scan.

We can work around that by loading a known class from the known jars and
asking for its location (via it's protection domain).   For a normal jar,
this gives us a file URI, which we can then use as the basis for a scan.

However, I expect that eventually JSTL will be converted to a module, and
then the location will just be something like jrt://apache.jstl.      Which
gives us our second problem - given a known or discovered module, how do we
discover what classes are within that module in order to scan them for
annotations?       We cannot find any API via Module nor the jrt URLHandler
that gives us access to a list of classes?

We can get a lists/sets of Packages,Exports etc. but nothing we can see
gives us the actual classes/resources contained in that module, nor the
location of the source jar file that we could open and thus determine the
contents ourselves.

So some guidance of how we are meant to do discovery of annotations,
fragments etc in a modularized deployment would be very useful.

regards


--
Greg Wilkins <[hidden email]> CTO http://webtide.com
Reply | Threaded
Open this post in threaded view
|

Re: Scanning modules?

Christian Stein
Hi Greg,

short disclaimer: I'm learning and testing the "--scan-module" magic at
the moment to implement a similar feature in JUnit 5. See [1] for details.
If I'm talking non-sense, please correct me jigsaw-devs.

To scan the contents of a module you'll need a ModuleReference[2] instance.
You may open() such a reference to obtain a ModuleReader[3] which allows
you to list() all or find() some of the contents of the referenced module.

How to get ModuleReference instances?

There're at least two ways: either ask the runtime to report all resolved
modules in a specific layer. See [4] for details and [5] how I try to obtain
all modules on the module-path.

Or you may build your own ModuleFinder [6] with well-known module locations.
Here [7] is an example how this could work.

Cheers and hope that helps,
Christian

[1] https://github.com/junit-team/junit5/pull/1057/files#diff-
96762765b7aa19e415e25de774366feb
[2] http://download.java.net/java/jdk9/docs/api/java/lang/
module/ModuleReference.html
[3] http://download.java.net/java/jdk9/docs/api/java/lang/
module/ModuleReader.html
[4] http://download.java.net/java/jdk9/docs/api/java/lang/ModuleLayer.html
[5]
https://github.com/junit-team/junit5/pull/1057/files#diff-6e80f389aa0b4148904a9c474b9e8fb8R43
[6] http://download.java.net/java/jdk9/docs/api/java/lang/
module/ModuleFinder.html
[7]
https://github.com/forax/pro/blob/master/src/main/java/com.github.forax.pro.helper/com/github/forax/pro/helper/ModuleHelper.java#L77

On Fri, Sep 15, 2017 at 6:01 AM, Greg Wilkins <[hidden email]> wrote:

> All,
> another question on scanning, but more jigsaw related than my other
> question.
>
> App containers have to scan jars deployed in EARs, WARs etc. for
> annotations, fragments, resources etc.  However, many containers also offer
> features to be able to optionally scan jars that are on the system
> classpath.
>
> For example, Jetty deploys the JSTL jar on the system classpath, but adds
> it to the annotation scanning for every webapp deployed.  Similarly when
> running in embedded mode, where all jars are on the system class path, we
> have been able to find and scan those that matched a pattern.
>
> However, with java9 modules, we have two problems if a jar like jstl.jar is
> deployed on at runtime.
>
> Firstly, the system loader is no longer a URLClassLoader, so we cannot scan
> to list of provided jar files looking for potential matches of known jars
> that we wish to scan.
>
> We can work around that by loading a known class from the known jars and
> asking for its location (via it's protection domain).   For a normal jar,
> this gives us a file URI, which we can then use as the basis for a scan.
>
> However, I expect that eventually JSTL will be converted to a module, and
> then the location will just be something like jrt://apache.jstl.      Which
> gives us our second problem - given a known or discovered module, how do we
> discover what classes are within that module in order to scan them for
> annotations?       We cannot find any API via Module nor the jrt URLHandler
> that gives us access to a list of classes?
>
> We can get a lists/sets of Packages,Exports etc. but nothing we can see
> gives us the actual classes/resources contained in that module, nor the
> location of the source jar file that we could open and thus determine the
> contents ourselves.
>
> So some guidance of how we are meant to do discovery of annotations,
> fragments etc in a modularized deployment would be very useful.
>
> regards
>
>
> --
> Greg Wilkins <[hidden email]> CTO http://webtide.com
>
Reply | Threaded
Open this post in threaded view
|

Re: Scanning modules?

Alan Bateman
In reply to this post by Greg Wilkins
On 15/09/2017 05:01, Greg Wilkins wrote:

> All,
> another question on scanning, but more jigsaw related than my other
> question.
>
> App containers have to scan jars deployed in EARs, WARs etc. for
> annotations, fragments, resources etc.  However, many containers also offer
> features to be able to optionally scan jars that are on the system
> classpath.
>
> For example, Jetty deploys the JSTL jar on the system classpath, but adds
> it to the annotation scanning for every webapp deployed.  Similarly when
> running in embedded mode, where all jars are on the system class path, we
> have been able to find and scan those that matched a pattern.
>
> However, with java9 modules, we have two problems if a jar like jstl.jar is
> deployed on at runtime.
>
> Firstly, the system loader is no longer a URLClassLoader, so we cannot scan
> to list of provided jar files looking for potential matches of known jars
> that we wish to scan.
>
> We can work around that by loading a known class from the known jars and
> asking for its location (via it's protection domain).   For a normal jar,
> this gives us a file URI, which we can then use as the basis for a scan.
>
> However, I expect that eventually JSTL will be converted to a module, and
> then the location will just be something like jrt://apache.jstl.      Which
> gives us our second problem - given a known or discovered module, how do we
> discover what classes are within that module in order to scan them for
> annotations?       We cannot find any API via Module nor the jrt URLHandler
> that gives us access to a list of classes?
>
> We can get a lists/sets of Packages,Exports etc. but nothing we can see
> gives us the actual classes/resources contained in that module, nor the
> location of the source jar file that we could open and thus determine the
> contents ourselves.
>
> So some guidance of how we are meant to do discovery of annotations,
> fragments etc in a modularized deployment would be very useful.
>
For the class path scanning then I assume you can use the
java.class.path property, the type of the application or system class
loader should not be interesting.

For the module path or other locations containing packaged modules then
you create a ModuleFinder and invoke its findAll method to get a module
reference to every module. Then you can use a ModuleReader to open the
module. ModuleReader defines `list` and other methods to access its
resources. My guess is this is where you will end up in that I suspect
you are selecting the candidate modules to resolve and these modules
will be instantiated in a child layer.

If you are really just looking to get at the contents of the modules in
the boot layer (the modules that are resolved at startup) then you'll do
something like this:

      ModuleLayer.boot().configuration().modules().stream()
             .map(ResolvedModule::reference)
             .forEach(mref -> {
                 System.out.println(mref.descriptor().name());
                 try (ModuleReader reader = mref.open()) {
                     reader.list().forEach(System.out::println);
                 } catch (IOException ioe) {
                     throw new UncheckedIOException(ioe);
                 }
             });

This code fragment processes the configuration for the boot layer,
opening up each module and listing the names of the resources in every
module.

You mention the jrt protocol handler. That is to support URL
connections. There is also a jrt file system provider for tools that
need to scan the contents of the current or other run-time image. I
suspect you won't need this for what you are doing.

-Alan.

Reply | Threaded
Open this post in threaded view
|

Re: Scanning modules?

Greg Wilkins
Alan & Christian,

thanks for that - I totally missed ResolvedModule.open().list()

So now a supplementary question... are ModuleReaders MR aware?  Ie if the
jar of the module is MR, will the resolved module only contain classes that
are appropriate for the current Runtime?

thanks in advance.

cheers


On 15 September 2017 at 17:15, Alan Bateman <[hidden email]> wrote:

> On 15/09/2017 05:01, Greg Wilkins wrote:
>
>> All,
>> another question on scanning, but more jigsaw related than my other
>> question.
>>
>> App containers have to scan jars deployed in EARs, WARs etc. for
>> annotations, fragments, resources etc.  However, many containers also
>> offer
>> features to be able to optionally scan jars that are on the system
>> classpath.
>>
>> For example, Jetty deploys the JSTL jar on the system classpath, but adds
>> it to the annotation scanning for every webapp deployed.  Similarly when
>> running in embedded mode, where all jars are on the system class path, we
>> have been able to find and scan those that matched a pattern.
>>
>> However, with java9 modules, we have two problems if a jar like jstl.jar
>> is
>> deployed on at runtime.
>>
>> Firstly, the system loader is no longer a URLClassLoader, so we cannot
>> scan
>> to list of provided jar files looking for potential matches of known jars
>> that we wish to scan.
>>
>> We can work around that by loading a known class from the known jars and
>> asking for its location (via it's protection domain).   For a normal jar,
>> this gives us a file URI, which we can then use as the basis for a scan.
>>
>> However, I expect that eventually JSTL will be converted to a module, and
>> then the location will just be something like jrt://apache.jstl.
>> Which
>> gives us our second problem - given a known or discovered module, how do
>> we
>> discover what classes are within that module in order to scan them for
>> annotations?       We cannot find any API via Module nor the jrt
>> URLHandler
>> that gives us access to a list of classes?
>>
>> We can get a lists/sets of Packages,Exports etc. but nothing we can see
>> gives us the actual classes/resources contained in that module, nor the
>> location of the source jar file that we could open and thus determine the
>> contents ourselves.
>>
>> So some guidance of how we are meant to do discovery of annotations,
>> fragments etc in a modularized deployment would be very useful.
>>
>> For the class path scanning then I assume you can use the java.class.path
> property, the type of the application or system class loader should not be
> interesting.
>
> For the module path or other locations containing packaged modules then
> you create a ModuleFinder and invoke its findAll method to get a module
> reference to every module. Then you can use a ModuleReader to open the
> module. ModuleReader defines `list` and other methods to access its
> resources. My guess is this is where you will end up in that I suspect you
> are selecting the candidate modules to resolve and these modules will be
> instantiated in a child layer.
>
> If you are really just looking to get at the contents of the modules in
> the boot layer (the modules that are resolved at startup) then you'll do
> something like this:
>
>      ModuleLayer.boot().configuration().modules().stream()
>             .map(ResolvedModule::reference)
>             .forEach(mref -> {
>                 System.out.println(mref.descriptor().name());
>                 try (ModuleReader reader = mref.open()) {
>                     reader.list().forEach(System.out::println);
>                 } catch (IOException ioe) {
>                     throw new UncheckedIOException(ioe);
>                 }
>             });
>
> This code fragment processes the configuration for the boot layer, opening
> up each module and listing the names of the resources in every module.
>
> You mention the jrt protocol handler. That is to support URL connections.
> There is also a jrt file system provider for tools that need to scan the
> contents of the current or other run-time image. I suspect you won't need
> this for what you are doing.
>
> -Alan.
>
>


--
Greg Wilkins <[hidden email]> CTO http://webtide.com
Reply | Threaded
Open this post in threaded view
|

Re: Scanning modules?

Alan Bateman
On 15/09/2017 08:40, Greg Wilkins wrote:
> Alan & Christian,
>
> thanks for that - I totally missed ResolvedModule.open().list()
>
> So now a supplementary question... are ModuleReaders MR aware?  Ie if
> the jar of the module is MR, will the resolved module only contain
> classes that are appropriate for the current Runtime?
>
Yes.