Module system and services directory in META-INF

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Module system and services directory in META-INF

Jochen Theodorou
hi all,

Do all files in META-INF/services now have to be well formed according
to the SPI strucutres in Java, or is it still valid to have other files
in there as well? Before the module system it was no problem to have in
there service decriptors, that are services, but do not use the
ServiceProvider structure. Nothing was disturbed as long as nobody tried
to load it that way. Now this seems to be different and I am wondering
where this got specified.

Because the spec does not try to define file layouts so much I have the
feeling that this is not really specified by the java module system. Was
it then a decision by some maven plugin writers or maybe even the javac
guys?

We have not yet decided if we want to go with the service provider
mechanism, because I have doubts, that unloading of service classes and
class loaders associated with those classes will still unload if we use
a structure as global as the service provider structure. I actually even
fail to imagine a structure in which you can say, that at this point you
do no longer require the services in a general and global service
provider situation - unless there is something like an "unload". In
other words... if you load a module dynamically, can you let it define
services and can those services then be unloaded again?

bye Jochen
Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

Jochen Theodorou
On 26.12.2017 19:30, Jochen Theodorou wrote:

> hi all,
>
> Do all files in META-INF/services now have to be well formed according
> to the SPI strucutres in Java, or is it still valid to have other files
> in there as well? Before the module system it was no problem to have in
> there service decriptors, that are services, but do not use the
> ServiceProvider structure. Nothing was disturbed as long as nobody tried
> to load it that way. Now this seems to be different and I am wondering
> where this got specified.
>
> Because the spec does not try to define file layouts so much I have the
> feeling that this is not really specified by the java module system. Was
> it then a decision by some maven plugin writers or maybe even the javac
> guys?

I found
https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Service_Provider 

Is there no Java 9 version? Also the spec is not specific enough for
this as it sounds more like implicit assumption than specification

bye Jochen

Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

David Holmes
On 27/12/2017 5:43 AM, Jochen Theodorou wrote:

> On 26.12.2017 19:30, Jochen Theodorou wrote:
>> hi all,
>>
>> Do all files in META-INF/services now have to be well formed according
>> to the SPI strucutres in Java, or is it still valid to have other
>> files in there as well? Before the module system it was no problem to
>> have in there service decriptors, that are services, but do not use
>> the ServiceProvider structure. Nothing was disturbed as long as nobody
>> tried to load it that way. Now this seems to be different and I am
>> wondering where this got specified.
>>
>> Because the spec does not try to define file layouts so much I have
>> the feeling that this is not really specified by the java module
>> system. Was it then a decision by some maven plugin writers or maybe
>> even the javac guys?
>
> I found
> https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Service_Provider 
>
> Is there no Java 9 version? Also the spec is not specific enough for
> this as it sounds more like implicit assumption than specification

https://docs.oracle.com/javase/9/docs/specs/jar/jar.html

but the Service provider section is gone. Also the document seems
mis-formatted.

David

> bye Jochen
>
Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

forax
I believe that the doc has been moved to the doc of the ServiceLoader
  https://docs.oracle.com/javase/9/docs/api/java/util/ServiceLoader.html

Rémi

----- Mail original -----
> De: "David Holmes" <[hidden email]>
> À: "Jochen Theodorou" <[hidden email]>, "jigsaw-dev" <[hidden email]>
> Envoyé: Mercredi 27 Décembre 2017 07:27:21
> Objet: Re: Module system and services directory in META-INF

> On 27/12/2017 5:43 AM, Jochen Theodorou wrote:
>> On 26.12.2017 19:30, Jochen Theodorou wrote:
>>> hi all,
>>>
>>> Do all files in META-INF/services now have to be well formed according
>>> to the SPI strucutres in Java, or is it still valid to have other
>>> files in there as well? Before the module system it was no problem to
>>> have in there service decriptors, that are services, but do not use
>>> the ServiceProvider structure. Nothing was disturbed as long as nobody
>>> tried to load it that way. Now this seems to be different and I am
>>> wondering where this got specified.
>>>
>>> Because the spec does not try to define file layouts so much I have
>>> the feeling that this is not really specified by the java module
>>> system. Was it then a decision by some maven plugin writers or maybe
>>> even the javac guys?
>>
>> I found
>> https://docs.oracle.com/javase/8/docs/technotes/guides/jar/jar.html#Service_Provider
>>
>> Is there no Java 9 version? Also the spec is not specific enough for
>> this as it sounds more like implicit assumption than specification
>
> https://docs.oracle.com/javase/9/docs/specs/jar/jar.html
>
> but the Service provider section is gone. Also the document seems
> mis-formatted.
>
> David
>
>> bye Jochen
Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

Cédric Champeau
Hi folks,

Let me rephrase what I think Jochen is saying. It seems there is an
implicit assertion, now, that every file found under META-INF/services are
for the exclusive use of ServiceLoader. It has been brought to the Groovy
language developers attention that Maven had a problem loading Groovy with
JDK 9, because of spurious class not found exception [1]. It turns out that
Maven makes use of java.lang.module.ModuleFinder.of(path).findAll(), which
scans for files under `META-INF/services` and assumes all of them are
service descriptors. This is *not* the case for Groovy, as we have used the
same directory to declare our "extension modules", as well as an additional
file named org.codehaus.groovy.source.Extensions which arguably has nothing
to do under `services` (but, it can be thought as a "service for Groovy").
This is the one responsible for the class not found exception, because this
file contains a single line with the name of a groovy file extension (that
is to say: "groovy") and it tries to load that class (which obviously
doesn't exist).

Now the problem we have is that we cannot move those files easily somewhere
else: this is the adhoc way of registering extension modules in Groovy. So
if we change the directory, all previously published extension modules will
never be found again, unless we scan for both the new and old location,
which has a cost. So there are 2 sides of the coin to this question:

1. is the spec saying somewhere that no-one should ever use
`META-INF/services` for something else than a service for `ServiceLoader`?
If not, then ModuleFinder should probably be patched to recognize that
sometimes it's not the case.
2. the 2d aspect of the question is what Jochen described as "would the
service loader infrastructure let us unload services".

Thanks and best wishes!

[1] http://markmail.org/message/obdyvuv24kqpxm6v

2017-12-27 13:10 GMT+01:00 Remi Forax <[hidden email]>:

> I believe that the doc has been moved to the doc of the ServiceLoader
>   https://docs.oracle.com/javase/9/docs/api/java/util/ServiceLoader.html
>
> Rémi
>
> ----- Mail original -----
> > De: "David Holmes" <[hidden email]>
> > À: "Jochen Theodorou" <[hidden email]>, "jigsaw-dev" <
> [hidden email]>
> > Envoyé: Mercredi 27 Décembre 2017 07:27:21
> > Objet: Re: Module system and services directory in META-INF
>
> > On 27/12/2017 5:43 AM, Jochen Theodorou wrote:
> >> On 26.12.2017 19:30, Jochen Theodorou wrote:
> >>> hi all,
> >>>
> >>> Do all files in META-INF/services now have to be well formed according
> >>> to the SPI strucutres in Java, or is it still valid to have other
> >>> files in there as well? Before the module system it was no problem to
> >>> have in there service decriptors, that are services, but do not use
> >>> the ServiceProvider structure. Nothing was disturbed as long as nobody
> >>> tried to load it that way. Now this seems to be different and I am
> >>> wondering where this got specified.
> >>>
> >>> Because the spec does not try to define file layouts so much I have
> >>> the feeling that this is not really specified by the java module
> >>> system. Was it then a decision by some maven plugin writers or maybe
> >>> even the javac guys?
> >>
> >> I found
> >> https://docs.oracle.com/javase/8/docs/technotes/
> guides/jar/jar.html#Service_Provider
> >>
> >> Is there no Java 9 version? Also the spec is not specific enough for
> >> this as it sounds more like implicit assumption than specification
> >
> > https://docs.oracle.com/javase/9/docs/specs/jar/jar.html
> >
> > but the Service provider section is gone. Also the document seems
> > mis-formatted.
> >
> > David
> >
> >> bye Jochen
>
Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

Alan Bateman
On 27/12/2017 12:22, Cédric Champeau wrote:
> :
>
> 1. is the spec saying somewhere that no-one should ever use
> `META-INF/services` for something else than a service for `ServiceLoader`?
> If not, then ModuleFinder should probably be patched to recognize that
> sometimes it's not the case.
The META-INF/services directory has been used for service configuration
files for a long time (I think originally specified in Java SE 1.3).
Adding properties files or other resources in that location may cause
problems for tools and libraries that scan that location.

I haven't read the thread that you linked to but I assume it's about JAR
files on the module path that are treated as automatic modules. In that
case, the relevant part of the ModuleFinder spec is:

"The contents of entries starting with META-INF/services/ are assumed to
be service configuration files (see ServiceLoader). If the name of a
file (that follows META-INF/services/) is a legal class name then it is
assumed to be the fully-qualified class name of a service type. The
entries in the file are assumed to be the fully-qualified class names of
provider classes."

> 2. the 2d aspect of the question is what Jochen described as "would the
> service loader infrastructure let us unload services".
>
Jochen's mail mentions loading modules dynamically so I assume this
means module layers. If so then unloading is possible and it works well
with services.

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

Alan Bateman
In reply to this post by David Holmes


On 27/12/2017 06:27, David Holmes wrote:
>
> https://docs.oracle.com/javase/9/docs/specs/jar/jar.html
>
> but the Service provider section is gone. Also the document seems
> mis-formatted.
Yes, there was a problem with the markdown. It's been fixed in JDK 10 docs.

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

Jochen Theodorou
In reply to this post by Cédric Champeau
On 27.12.2017 13:22, Cédric Champeau wrote:
[...]
> 1. is the spec saying somewhere that no-one should ever use
> `META-INF/services` for something else than a service for `ServiceLoader`?
> If not, then ModuleFinder should probably be patched to recognize that
> sometimes it's not the case.

or the javadoc of ServiceLoader should be cleared up as well as the jar
file spec should contain at least a pointer to the javadoc for "further"
specification.

> 2. the 2d aspect of the question is what Jochen described as "would the
> service loader infrastructure let us unload services".

I assume, that since the ServiceLoader is looking up services lazy
(which btw speaks against a requirement in 1), then a reload (which
really is more a reset) could in theory free resources, since the
services would first have to be rediscovered. Not sure if that is good
enough for us.

And before I start ranting how the extensive usage of CallerSensitive
makes a runtime practically require "all the rights" to run properly and
thus degrading the java module system or the runtime impossible I better
stop :(

bye Jochen
Reply | Threaded
Open this post in threaded view
|

Re: Module system and services directory in META-INF

Jochen Theodorou
In reply to this post by Alan Bateman
On 27.12.2017 13:55, Alan Bateman wrote:

> On 27/12/2017 12:22, Cédric Champeau wrote:
>> :
>>
>> 1. is the spec saying somewhere that no-one should ever use
>> `META-INF/services` for something else than a service for
>> `ServiceLoader`?
>> If not, then ModuleFinder should probably be patched to recognize that
>> sometimes it's not the case.
> The META-INF/services directory has been used for service configuration
> files for a long time (I think originally specified in Java SE 1.3).
> Adding properties files or other resources in that location may cause
> problems for tools and libraries that scan that location.

At least in the last 4 years nobody bothered reporting a problem with this.

> I haven't read the thread that you linked to but I assume it's about JAR
> files on the module path that are treated as automatic modules. In that
> case, the relevant part of the ModuleFinder spec is:
>
> "The contents of entries starting with META-INF/services/ are assumed to
> be service configuration files (see ServiceLoader). If the name of a
> file (that follows META-INF/services/) is a legal class name then it is
> assumed to be the fully-qualified class name of a service type. The
> entries in the file are assumed to be the fully-qualified class names of
> provider classes."

spec by javadoc... takes time getting used to it. At least now I know
where this is mentioned

>> 2. the 2d aspect of the question is what Jochen described as "would the
>> service loader infrastructure let us unload services".
>>
> Jochen's mail mentions loading modules dynamically so I assume this
> means module layers. If so then unloading is possible and it works well
> with services.

If not keeping a reference to layer, class and loader is enough, then I
am happy ;)

bye Jochen