Better tools for adjusting to strong encapsulation

classic Classic list List threaded Threaded
33 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Better tools for adjusting to strong encapsulation

mark.reinhold
By now we've heard quite a bit of feedback, both publicly and privately,
from developers who've migrated existing applications to JDK 9.  This can
be a non-trivial task, involving the iterative construction of a set of
command-line workarounds, since some popular libraries and frameworks use
the core reflection API to access internal APIs of the JDK.  On JDK 9
such components no longer work out-of-the-box because most internal APIs
are now strongly encapsulated in order to improve the integrity of the
platform.

In order to make it easier to adjust to the strong encapsulation of
internal APIs in JDK 9, in both the short term and over the long term,
we've introduced a temporary "big kill switch", described below.  This
switch will only be supported in JDK 9.  It's already available in the
current Jigsaw EA builds [1], and will shortly be merged into JDK 9.

Background
----------

Diagnosing the failures of libraries and frameworks that access internal
APIs of the JDK and then working around those failures is possible, but
tedious.  The initial symptom is usually an `IllegalAccessException` or
an `InaccessibleObjectException`.  The workaround is to use an
`--add-opens` command-line option whose arguments are based upon the
information shown in the exception message.  A large application that
uses several reflective components may suffer many such failures, so this
process may have to be repeated many times, resulting in a long list of
command-line options that will typically be recorded in a launch script
or an @-file.

These options are tedious to construct but, additionally, there's a
significant risk that once they're baked into a file they'll never be
removed, even long after the problems that they work around have been
resolved.  An application that needs some workarounds today might not
benefit as well as it could from strong encapsulation in the long term,
after all of the reflective components that it uses have been fixed to
replace their use of JDK-internal APIs with proper exported APIs.

To help the entire ecosystem migrate to the modular Java platform we want
to motivate such fixes as strongly as we possibly can, especially in the
near term.  We can do that by making it easier for you to get an existing
application to work while also ensuring that the run-time system reports
illegal reflective-access operations in a way that allows you to identify
which libraries and frameworks on your class path require illegal access
in order to work.  If a component's maintainers have already released a
new, fixed version that no longer requires illegal access then you can
consider upgrading to that version.  If, however, a component still needs
to be fixed then we encourage you to contact its maintainers and ask them
to stop using JDK-internal APIs [2].

With all this in mind we've introduced a "big kill switch", described
below, together with some new warning messages.  This switch will be
supported in JDK 9, in order to enable application migration and motivate
fixes to libraries and frameworks.  It will not be supported in JDK 10 or
later releases, in order to motivate fixes to libraries and frameworks
sooner rather than later.  If this switch were to be supported long-term
then that would tend to reduce the pressure on maintainers to fix their
components.

New command-line option: `--permit-illegal-access`
--------------------------------------------------

If this option is present at run time then it will permit illegal access
operations by code in any unnamed module (i.e., code on the class path)
to members of types in any named module via the standard reflective APIs
(`java.lang.reflect` and `java.lang.invoke`), including in particular
those that would otherwise result in an `IllegalAccessException` or an
`InaccessibleObjectException`.

            +----------------------------------------------+
            | This option will only be supported in JDK 9. |
            |         It will be removed in JDK 10.        |
            +----------------------------------------------+

This option does not permit illegal access operations by code in named
modules to members of types in other named modules; for that you must
use the `--add-opens` or `--add-exports` options.  Those options can be
combined with `--permit-illegal-access`.

Warnings of illegal reflective-access operations
------------------------------------------------

When an illegal reflective access operation succeeds due to the use of
the `--permit-illegal-access` option, or the use of an `--add-opens` or
`--add-exports` option, then a warning message of the following form is
written to the error stream:

    WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)

where:

  - $PERPETRATOR is the fully-qualified name of the type containing
    the code that invoked the reflective operation in question plus
    the code source (i.e., JAR-file path), if available,

  - $VICTIM is a string that describes the member being accessed,
    including the fully-qualified name of the enclosing type, and

  - $OPTION is the name of the command-line option that enabled this
    access, when that can be determined, or the first one of those
    options if more than one option had that effect.

The run-time system attempts to suppress duplicate warnings for the same
$PERPETRATOR and $VICTIM, but it's not always practical to do so.

For deeper diagnosis you can request a stack trace on each such warning
by setting the system property `sun.reflect.debugModuleAccessChecks` to
the value `access`, though this detail might change.  (That property can
also be helpful to diagnose mysterious failures due to illegal-access
exceptions that are caught and suppressed.)

In addition to displaying a warning on each illegal access operation, the
run-time system also shows new initial warning messages at startup time.
If `--permit-illegal-access` is used then a warning reports the imminent
demise of that option in the next major release.  If either `--add-opens`
or `--add-exports` are used then a warning reports a count of each type
of option used (i.e., opens vs. exports).

Here are some examples of these messages, from running Jython on a very
recent Jigsaw build:

  $ java --permit-illegal-access -jar jython-standalone-2.7.0.jar
  WARNING: --permit-illegal-access will be removed in the next major release
  WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to method sun.nio.ch.SelChImpl.getFD() (permitted by --permit-illegal-access)
  WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field sun.nio.ch.FileChannelImpl.fd (permitted by --permit-illegal-access)
  WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field java.io.FileDescriptor.fd (permitted by --permit-illegal-access)
  WARNING: Illegal access by org.python.core.PySystemState (file:/tmp/jython-standalone-2.7.0.jar) to method java.io.Console.encoding() (permitted by --permit-illegal-access)
  Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)
  [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java9-internal
  Type "help", "copyright", "credits" or "license" for more information.
  >>> ^D
  $


[1] http://openjdk.java.net/projects/jigsaw/ea
[2] This will usually but not always be possible, since there are still a
    few critical internal APIs without exported replacements, per JEP 260
    (http://openjdk.java.net/jeps/260).
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Simon Nash
These new messages are a big problem for an application that interacts with
the user via the console.  I understand that the messages may be useful to
application developers who need to "identify which libraries and frameworks
on your class path require illegal access in order to work".  For other
situations where the reason for the access is known to the application
developer and there is no way to avoid it, please consider giving the
application developer the option to suppress the messages or redirect them
to somewhere other than System.err.

Simon

On 21/03/2017 14:48, [hidden email] wrote:

> By now we've heard quite a bit of feedback, both publicly and privately,
> from developers who've migrated existing applications to JDK 9.  This can
> be a non-trivial task, involving the iterative construction of a set of
> command-line workarounds, since some popular libraries and frameworks use
> the core reflection API to access internal APIs of the JDK.  On JDK 9
> such components no longer work out-of-the-box because most internal APIs
> are now strongly encapsulated in order to improve the integrity of the
> platform.
>
> In order to make it easier to adjust to the strong encapsulation of
> internal APIs in JDK 9, in both the short term and over the long term,
> we've introduced a temporary "big kill switch", described below.  This
> switch will only be supported in JDK 9.  It's already available in the
> current Jigsaw EA builds [1], and will shortly be merged into JDK 9.
>
> Background
> ----------
>
> Diagnosing the failures of libraries and frameworks that access internal
> APIs of the JDK and then working around those failures is possible, but
> tedious.  The initial symptom is usually an `IllegalAccessException` or
> an `InaccessibleObjectException`.  The workaround is to use an
> `--add-opens` command-line option whose arguments are based upon the
> information shown in the exception message.  A large application that
> uses several reflective components may suffer many such failures, so this
> process may have to be repeated many times, resulting in a long list of
> command-line options that will typically be recorded in a launch script
> or an @-file.
>
> These options are tedious to construct but, additionally, there's a
> significant risk that once they're baked into a file they'll never be
> removed, even long after the problems that they work around have been
> resolved.  An application that needs some workarounds today might not
> benefit as well as it could from strong encapsulation in the long term,
> after all of the reflective components that it uses have been fixed to
> replace their use of JDK-internal APIs with proper exported APIs.
>
> To help the entire ecosystem migrate to the modular Java platform we want
> to motivate such fixes as strongly as we possibly can, especially in the
> near term.  We can do that by making it easier for you to get an existing
> application to work while also ensuring that the run-time system reports
> illegal reflective-access operations in a way that allows you to identify
> which libraries and frameworks on your class path require illegal access
> in order to work.  If a component's maintainers have already released a
> new, fixed version that no longer requires illegal access then you can
> consider upgrading to that version.  If, however, a component still needs
> to be fixed then we encourage you to contact its maintainers and ask them
> to stop using JDK-internal APIs [2].
>
> With all this in mind we've introduced a "big kill switch", described
> below, together with some new warning messages.  This switch will be
> supported in JDK 9, in order to enable application migration and motivate
> fixes to libraries and frameworks.  It will not be supported in JDK 10 or
> later releases, in order to motivate fixes to libraries and frameworks
> sooner rather than later.  If this switch were to be supported long-term
> then that would tend to reduce the pressure on maintainers to fix their
> components.
>
> New command-line option: `--permit-illegal-access`
> --------------------------------------------------
>
> If this option is present at run time then it will permit illegal access
> operations by code in any unnamed module (i.e., code on the class path)
> to members of types in any named module via the standard reflective APIs
> (`java.lang.reflect` and `java.lang.invoke`), including in particular
> those that would otherwise result in an `IllegalAccessException` or an
> `InaccessibleObjectException`.
>
>             +----------------------------------------------+
>             | This option will only be supported in JDK 9. |
>             |         It will be removed in JDK 10.        |
>             +----------------------------------------------+
>
> This option does not permit illegal access operations by code in named
> modules to members of types in other named modules; for that you must
> use the `--add-opens` or `--add-exports` options.  Those options can be
> combined with `--permit-illegal-access`.
>
> Warnings of illegal reflective-access operations
> ------------------------------------------------
>
> When an illegal reflective access operation succeeds due to the use of
> the `--permit-illegal-access` option, or the use of an `--add-opens` or
> `--add-exports` option, then a warning message of the following form is
> written to the error stream:
>
>     WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)
>
> where:
>
>   - $PERPETRATOR is the fully-qualified name of the type containing
>     the code that invoked the reflective operation in question plus
>     the code source (i.e., JAR-file path), if available,
>
>   - $VICTIM is a string that describes the member being accessed,
>     including the fully-qualified name of the enclosing type, and
>
>   - $OPTION is the name of the command-line option that enabled this
>     access, when that can be determined, or the first one of those
>     options if more than one option had that effect.
>
> The run-time system attempts to suppress duplicate warnings for the same
> $PERPETRATOR and $VICTIM, but it's not always practical to do so.
>
> For deeper diagnosis you can request a stack trace on each such warning
> by setting the system property `sun.reflect.debugModuleAccessChecks` to
> the value `access`, though this detail might change.  (That property can
> also be helpful to diagnose mysterious failures due to illegal-access
> exceptions that are caught and suppressed.)
>
> In addition to displaying a warning on each illegal access operation, the
> run-time system also shows new initial warning messages at startup time.
> If `--permit-illegal-access` is used then a warning reports the imminent
> demise of that option in the next major release.  If either `--add-opens`
> or `--add-exports` are used then a warning reports a count of each type
> of option used (i.e., opens vs. exports).
>
> Here are some examples of these messages, from running Jython on a very
> recent Jigsaw build:
>
>   $ java --permit-illegal-access -jar jython-standalone-2.7.0.jar
>   WARNING: --permit-illegal-access will be removed in the next major release
>   WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to method sun.nio.ch.SelChImpl.getFD() (permitted by --permit-illegal-access)
>   WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field sun.nio.ch.FileChannelImpl.fd (permitted by --permit-illegal-access)
>   WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field java.io.FileDescriptor.fd (permitted by --permit-illegal-access)
>   WARNING: Illegal access by org.python.core.PySystemState (file:/tmp/jython-standalone-2.7.0.jar) to method java.io.Console.encoding() (permitted by --permit-illegal-access)
>   Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)
>   [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java9-internal
>   Type "help", "copyright", "credits" or "license" for more information.
>   >>> ^D
>   $
>
>
> [1] http://openjdk.java.net/projects/jigsaw/ea
> [2] This will usually but not always be possible, since there are still a
>     few critical internal APIs without exported replacements, per JEP 260
>     (http://openjdk.java.net/jeps/260).
>




Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Phil Race
In reply to this post by mark.reinhold
Lest folks think they can now sit back and relax until JDK 10,
I'd like to point out that there are some cases where what has
happened in 9 is that a very small number of old deprecated
and non-standard APIs have actually been removed, and some
small number of APIs adjusted in behaviour.
Whilst these changes are generally tangential to the module system,
its advent made it necessary to determine what was supported and standard.

No "kill switch" will bring those back, so affected apps will
still need to migrate for JDK 9.

The JDK 9 migration guide should cover the specifics of these APIs.

-phil.

On 03/21/2017 07:48 AM, [hidden email] wrote:

> By now we've heard quite a bit of feedback, both publicly and privately,
> from developers who've migrated existing applications to JDK 9.  This can
> be a non-trivial task, involving the iterative construction of a set of
> command-line workarounds, since some popular libraries and frameworks use
> the core reflection API to access internal APIs of the JDK.  On JDK 9
> such components no longer work out-of-the-box because most internal APIs
> are now strongly encapsulated in order to improve the integrity of the
> platform.
>
> In order to make it easier to adjust to the strong encapsulation of
> internal APIs in JDK 9, in both the short term and over the long term,
> we've introduced a temporary "big kill switch", described below.  This
> switch will only be supported in JDK 9.  It's already available in the
> current Jigsaw EA builds [1], and will shortly be merged into JDK 9.
>
> Background
> ----------
>
> Diagnosing the failures of libraries and frameworks that access internal
> APIs of the JDK and then working around those failures is possible, but
> tedious.  The initial symptom is usually an `IllegalAccessException` or
> an `InaccessibleObjectException`.  The workaround is to use an
> `--add-opens` command-line option whose arguments are based upon the
> information shown in the exception message.  A large application that
> uses several reflective components may suffer many such failures, so this
> process may have to be repeated many times, resulting in a long list of
> command-line options that will typically be recorded in a launch script
> or an @-file.
>
> These options are tedious to construct but, additionally, there's a
> significant risk that once they're baked into a file they'll never be
> removed, even long after the problems that they work around have been
> resolved.  An application that needs some workarounds today might not
> benefit as well as it could from strong encapsulation in the long term,
> after all of the reflective components that it uses have been fixed to
> replace their use of JDK-internal APIs with proper exported APIs.
>
> To help the entire ecosystem migrate to the modular Java platform we want
> to motivate such fixes as strongly as we possibly can, especially in the
> near term.  We can do that by making it easier for you to get an existing
> application to work while also ensuring that the run-time system reports
> illegal reflective-access operations in a way that allows you to identify
> which libraries and frameworks on your class path require illegal access
> in order to work.  If a component's maintainers have already released a
> new, fixed version that no longer requires illegal access then you can
> consider upgrading to that version.  If, however, a component still needs
> to be fixed then we encourage you to contact its maintainers and ask them
> to stop using JDK-internal APIs [2].
>
> With all this in mind we've introduced a "big kill switch", described
> below, together with some new warning messages.  This switch will be
> supported in JDK 9, in order to enable application migration and motivate
> fixes to libraries and frameworks.  It will not be supported in JDK 10 or
> later releases, in order to motivate fixes to libraries and frameworks
> sooner rather than later.  If this switch were to be supported long-term
> then that would tend to reduce the pressure on maintainers to fix their
> components.
>
> New command-line option: `--permit-illegal-access`
> --------------------------------------------------
>
> If this option is present at run time then it will permit illegal access
> operations by code in any unnamed module (i.e., code on the class path)
> to members of types in any named module via the standard reflective APIs
> (`java.lang.reflect` and `java.lang.invoke`), including in particular
> those that would otherwise result in an `IllegalAccessException` or an
> `InaccessibleObjectException`.
>
>              +----------------------------------------------+
>              | This option will only be supported in JDK 9. |
>              |         It will be removed in JDK 10.        |
>              +----------------------------------------------+
>
> This option does not permit illegal access operations by code in named
> modules to members of types in other named modules; for that you must
> use the `--add-opens` or `--add-exports` options.  Those options can be
> combined with `--permit-illegal-access`.
>
> Warnings of illegal reflective-access operations
> ------------------------------------------------
>
> When an illegal reflective access operation succeeds due to the use of
> the `--permit-illegal-access` option, or the use of an `--add-opens` or
> `--add-exports` option, then a warning message of the following form is
> written to the error stream:
>
>      WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)
>
> where:
>
>    - $PERPETRATOR is the fully-qualified name of the type containing
>      the code that invoked the reflective operation in question plus
>      the code source (i.e., JAR-file path), if available,
>
>    - $VICTIM is a string that describes the member being accessed,
>      including the fully-qualified name of the enclosing type, and
>
>    - $OPTION is the name of the command-line option that enabled this
>      access, when that can be determined, or the first one of those
>      options if more than one option had that effect.
>
> The run-time system attempts to suppress duplicate warnings for the same
> $PERPETRATOR and $VICTIM, but it's not always practical to do so.
>
> For deeper diagnosis you can request a stack trace on each such warning
> by setting the system property `sun.reflect.debugModuleAccessChecks` to
> the value `access`, though this detail might change.  (That property can
> also be helpful to diagnose mysterious failures due to illegal-access
> exceptions that are caught and suppressed.)
>
> In addition to displaying a warning on each illegal access operation, the
> run-time system also shows new initial warning messages at startup time.
> If `--permit-illegal-access` is used then a warning reports the imminent
> demise of that option in the next major release.  If either `--add-opens`
> or `--add-exports` are used then a warning reports a count of each type
> of option used (i.e., opens vs. exports).
>
> Here are some examples of these messages, from running Jython on a very
> recent Jigsaw build:
>
>    $ java --permit-illegal-access -jar jython-standalone-2.7.0.jar
>    WARNING: --permit-illegal-access will be removed in the next major release
>    WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to method sun.nio.ch.SelChImpl.getFD() (permitted by --permit-illegal-access)
>    WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field sun.nio.ch.FileChannelImpl.fd (permitted by --permit-illegal-access)
>    WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field java.io.FileDescriptor.fd (permitted by --permit-illegal-access)
>    WARNING: Illegal access by org.python.core.PySystemState (file:/tmp/jython-standalone-2.7.0.jar) to method java.io.Console.encoding() (permitted by --permit-illegal-access)
>    Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)
>    [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java9-internal
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> ^D
>    $
>
>
> [1] http://openjdk.java.net/projects/jigsaw/ea
> [2] This will usually but not always be possible, since there are still a
>      few critical internal APIs without exported replacements, per JEP 260
>      (http://openjdk.java.net/jeps/260).

Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Gregg Wonderly
In reply to this post by mark.reinhold
Better yet, why aren’t the java.util.logging apis used for stuff like this?  It seems strange that there is not a nice compact formatter, yet, standard in the JDK.  The logging APIs seem highly neglected given that even the varargs access has not been fixed so that you can use varargs without an array argument on the end.  

Would be nice to see things like this that actually could greatly unify the management of the runtime environment get used, instead of using stderr and then creating that as a legacy interface which will now have to work forever since people will create tooling to use that.  If instead logging was used, it could be quenched with Level settings, or reformatted/managed with a custom handler so that everyone could have their cake and eat it too…

Gregg

> On Mar 21, 2017, at 12:56 PM, Simon Nash <[hidden email]> wrote:
>
> These new messages are a big problem for an application that interacts with
> the user via the console.  I understand that the messages may be useful to
> application developers who need to "identify which libraries and frameworks
> on your class path require illegal access in order to work".  For other
> situations where the reason for the access is known to the application
> developer and there is no way to avoid it, please consider giving the
> application developer the option to suppress the messages or redirect them
> to somewhere other than System.err.
>
> Simon
>
> On 21/03/2017 14:48, [hidden email] wrote:
>> By now we've heard quite a bit of feedback, both publicly and privately,
>> from developers who've migrated existing applications to JDK 9.  This can
>> be a non-trivial task, involving the iterative construction of a set of
>> command-line workarounds, since some popular libraries and frameworks use
>> the core reflection API to access internal APIs of the JDK.  On JDK 9
>> such components no longer work out-of-the-box because most internal APIs
>> are now strongly encapsulated in order to improve the integrity of the
>> platform.
>> In order to make it easier to adjust to the strong encapsulation of
>> internal APIs in JDK 9, in both the short term and over the long term,
>> we've introduced a temporary "big kill switch", described below.  This
>> switch will only be supported in JDK 9.  It's already available in the
>> current Jigsaw EA builds [1], and will shortly be merged into JDK 9.
>> Background
>> ----------
>> Diagnosing the failures of libraries and frameworks that access internal
>> APIs of the JDK and then working around those failures is possible, but
>> tedious.  The initial symptom is usually an `IllegalAccessException` or
>> an `InaccessibleObjectException`.  The workaround is to use an
>> `--add-opens` command-line option whose arguments are based upon the
>> information shown in the exception message.  A large application that
>> uses several reflective components may suffer many such failures, so this
>> process may have to be repeated many times, resulting in a long list of
>> command-line options that will typically be recorded in a launch script
>> or an @-file.
>> These options are tedious to construct but, additionally, there's a
>> significant risk that once they're baked into a file they'll never be
>> removed, even long after the problems that they work around have been
>> resolved.  An application that needs some workarounds today might not
>> benefit as well as it could from strong encapsulation in the long term,
>> after all of the reflective components that it uses have been fixed to
>> replace their use of JDK-internal APIs with proper exported APIs.
>> To help the entire ecosystem migrate to the modular Java platform we want
>> to motivate such fixes as strongly as we possibly can, especially in the
>> near term.  We can do that by making it easier for you to get an existing
>> application to work while also ensuring that the run-time system reports
>> illegal reflective-access operations in a way that allows you to identify
>> which libraries and frameworks on your class path require illegal access
>> in order to work.  If a component's maintainers have already released a
>> new, fixed version that no longer requires illegal access then you can
>> consider upgrading to that version.  If, however, a component still needs
>> to be fixed then we encourage you to contact its maintainers and ask them
>> to stop using JDK-internal APIs [2].
>> With all this in mind we've introduced a "big kill switch", described
>> below, together with some new warning messages.  This switch will be
>> supported in JDK 9, in order to enable application migration and motivate
>> fixes to libraries and frameworks.  It will not be supported in JDK 10 or
>> later releases, in order to motivate fixes to libraries and frameworks
>> sooner rather than later.  If this switch were to be supported long-term
>> then that would tend to reduce the pressure on maintainers to fix their
>> components.
>> New command-line option: `--permit-illegal-access`
>> --------------------------------------------------
>> If this option is present at run time then it will permit illegal access
>> operations by code in any unnamed module (i.e., code on the class path)
>> to members of types in any named module via the standard reflective APIs
>> (`java.lang.reflect` and `java.lang.invoke`), including in particular
>> those that would otherwise result in an `IllegalAccessException` or an
>> `InaccessibleObjectException`.
>>            +----------------------------------------------+
>>            | This option will only be supported in JDK 9. |
>>            |         It will be removed in JDK 10.        |
>>            +----------------------------------------------+
>> This option does not permit illegal access operations by code in named
>> modules to members of types in other named modules; for that you must
>> use the `--add-opens` or `--add-exports` options.  Those options can be
>> combined with `--permit-illegal-access`.
>> Warnings of illegal reflective-access operations
>> ------------------------------------------------
>> When an illegal reflective access operation succeeds due to the use of
>> the `--permit-illegal-access` option, or the use of an `--add-opens` or
>> `--add-exports` option, then a warning message of the following form is
>> written to the error stream:
>>    WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)
>> where:
>>  - $PERPETRATOR is the fully-qualified name of the type containing
>>    the code that invoked the reflective operation in question plus
>>    the code source (i.e., JAR-file path), if available,
>>  - $VICTIM is a string that describes the member being accessed,
>>    including the fully-qualified name of the enclosing type, and
>>  - $OPTION is the name of the command-line option that enabled this
>>    access, when that can be determined, or the first one of those
>>    options if more than one option had that effect.
>> The run-time system attempts to suppress duplicate warnings for the same
>> $PERPETRATOR and $VICTIM, but it's not always practical to do so.
>> For deeper diagnosis you can request a stack trace on each such warning
>> by setting the system property `sun.reflect.debugModuleAccessChecks` to
>> the value `access`, though this detail might change.  (That property can
>> also be helpful to diagnose mysterious failures due to illegal-access
>> exceptions that are caught and suppressed.)
>> In addition to displaying a warning on each illegal access operation, the
>> run-time system also shows new initial warning messages at startup time.
>> If `--permit-illegal-access` is used then a warning reports the imminent
>> demise of that option in the next major release.  If either `--add-opens`
>> or `--add-exports` are used then a warning reports a count of each type
>> of option used (i.e., opens vs. exports).
>> Here are some examples of these messages, from running Jython on a very
>> recent Jigsaw build:
>>  $ java --permit-illegal-access -jar jython-standalone-2.7.0.jar
>>  WARNING: --permit-illegal-access will be removed in the next major release
>>  WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to method sun.nio.ch.SelChImpl.getFD() (permitted by --permit-illegal-access)
>>  WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field sun.nio.ch.FileChannelImpl.fd (permitted by --permit-illegal-access)
>>  WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field java.io.FileDescriptor.fd (permitted by --permit-illegal-access)
>>  WARNING: Illegal access by org.python.core.PySystemState (file:/tmp/jython-standalone-2.7.0.jar) to method java.io.Console.encoding() (permitted by --permit-illegal-access)
>>  Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)   [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java9-internal
>>  Type "help", "copyright", "credits" or "license" for more information.
>>  >>> ^D
>>  $ [1] http://openjdk.java.net/projects/jigsaw/ea
>> [2] This will usually but not always be possible, since there are still a
>>    few critical internal APIs without exported replacements, per JEP 260
>>    (http://openjdk.java.net/jeps/260).
>
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Jochen Theodorou
In reply to this post by mark.reinhold
On 21.03.2017 15:48, [hidden email] wrote:
[...]

> When an illegal reflective access operation succeeds due to the use of
> the `--permit-illegal-access` option, or the use of an `--add-opens` or
> `--add-exports` option, then a warning message of the following form is
> written to the error stream:
>
>      WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)
>
> where:
>
>    - $PERPETRATOR is the fully-qualified name of the type containing
>      the code that invoked the reflective operation in question plus
>      the code source (i.e., JAR-file path), if available,
>
>    - $VICTIM is a string that describes the member being accessed,
>      including the fully-qualified name of the enclosing type, and
>
>    - $OPTION is the name of the command-line option that enabled this
>      access, when that can be determined, or the first one of those
>      options if more than one option had that effect.
>
> The run-time system attempts to suppress duplicate warnings for the same
> $PERPETRATOR and $VICTIM, but it's not always practical to do so.

that means we will get a lot of users asking us, why our runtime does
this, even though it is the "fault" of the code using the runtime and
not really ours. Because "type that invoked the reflective operation" is
very likely again something caller sensitive, right? Well, that will
point to our runtime, not to the real perpetrator. I think there is an
annotation that could be used to exclude our methods from getting the
caller information, but if memory serves me right this is no public api.

What will be the performance penalty of this?

bye Jochen
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Roel Spilker
In reply to this post by mark.reinhold
On Tue, Mar 21, 2017 at 3:48 PM, <[hidden email]> wrote:

> When an illegal reflective access operation succeeds due to the use of
> the `--permit-illegal-access` option, or the use of an `--add-opens` or
> `--add-exports` option, then a warning message of the following form is
> written to the error stream:
>
>     WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by
> $OPTION)
>


Hi Mark, others,

I really appreciate the work you do on helping developers to migrate their
code to the module system. I already use the messages in the exceptions to
solve problems successfully.

That said, I consider the output of my programs (the exit code, the output
stream, and the error stream) to be part of my public API. I consider extra
output, that is added to those streams outside my control, a breaking
change. Depending on my program flow and data in my program, these messages
appear in the middle of my output.

This is especially a problem because the users of my program cannot do
anything themselves to get rid of it. In my opinion, they should not be
bothered by messages intended for the developer.

Since this is a tool to help developers, can you either make it an opt-in
or at least an opt-out feature?

Kind regards,

Roel Spilker
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Michael Rasmussen
On 22 March 2017 at 12:28, Roel Spilker <[hidden email]> wrote:
> This is especially a problem because the users of my program cannot do
> anything themselves to get rid of it. In my opinion, they should not be
> bothered by messages intended for the developer.
>
> Since this is a tool to help developers, can you either make it an opt-in
> or at least an opt-out feature?
Well, in this case, as the warnings only appear because one of the
"override" JVM options were added, unless you're pushing those
settings to the end-user, they won't see these warnings.

But I do agree with the general gist of it (and what Jochen wrote).
JDK9 seems to be adding (too) many stderr outputs that are on by
default (not limited to jigsaw), and you have to go digging for
specific properties or special -XX JVM flags to set in order to
disable them (if at all possible).

Regards,
Michael Rasmussen
JRebel, ZeroTurnaround
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Alan Bateman
In reply to this post by Jochen Theodorou
On 21/03/2017 20:36, Jochen Theodorou wrote:

>
> that means we will get a lot of users asking us, why our runtime does
> this, even though it is the "fault" of the code using the runtime and
> not really ours. Because "type that invoked the reflective operation"
> is very likely again something caller sensitive, right? Well, that
> will point to our runtime, not to the real perpetrator. I think there
> is an annotation that could be used to exclude our methods from
> getting the caller information, but if memory serves me right this is
> no public api.
>
> What will be the performance penalty of this?
There isn't an annotation or anything else in this proposal to report a
caller from further up the stack but there are several debugging options
to reveal the stack traces when you need it (these debugging options
have been in JDK 9 for a long time and are very useful).

I assume your interest is Groovy where there are indeed some warnings
that look like Groovy is to blame. For example, I get the following with
a trivial script containing`println "Hello world!"` :

$ groovy hello.groovy
NOTE: Picked up the following options via JDK_JAVA_OPTIONS:
   --permit-illegal-access
WARNING: --permit-illegal-access will be removed in the next major release
WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
(file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
java.lang.Object.finalize() (permitted by --permit-illegal-access)
WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
(file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
java.lang.Object.clone() (permitted by --permit-illegal-access)
WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
(file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
java.lang.Object.registerNatives() (permitted by --permit-illegal-access)
WARNING: Illegal access by org.codehaus.groovy.vmplugin.v7.Java7$1
(file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to constructor
java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) (permitted by
--permit-illegal-access)
Hello world!

The user's script is innocent but maybe you are saying that if Groovy
spins code and that code calls back into the Groovy runtime to do its
dirty work then Groovy will be blamed?

On the question of performance then it may have an impact, it might not
as it depends on how much illegal access is going on. We see some
libraries doing some hacking with setAccessible(true) at startup and
then re-using the AccessibleObject over and over, they will hardly
notice anything.

In the case of Groovy then running with
JDK_JAVA_OPTIONS="-Dsun.reflect.debugModuleAccessChecks=true" reveals
several places where  InaccessibleObjectException is being thrown and
silently ignored, e.g:

java.lang.reflect.InaccessibleObjectException: Unable to make private
java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) accessible:
module java.base does not "opens java.lang.invoke" to unnamed module
@43bd930a
     at
java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:337)
     at
java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:281)
     at
java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:192)
     at
java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:185)
     at org.codehaus.groovy.vmplugin.v7.Java7$1.run(Java7.java:53)
     at java.base/java.security.AccessController.doPrivileged(Native Method)
     at org.codehaus.groovy.vmplugin.v7.Java7.<clinit>(Java7.java:50)
     at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method)
     at
java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
     at
java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
     at
java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
     at java.base/java.lang.Class.newInstance(Class.java:558)
     at
org.codehaus.groovy.vmplugin.VMPluginFactory.createPlugin(VMPluginFactory.java:59)
     at
org.codehaus.groovy.vmplugin.VMPluginFactory.<clinit>(VMPluginFactory.java:40)
     at
org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:102)
     at
org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:74)
     at groovy.lang.GroovySystem.<clinit>(GroovySystem.java:36)
     at
org.codehaus.groovy.runtime.InvokerHelper.<clinit>(InvokerHelper.java:66)
     at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)
     at groovy.lang.Binding.<init>(Binding.java:35)
     at groovy.lang.GroovyShell.<init>(GroovyShell.java:74)
     at groovy.ui.GroovyMain.processOnce(GroovyMain.java:651)
     at groovy.ui.GroovyMain.run(GroovyMain.java:384)
     at groovy.ui.GroovyMain.process(GroovyMain.java:370)
     at groovy.ui.GroovyMain.processArgs(GroovyMain.java:129)
     at groovy.ui.GroovyMain.main(GroovyMain.java:109)
     at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
     at
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
     at
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at java.base/java.lang.reflect.Method.invoke(Method.java:563)
     at
org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
     at org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
Hello world!

So it might be that the cost of these exceptions is already high and
that doing illegal access (with --permit-illegal-access) will not be
noticed while the issues are being fixed.

-Alan

Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Alan Bateman
In reply to this post by Gregg Wonderly
On 21/03/2017 19:03, Gregg Wonderly wrote:

> Better yet, why aren’t the java.util.logging apis used for stuff like this?  It seems strange that there is not a nice compact formatter, yet, standard in the JDK.  The logging APIs seem highly neglected given that even the varargs access has not been fixed so that you can use varargs without an array argument on the end.
>
> Would be nice to see things like this that actually could greatly unify the management of the runtime environment get used, instead of using stderr and then creating that as a legacy interface which will now have to work forever since people will create tooling to use that.  If instead logging was used, it could be quenched with Level settings, or reformatted/managed with a custom handler so that everyone could have their cake and eat it too…
>
The java.util.logging API is not appropriate here. There are many
reasons. One is that java.logging is its own module, you can't directly
use it in the java.base module. We can create run-time images that don't
include the java.logging module too. Then there is the issue that using
logging APIs can potentially run arbitrary code and that arbitrary code
will trigger access checks too.

On your comment about neglect then you might want to look at JEP 264 [1].

-Alan

[1] http://openjdk.java.net/jeps/264
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Andrew Dinn
On 22/03/17 11:42, Alan Bateman wrote:
> On your comment about neglect then you might want to look at JEP 264 [1].
> ...
> [1] http://openjdk.java.net/jeps/264

You might also want to look at this xkcd

  https://xkcd.com/927/

which is, perhaps, just as relevant.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Martijn Verburg
In reply to this post by mark.reinhold
Hi Mark,

Thanks for putting in this mechanism.  I appreciate some folks would
like some fine tuning / control of this, but it gets a thumbs up from
many of our customers and developers at the LJC.  It's crucial that
they're given time to migrate to safer libs (or even better help those
libs become safer).
Cheers,
Martijn


On 21 March 2017 at 14:48,  <[hidden email]> wrote:

> By now we've heard quite a bit of feedback, both publicly and privately,
> from developers who've migrated existing applications to JDK 9.  This can
> be a non-trivial task, involving the iterative construction of a set of
> command-line workarounds, since some popular libraries and frameworks use
> the core reflection API to access internal APIs of the JDK.  On JDK 9
> such components no longer work out-of-the-box because most internal APIs
> are now strongly encapsulated in order to improve the integrity of the
> platform.
>
> In order to make it easier to adjust to the strong encapsulation of
> internal APIs in JDK 9, in both the short term and over the long term,
> we've introduced a temporary "big kill switch", described below.  This
> switch will only be supported in JDK 9.  It's already available in the
> current Jigsaw EA builds [1], and will shortly be merged into JDK 9.
>
> Background
> ----------
>
> Diagnosing the failures of libraries and frameworks that access internal
> APIs of the JDK and then working around those failures is possible, but
> tedious.  The initial symptom is usually an `IllegalAccessException` or
> an `InaccessibleObjectException`.  The workaround is to use an
> `--add-opens` command-line option whose arguments are based upon the
> information shown in the exception message.  A large application that
> uses several reflective components may suffer many such failures, so this
> process may have to be repeated many times, resulting in a long list of
> command-line options that will typically be recorded in a launch script
> or an @-file.
>
> These options are tedious to construct but, additionally, there's a
> significant risk that once they're baked into a file they'll never be
> removed, even long after the problems that they work around have been
> resolved.  An application that needs some workarounds today might not
> benefit as well as it could from strong encapsulation in the long term,
> after all of the reflective components that it uses have been fixed to
> replace their use of JDK-internal APIs with proper exported APIs.
>
> To help the entire ecosystem migrate to the modular Java platform we want
> to motivate such fixes as strongly as we possibly can, especially in the
> near term.  We can do that by making it easier for you to get an existing
> application to work while also ensuring that the run-time system reports
> illegal reflective-access operations in a way that allows you to identify
> which libraries and frameworks on your class path require illegal access
> in order to work.  If a component's maintainers have already released a
> new, fixed version that no longer requires illegal access then you can
> consider upgrading to that version.  If, however, a component still needs
> to be fixed then we encourage you to contact its maintainers and ask them
> to stop using JDK-internal APIs [2].
>
> With all this in mind we've introduced a "big kill switch", described
> below, together with some new warning messages.  This switch will be
> supported in JDK 9, in order to enable application migration and motivate
> fixes to libraries and frameworks.  It will not be supported in JDK 10 or
> later releases, in order to motivate fixes to libraries and frameworks
> sooner rather than later.  If this switch were to be supported long-term
> then that would tend to reduce the pressure on maintainers to fix their
> components.
>
> New command-line option: `--permit-illegal-access`
> --------------------------------------------------
>
> If this option is present at run time then it will permit illegal access
> operations by code in any unnamed module (i.e., code on the class path)
> to members of types in any named module via the standard reflective APIs
> (`java.lang.reflect` and `java.lang.invoke`), including in particular
> those that would otherwise result in an `IllegalAccessException` or an
> `InaccessibleObjectException`.
>
>             +----------------------------------------------+
>             | This option will only be supported in JDK 9. |
>             |         It will be removed in JDK 10.        |
>             +----------------------------------------------+
>
> This option does not permit illegal access operations by code in named
> modules to members of types in other named modules; for that you must
> use the `--add-opens` or `--add-exports` options.  Those options can be
> combined with `--permit-illegal-access`.
>
> Warnings of illegal reflective-access operations
> ------------------------------------------------
>
> When an illegal reflective access operation succeeds due to the use of
> the `--permit-illegal-access` option, or the use of an `--add-opens` or
> `--add-exports` option, then a warning message of the following form is
> written to the error stream:
>
>     WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)
>
> where:
>
>   - $PERPETRATOR is the fully-qualified name of the type containing
>     the code that invoked the reflective operation in question plus
>     the code source (i.e., JAR-file path), if available,
>
>   - $VICTIM is a string that describes the member being accessed,
>     including the fully-qualified name of the enclosing type, and
>
>   - $OPTION is the name of the command-line option that enabled this
>     access, when that can be determined, or the first one of those
>     options if more than one option had that effect.
>
> The run-time system attempts to suppress duplicate warnings for the same
> $PERPETRATOR and $VICTIM, but it's not always practical to do so.
>
> For deeper diagnosis you can request a stack trace on each such warning
> by setting the system property `sun.reflect.debugModuleAccessChecks` to
> the value `access`, though this detail might change.  (That property can
> also be helpful to diagnose mysterious failures due to illegal-access
> exceptions that are caught and suppressed.)
>
> In addition to displaying a warning on each illegal access operation, the
> run-time system also shows new initial warning messages at startup time.
> If `--permit-illegal-access` is used then a warning reports the imminent
> demise of that option in the next major release.  If either `--add-opens`
> or `--add-exports` are used then a warning reports a count of each type
> of option used (i.e., opens vs. exports).
>
> Here are some examples of these messages, from running Jython on a very
> recent Jigsaw build:
>
>   $ java --permit-illegal-access -jar jython-standalone-2.7.0.jar
>   WARNING: --permit-illegal-access will be removed in the next major release
>   WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to method sun.nio.ch.SelChImpl.getFD() (permitted by --permit-illegal-access)
>   WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field sun.nio.ch.FileChannelImpl.fd (permitted by --permit-illegal-access)
>   WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field java.io.FileDescriptor.fd (permitted by --permit-illegal-access)
>   WARNING: Illegal access by org.python.core.PySystemState (file:/tmp/jython-standalone-2.7.0.jar) to method java.io.Console.encoding() (permitted by --permit-illegal-access)
>   Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)
>   [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java9-internal
>   Type "help", "copyright", "credits" or "license" for more information.
>   >>> ^D
>   $
>
>
> [1] http://openjdk.java.net/projects/jigsaw/ea
> [2] This will usually but not always be possible, since there are still a
>     few critical internal APIs without exported replacements, per JEP 260
>     (http://openjdk.java.net/jeps/260).
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Remi Forax
In reply to this post by Andrew Dinn
Hi Andrew,
it's not that relevant, we are beyond that point.

Introducing SystemLogger is an implicit way to recognize that java.logging is a bad API and that the JDK should cut its ties to that API (when it's possible, i.e. when Logger is not part of the public API).

I see the SystemLogger API is a way to get logging info from the JDK and to redirect them to whatever logger i want, so one can have the JDK log info and the application log info at the same place.

cheers,
Rémi

----- Mail original -----
> De: "Andrew Dinn" <[hidden email]>
> À: "Alan Bateman" <[hidden email]>
> Cc: [hidden email]
> Envoyé: Mercredi 22 Mars 2017 12:57:56
> Objet: Re: Better tools for adjusting to strong encapsulation

> On 22/03/17 11:42, Alan Bateman wrote:
>> On your comment about neglect then you might want to look at JEP 264 [1].
>> ...
>> [1] http://openjdk.java.net/jeps/264
>
> You might also want to look at this xkcd
>
>  https://xkcd.com/927/
>
> which is, perhaps, just as relevant.
>
> regards,
>
>
> Andrew Dinn
> -----------
> Senior Principal Software Engineer
> Red Hat UK Ltd
> Registered in England and Wales under Company Registration No. 03798903
> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Remi Forax
In reply to this post by mark.reinhold
I do not understand why the uses of --add-exports and --add-opens has to generate a warning,
they currently don't, this is a regression from my POV.

I do not think that introducing --permit-illegal-access worth the deal ion that case.

regards,
Rémi

----- Mail original -----
> De: "mark reinhold" <[hidden email]>
> À: [hidden email]
> Envoyé: Mardi 21 Mars 2017 15:48:13
> Objet: Better tools for adjusting to strong encapsulation

> By now we've heard quite a bit of feedback, both publicly and privately,
> from developers who've migrated existing applications to JDK 9.  This can
> be a non-trivial task, involving the iterative construction of a set of
> command-line workarounds, since some popular libraries and frameworks use
> the core reflection API to access internal APIs of the JDK.  On JDK 9
> such components no longer work out-of-the-box because most internal APIs
> are now strongly encapsulated in order to improve the integrity of the
> platform.
>
> In order to make it easier to adjust to the strong encapsulation of
> internal APIs in JDK 9, in both the short term and over the long term,
> we've introduced a temporary "big kill switch", described below.  This
> switch will only be supported in JDK 9.  It's already available in the
> current Jigsaw EA builds [1], and will shortly be merged into JDK 9.
>
> Background
> ----------
>
> Diagnosing the failures of libraries and frameworks that access internal
> APIs of the JDK and then working around those failures is possible, but
> tedious.  The initial symptom is usually an `IllegalAccessException` or
> an `InaccessibleObjectException`.  The workaround is to use an
> `--add-opens` command-line option whose arguments are based upon the
> information shown in the exception message.  A large application that
> uses several reflective components may suffer many such failures, so this
> process may have to be repeated many times, resulting in a long list of
> command-line options that will typically be recorded in a launch script
> or an @-file.
>
> These options are tedious to construct but, additionally, there's a
> significant risk that once they're baked into a file they'll never be
> removed, even long after the problems that they work around have been
> resolved.  An application that needs some workarounds today might not
> benefit as well as it could from strong encapsulation in the long term,
> after all of the reflective components that it uses have been fixed to
> replace their use of JDK-internal APIs with proper exported APIs.
>
> To help the entire ecosystem migrate to the modular Java platform we want
> to motivate such fixes as strongly as we possibly can, especially in the
> near term.  We can do that by making it easier for you to get an existing
> application to work while also ensuring that the run-time system reports
> illegal reflective-access operations in a way that allows you to identify
> which libraries and frameworks on your class path require illegal access
> in order to work.  If a component's maintainers have already released a
> new, fixed version that no longer requires illegal access then you can
> consider upgrading to that version.  If, however, a component still needs
> to be fixed then we encourage you to contact its maintainers and ask them
> to stop using JDK-internal APIs [2].
>
> With all this in mind we've introduced a "big kill switch", described
> below, together with some new warning messages.  This switch will be
> supported in JDK 9, in order to enable application migration and motivate
> fixes to libraries and frameworks.  It will not be supported in JDK 10 or
> later releases, in order to motivate fixes to libraries and frameworks
> sooner rather than later.  If this switch were to be supported long-term
> then that would tend to reduce the pressure on maintainers to fix their
> components.
>
> New command-line option: `--permit-illegal-access`
> --------------------------------------------------
>
> If this option is present at run time then it will permit illegal access
> operations by code in any unnamed module (i.e., code on the class path)
> to members of types in any named module via the standard reflective APIs
> (`java.lang.reflect` and `java.lang.invoke`), including in particular
> those that would otherwise result in an `IllegalAccessException` or an
> `InaccessibleObjectException`.
>
>            +----------------------------------------------+
>            | This option will only be supported in JDK 9. |
>            |         It will be removed in JDK 10.        |
>            +----------------------------------------------+
>
> This option does not permit illegal access operations by code in named
> modules to members of types in other named modules; for that you must
> use the `--add-opens` or `--add-exports` options.  Those options can be
> combined with `--permit-illegal-access`.
>
> Warnings of illegal reflective-access operations
> ------------------------------------------------
>
> When an illegal reflective access operation succeeds due to the use of
> the `--permit-illegal-access` option, or the use of an `--add-opens` or
> `--add-exports` option, then a warning message of the following form is
> written to the error stream:
>
>    WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)
>
> where:
>
>  - $PERPETRATOR is the fully-qualified name of the type containing
>    the code that invoked the reflective operation in question plus
>    the code source (i.e., JAR-file path), if available,
>
>  - $VICTIM is a string that describes the member being accessed,
>    including the fully-qualified name of the enclosing type, and
>
>  - $OPTION is the name of the command-line option that enabled this
>    access, when that can be determined, or the first one of those
>    options if more than one option had that effect.
>
> The run-time system attempts to suppress duplicate warnings for the same
> $PERPETRATOR and $VICTIM, but it's not always practical to do so.
>
> For deeper diagnosis you can request a stack trace on each such warning
> by setting the system property `sun.reflect.debugModuleAccessChecks` to
> the value `access`, though this detail might change.  (That property can
> also be helpful to diagnose mysterious failures due to illegal-access
> exceptions that are caught and suppressed.)
>
> In addition to displaying a warning on each illegal access operation, the
> run-time system also shows new initial warning messages at startup time.
> If `--permit-illegal-access` is used then a warning reports the imminent
> demise of that option in the next major release.  If either `--add-opens`
> or `--add-exports` are used then a warning reports a count of each type
> of option used (i.e., opens vs. exports).
>
> Here are some examples of these messages, from running Jython on a very
> recent Jigsaw build:
>
>  $ java --permit-illegal-access -jar jython-standalone-2.7.0.jar
>  WARNING: --permit-illegal-access will be removed in the next major release
>  WARNING: Illegal access by jnr.posix.JavaLibCHelper
>  (file:/tmp/jython-standalone-2.7.0.jar) to method sun.nio.ch.SelChImpl.getFD()
>  (permitted by --permit-illegal-access)
>  WARNING: Illegal access by jnr.posix.JavaLibCHelper
>  (file:/tmp/jython-standalone-2.7.0.jar) to field sun.nio.ch.FileChannelImpl.fd
>  (permitted by --permit-illegal-access)
>  WARNING: Illegal access by jnr.posix.JavaLibCHelper
>  (file:/tmp/jython-standalone-2.7.0.jar) to field java.io.FileDescriptor.fd
>  (permitted by --permit-illegal-access)
>  WARNING: Illegal access by org.python.core.PySystemState
>  (file:/tmp/jython-standalone-2.7.0.jar) to method java.io.Console.encoding()
>  (permitted by --permit-illegal-access)
>  Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)
>  [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java9-internal
>  Type "help", "copyright", "credits" or "license" for more information.
>  >>> ^D
>  $
>
>
> [1] http://openjdk.java.net/projects/jigsaw/ea
> [2] This will usually but not always be possible, since there are still a
>    few critical internal APIs without exported replacements, per JEP 260
>     (http://openjdk.java.net/jeps/260).
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Keimpe Bronkhorst
In reply to this post by mark.reinhold
Mark,

How do we (Oracle JDeveloper) turn these illegal reflective-access
operation warnings off. We don't want or need these warnings when
running JDeveloper except during specific developer sessions.

BTW, the big kill switch doesn't seem useful, it just hides everything
that needs work.

Keimpe Bronkhorst


On 3/21/2017 11:57 AM, [hidden email] wrote:

> Warnings of illegal reflective-access operations
> ------------------------------------------------
>
> When an illegal reflective access operation succeeds due to the use of
> the `--permit-illegal-access` option, or the use of an `--add-opens` or
> `--add-exports` option, then a warning message of the following form is
> written to the error stream:
>
>      WARNING: Illegal access by $PERPETRATOR to $VICTIM (permitted by $OPTION)
>
> where:
>
>    - $PERPETRATOR is the fully-qualified name of the type containing
>      the code that invoked the reflective operation in question plus
>      the code source (i.e., JAR-file path), if available,
>
>    - $VICTIM is a string that describes the member being accessed,
>      including the fully-qualified name of the enclosing type, and
>
>    - $OPTION is the name of the command-line option that enabled this
>      access, when that can be determined, or the first one of those
>      options if more than one option had that effect.
>
> The run-time system attempts to suppress duplicate warnings for the same
> $PERPETRATOR and $VICTIM, but it's not always practical to do so.
>
> For deeper diagnosis you can request a stack trace on each such warning
> by setting the system property `sun.reflect.debugModuleAccessChecks` to
> the value `access`, though this detail might change.  (That property can
> also be helpful to diagnose mysterious failures due to illegal-access
> exceptions that are caught and suppressed.)
>
> In addition to displaying a warning on each illegal access operation, the
> run-time system also shows new initial warning messages at startup time.
> If `--permit-illegal-access` is used then a warning reports the imminent
> demise of that option in the next major release.  If either `--add-opens`
> or `--add-exports` are used then a warning reports a count of each type
> of option used (i.e., opens vs. exports).
>
> Here are some examples of these messages, from running Jython on a very
> recent Jigsaw build:
>
>    $ java --permit-illegal-access -jar jython-standalone-2.7.0.jar
>    WARNING: --permit-illegal-access will be removed in the next major release
>    WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to method sun.nio.ch.SelChImpl.getFD() (permitted by --permit-illegal-access)
>    WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field sun.nio.ch.FileChannelImpl.fd (permitted by --permit-illegal-access)
>    WARNING: Illegal access by jnr.posix.JavaLibCHelper (file:/tmp/jython-standalone-2.7.0.jar) to field java.io.FileDescriptor.fd (permitted by --permit-illegal-access)
>    WARNING: Illegal access by org.python.core.PySystemState (file:/tmp/jython-standalone-2.7.0.jar) to method java.io.Console.encoding() (permitted by --permit-illegal-access)
>    Jython 2.7.0 (default:9987c746f838, Apr 29 2015, 02:25:11)
>    [OpenJDK 64-Bit Server VM (Oracle Corporation)] on java9-internal
>    Type "help", "copyright", "credits" or "license" for more information.
>    >>> ^D
>    $
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Jochen Theodorou
In reply to this post by Alan Bateman
On 22.03.2017 12:23, Alan Bateman wrote:
[...]

> I assume your interest is Groovy where there are indeed some warnings
> that look like Groovy is to blame. For example, I get the following with
> a trivial script containing`println "Hello world!"` :
>
> $ groovy hello.groovy
> NOTE: Picked up the following options via JDK_JAVA_OPTIONS:
>    --permit-illegal-access
> WARNING: --permit-illegal-access will be removed in the next major release
> WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
> java.lang.Object.finalize() (permitted by --permit-illegal-access)
> WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
> java.lang.Object.clone() (permitted by --permit-illegal-access)
> WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
> java.lang.Object.registerNatives() (permitted by --permit-illegal-access)
> WARNING: Illegal access by org.codehaus.groovy.vmplugin.v7.Java7$1
> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to constructor
> java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) (permitted by
> --permit-illegal-access)
> Hello world!
>
> The user's script is innocent but maybe you are saying that if Groovy
> spins code and that code calls back into the Groovy runtime to do its
> dirty work then Groovy will be blamed?

the warnings you are seeing are from the groovy runtime trying to
determine - in a pre JDK9 compatible way - if it is allowed to call the
methods on Objecz. This happens in a first step by setAccessible on all
methods using the array version, and if that fails by using the normal
setAccessible on each method (which will contribute to much longer
startup times). At no point our runtime or the program you wrote are
trying to call finalize, clone or registerNatives.

The only one we are to be blamed for is
MethodHandles$Lookup(java.lang.Class,int), for which I haven´t
implemented the alternative yet

If you did other programs... like for example running a gradle build...
you would see many many more such lines

> On the question of performance then it may have an impact, it might not
> as it depends on how much illegal access is going on. We see some
> libraries doing some hacking with setAccessible(true) at startup and
> then re-using the AccessibleObject over and over, they will hardly
> notice anything.

So it will not be on a invocation base with some kind of check if that
has been locked already? Then I misunderstood

> In the case of Groovy then running with
> JDK_JAVA_OPTIONS="-Dsun.reflect.debugModuleAccessChecks=true" reveals
> several places where  InaccessibleObjectException is being thrown and
> silently ignored, e.g:

silently and purposely ignored.

> java.lang.reflect.InaccessibleObjectException: Unable to make private
> java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) accessible:
> module java.base does not "opens java.lang.invoke" to unnamed module
> @43bd930a
>      at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:337)
>
>      at
> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:281)
>
>      at
> java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:192)
>
>      at
> java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:185)
>      at org.codehaus.groovy.vmplugin.v7.Java7$1.run(Java7.java:53)
>      at java.base/java.security.AccessController.doPrivileged(Native
> Method)
>      at org.codehaus.groovy.vmplugin.v7.Java7.<clinit>(Java7.java:50)
>      at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>      at
> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
>      at
> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
>      at
> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
>      at java.base/java.lang.Class.newInstance(Class.java:558)
>      at
> org.codehaus.groovy.vmplugin.VMPluginFactory.createPlugin(VMPluginFactory.java:59)
>
>      at
> org.codehaus.groovy.vmplugin.VMPluginFactory.<clinit>(VMPluginFactory.java:40)
>
>      at
> org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:102)
>
>      at
> org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:74)
>
>      at groovy.lang.GroovySystem.<clinit>(GroovySystem.java:36)
>      at
> org.codehaus.groovy.runtime.InvokerHelper.<clinit>(InvokerHelper.java:66)
>      at groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)
>      at groovy.lang.Binding.<init>(Binding.java:35)
>      at groovy.lang.GroovyShell.<init>(GroovyShell.java:74)
>      at groovy.ui.GroovyMain.processOnce(GroovyMain.java:651)
>      at groovy.ui.GroovyMain.run(GroovyMain.java:384)
>      at groovy.ui.GroovyMain.process(GroovyMain.java:370)
>      at groovy.ui.GroovyMain.processArgs(GroovyMain.java:129)
>      at groovy.ui.GroovyMain.main(GroovyMain.java:109)
>      at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
>      at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>
>      at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
>      at java.base/java.lang.reflect.Method.invoke(Method.java:563)
>      at
> org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
>      at
> org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
> Hello world!
>
> So it might be that the cost of these exceptions is already high and
> that doing illegal access (with --permit-illegal-access) will not be
> noticed while the issues are being fixed.

So you tell me that if I add an add-opens to java.base for all unnamed
modules, I will still pay a performance penalty here? on each method
invocation or only once?

Also... be assured that people will notice, even if that means only that
Groovy is much slower on JDK9 compared to JDK8

bye Jochen

Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Claes Redestad


On 2017-03-22 22:07, Jochen Theodorou wrote:

> On 22.03.2017 12:23, Alan Bateman wrote:
> [...]
>> I assume your interest is Groovy where there are indeed some warnings
>> that look like Groovy is to blame. For example, I get the following with
>> a trivial script containing`println "Hello world!"` :
>>
>> $ groovy hello.groovy
>> NOTE: Picked up the following options via JDK_JAVA_OPTIONS:
>>    --permit-illegal-access
>> WARNING: --permit-illegal-access will be removed in the next major
>> release
>> WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
>> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
>> java.lang.Object.finalize() (permitted by --permit-illegal-access)
>> WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
>> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
>> java.lang.Object.clone() (permitted by --permit-illegal-access)
>> WARNING: Illegal access by org.codehaus.groovy.reflection.CachedClass
>> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to method
>> java.lang.Object.registerNatives() (permitted by --permit-illegal-access)
>> WARNING: Illegal access by org.codehaus.groovy.vmplugin.v7.Java7$1
>> (file:/groovy-2.4.10/lib/groovy-2.4.10.jar) to constructor
>> java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) (permitted by
>> --permit-illegal-access)
>> Hello world!
>>
>> The user's script is innocent but maybe you are saying that if Groovy
>> spins code and that code calls back into the Groovy runtime to do its
>> dirty work then Groovy will be blamed?
>
> the warnings you are seeing are from the groovy runtime trying to
> determine - in a pre JDK9 compatible way - if it is allowed to call the
> methods on Objecz. This happens in a first step by setAccessible on all
> methods using the array version, and if that fails by using the normal
> setAccessible on each method (which will contribute to much longer
> startup times). At no point our runtime or the program you wrote are
> trying to call finalize, clone or registerNatives.

For determining capabilities in this manner it might be possible to
use the new AccessibleObject.canAccess[1], which should avoid any
warnings?

/Claes

[1]
http://download.java.net/java/jdk9/docs/api/java/lang/reflect/AccessibleObject.html#canAccess-java.lang.Object-


>
> The only one we are to be blamed for is
> MethodHandles$Lookup(java.lang.Class,int), for which I haven´t
> implemented the alternative yet
>
> If you did other programs... like for example running a gradle build...
> you would see many many more such lines
>
>> On the question of performance then it may have an impact, it might not
>> as it depends on how much illegal access is going on. We see some
>> libraries doing some hacking with setAccessible(true) at startup and
>> then re-using the AccessibleObject over and over, they will hardly
>> notice anything.
>
> So it will not be on a invocation base with some kind of check if that
> has been locked already? Then I misunderstood
>
>> In the case of Groovy then running with
>> JDK_JAVA_OPTIONS="-Dsun.reflect.debugModuleAccessChecks=true" reveals
>> several places where  InaccessibleObjectException is being thrown and
>> silently ignored, e.g:
>
> silently and purposely ignored.
>
>> java.lang.reflect.InaccessibleObjectException: Unable to make private
>> java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) accessible:
>> module java.base does not "opens java.lang.invoke" to unnamed module
>> @43bd930a
>>      at
>> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:337)
>>
>>
>>      at
>> java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:281)
>>
>>
>>      at
>> java.base/java.lang.reflect.Constructor.checkCanSetAccessible(Constructor.java:192)
>>
>>
>>      at
>> java.base/java.lang.reflect.Constructor.setAccessible(Constructor.java:185)
>>
>>      at org.codehaus.groovy.vmplugin.v7.Java7$1.run(Java7.java:53)
>>      at java.base/java.security.AccessController.doPrivileged(Native
>> Method)
>>      at org.codehaus.groovy.vmplugin.v7.Java7.<clinit>(Java7.java:50)
>>      at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native
>>
>> Method)
>>      at
>> java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>
>>
>>      at
>> java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>
>>
>>      at
>> java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:488)
>>      at java.base/java.lang.Class.newInstance(Class.java:558)
>>      at
>> org.codehaus.groovy.vmplugin.VMPluginFactory.createPlugin(VMPluginFactory.java:59)
>>
>>
>>      at
>> org.codehaus.groovy.vmplugin.VMPluginFactory.<clinit>(VMPluginFactory.java:40)
>>
>>
>>      at
>> org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:102)
>>
>>
>>      at
>> org.codehaus.groovy.runtime.metaclass.MetaClassRegistryImpl.<init>(MetaClassRegistryImpl.java:74)
>>
>>
>>      at groovy.lang.GroovySystem.<clinit>(GroovySystem.java:36)
>>      at
>> org.codehaus.groovy.runtime.InvokerHelper.<clinit>(InvokerHelper.java:66)
>>      at
>> groovy.lang.GroovyObjectSupport.<init>(GroovyObjectSupport.java:34)
>>      at groovy.lang.Binding.<init>(Binding.java:35)
>>      at groovy.lang.GroovyShell.<init>(GroovyShell.java:74)
>>      at groovy.ui.GroovyMain.processOnce(GroovyMain.java:651)
>>      at groovy.ui.GroovyMain.run(GroovyMain.java:384)
>>      at groovy.ui.GroovyMain.process(GroovyMain.java:370)
>>      at groovy.ui.GroovyMain.processArgs(GroovyMain.java:129)
>>      at groovy.ui.GroovyMain.main(GroovyMain.java:109)
>>      at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>>      at
>> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>>
>>
>>      at
>> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>>
>>
>>      at java.base/java.lang.reflect.Method.invoke(Method.java:563)
>>      at
>> org.codehaus.groovy.tools.GroovyStarter.rootLoader(GroovyStarter.java:109)
>>
>>      at
>> org.codehaus.groovy.tools.GroovyStarter.main(GroovyStarter.java:131)
>> Hello world!
>>
>> So it might be that the cost of these exceptions is already high and
>> that doing illegal access (with --permit-illegal-access) will not be
>> noticed while the issues are being fixed.
>
> So you tell me that if I add an add-opens to java.base for all unnamed
> modules, I will still pay a performance penalty here? on each method
> invocation or only once?
>
> Also... be assured that people will notice, even if that means only that
> Groovy is much slower on JDK9 compared to JDK8
>
> bye Jochen
>
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

mark.reinhold
In reply to this post by mark.reinhold
Thanks to everyone for all the feedback on this topic.

It appears that issuing warning messages for illegal-access operations
enabled by the precise `--add-opens` and `--add-exports` options is a
bit too aggressive, at least for JDK 9.  Perhaps we can enable that in
JDK 10 after there's been more time for libraries, frameworks, and even
the JDK itself to adjust to the realities of strong encapsulation.

For now I suggest that we revert to the previous behavior of these two
options, so that they do not cause warning messages to be issued.  The
new `--permit-illegal-access` option will continue to generate warning
messages, as proposed.  If those messages are a problem in a particular
scenario then they can be eliminated by switching to an appropriate set
of `--add-opens` options, which can be constructed from the information
contained in those messages.

Comments?

- Mark
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Remi Forax
good to me.

Rémi


On March 23, 2017 4:30:00 AM GMT+01:00, [hidden email] wrote:

>Thanks to everyone for all the feedback on this topic.
>
>It appears that issuing warning messages for illegal-access operations
>enabled by the precise `--add-opens` and `--add-exports` options is a
>bit too aggressive, at least for JDK 9.  Perhaps we can enable that in
>JDK 10 after there's been more time for libraries, frameworks, and even
>the JDK itself to adjust to the realities of strong encapsulation.
>
>For now I suggest that we revert to the previous behavior of these two
>options, so that they do not cause warning messages to be issued.  The
>new `--permit-illegal-access` option will continue to generate warning
>messages, as proposed.  If those messages are a problem in a particular
>scenario then they can be eliminated by switching to an appropriate set
>of `--add-opens` options, which can be constructed from the information
>contained in those messages.
>
>Comments?
>
>- Mark

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Alan Bateman
In reply to this post by Jochen Theodorou
On 22/03/2017 21:07, Jochen Theodorou wrote:

> the warnings you are seeing are from the groovy runtime trying to
> determine - in a pre JDK9 compatible way - if it is allowed to call
> the methods on Objecz. This happens in a first step by setAccessible
> on all methods using the array version, and if that fails by using the
> normal setAccessible on each method (which will contribute to much
> longer startup times). At no point our runtime or the program you
> wrote are trying to call finalize, clone or registerNatives.
This sounds like Object.class.getDeclaredMethods() and then invoking
setAccessible(true) on every method, is that right? If so then it will
trigger warnings for non-public methods. Is there any reason why this
code couldn't just look at the modifiers? Claes mentions the new
canAccess which might be useful going forward.

>
> The only one we are to be blamed for is
> MethodHandles$Lookup(java.lang.Class,int), for which I haven´t
> implemented the alternative yet
>
> If you did other programs... like for example running a gradle
> build... you would see many many more such lines
Lots of lines but not clear to me that it's Gradle that wants to hack
into every non-public member of java.util.ArrayList.

$ gradle
NOTE: Picked up the following options via JDK_JAVA_OPTIONS:
   --permit-illegal-access
WARNING: --permit-illegal-access will be removed in the next major release
WARNING: Illegal access by org.gradle.internal.reflect.JavaMethod
(file:/gradle-3.4.1/lib/gradle-base-services-3.4.1.jar) to method
java.lang.ClassLoader.getPackages() (permitted by --permit-illegal-access)
Starting a Gradle Daemon, 1 stopped Daemon could not be reused, use
--status for details
WARNING: Illegal access by org.gradle.internal.reflect.JavaMethod
(file:/gradle-3.4.1/lib/gradle-base-services-3.4.1.jar) to method
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
(permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.lang.Object.finalize() (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.lang.Object.clone() (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.lang.Object.registerNatives() (permitted by --permit-illegal-access)
WARNING: Illegal access by org.codehaus.groovy.vmplugin.v7.Java7$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to constructor
java.lang.invoke.MethodHandles$Lookup(java.lang.Class,int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by org.gradle.internal.reflect.JavaMethod
(file:/gradle-3.4.1/lib/gradle-base-services-3.4.1.jar) to method
java.lang.ClassLoader.defineClass(java.lang.String,byte[],int,int)
(permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.AbstractCollection.hugeCapacity(int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.AbstractCollection.finishToArray(java.lang.Object[],java.util.Iterator)
(permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.AbstractList.subListRangeCheck(int,int,int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.AbstractList.removeRange(int,int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.AbstractList.rangeCheckForAdd(int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.AbstractList.outOfBoundsMsg(int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.add(java.lang.Object,java.lang.Object[],int)
(permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.access$000(java.util.ArrayList) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.elementAt(java.lang.Object[],int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.readObject(java.io.ObjectInputStream) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.writeObject(java.io.ObjectOutputStream) (permitted
by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.newCapacity(int) (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.hugeCapacity(int) (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.removeIf(java.util.function.Predicate,int,int)
(permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.removeRange(int,int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.rangeCheckForAdd(int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.outOfBoundsMsg(int,int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.outOfBoundsMsg(int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.elementData(int) (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.grow() (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.grow(int) (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.nBits(int) (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.setBit(long[],int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.isClear(long[],int) (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.shiftTailOverGap(java.lang.Object[],int,int)
(permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.checkInvariants() (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.fastRemove(int) (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$3$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to method
java.util.ArrayList.batchRemove(java.util.Collection,boolean,int,int)
(permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.AbstractCollection.MAX_ARRAY_SIZE (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.AbstractList.modCount (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.ArrayList.serialVersionUID (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.ArrayList.DEFAULT_CAPACITY (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.ArrayList.EMPTY_ELEMENTDATA (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.ArrayList.DEFAULTCAPACITY_EMPTY_ELEMENTDATA (permitted by
--permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.ArrayList.elementData (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.ArrayList.size (permitted by --permit-illegal-access)
WARNING: Illegal access by
org.codehaus.groovy.reflection.CachedClass$1$1
(file:/gradle-3.4.1/lib/groovy-all-2.4.7.jar) to field
java.util.ArrayList.MAX_ARRAY_SIZE (permitted by --permit-illegal-access)
:help

Welcome to Gradle 3.4.1.

To run a build, run gradle <task> ...

To see a list of available tasks, run gradle tasks

To see a list of command-line options, run gradle --help

To see more detail about a task, run gradle help --task <task>

BUILD SUCCESSFUL

Total time: 3.584 secs


>
> So it will not be on a invocation base with some kind of check if that
> has been locked already? Then I misunderstood
Once setAccessible succeeds then further use of that AccessiblecObject
does not do an access check so there won't be subsequent warnings.


> :
>
> So you tell me that if I add an add-opens to java.base for all unnamed
> modules, I will still pay a performance penalty here? on each method
> invocation or only once?
A discussion point is whether the precise options (--add-exports and
--add-opens) triggers warnings in JDK 9 or not. If they do not then
there will be no performance overhead.

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: Better tools for adjusting to strong encapsulation

Jochen Theodorou
In reply to this post by Claes Redestad


On 23.03.2017 01:16, Claes Redestad wrote:
[...]

>> the warnings you are seeing are from the groovy runtime trying to
>> determine - in a pre JDK9 compatible way - if it is allowed to call the
>> methods on Objecz. This happens in a first step by setAccessible on all
>> methods using the array version, and if that fails by using the normal
>> setAccessible on each method (which will contribute to much longer
>> startup times). At no point our runtime or the program you wrote are
>> trying to call finalize, clone or registerNatives.
>
> For determining capabilities in this manner it might be possible to
> use the new AccessibleObject.canAccess[1], which should avoid any
> warnings?
>
> /Claes
>
> [1]
> http://download.java.net/java/jdk9/docs/api/java/lang/reflect/AccessibleObject.html#canAccess-java.lang.Object-

I guess so. I mean normally we request that you open up the Objects to
the runtime and the runtime then decides by itself if class X has access
to class Y via runtime. canAccess will avoid the warnings, and
completely avoid them I think for the case of everything being on the
classpath. It will not avoid the add-opens required for the runtime to
access objects from class X in module M to class Y in the same Module
via Groovy runtime. Because even if we use MethodHandles, the handle is
still produced from a Method object in reflection and that will mean we
still require the runtime accessing this.

And actually I don't think we can exclude the reflective part, because
we have to do runtime method selection. We cannot query for a method
directly, we have to query for all methods of a certain name, which is
impossible, so we have to take a look at all methods of a class. And I
mean independent of the accessibility modifier.

So does canAccess help? Yes, in the classpath scenario, not in the
module scenario. But at least that scenario is currently not likely

bye Jochen
12