Reflection: how does one access a protected member in a superclass reflectively?

classic Classic list List threaded Threaded
30 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Peter Levart


On 01/22/2018 12:18 PM, Alan Bateman wrote:

> On 22/01/2018 09:58, Peter Levart wrote:
>> :
>>
>> The 2nd problem is not trivial as you want to access a protected
>> member on behalf of some other sub-class of the member's declaring
>> class which is not cooperating (voluntarily handing you an instance
>> of its Lookup object). This currently requires the package containing
>> the member's declaring class to be opened at least to you (the Rexx
>> interpreter) and using the member.setAccessible(true) trick or
>> MethodHandles.privateLookupIn(declaringClass) equivalent for method
>> handles. Which is awkward because libraries packed as modules would
>> normally not specify that in their module descriptors and system
>> modules don't either. So you are left with either --add-opens command
>> line switches or deploying a javaagent to the JVM and using it's API
>> point java.lang.instrument.Instrumentation#redefineModule to add
>> opens to modules that way. Both approaches are not elegant, but
>> that's what is currently available, I think.
>>
> I suspect it may be just a misunderstanding. One of Rony's mails had
> this example:
>
> o=.bsf~new("mtest3.Class03A")             -- create Java object, get
> and assign proxy ooRexx object
> say "o:" o "o~myClassName:" o~myClassName -- get (static) field value
> in "mtest1.Class01A", accessible via inheritance
>
> I read this as the Rexx script doing the equivalent of "new
> mtest3.Class03A()", in which case should be no expectation that
> protected members are accessible to the Rexx code.
>
> -Alan.

I was asking myself the same question, yes. If Rony wants to call
protected methods on "behalf" of subclasses, then Rexx runtime has to
have features to subclass existing Java classes.

@Rony: So why does Rexx script want to call protected members? Is that
because Rexx can extend existing Java classes?

Regards, Peter

Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Peter Levart
In reply to this post by Peter Levart
@Potential implementer of below trick BEWARE!

While I tried to be smart by "injecting" special java agent powers into
designated trusted class, the presented mechanism is NOT SAFE as I
identify the class only by it's name. An attacker might create it's own
pair of classes with same names (runtime.Opener, runtime.Opener$Holder)
loaded in some isolated class loader and the presented agent will
happily inject the powers into the attacker's runtime.Opener$Holder. For
safe variant of such agent, the Opener$Holder class would have to prove
the authenticity to the Agent 1st. The mechanism of such authentication
will be left as an exercise to the reader...

Regards, Peter

On 01/22/2018 02:04 PM, Peter Levart wrote:

> Hi Rony,
>
> On 01/22/2018 10:58 AM, Peter Levart wrote:
>> The 2nd problem is not trivial as you want to access a protected
>> member on behalf of some other sub-class of the member's declaring
>> class which is not cooperating (voluntarily handing you an instance
>> of its Lookup object). This currently requires the package containing
>> the member's declaring class to be opened at least to you (the Rexx
>> interpreter) and using the member.setAccessible(true) trick or
>> MethodHandles.privateLookupIn(declaringClass) equivalent for method
>> handles. Which is awkward because libraries packed as modules would
>> normally not specify that in their module descriptors and system
>> modules don't either. So you are left with either --add-opens command
>> line switches or deploying a javaagent to the JVM and using it's API
>> point java.lang.instrument.Instrumentation#redefineModule to add
>> opens to modules that way. Both approaches are not elegant, but
>> that's what is currently available, I think.
>
> Just one more thing... While solutions for tackling the 2nd problem
> might seem attractive to use for solving the 1st problem too, I would
> recommend not doing that. Opening all the packages of public API(s)
> might inhibit possible optimizations John Rose has been talking about.
> For reflective access to public API(s) you don't need to open the
> packages because public API(s) are in exported packages and all the
> "static" types that are needed to access them (field types, method
> return an parameter types) are also guaranteed to be part of public
> API(s) (at least good modules guarantee that). Public API(s) are
> transitively public. For public API(s) it is just a matter of finding
> the accessible member in the hierarchy where there will always be at
> least one.
>
> For the 2nd problem, the main difficulty seems to be how to open just
> the packages that are involved in accessing the protected members on
> behalf of subclasses hoping that those packages are in minority.
> Here's one trick by using javaagent. Suppose your Rexx runtime had the
> following nonpublic class in its heart:
>
> package runtime;
>
> import java.util.function.BiConsumer;
>
> class Opener {
>     private static class Holder {
>         static BiConsumer<Class<?>, Module> opener;
>     }
>
>     static void openPackageOfTo(Class<?> clazz, Module module) {
>         Holder.opener.accept(clazz, module);
>     }
> }
>
>
> Now if you start the JVM by supplying the -javaagent:agent.jar command
> line in addition to everything else and pack the following compiled
> code into agent.jar with the following MANIFEST:
>
> Manifest-Version: 1.0
> Premain-Class: agent.Agent
>
> ---
> package agent;
>
> import java.lang.instrument.ClassFileTransformer;
> import java.lang.instrument.IllegalClassFormatException;
> import java.lang.instrument.Instrumentation;
> import java.lang.reflect.Field;
> import java.security.ProtectionDomain;
> import java.util.Map;
> import java.util.Set;
> import java.util.function.BiConsumer;
>
> public class Agent {
>     private static final String OPENER_BINARY_CLASS_NAME =
> "runtime/Opener";
>     private static final String HOLDER_CLASS_NAME =
> "runtime.Opener$Holder";
>     private static final String OPENER_FIELD_NAME = "opener";
>
>     private static Instrumentation instrumentation;
>
>     public static void premain(String agentArgs, Instrumentation inst) {
>         instrumentation = inst;
>         inst.addTransformer(new ClassFileTransformer() {
>             @Override
>             public byte[] transform(Module module,
>                                     ClassLoader loader,
>                                     String className,
>                                     Class<?> classBeingRedefined,
>                                     ProtectionDomain protectionDomain,
>                                     byte[] classfileBuffer) throws
> IllegalClassFormatException {
>
>                 // when runtime.Opener starts loading...
>                 if (className.equals(OPENER_BINARY_CLASS_NAME) &&
> classBeingRedefined == null) {
>                     try {
>                         // ...load runtime.Opener$Holder upfront using
> the same classloader
>                         Class<?> holderClass =
> Class.forName(HOLDER_CLASS_NAME, true, loader);
>                         // find the runtime.Opener$Holder#opener field
>                         Field openerField =
> holderClass.getDeclaredField(OPENER_FIELD_NAME);
>                         // and make it accessible
>                         openerField.setAccessible(true);
>                         // inject the BiConsumer
>                         openerField.set(null, (BiConsumer<Class<?>,
> Module>) Agent::openPackageOfTo);
>                     } catch (ReflectiveOperationException e) {
>                         throw new InternalError(e);
>                     }
>                 }
>
>                 // perform no actual transformation
>                 return null;
>             }
>         }, false);
>     }
>
>     static void openPackageOfTo(Class<?> clazz, Module module) {
>         String pn = clazz.getPackageName();
>         System.out.println("Opening package " + pn + " to " + module);
>         instrumentation.redefineModule(
>             clazz.getModule(),
>             Set.of(),
>             Map.of(),
>             Map.of(pn, Set.of(module)), // extra opens
>             Set.of(),
>             Map.of()
>         );
>     }
> }
>
>
> With such Rexx runtime specific helper agent jar you can extend the
> controlled power of java agent to your Rexx interpreter so you now
> have the power to dynamically add just those opens that are absolutely
> necessary for performing the accesses to protected members.
>
>
> Regards, Peter
>

Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Rony G. Flatscher
In reply to this post by Alan Bateman
On 19.01.2018 15:38, Alan Bateman wrote:

> On 18/01/2018 15:07, Rony G. Flatscher wrote:
>> An attachment in the email has been found to contain executable code and has been removed.
>>
>> File removed : java9modules.zip, zip,cmd
>> ----------------------------------------------------------------------------------------------------
>>
>> Dear Alan:
>>
>> tried to come up with a "cleaner" version to zip it up, however the error would not occur there.
> The attachment was dropped too.
OK, next time, I will use Dropbox or the like instead.

> When you say "the error would not occur there" then do you mean it won't compile? I wouldn't
> expect the test to compile with a reference to the protected member so maybe the issue was that
> the code was compiled in a different way and so only found when you ran.
Looked into it and you are probably right, the class file was a few minutes older than the latest
version of the java file. Deleting the class file and trying to recompile yields the expected error
"...has protected access in Class01A..." and there is no public getter defined for that field in any
of the classes in the exported packages.

---rony


Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Roger Riggs
Hi,

Contributions need to be submitted using the OpenJDK infrastructure to
adhere to the IP requirements.
The mail lists shold pass attachments that are text and patches though
you may need to be
sure your mailer attaches them with the correct mime-types and/or
extensions.

Roger

p.s.
The current settings have pass_mime_types:

    multipart/mixed
    multipart/alternative
    text/plain
    text/x-diff
    text/x-patch
    message/rfc822
    multipart/signed



On 1/22/2018 10:18 AM, Rony G. Flatscher wrote:

> On 19.01.2018 15:38, Alan Bateman wrote:
>> On 18/01/2018 15:07, Rony G. Flatscher wrote:
>>> An attachment in the email has been found to contain executable code and has been removed.
>>>
>>> File removed : java9modules.zip, zip,cmd
>>> ----------------------------------------------------------------------------------------------------
>>>
>>> Dear Alan:
>>>
>>> tried to come up with a "cleaner" version to zip it up, however the error would not occur there.
>> The attachment was dropped too.
> OK, next time, I will use Dropbox or the like instead.
Sorry, only inline, attached or on cr.openjdk.java.net.

Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Rony G. Flatscher
In reply to this post by Peter Levart
Hi Peter,

thank you *very* much also for your kind explanations and even coming up with agent code to
demonstrate how one could use that approach!

In fact I have been doing a static analysis in the past (since Java 1.1) to determine whether
members should be accessible to Rexx, restricting access to public members and to protected
inherited members, if accessing them from a dynamically created subclass.

One thing I have to make sure is, that I keep compatible with the Java 1.6/6.0 baseline, as there
are some deployments where the shops are still employing that environment. This should not really be
a problem as I am rewriting the entire reflection part and will be able to employ different
approaches for pre-Java 9 deplyoments, where the reflection class employed is dependent on the Java
runtime version.

Currently I am exploring the implications of the new Java 9 module system, trying to adhere to its
rules. In essence the goal is to allow reflectively everything for Rexx peers that a compiled Java
program allows for in the Java 9 environment. For that the reflecting class (currently in the
unnamed module) goes up the inheritance tree until it finds an exported class and analyses it
reflectively.

Experimenting with variations of classes residing in different modules with different exports, it is
possible to mix-up the relationships, what gets exported to what, and when should protected members
in superclasses be accessible and when not (and yes, this part should belong to dynamically created
subclasses, which also need adjustments to the module system).

Best regards,

---rony


On 22.01.2018 10:58, Peter Levart wrote:

> Hi Rony,
>
> On 01/18/2018 04:11 PM, Rony G. Flatscher wrote:
>> On 18.01.2018 10:58, Alan Bateman wrote:
>>> On 17/01/2018 18:53, Rony G. Flatscher wrote:
>>>> :
>>>>
>>>> Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
>>>> module, but having a need to access public types from exported packages and get reflective access
>>>> to objects supertype's protected members?
>>> I think it would be better to start with public members as protected is complicated (and hasn't
>>> changed with modules once you establish the class declaring the member is accessible).
>>>
>>> For your example, you've got a reference to a java.awt.Graphics2D object, the actual
>>> implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
>>> public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
>>> bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
>>> superclass and interfaces to find a public class or interface that defines the target
>>> setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
>>> were non public.
>>>
>>> Can you extend this check to test if the class is in a package exported by its module. For the
>>> example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
>>> sun.java2d to everyone. Here is a code fragment to test this:
>>>
>>> Class<?> clazz = graphicsObj.getClass();
>>> boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
>>>
>>> (I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
>>>
>>> If you can incorporate this check into the bridge then I suspect you'll find most of the examples
>>> will work.
>> Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
>> find an accessible member in a superclass, which means to have a need to also access protected
>> members in the superclass) and that is actually my current approach. However, I started out with
>> reflecting Fields first and see, whether I can reflectively get access.
>>
>> The rewritten method resolution would follow next, which would allow me to tackle that warning and
>> see whether I can get rid of it. However, before going a wrong route I would like to learn what the
>> "official" Java 9 solution would be and try to implement that.
>>
>> ---rony
>
> Yes, I think you are dealing with two problems here which you have been using the same solution
> for in the past.
>
> The 1st thing you have been doing incorrectly for Java 9, as Alan explained, is the idiom:
> o.getClass().getMethod(...) and the 2nd is that you are trying to access protected members on
> behalf of some other class which is a subclass of the protected member's declaring class.
>
> The 1st problem has different solutions which are all doable in Java 9, since you are dealing
> within the confines of public types, public members and exported packages. One solution is to
> search for the most specific member in the inheritance hierarchy which is also accessible
> (declared in public type in exported package) which is what Alan suggests.
>
> There might also be another elegant solution which requires some re-design of your Rexx
> interpreter.  When you deal with reference values in Rexx (the values that refer to objects in
> Java), you could track not only the value itself but also the "static" type of that value. A
> reference value is always obtained either by calling a constructor, accessing a field (either
> static or instance), by calling a method (static or instance) or by accessing an element of some
> array:
>
> - calling constructor: the "static type" is the class upon which the constructor has been called
> - accessing a field: the "static type" is the type of the field (i.e. Field.getDeclaringClass())
> - calling a method: the "static type" is the return type of the method (i.e. Method.getReturnType())
> - accessing an element of some array: the "static type" is the array's "static type"'s component
> type (i.e. Class.getComponentType() invoked on array's "static type" Class).
>
> When you take the "static" type as the starting Class when searching for a public member with
> standard Class.getMethod() or Class.getField(), you would then get the correct publicly accessible
> reflected member. With a caveat that this only works when there's no generics involved. If there's
> generics, the logic to compute the correct static type is more involved and would sometimes
> require passing the generic type parameters (when invoking constructors of generic classes or
> generic methods) in the syntax of your Rexx language. So you may or may not want to do that.
> Perhaps some library for deep resolving could be of help here (Google Guava has some support for
> that). I guess searching for the most specific member in the hierarchy that is also accessible is
> your best bet currently if the goal is to be syntactically backwards compatible in the Rexx language.
>
> The 2nd problem is not trivial as you want to access a protected member on behalf of some other
> sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
> instance of its Lookup object). This currently requires the package containing the member's
> declaring class to be opened at least to you (the Rexx interpreter) and using the
> member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
> method handles. Which is awkward because libraries packed as modules would normally not specify
> that in their module descriptors and system modules don't either. So you are left with either
> --add-opens command line switches or deploying a javaagent to the JVM and using it's API point
> java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
> approaches are not elegant, but that's what is currently available, I think.
>
> Regards, Peter

Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Rony G. Flatscher
In reply to this post by Alan Bateman

On 22.01.2018 12:18, Alan Bateman wrote:

> On 22/01/2018 09:58, Peter Levart wrote:
>> :
>>
>> The 2nd problem is not trivial as you want to access a protected member on behalf of some other
>> sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
>> instance of its Lookup object). This currently requires the package containing the member's
>> declaring class to be opened at least to you (the Rexx interpreter) and using the
>> member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
>> method handles. Which is awkward because libraries packed as modules would normally not specify
>> that in their module descriptors and system modules don't either. So you are left with either
>> --add-opens command line switches or deploying a javaagent to the JVM and using it's API point
>> java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
>> approaches are not elegant, but that's what is currently available, I think.
>>
> I suspect it may be just a misunderstanding. One of Rony's mails had this example:
>
> o=.bsf~new("mtest3.Class03A")             -- create Java object, get and assign proxy ooRexx object
> say "o:" o "o~myClassName:" o~myClassName -- get (static) field value in "mtest1.Class01A",
> accessible via inheritance
>
> I read this as the Rexx script doing the equivalent of "new mtest3.Class03A()", in which case
> should be no expectation that protected members are accessible to the Rexx code.
Yes, you are right! There need to be a public member that is capable of accessing the protected ones
in this case.

Obviously I have used too many variants and mixed up the use cases, really sorry for the noise! :(

---rony
Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Rony G. Flatscher
In reply to this post by Roger Riggs
Hi,

On 22.01.2018 16:24, Roger Riggs wrote:
> Contributions need to be submitted using the OpenJDK infrastructure to adhere to the IP requirements.
Would you have a link which OpenJFDK infrastructure to use in this case?

> The mail lists shold pass attachments that are text and patches though you may need to be
> sure your mailer attaches them with the correct mime-types and/or extensions.
>
> Roger
>
> p.s.
> The current settings have pass_mime_types:
>
>    multipart/mixed
>    multipart/alternative
>    text/plain
>    text/x-diff
>    text/x-patch
>    message/rfc822
>    multipart/signed
Thanks for this information!

Regards,

---rony

> On 1/22/2018 10:18 AM, Rony G. Flatscher wrote:
>> On 19.01.2018 15:38, Alan Bateman wrote:
>>> On 18/01/2018 15:07, Rony G. Flatscher wrote:
>>>> An attachment in the email has been found to contain executable code and has been removed.
>>>>
>>>> File removed : java9modules.zip, zip,cmd
>>>> ----------------------------------------------------------------------------------------------------
>>>>
>>>>
>>>> Dear Alan:
>>>>
>>>> tried to come up with a "cleaner" version to zip it up, however the error would not occur there.
>>> The attachment was dropped too.
>> OK, next time, I will use Dropbox or the like instead.
> Sorry, only inline, attached or on cr.openjdk.java.net.
>

Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access ...

Roger Riggs
Hi,

Attaching the patch as a .patch or .txt file should be sufficient for
small contributions and example code.

Roger

On 1/22/2018 10:39 AM, Rony G. Flatscher wrote:

> Hi,
>
> On 22.01.2018 16:24, Roger Riggs wrote:
>> Contributions need to be submitted using the OpenJDK infrastructure to adhere to the IP requirements.
> Would you have a link which OpenJFDK infrastructure to use in this case?
>
>> The mail lists shold pass attachments that are text and patches though you may need to be
>> sure your mailer attaches them with the correct mime-types and/or extensions.
>>
>> Roger
>>
>> p.s.
>> The current settings have pass_mime_types:
>>
>>     multipart/mixed
>>     multipart/alternative
>>     text/plain
>>     text/x-diff
>>     text/x-patch
>>     message/rfc822
>>     multipart/signed
> Thanks for this information!
>
> Regards,
>
> ---rony
>
>> On 1/22/2018 10:18 AM, Rony G. Flatscher wrote:
>>> On 19.01.2018 15:38, Alan Bateman wrote:
>>>> On 18/01/2018 15:07, Rony G. Flatscher wrote:
>>>>> An attachment in the email has been found to contain executable code and has been removed.
>>>>>
>>>>> File removed : java9modules.zip, zip,cmd
>>>>> ----------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>> Dear Alan:
>>>>>
>>>>> tried to come up with a "cleaner" version to zip it up, however the error would not occur there.
>>>> The attachment was dropped too.
>>> OK, next time, I will use Dropbox or the like instead.
>> Sorry, only inline, attached or on cr.openjdk.java.net.
>>

Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Peter Levart
In reply to this post by Rony G. Flatscher
Hi Rony,

On 01/22/18 16:35, Rony G. Flatscher wrote:
> Hi Peter,
>
> thank you *very* much also for your kind explanations and even coming up with agent code to
> demonstrate how one could use that approach!
>
> In fact I have been doing a static analysis in the past (since Java 1.1) to determine whether
> members should be accessible to Rexx, restricting access to public members and to protected
> inherited members, if accessing them from a dynamically created subclass.

So you *are* loading dynamically generated subclasses. It's just that
their logic is implemented in Rexx scripting and therefore delegates
invocations to Rexx runtime and those invocations include invocations to
protected methods of their superclasses.

In that case, why don't you come up with some mechanism for dynamically
generated classes to hand over their Lookup(s) to Rexx runtime. For
example, the Rexx runtime could have the following public registration API:

---
package runtime;

import java.lang.invoke.MethodHandles;
import java.util.Map;
import java.util.concurrent.ConcurrentHashMap;

public class LookupRegistry {
     private static final Map<Class<?>, MethodHandles.Lookup> registry =
         new ConcurrentHashMap<>();

     public static void register(MethodHandles.Lookup lookup) {
         if (registry.putIfAbsent(lookup.lookupClass(), lookup) != null) {
             throw new IllegalStateException(
                 "Lookup for " + lookup.lookupClass() + " is already
registered");
         }
     }

     /* non-public */ static MethodHandles.Lookup getLookup(Class<?>
clazz) {
         return registry.get(clazz);
     }
}


...then each Rexx dynamically generated class could include the
following static initialization block:


public class GeneratedClass {
     static {
         runtime.LookupRegistry.register(
             java.lang.invoke.MethodHandles.lookup()
         );
     }
...


That way Rexx runtime could have access to Lookup(s) of each of it's
dynamically generated classes as soon as they are loaded and initialized
and can use a particular class's Lookup to lookup a MethodHandle for the
protected methods of its superclass. It can even simulate a
"super.method()" invocation, etc...


Regards, Peter

> One thing I have to make sure is, that I keep compatible with the Java 1.6/6.0 baseline, as there
> are some deployments where the shops are still employing that environment. This should not really be
> a problem as I am rewriting the entire reflection part and will be able to employ different
> approaches for pre-Java 9 deplyoments, where the reflection class employed is dependent on the Java
> runtime version.
>
> Currently I am exploring the implications of the new Java 9 module system, trying to adhere to its
> rules. In essence the goal is to allow reflectively everything for Rexx peers that a compiled Java
> program allows for in the Java 9 environment. For that the reflecting class (currently in the
> unnamed module) goes up the inheritance tree until it finds an exported class and analyses it
> reflectively.
>
> Experimenting with variations of classes residing in different modules with different exports, it is
> possible to mix-up the relationships, what gets exported to what, and when should protected members
> in superclasses be accessible and when not (and yes, this part should belong to dynamically created
> subclasses, which also need adjustments to the module system).
>
> Best regards,
>
> ---rony
>
>
> On 22.01.2018 10:58, Peter Levart wrote:
>> Hi Rony,
>>
>> On 01/18/2018 04:11 PM, Rony G. Flatscher wrote:
>>> On 18.01.2018 10:58, Alan Bateman wrote:
>>>> On 17/01/2018 18:53, Rony G. Flatscher wrote:
>>>>> :
>>>>>
>>>>> Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
>>>>> module, but having a need to access public types from exported packages and get reflective access
>>>>> to objects supertype's protected members?
>>>> I think it would be better to start with public members as protected is complicated (and hasn't
>>>> changed with modules once you establish the class declaring the member is accessible).
>>>>
>>>> For your example, you've got a reference to a java.awt.Graphics2D object, the actual
>>>> implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
>>>> public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
>>>> bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
>>>> superclass and interfaces to find a public class or interface that defines the target
>>>> setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
>>>> were non public.
>>>>
>>>> Can you extend this check to test if the class is in a package exported by its module. For the
>>>> example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
>>>> sun.java2d to everyone. Here is a code fragment to test this:
>>>>
>>>> Class<?> clazz = graphicsObj.getClass();
>>>> boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
>>>>
>>>> (I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
>>>>
>>>> If you can incorporate this check into the bridge then I suspect you'll find most of the examples
>>>> will work.
>>> Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
>>> find an accessible member in a superclass, which means to have a need to also access protected
>>> members in the superclass) and that is actually my current approach. However, I started out with
>>> reflecting Fields first and see, whether I can reflectively get access.
>>>
>>> The rewritten method resolution would follow next, which would allow me to tackle that warning and
>>> see whether I can get rid of it. However, before going a wrong route I would like to learn what the
>>> "official" Java 9 solution would be and try to implement that.
>>>
>>> ---rony
>> Yes, I think you are dealing with two problems here which you have been using the same solution
>> for in the past.
>>
>> The 1st thing you have been doing incorrectly for Java 9, as Alan explained, is the idiom:
>> o.getClass().getMethod(...) and the 2nd is that you are trying to access protected members on
>> behalf of some other class which is a subclass of the protected member's declaring class.
>>
>> The 1st problem has different solutions which are all doable in Java 9, since you are dealing
>> within the confines of public types, public members and exported packages. One solution is to
>> search for the most specific member in the inheritance hierarchy which is also accessible
>> (declared in public type in exported package) which is what Alan suggests.
>>
>> There might also be another elegant solution which requires some re-design of your Rexx
>> interpreter.  When you deal with reference values in Rexx (the values that refer to objects in
>> Java), you could track not only the value itself but also the "static" type of that value. A
>> reference value is always obtained either by calling a constructor, accessing a field (either
>> static or instance), by calling a method (static or instance) or by accessing an element of some
>> array:
>>
>> - calling constructor: the "static type" is the class upon which the constructor has been called
>> - accessing a field: the "static type" is the type of the field (i.e. Field.getDeclaringClass())
>> - calling a method: the "static type" is the return type of the method (i.e. Method.getReturnType())
>> - accessing an element of some array: the "static type" is the array's "static type"'s component
>> type (i.e. Class.getComponentType() invoked on array's "static type" Class).
>>
>> When you take the "static" type as the starting Class when searching for a public member with
>> standard Class.getMethod() or Class.getField(), you would then get the correct publicly accessible
>> reflected member. With a caveat that this only works when there's no generics involved. If there's
>> generics, the logic to compute the correct static type is more involved and would sometimes
>> require passing the generic type parameters (when invoking constructors of generic classes or
>> generic methods) in the syntax of your Rexx language. So you may or may not want to do that.
>> Perhaps some library for deep resolving could be of help here (Google Guava has some support for
>> that). I guess searching for the most specific member in the hierarchy that is also accessible is
>> your best bet currently if the goal is to be syntactically backwards compatible in the Rexx language.
>>
>> The 2nd problem is not trivial as you want to access a protected member on behalf of some other
>> sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
>> instance of its Lookup object). This currently requires the package containing the member's
>> declaring class to be opened at least to you (the Rexx interpreter) and using the
>> member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
>> method handles. Which is awkward because libraries packed as modules would normally not specify
>> that in their module descriptors and system modules don't either. So you are left with either
>> --add-opens command line switches or deploying a javaagent to the JVM and using it's API point
>> java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
>> approaches are not elegant, but that's what is currently available, I think.
>>
>> Regards, Peter

Reply | Threaded
Open this post in threaded view
|

Re: Reflection: how does one access a protected member in a superclass reflectively?

Rony G. Flatscher
Hi Peter,

firstly: thank you *very* much again for your kind hints and help!

Ad Rexx and Java: the aim is to have Rexx (dynamically typed, caseless) behave 1:1 like a compilable
and runnable Java program. Over the course of time I looked into ASM (many, many years ago) to add
the ability to create dynamically subclasses. In the meantime Janino is at play as I really needed
just the ability to create dynamically simple subclasses (again quite some time ago). This support
was needed to allow abstract Java methods (and Java interface classes) to be implemented in ooRexx.
It was also implemented such, that concrete Java classes can be subclassed at runtime as well
(supplying a list of Java methods that should be interceptable by Rexx methods, but still allow Rexx
to invoke the Java methods in superclasses).

The reflection infrastructure in the bridge has been developed over the course of more than 15
years. It worked until Java 1.8/8 (introduction of new static and default methods) unchanged for
many years. Now with Java 9 clearly the reflection part needs to be redone to work with Java 9 and I
take the opportunity to rewrite it altogether in a way (hoping) that the module related problems can
be solved via reflection only, such that I can also create stripped down versions of the rewritten
reflection part for Java 1.8/8 and another for Java 1.6/6 and 1.7/7. Hence my desire to stick to
reflection for the time being.

If that is not possible with some of the needed functionality then I would immediately turn to
MethodHandles and try to follow your advice.

The task of the bridge is in principle is to allow Java programs to be written in Rexx and when the
Rexx program executes it should execute as if it was a Java program (if that can be compiled and
run)! :) [The bridge allows Java to employ Rexx as a scripting language as well, implementing
Apache's BSF and Java's 1.6/6 javax.script.ScriptEngine and the like.]

The Java objects to work with reflectively are received by invoking Java methods so are out of
control of the bridge. All interactions with Java have been done using the Java reflection mechanism
(including with the dynamically created Java classes at runtime) since Java 1.1.

Again, thank you very much for your kind help and advice!

---rony

P.S.: Currently I have been tracing down accessing public fields in a superclass of a type which
package is exported from the unnamed module (Rexx reflection at this stage works out of the unnamed
module), but the superclass resides in a non-exported package. A Java program from the unnamed
module is able to access those public fields, doing the same with reflection has not been successful
yet. Will double-check the example and post it under a new thread (javac 9 creates quite different
code for that Java program depending whether using the module or the classpath where the module
version uses MethodHandles; however both versions are able to access those public fields, which I
cannot access via reflection yet).




On 23.01.2018 08:10, Peter Levart wrote:

> Hi Rony,
>
> On 01/22/18 16:35, Rony G. Flatscher wrote:
>> Hi Peter,
>>
>> thank you *very* much also for your kind explanations and even coming up with agent code to
>> demonstrate how one could use that approach!
>>
>> In fact I have been doing a static analysis in the past (since Java 1.1) to determine whether
>> members should be accessible to Rexx, restricting access to public members and to protected
>> inherited members, if accessing them from a dynamically created subclass.
>
> So you *are* loading dynamically generated subclasses. It's just that their logic is implemented
> in Rexx scripting and therefore delegates invocations to Rexx runtime and those invocations
> include invocations to protected methods of their superclasses.
>
> In that case, why don't you come up with some mechanism for dynamically generated classes to hand
> over their Lookup(s) to Rexx runtime. For example, the Rexx runtime could have the following
> public registration API:
>
> ---
> package runtime;
>
> import java.lang.invoke.MethodHandles;
> import java.util.Map;
> import java.util.concurrent.ConcurrentHashMap;
>
> public class LookupRegistry {
>     private static final Map<Class<?>, MethodHandles.Lookup> registry =
>         new ConcurrentHashMap<>();
>
>     public static void register(MethodHandles.Lookup lookup) {
>         if (registry.putIfAbsent(lookup.lookupClass(), lookup) != null) {
>             throw new IllegalStateException(
>                 "Lookup for " + lookup.lookupClass() + " is already registered");
>         }
>     }
>
>     /* non-public */ static MethodHandles.Lookup getLookup(Class<?> clazz) {
>         return registry.get(clazz);
>     }
> }
>
>
> ...then each Rexx dynamically generated class could include the following static initialization block:
>
>
> public class GeneratedClass {
>     static {
>         runtime.LookupRegistry.register(
>             java.lang.invoke.MethodHandles.lookup()
>         );
>     }
> ...
>
>
> That way Rexx runtime could have access to Lookup(s) of each of it's dynamically generated classes
> as soon as they are loaded and initialized and can use a particular class's Lookup to lookup a
> MethodHandle for the protected methods of its superclass. It can even simulate a "super.method()"
> invocation, etc...
>
>
> Regards, Peter
>
>> One thing I have to make sure is, that I keep compatible with the Java 1.6/6.0 baseline, as there
>> are some deployments where the shops are still employing that environment. This should not really be
>> a problem as I am rewriting the entire reflection part and will be able to employ different
>> approaches for pre-Java 9 deplyoments, where the reflection class employed is dependent on the Java
>> runtime version.
>>
>> Currently I am exploring the implications of the new Java 9 module system, trying to adhere to its
>> rules. In essence the goal is to allow reflectively everything for Rexx peers that a compiled Java
>> program allows for in the Java 9 environment. For that the reflecting class (currently in the
>> unnamed module) goes up the inheritance tree until it finds an exported class and analyses it
>> reflectively.
>>
>> Experimenting with variations of classes residing in different modules with different exports, it is
>> possible to mix-up the relationships, what gets exported to what, and when should protected members
>> in superclasses be accessible and when not (and yes, this part should belong to dynamically created
>> subclasses, which also need adjustments to the module system).
>>
>> Best regards,
>>
>> ---rony
>>
>>
>> On 22.01.2018 10:58, Peter Levart wrote:
>>> Hi Rony,
>>>
>>> On 01/18/2018 04:11 PM, Rony G. Flatscher wrote:
>>>> On 18.01.2018 10:58, Alan Bateman wrote:
>>>>> On 17/01/2018 18:53, Rony G. Flatscher wrote:
>>>>>> :
>>>>>>
>>>>>> Would you have concrete suggestions for this use-case, i.e. a framework that is not part of a
>>>>>> module, but having a need to access public types from exported packages and get reflective access
>>>>>> to objects supertype's protected members?
>>>>> I think it would be better to start with public members as protected is complicated (and hasn't
>>>>> changed with modules once you establish the class declaring the member is accessible).
>>>>>
>>>>> For your example, you've got a reference to a java.awt.Graphics2D object, the actual
>>>>> implementation type is sun.java2d.SunGraphics2D. The user is attempting to invoke one of the
>>>>> public setRenderingHint methods that Graphics2D defines. You said in one of your mails that the
>>>>> bridge "iterates over all its superclasses" which I take to mean that it recursively looks at the
>>>>> superclass and interfaces to find a public class or interface that defines the target
>>>>> setRenderingHint method. In the example, I expect it would skip sun.java2d.SunGraphics2D if it
>>>>> were non public.
>>>>>
>>>>> Can you extend this check to test if the class is in a package exported by its module. For the
>>>>> example, sun.java2d.SunGraphics2D is in the java.desktop module and this module does not export
>>>>> sun.java2d to everyone. Here is a code fragment to test this:
>>>>>
>>>>> Class<?> clazz = graphicsObj.getClass();
>>>>> boolean isExportedToAll = clazz.getModule().isExported(clazz.getPackageName());
>>>>>
>>>>> (I'm deliberately avoiding the 2-arg isExported to keep things simple for this discussion).
>>>>>
>>>>> If you can incorporate this check into the bridge then I suspect you'll find most of the examples
>>>>> will work.
>>>> Yes, I understand (not being able to use methods in an unexported type's instance, hence the need to
>>>> find an accessible member in a superclass, which means to have a need to also access protected
>>>> members in the superclass) and that is actually my current approach. However, I started out with
>>>> reflecting Fields first and see, whether I can reflectively get access.
>>>>
>>>> The rewritten method resolution would follow next, which would allow me to tackle that warning and
>>>> see whether I can get rid of it. However, before going a wrong route I would like to learn what the
>>>> "official" Java 9 solution would be and try to implement that.
>>>>
>>>> ---rony
>>> Yes, I think you are dealing with two problems here which you have been using the same solution
>>> for in the past.
>>>
>>> The 1st thing you have been doing incorrectly for Java 9, as Alan explained, is the idiom:
>>> o.getClass().getMethod(...) and the 2nd is that you are trying to access protected members on
>>> behalf of some other class which is a subclass of the protected member's declaring class.
>>>
>>> The 1st problem has different solutions which are all doable in Java 9, since you are dealing
>>> within the confines of public types, public members and exported packages. One solution is to
>>> search for the most specific member in the inheritance hierarchy which is also accessible
>>> (declared in public type in exported package) which is what Alan suggests.
>>>
>>> There might also be another elegant solution which requires some re-design of your Rexx
>>> interpreter.  When you deal with reference values in Rexx (the values that refer to objects in
>>> Java), you could track not only the value itself but also the "static" type of that value. A
>>> reference value is always obtained either by calling a constructor, accessing a field (either
>>> static or instance), by calling a method (static or instance) or by accessing an element of some
>>> array:
>>>
>>> - calling constructor: the "static type" is the class upon which the constructor has been called
>>> - accessing a field: the "static type" is the type of the field (i.e. Field.getDeclaringClass())
>>> - calling a method: the "static type" is the return type of the method (i.e. Method.getReturnType())
>>> - accessing an element of some array: the "static type" is the array's "static type"'s component
>>> type (i.e. Class.getComponentType() invoked on array's "static type" Class).
>>>
>>> When you take the "static" type as the starting Class when searching for a public member with
>>> standard Class.getMethod() or Class.getField(), you would then get the correct publicly accessible
>>> reflected member. With a caveat that this only works when there's no generics involved. If there's
>>> generics, the logic to compute the correct static type is more involved and would sometimes
>>> require passing the generic type parameters (when invoking constructors of generic classes or
>>> generic methods) in the syntax of your Rexx language. So you may or may not want to do that.
>>> Perhaps some library for deep resolving could be of help here (Google Guava has some support for
>>> that). I guess searching for the most specific member in the hierarchy that is also accessible is
>>> your best bet currently if the goal is to be syntactically backwards compatible in the Rexx language.
>>>
>>> The 2nd problem is not trivial as you want to access a protected member on behalf of some other
>>> sub-class of the member's declaring class which is not cooperating (voluntarily handing you an
>>> instance of its Lookup object). This currently requires the package containing the member's
>>> declaring class to be opened at least to you (the Rexx interpreter) and using the
>>> member.setAccessible(true) trick or MethodHandles.privateLookupIn(declaringClass) equivalent for
>>> method handles. Which is awkward because libraries packed as modules would normally not specify
>>> that in their module descriptors and system modules don't either. So you are left with either
>>> --add-opens command line switches or deploying a javaagent to the JVM and using it's API point
>>> java.lang.instrument.Instrumentation#redefineModule to add opens to modules that way. Both
>>> approaches are not elegant, but that's what is currently available, I think.
>>>
>>> Regards, Peter
>

12