Accessing module internals from bytecode rewriting agent

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Accessing module internals from bytecode rewriting agent

Martin Buchholz
We've been playing with byteman.  If I have a simple rule that accesses a
private field,

RULE trace java.util.PriorityQueue_add
CLASS java.util.PriorityQueue
METHOD add
AT EXIT
IF true
DO traceln(java.util.Arrays.toString($0.queue))
ENDRULE

Then with jdk8 I get output like:

 [1, 2, 3]

but in jdk9 we fail with

java.lang.IllegalArgumentException: illegal lookupClass: class
java.util.PriorityQueue

Bytecode rewriting agents have the power to inject code into classes; they
should somehow also be able to reflectively inspect those same classes!
But how?  We ran into similar problems trying to port java agents at Google
to jdk9.

Details:
JAVA_TOOL_OPTIONS="-javaagent:${BYTEMAN_JAR}=script:${BYTEMAN_SCRIPT},boot:${BYTEMAN_JAR}"

     [java] java.lang.IllegalArgumentException: illegal lookupClass: class
java.util.PriorityQueue
     [java] at
java.base/java.lang.invoke.MethodHandleStatics.newIllegalArgumentException(MethodHandleStatics.java:123)
     [java] at
java.base/java.lang.invoke.MethodHandles$Lookup.checkUnprivilegedlookupClass(MethodHandles.java:1003)
     [java] at
java.base/java.lang.invoke.MethodHandles$Lookup.<init>(MethodHandles.java:750)
     [java] at
java.base/java.lang.invoke.MethodHandles.privateLookupIn(MethodHandles.java:205)
     [java] at
org.jboss.byteman.jigsaw/org.jboss.byteman.jigsaw.JigsawAccessEnabler.createFieldGetter(JigsawAccessEnabler.java)
     [java] at
org.jboss.byteman.rule.Rule.addAccessibleFieldGetter(Rule.java:1043)
     [java] at
org.jboss.byteman.rule.expression.FieldExpression.lookupField(FieldExpression.java:549)
     [java] at
org.jboss.byteman.rule.expression.FieldExpression.typeCheckAny(FieldExpression.java:201)
     [java] at
org.jboss.byteman.rule.expression.FieldExpression.typeCheck(FieldExpression.java:110)
     [java] at
org.jboss.byteman.rule.expression.MethodExpression.findMethod(MethodExpression.java:297)
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Alan Bateman
On 25/04/2017 04:26, Martin Buchholz wrote:

> :
>
> java.lang.IllegalArgumentException: illegal lookupClass: class
> java.util.PriorityQueue
>
> Bytecode rewriting agents have the power to inject code into classes; they
> should somehow also be able to reflectively inspect those same classes!
> But how?  We ran into similar problems trying to port java agents at Google
> to jdk9.
On the general point: agents (both java and JVM TI agents) can redefine
any module in support of instrumentation and any amount of deep
reflection that the agent might do. So I wouldn't expect too many issues
although some java agents may need updates to work on JDK 9 without
encapsulation busting command line options.

On Byteman, Andrew Dinn has been working with us on jigsaw-dev on the
agent support. From the stack trace then I suspect the issue you are
running into is that Byteman has also changed to use the new reflection
API (java.lang.invoke) and is running into a long standing check to
prevent full power lookups to full privileged classes from leaking.
Andrew brought this up on jigsaw-dev [1] a few days ago and we could
probably move that thread to core-libs-dev.

-Alan

[1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-April/012267.html
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Jeremy Manson
In reply to this post by Martin Buchholz
Just to underline this - the conversation with Martin about this came from
my trying to test my allocation instrumenter
<https://github.com/google/allocation-instrumenter> with JDK 9.

The instrument API specifically supports a Boot-Class-Path attribute in the
MANIFEST, as well as Instrumentation.appendToBootStrapClassLoaderSearch.  I
had been using that as a way to make it possible to rewrite JDK internals
(add allocation instrumentation to them).  I hadn't finished testing when
Martin sent this email, but I hadn't been able to get those things to work,
and what byteman does is pretty similar to what I was trying to test.

It sure would be nice if there were still a supported way to rewrite system
internals.  Bytecode rewriting is the recommended way to track allocations
<http://docs.oracle.com/javase/8/docs/platform/jvmti/jvmti.html#bci>, and
it is very reasonable to want to do this kind of thing in the system
classes.

Jeremy

On Mon, Apr 24, 2017 at 8:26 PM, Martin Buchholz <[hidden email]>
wrote:

> We've been playing with byteman.  If I have a simple rule that accesses a
> private field,
>
> RULE trace java.util.PriorityQueue_add
> CLASS java.util.PriorityQueue
> METHOD add
> AT EXIT
> IF true
> DO traceln(java.util.Arrays.toString($0.queue))
> ENDRULE
>
> Then with jdk8 I get output like:
>
>  [1, 2, 3]
>
> but in jdk9 we fail with
>
> java.lang.IllegalArgumentException: illegal lookupClass: class
> java.util.PriorityQueue
>
> Bytecode rewriting agents have the power to inject code into classes; they
> should somehow also be able to reflectively inspect those same classes!
> But how?  We ran into similar problems trying to port java agents at Google
> to jdk9.
>
> Details:
> JAVA_TOOL_OPTIONS="-javaagent:${BYTEMAN_JAR}=script:${
> BYTEMAN_SCRIPT},boot:${BYTEMAN_JAR}"
>
>      [java] java.lang.IllegalArgumentException: illegal lookupClass:
> class java.util.PriorityQueue
>      [java] at java.base/java.lang.invoke.MethodHandleStatics.
> newIllegalArgumentException(MethodHandleStatics.java:123)
>      [java] at java.base/java.lang.invoke.MethodHandles$Lookup.
> checkUnprivilegedlookupClass(MethodHandles.java:1003)
>      [java] at java.base/java.lang.invoke.MethodHandles$Lookup.<init>(
> MethodHandles.java:750)
>      [java] at java.base/java.lang.invoke.MethodHandles.privateLookupIn(
> MethodHandles.java:205)
>      [java] at org.jboss.byteman.jigsaw/org.jboss.byteman.jigsaw.
> JigsawAccessEnabler.createFieldGetter(JigsawAccessEnabler.java)
>      [java] at org.jboss.byteman.rule.Rule.addAccessibleFieldGetter(Rule.
> java:1043)
>      [java] at org.jboss.byteman.rule.expression.FieldExpression.
> lookupField(FieldExpression.java:549)
>      [java] at org.jboss.byteman.rule.expression.FieldExpression.
> typeCheckAny(FieldExpression.java:201)
>      [java] at org.jboss.byteman.rule.expression.FieldExpression.
> typeCheck(FieldExpression.java:110)
>      [java] at org.jboss.byteman.rule.expression.MethodExpression.
> findMethod(MethodExpression.java:297)
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Jeremy Manson
In reply to this post by Alan Bateman
Ah, then it sounds as if I just don't know what I'm doing.  Any pointers?

Jeremy

On Mon, Apr 24, 2017 at 11:22 PM, Alan Bateman <[hidden email]>
wrote:

> On 25/04/2017 04:26, Martin Buchholz wrote:
>
> :
>>
>> java.lang.IllegalArgumentException: illegal lookupClass: class
>> java.util.PriorityQueue
>>
>> Bytecode rewriting agents have the power to inject code into classes; they
>> should somehow also be able to reflectively inspect those same classes!
>> But how?  We ran into similar problems trying to port java agents at
>> Google
>> to jdk9.
>>
> On the general point: agents (both java and JVM TI agents) can redefine
> any module in support of instrumentation and any amount of deep reflection
> that the agent might do. So I wouldn't expect too many issues although some
> java agents may need updates to work on JDK 9 without encapsulation busting
> command line options.
>
> On Byteman, Andrew Dinn has been working with us on jigsaw-dev on the
> agent support. From the stack trace then I suspect the issue you are
> running into is that Byteman has also changed to use the new reflection API
> (java.lang.invoke) and is running into a long standing check to prevent
> full power lookups to full privileged classes from leaking. Andrew brought
> this up on jigsaw-dev [1] a few days ago and we could probably move that
> thread to core-libs-dev.
>
> -Alan
>
> [1] http://mail.openjdk.java.net/pipermail/jigsaw-dev/2017-April
> /012267.html
>
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Alan Bateman
In reply to this post by Jeremy Manson
On 25/04/2017 07:22, Jeremy Manson wrote:

> :
>
> The instrument API specifically supports a Boot-Class-Path attribute in the
> MANIFEST, as well as Instrumentation.appendToBootStrapClassLoaderSearch.  I
> had been using that as a way to make it possible to rewrite JDK internals
> (add allocation instrumentation to them).
I wouldn't expect any issues with that scenario as the VM adds the
readability needed for this to work. From the javadoc:

"As an aid to agents that deploy supporting classes on the search path
of the bootstrap class loader, or the search path of the class loader
that loads the main agent class, the Java virtual machine arranges for
the module of transformed classes to read the unnamed module of both
class loaders."

Instead, I think the issue in Martin's mail is more to do with Byteman
moving to java.lang.invoke and running into an issue getting full power
lookups to fully privileged classes.

-Alan.
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Jeremy Manson
Thanks, Alan.  I don't use byteman, and my agent wasn't working out of the
box to redefine JDK classes...  I guess I can keep digging.

Jeremy

On Mon, Apr 24, 2017 at 11:35 PM, Alan Bateman <[hidden email]>
wrote:

> On 25/04/2017 07:22, Jeremy Manson wrote:
>
> :
>>
>> The instrument API specifically supports a Boot-Class-Path attribute in
>> the
>> MANIFEST, as well as Instrumentation.appendToBootStrapClassLoaderSearch.
>> I
>> had been using that as a way to make it possible to rewrite JDK internals
>> (add allocation instrumentation to them).
>>
> I wouldn't expect any issues with that scenario as the VM adds the
> readability needed for this to work. From the javadoc:
>
> "As an aid to agents that deploy supporting classes on the search path of
> the bootstrap class loader, or the search path of the class loader that
> loads the main agent class, the Java virtual machine arranges for the
> module of transformed classes to read the unnamed module of both class
> loaders."
>
> Instead, I think the issue in Martin's mail is more to do with Byteman
> moving to java.lang.invoke and running into an issue getting full power
> lookups to fully privileged classes.
>
> -Alan.
>
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Alan Bateman
In reply to this post by Jeremy Manson
On 25/04/2017 07:29, Jeremy Manson wrote:

> Ah, then it sounds as if I just don't know what I'm doing.  Any pointers?
>
This should help:

http://download.java.net/java/jdk9/docs/api/java/lang/instrument/Instrumentation.html#redefineModule-java.lang.Module-java.util.Set-java.util.Map-java.util.Map-java.util.Set-java.util.Map-

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Jeremy Manson
Thanks.

It wouldn't have occurred to me that I needed to use a Java 9 API.  I can't
remember ever having an upgrade path for anything in any Java version for
which I couldn't use the same code in Java N and Java N+1, if the use case
was a supported part of the spec in both.

The usual migration path we have is to make all of our code work in both
JDK N and JDK N+1, and then flip the switch to migrate to N+1.  I expect we
are not alone in that.

I guess I can call it reflectively.

This is pretty nice, otherwise, because we have lots of reflective hacks
into the JDK in our codebase, and it gives us a way to do the ones we can't
do with Unsafe. :)

Jeremy

On Mon, Apr 24, 2017 at 11:50 PM, Alan Bateman <[hidden email]>
wrote:

> On 25/04/2017 07:29, Jeremy Manson wrote:
>
> Ah, then it sounds as if I just don't know what I'm doing.  Any pointers?
>>
>> This should help:
>
> http://download.java.net/java/jdk9/docs/api/java/lang/instru
> ment/Instrumentation.html#redefineModule-java.lang.Modul
> e-java.util.Set-java.util.Map-java.util.Map-java.util.Set-java.util.Map-
>
> -Alan
>
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Andrew Dinn
In reply to this post by Alan Bateman
On 25/04/17 07:22, Alan Bateman wrote:

> On 25/04/2017 04:26, Martin Buchholz wrote:
>
>> :
>>
>> java.lang.IllegalArgumentException: illegal lookupClass: class
>> java.util.PriorityQueue
>>
>> Bytecode rewriting agents have the power to inject code into classes;
>> they
>> should somehow also be able to reflectively inspect those same classes!
>> But how?  We ran into similar problems trying to port java agents at
>> Google
>> to jdk9.
> <snip>
> On Byteman, Andrew Dinn has been working with us on jigsaw-dev on the
> agent support. From the stack trace then I suspect the issue you are
> running into is that Byteman has also changed to use the new reflection
> API (java.lang.invoke) and is running into a long standing check to
> prevent full power lookups to full privileged classes from leaking.
> Andrew brought this up on jigsaw-dev [1] a few days ago and we could
> probably move that thread to core-libs-dev.

This discussion really ought to be happening on the Byteman forum but
anyway ...

Yes, Alan is right that this is exactly what is going on. Byteman on
jdk9 (the 4.0.0-BETA release series) now uses method handles in place of
reflection. Unfortunately, it is limited by a slightly arbitrary
restriction on the use of method handles in java.* and sun.* packages (a
blunt instrument to provide some security guarantees).

This is being addressed on two fronts. I am in the process of adding
unit tests to complete a workaround which remedies this problem by
falling back to reflection in the specific cases where the restriction
applies (yes, that means there is probably going to be another
4.0.0-BETA release before jdk9 goes live %-/ ).

Meanwhile, Mandy Chung has kindly offered to look into the security
considerations that are at play and come up with a less restrictive
policy which enforces the security needs more accurately. At that point
I will probably remove the fallback -- in part because I hope that by
then both reflection and method handles will employ a common blacklist
of inaccessible methods. I think it would make sense for them both to
offer equivalent access and for that access to take security seriously.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Remi Forax
[...]

>
> Meanwhile, Mandy Chung has kindly offered to look into the security
> considerations that are at play and come up with a less restrictive
> policy which enforces the security needs more accurately. At that point
> I will probably remove the fallback -- in part because I hope that by
> then both reflection and method handles will employ a common blacklist
> of inaccessible methods. I think it would make sense for them both to
> offer equivalent access and for that access to take security seriously.

I've seen this kind of fallback in different language runtimes, mine included,
having a "less blunt" checks would be great.

>
> regards,
>
>
> Andrew Dinn

RĂ©mi
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Martin Buchholz
In reply to this post by Andrew Dinn
Thanks, all.  I should have remembered the special casing of
java.util.concurrent in checkUnprivilegedlookupClass.  It's surprising to
see different behavior crossing the java.util.concurrent boundary, as I
observed trying to access PriorityQueue.queue (fails!) vs.
PriorityBlockingQueue.queue (succeeds!).  Tools such as java agents should
have a single consistent mechanism that works for all target classes.

On Tue, Apr 25, 2017 at 1:19 AM, Andrew Dinn <[hidden email]> wrote:

> On 25/04/17 07:22, Alan Bateman wrote:
> > On 25/04/2017 04:26, Martin Buchholz wrote:
> >
> >> :
> >>
> >> java.lang.IllegalArgumentException: illegal lookupClass: class
> >> java.util.PriorityQueue
> >>
> >> Bytecode rewriting agents have the power to inject code into classes;
> >> they
> >> should somehow also be able to reflectively inspect those same classes!
> >> But how?  We ran into similar problems trying to port java agents at
> >> Google
> >> to jdk9.
> > <snip>
> > On Byteman, Andrew Dinn has been working with us on jigsaw-dev on the
> > agent support. From the stack trace then I suspect the issue you are
> > running into is that Byteman has also changed to use the new reflection
> > API (java.lang.invoke) and is running into a long standing check to
> > prevent full power lookups to full privileged classes from leaking.
> > Andrew brought this up on jigsaw-dev [1] a few days ago and we could
> > probably move that thread to core-libs-dev.
>
> This discussion really ought to be happening on the Byteman forum but
> anyway ...
>
> Yes, Alan is right that this is exactly what is going on. Byteman on
> jdk9 (the 4.0.0-BETA release series) now uses method handles in place of
> reflection. Unfortunately, it is limited by a slightly arbitrary
> restriction on the use of method handles in java.* and sun.* packages (a
> blunt instrument to provide some security guarantees).
>
> This is being addressed on two fronts. I am in the process of adding
> unit tests to complete a workaround which remedies this problem by
> falling back to reflection in the specific cases where the restriction
> applies (yes, that means there is probably going to be another
> 4.0.0-BETA release before jdk9 goes live %-/ ).
>
> Meanwhile, Mandy Chung has kindly offered to look into the security
> considerations that are at play and come up with a less restrictive
> policy which enforces the security needs more accurately. At that point
> I will probably remove the fallback -- in part because I hope that by
> then both reflection and method handles will employ a common blacklist
> of inaccessible methods. I think it would make sense for them both to
> offer equivalent access and for that access to take security seriously.
>
> regards,
>
>
> Andrew Dinn
> -----------
> Senior Principal Software Engineer
> Red Hat UK Ltd
> Registered in England and Wales under Company Registration No. 03798903
> Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
>
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Brian Goetz
In reply to this post by Jeremy Manson
Or use multi-release JARs.

On 4/25/2017 3:14 AM, Jeremy Manson wrote:
> I guess I can call it reflectively.

Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Jeremy Manson
Maybe when there's something breaking in Java 10 or 11, I can do that.
Right now, that approach requires me to have a fork of my code, have
special build goop, rely on yet another Java 9 feature, and make my
relatively widely used OSS project require Java 9 to build *prior to Java
9's release -* just so I can save maybe 15-20 lines of code.

Meanwhile, I'm not having a lot of luck - adding what seems to be the right
reflective call doesn't seem to allow me to modify classes like ArrayList.
Trying to create a boiled down repro (in 30 minute chunks between
management tasks - sigh).

OffTopic, but possibly of interest to you: it has also been challenging to
figure out how we're going to migrate our users away from features like
Xbootclasspath.  Our standard operating procedure (which has worked from 5
to 6 to 7 to 8) is to get the codebase into a state where it can work both
with the previous version and the new version of the JDK, and then throw
the switch.

If we follow this path, before we migrate to Java 9, we would need to make
sure all of our code builds and the tests pass with Java 9 and Java 8.  We
can't make all of the code build and the tests pass with Java 9 as-is,
because many of them use options like Xbootclasspath, which have been
dropped.  We can't migrate those tests to use the new command line flags
before we migrate to Java 9, because if we do, they will stop working with
Java 8.  Due to the size of our code base, it's pretty much impossible to
migrate all of the users to a new set of command line flags at the same
time as we migrate to Java 9.

Between that and the amount of reflective access we do into the JDK, we'll
probably just end up re-adding features like Xbootclasspath in our fork and
enabling the big kill switch for everyone, and then removing it gradually
over the lifetime of Java 9.  We've had to do that for some minor stuff
before, but nothing this big.  I'm glad we have options like that, and
don't envy people with large codebases who don't.

That said, we probably abuse the ability to crack open the JDK more than
most people.

I've been dealing with some of the concerns about upgradability for some of
Google's open-source projects lately, and I am insisting that we have a
rule that there is always an intermediate version that provides backwards
compatibility for any desired change, so that users can always perform
stepwise migrations.  Thanks for the inspiration. :)

Jeremy

On Mon, May 1, 2017 at 5:38 PM, Brian Goetz <[hidden email]> wrote:

> Or use multi-release JARs.
>
>
> On 4/25/2017 3:14 AM, Jeremy Manson wrote:
>
>> I guess I can call it reflectively.
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Alan Bateman
On 02/05/2017 07:50, Jeremy Manson wrote:

> :
>
> If we follow this path, before we migrate to Java 9, we would need to
> make sure all of our code builds and the tests pass with Java 9 and
> Java 8.  We can't make all of the code build and the tests pass with
> Java 9 as-is, because many of them use options like Xbootclasspath,
> which have been dropped.  We can't migrate those tests to use the new
> command line flags before we migrate to Java 9, because if we do, they
> will stop working with Java 8.  Due to the size of our code base, it's
> pretty much impossible to migrate all of the users to a new set of
> command line flags at the same time as we migrate to Java 9.
So I'm curious about -Xbootclasspath as that completely overrides the
runtime classes. Is this static instrumentation or combining the
VM/launcher from one build with the classes from a different JDK build?
In any case, and as you have found, -Xbootclasspath  and
-Xbootclasspath/p are meaningless now, a consequence of moving all the
core APIs to modules. The -Xbootclasspath/a option works as before with
the caveat that not all platform classes are visible to the boot class
loader now (a consequence of ongoing security work to move non-core
components away from the fully privileged boot loader).

It might be that you mean the `javac -bootclasspath,  this works as
before when you target 8 or older.

-Alan

Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Andrew Dinn
In reply to this post by Andrew Dinn
On 25/04/17 09:19, Andrew Dinn wrote:

> This discussion really ought to be happening on the Byteman forum but
> anyway ...
>
> Yes, Alan is right that this is exactly what is going on. Byteman on
> jdk9 (the 4.0.0-BETA release series) now uses method handles in place of
> reflection. Unfortunately, it is limited by a slightly arbitrary
> restriction on the use of method handles in java.* and sun.* packages (a
> blunt instrument to provide some security guarantees).
>
> This is being addressed on two fronts. I am in the process of adding
> unit tests to complete a workaround which remedies this problem by
> falling back to reflection in the specific cases where the restriction
> applies (yes, that means there is probably going to be another
> 4.0.0-BETA release before jdk9 goes live %-/ ).

Just wanted to post a heads-up that this fall-back behaviour has now
been implemented in Byteman release 4.0.0-BETA5.

regards,


Andrew Dinn
-----------
Senior Principal Software Engineer
Red Hat UK Ltd
Registered in England and Wales under Company Registration No. 03798903
Directors: Michael Cunningham, Michael ("Mike") O'Neill, Eric Shander
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Brian Goetz
In reply to this post by Jeremy Manson
8 can deal with multi-release JARs.  The extra goop is ignored. So a
7/8/9 MRJar is perfectly possible (and in fact, the whole point of doing
them in the first place, so library writers could publish JARs that span
7/8/9.)

On 5/2/2017 2:50 AM, Jeremy Manson wrote:

> Maybe when there's something breaking in Java 10 or 11, I can do
> that.  Right now, that approach requires me to have a fork of my code,
> have special build goop, rely on yet another Java 9 feature, and make
> my relatively widely used OSS project require Java 9 to build /prior
> to Java 9's release -/ just so I can save maybe 15-20 lines of code.
>
> Meanwhile, I'm not having a lot of luck - adding what seems to be the
> right reflective call doesn't seem to allow me to modify classes like
> ArrayList.  Trying to create a boiled down repro (in 30 minute chunks
> between management tasks - sigh).
>
> OffTopic, but possibly of interest to you: it has also been
> challenging to figure out how we're going to migrate our users away
> from features like Xbootclasspath.  Our standard operating procedure
> (which has worked from 5 to 6 to 7 to 8) is to get the codebase into a
> state where it can work both with the previous version and the new
> version of the JDK, and then throw the switch.
>
> If we follow this path, before we migrate to Java 9, we would need to
> make sure all of our code builds and the tests pass with Java 9 and
> Java 8.  We can't make all of the code build and the tests pass with
> Java 9 as-is, because many of them use options like Xbootclasspath,
> which have been dropped.  We can't migrate those tests to use the new
> command line flags before we migrate to Java 9, because if we do, they
> will stop working with Java 8.  Due to the size of our code base, it's
> pretty much impossible to migrate all of the users to a new set of
> command line flags at the same time as we migrate to Java 9.
>
> Between that and the amount of reflective access we do into the JDK,
> we'll probably just end up re-adding features like Xbootclasspath in
> our fork and enabling the big kill switch for everyone, and then
> removing it gradually over the lifetime of Java 9.  We've had to do
> that for some minor stuff before, but nothing this big.  I'm glad we
> have options like that, and don't envy people with large codebases who
> don't.
>
> That said, we probably abuse the ability to crack open the JDK more
> than most people.
>
> I've been dealing with some of the concerns about upgradability for
> some of Google's open-source projects lately, and I am insisting that
> we have a rule that there is always an intermediate version that
> provides backwards compatibility for any desired change, so that users
> can always perform stepwise migrations.  Thanks for the inspiration. :)
>
> Jeremy
>
> On Mon, May 1, 2017 at 5:38 PM, Brian Goetz <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Or use multi-release JARs.
>
>
>     On 4/25/2017 3:14 AM, Jeremy Manson wrote:
>
>         I guess I can call it reflectively.
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Michael Nascimento
On Tue, May 2, 2017 at 11:51 AM, Brian Goetz <[hidden email]> wrote:

> 8 can deal with multi-release JARs.
>

I missed that. Starting with what update? Is there any link I could use to
share this info?

Regards,
Michael
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Jeremy Manson
In reply to this post by Brian Goetz
Brian:  Bit of a miscommunication on my part: I know 8 can deal with
multirelease JARs.  I just can't build them with 8 (although maybe I could
with enough build magic).  When I said "rely on another 9 feature", I just
meant that I would be building something else in that required 9, which
would be bad because I would require a pre-release 9 to build.  I'm not
going to inflict that on my users (many of whom have forked it and build it
themselves).

(This project isn't Guava or Dagger or something - it's just a small-ball
instrumentation agent.)

Alan: Most of our code runs with our home-grown launcher (so we can take
advantage of JEP 178) and an OpenJDK-based JVM, although we also still
build and distribute the OpenJDK launcher, and many people use it.  People
are using Xbootclasspath for a variety of things.  We use it in various
non-OpenJDK libraries configurations (like with GWT and j2objc) to run with
their class libraries, which are not OpenJDK-based.  We also have folks
using Jetty ALPN, which requires it.  We also have a bunch of people using
it for instrumentation agents (it's a little tricky to get
X-Boot-Class-Path to work in our environment), but there should be
workarounds for that.  We have a few people using it for...
not-immediately-obvious reasons.

I didn't realize that bootclasspath/a still partially worked - that might
be enough for some of these use cases.  We'll have to see when we get to
it.  And see what happens when the stuff hits the fan.

We don't actually use javac at all - we have our own wrapper around the
javac API.

Anyway, I can see we have an interesting upgrade path ahead of us.  I'm
glad that we solved so many recurring problems permanently (like our hash
randomization change).

Jeremy


On Tue, May 2, 2017 at 7:51 AM, Brian Goetz <[hidden email]> wrote:

> 8 can deal with multi-release JARs.  The extra goop is ignored.  So a
> 7/8/9 MRJar is perfectly possible (and in fact, the whole point of doing
> them in the first place, so library writers could publish JARs that span
> 7/8/9.)
>
>
> On 5/2/2017 2:50 AM, Jeremy Manson wrote:
>
> Maybe when there's something breaking in Java 10 or 11, I can do that.
> Right now, that approach requires me to have a fork of my code, have
> special build goop, rely on yet another Java 9 feature, and make my
> relatively widely used OSS project require Java 9 to build *prior to Java
> 9's release -* just so I can save maybe 15-20 lines of code.
>
> Meanwhile, I'm not having a lot of luck - adding what seems to be the
> right reflective call doesn't seem to allow me to modify classes like
> ArrayList.  Trying to create a boiled down repro (in 30 minute chunks
> between management tasks - sigh).
>
> OffTopic, but possibly of interest to you: it has also been challenging to
> figure out how we're going to migrate our users away from features like
> Xbootclasspath.  Our standard operating procedure (which has worked from 5
> to 6 to 7 to 8) is to get the codebase into a state where it can work both
> with the previous version and the new version of the JDK, and then throw
> the switch.
>
> If we follow this path, before we migrate to Java 9, we would need to make
> sure all of our code builds and the tests pass with Java 9 and Java 8.  We
> can't make all of the code build and the tests pass with Java 9 as-is,
> because many of them use options like Xbootclasspath, which have been
> dropped.  We can't migrate those tests to use the new command line flags
> before we migrate to Java 9, because if we do, they will stop working with
> Java 8.  Due to the size of our code base, it's pretty much impossible to
> migrate all of the users to a new set of command line flags at the same
> time as we migrate to Java 9.
>
> Between that and the amount of reflective access we do into the JDK, we'll
> probably just end up re-adding features like Xbootclasspath in our fork and
> enabling the big kill switch for everyone, and then removing it gradually
> over the lifetime of Java 9.  We've had to do that for some minor stuff
> before, but nothing this big.  I'm glad we have options like that, and
> don't envy people with large codebases who don't.
>
> That said, we probably abuse the ability to crack open the JDK more than
> most people.
>
> I've been dealing with some of the concerns about upgradability for some
> of Google's open-source projects lately, and I am insisting that we have a
> rule that there is always an intermediate version that provides backwards
> compatibility for any desired change, so that users can always perform
> stepwise migrations.  Thanks for the inspiration. :)
>
> Jeremy
>
> On Mon, May 1, 2017 at 5:38 PM, Brian Goetz <[hidden email]>
> wrote:
>
>> Or use multi-release JARs.
>>
>>
>> On 4/25/2017 3:14 AM, Jeremy Manson wrote:
>>
>>> I guess I can call it reflectively.
>>>
>>
>>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Paul Sandoz
In reply to this post by Michael Nascimento

> On 2 May 2017, at 09:45, Michael Nascimento <[hidden email]> wrote:
>
> On Tue, May 2, 2017 at 11:51 AM, Brian Goetz <[hidden email]> wrote:
>
>> 8 can deal with multi-release JARs.
>>
>
> I missed that. Starting with what update? Is there any link I could use to
> share this info?
>

There have been no modifications to the Java 8 runtime, it will just ignore stuff in the META-INF/versions/{N} directories.

At one point we were mulling back porting the Java 9 runtime changes to Java 8 and therefore we could go back to Java 7, but we decided not to proceed with that, which is what i suspect you may be thinking about.

Paul.
Reply | Threaded
Open this post in threaded view
|

Re: Accessing module internals from bytecode rewriting agent

Michael Nascimento
On Tue, May 2, 2017 at 2:15 PM, Paul Sandoz <[hidden email]> wrote:

> At one point we were mulling back porting the Java 9 runtime changes to
> Java 8 and therefore we could go back to Java 7, but we decided not to
> proceed with that, which is what i suspect you may be thinking about.
>
>
Yes, I was hopeful about the last bit. Thank you for the clarification!

Regards,
Michael
12