Compiling module descriptors is not reproducible?

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Compiling module descriptors is not reproducible?

Mark Raynsford
Hello!

I've been interested for a while in making all of my software builds
reproducible [0]. I don't think there's ever been any kind of
written guarantee that the output of javac will be completely
deterministic, but to date it seems like it actually has been. However,
I was a bit disappointed today to find that the output of one of the
builds at AdoptOpenJDK [1] seems to be inserting compiler version
information into compiled module descriptors. It doesn't seem to be
doing this for any other class files.

Here's the output of "openjdk version 10.0.1 2018-04-17" on Linux
(not an AdoptOpenJDK build) when compiling a module descriptor:

---
module com.io7m.jtensors.core
  minor version: 0
  major version: 53
  flags: (0x8000) ACC_MODULE
  this_class: #1                          // "module-info"
  super_class: #0
  interfaces: 0, fields: 0, methods: 0, attributes: 2
Constant pool:
   #1 = Class              #17            // "module-info"
   #2 = Utf8               SourceFile
   #3 = Utf8               module-info.java
   #4 = Utf8               Module
   #5 = Module             #18            // "com.io7m.jtensors.core"
   #6 = Module             #19            // "java.base"
   #7 = Module             #20            // "org.immutables.value"
   #8 = Module             #21            // "com.io7m.junreachable.core"
   #9 = Package            #22            // com/io7m/jtensors/core/determinants
  #10 = Package            #23            // com/io7m/jtensors/core/dotproducts
  #11 = Package            #24            // com/io7m/jtensors/core/parameterized/matrices
  #12 = Package            #25            // com/io7m/jtensors/core/parameterized/vectors
  #13 = Package            #26            // com/io7m/jtensors/core/quaternions
  #14 = Package            #27            // com/io7m/jtensors/core/unparameterized/matrices
  #15 = Package            #28            // com/io7m/jtensors/core/unparameterized/vectors
  #16 = Package            #29            // com/io7m/jtensors/core
  #17 = Utf8               module-info
  #18 = Utf8               com.io7m.jtensors.core
  #19 = Utf8               java.base
  #20 = Utf8               org.immutables.value
  #21 = Utf8               com.io7m.junreachable.core
  #22 = Utf8               com/io7m/jtensors/core/determinants
  #23 = Utf8               com/io7m/jtensors/core/dotproducts
  #24 = Utf8               com/io7m/jtensors/core/parameterized/matrices
  #25 = Utf8               com/io7m/jtensors/core/parameterized/vectors
  #26 = Utf8               com/io7m/jtensors/core/quaternions
  #27 = Utf8               com/io7m/jtensors/core/unparameterized/matrices
  #28 = Utf8               com/io7m/jtensors/core/unparameterized/vectors
  #29 = Utf8               com/io7m/jtensors/core
{
}
SourceFile: "module-info.java"
Module:
  #5,0                                    // "com.io7m.jtensors.core"
  #0
  3                                       // requires
    #6,8000                                 // "java.base" ACC_MANDATED
    #0
    #7,40                                   // "org.immutables.value" ACC_STATIC_PHASE
    #0
    #8,0                                    // "com.io7m.junreachable.core"
    #0
  8                                       // exports
    #9,0                                    // com/io7m/jtensors/core/determinants
    #10,0                                   // com/io7m/jtensors/core/dotproducts
    #11,0                                   // com/io7m/jtensors/core/parameterized/matrices
    #12,0                                   // com/io7m/jtensors/core/parameterized/vectors
    #13,0                                   // com/io7m/jtensors/core/quaternions
    #14,0                                   // com/io7m/jtensors/core/unparameterized/matrices
    #15,0                                   // com/io7m/jtensors/core/unparameterized/vectors
    #16,0                                   // com/io7m/jtensors/core
  0                                       // opens
  0                                       // uses
  0                                       // provides
---

Here's the same module descriptor compiled on "openjdk version
10-internal" on MacOS (this is actually OpenJDK 10+23):

---
module com.io7m.jtensors.core
  minor version: 0
  major version: 53
  flags: (0x8000) ACC_MODULE
  this_class: #1                          // "module-info"
  super_class: #0
  interfaces: 0, fields: 0, methods: 0, attributes: 2
Constant pool:
   #1 = Class              #18            // "module-info"
   #2 = Utf8               SourceFile
   #3 = Utf8               module-info.java
   #4 = Utf8               Module
   #5 = Module             #19            // "com.io7m.jtensors.core"
   #6 = Module             #20            // "java.base"
   #7 = Utf8               10-internal
   #8 = Module             #21            // "org.immutables.value"
   #9 = Module             #22            // "com.io7m.junreachable.core"
  #10 = Package            #23            // com/io7m/jtensors/core/determinants
  #11 = Package            #24            // com/io7m/jtensors/core/dotproducts
  #12 = Package            #25            // com/io7m/jtensors/core/parameterized/matrices
  #13 = Package            #26            // com/io7m/jtensors/core/parameterized/vectors
  #14 = Package            #27            // com/io7m/jtensors/core/quaternions
  #15 = Package            #28            // com/io7m/jtensors/core/unparameterized/matrices
  #16 = Package            #29            // com/io7m/jtensors/core/unparameterized/vectors
  #17 = Package            #30            // com/io7m/jtensors/core
  #18 = Utf8               module-info
  #19 = Utf8               com.io7m.jtensors.core
  #20 = Utf8               java.base
  #21 = Utf8               org.immutables.value
  #22 = Utf8               com.io7m.junreachable.core
  #23 = Utf8               com/io7m/jtensors/core/determinants
  #24 = Utf8               com/io7m/jtensors/core/dotproducts
  #25 = Utf8               com/io7m/jtensors/core/parameterized/matrices
  #26 = Utf8               com/io7m/jtensors/core/parameterized/vectors
  #27 = Utf8               com/io7m/jtensors/core/quaternions
  #28 = Utf8               com/io7m/jtensors/core/unparameterized/matrices
  #29 = Utf8               com/io7m/jtensors/core/unparameterized/vectors
  #30 = Utf8               com/io7m/jtensors/core
{
}
SourceFile: "module-info.java"
Module:
  #5,0                                    // "com.io7m.jtensors.core"
  #0
  3                                       // requires
    #6,8000                                 // "java.base" ACC_MANDATED
    #7                                      // 10-internal
    #8,40                                   // "org.immutables.value" ACC_STATIC_PHASE
    #0
    #9,0                                    // "com.io7m.junreachable.core"
    #0
  8                                       // exports
    #10,0                                   // com/io7m/jtensors/core/determinants
    #11,0                                   // com/io7m/jtensors/core/dotproducts
    #12,0                                   // com/io7m/jtensors/core/parameterized/matrices
    #13,0                                   // com/io7m/jtensors/core/parameterized/vectors
    #14,0                                   // com/io7m/jtensors/core/quaternions
    #15,0                                   // com/io7m/jtensors/core/unparameterized/matrices
    #16,0                                   // com/io7m/jtensors/core/unparameterized/vectors
    #17,0                                   // com/io7m/jtensors/core
  0                                       // opens
  0                                       // uses
  0                                       // provides
---

Note the insertion of the "10-internal" string as constant pool entry
7.

Does anyone have any idea why this would happen? I'm considering filing
a bug at the AdoptOpenJDK project, but I'd like to be sure as to
whether or not this is actually intended behaviour by newer compilers.
Inserting version information obviously breaks reproducibility, so it'd
be nice if it could be turned off!

The sources to the above module descriptor are available [2] if anyone
wants to try compiling this for themselves. The SHA256 of the resulting
module-info.class should be
e7a94b5a2788a3c5cd6d7f586e70a0f2138a2eaa0e75144b6a16e75c4b297870 if
your compiler produces the same output as mine.

[0] https://reproducible-builds.org/
[1] https://adoptopenjdk.net/
[2] https://github.com/io7m/jtensors

--
Mark Raynsford | http://www.io7m.com

Reply | Threaded
Open this post in threaded view
|

Re: Compiling module descriptors is not reproducible?

Alan Bateman
On 19/05/2018 10:18, Mark Raynsford wrote:

> Hello!
>
> I've been interested for a while in making all of my software builds
> reproducible [0]. I don't think there's ever been any kind of
> written guarantee that the output of javac will be completely
> deterministic, but to date it seems like it actually has been. However,
> I was a bit disappointed today to find that the output of one of the
> builds at AdoptOpenJDK [1] seems to be inserting compiler version
> information into compiled module descriptors. It doesn't seem to be
> doing this for any other class files.
The requires table in the Module attribute can be used by compilers to
record version information about dependences. javac does record the
version information, I don't know if there is a way to disable that. In
any case, I assume your question will be partly answered by looking at
the output of `java --list-modules` on the different builds. If you
download OpenJDK builds from jdk.java.net/10 then you should see that
the standard and JDK-specific modules all report their version string as
"10" or "10.0.1" as they have been built with configure options that
make it so. At a guess, your mail may have included the output from an
exploded (as opposed to images) build.

-Alan
Reply | Threaded
Open this post in threaded view
|

Re: Compiling module descriptors is not reproducible?

Remi Forax
In reply to this post by Mark Raynsford
----- Mail original -----
> De: "org openjdk" <[hidden email]>
> À: "jigsaw-dev" <[hidden email]>
> Envoyé: Samedi 19 Mai 2018 11:18:09
> Objet: Compiling module descriptors is not reproducible?

> Hello!

Hi Mark,

>
> I've been interested for a while in making all of my software builds
> reproducible [0]. I don't think there's ever been any kind of
> written guarantee that the output of javac will be completely
> deterministic, but to date it seems like it actually has been. However,
> I was a bit disappointed today to find that the output of one of the
> builds at AdoptOpenJDK [1] seems to be inserting compiler version
> information into compiled module descriptors. It doesn't seem to be
> doing this for any other class files.
>
> Here's the output of "openjdk version 10.0.1 2018-04-17" on Linux
> (not an AdoptOpenJDK build) when compiling a module descriptor:
>
> ---
> module com.io7m.jtensors.core
>  minor version: 0
>  major version: 53
>  flags: (0x8000) ACC_MODULE
>  this_class: #1                          // "module-info"
>  super_class: #0
>  interfaces: 0, fields: 0, methods: 0, attributes: 2
> Constant pool:
>   #1 = Class              #17            // "module-info"
>   #2 = Utf8               SourceFile
>   #3 = Utf8               module-info.java
>   #4 = Utf8               Module
>   #5 = Module             #18            // "com.io7m.jtensors.core"
>   #6 = Module             #19            // "java.base"
>   #7 = Module             #20            // "org.immutables.value"
>   #8 = Module             #21            // "com.io7m.junreachable.core"
>   #9 = Package            #22            // com/io7m/jtensors/core/determinants
>  #10 = Package            #23            // com/io7m/jtensors/core/dotproducts
>  #11 = Package            #24            //
>  com/io7m/jtensors/core/parameterized/matrices
>  #12 = Package            #25            //
>  com/io7m/jtensors/core/parameterized/vectors
>  #13 = Package            #26            // com/io7m/jtensors/core/quaternions
>  #14 = Package            #27            //
>  com/io7m/jtensors/core/unparameterized/matrices
>  #15 = Package            #28            //
>  com/io7m/jtensors/core/unparameterized/vectors
>  #16 = Package            #29            // com/io7m/jtensors/core
>  #17 = Utf8               module-info
>  #18 = Utf8               com.io7m.jtensors.core
>  #19 = Utf8               java.base
>  #20 = Utf8               org.immutables.value
>  #21 = Utf8               com.io7m.junreachable.core
>  #22 = Utf8               com/io7m/jtensors/core/determinants
>  #23 = Utf8               com/io7m/jtensors/core/dotproducts
>  #24 = Utf8               com/io7m/jtensors/core/parameterized/matrices
>  #25 = Utf8               com/io7m/jtensors/core/parameterized/vectors
>  #26 = Utf8               com/io7m/jtensors/core/quaternions
>  #27 = Utf8               com/io7m/jtensors/core/unparameterized/matrices
>  #28 = Utf8               com/io7m/jtensors/core/unparameterized/vectors
>  #29 = Utf8               com/io7m/jtensors/core
> {
> }
> SourceFile: "module-info.java"
> Module:
>  #5,0                                    // "com.io7m.jtensors.core"
>  #0
>  3                                       // requires
>    #6,8000                                 // "java.base" ACC_MANDATED
>    #0
>    #7,40                                   // "org.immutables.value"
>    ACC_STATIC_PHASE
>    #0
>    #8,0                                    // "com.io7m.junreachable.core"
>    #0
>  8                                       // exports
>    #9,0                                    // com/io7m/jtensors/core/determinants
>    #10,0                                   // com/io7m/jtensors/core/dotproducts
>    #11,0                                   //
>    com/io7m/jtensors/core/parameterized/matrices
>    #12,0                                   //
>    com/io7m/jtensors/core/parameterized/vectors
>    #13,0                                   // com/io7m/jtensors/core/quaternions
>    #14,0                                   //
>    com/io7m/jtensors/core/unparameterized/matrices
>    #15,0                                   //
>    com/io7m/jtensors/core/unparameterized/vectors
>    #16,0                                   // com/io7m/jtensors/core
>  0                                       // opens
>  0                                       // uses
>  0                                       // provides
> ---
>
> Here's the same module descriptor compiled on "openjdk version
> 10-internal" on MacOS (this is actually OpenJDK 10+23):
>
> ---
> module com.io7m.jtensors.core
>  minor version: 0
>  major version: 53
>  flags: (0x8000) ACC_MODULE
>  this_class: #1                          // "module-info"
>  super_class: #0
>  interfaces: 0, fields: 0, methods: 0, attributes: 2
> Constant pool:
>   #1 = Class              #18            // "module-info"
>   #2 = Utf8               SourceFile
>   #3 = Utf8               module-info.java
>   #4 = Utf8               Module
>   #5 = Module             #19            // "com.io7m.jtensors.core"
>   #6 = Module             #20            // "java.base"
>   #7 = Utf8               10-internal
>   #8 = Module             #21            // "org.immutables.value"
>   #9 = Module             #22            // "com.io7m.junreachable.core"
>  #10 = Package            #23            // com/io7m/jtensors/core/determinants
>  #11 = Package            #24            // com/io7m/jtensors/core/dotproducts
>  #12 = Package            #25            //
>  com/io7m/jtensors/core/parameterized/matrices
>  #13 = Package            #26            //
>  com/io7m/jtensors/core/parameterized/vectors
>  #14 = Package            #27            // com/io7m/jtensors/core/quaternions
>  #15 = Package            #28            //
>  com/io7m/jtensors/core/unparameterized/matrices
>  #16 = Package            #29            //
>  com/io7m/jtensors/core/unparameterized/vectors
>  #17 = Package            #30            // com/io7m/jtensors/core
>  #18 = Utf8               module-info
>  #19 = Utf8               com.io7m.jtensors.core
>  #20 = Utf8               java.base
>  #21 = Utf8               org.immutables.value
>  #22 = Utf8               com.io7m.junreachable.core
>  #23 = Utf8               com/io7m/jtensors/core/determinants
>  #24 = Utf8               com/io7m/jtensors/core/dotproducts
>  #25 = Utf8               com/io7m/jtensors/core/parameterized/matrices
>  #26 = Utf8               com/io7m/jtensors/core/parameterized/vectors
>  #27 = Utf8               com/io7m/jtensors/core/quaternions
>  #28 = Utf8               com/io7m/jtensors/core/unparameterized/matrices
>  #29 = Utf8               com/io7m/jtensors/core/unparameterized/vectors
>  #30 = Utf8               com/io7m/jtensors/core
> {
> }
> SourceFile: "module-info.java"
> Module:
>  #5,0                                    // "com.io7m.jtensors.core"
>  #0
>  3                                       // requires
>    #6,8000                                 // "java.base" ACC_MANDATED
>    #7                                      // 10-internal
>    #8,40                                   // "org.immutables.value"
>    ACC_STATIC_PHASE
>    #0
>    #9,0                                    // "com.io7m.junreachable.core"
>    #0
>  8                                       // exports
>    #10,0                                   // com/io7m/jtensors/core/determinants
>    #11,0                                   // com/io7m/jtensors/core/dotproducts
>    #12,0                                   //
>    com/io7m/jtensors/core/parameterized/matrices
>    #13,0                                   //
>    com/io7m/jtensors/core/parameterized/vectors
>    #14,0                                   // com/io7m/jtensors/core/quaternions
>    #15,0                                   //
>    com/io7m/jtensors/core/unparameterized/matrices
>    #16,0                                   //
>    com/io7m/jtensors/core/unparameterized/vectors
>    #17,0                                   // com/io7m/jtensors/core
>  0                                       // opens
>  0                                       // uses
>  0                                       // provides
> ---
>
> Note the insertion of the "10-internal" string as constant pool entry
> 7.
>
> Does anyone have any idea why this would happen? I'm considering filing
> a bug at the AdoptOpenJDK project, but I'd like to be sure as to
> whether or not this is actually intended behaviour by newer compilers.
> Inserting version information obviously breaks reproducibility, so it'd
> be nice if it could be turned off!
>

I think it's the opposite, by default the compiler add the version of the required module,
so the AdoptOpenJDK version does the right thing, you should have the module compiled version by default,
the real question is why the compiler of the openjdk version 10.0.1 doesn't generate the required module version.

> The sources to the above module descriptor are available [2] if anyone
> wants to try compiling this for themselves. The SHA256 of the resulting
> module-info.class should be
> e7a94b5a2788a3c5cd6d7f586e70a0f2138a2eaa0e75144b6a16e75c4b297870 if
> your compiler produces the same output as mine.
>
> [0] https://reproducible-builds.org/
> [1] https://adoptopenjdk.net/
> [2] https://github.com/io7m/jtensors
>
> --
> Mark Raynsford | http://www.io7m.com


Rémi
Reply | Threaded
Open this post in threaded view
|

Re: Compiling module descriptors is not reproducible?

Remi Forax
In reply to this post by Alan Bateman
----- Mail original -----
> De: "Alan Bateman" <[hidden email]>
> À: "org openjdk" <[hidden email]>, "jigsaw-dev" <[hidden email]>
> Envoyé: Samedi 19 Mai 2018 13:26:55
> Objet: Re: Compiling module descriptors is not reproducible?

> On 19/05/2018 10:18, Mark Raynsford wrote:
>> Hello!
>>
>> I've been interested for a while in making all of my software builds
>> reproducible [0]. I don't think there's ever been any kind of
>> written guarantee that the output of javac will be completely
>> deterministic, but to date it seems like it actually has been. However,
>> I was a bit disappointed today to find that the output of one of the
>> builds at AdoptOpenJDK [1] seems to be inserting compiler version
>> information into compiled module descriptors. It doesn't seem to be
>> doing this for any other class files.
> The requires table in the Module attribute can be used by compilers to
> record version information about dependences. javac does record the
> version information, I don't know if there is a way to disable that.

It seems there is no way:
http://hg.openjdk.java.net/jdk/jdk/file/5ec7380f671d/src/jdk.compiler/share/classes/com/sun/tools/javac/jvm/ClassWriter.java#l986

> In any case, I assume your question will be partly answered by looking at
> the output of `java --list-modules` on the different builds. If you
> download OpenJDK builds from jdk.java.net/10 then you should see that
> the standard and JDK-specific modules all report their version string as
> "10" or "10.0.1" as they have been built with configure options that
> make it so. At a guess, your mail may have included the output from an
> exploded (as opposed to images) build.
>
> -Alan

Rémi
Reply | Threaded
Open this post in threaded view
|

Re: Compiling module descriptors is not reproducible?

Mark Raynsford
In reply to this post by Alan Bateman
On 2018-05-19T12:26:55 +0100
Alan Bateman <[hidden email]> wrote:

>
> The requires table in the Module attribute can be used by compilers to
> record version information about dependences. javac does record the
> version information, I don't know if there is a way to disable that. In
> any case, I assume your question will be partly answered by looking at
> the output of `java --list-modules` on the different builds. If you
> download OpenJDK builds from jdk.java.net/10 then you should see that
> the standard and JDK-specific modules all report their version string as
> "10" or "10.0.1" as they have been built with configure options that
> make it so. At a guess, your mail may have included the output from an
> exploded (as opposed to images) build.

That's a good point: What I thought I was looking at was the compiler
inserting it's own version into the class file (as in "this class file
was compiled by compiler version 10-internal") as opposed to the
versions of the modules being inserted.

Having the platform module versions present still implies
non-reproducibility but perhaps that's workable in some way.

On 2018-05-19T13:49:40 +0200
Remi Forax <[hidden email]> wrote:

> I think it's the opposite, by default the compiler add the version of the required module,
> so the AdoptOpenJDK version does the right thing, you should have the module compiled version by default,
> the real question is why the compiler of the openjdk version 10.0.1 doesn't generate the required module version.

That is another good point! This build is the one distributed on Arch
Linux. This was the procedure used to build it (no patches are applied):

  https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/java10-openjdk#n58

--
Mark Raynsford | http://www.io7m.com