Re: RFR: JDK-8241463 Move build tools to respective modules

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Re: RFR: JDK-8241463 Move build tools to respective modules

Mandy Chung
Hi Magnus,

Modularizing the build tools is a good move.    This patch suggests to
place the build tools under
     src/$MODULE/share/tools/$PACKAGE/*.java

I think the modular source location of the build tools needs more
discussion, including jigsaw-dev for this discussion.

The JDK source as specified in JEP 201 is under:
     src/$MODULE/{share,$OS}/classes/$PACKAGE/*.java

Compiling the source files from the `src` directory are the intermediate
input to build the resulting image.    Build tools are used to generate
additional intermediate input (that is not part of the `src` directory)
to build the image.   So I wonder if make/$MODULE/share/tools or
make/tools/$MODULE  may be better location for the build tools.

Mandy

On 3/23/20 12:03 PM, Magnus Ihse Bursie wrote:

> The build tools (small java tools that are run during the build to
> generate source code, or data, needed in the JDK) have historically
> been placed in the "make" directory. This maybe made sense long time
> ago, but does not do so anymore.
>
> Instead, the build tools source code should move the the module that
> needs them. For instance, compilefontconfig should move to
> java.desktop, etc.
>
> There are multiple reasons for this:
>
> * Currently we build *all* build tools at once, which mean that we
> cannot compile java.base until e.g. the compilefontconfig tool is
> compiled, even though it is not needed.
>
> * If a build tool, e.g. compilefontconfig is modified, all build tools
> are recompiled, which triggers a rebuild of more or less the entire
> JDK. This makes development of the build tools unnecessary tedious.
>
> * When the build tools are modified, the group owning the
> corresponding module is the proper review instance, not the build
> team. But since they reside under "make", the review mails often
> include build-dev, but this is mostly noise for us. With this move,
> the ownership is made clear.
>
> In this patch, I have not modified how and when the build tools are
> compiled, but this shuffle is the prerequisite for continuing with
> that in a follow-up patch.
>
> I have also moved the build tools to the org.openjdk.buildtools.*
> package name space (inspired by Skara), instead of the strangely named
> build.tools.* name space.
>
> A few build tools are not moved in this patch. Two of them,
> charsetmapping and cldrconverter, are shared between two modules. (I
> think they should move to modules nevertheless, but they need some
> more thought to make sure I do this right.) The rest are tools that
> are needed for the build in general, like linking or javadoc support.
> I'll move this to a better location too, but in a separate patch.
>
> Bug: https://bugs.openjdk.java.net/browse/JDK-8241463
> WebRev:
> http://cr.openjdk.java.net/~ihse/JDK-8241463-move-build-tools-to-modules/webrev.01
>
> /Magnus
>

Reply | Threaded
Open this post in threaded view
|

Re: RFR: JDK-8241463 Move build tools to respective modules

Magnus Ihse Bursie
On 2020-03-23 21:19, Mandy Chung wrote:
> Hi Magnus,
>
> Modularizing the build tools is a good move.
Thanks!
> This patch suggests to place the build tools under
>     src/$MODULE/share/tools/$PACKAGE/*.java
>
> I think the modular source location of the build tools needs more
> discussion, including jigsaw-dev for this discussion.

Ok, I've pruned the recipient list down to just jigsaw-dev and
build-dev. Let's keep this thread for discussing where to put the code
in the source tree, and when we agree on that I can make an updated
webrev and get buy-in from the component owners.

>
> The JDK source as specified in JEP 201 is under:
>     src/$MODULE/{share,$OS}/classes/$PACKAGE/*.java
>
> Compiling the source files from the `src` directory are the
> intermediate input to build the resulting image.    Build tools are
> used to generate additional intermediate input (that is not part of
> the `src` directory) to build the image.   So I wonder if
> make/$MODULE/share/tools or make/tools/$MODULE  may be better location
> for the build tools.
That's certainly a possibility. Ever since the big jigsaw restructuring,
Erik and I have discussed how to modularize the build system. I did not
even like the original jigsaw push, but there was not enough time to fix
the build system, and as a compromise we opened a bunch of JBS issues to
fix this in a follow-up fashion. Two are still remaining: JDK-8055192
and JDK-8055193, and JDK-8241463 is a part of that, even though it seems
to have not been written down in a bug report at that time.

At the core, we'd like to "invert" the current structure where we have
files like:
make/lib/Lib-java.base.gmk
make/lib/Lib-java.desktop.gmk
make/gensrc/Gensrc-java.base.gmk
make/gensrc/Gensrc-java.desktop.gmk
... etc

and instead have like:
make/modules/java.base/lib.gmk
make/modules/java.base/gensrc.gmk
make/modules/java.desktop/lib.gmk
make/modules/java.desktop/gensrc.gmk

However, this stuff is clearly a core part of the build system, so there
is no question that this belongs under "make".

In contrast, the build tools are only interacting with the build system
on the surface. There is some kind of an API here -- the build tools
have some kind of calling convention, wants some input and produces some
output, which is consumed by the build system. But the actual workings
of the build tools have 100% to do with the component. In some cases,
the original developer found it more suitable to create a tool for
generating programmatically a number of classes, rather than doing that
once, for hand, and check in the code. (I'm a bit skeptical towards
those build tools, btw, even though I understand the reason.) But this
clearly is just another way to express the core functionality of that
module. If anything is causing confusion right now, it's when the
component teams do not know about or take responsibility for the build
tools.

So I think the build tools are just like much of the rest of the product
-- the build system needs to be aware of them and how they work, but
they are really the expression -- and responsibility -- of the component
owners. And thus, I believe they are much better suited in the src tree.

Furthermore, we have JDK-8055193, about modularizing the make/data
directory. This is something I care even more about. And here I'm quite
firm in my conviction that this has nothing to do in the make directory.
It's just a remnant of the old thinking of "Oh, I don't know where to
put this. Let's just continue using 'make' as our misc trashcan". Things
like make/data/macosxicons should move to
src/java.desktop/data/macosxicons, and make/data/publicsuffixlist to
src/java.base/data/publicsuffixlist, and so forth. There's absolutely
nothing here that has anything to do with the build system. (With the
possible exception of product-wide stuff, I know these exists among the
tools, I'm not sure about the data.)

And since there is a strong correlation between the data files and the
build tools -- many build tools are custom made to process things in the
data directory -- I think it makes much sense to move the source code of
the tools along with them.

We already have collected everything else that belongs to a module under
src/$module/share. Apart from classes, and native, we have:
* conf
* lib
* legal
* man
for those modules that require them.

My suggestion is that we add, for those module that require them:
* data
* tools

If the name of the latter is not good enough, I'm open for suggestions.
Maybe "build-tools", "buildtools", "tools-src", "tools-classes",
"classes-tools"...? My idea for picking "tools" was that it was short,
and seemed to fit in well with the style of the other names. An
additional  benefit was that it was not as limiting as "buildtools", and
thus allowing for the modules to put other kinds of tools there that
might not be needed at build time -- for instance, tools that are needed
every once in a while to update some checked-in files, like
make/script/generate-symbol-data.sh (which really belongs to jdk.compiler).

And regarding JEP 201, as far as I can tell the additional "man" and
"lib" directories do not seem to be reflected in JEP 201. I do not know
if this is considered an oversight, or just reflecting the fact that the
directory structure will continue to evolve after JEP 201 were
delivered. But if you think JEP 201 needs to be updated, fine, I'll
gladly help with that.

/Magnus

>
> Mandy
>
> On 3/23/20 12:03 PM, Magnus Ihse Bursie wrote:
>> The build tools (small java tools that are run during the build to
>> generate source code, or data, needed in the JDK) have historically
>> been placed in the "make" directory. This maybe made sense long time
>> ago, but does not do so anymore.
>>
>> Instead, the build tools source code should move the the module that
>> needs them. For instance, compilefontconfig should move to
>> java.desktop, etc.
>>
>> There are multiple reasons for this:
>>
>> * Currently we build *all* build tools at once, which mean that we
>> cannot compile java.base until e.g. the compilefontconfig tool is
>> compiled, even though it is not needed.
>>
>> * If a build tool, e.g. compilefontconfig is modified, all build
>> tools are recompiled, which triggers a rebuild of more or less the
>> entire JDK. This makes development of the build tools unnecessary
>> tedious.
>>
>> * When the build tools are modified, the group owning the
>> corresponding module is the proper review instance, not the build
>> team. But since they reside under "make", the review mails often
>> include build-dev, but this is mostly noise for us. With this move,
>> the ownership is made clear.
>>
>> In this patch, I have not modified how and when the build tools are
>> compiled, but this shuffle is the prerequisite for continuing with
>> that in a follow-up patch.
>>
>> I have also moved the build tools to the org.openjdk.buildtools.*
>> package name space (inspired by Skara), instead of the strangely
>> named build.tools.* name space.
>>
>> A few build tools are not moved in this patch. Two of them,
>> charsetmapping and cldrconverter, are shared between two modules. (I
>> think they should move to modules nevertheless, but they need some
>> more thought to make sure I do this right.) The rest are tools that
>> are needed for the build in general, like linking or javadoc support.
>> I'll move this to a better location too, but in a separate patch.
>>
>> Bug: https://bugs.openjdk.java.net/browse/JDK-8241463
>> WebRev:
>> http://cr.openjdk.java.net/~ihse/JDK-8241463-move-build-tools-to-modules/webrev.01
>>
>> /Magnus
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: RFR: JDK-8241463 Move build tools to respective modules

Mandy Chung
Hi Magnus,

On 3/24/20 4:16 AM, Magnus Ihse Bursie wrote:

>
> At the core, we'd like to "invert" the current structure where we have
> files like:
> make/lib/Lib-java.base.gmk
> make/lib/Lib-java.desktop.gmk
> make/gensrc/Gensrc-java.base.gmk
> make/gensrc/Gensrc-java.desktop.gmk
> ... etc
>
> and instead have like:
> make/modules/java.base/lib.gmk
> make/modules/java.base/gensrc.gmk
> make/modules/java.desktop/lib.gmk
> make/modules/java.desktop/gensrc.gmk
>
> :
>
> We already have collected everything else that belongs to a module
> under src/$module/share. Apart from classes, and native, we have:
> * conf
> * lib
> * legal
> * man
> for those modules that require them.
>
> My suggestion is that we add, for those module that require them:
> * data
> * tools
>

I think we should take at modularizing make/data, build tools,
make/gensrc etc as a whole and put down other options considered.

I haven't had time looking closely but I try to understand how make/data
are used.  Here are some examples of it that produces either
gensrc/$MODULE/$PACKAGE/*.java files or files in jdk build output.

    - generate source files from gensrc/$MODULE/$PACKAGE/*.java.template
    - generate resource bundle from
gensrc/$MODULE/$PACKAGE/resource/*.properties
    - generate CLDR locale data from make/data/cldr files
    - generate jdk/lib/tzdb.dat from make/data/tzdata
    - generate jdk/lib/ct.sym from make/data/symbol
    - generate jdk/lib/security/cacerts from make/data/cacerts
    - generate jdk/lib/security/backlisted.certs from
make/data/blacklistedcertsconverter
    - generate jdk/lib/security/public_suffix_list.data from
make/data/publicsuffixlist

I can see why you propose the data files are moved to the source. There
could be other options to explore.

Another observation:  Some build tools are module-specific (e.g.
generate icons .png.java files) that is clearly used only by one
module.   There are other build tools that can be used by any module
e.g. generate resource bundle .java source.

> And regarding JEP 201, as far as I can tell the additional "man" and
> "lib" directories do not seem to be reflected in JEP 201. I do not
> know if this is considered an oversight, or just reflecting the fact
> that the directory structure will continue to evolve after JEP 201
> were delivered. But if you think JEP 201 needs to be updated, fine,
> I'll gladly help with that.
>

JEP 299 discusses the reorganization of man pages and specification and
specifies the man directory.  When JEP 201 was written, the man
directory was not present (part of moving the specs effort to the
openjdk source).   It was added in JDK 12.  "lib" directory is an oversight.

JEP 201 may need update for this build modularization work too.

Mandy