native cmds and libs in the module library

classic Classic list List threaded Threaded
20 messages Options
Reply | Threaded
Open this post in threaded view
|

native cmds and libs in the module library

chris.hegarty
Hi,

I've been looking into where native commands and native libraries get
installed in the modules library. Today the files in the NATIVE_LIBS and
NATIVE_CMDS sections of jmod files [1] are installed into directories in
the the module library. This is not ideal for a few reasons:

1. Native commands hidden down in modules directory are not easy to
    find, and complicates the PATH.

2. One native library may have a dependency on another, which
    complicates the LD_LIBRARY_PATH. Again, having to poke around
    in the modules directory of each module.

3. JDK images cannot be built with simple jmod create/install. This
    would be nice! Currently, in the build native libs and cmds are
    copied into place.

4. At some point you could imagine it would be desirable to install
    native commands into /usr/bin.

I've prototyped changes to the SimpleLibrary where, at the point of
creation (jmod create), you can optionally specify a path for native
libs and cmds. The creator of the library can decide where these get
installed. Then, when a jmod file is installed the native cmds and libs
are placed in the libraries appropriate directory.

If 'jmod create' is called without these optional arguments then the
current behavior is maintained, native libs and cmds are installed in
each module under its 'lib' and 'bin' directories, respectively. I think
this behavior may be desirable in some cases, a module has a single
native library that implements some platform specific operation.

http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.00/webrev/

Open issues: impact on native deb packages when natlibs/cmds is
specified outside of the module library.

Thanks,
-Chris.

[1] http://cr.openjdk.java.net/~mr/jigsaw/notes/module-file-format/
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Alan Bateman
On 12/12/2011 16:46, Chris Hegarty wrote:

> Hi,
>
> I've been looking into where native commands and native libraries get
> installed in the modules library. Today the files in the NATIVE_LIBS
> and NATIVE_CMDS sections of jmod files [1] are installed into
> directories in the the module library. This is not ideal for a few
> reasons:
>
> 1. Native commands hidden down in modules directory are not easy to
>    find, and complicates the PATH.
>
> 2. One native library may have a dependency on another, which
>    complicates the LD_LIBRARY_PATH. Again, having to poke around
>    in the modules directory of each module.
>
> 3. JDK images cannot be built with simple jmod create/install. This
>    would be nice! Currently, in the build native libs and cmds are
>    copied into place.
>
> 4. At some point you could imagine it would be desirable to install
>    native commands into /usr/bin.
>
> I've prototyped changes to the SimpleLibrary where, at the point of
> creation (jmod create), you can optionally specify a path for native
> libs and cmds. The creator of the library can decide where these get
> installed. Then, when a jmod file is installed the native cmds and
> libs are placed in the libraries appropriate directory.
>
> If 'jmod create' is called without these optional arguments then the
> current behavior is maintained, native libs and cmds are installed in
> each module under its 'lib' and 'bin' directories, respectively. I
> think this behavior may be desirable in some cases, a module has a
> single native library that implements some platform specific operation.
>
> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.00/webrev/
The approach to optionally specify the location for commands and native
libraries make sense to me. I suspect defaulting to directories in each
module will need a bit of thought, at least for legacy mode.

Once commands and native libraries are installed to locations outside of
the per module directory then it does mean handling the case that files
already exist. It also means the module library needs somewhere to track
these files so that removing a module knows to remove the files in these
other locations.

The proposed option to jmod create are -natcmd and -natlibs but as they
are long names then I assume they should be --.

-Alan.



Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Sean Mullan
On most systems, you'll probably also run into file permission issues if you
aren't superuser or an administrator. Is this something you are going to detect
up-front and warn the user or will the command just fail when it tries to
install the files?

--Sean

On 12/19/11 10:34 AM, Alan Bateman wrote:

> On 12/12/2011 16:46, Chris Hegarty wrote:
>> Hi,
>>
>> I've been looking into where native commands and native libraries get
>> installed in the modules library. Today the files in the NATIVE_LIBS
>> and NATIVE_CMDS sections of jmod files [1] are installed into
>> directories in the the module library. This is not ideal for a few
>> reasons:
>>
>> 1. Native commands hidden down in modules directory are not easy to
>>    find, and complicates the PATH.
>>
>> 2. One native library may have a dependency on another, which
>>    complicates the LD_LIBRARY_PATH. Again, having to poke around
>>    in the modules directory of each module.
>>
>> 3. JDK images cannot be built with simple jmod create/install. This
>>    would be nice! Currently, in the build native libs and cmds are
>>    copied into place.
>>
>> 4. At some point you could imagine it would be desirable to install
>>    native commands into /usr/bin.
>>
>> I've prototyped changes to the SimpleLibrary where, at the point of
>> creation (jmod create), you can optionally specify a path for native
>> libs and cmds. The creator of the library can decide where these get
>> installed. Then, when a jmod file is installed the native cmds and
>> libs are placed in the libraries appropriate directory.
>>
>> If 'jmod create' is called without these optional arguments then the
>> current behavior is maintained, native libs and cmds are installed in
>> each module under its 'lib' and 'bin' directories, respectively. I
>> think this behavior may be desirable in some cases, a module has a
>> single native library that implements some platform specific operation.
>>
>> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.00/webrev/
> The approach to optionally specify the location for commands and native
> libraries make sense to me. I suspect defaulting to directories in each
> module will need a bit of thought, at least for legacy mode.
>
> Once commands and native libraries are installed to locations outside of
> the per module directory then it does mean handling the case that files
> already exist. It also means the module library needs somewhere to track
> these files so that removing a module knows to remove the files in these
> other locations.
>
> The proposed option to jmod create are -natcmd and -natlibs but as they
> are long names then I assume they should be --.
>
> -Alan.
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Alan Bateman
On 19/12/2011 16:23, Sean Mullan wrote:
> On most systems, you'll probably also run into file permission issues if you
> aren't superuser or an administrator. Is this something you are going to detect
> up-front and warn the user or will the command just fail when it tries to
> install the files?
>
> --Sean
>
If the "jmod create" command creates the bin and lib directories and it
fails then that gives us the error. If it doesn't create the directories
then it could emit a warning if they aren't writable. In any case, if
the install fails for any reason then it should rollback all changes.

-Alan.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

chris.hegarty
On 12/20/11 01:14 PM, Alan Bateman wrote:

> On 19/12/2011 16:23, Sean Mullan wrote:
>> On most systems, you'll probably also run into file permission issues
>> if you
>> aren't superuser or an administrator. Is this something you are going
>> to detect
>> up-front and warn the user or will the command just fail when it tries to
>> install the files?
>>
>> --Sean
>>
> If the "jmod create" command creates the bin and lib directories and it
> fails then that gives us the error. If it doesn't create the directories
> then it could emit a warning if they aren't writable. In any case, if
> the install fails for any reason then it should rollback all changes.

Right, I'm in the process of making this change. Currently we only
remove the module directory, we need to track out of module content and
remove it if necessary. This will hopefully be useful for a 'jmod
remove' command in the future too.

I was planning to use the same rollback strategy if a conflict, a native
library or command already exists with the same name, occurs. I'm also
incorporating your feedback from yesterday.

-Chris.

>
> -Alan.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

chris.hegarty
Here is an updated webrev incorporating the feedback I received.

http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.01/webrev/

1) '-natlibs' -> '--natlibs', '-natcmds' -> '--natcmds'
2) detect during 'jmod create' if a directory is creatable/writable
3) detect conflicts and rollback changes if there is a conflict
4) track files installed outside of the module directory

It would be nice to allow an override of files in the module libraries
natlibs or natcmds directories if a newer version of the same module is
installed. Currently this will fail due to a conflict on these files.
This shouldn't be difficult to add later.

-Chris.

On 20/12/2011 13:22, Chris Hegarty wrote:

> On 12/20/11 01:14 PM, Alan Bateman wrote:
>> On 19/12/2011 16:23, Sean Mullan wrote:
>>> On most systems, you'll probably also run into file permission issues
>>> if you
>>> aren't superuser or an administrator. Is this something you are going
>>> to detect
>>> up-front and warn the user or will the command just fail when it
>>> tries to
>>> install the files?
>>>
>>> --Sean
>>>
>> If the "jmod create" command creates the bin and lib directories and it
>> fails then that gives us the error. If it doesn't create the directories
>> then it could emit a warning if they aren't writable. In any case, if
>> the install fails for any reason then it should rollback all changes.
>
> Right, I'm in the process of making this change. Currently we only
> remove the module directory, we need to track out of module content and
> remove it if necessary. This will hopefully be useful for a 'jmod
> remove' command in the future too.
>
> I was planning to use the same rollback strategy if a conflict, a native
> library or command already exists with the same name, occurs. I'm also
> incorporating your feedback from yesterday.
>
> -Chris.
>
>>
>> -Alan.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Alan Bateman
On 06/01/2012 16:14, Chris Hegarty wrote:

> Here is an updated webrev incorporating the feedback I received.
>
> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.01/webrev/
>
> 1) '-natlibs' -> '--natlibs', '-natcmds' -> '--natcmds'
> 2) detect during 'jmod create' if a directory is creatable/writable
> 3) detect conflicts and rollback changes if there is a conflict
> 4) track files installed outside of the module directory
>
> It would be nice to allow an override of files in the module libraries
> natlibs or natcmds directories if a newer version of the same module
> is installed. Currently this will fail due to a conflict on these
> files. This shouldn't be difficult to add later.
>
> -Chris.
>
I went through this and I think it's starting to come good.

One concern is that ModuleFile now only works with a SimpleLibrary
whereas I think it should be able to work with any Library implementation.

Also "oomfiles" is a strange name to have in the module library.
Something like "otherfiles" or just "files" might be better. For
consistency and portability it would be better to have it in UTF8 rather
than the platform encoding.

It might be worth combining getSubdirOfSection and librarySectionPath
into one instance method that returns the location to install the
section (be in the module library or elsewhere).

Security is a concern where an adversary may include a sequence of ../
in the relative path to a library. I think it would be good to include a
couple of tests for this type of issue.

Otherwise, I think the approach is okay, just needs clean-up and
ensuring that the coding style and naming is consistent.

-Alan.













Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Mandy Chung
In reply to this post by chris.hegarty
On 1/6/2012 8:14 AM, Chris Hegarty wrote:
> Here is an updated webrev incorporating the feedback I received.
>
> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.01/webrev/
>

This is a good start.  We can extend this approach for the
configuration files section as these files are intended
to be edited by the users; otherwise the users would require
to know the module library layout to find the config files.

The jdk modules in the current jigsaw prototype don't have
the "configs" section because it requires changes in the
JDK implementation to get the properties file at the new
location (possibly add a new API to obtain it) that hasn't
happened yet.  The properties files currently remains under
the $JAVA_HOME/lib directory or its subdirectory as in the
legacy image.  It also requires changes in the jdk build.
It would be good to add the support now to prepare the future
transition.

It makes sense to me to specify an alternative path for the
native cmds and config files when creating the module library
but I am not quite sure about the native libraries.  Native
libraries in a module are typically the implementation to support
its own module and not have a dependency on another.  There
are 7 jdk modules that have native libs but no native cmd nor
config files.  jdk.sctp is the only module whose native lib
(libsctp.so) has dependency on native libs on another module.
There are 6 jdk modules that have native libs and also either
native cmds or config files.  Attached is the matrix I have
from the jdk I built with module views prototype.

I am wondering what the common use cases are and whether
--natlib option should be a per-module base or per-library base?

For the per-library base as in your proposal, it's simpler and
consistent with other files (native cmds and config files) but
modules with native library but no dependency on other module's
native library can only support one single version unless the
native library uses shared library versioning.  Would it make
sense to support the --natlib option as per-module base
(i.e. jmod install option)?

> It would be nice to allow an override of files in the module libraries
> natlibs or natcmds directories if a newer version of the same module
> is installed. Currently this will fail due to a conflict on these
> files. This shouldn't be difficult to add later.

In addition, when we remove a module of a specific version, it
should restore natcmds/natlibs in the latest version of the same
module in the module library, if present.

As for my comments to the implementation, I agree with Alan
that the filename "oomfiles" is a bit strange and simply
naming it "files" seems good to me.  Some other comments:

ModuleFile.java
    L129: Rather than passing SimpleLibrary as a parameter
    to the Reader constructor, perhaps it's better to pass
    the paths to the readRest method to specify the destination
    to install section content for different section types.

    L450-454&  472-476: it might be good to combine the
    markNativeCodeExecutable and trackOutOfModuleContent methods
    as both are post-extract steps.

    L567-587: as Alan also pointed out, this might be worth
    having a method to return the path of a given section type.

SimpleLibrary.java
    L264: Files.ensureIsDirectory is the same utility method
    but is private.  Perhaps just use the existing one.
    L287, 302: looks like these lines can just call the
    ensureDirectory method.

Librarian.java
    L50-51: this can be replaced with Library.systemLibraryPath().

Mandy


jdk-modules.txt (728 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

chris.hegarty
Here is an updated webrev based on the comments and feedback I received
on this.

  - An additional option is added for supporting config files
  - jdk/jre images are created by installing jmod files ( if available )
  - Some code cleanup and style changes

http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.03/webrev/

Issues I intend to address separately:

  - Separate out native libs and config files during the build, so they
    can be put into the right sections of the jmod file
  - Investigate the potential for global configuration files versus
    non-editable per-module config files.

Thanks,
-Chris.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Mandy Chung
On 1/17/2012 6:28 AM, Chris Hegarty wrote:

> Here is an updated webrev based on the comments and feedback I received
> on this.
>
>   - An additional option is added for supporting config files
If a relative path is specified as an argument to --natlib, --natbin,
or --config option, what is it relative to and what gets recorded
in the module library header? I believe the implementation currently
uses cwd (i.e. the directory where jmod install is invoked) which
means that we can't simply copy the module library and the natlibs
to a different location to use?? See my comment for Modules.gmk.


> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.03/webrev/
Modules.gmk
    L141: --natlib $mlib/.. --natcmd $mlib/../../bin --config $$mlib/..

$mlib is an absolute path.  Since these alt paths are recorded
in the module library header, if we zip up the jdk-module-image and
unzip it at a different location, these paths need fixup in order
to install new modules in it.  It seems that we should keep
these paths relative to the system module library when creating
the jdk modules image.

Similarly, the realpath of the files are stored in "files"
which would need fixup too which is not ideal.

SimpleLibrary.java
    L346-348: I wonder if it's cleaner to initialize these instance
    variables to the default path, i.e. new File(root, "bin") if the
    given argument is null so that the null check is not needed in
    various place.

    L301: I think this warning should be output in the Librarian
    implementation rather than in the SimpleLibrary.  Instead
    it should throw an exception if the natlibs is not writeable
    and when it writes a native library.

    L1289: it's a good question when we should check the extension.
    It depends if the specification requires the extension of a
    module file be a specific value.  I think we can leave as it
    is and get back to it later.

ModuleFile.java
    L199-201: may be cleaner if these variables are initialized
    to the default path e.g. File(destination, "lib") and the
    non-null check in the getDirOfSection method will not be needed.

    L581-585: minor: since they are interested in 3 section types,
    would it be better to handle each section type in this postExtract
    method rather than checking the section type in both
    markNativeCodeExecutable and trackFiles methods. Probably the
    markNativeCodeExecutable method can be merged with this postExtract
    method and remove L501-504 as well.

Librarian.java
    L55, 151: this comment line can be removed.

hello-native.sh
    It might be good to verify if the path of the native library
    is expected.


> Issues I intend to address separately:
>
>  - Separate out native libs and config files during the build, so they
>    can be put into the right sections of the jmod file
>  - Investigate the potential for global configuration files versus
>    non-editable per-module config files.

That's fine with me.

Mandy

>
> Thanks,
> -Chris.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

chris.hegarty
Thanks Mandy,

You are correct. I was storing absolute paths.

Updated webrev:
   http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.04/webrev/

  - Paths to 'jmod create -L lib -natlib <dir> ...' are resolved
    relative to the current working dir, as you would expect.
  - natlib, natcmd, config, paths are stored relative to the module
    library root, in the libraries metadata.
  - The per module 'files' contains paths relative to the modules
    install directory.

  - I made most of the specific code suggestions you said, with the
    exception that natlibs, natcmds, and configs can still be null
    in SimpleLibrary. I use this to differentiate between a passed
    path and a per module path.

-Chris.

On 01/18/12 07:03 AM, Mandy Chung wrote:

> On 1/17/2012 6:28 AM, Chris Hegarty wrote:
>
>> Here is an updated webrev based on the comments and feedback I received
>> on this.
>>
>> - An additional option is added for supporting config files
> If a relative path is specified as an argument to --natlib, --natbin,
> or --config option, what is it relative to and what gets recorded
> in the module library header? I believe the implementation currently
> uses cwd (i.e. the directory where jmod install is invoked) which
> means that we can't simply copy the module library and the natlibs
> to a different location to use?? See my comment for Modules.gmk.
>
>
>> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.03/webrev/
> Modules.gmk
> L141: --natlib $mlib/.. --natcmd $mlib/../../bin --config $$mlib/..
>
> $mlib is an absolute path. Since these alt paths are recorded
> in the module library header, if we zip up the jdk-module-image and
> unzip it at a different location, these paths need fixup in order
> to install new modules in it. It seems that we should keep
> these paths relative to the system module library when creating
> the jdk modules image.
>
> Similarly, the realpath of the files are stored in "files"
> which would need fixup too which is not ideal.
>
> SimpleLibrary.java
> L346-348: I wonder if it's cleaner to initialize these instance
> variables to the default path, i.e. new File(root, "bin") if the
> given argument is null so that the null check is not needed in
> various place.
>
> L301: I think this warning should be output in the Librarian
> implementation rather than in the SimpleLibrary. Instead
> it should throw an exception if the natlibs is not writeable
> and when it writes a native library.
>
> L1289: it's a good question when we should check the extension.
> It depends if the specification requires the extension of a
> module file be a specific value. I think we can leave as it
> is and get back to it later.
>
> ModuleFile.java
> L199-201: may be cleaner if these variables are initialized
> to the default path e.g. File(destination, "lib") and the
> non-null check in the getDirOfSection method will not be needed.
>
> L581-585: minor: since they are interested in 3 section types,
> would it be better to handle each section type in this postExtract
> method rather than checking the section type in both
> markNativeCodeExecutable and trackFiles methods. Probably the
> markNativeCodeExecutable method can be merged with this postExtract
> method and remove L501-504 as well.
>
> Librarian.java
> L55, 151: this comment line can be removed.
>
> hello-native.sh
> It might be good to verify if the path of the native library
> is expected.
>
>
>> Issues I intend to address separately:
>>
>> - Separate out native libs and config files during the build, so they
>> can be put into the right sections of the jmod file
>> - Investigate the potential for global configuration files versus
>> non-editable per-module config files.
>
> That's fine with me.
>
> Mandy
>
>>
>> Thanks,
>> -Chris.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Mandy Chung
Thanks, Chris. This looks much better.

Modules.gmk
   L141: I would suggest to replace "$$mlib/.." with
   "$(MODULE_IMAGES_DIR)/$$image/bin" to make the path explicit and
   similarly for "$$mlib/../../bin".  Perhaps define a new variable for
   the root "$(MODULE_IMAGES_DIR)/$$image" would help.

   L191: since the jmod files have been created, do we really need
   L195-197?

SimpleLibrary.java
   L228-232: should they be final fields?

ModuleFile.java
   L295: extra blank line.

Mandy

On 1/20/2012 9:56 AM, Chris Hegarty wrote:

> Thanks Mandy,
>
> You are correct. I was storing absolute paths.
>
> Updated webrev:
>   http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.04/webrev/
>
>  - Paths to 'jmod create -L lib -natlib <dir> ...' are resolved
>    relative to the current working dir, as you would expect.
>  - natlib, natcmd, config, paths are stored relative to the module
>    library root, in the libraries metadata.
>  - The per module 'files' contains paths relative to the modules
>    install directory.
>
>  - I made most of the specific code suggestions you said, with the
>    exception that natlibs, natcmds, and configs can still be null
>    in SimpleLibrary. I use this to differentiate between a passed
>    path and a per module path.
>
> -Chris.
>
> On 01/18/12 07:03 AM, Mandy Chung wrote:
>> On 1/17/2012 6:28 AM, Chris Hegarty wrote:
>>
>>> Here is an updated webrev based on the comments and feedback I received
>>> on this.
>>>
>>> - An additional option is added for supporting config files
>> If a relative path is specified as an argument to --natlib, --natbin,
>> or --config option, what is it relative to and what gets recorded
>> in the module library header? I believe the implementation currently
>> uses cwd (i.e. the directory where jmod install is invoked) which
>> means that we can't simply copy the module library and the natlibs
>> to a different location to use?? See my comment for Modules.gmk.
>>
>>
>>> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.03/webrev/
>> Modules.gmk
>> L141: --natlib $mlib/.. --natcmd $mlib/../../bin --config $$mlib/..
>>
>> $mlib is an absolute path. Since these alt paths are recorded
>> in the module library header, if we zip up the jdk-module-image and
>> unzip it at a different location, these paths need fixup in order
>> to install new modules in it. It seems that we should keep
>> these paths relative to the system module library when creating
>> the jdk modules image.
>>
>> Similarly, the realpath of the files are stored in "files"
>> which would need fixup too which is not ideal.
>>
>> SimpleLibrary.java
>> L346-348: I wonder if it's cleaner to initialize these instance
>> variables to the default path, i.e. new File(root, "bin") if the
>> given argument is null so that the null check is not needed in
>> various place.
>>
>> L301: I think this warning should be output in the Librarian
>> implementation rather than in the SimpleLibrary. Instead
>> it should throw an exception if the natlibs is not writeable
>> and when it writes a native library.
>>
>> L1289: it's a good question when we should check the extension.
>> It depends if the specification requires the extension of a
>> module file be a specific value. I think we can leave as it
>> is and get back to it later.
>>
>> ModuleFile.java
>> L199-201: may be cleaner if these variables are initialized
>> to the default path e.g. File(destination, "lib") and the
>> non-null check in the getDirOfSection method will not be needed.
>>
>> L581-585: minor: since they are interested in 3 section types,
>> would it be better to handle each section type in this postExtract
>> method rather than checking the section type in both
>> markNativeCodeExecutable and trackFiles methods. Probably the
>> markNativeCodeExecutable method can be merged with this postExtract
>> method and remove L501-504 as well.
>>
>> Librarian.java
>> L55, 151: this comment line can be removed.
>>
>> hello-native.sh
>> It might be good to verify if the path of the native library
>> is expected.
>>
>>
>>> Issues I intend to address separately:
>>>
>>> - Separate out native libs and config files during the build, so they
>>> can be put into the right sections of the jmod file
>>> - Investigate the potential for global configuration files versus
>>> non-editable per-module config files.
>>
>> That's fine with me.
>>
>> Mandy
>>
>>>
>>> Thanks,
>>> -Chris.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Alan Bateman
In reply to this post by chris.hegarty
On 20/01/2012 17:56, Chris Hegarty wrote:

> Thanks Mandy,
>
> You are correct. I was storing absolute paths.
>
> Updated webrev:
>   http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.04/webrev/
>
>  - Paths to 'jmod create -L lib -natlib <dir> ...' are resolved
>    relative to the current working dir, as you would expect.
>  - natlib, natcmd, config, paths are stored relative to the module
>    library root, in the libraries metadata.
>  - The per module 'files' contains paths relative to the modules
>    install directory.
>
>  - I made most of the specific code suggestions you said, with the
>    exception that natlibs, natcmds, and configs can still be null
>    in SimpleLibrary. I use this to differentiate between a passed
>    path and a per module path.
>
> -Chris.
I think this quite good.

Can you briefly summarize how one would use this on Windows? I assume it
requires specifying the --natcmd and --natlib to be the same directory.

In the simple library layout I added a flags word some time ago and it
has many spare bits. Rather than storing 0 or 1 before each path then an
alternative would be to use some of the spare bits.

A minor nit but the code style with if-then-else is a bit inconsistent.
SimpleLibrary.storePath, SimpleLibrary.resolveAndEnsurePath,
Librarian.Create.go are just a couple of examples that jump out (there
are many more).

A general comment is that we'll need to go over this again once we get
to clean-up the implementation, improve performance, make library
operations atomic, etc. I mention this because what we have now is a bit
inefficient and there's also quite a bit of hopping between new and old
APIs. It will need a major clean-up at some point.

On the tests then I think an important test to include is a test that
attempts to install a library or binary outside of the bin and lib
trees. I think the validation code that you've added will catch this but
I think a test is important.

-Alan.





Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

chris.hegarty
Alan, Mandy,

Thank you for your comments.

In short, I would like to integrate what is in the latest webrev (with
your approval/review), and then continue the remainder of this work as a
separate task.

Latest webrev
   http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.05/webrev/

Changes:
  - SimpleLibrary parent, natlib, natcmd, and config are now final
    (Mandys comment).
  - Cleanup some style issues.
  - Backout changes to Modules.gmk (install from jmod files).

Long description.

I think it best to postpone to the next phase of this work creating the
images by installing from jmod files. The reason being that this doesn't
work very well on Windows yet.

The jmod files for Windows contain both commands and libraries in the
NATIVE_CMDS section of the jmod file, and config/property files in the
NATIVE_LIBS section. It will not work to simply set --natcmd bin, and
--natlib lib, as the findLocalNativeLibrary will then be searching in
the wrong place.

The solution is to have the appropriate files put in the correct section
of the jmod file in the first place, and this will require separating
out commands, libraries, and config files during the build <sigh>. I
think this is best done as a separate task.

Thanks,
-Chris.


On 24/01/2012 14:15, Alan Bateman wrote:

> ....
> Can you briefly summarize how one would use this on Windows? I assume it
> requires specifying the --natcmd and --natlib to be the same directory.
>
> In the simple library layout I added a flags word some time ago and it
> has many spare bits. Rather than storing 0 or 1 before each path then an
> alternative would be to use some of the spare bits.
>
> A minor nit but the code style with if-then-else is a bit inconsistent.
> SimpleLibrary.storePath, SimpleLibrary.resolveAndEnsurePath,
> Librarian.Create.go are just a couple of examples that jump out (there
> are many more).
>
> A general comment is that we'll need to go over this again once we get
> to clean-up the implementation, improve performance, make library
> operations atomic, etc. I mention this because what we have now is a bit
> inefficient and there's also quite a bit of hopping between new and old
> APIs. It will need a major clean-up at some point.
>
> On the tests then I think an important test to include is a test that
> attempts to install a library or binary outside of the bin and lib
> trees. I think the validation code that you've added will catch this but
> I think a test is important.
>
> -Alan.
>
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Mandy Chung
On 1/25/12 7:28 AM, Chris Hegarty wrote:
>
> In short, I would like to integrate what is in the latest webrev (with
> your approval/review), and then continue the remainder of this work as
> a separate task.
>
That's fine with me.  Your new webrev looks good.   It's also better for
this changeset to go first before the module views that I will do the merge.

>
> The solution is to have the appropriate files put in the correct
> section of the jmod file in the first place, and this will require
> separating out commands, libraries, and config files during the build
> <sigh>. I think this is best done as a separate task.

Agree and this would require quite some amount of works.  I can help
working with you on that.

Thanks
Mandy
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Alan Bateman
In reply to this post by chris.hegarty
On 25/01/2012 15:28, Chris Hegarty wrote:

> Alan, Mandy,
>
> Thank you for your comments.
>
> In short, I would like to integrate what is in the latest webrev (with
> your approval/review), and then continue the remainder of this work as
> a separate task.
>
> Latest webrev
>   http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.05/webrev/
>
> Changes:
>  - SimpleLibrary parent, natlib, natcmd, and config are now final
>    (Mandys comment).
>  - Cleanup some style issues.
>  - Backout changes to Modules.gmk (install from jmod files).
>
> Long description.
>
> I think it best to postpone to the next phase of this work creating
> the images by installing from jmod files. The reason being that this
> doesn't work very well on Windows yet.
>
> The jmod files for Windows contain both commands and libraries in the
> NATIVE_CMDS section of the jmod file, and config/property files in the
> NATIVE_LIBS section. It will not work to simply set --natcmd bin, and
> --natlib lib, as the findLocalNativeLibrary will then be searching in
> the wrong place.
>
> The solution is to have the appropriate files put in the correct
> section of the jmod file in the first place, and this will require
> separating out commands, libraries, and config files during the build
> <sigh>. I think this is best done as a separate task.
>
> Thanks,
> -Chris.
>
I agree with Mandy, I think you should just push the patch that you have
now and address the separation and Windows issues in a second phase.

-Alan.

Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

chris.hegarty

On 01/26/12 11:52 AM, Alan Bateman wrote:
> .....
>>
> I agree with Mandy, I think you should just push the patch that you have
> now and address the separation and Windows issues in a second phase.

Thanks Alan, and Mandy.

I will push this patch soon.

-Chris.

>
> -Alan.
>
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

David Holmes
In reply to this post by Alan Bateman
So this may be the extra work you are referring to but I was surprised
to discover that doing:

jmod create -L foo --natcmd bin --natlib lib

didn't create foo/bin and foo/lib but rather created ./bin and ./lib

David

On 26/01/2012 9:52 PM, Alan Bateman wrote:

> On 25/01/2012 15:28, Chris Hegarty wrote:
>> Alan, Mandy,
>>
>> Thank you for your comments.
>>
>> In short, I would like to integrate what is in the latest webrev (with
>> your approval/review), and then continue the remainder of this work as
>> a separate task.
>>
>> Latest webrev
>> http://cr.openjdk.java.net/~chegar/jigsaw/libbin_webrev.05/webrev/
>>
>> Changes:
>> - SimpleLibrary parent, natlib, natcmd, and config are now final
>> (Mandys comment).
>> - Cleanup some style issues.
>> - Backout changes to Modules.gmk (install from jmod files).
>>
>> Long description.
>>
>> I think it best to postpone to the next phase of this work creating
>> the images by installing from jmod files. The reason being that this
>> doesn't work very well on Windows yet.
>>
>> The jmod files for Windows contain both commands and libraries in the
>> NATIVE_CMDS section of the jmod file, and config/property files in the
>> NATIVE_LIBS section. It will not work to simply set --natcmd bin, and
>> --natlib lib, as the findLocalNativeLibrary will then be searching in
>> the wrong place.
>>
>> The solution is to have the appropriate files put in the correct
>> section of the jmod file in the first place, and this will require
>> separating out commands, libraries, and config files during the build
>> <sigh>. I think this is best done as a separate task.
>>
>> Thanks,
>> -Chris.
>>
> I agree with Mandy, I think you should just push the patch that you have
> now and address the separation and Windows issues in a second phase.
>
> -Alan.
>
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

Alan Bateman
On 31/01/2012 05:19, David Holmes wrote:
> So this may be the extra work you are referring to but I was surprised
> to discover that doing:
>
> jmod create -L foo --natcmd bin --natlib lib
>
> didn't create foo/bin and foo/lib but rather created ./bin and ./lib
Right, because if you specify relative directories then they are
relative to pwd.

If it helps then creating a custom image containing the system library
then it can be done with steps such as:

mkdir -p image/lib
jmod create -L image/lib/modules --natcmd image/bin --natlib image/lib
jmod install -L image/lib/modules [hidden email]

and that will give you a layout equivalent of the build's jre-base-image.

-Alan.
Reply | Threaded
Open this post in threaded view
|

Re: native cmds and libs in the module library

David Holmes
On 31/01/2012 6:04 PM, Alan Bateman wrote:
> On 31/01/2012 05:19, David Holmes wrote:
>> So this may be the extra work you are referring to but I was surprised
>> to discover that doing:
>>
>> jmod create -L foo --natcmd bin --natlib lib
>>
>> didn't create foo/bin and foo/lib but rather created ./bin and ./lib
> Right, because if you specify relative directories then they are
> relative to pwd.

I was thinking of the Library as being a "root" directory for
everything, but its just the location to place the %jigsaw-library file,
and the installed modules.

> If it helps then creating a custom image containing the system library
> then it can be done with steps such as:
>
> mkdir -p image/lib
> jmod create -L image/lib/modules --natcmd image/bin --natlib image/lib
> jmod install -L image/lib/modules [hidden email]
>
> and that will give you a layout equivalent of the build's jre-base-image.

Many thanks - and that worked without the IllegalArgumentException I
reported in separate email.

David
-----

> -Alan.