Setting plugin path programmatically

Hdf-forum archives
1

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via
HDgetenv
which is defined as
#define HDgetenv(S) getenv(S)

According to

"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

···

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

2

Hi!

  It may be a problem with calling those functions before HDF5 filters are invoked and plugins are loaded. Such could happen in global constructors before a main() function is called, where this path is modified. Getting control over the order of global constructors in a C++ application is possible but can be difficult. Having such a runtime-API to modify the load path would solve some problems, for instance loading plugins from the application's install directory rather than a fixed compiled-in path - which is something I'd need in my applications. But for this I'm currently using a patch such that the HDF5_PLUGIN_PATH may contain a special two-character as placeholder for the application's install directory - for now, that is "@0" - and HDF5 replaces the statically compiled HDF5_PLUGIN_PATH containing this two-character with the directory available at runtime. This solves that problem in a constructor-safe way. Your scenario may be different though.

       Werner

···

On 03.03.2017 09:43, Андрей Парамонов wrote:

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:
https://support.hdfgroup.org/HDF5/doc1.8/Advanced/DynamicallyLoadedFilters/HDF5DynamicallyLoadedFilters.pdf

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via
HDgetenv
which is defined as
#define HDgetenv(S) getenv(S)

According to
http://stackoverflow.com/questions/10636768/does-getenv-cache-the-result/10636993#10636993

"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:
https://msdn.microsoft.com/ru-ru/library/windows/desktop/ms686206(v=vs.85).aspx

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

--
___________________________________________________________________________
Dr. Werner Benger Visualization Research
Center for Computation & Technology at Louisiana State University (CCT/LSU)
2019 Digital Media Center, Baton Rouge, Louisiana 70803
Tel.: +1 225 578 4809 Fax.: +1 225 578-5362

3

Hi Andrey,

I'll create an issue for this. We have a Windows layer that substitutes Win32 equivalents for some POSIX commands and it should be straightforward to use SetEnvironmentVariable() on Windows.

Dana Robinson
Software Engineer
The HDF Group

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Friday, March 3, 2017 3:43 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Setting plugin path programmatically

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via HDgetenv which is defined as
#define HDgetenv(S) getenv(S)

According to

"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

4

Oops. Sorry, I read this too quickly. I'll have to think more about your problem. I will create an issue for this, though.

Dana

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Friday, March 3, 2017 3:43 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Setting plugin path programmatically

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via HDgetenv which is defined as
#define HDgetenv(S) getenv(S)

According to

"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

5

Hi Dana,

Setting the plugin path programmatically will be very useful for those applications that bundle filters too as this will simplify their installation procedure.

Regards,
Peter

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Dana Robinson
Sent: 03 March 2017 15:58
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] Setting plugin path programmatically

Hi Andrey,

I'll create an issue for this. We have a Windows layer that substitutes Win32 equivalents for some POSIX commands and it should be straightforward to use SetEnvironmentVariable() on Windows.

Dana Robinson
Software Engineer
The HDF Group

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Friday, March 3, 2017 3:43 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Setting plugin path programmatically

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via HDgetenv which is defined as
#define HDgetenv(S) getenv(S)

According to

"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

6

Hello Dana!

03.03.2017 18:57, Dana Robinson пишет:

I'll create an issue for this. We have a Windows layer that substitutes Win32 equivalents for some POSIX commands and it should be straightforward to use SetEnvironmentVariable() on Windows.

I'm not sure whether this issue is in fact specific for Windows.

On one hand, on POSIX system getenv also uses cached values
(But is there a way to modify env.variables other than through setenv?)

On the other hand, POSIX culture of application deployment differs much from that of Windows, and distributing plugins in a local dir is probably not what people would actually like to do.

A simple (but probably incorrect) way to fix the issue if it's considered Windows-specific would be to define HDgetenv via GetEnvironmentVariable.

Best wishes,
Andrey Paramonov

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Friday, March 3, 2017 3:43 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Setting plugin path programmatically

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:
https://support.hdfgroup.org/HDF5/doc1.8/Advanced/DynamicallyLoadedFilters/HDF5DynamicallyLoadedFilters.pdf

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via HDgetenv which is defined as
#define HDgetenv(S) getenv(S)

According to
http://stackoverflow.com/questions/10636768/does-getenv-cache-the-result/10636993#10636993
"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:
https://msdn.microsoft.com/ru-ru/library/windows/desktop/ms686206(v=vs.85).aspx

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
This message has been scanned for viruses and
dangerous content by MailScanner, and is
believed to be clean.

7

Hi all,

I've created a JIRA issue for this request. For future reference, it's HDFFV-10143 in our system.

Cheers,

Dana

···

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Peter.Chang@Diamond.ac.uk
Sent: Friday, March 3, 2017 11:06 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] Setting plugin path programmatically

Hi Dana,

Setting the plugin path programmatically will be very useful for those applications that bundle filters too as this will simplify their installation procedure.

Regards,
Peter

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Dana Robinson
Sent: 03 March 2017 15:58
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] Setting plugin path programmatically

Hi Andrey,

I'll create an issue for this. We have a Windows layer that substitutes Win32 equivalents for some POSIX commands and it should be straightforward to use SetEnvironmentVariable() on Windows.

Dana Robinson
Software Engineer
The HDF Group

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Friday, March 3, 2017 3:43 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Setting plugin path programmatically

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via HDgetenv which is defined as
#define HDgetenv(S) getenv(S)

According to

"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

--
This e-mail and any attachments may contain confidential, copyright and or privileged material, and are for the use of the intended addressee only. If you are not the intended addressee or an authorised recipient of the addressee please notify us of receipt by returning the e-mail and do not use, copy, retain, distribute or disclose the information in or attached to the e-mail.
Any opinions expressed within this e-mail are those of the individual and not necessarily of Diamond Light Source Ltd.
Diamond Light Source Ltd. cannot guarantee that this e-mail or any attachments are free from viruses and we cannot accept liability for any damage which you may sustain as a result of software viruses which may be transmitted in or with the message.
Diamond Light Source Limited (company no. 4375679). Registered in England and Wales with its registered office at Diamond House, Harwell Science and Innovation Campus, Didcot, Oxfordshire, OX11 0DE, United Kingdom

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

8

Hi Dana,

You may want to look at using SetDllDirectory <https://msdn.microsoft.com/en-us/library/windows/desktop/ms686203(v=vs.85).aspx> as this allows the DLL directory to be explicitly specified.

Regards,
Dave

···

On 4/03/17 5:25 AM, Dana Robinson wrote:

Hi all,

I've created a JIRA issue for this request. For future reference, it's HDFFV-10143 in our system.

Cheers,

Dana

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Peter.Chang@Diamond.ac.uk
Sent: Friday, March 3, 2017 11:06 AM
To: hdf-forum@lists.hdfgroup.org
Subject: Re: [Hdf-forum] Setting plugin path programmatically

Hi Dana,

Setting the plugin path programmatically will be very useful for those applications that bundle filters too as this will simplify their installation procedure.

Regards,
  Peter

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of Dana Robinson
Sent: 03 March 2017 15:58
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: Re: [Hdf-forum] Setting plugin path programmatically

Hi Andrey,

I'll create an issue for this. We have a Windows layer that substitutes Win32 equivalents for some POSIX commands and it should be straightforward to use SetEnvironmentVariable() on Windows.

Dana Robinson
Software Engineer
The HDF Group

-----Original Message-----
From: Hdf-forum [mailto:hdf-forum-bounces@lists.hdfgroup.org] On Behalf Of ??? ???
Sent: Friday, March 3, 2017 3:43 AM
To: HDF Users Discussion List <hdf-forum@lists.hdfgroup.org>
Subject: [Hdf-forum] Setting plugin path programmatically

Hello!

My usage scenario is a big Windows application with a lot of dlls, some of which use HDF5 library. HDF5 code uses Dynamically Loaded Plugin Filters:
https://support.hdfgroup.org/HDF5/doc1.8/Advanced/DynamicallyLoadedFilters/HDF5DynamicallyLoadedFilters.pdf

The filters are deployed in a local directory which is activated via modifying process HDF5_PLUGIN_PATH env.variable, just prior to loading the library. That works fine, and important thing is that user can just drop new plugins in that directory to support newer files.

However, it appears that in some cases plugin search fails. HDF5 library retrieves HDF5_PLUGIN_PATH via HDgetenv which is defined as
#define HDgetenv(S) getenv(S)

According to
http://stackoverflow.com/questions/10636768/does-getenv-cache-the-result/10636993#10636993
"A process inherits the environment from the process creating the new process. This is held in memory."

In the application I do not use C runtime (the implementation language is not C), so I use SetEnvironmentVariable:
https://msdn.microsoft.com/ru-ru/library/windows/desktop/ms686206(v=vs.85).aspx

It appears that if I modify HDF5_PLUGIN_PATH before C runtime is loaded for the first time in the application, all is fine. However if I call SetEnvironmentVariable after C runtime is already loaded, it has no effect for HDF5.

The application is huge, and it's very hard to control the moments C runtime and HDF5 library are loaded. I wonder -- does it make sense to add HDF5 procedures to verbosely modify plugin search path, something like

herr_t H5PLadd_path(const char *name)
herr_t H5PLrem_path(const char *name)

?

The above procedures could use HDsetenv, which is otherwise not accessible (from non-C applications).
(I think similar motivation led to exposing H5free_memory procedure.)

What do you think?

Best wishes,
Andrey Paramonov

--
This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@lists.hdfgroup.org
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
Twitter: https://twitter.com/hdf5