# can not open/close a file more than 1010 times **URL:** \ **Category:** Hdf-forum archives\ **Created:** [February 5, 2012, 10:20pm UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458 "2012-02-05T22:20:39Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![steph](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/s/22d042/32.png) [@steph](https://forum.hdfgroup.org/u/steph)\ **Post date:** [February 5, 2012, 10:20pm UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/1 "2012-02-05T22:20:39Z")
Dear all, I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. Problem: With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two: is there a limit of the number of hdf5 files opening allowed? Observations: This problem can be reproduced, still the same given number of opening/closing allowed. This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. The hdf5 file created and used before the crash is still readable and all data written in it before are ok. Setup: This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". Here is a part of the standard error, produced when the code reached the 1010th opening: HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0:   #000: H5F.c line 1495 in H5Fopen(): unable to open file     major: File accessability     minor: Unable to open file   #001: H5F.c line 1195 in H5F\_open(): unable to open file     major: File accessability     minor: Unable to open file   #002: H5FD.c line 1088 in H5FD\_open(): open failed     major: Virtual File Layer     minor: Unable to initialize object   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed     major: Internal error (too specific to document in detail)     minor: Some MPI function failed   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files     major: Internal error (too specific to document in detail)     minor: MPI Error String HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0:   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier     major: Invalid arguments to routine     minor: Inappropriate type HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1:   #000: H5F.c line 1495 in H5Fopen(): unable to open file     major: File accessability     minor: Unable to open file   #001: H5F.c line 1195 in H5F\_open(): unable to open file     major: File accessability     minor: Unable to open file   #002: H5FD.c line 1088 in H5FD\_open(): open failed     major: Virtual File Layer     minor: Unable to initialize object HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) HDF5-DIAG: Error detected in HDF5 (1.8.5) : HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: :   #000: H5F.c line 1495 in H5Fopen(): unable to open file     minor: Some MPI function failed   #000: H5F.c line 1495 in H5Fopen(): unable to open file   #000: H5F.c line 1495 in H5Fopen(): unable to open file     major: File accessability     major: File accessability I also attach the whole standard error. I will appreciate any help! Do not hesitate to ask me some additional details if needed! Thanks, Best regards, Stephane [main.F](https://hdf-discourse-1.s3.amazonaws.com/original/1X/a676c007aa4c13289f55011532e3e4eb42549d53.F) (1.99 KB) [test-hdf5.out.bz2](https://hdf-discourse-1.s3.amazonaws.com/original/1X/1f1f22186deaaebc3ca753c367db2d06ef989944.bz2) (151 KB) ---
**Author:** ![Mohamad\_Chaarawi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@Mohamad\_Chaarawi](https://forum.hdfgroup.org/u/Mohamad_Chaarawi)\ **Post date:** [February 6, 2012, 3:25pm UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/2 "2012-02-06T15:25:10Z")
Hi Stephane, I could not replicate the problem you are seeing. I ran the sample program that you sent me on an XE6 and it worked fine.. I am not aware of any limit on the number of times you can open/close a file (at least from the HDF5 side of things). Not sure if the filesystem you are working on imposes such a limitation, but i don't see why it would.. Thanks, Mohamad > **···** > > On 02/05/2012 04:20 PM, St�phane Backaert wrote: > > > Dear all, > > > > I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. > > > > Problem: > > With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two:\* is there a limit of the number of hdf5 files opening allowed?\* > > > > Observations: > > This problem can be reproduced, still the same given number of opening/closing allowed. > > This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). > > We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. > > The hdf5 file created and used before the crash is still readable and all data written in it before are ok. > > > > Setup: > > This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). > > > > A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". > > > > Here is a part of the standard error, produced when the code reached the 1010th opening: > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > >     major: Virtual File Layer > >     minor: Unable to initialize object > >   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > >     major: Internal error (too specific to document in detail) > >     minor: Some MPI function failed > >   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: > > ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files > >     major: Internal error (too specific to document in detail) > >     minor: MPI Error String > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > >   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier > >     major: Invalid arguments to routine > >     minor: Inappropriate type > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1: > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > >     major: Virtual File Layer > >     minor: Unable to initialize object > > HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) > > HDF5-DIAG: Error detected in HDF5 (1.8.5) : > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: > > : > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     minor: Some MPI function failed > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     major: File accessability > > > > I also attach the whole standard error. > > > > I will appreciate any help! Do not hesitate to ask me some additional details if needed! > > > > Thanks, > > > > Best regards, > > > > Stephane > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) ---
**Author:** ![John\_Biddiscombe](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/ee7513/32.png) [@John\_Biddiscombe](https://forum.hdfgroup.org/u/John_Biddiscombe)\ **Post date:** [February 7, 2012, 9:39am UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/3 "2012-02-07T09:39:47Z")
Sephane, Make sure you point out that this only occurs when using a lustre filesystem. JB > **···** > > From: [hdf-forum-bounces@hdfgroup.org](mailto:hdf-forum-bounces@hdfgroup.org) [[mailto:hdf-forum-bounces@hdfgroup.org](mailto:hdf-forum-bounces@hdfgroup.org)] On Behalf Of Stéphane Backaert > Sent: 05 February 2012 23:21 > To: [hdf-forum@hdfgroup.org](mailto:hdf-forum@hdfgroup.org) > Subject: [Hdf-forum] can not open/close a file more than 1010 times > > Dear all, > > I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. > > Problem: > With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two: is there a limit of the number of hdf5 files opening allowed? > > Observations: > This problem can be reproduced, still the same given number of opening/closing allowed. > This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). > We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. > The hdf5 file created and used before the crash is still readable and all data written in it before are ok. > > Setup: > This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). > > A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". > > Here is a part of the standard error, produced when the code reached the 1010th opening: > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: >   #000: H5F.c line 1495 in H5Fopen(): unable to open file >     major: File accessability >     minor: Unable to open file >   #001: H5F.c line 1195 in H5F\_open(): unable to open file >     major: File accessability >     minor: Unable to open file >   #002: H5FD.c line 1088 in H5FD\_open(): open failed >     major: Virtual File Layer >     minor: Unable to initialize object >   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed >     major: Internal error (too specific to document in detail) >     minor: Some MPI function failed >   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: > ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files >     major: Internal error (too specific to document in detail) >     minor: MPI Error String > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: >   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier >     major: Invalid arguments to routine >     minor: Inappropriate type > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1: >   #000: H5F.c line 1495 in H5Fopen(): unable to open file >     major: File accessability >     minor: Unable to open file >   #001: H5F.c line 1195 in H5F\_open(): unable to open file >     major: File accessability >     minor: Unable to open file >   #002: H5FD.c line 1088 in H5FD\_open(): open failed >     major: Virtual File Layer >     minor: Unable to initialize object > HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) > HDF5-DIAG: Error detected in HDF5 (1.8.5) : > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: > : >   #000: H5F.c line 1495 in H5Fopen(): unable to open file >     minor: Some MPI function failed >   #000: H5F.c line 1495 in H5Fopen(): unable to open file >   #000: H5F.c line 1495 in H5Fopen(): unable to open file >     major: File accessability >     major: File accessability > > I also attach the whole standard error. > > I will appreciate any help! Do not hesitate to ask me some additional details if needed! > > Thanks, > > Best regards, > > Stephane ---
**Author:** ![John\_Shalf](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/edb3f5/32.png) [@John\_Shalf](https://forum.hdfgroup.org/u/John_Shalf)\ **Post date:** [February 7, 2012, 11:51am UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/4 "2012-02-07T11:51:25Z")
Which XE6 is this? (which OS rev?) -john > **···** > > On Feb 7, 2012, at 1:39 AM, Biddiscombe, John A. wrote: > > > Sephane, > > > > Make sure you point out that this only occurs when using a lustre filesystem. > > > > JB > > > > From: hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] On Behalf Of Stéphane Backaert > > Sent: 05 February 2012 23:21 > > To: hdf-forum@hdfgroup.org > > Subject: [Hdf-forum] can not open/close a file more than 1010 times > > > > Dear all, > > > > I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. > > > > Problem: > > With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two: is there a limit of the number of hdf5 files opening allowed? > > > > Observations: > > This problem can be reproduced, still the same given number of opening/closing allowed. > > This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). > > We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. > > The hdf5 file created and used before the crash is still readable and all data written in it before are ok. > > > > Setup: > > This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). > > > > A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". > > > > Here is a part of the standard error, produced when the code reached the 1010th opening: > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > >     major: Virtual File Layer > >     minor: Unable to initialize object > >   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > >     major: Internal error (too specific to document in detail) > >     minor: Some MPI function failed > >   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: > > ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files > >     major: Internal error (too specific to document in detail) > >     minor: MPI Error String > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > >   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier > >     major: Invalid arguments to routine > >     minor: Inappropriate type > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1: > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > >     major: Virtual File Layer > >     minor: Unable to initialize object > > HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) > > HDF5-DIAG: Error detected in HDF5 (1.8.5) : > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: > > : > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     minor: Some MPI function failed > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > >     major: File accessability > >     major: File accessability > > > > I also attach the whole standard error. > > > > I will appreciate any help! Do not hesitate to ask me some additional details if needed! > > > > Thanks, > > > > Best regards, > > > > Stephane > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) ---
**Author:** ![Jerome\_Soumagne](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/6bbea6/32.png) [@Jerome\_Soumagne](https://forum.hdfgroup.org/u/Jerome_Soumagne)\ **Post date:** [February 7, 2012, 11:22am UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/5 "2012-02-07T11:22:39Z")
Stephane, I just tried on an XK6 system using gnu compilers and I do get the same issue. I tried with 1.8.8 (compiled myself) and 1.8.7 (from cray). It seems that it only appears when I write to the lustre file system... and it does not seem to occur otherwise.          1002          1003          1004          1005 HDF5-DIAG: Error detected in HDF5 (1.8.8) MPI-process 0:    #000: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1522 in H5Fopen(): unable to open file      major: File accessability      minor: Unable to open file    #001: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1211 in H5F\_open(): unable to open file: time = Tue Feb 7 10:42:17 2012 , name = 'test.h5', tent\_flags = 1      major: File accessability      minor: Unable to open file regards, Jerome > **···** > > On 02/07/2012 10:39 AM, Biddiscombe, John A. wrote: > > > Sephane, > > > > Make sure you point out that this only occurs when using a lustre filesystem. > > > > JB > > > > \*From:\*hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] \*On Behalf Of \*St�phane Backaert > > \*Sent:\* 05 February 2012 23:21 > > \*To:\* hdf-forum@hdfgroup.org > > \*Subject:\* [Hdf-forum] can not open/close a file more than 1010 times > > > > Dear all, > > > > I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. > > > > Problem: > > > > With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two:\* is there a limit of the number of hdf5 files opening allowed?\* > > > > Observations: > > > > This problem can be reproduced, still the same given number of opening/closing allowed. > > > > This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). > > > > We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. > > > > The hdf5 file created and used before the crash is still readable and all data written in it before are ok. > > > > Setup: > > > > This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). > > > > A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". > > > > Here is a part of the standard error, produced when the code reached the 1010th opening: > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > >     major: File accessability > > > >     minor: Unable to open file > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > >     major: File accessability > > > >     minor: Unable to open file > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > >     major: Virtual File Layer > > > >     minor: Unable to initialize object > > > >   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > >     major: Internal error (too specific to document in detail) > > > >     minor: Some MPI function failed > > > >   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: > > > > ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files > > > >     major: Internal error (too specific to document in detail) > > > >     minor: MPI Error String > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > >   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier > > > >     major: Invalid arguments to routine > > > >     minor: Inappropriate type > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1: > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > >     major: File accessability > > > >     minor: Unable to open file > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > >     major: File accessability > > > >     minor: Unable to open file > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > >     major: Virtual File Layer > > > >     minor: Unable to initialize object > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > > MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) : > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: > > > > : > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > >     minor: Some MPI function failed > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > >     major: File accessability > > > >     major: File accessability > > > > I also attach the whole standard error. > > > > I will appreciate any help! Do not hesitate to ask me some additional details if needed! > > > > Thanks, > > > > Best regards, > > > > Stephane > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) ---
**Author:** ![Mohamad\_Chaarawi](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/m/e47c2d/32.png) [@Mohamad\_Chaarawi](https://forum.hdfgroup.org/u/Mohamad_Chaarawi)\ **Post date:** [February 7, 2012, 2:38pm UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/6 "2012-02-07T14:38:15Z")
I tried this on hopper on /scratch2 which is Lustre also, but it works fine for me.. I used the Cray provided HDF5 (1.8.7) Since I could not replicate the problem, did you try opening/closing the file using plain MPI-I/O:      MPI\_File\_open()      MPI\_File\_close() If you get the same problem there, I would go one step down further using plain posix I/O, just to try and get an idea at what layer you see that issue. Thanks, Mohamad > **···** > > On 02/07/2012 05:22 AM, Jerome Soumagne wrote: > > > Stephane, > > > > I just tried on an XK6 system using gnu compilers and I do get the same issue. I tried with 1.8.8 (compiled myself) and 1.8.7 (from cray). It seems that it only appears when I write to the lustre file system... and it does not seem to occur otherwise. > > > >         1002 > >         1003 > >         1004 > >         1005 > > HDF5-DIAG: Error detected in HDF5 (1.8.8) MPI-process 0: > >   #000: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1522 in H5Fopen(): unable to open file > >     major: File accessability > >     minor: Unable to open file > >   #001: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1211 in H5F\_open(): unable to open file: time = Tue Feb 7 10:42:17 2012 > > , name = 'test.h5', tent\_flags = 1 > >     major: File accessability > >     minor: Unable to open file > > > > regards, > > > > Jerome > > > > On 02/07/2012 10:39 AM, Biddiscombe, John A. wrote: > > > > > Sephane, > > > > > > Make sure you point out that this only occurs when using a lustre filesystem. > > > > > > JB > > > > > > \*From:\*hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] \*On Behalf Of \*St�phane Backaert > > > \*Sent:\* 05 February 2012 23:21 > > > \*To:\* hdf-forum@hdfgroup.org > > > \*Subject:\* [Hdf-forum] can not open/close a file more than 1010 times > > > > > > Dear all, > > > > > > I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. > > > > > > Problem: > > > > > > With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two:\* is there a limit of the number of hdf5 files opening allowed?\* > > > > > > Observations: > > > > > > This problem can be reproduced, still the same given number of opening/closing allowed. > > > > > > This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). > > > > > > We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. > > > > > > The hdf5 file created and used before the crash is still readable and all data written in it before are ok. > > > > > > Setup: > > > > > > This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). > > > > > > A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". > > > > > > Here is a part of the standard error, produced when the code reached the 1010th opening: > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > >     major: File accessability > > > > > >     minor: Unable to open file > > > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > > > >     major: File accessability > > > > > >     minor: Unable to open file > > > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > > > >     major: Virtual File Layer > > > > > >     minor: Unable to initialize object > > > > > >   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > > > >     major: Internal error (too specific to document in detail) > > > > > >     minor: Some MPI function failed > > > > > >   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: > > > > > > ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files > > > > > >     major: Internal error (too specific to document in detail) > > > > > >     minor: MPI Error String > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > > > >   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier > > > > > >     major: Invalid arguments to routine > > > > > >     minor: Inappropriate type > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1: > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > >     major: File accessability > > > > > >     minor: Unable to open file > > > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > > > >     major: File accessability > > > > > >     minor: Unable to open file > > > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > > > >     major: Virtual File Layer > > > > > >     minor: Unable to initialize object > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > > > > MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) : > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: > > > > > > : > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > >     minor: Some MPI function failed > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > >     major: File accessability > > > > > >     major: File accessability > > > > > > I also attach the whole standard error. > > > > > > I will appreciate any help! Do not hesitate to ask me some additional details if needed! > > > > > > Thanks, > > > > > > Best regards, > > > > > > Stephane > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Hdf-forum is for HDF software users discussion. > > > Hdf-forum@hdfgroup.org > > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) ---
**Author:** ![Jerome\_Soumagne](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/6bbea6/32.png) [@Jerome\_Soumagne](https://forum.hdfgroup.org/u/Jerome_Soumagne)\ **Post date:** [February 7, 2012, 3:16pm UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/7 "2012-02-07T15:16:31Z")
I've just tried with an MPI\_file\_open/close and I get a "Failure to get stripe info: Bad file descriptor" message after about 1012 iterations, so that's definitely not a problem with HDF5. Jerome > **···** > > On 02/07/2012 03:38 PM, Mohamad Chaarawi wrote: > > > I tried this on hopper on /scratch2 which is Lustre also, but it works fine for me.. > > I used the Cray provided HDF5 (1.8.7) > > > > Since I could not replicate the problem, did you try opening/closing the file using plain MPI-I/O: > >     MPI\_File\_open() > >     MPI\_File\_close() > > > > If you get the same problem there, I would go one step down further using plain posix I/O, just to try and get an idea at what layer you see that issue. > > > > Thanks, > > Mohamad > > > > On 02/07/2012 05:22 AM, Jerome Soumagne wrote: > > > > > Stephane, > > > > > > I just tried on an XK6 system using gnu compilers and I do get the same issue. I tried with 1.8.8 (compiled myself) and 1.8.7 (from cray). It seems that it only appears when I write to the lustre file system... and it does not seem to occur otherwise. > > > > > >         1002 > > >         1003 > > >         1004 > > >         1005 > > > HDF5-DIAG: Error detected in HDF5 (1.8.8) MPI-process 0: > > >   #000: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1522 in H5Fopen(): unable to open file > > >     major: File accessability > > >     minor: Unable to open file > > >   #001: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1211 in H5F\_open(): unable to open file: time = Tue Feb 7 10:42:17 2012 > > > , name = 'test.h5', tent\_flags = 1 > > >     major: File accessability > > >     minor: Unable to open file > > > > > > regards, > > > > > > Jerome > > > > > > On 02/07/2012 10:39 AM, Biddiscombe, John A. wrote: > > > > > > > Sephane, > > > > > > > > Make sure you point out that this only occurs when using a lustre filesystem. > > > > > > > > JB > > > > > > > > \*From:\*hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] \*On Behalf Of \*St�phane Backaert > > > > \*Sent:\* 05 February 2012 23:21 > > > > \*To:\* hdf-forum@hdfgroup.org > > > > \*Subject:\* [Hdf-forum] can not open/close a file more than 1010 times > > > > > > > > Dear all, > > > > > > > > I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. > > > > > > > > Problem: > > > > > > > > With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two:\* is there a limit of the number of hdf5 files opening allowed?\* > > > > > > > > Observations: > > > > > > > > This problem can be reproduced, still the same given number of opening/closing allowed. > > > > > > > > This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). > > > > > > > > We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. > > > > > > > > The hdf5 file created and used before the crash is still readable and all data written in it before are ok. > > > > > > > > Setup: > > > > > > > > This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). > > > > > > > > A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". > > > > > > > > Here is a part of the standard error, produced when the code reached the 1010th opening: > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > >     major: File accessability > > > > > > > >     minor: Unable to open file > > > > > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > > > > > >     major: File accessability > > > > > > > >     minor: Unable to open file > > > > > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > > > > > >     major: Virtual File Layer > > > > > > > >     minor: Unable to initialize object > > > > > > > >   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > > > > > >     major: Internal error (too specific to document in detail) > > > > > > > >     minor: Some MPI function failed > > > > > > > >   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: > > > > > > > > ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files > > > > > > > >     major: Internal error (too specific to document in detail) > > > > > > > >     minor: MPI Error String > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > > > > > >   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier > > > > > > > >     major: Invalid arguments to routine > > > > > > > >     minor: Inappropriate type > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1: > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > >     major: File accessability > > > > > > > >     minor: Unable to open file > > > > > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > > > > > >     major: File accessability > > > > > > > >     minor: Unable to open file > > > > > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > > > > > >     major: Virtual File Layer > > > > > > > >     minor: Unable to initialize object > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > > > > > > MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) : > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: > > > > > > > > : > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > >     minor: Some MPI function failed > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > >     major: File accessability > > > > > > > >     major: File accessability > > > > > > > > I also attach the whole standard error. > > > > > > > > I will appreciate any help! Do not hesitate to ask me some additional details if needed! > > > > > > > > Thanks, > > > > > > > > Best regards, > > > > > > > > Stephane > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > Hdf-forum is for HDF software users discussion. > > > > Hdf-forum@hdfgroup.org > > > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Hdf-forum is for HDF software users discussion. > > > Hdf-forum@hdfgroup.org > > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) ---
**Author:** ![Jerome\_Soumagne](https://forum.hdfgroup.org/letter_avatar_proxy/v4/letter/j/6bbea6/32.png) [@Jerome\_Soumagne](https://forum.hdfgroup.org/u/Jerome_Soumagne)\ **Post date:** [February 8, 2012, 8:43am UTC](https://forum.hdfgroup.org/t/can-not-open-close-a-file-more-than-1010-times/2458/8 "2012-02-08T08:43:29Z")
Just for information, this issue seems to have been introduced with Cray MPT versions \>= 5.4.1. It has been reported to Cray. Jerome > **···** > > On 02/07/2012 04:16 PM, Jerome Soumagne wrote: > > > I've just tried with an MPI\_file\_open/close and I get a "Failure to get stripe info: Bad file descriptor" message after about 1012 iterations, so that's definitely not a problem with HDF5. > > > > Jerome > > > > On 02/07/2012 03:38 PM, Mohamad Chaarawi wrote: > > > > > I tried this on hopper on /scratch2 which is Lustre also, but it works fine for me.. > > > I used the Cray provided HDF5 (1.8.7) > > > > > > Since I could not replicate the problem, did you try opening/closing the file using plain MPI-I/O: > > >     MPI\_File\_open() > > >     MPI\_File\_close() > > > > > > If you get the same problem there, I would go one step down further using plain posix I/O, just to try and get an idea at what layer you see that issue. > > > > > > Thanks, > > > Mohamad > > > > > > On 02/07/2012 05:22 AM, Jerome Soumagne wrote: > > > > > > > Stephane, > > > > > > > > I just tried on an XK6 system using gnu compilers and I do get the same issue. I tried with 1.8.8 (compiled myself) and 1.8.7 (from cray). It seems that it only appears when I write to the lustre file system... and it does not seem to occur otherwise. > > > > > > > >         1002 > > > >         1003 > > > >         1004 > > > >         1005 > > > > HDF5-DIAG: Error detected in HDF5 (1.8.8) MPI-process 0: > > > >   #000: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1522 in H5Fopen(): unable to open file > > > >     major: File accessability > > > >     minor: Unable to open file > > > >   #001: /project/csvis/soumagne/apps/src/rosa/hdf5-vfd-1.8.8/source/src/H5F.c line 1211 in H5F\_open(): unable to open file: time = Tue Feb 7 10:42:17 2012 > > > > , name = 'test.h5', tent\_flags = 1 > > > >     major: File accessability > > > >     minor: Unable to open file > > > > > > > > regards, > > > > > > > > Jerome > > > > > > > > On 02/07/2012 10:39 AM, Biddiscombe, John A. wrote: > > > > > > > > > Sephane, > > > > > > > > > > Make sure you point out that this only occurs when using a lustre filesystem. > > > > > > > > > > JB > > > > > > > > > > \*From:\*hdf-forum-bounces@hdfgroup.org [mailto:hdf-forum-bounces@hdfgroup.org] \*On Behalf Of \*St�phane Backaert > > > > > \*Sent:\* 05 February 2012 23:21 > > > > > \*To:\* hdf-forum@hdfgroup.org > > > > > \*Subject:\* [Hdf-forum] can not open/close a file more than 1010 times > > > > > > > > > > Dear all, > > > > > > > > > > I am using the HDF5-parallel API on a Cray XE6. This had been working fine until a few weeks. This is probably something related to this machine (since a maintenance?) so I sent my issue to the helpdesk of this Cray, but I submit my problem to you as well. I have this problem only with this machine, no problem with another machine (not Cray) used. > > > > > > > > > > Problem: > > > > > > > > > > With parallel version of HDF5, we can not open-and-close the same hdf5 file more than a fixed number of times (1010). Moreover, if we try to open-and-close, for example, two different hdf5 files, this maximum number is divided by two:\* is there a limit of the number of hdf5 files opening allowed?\* > > > > > > > > > > Observations: > > > > > > > > > > This problem can be reproduced, still the same given number of opening/closing allowed. > > > > > > > > > > This problem does not depend on the number of mpi procs involved (test up to 512 cores, still 1010 opening before crash), or on the quantity written, or on the actions made with an opened file ( like create dataset, attributes or group). > > > > > > > > > > We checked the status of each hdf5 operations (hdferror argument): no one complains before the crash. So every file seems to be correctly opened/closed. No different behavior if we force the file opening property with H5F\_CLOSE\_STRONG\_F. > > > > > > > > > > The hdf5 file created and used before the crash is still readable and all data written in it before are ok. > > > > > > > > > > Setup: > > > > > > > > > > This problem occurs with the hdf5-parallel version 1.5.8.0 and 1.8.8, both compiled with the intel compiler 12.0.3.174, only on a Cray XE6 (I could send the module config used). > > > > > > > > > > A basic code reproduces our hdf5 calls structure. The attached code produced the described error. It is compiled with the command "h5pfc -FR main.F -o test". > > > > > > > > > > Here is a part of the standard error, produced when the code reached the 1010th opening: > > > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > > > >     major: File accessability > > > > > > > > > >     minor: Unable to open file > > > > > > > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > > > > > > > >     major: File accessability > > > > > > > > > >     minor: Unable to open file > > > > > > > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > > > > > > > >     major: Virtual File Layer > > > > > > > > > >     minor: Unable to initialize object > > > > > > > > > >   #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > > > > > > > >     major: Internal error (too specific to document in detail) > > > > > > > > > >     minor: Some MPI function failed > > > > > > > > > >   #004: H5FDmpio.c line 999 in H5FD\_mpio\_open(): Other I/O error , error stack: > > > > > > > > > > ADIOI\_UFS\_OPEN(108): Other I/O error Too many open files > > > > > > > > > >     major: Internal error (too specific to document in detail) > > > > > > > > > >     minor: MPI Error String > > > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 0: > > > > > > > > > >   #000: H5F.c line 1943 in H5Fclose(): invalid file identifier > > > > > > > > > >     major: Invalid arguments to routine > > > > > > > > > >     minor: Inappropriate type > > > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 1: > > > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > > > >     major: File accessability > > > > > > > > > >     minor: Unable to open file > > > > > > > > > >   #001: H5F.c line 1195 in H5F\_open(): unable to open file > > > > > > > > > >     major: File accessability > > > > > > > > > >     minor: Unable to open file > > > > > > > > > >   #002: H5FD.c line 1088 in H5FD\_open(): open failed > > > > > > > > > >     major: Virtual File Layer > > > > > > > > > >     minor: Unable to initialize object > > > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) HDF5-DIAG: Error detected in HDF5 (1.8.5) #003: H5FDmpio.c line 999 in H5FD\_mpio\_open(): MPI\_File\_open failed > > > > > > > > > > MPI-process 3MPI-process 6 major: Internal error (too specific to document in detail) > > > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) : > > > > > > > > > > HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 7HDF5-DIAG: Error detected in HDF5 (1.8.5) MPI-process 2: > > > > > > > > > > : > > > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > > > >     minor: Some MPI function failed > > > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > > > >   #000: H5F.c line 1495 in H5Fopen(): unable to open file > > > > > > > > > >     major: File accessability > > > > > > > > > >     major: File accessability > > > > > > > > > > I also attach the whole standard error. > > > > > > > > > > I will appreciate any help! Do not hesitate to ask me some additional details if needed! > > > > > > > > > > Thanks, > > > > > > > > > > Best regards, > > > > > > > > > > Stephane > > > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > > Hdf-forum is for HDF software users discussion. > > > > > Hdf-forum@hdfgroup.org > > > > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > > Hdf-forum is for HDF software users discussion. > > > > Hdf-forum@hdfgroup.org > > > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > > Hdf-forum is for HDF software users discussion. > > > Hdf-forum@hdfgroup.org > > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org) > > > > \_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_\_ > > Hdf-forum is for HDF software users discussion. > > Hdf-forum@hdfgroup.org > > [http://mail.hdfgroup.org/mailman/listinfo/hdf-forum\_hdfgroup.org](http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org)