memcpy overlap during H5Awrite

Hi all,
I've used valgrind to check my hdf5 use and it returns this:

==16072== Source and destination overlap in memcpy(0x4FC9CC0, 0x4FC9CC0, 33)
==16072== at 0x4906884: memcpy (mac_replace_strmem.c:394)
==16072== by 0x4C6A27A: H5O_attr_write_cb (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4C7FEEC: H5O_msg_iterate_real (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4C6A620: H5O_attr_write (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4B84F64: H5Awrite (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4A13C1E: HDF_Add_Attribute_As_String (l3.c:761)
==16072== by 0x4A1434E: HDF_Create_Root (l3.c:876)
==16072== by 0x4A16C90: L3_openFile (l3.c:1359)
==16072== by 0x401882: main (test02.c:19)

even when I set the --enable-using-memchecker.
Then I should have a bug in my code...
I cannot find it and when I run my test suite I has correct results (which is far to be a proof...)
Here's the piece of code, I would appreciate any help, hint or whatever (except putting the
valgrind message in the suppression list !).
I've suspected a misused of SCALAR data space and strings, I've seen examples with SIMPLE dataspaces
instead of SCALAR for strings but the use of a fixed length string should be possible ?

notes:
  * nodeid is a group
  * L3C_MAX_ATTRIB_SIZE > L3C_MAX_DTYPE
  * using hdf 1.8.5 thread-safe
  * using valgrind 3.5
  * x86 platform with gcc

int HDF_Add_Attribute_As_String(L3_Cursor_t *ctxt, hid_t nodeid, const char *name, const char *value)
{
   hid_t sid,tid,aid;
   herr_t status;
   hsize_t dim;
   char buff[L3C_MAX_ATTRIB_SIZE+1];

   if (!strcmp(name,L3S_DTYPE))
   {
     dim=(hsize_t)(L3C_MAX_DTYPE+1);
   }
   else
   {
     dim=(hsize_t)(L3C_MAX_ATTRIB_SIZE+1);
   }
   sid = H5Screate(H5S_SCALAR);
   if (sid < 0)
   {
     return 0;
   }
   tid = H5Tcopy(H5T_C_S1);
   if (tid < 0)
   {
     H5Sclose(sid);
     return 0;
   }
   if (H5Tset_size(tid,dim)<0)
   {
     H5Tclose(tid);
     H5Sclose(sid);
     return 0;
   }
   aid = H5Acreate(nodeid,name,tid,sid,H5P_DEFAULT,H5P_DEFAULT);
   if (aid < 0)
   {
     H5Tclose(tid);
     H5Sclose(sid);
     return 0;
   }
   memset(buff, 0, dim);
   strcpy(buff, value);
   status = H5Awrite(aid, tid, buff);

   H5Aclose(aid);
   H5Tclose(tid);
   H5Sclose(sid);

   if (status < 0)
   {
     return 0;
   }
   return 1;
}

-MP-

···

-----------------------------------------------------------------------
  Marc POINOT [ONERA/DSNA] Tel:+33.1.46.73.42.84 Fax:+33.1.46.73.41.66
  Avertissement/disclaimer http://www.onera.fr/onera-en/emails-terms

Hi Marc,

Hi all,
I've used valgrind to check my hdf5 use and it returns this:

==16072== Source and destination overlap in memcpy(0x4FC9CC0, 0x4FC9CC0, 33)
==16072== at 0x4906884: memcpy (mac_replace_strmem.c:394)
==16072== by 0x4C6A27A: H5O_attr_write_cb (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4C7FEEC: H5O_msg_iterate_real (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4C6A620: H5O_attr_write (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4B84F64: H5Awrite (in /home/poinot/Tools-2/lib/libhdf5.so.6.0.4)
==16072== by 0x4A13C1E: HDF_Add_Attribute_As_String (l3.c:761)
==16072== by 0x4A1434E: HDF_Create_Root (l3.c:876)
==16072== by 0x4A16C90: L3_openFile (l3.c:1359)
==16072== by 0x401882: main (test02.c:19)

even when I set the --enable-using-memchecker.
Then I should have a bug in my code...
I cannot find it and when I run my test suite I has correct results (which is far to be a proof...)
Here's the piece of code, I would appreciate any help, hint or whatever (except putting the
valgrind message in the suppression list !).
I've suspected a misused of SCALAR data space and strings, I've seen examples with SIMPLE dataspaces
instead of SCALAR for strings but the use of a fixed length string should be possible ?

  Your code looks reasonable. We've been working on valgrind issues recently - can you check out the latest code from:

  http://svn.hdfgroup.uiuc.edu/hdf5/branches/hdf5_1_8/

  See if these recent changes help.

    Quincey

···

On Aug 24, 2010, at 2:09 AM, Marc POINOT wrote:

notes:
* nodeid is a group
* L3C_MAX_ATTRIB_SIZE > L3C_MAX_DTYPE
* using hdf 1.8.5 thread-safe
* using valgrind 3.5
* x86 platform with gcc

int HDF_Add_Attribute_As_String(L3_Cursor_t *ctxt, hid_t nodeid, const char *name, const char *value)
{
hid_t sid,tid,aid;
herr_t status;
hsize_t dim;
char buff[L3C_MAX_ATTRIB_SIZE+1];

if (!strcmp(name,L3S_DTYPE))
{
   dim=(hsize_t)(L3C_MAX_DTYPE+1);
}
else
{
   dim=(hsize_t)(L3C_MAX_ATTRIB_SIZE+1);
}
sid = H5Screate(H5S_SCALAR);
if (sid < 0)
{
   return 0;
}
tid = H5Tcopy(H5T_C_S1);
if (tid < 0)
{
   H5Sclose(sid);
   return 0;
}
if (H5Tset_size(tid,dim)<0)
{
   H5Tclose(tid);
   H5Sclose(sid);
   return 0;
}
aid = H5Acreate(nodeid,name,tid,sid,H5P_DEFAULT,H5P_DEFAULT);
if (aid < 0)
{
   H5Tclose(tid);
   H5Sclose(sid);
   return 0;
}
memset(buff, 0, dim);
strcpy(buff, value);
status = H5Awrite(aid, tid, buff);

H5Aclose(aid);
H5Tclose(tid);
H5Sclose(sid);

if (status < 0)
{
   return 0;
}
return 1;
}

-MP-
-----------------------------------------------------------------------
Marc POINOT [ONERA/DSNA] Tel:+33.1.46.73.42.84 Fax:+33.1.46.73.41.66
Avertissement/disclaimer http://www.onera.fr/onera-en/emails-terms

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

Quincey Koziol wrote:

  Your code looks reasonable. We've been working on valgrind issues recently - can you check out the latest code from:

  http://svn.hdfgroup.uiuc.edu/hdf5/branches/hdf5_1_8/

  See if these recent changes help.

Great, this removes all Valgrind weird messages I had on some hdf5 calls.
There's still one I'm investigating now (see below), only one test in my test suite
raises this message, I'm trying to find out what's different in this test.

==24685== Syscall param write(buf) points to uninitialised byte(s)
==24685== at 0x3E5070B012: __write_nocancel (in /lib64/tls/libpthread-2.3.4.so)
==24685== by 0x4C589C4: H5FD_sec2_write (H5FDsec2.c:839)
==24685== by 0x4C4E603: H5FD_write (H5FDint.c:184)
==24685== by 0x4C32FC4: H5F_accum_write (H5Faccum.c:580)
==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
==24685== by 0x4BC9F84: H5C_flush_cache (H5C.c:1801)
==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)
==24685== Address 0x534713C is 6,852 bytes inside a block of size 8,192 alloc'd
==24685== at 0x4905E12: realloc (vg_replace_malloc.c:306)
==24685== by 0x4CFCA97: H5MM_realloc (H5MM.c:140)
==24685== by 0x4C320C2: H5F_accum_adjust (H5Faccum.c:335)
==24685== by 0x4C32558: H5F_accum_write (H5Faccum.c:416)
==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
==24685== by 0x4BCA036: H5C_flush_cache (H5C.c:1824)
==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)

Thanks Quincey

-MP-

···

-----------------------------------------------------------------------
  Marc POINOT [ONERA/DSNA] Tel:+33.1.46.73.42.84 Fax:+33.1.46.73.41.66
  Avertissement/disclaimer http://www.onera.fr/onera-en/emails-terms

Hi marc,

Quincey Koziol wrote:

  Your code looks reasonable. We've been working on valgrind issues recently - can you check out the latest code from:
  http://svn.hdfgroup.uiuc.edu/hdf5/branches/hdf5_1_8/
  See if these recent changes help.

Great, this removes all Valgrind weird messages I had on some hdf5 calls.

  Good! :slight_smile:

There's still one I'm investigating now (see below), only one test in my test suite
raises this message, I'm trying to find out what's different in this test.

==24685== Syscall param write(buf) points to uninitialised byte(s)
==24685== at 0x3E5070B012: __write_nocancel (in /lib64/tls/libpthread-2.3.4.so)
==24685== by 0x4C589C4: H5FD_sec2_write (H5FDsec2.c:839)
==24685== by 0x4C4E603: H5FD_write (H5FDint.c:184)
==24685== by 0x4C32FC4: H5F_accum_write (H5Faccum.c:580)
==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
==24685== by 0x4BC9F84: H5C_flush_cache (H5C.c:1801)
==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)
==24685== Address 0x534713C is 6,852 bytes inside a block of size 8,192 alloc'd
==24685== at 0x4905E12: realloc (vg_replace_malloc.c:306)
==24685== by 0x4CFCA97: H5MM_realloc (H5MM.c:140)
==24685== by 0x4C320C2: H5F_accum_adjust (H5Faccum.c:335)
==24685== by 0x4C32558: H5F_accum_write (H5Faccum.c:416)
==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
==24685== by 0x4BCA036: H5C_flush_cache (H5C.c:1824)
==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)

  Well, I didn't say we were completely done yet. :wink: If you can get this warning (which is most likely harmless) down to a simple program, I can see about fixing it.

  Quincey

···

On Aug 24, 2010, at 7:32 AM, Marc POINOT wrote:

Quincey Koziol wrote:

  Well, I didn't say we were completely done yet. :wink: If you can get this warning (which is most likely harmless) down to a simple program, I can see about fixing it.

fixed: bug on my side, bad memory write.

I'll continue my tests on my HDF5 application with this HDF5 dev version, I'll let you know.

-MP-

···

-----------------------------------------------------------------------
  Marc POINOT [ONERA/DSNA] Tel:+33.1.46.73.42.84 Fax:+33.1.46.73.41.66
  Avertissement/disclaimer http://www.onera.fr/onera-en/emails-terms

FYI, I've been seeing same memcpy src/dst overlap problem on our BG/P
system. It seems to be happening in one of the first attributes ever
written to the file, a tiny single 4 byte integer value'd attribute. The
pointer for both src and dst is the same value.

I cannot find a way to re-produce on something 'small'.

Mark

···

On Tue, 2010-08-24 at 06:13, Quincey Koziol wrote:

Hi marc,

On Aug 24, 2010, at 7:32 AM, Marc POINOT wrote:

> Quincey Koziol wrote:
>> Your code looks reasonable. We've been working on valgrind issues recently - can you check out the latest code from:
>> http://*svn.hdfgroup.uiuc.edu/hdf5/branches/hdf5_1_8/
>> See if these recent changes help.
>
> Great, this removes all Valgrind weird messages I had on some hdf5 calls.

  Good! :slight_smile:

> There's still one I'm investigating now (see below), only one test in my test suite
> raises this message, I'm trying to find out what's different in this test.
>
> ==24685== Syscall param write(buf) points to uninitialised byte(s)
> ==24685== at 0x3E5070B012: __write_nocancel (in /lib64/tls/libpthread-2.3.4.so)
> ==24685== by 0x4C589C4: H5FD_sec2_write (H5FDsec2.c:839)
> ==24685== by 0x4C4E603: H5FD_write (H5FDint.c:184)
> ==24685== by 0x4C32FC4: H5F_accum_write (H5Faccum.c:580)
> ==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
> ==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
> ==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
> ==24685== by 0x4BC9F84: H5C_flush_cache (H5C.c:1801)
> ==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
> ==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
> ==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
> ==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)
> ==24685== Address 0x534713C is 6,852 bytes inside a block of size 8,192 alloc'd
> ==24685== at 0x4905E12: realloc (vg_replace_malloc.c:306)
> ==24685== by 0x4CFCA97: H5MM_realloc (H5MM.c:140)
> ==24685== by 0x4C320C2: H5F_accum_adjust (H5Faccum.c:335)
> ==24685== by 0x4C32558: H5F_accum_write (H5Faccum.c:416)
> ==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
> ==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
> ==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
> ==24685== by 0x4BCA036: H5C_flush_cache (H5C.c:1824)
> ==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
> ==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
> ==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
> ==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)

  Well, I didn't say we were completely done yet. :wink: If you can get this warning (which is most likely harmless) down to a simple program, I can see about fixing it.

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

Mark,

This should be fixed in upcoming 1.8.6 release (October 15). Hopefully a pre-released tar ball will be available for testing next week. We will announce it on FORUM.

Elena

···

On Sep 18, 2010, at 5:09 PM, Mark Miller wrote:

FYI, I've been seeing same memcpy src/dst overlap problem on our BG/P
system. It seems to be happening in one of the first attributes ever
written to the file, a tiny single 4 byte integer value'd attribute. The
pointer for both src and dst is the same value.

I cannot find a way to re-produce on something 'small'.

Mark

On Tue, 2010-08-24 at 06:13, Quincey Koziol wrote:

Hi marc,

On Aug 24, 2010, at 7:32 AM, Marc POINOT wrote:

Quincey Koziol wrote:

  Your code looks reasonable. We've been working on valgrind issues recently - can you check out the latest code from:
  http://*svn.hdfgroup.uiuc.edu/hdf5/branches/hdf5_1_8/
  See if these recent changes help.

Great, this removes all Valgrind weird messages I had on some hdf5 calls.

  Good! :slight_smile:

There's still one I'm investigating now (see below), only one test in my test suite
raises this message, I'm trying to find out what's different in this test.

==24685== Syscall param write(buf) points to uninitialised byte(s)
==24685== at 0x3E5070B012: __write_nocancel (in /lib64/tls/libpthread-2.3.4.so)
==24685== by 0x4C589C4: H5FD_sec2_write (H5FDsec2.c:839)
==24685== by 0x4C4E603: H5FD_write (H5FDint.c:184)
==24685== by 0x4C32FC4: H5F_accum_write (H5Faccum.c:580)
==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
==24685== by 0x4BC9F84: H5C_flush_cache (H5C.c:1801)
==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)
==24685== Address 0x534713C is 6,852 bytes inside a block of size 8,192 alloc'd
==24685== at 0x4905E12: realloc (vg_replace_malloc.c:306)
==24685== by 0x4CFCA97: H5MM_realloc (H5MM.c:140)
==24685== by 0x4C320C2: H5F_accum_adjust (H5Faccum.c:335)
==24685== by 0x4C32558: H5F_accum_write (H5Faccum.c:416)
==24685== by 0x4C34F7F: H5F_block_write (H5Fio.c:162)
==24685== by 0x4D2158F: H5O_flush (H5Ocache.c:486)
==24685== by 0x4BD765D: H5C_flush_single_entry (H5C.c:7606)
==24685== by 0x4BCA036: H5C_flush_cache (H5C.c:1824)
==24685== by 0x4BA1EB6: H5AC_flush (H5AC.c:843)
==24685== by 0x4C2CA29: H5F_flush (H5F.c:1685)
==24685== by 0x4C2A044: H5F_dest (H5F.c:994)
==24685== by 0x4C2D2F3: H5F_try_close (H5F.c:1909)

  Well, I didn't say we were completely done yet. :wink: If you can get this warning (which is most likely harmless) down to a simple program, I can see about fixing it.

  Quincey

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://*mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

--
Mark C. Miller, Lawrence Livermore National Laboratory
================!!LLNL BUSINESS ONLY!!================
miller86@llnl.gov urgent: miller86@pager.llnl.gov
T:8-6 (925)-423-5901 M/W/Th:7-12,2-7 (530)-753-8511

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org