writing data as fast as possible

Hdf-forum archives
1

We have an instrument generating several GBytes/second.
We would like to dump these data as fast as possible to disk (of course
in a parallel way), preferably with O_DIRECT to bypass the OS file cache
and to avoid seeks to maintain meta data.
Yet, it would be nice if these data can be seen as an HDF5 dataset.

My question is if it possible to do something like this. Either that
HDF5 has the option to write data in this way or through some mechanism
that HDF5 can treat such an external file as HDF5 data, possibly by
dynamically loading a shared library that knows how to interpret the
data.

I'm asking this because we have the impression that something like an
extendible dataset does not give us the performance/robustness we need.
In case of a crash we would like to lose as few data as possible, which
means that a regular flush has to be done which might kill performance.
I guess that without a flush the indices are only written at the end, so
the data cannot be found in case of a crash.

Cheers,
Ger van Diepen

begin 644 TEXT.htm
M/&AT;6P^"B`@/&AE860^"@H@(#PO:&5A9#X*("`\8F]D>2!S='EL93TB9F]N
M="UV87)I86YT.B!N;W)M86P[(&QI;F4M:&5I9VAT.B!N;W)M86P[(&UA<F=I
M;BUR:6=H=#H@-'!X.R!M87)G:6XM=&]P.B`T<'@[(&UA<F=I;BUB;W1T;VTZ
M(#%P>#L@;6%R9VEN+6QE9G0Z(#1P>"(^"B`@("`\<"!S='EL93TB;6%R9VEN
M+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^"B`@("`@(#QF;VYT(&9A8V4]
M(DQU8VED82!'<F%N9&4B('-I>F4](C,B/E=E(&AA=F4@86X@:6YS=')U;65N
M="!G96YE<F%T:6YG('-E=F5R86P@1T)Y=&5S+W-E8V]N9"X\+V9O;G0^("`@
M(#PO<#X*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM
M=&]P.B`P(CX*("`@("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ
M93TB,R(^5V4@=V]U;&0@;&EK92!T;R!D=6UP('1H97-E(&1A=&$@87,@9F%S
M="!A<R!P;W-S:6)L92!T;R!D:7-K("8C-#`[;V8@8V]U<G-E(&EN(&$@<&%R
M86QL96P@=V%Y)B,T,3LF(S0T.R!P<F5F97)A8FQY('=I=&@@3U]$25)%0U0@
M=&\@8GEP87-S('1H92!/4R!F:6QE(&-A8VAE(&%N9"!T;R!A=F]I9"!S965K
M<R!T;R!M86EN=&%I;B!M971A(&1A=&$N/"]F;VYT/B`@("`\+W`^"B`@("`\
M<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^"B`@
M("`@(#QF;VYT(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/EEE="8C
M-#0[(&ET('=O=6QD(&)E(&YI8V4@:68@=&AE<V4@9&%T82!C86X@8F4@<V5E
M;B!A<R!A;B!(1$8U(&1A=&%S970N/"]F;VYT/B`@("`\+W`^"CQB<CX@("`@
M("`*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P
M.B`P(CX*("`@("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ93TB
M,R(^37D@<75E<W1I;VX@:7,@:68@:70@<&]S<VEB;&4@=&\@9&\@<V]M971H
M:6YG(&QI:V4@=&AI<RX@16ET:&5R('1H870@2$1&-2!H87,@=&AE(&]P=&EO
M;B!T;R!W<FET92!D871A(&EN('1H:7,@=V%Y(&]R('1H<F]U9V@@<V]M92!M
M96-H86YI<VT@=&AA="!(1$8U(&-A;B!T<F5A="!S=6-H(&%N(&5X=&5R;F%L
M(&9I;&4@87,@2$1&-2!D871A)B,T-#L@<&]S<VEB;'D@8GD@9'EN86UI8V%L
M;'D@;&]A9&EN9R!A('-H87)E9"!L:6)R87)Y('1H870@:VYO=W,@:&]W('1O
M(&EN=&5R<')E="!T:&4@9&%T82X\+V9O;G0^("`@(#PO<#X*/&)R/B`@("`@
M(`H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T;VTZ(#`[(&UA<F=I;BUT;W`Z
M(#`B/@H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S
M(CY))B,S.3MM(&%S:VEN9R!T:&ES(&)E8V%U<V4@=V4@:&%V92!T:&4@:6UP
M<F5S<VEO;B!T:&%T('-O;65T:&EN9R!L:6ME(&%N(&5X=&5N9&EB;&4@9&%T
M87-E="!D;V5S(&YO="!G:79E('5S('1H92!P97)F;W)M86YC92]R;V)U<W1N
M97-S('=E(&YE960N($EN(&-A<V4@;V8@82!C<F%S:"!W92!W;W5L9"!L:6ME
M('1O(&QO<V4@87,@9F5W(&1A=&$@87,@<&]S<VEB;&4F(S0T.R!W:&EC:"!M
M96%N<R!T:&%T(&$@<F5G=6QA<B!F;'5S:"!H87,@=&\@8F4@9&]N92!W:&EC
M:"!M:6=H="!K:6QL('!E<F9O<FUA;F-E+B!)(&=U97-S('1H870@=VET:&]U
M="!A(&9L=7-H('1H92!I;F1I8V5S(&%R92!O;FQY('=R:71T96X@870@=&AE
M(&5N9"8C-#0[('-O('1H92!D871A(&-A;FYO="!B92!F;W5N9"!I;B!C87-E
M(&]F(&$@8W)A<V@N/"]F;VYT/B`@("`\+W`^"CQB<CX@("`@("`*("`@(#QP
M('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX*("`@
M("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ93TB,R(^0VAE97)S
M)B,T-#L\+V9O;G0^("`@(#PO<#X*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T
M=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX*("`@("`@/&9O;G0@9F%C93TB3'5C
M:61A($=R86YD92(@<VEZ93TB,R(^1V5R('9A;B!$:65P96X\+V9O;G0^("`@
8(#PO<#X*("`\+V)O9'D^"CPO:'1M;#X*
`
end

2

Ger

couldn't you use the multi VFL driver? In any case, the nGB/sec rate will
not be sustainable by any hard drive today in the market right? Even if you
buffer in SSDs still the disk write speed is your bottleneck rather than
hdf5...

HTH

-- dimitris

···

2010/1/15 Ger van Diepen <diepen@astron.nl>

We have an instrument generating several GBytes/second.
We would like to dump these data as fast as possible to disk (of course
in a parallel way), preferably with O_DIRECT to bypass the OS file cache
and to avoid seeks to maintain meta data.
Yet, it would be nice if these data can be seen as an HDF5 dataset.

My question is if it possible to do something like this. Either that
HDF5 has the option to write data in this way or through some mechanism
that HDF5 can treat such an external file as HDF5 data, possibly by
dynamically loading a shared library that knows how to interpret the
data.

I'm asking this because we have the impression that something like an
extendible dataset does not give us the performance/robustness we need.
In case of a crash we would like to lose as few data as possible, which
means that a regular flush has to be done which might kill performance.
I guess that without a flush the indices are only written at the end, so
the data cannot be found in case of a crash.

Cheers,
Ger van Diepen

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

3

Hi Ger,

We have an instrument generating several GBytes/second.
We would like to dump these data as fast as possible to disk (of course
in a parallel way), preferably with O_DIRECT to bypass the OS file cache
and to avoid seeks to maintain meta data.
Yet, it would be nice if these data can be seen as an HDF5 dataset.

My question is if it possible to do something like this. Either that
HDF5 has the option to write data in this way or through some mechanism
that HDF5 can treat such an external file as HDF5 data, possibly by
dynamically loading a shared library that knows how to interpret the
data.

I'm asking this because we have the impression that something like an
extendible dataset does not give us the performance/robustness we need.
In case of a crash we would like to lose as few data as possible, which
means that a regular flush has to be done which might kill performance.
I guess that without a flush the indices are only written at the end, so
the data cannot be found in case of a crash.

  You might want the "external" dataset storage feature - try H5Pset_external(). However, I'll also mention that we are working on parallel I/O improvements (and extensible dataset improvements) currently, so it may not be necessary to use external storage and still get the performance you are looking for.

  Quincey

···

On Jan 15, 2010, at 4:28 AM, Ger van Diepen wrote:

Cheers,
Ger van Diepen

begin 644 TEXT.htm
M/&AT;6P^"B`@/&AE860^"@H@(#PO:&5A9#X*("`\8F]D>2!S='EL93TB9F]N
M="UV87)I86YT.B!N;W)M86P[(&QI;F4M:&5I9VAT.B!N;W)M86P[(&UA<F=I
M;BUR:6=H=#H@-'!X.R!M87)G:6XM=&]P.B`T<'@[(&UA<F=I;BUB;W1T;VTZ
M(#%P>#L@;6%R9VEN+6QE9G0Z(#1P>"(^"B`@("`\<"!S='EL93TB;6%R9VEN
M+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^"B`@("`@(#QF;VYT(&9A8V4]
M(DQU8VED82!'<F%N9&4B('-I>F4](C,B/E=E(&AA=F4@86X@:6YS=')U;65N
M="!G96YE<F%T:6YG('-E=F5R86P@1T)Y=&5S+W-E8V]N9"X\+V9O;G0^("`@
M(#PO<#X*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM
M=&]P.B`P(CX*("`@("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ
M93TB,R(^5V4@=V]U;&0@;&EK92!T;R!D=6UP('1H97-E(&1A=&$@87,@9F%S
M="!A<R!P;W-S:6)L92!T;R!D:7-K("8C-#`[;V8@8V]U<G-E(&EN(&$@<&%R
M86QL96P@=V%Y)B,T,3LF(S0T.R!P<F5F97)A8FQY('=I=&@@3U]$25)%0U0@
M=&\@8GEP87-S('1H92!/4R!F:6QE(&-A8VAE(&%N9"!T;R!A=F]I9"!S965K
M<R!T;R!M86EN=&%I;B!M971A(&1A=&$N/"]F;VYT/B`@("`\+W`^"B`@("`\
M<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^"B`@
M("`@(#QF;VYT(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/EEE="8C
M-#0[(&ET('=O=6QD(&)E(&YI8V4@:68@=&AE<V4@9&%T82!C86X@8F4@<V5E
M;B!A<R!A;B!(1$8U(&1A=&%S970N/"]F;VYT/B`@("`\+W`^"CQB<CX@("`@
M("`*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P
M.B`P(CX*("`@("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ93TB
M,R(^37D@<75E<W1I;VX@:7,@:68@:70@<&]S<VEB;&4@=&\@9&\@<V]M971H
M:6YG(&QI:V4@=&AI<RX@16ET:&5R('1H870@2$1&-2!H87,@=&AE(&]P=&EO
M;B!T;R!W<FET92!D871A(&EN('1H:7,@=V%Y(&]R('1H<F]U9V@@<V]M92!M
M96-H86YI<VT@=&AA="!(1$8U(&-A;B!T<F5A="!S=6-H(&%N(&5X=&5R;F%L
M(&9I;&4@87,@2$1&-2!D871A)B,T-#L@<&]S<VEB;'D@8GD@9'EN86UI8V%L
M;'D@;&]A9&EN9R!A('-H87)E9"!L:6)R87)Y('1H870@:VYO=W,@:&]W('1O
M(&EN=&5R<')E="!T:&4@9&%T82X\+V9O;G0^("`@(#PO<#X*/&)R/B`@("`@
M(`H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T;VTZ(#`[(&UA<F=I;BUT;W`Z
M(#`B/@H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S
M(CY))B,S.3MM(&%S:VEN9R!T:&ES(&)E8V%U<V4@=V4@:&%V92!T:&4@:6UP
M<F5S<VEO;B!T:&%T('-O;65T:&EN9R!L:6ME(&%N(&5X=&5N9&EB;&4@9&%T
M87-E="!D;V5S(&YO="!G:79E('5S('1H92!P97)F;W)M86YC92]R;V)U<W1N
M97-S('=E(&YE960N($EN(&-A<V4@;V8@82!C<F%S:"!W92!W;W5L9"!L:6ME
M('1O(&QO<V4@87,@9F5W(&1A=&$@87,@<&]S<VEB;&4F(S0T.R!W:&EC:"!M
M96%N<R!T:&%T(&$@<F5G=6QA<B!F;'5S:"!H87,@=&\@8F4@9&]N92!W:&EC
M:"!M:6=H="!K:6QL('!E<F9O<FUA;F-E+B!)(&=U97-S('1H870@=VET:&]U
M="!A(&9L=7-H('1H92!I;F1I8V5S(&%R92!O;FQY('=R:71T96X@870@=&AE
M(&5N9"8C-#0[('-O('1H92!D871A(&-A;FYO="!B92!F;W5N9"!I;B!C87-E
M(&]F(&$@8W)A<V@N/"]F;VYT/B`@("`\+W`^"CQB<CX@("`@("`*("`@(#QP
M('-T>6QE/2)M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX*("`@
M("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ93TB,R(^0VAE97)S
M)B,T-#L\+V9O;G0^("`@(#PO<#X*("`@(#QP('-T>6QE/2)M87)G:6XM8F]T
M=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX*("`@("`@/&9O;G0@9F%C93TB3'5C
M:61A($=R86YD92(@<VEZ93TB,R(^1V5R('9A;B!$:65P96X\+V9O;G0^("`@
8(#PO<#X*("`\+V)O9'D^"CPO:'1M;#X*
`
end

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

4

Hi Dimitris,

Thanks for your suggestion.
I'll look into the VFL possibility. A disadvantage I see is that you
have to make it known, so common tools won't recognize it. I guess it is
not possible to load it dynamically. In principle it should be possible
by storing the driver name and use it as the name of the shared library.

Does using a specific VFL mean that all data are written through that
VFL and that it is not possible to store the meta data in the 'normal'
way and only store the bulk data through that VFL?
Preferably I would like to bypass HDF5 entirely when writing (apart
from telling the VFL) and use that VFL when reading. In that way I can
write as I like and still use tools like h5view to look at the data
(provided the VFL can be loaded dynamically).

As I said writing happens in parallel, so each disk has to part of the
job. But we would like to keep the amount of resources needed to a
minimum, for which optimal write performance is needed. Reading is less
of an issue.

Cheers,
Ger

Dimitris Servis <servisster@gmail.com> 1/15/2010 11:47 AM >>>

Ger

couldn't you use the multi VFL driver? In any case, the nGB/sec rate
will not be sustainable by any hard drive today in the market right?
Even if you buffer in SSDs still the disk write speed is your bottleneck
rather than hdf5...

HTH

-- dimitris

···

2010/1/15 Ger van Diepen <diepen@astron.nl>

We have an instrument generating several GBytes/second.
We would like to dump these data as fast as possible to disk (of
course
in a parallel way), preferably with O_DIRECT to bypass the OS file
cache
and to avoid seeks to maintain meta data.
Yet, it would be nice if these data can be seen as an HDF5 dataset.

My question is if it possible to do something like this. Either that
HDF5 has the option to write data in this way or through some
mechanism
that HDF5 can treat such an external file as HDF5 data, possibly by
dynamically loading a shared library that knows how to interpret the
data.

I'm asking this because we have the impression that something like an
extendible dataset does not give us the performance/robustness we
need.
In case of a crash we would like to lose as few data as possible,
which
means that a regular flush has to be done which might kill
performance.
I guess that without a flush the indices are only written at the end,
so
the data cannot be found in case of a crash.

Cheers,
Ger van Diepen

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

begin 644 TEXT.htm
M/&AT;6P^#0H@(#QH96%D/@T*#0H@(#PO:&5A9#X-"B`@/&)O9'D@<W1Y;&4]
M(F9O;G0M=F%R:6%N=#H@;F]R;6%L.R!L:6YE+6AE:6=H=#H@;F]R;6%L.R!M
M87)G:6XM<FEG:'0Z(#1P>#L@;6%R9VEN+71O<#H@-'!X.R!M87)G:6XM8F]T
M=&]M.B`Q<'@[(&UA<F=I;BUL969T.B`T<'@B/@T*("`@(#QP('-T>6QE/2)M
M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QF;VYT
M(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/DAI($1I;6ET<FES)B,T
M-#L\+V9O;G0^("`@(#PO<#X-"CQB<CX@("`@("`-"B`@("`\<"!S='EL93TB
M;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^#0H@("`@("`\9F]N
M="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S(CY4:&%N:W,@9F]R('EO
M=7(@<W5G9V5S=&EO;BX\+V9O;G0^("`@(#PO<#X-"B`@("`\<"!S='EL93TB
M;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^#0H@("`@("`\9F]N
M="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S(CY))B,S.3ML;"!L;V]K
M(&EN=&\@=&AE(%9&3"!P;W-S:6)I;&ET>2X@02!D:7-A9'9A;G1A9V4@22!S
M964@:7,@=&AA="!Y;W4@:&%V92!T;R!M86ME(&ET(&MN;W=N)B,T-#L@<V\@
M8V]M;6]N('1O;VQS('=O;B8C,SD[="!R96-O9VYI>F4@:70N($D@9W5E<W,@
M:70@:7,@;F]T("9N8G-P.W!O<W-I8FQE('1O(&QO860@:70@9'EN86UI8V%L
M;'DN($EN('!R:6YC:7!L92!I="!S:&]U;&0@8F4@<&]S<VEB;&4@8GD@<W1O
M<FEN9R!T:&4@9')I=F5R(&YA;64@86YD('5S92!I="!A<R!T:&4@;F%M92!O
M9B!T:&4@<VAA<F5D(&QI8G)A<GDN/"]F;VYT/B`@("`\+W`^#0H\8G(^("`@
M("`@#0H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T;VTZ(#`[(&UA<F=I;BUT
M;W`Z(#`B/@T*("`@("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ
M93TB,R(^1&]E<R!U<VEN9R!A('-P96-I9FEC(%9&3"!M96%N('1H870@86QL
M(&1A=&$@87)E('=R:71T96X@=&AR;W5G:"!T:&%T(%9&3"!A;F0@=&AA="!I
M="!I<R!N;W0@<&]S<VEB;&4@=&\@<W1O<F4@=&AE(&UE=&$@9&%T82!I;B!T
M:&4@)B,S.3MN;W)M86PF(S,Y.R!W87D@86YD(&]N;'D@<W1O<F4@=&AE(&)U
M;&L@9&%T82!T:')O=6=H('1H870@5D9,)B,V,SL\+V9O;G0^("`@(#PO<#X-
M"B`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@
M,"(^#0H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S
M(CY0<F5F97)A8FQY($D@=V]U;&0@;&EK92!T;R!B>7!A<W,@2$1&-2!E;G1I
M<F5L>2!W:&5N('=R:71I;F<@)B,T,#MA<&%R="!F<F]M('1E;&QI;F<@=&AE
M(%9&3"8C-#$[(&%N9"!U<V4@=&AA="!61DP@=VAE;B!R96%D:6YG+B!);B!T
M:&%T('=A>2!)(&-A;B!W<FET92!A<R!)(&QI:V4@86YD('-T:6QL('5S92!T
M;V]L<R!L:6ME(&@U=FEE=R!T;R!L;V]K(&%T('1H92!D871A("8C-#`[<')O
M=FED960@=&AE(%9&3"!C86X@8F4@;&]A9&5D(&1Y;F%M:6-A;&QY)B,T,3LN
M(#PO9F]N=#X@("`@/"]P/@T*/&)R/B`@("`@(`T*("`@(#QP('-T>6QE/2)M
M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QF;VYT
M(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/D%S($D@<V%I9"!W<FET
M:6YG(&AA<'!E;G,@:6X@<&%R86QL96PF(S0T.R!S;R!E86-H(&1I<VL@:&%S
M('1O('!A<G0@;V8@=&AE(&IO8BX@0G5T('=E('=O=6QD(&QI:V4@=&\@:V5E
M<"!T:&4@86UO=6YT(&]F(')E<V]U<F-E<R!N965D960@=&\@82!M:6YI;75M
M)B,T-#L@9F]R('=H:6-H(&]P=&EM86P@=W)I=&4@<&5R9F]R;6%N8V4@:7,@
M;F5E9&5D+B!296%D:6YG(&ES(&QE<W,@;V8@86X@:7-S=64N/"]F;VYT/B`@
M("`\+W`^#0H\8G(^("`@("`@#0H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T
M;VTZ(#`[(&UA<F=I;BUT;W`Z(#`B/@T*("`@("`@/&9O;G0@9F%C93TB3'5C
M:61A($=R86YD92(@<VEZ93TB,R(^0VAE97)S)B,T-#L\+V9O;G0^("`@(#PO
M<#X-"B`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O
M<#H@,"(^#0H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE
M/2(S(CY'97(\+V9O;G0^/&)R/CQB<CXF9W0[)F=T.R9G=#L@1&EM:71R:7,@
M4V5R=FES("9L=#MS97)V:7-S=&5R0&=M86EL+F-O;29G=#L@,2\Q-2\R,#$P
M(#$Q.C0W($%-("9G=#LF9W0[)F=T.SQB<CY'97(\8G(^/&)R/F-O=6QD;B8C
M,SD[="!Y;W4@=7-E('1H92!M=6QT:2!61DP@9')I=F5R)B,V,SL@26X@86YY
M(&-A<V4F(S0T.R!T:&4@;D="+W-E8R!R871E('=I;&P@;F]T(&)E('-U<W1A
M:6YA8FQE(&)Y(&%N>2!H87)D(&1R:79E('1O9&%Y(&EN('1H92!M87)K970@
M<FEG:'0F(S8S.R!%=F5N(&EF('EO=2!B=69F97(@:6X@4U-$<R!S=&EL;"!T
M:&4@9&ES:R!W<FET92!S<&5E9"!I<R!Y;W5R(&)O='1L96YE8VL@<F%T:&5R
M('1H86X@:&1F-2XN+CQB<CX\8G(^2%1(/&)R/CQB<CXM+2!D:6UI=')I<SQB
M<CX\8G(^("`@(#PO<#X-"B`@("`\9&EV(&-L87-S/2)G;6%I;%]Q=6]T92(^
M#0H@("`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O
M<#H@,"(^#0H@("`@("`@(#(P,3`O,2\Q-2!'97(@=F%N($1I97!E;B`\<W!A
M;B!D:7(](FQT<B(^)FQT.SQA(&AR968](FUA:6QT;SID:65P96Y`87-T<F]N
M+FYL(CYD:65P96Y`87-T<F]N+FYL/"]A/B9G=#L\+W-P86X^/&)R/B`@("`@
M(#PO<#X-"B`@("`@(#QB;&]C:W%U;W1E('-T>6QE/2)P861D:6YG+6QE9G0Z
M(#`[(&)O<F1E<BUL969T.B`Q<'@@<V]L:60@<F=B*#(P-"P@,C`T+"`R,#0I
M.R!M87)G:6XM<FEG:'0Z(#!P=#L@;6%R9VEN+71O<#H@,'!T.R!M87)G:6XM
M8F]T=&]M.B`P<'0[(&UA<F=I;BUL969T.B`P(B!C;&%S<STB9VUA:6Q?<75O
M=&4B/@T*("`@("`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R
M9VEN+71O<#H@,"(^#0H@("`@("`@("`@/&)R/@T*("`@("`@("`@(%=E(&AA
M=F4@86X@:6YS=')U;65N="!G96YE<F%T:6YG('-E=F5R86P@1T)Y=&5S+W-E
M8V]N9"X\8G(^5V4@=V]U;&0@;&EK92!T;R!D=6UP('1H97-E(&1A=&$@87,@
M9F%S="!A<R!P;W-S:6)L92!T;R!D:7-K("8C-#`[;V8@8V]U<G-E/&)R/FEN
M(&$@<&%R86QL96P@=V%Y)B,T,3LF(S0T.R!P<F5F97)A8FQY('=I=&@@3U]$
M25)%0U0@=&\@8GEP87-S('1H92!/4R!F:6QE(&-A8VAE/&)R/F%N9"!T;R!A
M=F]I9"!S965K<R!T;R!M86EN=&%I;B!M971A(&1A=&$N/&)R/EEE="8C-#0[
M(&ET('=O=6QD(&)E(&YI8V4@:68@=&AE<V4@9&%T82!C86X@8F4@<V5E;B!A
M<R!A;B!(1$8U(&1A=&%S970N/&)R/CQB<CY->2!Q=65S=&EO;B!I<R!I9B!I
M="!P;W-S:6)L92!T;R!D;R!S;VUE=&AI;F<@;&EK92!T:&ES+B!%:71H97(@
M=&AA=#QB<CY(1$8U(&AA<R!T:&4@;W!T:6]N('1O('=R:71E(&1A=&$@:6X@
M=&AI<R!W87D@;W(@=&AR;W5G:"!S;VUE(&UE8VAA;FES;3QB<CYT:&%T($A$
M1C4@8V%N('1R96%T('-U8V@@86X@97AT97)N86P@9FEL92!A<R!(1$8U(&1A
M=&$F(S0T.R!P;W-S:6)L>2!B>3QB<CYD>6YA;6EC86QL>2!L;V%D:6YG(&$@
M<VAA<F5D(&QI8G)A<GD@=&AA="!K;F]W<R!H;W<@=&\@:6YT97)P<F5T('1H
M93QB<CYD871A+CQB<CX\8G(^228C,SD[;2!A<VMI;F<@=&AI<R!B96-A=7-E
M('=E(&AA=F4@=&AE(&EM<')E<W-I;VX@=&AA="!S;VUE=&AI;F<@;&EK92!A
M;CQB<CYE>'1E;F1I8FQE(&1A=&%S970@9&]E<R!N;W0@9VEV92!U<R!T:&4@
M<&5R9F]R;6%N8V4O<F]B=7-T;F5S<R!W92!N965D+CQB<CY);B!C87-E(&]F
M(&$@8W)A<V@@=V4@=V]U;&0@;&EK92!T;R!L;W-E(&%S(&9E=R!D871A(&%S
M('!O<W-I8FQE)B,T-#L@=VAI8V@\8G(^;65A;G,@=&AA="!A(')E9W5L87(@
M9FQU<V@@:&%S('1O(&)E(&1O;F4@=VAI8V@@;6EG:'0@:VEL;"!P97)F;W)M
M86YC92X\8G(^22!G=65S<R!T:&%T('=I=&AO=70@82!F;'5S:"!T:&4@:6YD
M:6-E<R!A<F4@;VYL>2!W<FET=&5N(&%T('1H92!E;F0F(S0T.R!S;SQB<CYT
M:&4@9&%T82!C86YN;W0@8F4@9F]U;F0@:6X@8V%S92!O9B!A(&-R87-H+CQB
M<CX\8G(^0VAE97)S)B,T-#L\8G(^1V5R('9A;B!$:65P96X\8G(^/&)R/CQB
M<CY?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?
M7U]?7SQB<CY(9&8M9F]R=6T@:7,@9F]R($A$1B!S;V9T=V%R92!U<V5R<R!D
M:7-C=7-S:6]N+CQB<CX\82!H<F5F/2)M86EL=&\Z2&1F+69O<G5M0&AD9F=R
M;W5P+F]R9R(^2&1F+69O<G5M0&AD9F=R;W5P+F]R9SPO83X\8G(^/&$@:')E
M9CTB:'1T<#HO+VUA:6PN:&1F9W)O=7`N;W)G+VUA:6QM86XO;&ES=&EN9F\O
M:&1F+69O<G5M7VAD9F=R;W5P+F]R9R(@=&%R9V5T/2)?8FQA;FLB/FAT='`Z
M+R]M86EL+FAD9F=R;W5P+F]R9R]M86EL;6%N+VQI<W1I;F9O+VAD9BUF;W)U
M;5]H9&9G<F]U<"YO<F<\+V$^/&)R/B`@("`@("`@/"]P/@T*("`@("`@/"]B
M;&]C:W%U;W1E/@T*("`@(#PO9&EV/@T*("`@(#QP('-T>6QE/2)M87)G:6XM
M8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QB<CX-"B`@("`\
9+W`^#0H@(#PO8F]D>3X-"CPO:'1M;#X-"G9A
`
end

5

Hi Ger

I am not sure if you can compose a multi file afterwards. The final result
is what you want: multiple raw data files and multiple metadata files
physically separated. Maybe with a little tweak of the VFL driver (not a
very difficult task) and some prerequisites (raw data file size and type)
you can easily force HDF5 to use the address spaces of the separate files in
datasets. After all, 1GB of doubles on disk is little different that 1GB of
doubles in memory... Maybe this can give you a hint?

http://www.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetExternal

HTH

--ds

···

2010/1/15 Ger van Diepen <diepen@astron.nl>

Hi Dimitris,

Thanks for your suggestion.
I'll look into the VFL possibility. A disadvantage I see is that you
have to make it known, so common tools won't recognize it. I guess it is
not possible to load it dynamically. In principle it should be possible
by storing the driver name and use it as the name of the shared library.

Does using a specific VFL mean that all data are written through that
VFL and that it is not possible to store the meta data in the 'normal'
way and only store the bulk data through that VFL?
Preferably I would like to bypass HDF5 entirely when writing (apart
from telling the VFL) and use that VFL when reading. In that way I can
write as I like and still use tools like h5view to look at the data
(provided the VFL can be loaded dynamically).

As I said writing happens in parallel, so each disk has to part of the
job. But we would like to keep the amount of resources needed to a
minimum, for which optimal write performance is needed. Reading is less
of an issue.

Cheers,
Ger

>>> Dimitris Servis <servisster@gmail.com> 1/15/2010 11:47 AM >>>
Ger

couldn't you use the multi VFL driver? In any case, the nGB/sec rate
will not be sustainable by any hard drive today in the market right?
Even if you buffer in SSDs still the disk write speed is your bottleneck
rather than hdf5...

HTH

-- dimitris

2010/1/15 Ger van Diepen <diepen@astron.nl>

We have an instrument generating several GBytes/second.
We would like to dump these data as fast as possible to disk (of
course
in a parallel way), preferably with O_DIRECT to bypass the OS file
cache
and to avoid seeks to maintain meta data.
Yet, it would be nice if these data can be seen as an HDF5 dataset.

My question is if it possible to do something like this. Either that
HDF5 has the option to write data in this way or through some
mechanism
that HDF5 can treat such an external file as HDF5 data, possibly by
dynamically loading a shared library that knows how to interpret the
data.

I'm asking this because we have the impression that something like an
extendible dataset does not give us the performance/robustness we
need.
In case of a crash we would like to lose as few data as possible,
which
means that a regular flush has to be done which might kill
performance.
I guess that without a flush the indices are only written at the end,
so
the data cannot be found in case of a crash.

Cheers,
Ger van Diepen

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

6

Hi Ger,

Hi Dimitris,

Thanks for your suggestion.
I'll look into the VFL possibility. A disadvantage I see is that you
have to make it known, so common tools won't recognize it. I guess it is
not possible to load it dynamically. In principle it should be possible
by storing the driver name and use it as the name of the shared library.

Does using a specific VFL mean that all data are written through that
VFL and that it is not possible to store the meta data in the 'normal'
way and only store the bulk data through that VFL?
Preferably I would like to bypass HDF5 entirely when writing (apart
from telling the VFL) and use that VFL when reading. In that way I can
write as I like and still use tools like h5view to look at the data
(provided the VFL can be loaded dynamically).

As I said writing happens in parallel, so each disk has to part of the
job. But we would like to keep the amount of resources needed to a
minimum, for which optimal write performance is needed. Reading is less
of an issue.

  The "split" file driver may also be a way to get what you'd like, although it stores all the raw datasets in one file, which may be difficult to get your application to play nicely with. If possible I would try to stay with just using the MPI file driver and see where the bottlenecks are, so we can correct them.

  Quincey

···

On Jan 15, 2010, at 5:30 AM, Ger van Diepen wrote:

Cheers,
Ger

Dimitris Servis <servisster@gmail.com> 1/15/2010 11:47 AM >>>

Ger

couldn't you use the multi VFL driver? In any case, the nGB/sec rate
will not be sustainable by any hard drive today in the market right?
Even if you buffer in SSDs still the disk write speed is your bottleneck
rather than hdf5...

HTH

-- dimitris

2010/1/15 Ger van Diepen <diepen@astron.nl>

We have an instrument generating several GBytes/second.
We would like to dump these data as fast as possible to disk (of
course
in a parallel way), preferably with O_DIRECT to bypass the OS file
cache
and to avoid seeks to maintain meta data.
Yet, it would be nice if these data can be seen as an HDF5 dataset.

My question is if it possible to do something like this. Either that
HDF5 has the option to write data in this way or through some
mechanism
that HDF5 can treat such an external file as HDF5 data, possibly by
dynamically loading a shared library that knows how to interpret the
data.

I'm asking this because we have the impression that something like an
extendible dataset does not give us the performance/robustness we
need.
In case of a crash we would like to lose as few data as possible,
which
means that a regular flush has to be done which might kill
performance.
I guess that without a flush the indices are only written at the end,
so
the data cannot be found in case of a crash.

Cheers,
Ger van Diepen

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org

begin 644 TEXT.htm
M/&AT;6P^#0H@(#QH96%D/@T*#0H@(#PO:&5A9#X-"B`@/&)O9'D@<W1Y;&4]
M(F9O;G0M=F%R:6%N=#H@;F]R;6%L.R!L:6YE+6AE:6=H=#H@;F]R;6%L.R!M
M87)G:6XM<FEG:'0Z(#1P>#L@;6%R9VEN+71O<#H@-'!X.R!M87)G:6XM8F]T
M=&]M.B`Q<'@[(&UA<F=I;BUL969T.B`T<'@B/@T*("`@(#QP('-T>6QE/2)M
M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QF;VYT
M(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/DAI($1I;6ET<FES)B,T
M-#L\+V9O;G0^("`@(#PO<#X-"CQB<CX@("`@("`-"B`@("`\<"!S='EL93TB
M;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^#0H@("`@("`\9F]N
M="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S(CY4:&%N:W,@9F]R('EO
M=7(@<W5G9V5S=&EO;BX\+V9O;G0^("`@(#PO<#X-"B`@("`\<"!S='EL93TB
M;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@,"(^#0H@("`@("`\9F]N
M="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S(CY))B,S.3ML;"!L;V]K
M(&EN=&\@=&AE(%9&3"!P;W-S:6)I;&ET>2X@02!D:7-A9'9A;G1A9V4@22!S
M964@:7,@=&AA="!Y;W4@:&%V92!T;R!M86ME(&ET(&MN;W=N)B,T-#L@<V\@
M8V]M;6]N('1O;VQS('=O;B8C,SD[="!R96-O9VYI>F4@:70N($D@9W5E<W,@
M:70@:7,@;F]T("9N8G-P.W!O<W-I8FQE('1O(&QO860@:70@9'EN86UI8V%L
M;'DN($EN('!R:6YC:7!L92!I="!S:&]U;&0@8F4@<&]S<VEB;&4@8GD@<W1O
M<FEN9R!T:&4@9')I=F5R(&YA;64@86YD('5S92!I="!A<R!T:&4@;F%M92!O
M9B!T:&4@<VAA<F5D(&QI8G)A<GDN/"]F;VYT/B`@("`\+W`^#0H\8G(^("`@
M("`@#0H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T;VTZ(#`[(&UA<F=I;BUT
M;W`Z(#`B/@T*("`@("`@/&9O;G0@9F%C93TB3'5C:61A($=R86YD92(@<VEZ
M93TB,R(^1&]E<R!U<VEN9R!A('-P96-I9FEC(%9&3"!M96%N('1H870@86QL
M(&1A=&$@87)E('=R:71T96X@=&AR;W5G:"!T:&%T(%9&3"!A;F0@=&AA="!I
M="!I<R!N;W0@<&]S<VEB;&4@=&\@<W1O<F4@=&AE(&UE=&$@9&%T82!I;B!T
M:&4@)B,S.3MN;W)M86PF(S,Y.R!W87D@86YD(&]N;'D@<W1O<F4@=&AE(&)U
M;&L@9&%T82!T:')O=6=H('1H870@5D9,)B,V,SL\+V9O;G0^("`@(#PO<#X-
M"B`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O<#H@
M,"(^#0H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE/2(S
M(CY0<F5F97)A8FQY($D@=V]U;&0@;&EK92!T;R!B>7!A<W,@2$1&-2!E;G1I
M<F5L>2!W:&5N('=R:71I;F<@)B,T,#MA<&%R="!F<F]M('1E;&QI;F<@=&AE
M(%9&3"8C-#$[(&%N9"!U<V4@=&AA="!61DP@=VAE;B!R96%D:6YG+B!);B!T
M:&%T('=A>2!)(&-A;B!W<FET92!A<R!)(&QI:V4@86YD('-T:6QL('5S92!T
M;V]L<R!L:6ME(&@U=FEE=R!T;R!L;V]K(&%T('1H92!D871A("8C-#`[<')O
M=FED960@=&AE(%9&3"!C86X@8F4@;&]A9&5D(&1Y;F%M:6-A;&QY)B,T,3LN
M(#PO9F]N=#X@("`@/"]P/@T*/&)R/B`@("`@(`T*("`@(#QP('-T>6QE/2)M
M87)G:6XM8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QF;VYT
M(&9A8V4](DQU8VED82!'<F%N9&4B('-I>F4](C,B/D%S($D@<V%I9"!W<FET
M:6YG(&AA<'!E;G,@:6X@<&%R86QL96PF(S0T.R!S;R!E86-H(&1I<VL@:&%S
M('1O('!A<G0@;V8@=&AE(&IO8BX@0G5T('=E('=O=6QD(&QI:V4@=&\@:V5E
M<"!T:&4@86UO=6YT(&]F(')E<V]U<F-E<R!N965D960@=&\@82!M:6YI;75M
M)B,T-#L@9F]R('=H:6-H(&]P=&EM86P@=W)I=&4@<&5R9F]R;6%N8V4@:7,@
M;F5E9&5D+B!296%D:6YG(&ES(&QE<W,@;V8@86X@:7-S=64N/"]F;VYT/B`@
M("`\+W`^#0H\8G(^("`@("`@#0H@("`@/'`@<W1Y;&4](FUA<F=I;BUB;W1T
M;VTZ(#`[(&UA<F=I;BUT;W`Z(#`B/@T*("`@("`@/&9O;G0@9F%C93TB3'5C
M:61A($=R86YD92(@<VEZ93TB,R(^0VAE97)S)B,T-#L\+V9O;G0^("`@(#PO
M<#X-"B`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O
M<#H@,"(^#0H@("`@("`\9F]N="!F86-E/2),=6-I9&$@1W)A;F1E(B!S:7IE
M/2(S(CY'97(\+V9O;G0^/&)R/CQB<CXF9W0[)F=T.R9G=#L@1&EM:71R:7,@
M4V5R=FES("9L=#MS97)V:7-S=&5R0&=M86EL+F-O;29G=#L@,2\Q-2\R,#$P
M(#$Q.C0W($%-("9G=#LF9W0[)F=T.SQB<CY'97(\8G(^/&)R/F-O=6QD;B8C
M,SD[="!Y;W4@=7-E('1H92!M=6QT:2!61DP@9')I=F5R)B,V,SL@26X@86YY
M(&-A<V4F(S0T.R!T:&4@;D="+W-E8R!R871E('=I;&P@;F]T(&)E('-U<W1A
M:6YA8FQE(&)Y(&%N>2!H87)D(&1R:79E('1O9&%Y(&EN('1H92!M87)K970@
M<FEG:'0F(S8S.R!%=F5N(&EF('EO=2!B=69F97(@:6X@4U-$<R!S=&EL;"!T
M:&4@9&ES:R!W<FET92!S<&5E9"!I<R!Y;W5R(&)O='1L96YE8VL@<F%T:&5R
M('1H86X@:&1F-2XN+CQB<CX\8G(^2%1(/&)R/CQB<CXM+2!D:6UI=')I<SQB
M<CX\8G(^("`@(#PO<#X-"B`@("`\9&EV(&-L87-S/2)G;6%I;%]Q=6]T92(^
M#0H@("`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R9VEN+71O
M<#H@,"(^#0H@("`@("`@(#(P,3`O,2\Q-2!'97(@=F%N($1I97!E;B`\<W!A
M;B!D:7(](FQT<B(^)FQT.SQA(&AR968](FUA:6QT;SID:65P96Y`87-T<F]N
M+FYL(CYD:65P96Y`87-T<F]N+FYL/"]A/B9G=#L\+W-P86X^/&)R/B`@("`@
M(#PO<#X-"B`@("`@(#QB;&]C:W%U;W1E('-T>6QE/2)P861D:6YG+6QE9G0Z
M(#`[(&)O<F1E<BUL969T.B`Q<'@@<V]L:60@<F=B*#(P-"P@,C`T+"`R,#0I
M.R!M87)G:6XM<FEG:'0Z(#!P=#L@;6%R9VEN+71O<#H@,'!T.R!M87)G:6XM
M8F]T=&]M.B`P<'0[(&UA<F=I;BUL969T.B`P(B!C;&%S<STB9VUA:6Q?<75O
M=&4B/@T*("`@("`@("`\<"!S='EL93TB;6%R9VEN+6)O='1O;3H@,#L@;6%R
M9VEN+71O<#H@,"(^#0H@("`@("`@("`@/&)R/@T*("`@("`@("`@(%=E(&AA
M=F4@86X@:6YS=')U;65N="!G96YE<F%T:6YG('-E=F5R86P@1T)Y=&5S+W-E
M8V]N9"X\8G(^5V4@=V]U;&0@;&EK92!T;R!D=6UP('1H97-E(&1A=&$@87,@
M9F%S="!A<R!P;W-S:6)L92!T;R!D:7-K("8C-#`[;V8@8V]U<G-E/&)R/FEN
M(&$@<&%R86QL96P@=V%Y)B,T,3LF(S0T.R!P<F5F97)A8FQY('=I=&@@3U]$
M25)%0U0@=&\@8GEP87-S('1H92!/4R!F:6QE(&-A8VAE/&)R/F%N9"!T;R!A
M=F]I9"!S965K<R!T;R!M86EN=&%I;B!M971A(&1A=&$N/&)R/EEE="8C-#0[
M(&ET('=O=6QD(&)E(&YI8V4@:68@=&AE<V4@9&%T82!C86X@8F4@<V5E;B!A
M<R!A;B!(1$8U(&1A=&%S970N/&)R/CQB<CY->2!Q=65S=&EO;B!I<R!I9B!I
M="!P;W-S:6)L92!T;R!D;R!S;VUE=&AI;F<@;&EK92!T:&ES+B!%:71H97(@
M=&AA=#QB<CY(1$8U(&AA<R!T:&4@;W!T:6]N('1O('=R:71E(&1A=&$@:6X@
M=&AI<R!W87D@;W(@=&AR;W5G:"!S;VUE(&UE8VAA;FES;3QB<CYT:&%T($A$
M1C4@8V%N('1R96%T('-U8V@@86X@97AT97)N86P@9FEL92!A<R!(1$8U(&1A
M=&$F(S0T.R!P;W-S:6)L>2!B>3QB<CYD>6YA;6EC86QL>2!L;V%D:6YG(&$@
M<VAA<F5D(&QI8G)A<GD@=&AA="!K;F]W<R!H;W<@=&\@:6YT97)P<F5T('1H
M93QB<CYD871A+CQB<CX\8G(^228C,SD[;2!A<VMI;F<@=&AI<R!B96-A=7-E
M('=E(&AA=F4@=&AE(&EM<')E<W-I;VX@=&AA="!S;VUE=&AI;F<@;&EK92!A
M;CQB<CYE>'1E;F1I8FQE(&1A=&%S970@9&]E<R!N;W0@9VEV92!U<R!T:&4@
M<&5R9F]R;6%N8V4O<F]B=7-T;F5S<R!W92!N965D+CQB<CY);B!C87-E(&]F
M(&$@8W)A<V@@=V4@=V]U;&0@;&EK92!T;R!L;W-E(&%S(&9E=R!D871A(&%S
M('!O<W-I8FQE)B,T-#L@=VAI8V@\8G(^;65A;G,@=&AA="!A(')E9W5L87(@
M9FQU<V@@:&%S('1O(&)E(&1O;F4@=VAI8V@@;6EG:'0@:VEL;"!P97)F;W)M
M86YC92X\8G(^22!G=65S<R!T:&%T('=I=&AO=70@82!F;'5S:"!T:&4@:6YD
M:6-E<R!A<F4@;VYL>2!W<FET=&5N(&%T('1H92!E;F0F(S0T.R!S;SQB<CYT
M:&4@9&%T82!C86YN;W0@8F4@9F]U;F0@:6X@8V%S92!O9B!A(&-R87-H+CQB
M<CX\8G(^0VAE97)S)B,T-#L\8G(^1V5R('9A;B!$:65P96X\8G(^/&)R/CQB
M<CY?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?7U]?
M7U]?7SQB<CY(9&8M9F]R=6T@:7,@9F]R($A$1B!S;V9T=V%R92!U<V5R<R!D
M:7-C=7-S:6]N+CQB<CX\82!H<F5F/2)M86EL=&\Z2&1F+69O<G5M0&AD9F=R
M;W5P+F]R9R(^2&1F+69O<G5M0&AD9F=R;W5P+F]R9SPO83X\8G(^/&$@:')E
M9CTB:'1T<#HO+VUA:6PN:&1F9W)O=7`N;W)G+VUA:6QM86XO;&ES=&EN9F\O
M:&1F+69O<G5M7VAD9F=R;W5P+F]R9R(@=&%R9V5T/2)?8FQA;FLB/FAT='`Z
M+R]M86EL+FAD9F=R;W5P+F]R9R]M86EL;6%N+VQI<W1I;F9O+VAD9BUF;W)U
M;5]H9&9G<F]U<"YO<F<\+V$^/&)R/B`@("`@("`@/"]P/@T*("`@("`@/"]B
M;&]C:W%U;W1E/@T*("`@(#PO9&EV/@T*("`@(#QP('-T>6QE/2)M87)G:6XM
M8F]T=&]M.B`P.R!M87)G:6XM=&]P.B`P(CX-"B`@("`@(#QB<CX-"B`@("`\
9+W`^#0H@(#PO8F]D>3X-"CPO:'1M;#X-"G9A
`
end

_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@hdfgroup.org
http://mail.hdfgroup.org/mailman/listinfo/hdf-forum_hdfgroup.org