Compound type with an array inside

[rezahousseini]

I have a dataset with a compound type and inside this compound type an array. Is there a way to read this compound type into Julia? It seems that Array types are not supported yet, but perhaps it's possible to patch this scenario with the low-level routines?

[timholy]

Array types are definitely supported (see the examples in test/), and JLD supports saving a loading Julia types (which are compound objects). However, our support for HDF5 compound objects is still poor, largely because C structure support is still unfinished in Julia itself JuliaLang/julia#2818.

Am I correct in guessing that you have an HDF5 file written by other software that you want to read into Julia? If you can email me a small test file, I'd be happy to play with this a bit.

[rezahousseini]

I sent you a small dataset

[timholy]

Thanks, got it. Indeed we don't yet support the fixed-size H5T_ARRAY type. I've looked into it a bit, and it's not entirely trivial, because Julia doesn't support fixed-size arrays. The hdf5_to_julia infrastructure might need a little reworking.

Because my ability to work on this over the next couple of weeks will be limited, I've started a branch called h5array that contains my initial efforts. If you or anyone else wants to take a stab at this, be my guest. Otherwise I'll plug away as time allows, but my guess is that not much will happen until near the end of this month. If you're up to it, it would be a great contribution.

[timholy]

The h5ex_t_array.h5 example on http://www.hdfgroup.org/ftp/HDF5/examples/examples-by-api/api18-c.html is relevant, and slightly simpler than the file you sent me (since it doesn't involve nesting array types inside a compound object).

[rezahousseini]

Thanks a lot Tim. I'm not sure sure if I have the ability to proceed on this. I tried a workaround with PyCall and the python module h5py but it was unbelievable slow and couldn't handle integer conversion.

[timholy]

I fully understand. I will get to this as soon as I can. In the meantime, is there an option to convert the files to a different saving style? For example, instead of a compound type can you use 3 different variables, and instead of a fixed-size array can you use the (more common) variable-length arrays?

[timholy]

We're halfway there, see 5679103.

[kleinhenz]

closed by #592.