python - HDF5 possible data corruption or loss? -

-

on wikipedia 1 can read following criticism hdf5:

criticism of hdf5 follows monolithic design , lengthy specification. though 150-page open standard, there single c implementation of hdf5, meaning bindings share bugs , performance issues. compounded lack of journaling, documented bugs in current stable release capable of corrupting entire hdf5 databases. although 1.10-alpha adds journaling, backwards-incompatible previous versions. hdf5 not support utf-8 well, necessitating ascii in places. furthermore in latest draft, array data can never deleted.

i wondering if applying c implementation of hdf5 or if general flaw of hdf5?

i doing scientific experiments generate gigabytes of data , in cases @ least several hundred megabytes of data. data loss , corruption huge disadvantage me.

my scripts have python api, hence using h5py (version 2.5.0).

so, criticism relevant me , should concerned corrupted data?

declaration front: maintain h5py, have bias etc.

the wikipedia page has changed since question posted, here's see:

criticism

criticism of hdf5 follows monolithic design , lengthy specification.

  • though 150-page open standard, other c implementation of hdf5 hdf5 reader.
  • hdf5 not enforce use of utf-8, client applications may expecting ascii in places.
  • dataset data cannot freed in file without generating file copy using external tool (h5repack).

i'd pretty sums problems hdf5, it's complex (but people need complexity, see virtual dataset support), it's got long history backwards compatibly it's focus, , it's not designed allow massive changes in files. it's not best on windows (due how deals filenames).

i picked hdf5 research because of available options, had decent metadata support (hdf5 @ least allows utf-8, formats fits don't have that), support multidimensional arrays (which formats protocol buffers don't support), , supports more 64 bit floats (which rare).

i can't comment known bugs, have seen corruption (this happened when writing file , linux oom'd script). however, shouldn't concern long have proper data hygiene practices (as mentioned in hackernews link), in case not continuously write same file, each run create new file. should not modify file, instead data reduction should produce new files, , should backup originals.

finally, worth pointing out there alternatives hdf5, depending on requirements are: sql databases may fit needs better (and sqlite comes python default, it's easy experiment with), simple csv file. recommend against custom/non-portable formats (e.g. pickle , similar), they're neither more robust hdf5, , more complex csv file.


Comments

Post a Comment

Popular posts from this blog

java - Run spring boot application error: Cannot instantiate interface org.springframework.context.ApplicationListener -

-
i have spring project , try make use spring boot , run @ embadded tomcat following : https://spring.io/guides/gs/rest-service/ this application //@configuration //@enableaspectjautoproxy @springbootapplication @componentscan(basepackages = "gux.prome") public class application { public static void main(string[] args) { springapplication.run(application.class, args); } } if use maven command: mvn spring-boot:run , project starts fine, need debug , run main method @ intelij, exception occurs: exception in thread "main" java.lang.illegalargumentexception: cannot instantiate interface org.springframework.context.applicationlistener : org.springframework.boot.logging.classpathloggingapplicationlistener @ org.springframework.boot.springapplication.createspringfactoriesinstances(springapplication.java:414) @ org.springframework.boot.springapplication.getspringfactoriesinstances(springapplication.java:394) @ org.springframework.bo...
Read more

python - pip wont install .WHL files -

-
when try install .whl gohike typing cmd c:\users\owner\downloads>pip install ta_lib-0.4.9-cp27-none-win_amd64.whl i print out: `processing c:\users\owner\downloads\ta_lib-0.4.9-cp27-none-win_amd64.whl complete output command python setup.py egg_info: traceback <most recent call last>: file "<string>", line 1, in <module> ioerror: [errno 2] no such file or directory:'c:\\users\\owner\\appdata\\lo cal\\temp\\pip-9gwk2c-build\\setup.py' ----------------------------------------- command "python setup.py egg_info" failed error code 1 in c:\users\owner\ap pdata\local\temp\pip-9gwk2c-build\` the part between pip- , -build in pip-9gwk2c-build different on same file. i asked python 2.7.11 pip 8.0.3 can't install gohike whl of ta-lib 0.4.9 few days ago , decided ask differently. can me! you have outdated pip not understand wheels. wheels not supposed have setup.py inside. try upgrading pip pip in...
Read more

Excel VBA "Microsoft Windows Common Controls 6.0 (SP6)" Location Changes -

-
i have excel workbook used generate quotations customers. there main page user inputs data, , there buttons generate various pdfs , send them customer. however, 1 user consistently getting error preventing generating , sending pdfs. i beleive "microsoft windows common controls 6.0 (sp6)". it's installed on users' computers. on mine, location "c:\windows\syswow64\mscomctl.ocx" on others "c:\windows\system32\mscomctl.ocx" the user experiencing issues on 32-bit machine, "mscomctl.ocx" file should in system32. have manually put in there, , pointed common controls reference in vba it. issue seems keep reverting looking in syswow64. since folder doesn't exist, can't find it, , can't produce pdfs. i thought perhaps issue user running older version of office (2007) there users on 2013 , 2016. thought perhaps mixture of 64-bit , 32-bit users. user gets sent workbook other users (as quotation process goes further, gets passed ...
Read more