python - Numpy savez exceptions -

-

i found statement:

np.savez not multi-process safe, because uses gettempdir() + key + ".npy". so, if you're running same script on different data sets, roach own data when 1 process removes /tmp/arr_0.npy of process.

as luck have it, i'm in exact situation, have example error:

traceback (most recent call last):    file "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 99, in <module> main()     file "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 90, in  main  np.savez("%smask-t%0.2f" % (outfile,threshold), result)     file "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 230, in savez  zip.write(filename, arcname=fname)      file "/usr/lib/python2.5/zipfile.py", line 541, in write   st = os.stat(filename)  oserror: [errno 2] no such file or directory: '/tmp/arr_0.npy'

fortunately, replacing tempfile.gettempdir() tempfile.mkdtemp() , remembering clean @ end os.rmdir(direc) takes fix far can tell.

what mean , how can avoid this? numpy store data written .npz file in temporary files, potentially named other temporary .npz files in other scripts?!

i have scientific experiments run on several hours , save results via np.savez. destination path results saved savez different, every experiment has own result path. however, scripts located in same directory.

interestingly scripts run several hours before error occurs:

> traceback (most recent call last):   file > "/work/var/slurmd/state.node348.d/job20832/slurm_script", line 53, in > <module> >     e = exp6_06()   file > "/work/experiments/s06/cs06_ex06.py", line > 150, in __init__ >     self.__start()   file "/work/experiments/s06/cs06_ex06.py", line > 374, in __start >     file "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py", > line 600, in savez_compressed >     _savez(file, args, kwds, true)   file "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py", > line 630, in _savez >     fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')   file "/sw/env/openmpi/2q4/lib/python2.7/tempfile.py", > line 314, in mkstemp >     return _mkstemp_inner(dir, prefix, suffix, flags)   file "/sw/env/openmpi/2q4/lib/python2.7/tempfile.py", > line 244, in _mkstemp_inner >     fd = _os.open(file, flags, 0600) oserror: [errno 2] no such file or directory: > '/work/tmp/node001.admin.2016-03-06-152506.fx092.27432/tmp5uulez-numpy.npy'

used numpy version: 1.10.4

your error occurs in mkstemp. quote suggests using mkdtemp instead:

https://docs.python.org/2/library/tempfile.html

tempfile.mkstemp([suffix=''[, prefix='tmp'[, dir=none[, text=false]]]])

creates temporary file in secure manner possible. there no race conditions in file’s creation, assuming platform implements os.o_excl flag os.open(). file readable , writable creating user id. if platform uses permission bits indicate whether file executable, file executable no one. file descriptor not inherited child processes.

and

tempfile.mkdtemp([suffix=''[, prefix='tmp'[, dir=none]]])

creates temporary directory in secure manner possible. there no race conditions in directory’s creation. directory readable, writable, , searchable creating user id.

the user of mkdtemp() responsible deleting temporary directory , contents when done it.

the quoted error appears occur after temp files have been created, , being collected zip. error appears occur while creating 1 of temp files.

i haven't used these functions enough understand difference. may need study _savez in lib/npyio.py.

what os using? linux, mac, windows?

here's core of _savez, in lib/npyio.py file:

def _savez(file, args, kwds, compress):     ....     zip = zipfile_factory(file, mode="w", compression=compression)      # stage arrays in temporary file on disk, before writing zip. =>  fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')     os.close(fd)     try:         key, val in namedict.items():             fname = key + '.npy'             fid = open(tmpfile, 'wb')             try:                 format.write_array(fid, np.asanyarray(val))                 fid.close()                 fid = none =>              zip.write(tmpfile, arcname=fname)             finally:                 if fid:                     fid.close()     finally:         os.remove(tmpfile)     zip.close()

the quoted error occurs toward end of loop, when temp 'save' file added archive. error occurs early, when getting temp dir , temp file (fd). note discards open file handle, , uses name (repeatedly each array).

it's surprising temp directory should have disappeared during mkstemp function call. feels openmpi problem rather savez one.


Comments

Post a Comment

Popular posts from this blog

java - Run spring boot application error: Cannot instantiate interface org.springframework.context.ApplicationListener -

-
i have spring project , try make use spring boot , run @ embadded tomcat following : https://spring.io/guides/gs/rest-service/ this application //@configuration //@enableaspectjautoproxy @springbootapplication @componentscan(basepackages = "gux.prome") public class application { public static void main(string[] args) { springapplication.run(application.class, args); } } if use maven command: mvn spring-boot:run , project starts fine, need debug , run main method @ intelij, exception occurs: exception in thread "main" java.lang.illegalargumentexception: cannot instantiate interface org.springframework.context.applicationlistener : org.springframework.boot.logging.classpathloggingapplicationlistener @ org.springframework.boot.springapplication.createspringfactoriesinstances(springapplication.java:414) @ org.springframework.boot.springapplication.getspringfactoriesinstances(springapplication.java:394) @ org.springframework.bo...
Read more

reactjs - React router and this.props.children - how to pass state to this.props.children -

-
i'm using react-router first time , don't know how think in yet. here's how i'm loading components in nested routes. entry point .js reactdom.render( <router history={hashhistory} > <route path="/" component={app}> <route path="models" component={content}> </route> </router>, document.getelementbyid('app') ); app.js render: function() { return ( <div> <header /> {this.props.children} </div> ); } so child of app content component sent in. i'm using flux , app.js has state , listens changes, don't know how pass state down this.props.children. before using react-router app.js defines children explicitly, passing state natural don't see how now. using couple of react helper methods can add state, props , whatever else this.props.children render: function() { var children = react.ch...
Read more

Excel VBA "Microsoft Windows Common Controls 6.0 (SP6)" Location Changes -

-
i have excel workbook used generate quotations customers. there main page user inputs data, , there buttons generate various pdfs , send them customer. however, 1 user consistently getting error preventing generating , sending pdfs. i beleive "microsoft windows common controls 6.0 (sp6)". it's installed on users' computers. on mine, location "c:\windows\syswow64\mscomctl.ocx" on others "c:\windows\system32\mscomctl.ocx" the user experiencing issues on 32-bit machine, "mscomctl.ocx" file should in system32. have manually put in there, , pointed common controls reference in vba it. issue seems keep reverting looking in syswow64. since folder doesn't exist, can't find it, , can't produce pdfs. i thought perhaps issue user running older version of office (2007) there users on 2013 , 2016. thought perhaps mixture of 64-bit , 32-bit users. user gets sent workbook other users (as quotation process goes further, gets passed ...
Read more