python - Numpy savez exceptions -

i found statement:

np.savez not multi-process safe, because uses gettempdir() + key + ".npy". so, if you're running same script on different data sets, roach own data when 1 process removes /tmp/arr_0.npy of process.

as luck have it, i'm in exact situation, have example error:

traceback (most recent call last):    file "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 99, in <module> main()     file "/home/pedro/prospectus/prelim/bbvs/bin/skldthresh", line 90, in  main  np.savez("%smask-t%0.2f" % (outfile,threshold), result)     file "/usr/lib/python2.5/site-packages/numpy/lib/io.py", line 230, in savez  zip.write(filename, arcname=fname)      file "/usr/lib/python2.5/zipfile.py", line 541, in write   st = os.stat(filename)  oserror: [errno 2] no such file or directory: '/tmp/arr_0.npy' 

fortunately, replacing tempfile.gettempdir() tempfile.mkdtemp() , remembering clean @ end os.rmdir(direc) takes fix far can tell.

what mean , how can avoid this? numpy store data written .npz file in temporary files, potentially named other temporary .npz files in other scripts?!

i have scientific experiments run on several hours , save results via np.savez. destination path results saved savez different, every experiment has own result path. however, scripts located in same directory.

interestingly scripts run several hours before error occurs:

> traceback (most recent call last):   file > "/work/var/slurmd/state.node348.d/job20832/slurm_script", line 53, in > <module> >     e = exp6_06()   file > "/work/experiments/s06/cs06_ex06.py", line > 150, in __init__ >     self.__start()   file "/work/experiments/s06/cs06_ex06.py", line > 374, in __start >     file "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py", > line 600, in savez_compressed >     _savez(file, args, kwds, true)   file "/home/fx092/.local/lib/python2.7/site-packages/numpy/lib/npyio.py", > line 630, in _savez >     fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')   file "/sw/env/openmpi/2q4/lib/python2.7/tempfile.py", > line 314, in mkstemp >     return _mkstemp_inner(dir, prefix, suffix, flags)   file "/sw/env/openmpi/2q4/lib/python2.7/tempfile.py", > line 244, in _mkstemp_inner >     fd = _os.open(file, flags, 0600) oserror: [errno 2] no such file or directory: > '/work/tmp/node001.admin.2016-03-06-152506.fx092.27432/tmp5uulez-numpy.npy' 

used numpy version: 1.10.4

your error occurs in mkstemp. quote suggests using mkdtemp instead:

https://docs.python.org/2/library/tempfile.html

tempfile.mkstemp([suffix=''[, prefix='tmp'[, dir=none[, text=false]]]])

creates temporary file in secure manner possible. there no race conditions in file’s creation, assuming platform implements os.o_excl flag os.open(). file readable , writable creating user id. if platform uses permission bits indicate whether file executable, file executable no one. file descriptor not inherited child processes.

and

tempfile.mkdtemp([suffix=''[, prefix='tmp'[, dir=none]]])

creates temporary directory in secure manner possible. there no race conditions in directory’s creation. directory readable, writable, , searchable creating user id.

the user of mkdtemp() responsible deleting temporary directory , contents when done it.

the quoted error appears occur after temp files have been created, , being collected zip. error appears occur while creating 1 of temp files.

i haven't used these functions enough understand difference. may need study _savez in lib/npyio.py.

what os using? linux, mac, windows?

---

here's core of _savez, in lib/npyio.py file:

def _savez(file, args, kwds, compress):     ....     zip = zipfile_factory(file, mode="w", compression=compression)      # stage arrays in temporary file on disk, before writing zip. =>  fd, tmpfile = tempfile.mkstemp(suffix='-numpy.npy')     os.close(fd)     try:         key, val in namedict.items():             fname = key + '.npy'             fid = open(tmpfile, 'wb')             try:                 format.write_array(fid, np.asanyarray(val))                 fid.close()                 fid = none =>              zip.write(tmpfile, arcname=fname)             finally:                 if fid:                     fid.close()     finally:         os.remove(tmpfile)     zip.close() 

the quoted error occurs toward end of loop, when temp 'save' file added archive. error occurs early, when getting temp dir , temp file (fd). note discards open file handle, , uses name (repeatedly each array).

it's surprising temp directory should have disappeared during mkstemp function call. feels openmpi problem rather savez one.