Submission 13295

Operator message: 'Submissions tested 3 times, the same error appeared every time.'
Logs:
ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', 'solution.py']' command failed.  (See above for error)
2020-12-12 08:09:00.772756: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:
2020-12-12 08:09:00.772934: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:
2020-12-12 08:09:00.772948: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
DEBUG:commons:version: 6.1.7 *
INFO:typing:version: 6.1.8
DEBUG:aido_schemas:aido-protocols version 6.0.33 path /home/jbr/workspace/env/lib/python3.6/site-packages
INFO:nodes:version 6.1.1 path /home/jbr/workspace/env/lib/python3.6/site-packages pyparsing 2.4.7
2020-12-12 08:09:02.971270: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-12-12 08:09:02.988059: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2020-12-12 08:09:02.988357: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-12-12 08:09:02.990522: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-12-12 08:09:02.992403: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-12-12 08:09:02.992693: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-12-12 08:09:02.994568: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-12-12 08:09:02.995350: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-12-12 08:09:02.999902: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-12 08:09:03.001367: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-12-12 08:09:03.968869: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-12-12 08:09:03.995639: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599935000 Hz
2020-12-12 08:09:03.996936: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637c142ce00 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-12 08:09:03.996985: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-12 08:09:04.276036: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x5637c172b570 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
2020-12-12 08:09:04.276098: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-12-12 08:09:04.277602: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2020-12-12 08:09:04.277790: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
Checkpoint <tensorflow.python.training.tracking.util.CheckpointLoadStatus object at 0x7fcfd07d3860> is loaded successfully!

2020-12-12 08:09:04.277836: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-12-12 08:09:04.277870: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-12-12 08:09:04.277904: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-12-12 08:09:04.277941: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-12-12 08:09:04.277978: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-12-12 08:09:04.278016: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-12 08:09:04.283428: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-12-12 08:09:04.283524: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-12-12 08:09:04.285146: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-12 08:09:04.285196: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2020-12-12 08:09:04.285214: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 
2020-12-12 08:09:04.288043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 9022 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
/home/jbr/workspace/configuration.py:22: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  doc = yaml.load(stream)
DEBUG:ipce:version 6.0.36 path /home/jbr/workspace/env/lib/python3.6/site-packages
INFO:nodes_wrapper:checking implementation
INFO:nodes_wrapper:checking implementation OK
DEBUG:nodes_wrapper:run_loop
  fin: /fifos/ego0-in
 fout: fifo:/fifos/ego0-out
Traceback (most recent call last):
  File "solution.py", line 80, in <module>
    main()
  File "solution.py", line 76, in main
    wrap_direct(node=node, protocol=protocol)
  File "/home/jbr/workspace/env/lib/python3.6/site-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
    run_loop(node, protocol, args)
  File "/home/jbr/workspace/env/lib/python3.6/site-packages/zuper_nodes_wrapper/wrapper.py", line 234, in run_loop
    fo = open_for_write(fout)
  File "/home/jbr/workspace/env/lib/python3.6/site-packages/zuper_nodes_wrapper/streams.py", line 62, in open_for_write
    os.mkfifo(fout)
PermissionError: [Errno 13] Permission denied

ERROR conda.cli.main_run:execute(33): Subprocess for 'conda run ['python', 'solution.py']' command failed.  (See above for error)
2020-12-12 07:47:11.803237: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer.so.6'; dlerror: libnvinfer.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:
2020-12-12 07:47:11.803415: W tensorflow/stream_executor/platform/default/dso_loader.cc:55] Could not load dynamic library 'libnvinfer_plugin.so.6'; dlerror: libnvinfer_plugin.so.6: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:
2020-12-12 07:47:11.803433: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:30] Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
DEBUG:commons:version: 6.1.7 *
INFO:typing:version: 6.1.8
DEBUG:aido_schemas:aido-protocols version 6.0.33 path /home/jbr/workspace/env/lib/python3.6/site-packages
INFO:nodes:version 6.1.1 path /home/jbr/workspace/env/lib/python3.6/site-packages pyparsing 2.4.7
2020-12-12 07:47:14.002977: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
2020-12-12 07:47:14.020229: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2020-12-12 07:47:14.020486: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-12-12 07:47:14.022417: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-12-12 07:47:14.023991: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-12-12 07:47:14.024269: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-12-12 07:47:14.025942: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-12-12 07:47:14.026783: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-12-12 07:47:14.032808: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-12 07:47:14.034230: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-12-12 07:47:14.968942: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-12-12 07:47:14.980205: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2599935000 Hz
2020-12-12 07:47:14.981050: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e071633360 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-12 07:47:14.981081: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-12 07:47:15.184936: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x55e0717dfbe0 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
Checkpoint <tensorflow.python.training.tracking.util.CheckpointLoadStatus object at 0x7f0e9c194860> is loaded successfully!

2020-12-12 07:47:15.184998: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): GeForce GTX 1080 Ti, Compute Capability 6.1
2020-12-12 07:47:15.186788: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties: 
pciBusID: 0000:03:00.0 name: GeForce GTX 1080 Ti computeCapability: 6.1
coreClock: 1.582GHz coreCount: 28 deviceMemorySize: 10.91GiB deviceMemoryBandwidth: 451.17GiB/s
2020-12-12 07:47:15.186871: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-12-12 07:47:15.186907: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
2020-12-12 07:47:15.186940: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
2020-12-12 07:47:15.186969: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
2020-12-12 07:47:15.187000: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
2020-12-12 07:47:15.187048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
2020-12-12 07:47:15.187082: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
2020-12-12 07:47:15.192653: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0
2020-12-12 07:47:15.192721: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
2020-12-12 07:47:15.193675: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1096] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-12-12 07:47:15.193697: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1102]      0 
2020-12-12 07:47:15.193709: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1115] 0:   N 
2020-12-12 07:47:15.195377: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8900 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:03:00.0, compute capability: 6.1)
/home/jbr/workspace/configuration.py:22: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  doc = yaml.load(stream)
DEBUG:ipce:version 6.0.36 path /home/jbr/workspace/env/lib/python3.6/site-packages
INFO:nodes_wrapper:checking implementation
INFO:nodes_wrapper:checking implementation OK
DEBUG:nodes_wrapper:run_loop
  fin: /fifos/ego0-in
 fout: fifo:/fifos/ego0-out
Traceback (most recent call last):
  File "solution.py", line 80, in <module>
    main()
  File "solution.py", line 76, in main
    wrap_direct(node=node, protocol=protocol)
  File "/home/jbr/workspace/env/lib/python3.6/site-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
    run_loop(node, protocol, args)
  File "/home/jbr/workspace/env/lib/python3.6/site-packages/zuper_nodes_wrapper/wrapper.py", line 234, in run_loop
    fo = open_for_write(fout)
  File "/home/jbr/workspace/env/lib/python3.6/site-packages/zuper_nodes_wrapper/streams.py", line 62, in open_for_write
    os.mkfifo(fout)
PermissionError: [Errno 13] Permission denied

Job shutdown

Job shutdown