Duckietown Challenges Home Challenges Submissions

Job 62189

Job ID62189
submission13700
userFrank (Chude) Qian 🇨🇦
user labelBC Net V2
challengeaido-LF-sim-validation
step347
statushost-error
up to date Note that this job is not up to date; the challenge has been changed.
evaluatorreg02
date started
date completed
duration0:01:01
message
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     limit_gpu_memory()
              ||   File "solution.py", line 29, in limit_gpu_memory
              ||     logical_gpus = tf.config.experimental.list_logical_devices('GPU')
              ||   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
              ||     return context.context().list_logical_devices(device_type=device_type)
              ||   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
              ||     self.ensure_initialized()
              ||   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
              ||     context_handle = pywrap_tfe.TFE_NewContext(opts)
              || tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     limit_gpu_memory()
              || |   File "solution.py", line 29, in limit_gpu_memory
              || |     logical_gpus = tf.config.experimental.list_logical_devices('GPU')
              || |   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
              || |     return context.context().list_logical_devices(device_type=device_type)
              || |   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
              || |     self.ensure_initialized()
              || |   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
              || |     context_handle = pywrap_tfe.TFE_NewContext(opts)
              || | tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     limit_gpu_memory()
              ||   File "solution.py", line 29, in limit_gpu_memory
              ||     logical_gpus = tf.config.experimental.list_logical_devices('GPU')
              ||   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
              ||     return context.context().list_logical_devices(device_type=device_type)
              ||   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
              ||     self.ensure_initialized()
              ||   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
              ||     context_handle = pywrap_tfe.TFE_NewContext(opts)
              || tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 339, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     limit_gpu_memory()
              || |   File "solution.py", line 29, in limit_gpu_memory
              || |     logical_gpus = tf.config.experimental.list_logical_devices('GPU')
              || |   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/framework/config.py", line 439, in list_logical_devices
              || |     return context.context().list_logical_devices(device_type=device_type)
              || |   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 1368, in list_logical_devices
              || |     self.ensure_initialized()
              || |   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/eager/context.py", line 526, in ensure_initialized
              || |     context_handle = pywrap_tfe.TFE_NewContext(opts)
              || | tensorflow.python.framework.errors_impl.InternalError: cudaGetDevice() failed. Status: CUDA driver version is insufficient for CUDA runtime version
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

Highlights

Artifacts

The artifacts are hidden.

Container logs

The logs are hidden.