Duckietown Challenges Home Challenges Submissions

Evaluator 4883

ID4883
evaluatornogpu-prod-04
ownerI don't have one 😀
machinenogpu-prod_af4c8b961571
processnogpu-prod-04_af4c8b961571
version6.2.5
first heard
last heard
statusinactive
# evaluating
# success18 53908
# timeout2 56493
# failed
# error1 54087
# aborted3 57179
# host-error39 54304
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
5757410891Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simtimeoutyesnogpu-prod-04----No reset possible
5755510933Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simabortedyesnogpu-prod-040:03:15
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5739910942Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-040:42:26
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.0
survival_time_median59.99999999999873
deviation-center-line_median1.2422730096440104
in-drivable-lane_median0.0


other stats
agent_compute-ego0_max0.013062841389995134
agent_compute-ego0_mean0.012267349562378946
agent_compute-ego0_median0.012303291510582764
agent_compute-ego0_min0.011399973838355122
complete-iteration_max0.3459626064014673
complete-iteration_mean0.2960075480455562
complete-iteration_median0.2918289627858145
complete-iteration_min0.2544096602091285
deviation-center-line_max4.053503393024394
deviation-center-line_mean1.731069028434027
deviation-center-line_min0.386226701423694
deviation-heading_max27.859809596736422
deviation-heading_mean14.925250437835436
deviation-heading_median14.309269950178932
deviation-heading_min3.22265225424745
driven_any_max2.6645352591003757e-13
driven_any_mean1.9984014443252818e-13
driven_any_median2.6645352591003757e-13
driven_any_min0.0
driven_lanedir_consec_max0.000286102294921875
driven_lanedir_consec_mean7.152557373046875e-05
driven_lanedir_consec_min0.0
driven_lanedir_max0.000286102294921875
driven_lanedir_mean7.152557373046875e-05
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max1.49721309207659e-06
get_duckie_state_mean1.4112057237204268e-06
get_duckie_state_median1.418302696412251e-06
get_duckie_state_min1.3110044099806152e-06
get_robot_state_max0.003871170507680367
get_robot_state_mean0.003717429532695074
get_robot_state_median0.003708521491978984
get_robot_state_min0.0035815046391419626
get_state_dump_max0.00477945258674971
get_state_dump_mean0.0046977081167807085
get_state_dump_median0.004699231782225546
get_state_dump_min0.004612916315922034
get_ui_image_max0.03544453042035893
get_ui_image_mean0.030744707802749495
get_ui_image_median0.029921524034352425
get_ui_image_min0.027691252721934196
in-drivable-lane_max0.0
in-drivable-lane_mean0.0
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.027691252721934196, "step_physics": 0.2068831745929067, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004657848391505105, "get_robot_state": 0.0035815046391419626, "sim_render-ego0": 0.0037496389695547898, "get_duckie_state": 1.449569079600008e-06, "in-drivable-lane": 0.0, "deviation-heading": 22.66279310353771, "agent_compute-ego0": 0.011399973838355122, "complete-iteration": 0.2704767609119018, "set_robot_commands": 0.002179997052677863, "deviation-center-line": 4.053503393024394, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008337911022989876, "sim_compute_performance-ego0": 0.0019153296004524832}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.0, "get_ui_image": 0.03544453042035893, "step_physics": 0.2693547219856891, "survival_time": 59.99999999999873, "driven_lanedir": 0.000286102294921875, "get_state_dump": 0.00477945258674971, "get_robot_state": 0.0036779252019750385, "sim_render-ego0": 0.0038920499005980734, "get_duckie_state": 1.3110044099806152e-06, "in-drivable-lane": 0.0, "deviation-heading": 27.859809596736422, "agent_compute-ego0": 0.01295041482117055, "complete-iteration": 0.3459626064014673, "set_robot_commands": 0.0021968979720370556, "deviation-center-line": 1.0457540566888746, "driven_lanedir_consec": 0.000286102294921875, "sim_compute_sim_state": 0.011596825994321647, "sim_compute_performance-ego0": 0.0019881217108479545}, "LF-norm-techtrack-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.03213658479727079, "step_physics": 0.24376661870799196, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004612916315922034, "get_robot_state": 0.00373911778198293, "sim_render-ego0": 0.003895538236378234, "get_duckie_state": 1.49721309207659e-06, "in-drivable-lane": 0.0, "deviation-heading": 3.22265225424745, "agent_compute-ego0": 0.01165616819999498, "complete-iteration": 0.31318116465972723, "set_robot_commands": 0.0021433268458916684, "deviation-center-line": 1.4387919625991463, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.009080597998994673, "sim_compute_performance-ego0": 0.0020583890856156043}, "LF-norm-small_loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.027706463271434063, "step_physics": 0.19046693618450436, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004740615172945987, "get_robot_state": 0.003871170507680367, "sim_render-ego0": 0.003963057742726297, "get_duckie_state": 1.387036313224494e-06, "in-drivable-lane": 0.0, "deviation-heading": 5.955746796820156, "agent_compute-ego0": 0.013062841389995134, "complete-iteration": 0.2544096602091285, "set_robot_commands": 0.0022503638843215574, "deviation-center-line": 0.386226701423694, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.006249781750719514, "sim_compute_performance-ego0": 0.0020128650331775115}}
set_robot_commands_max0.0022503638843215574
set_robot_commands_mean0.0021926464387320363
set_robot_commands_median0.002188447512357459
set_robot_commands_min0.0021433268458916684
sim_compute_performance-ego0_max0.0020583890856156043
sim_compute_performance-ego0_mean0.0019936763575233887
sim_compute_performance-ego0_median0.002000493372012733
sim_compute_performance-ego0_min0.0019153296004524832
sim_compute_sim_state_max0.011596825994321647
sim_compute_sim_state_mean0.008816279191756426
sim_compute_sim_state_median0.008709254510992274
sim_compute_sim_state_min0.006249781750719514
sim_render-ego0_max0.003963057742726297
sim_render-ego0_mean0.003875071212314348
sim_render-ego0_median0.0038937940684881534
sim_render-ego0_min0.0037496389695547898
simulation-passed1
step_physics_max0.2693547219856891
step_physics_mean0.22761786286777305
step_physics_median0.22532489665044933
step_physics_min0.19046693618450436
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
5737311010Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simabortedyesnogpu-prod-040:02:47
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5728010980Fernanda Custodio Pereira do Carmo 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-040:21:21
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median4.654618826563127
survival_time_median38.04999999999978
deviation-center-line_median1.3962319994832029
in-drivable-lane_median12.874999999999645


other stats
agent_compute-ego0_max0.040835550439502544
agent_compute-ego0_mean0.03724212952602486
agent_compute-ego0_median0.038999604554279046
agent_compute-ego0_min0.030133758556038812
complete-iteration_max0.2139413762189116
complete-iteration_mean0.188871919143701
complete-iteration_median0.1839080326663501
complete-iteration_min0.17373023502319213
deviation-center-line_max3.700738831343764
deviation-center-line_mean1.8162354212065897
deviation-center-line_min0.7717388545161897
deviation-heading_max7.771859218641016
deviation-heading_mean3.893281181120575
deviation-heading_median3.1635858940602812
deviation-heading_min1.4740937177207212
driven_any_max11.330327077043858
driven_any_mean6.886646690918119
driven_any_median7.038027771084406
driven_any_min2.140204144459806
driven_lanedir_consec_max6.9609594130439305
driven_lanedir_consec_mean4.494687689535361
driven_lanedir_consec_min1.7085536919712574
driven_lanedir_max6.9609594130439305
driven_lanedir_mean4.494687689535361
driven_lanedir_median4.654618826563127
driven_lanedir_min1.7085536919712574
get_duckie_state_max1.3999398999369016e-06
get_duckie_state_mean1.2555295889408078e-06
get_duckie_state_median1.2594446208016549e-06
get_duckie_state_min1.10328921422302e-06
get_robot_state_max0.003816755089930551
get_robot_state_mean0.003560721255722305
get_robot_state_median0.003528258358966525
get_robot_state_min0.0033696132150256197
get_state_dump_max0.004844008635521729
get_state_dump_mean0.004582507808371453
get_state_dump_median0.004585177886873407
get_state_dump_min0.0043156668242172675
get_ui_image_max0.03213575108331225
get_ui_image_mean0.028897052380048836
get_ui_image_median0.029003871051030698
get_ui_image_min0.025444716334821708
in-drivable-lane_max21.24999999999946
in-drivable-lane_mean12.449999999999696
in-drivable-lane_min2.8000000000000353
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 4.812496026221973, "get_ui_image": 0.02634181940194332, "step_physics": 0.08481536534699527, "survival_time": 26.35000000000024, "driven_lanedir": 3.987910884151872, "get_state_dump": 0.00436451489275152, "get_robot_state": 0.003427836931113041, "sim_render-ego0": 0.003649142655459317, "get_duckie_state": 1.240860332142223e-06, "in-drivable-lane": 4.90000000000007, "deviation-heading": 1.6468742533219165, "agent_compute-ego0": 0.038178570342786385, "complete-iteration": 0.17451408369974655, "set_robot_commands": 0.0020791504419211187, "deviation-center-line": 0.7717388545161897, "driven_lanedir_consec": 3.987910884151872, "sim_compute_sim_state": 0.009726865725083784, "sim_compute_performance-ego0": 0.0018523452859936335}, "LF-norm-zigzag-000-ego0": {"driven_any": 2.140204144459806, "get_ui_image": 0.03213575108331225, "step_physics": 0.11592874642808428, "survival_time": 12.30000000000004, "driven_lanedir": 1.7085536919712574, "get_state_dump": 0.0043156668242172675, "get_robot_state": 0.0033696132150256197, "sim_render-ego0": 0.0036053966414107967, "get_duckie_state": 1.10328921422302e-06, "in-drivable-lane": 2.8000000000000353, "deviation-heading": 1.4740937177207212, "agent_compute-ego0": 0.040835550439502544, "complete-iteration": 0.2139413762189116, "set_robot_commands": 0.0020203078806641612, "deviation-center-line": 0.8808883417988647, "driven_lanedir_consec": 1.7085536919712574, "sim_compute_sim_state": 0.009884600697258706, "sim_compute_performance-ego0": 0.001767895964958407}, "LF-norm-techtrack-000-ego0": {"driven_any": 11.330327077043858, "get_ui_image": 0.03166592270011807, "step_physics": 0.10239240291414411, "survival_time": 59.99999999999873, "driven_lanedir": 6.9609594130439305, "get_state_dump": 0.004844008635521729, "get_robot_state": 0.003816755089930551, "sim_render-ego0": 0.004032407374703616, "get_duckie_state": 1.3999398999369016e-06, "in-drivable-lane": 21.24999999999946, "deviation-heading": 7.771859218641016, "agent_compute-ego0": 0.030133758556038812, "complete-iteration": 0.19330198163295367, "set_robot_commands": 0.0023272351162518987, "deviation-center-line": 3.700738831343764, "driven_lanedir_consec": 6.9609594130439305, "sim_compute_sim_state": 0.01193663917115884, "sim_compute_performance-ego0": 0.0020585923667355045}, "LF-norm-small_loop-000-ego0": {"driven_any": 9.263559515946838, "get_ui_image": 0.025444716334821708, "step_physics": 0.08607032141053533, "survival_time": 49.74999999999931, "driven_lanedir": 5.321326768974384, "get_state_dump": 0.0048058408809952945, "get_robot_state": 0.0036286797868200094, "sim_render-ego0": 0.0038131463479804226, "get_duckie_state": 1.2780289094610864e-06, "in-drivable-lane": 20.84999999999922, "deviation-heading": 4.680297534798646, "agent_compute-ego0": 0.03982063876577171, "complete-iteration": 0.17373023502319213, "set_robot_commands": 0.0022577454287364303, "deviation-center-line": 1.911575657167541, "driven_lanedir_consec": 5.321326768974384, "sim_compute_sim_state": 0.005875195844106406, "sim_compute_performance-ego0": 0.00193348395297805}}
set_robot_commands_max0.0023272351162518987
set_robot_commands_mean0.0021711097168934023
set_robot_commands_median0.0021684479353287745
set_robot_commands_min0.0020203078806641612
sim_compute_performance-ego0_max0.0020585923667355045
sim_compute_performance-ego0_mean0.0019030793926663988
sim_compute_performance-ego0_median0.0018929146194858416
sim_compute_performance-ego0_min0.001767895964958407
sim_compute_sim_state_max0.01193663917115884
sim_compute_sim_state_mean0.009355825359401932
sim_compute_sim_state_median0.009805733211171244
sim_compute_sim_state_min0.005875195844106406
sim_render-ego0_max0.004032407374703616
sim_render-ego0_mean0.003775023254888538
sim_render-ego0_median0.00373114450171987
sim_render-ego0_min0.0036053966414107967
simulation-passed1
step_physics_max0.11592874642808428
step_physics_mean0.09730170902493974
step_physics_median0.09423136216233972
step_physics_min0.08481536534699527
survival_time_max59.99999999999873
survival_time_mean37.09999999999958
survival_time_min12.30000000000004
No reset possible
5719810994Mo Kleit 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-040:23:16
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median1.7516836624689902
survival_time_median39.224999999999426
deviation-center-line_median1.1374611116559985
in-drivable-lane_median9.60000000000012


other stats
agent_compute-ego0_max0.013111895943289676
agent_compute-ego0_mean0.012792874708905964
agent_compute-ego0_median0.012842292888873223
agent_compute-ego0_min0.012375017114587732
complete-iteration_max0.1884661284612676
complete-iteration_mean0.17024317088622176
complete-iteration_median0.16820816285307932
complete-iteration_min0.15609022937746073
deviation-center-line_max4.408665267074139
deviation-center-line_mean1.8253448791423856
deviation-center-line_min0.6177920261834066
deviation-heading_max10.73000726254849
deviation-heading_mean4.39934653655751
deviation-heading_median2.613259236683663
deviation-heading_min1.640860410314223
driven_any_max6.618033478166794
driven_any_mean3.732259003446574
driven_any_median3.508242416385705
driven_any_min1.2945177028480936
driven_lanedir_consec_max6.297743214185447
driven_lanedir_consec_mean2.657459158647139
driven_lanedir_consec_min0.8287260954651285
driven_lanedir_max6.297743214185447
driven_lanedir_mean2.657459158647139
driven_lanedir_median1.7516836624689902
driven_lanedir_min0.8287260954651285
get_duckie_state_max1.39215391141007e-06
get_duckie_state_mean1.3339054993782906e-06
get_duckie_state_median1.3368775768010906e-06
get_duckie_state_min1.2697129325009107e-06
get_robot_state_max0.003731898920025853
get_robot_state_mean0.0036394595275333814
get_robot_state_median0.0036443654324768
get_robot_state_min0.0035372083251540725
get_state_dump_max0.0047113760036624945
get_state_dump_mean0.004569157656652017
get_state_dump_median0.004559582516513796
get_state_dump_min0.004446089589917982
get_ui_image_max0.03464634293422016
get_ui_image_mean0.029687782652702197
get_ui_image_median0.02898846785220682
get_ui_image_min0.026127851972174983
in-drivable-lane_max37.64999999999854
in-drivable-lane_mean14.362499999999688
in-drivable-lane_min0.5999999999999659
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 1.7216128615304926, "get_ui_image": 0.026428149841927193, "step_physics": 0.09355018783260036, "survival_time": 18.450000000000127, "driven_lanedir": 0.884215966895221, "get_state_dump": 0.004446089589917982, "get_robot_state": 0.0035372083251540725, "sim_render-ego0": 0.003700459325635755, "get_duckie_state": 1.3248340503589527e-06, "in-drivable-lane": 12.20000000000014, "deviation-heading": 1.640860410314223, "agent_compute-ego0": 0.012375017114587732, "complete-iteration": 0.15695315953847525, "set_robot_commands": 0.002089231723063701, "deviation-center-line": 0.6630741433797039, "driven_lanedir_consec": 0.884215966895221, "sim_compute_sim_state": 0.00888031108959301, "sim_compute_performance-ego0": 0.0018608164142917944}, "LF-norm-zigzag-000-ego0": {"driven_any": 6.618033478166794, "get_ui_image": 0.03464634293422016, "step_physics": 0.11278683299525988, "survival_time": 59.99999999999873, "driven_lanedir": 6.297743214185447, "get_state_dump": 0.004530538825766431, "get_robot_state": 0.003613785244245315, "sim_render-ego0": 0.003893471875853781, "get_duckie_state": 1.2697129325009107e-06, "in-drivable-lane": 0.5999999999999659, "deviation-heading": 10.73000726254849, "agent_compute-ego0": 0.012824751753096376, "complete-iteration": 0.1884661284612676, "set_robot_commands": 0.002177693266157902, "deviation-center-line": 4.408665267074139, "driven_lanedir_consec": 6.297743214185447, "sim_compute_sim_state": 0.011947048792334818, "sim_compute_performance-ego0": 0.001956574823536742}, "LF-norm-techtrack-000-ego0": {"driven_any": 1.2945177028480936, "get_ui_image": 0.031548785862486446, "step_physics": 0.10918433658708156, "survival_time": 15.80000000000009, "driven_lanedir": 0.8287260954651285, "get_state_dump": 0.0047113760036624945, "get_robot_state": 0.003674945620708285, "sim_render-ego0": 0.003932422643956325, "get_duckie_state": 1.39215391141007e-06, "in-drivable-lane": 7.0000000000000995, "deviation-heading": 1.7899645865531544, "agent_compute-ego0": 0.013111895943289676, "complete-iteration": 0.1794631661676834, "set_robot_commands": 0.0022171739524095217, "deviation-center-line": 0.6177920261834066, "driven_lanedir_consec": 0.8287260954651285, "sim_compute_sim_state": 0.009005117115538202, "sim_compute_performance-ego0": 0.0019824008460300577}, "LF-norm-small_loop-000-ego0": {"driven_any": 5.294871971240918, "get_ui_image": 0.026127851972174983, "step_physics": 0.09422843164448734, "survival_time": 59.99999999999873, "driven_lanedir": 2.619151358042759, "get_state_dump": 0.004588626207261161, "get_robot_state": 0.003731898920025853, "sim_render-ego0": 0.00389885485519676, "get_duckie_state": 1.3489211032432284e-06, "in-drivable-lane": 37.64999999999854, "deviation-heading": 3.4365538868141723, "agent_compute-ego0": 0.012859834024650072, "complete-iteration": 0.15609022937746073, "set_robot_commands": 0.002299629182839374, "deviation-center-line": 1.6118480799322925, "driven_lanedir_consec": 2.619151358042759, "sim_compute_sim_state": 0.0062374506862237, "sim_compute_performance-ego0": 0.002026560900114061}}
set_robot_commands_max0.002299629182839374
set_robot_commands_mean0.0021959320311176244
set_robot_commands_median0.0021974336092837115
set_robot_commands_min0.002089231723063701
sim_compute_performance-ego0_max0.002026560900114061
sim_compute_performance-ego0_mean0.001956588245993164
sim_compute_performance-ego0_median0.0019694878347834
sim_compute_performance-ego0_min0.0018608164142917944
sim_compute_sim_state_max0.011947048792334818
sim_compute_sim_state_mean0.009017481920922433
sim_compute_sim_state_median0.008942714102565606
sim_compute_sim_state_min0.0062374506862237
sim_render-ego0_max0.003932422643956325
sim_render-ego0_mean0.003856302175160656
sim_render-ego0_median0.0038961633655252704
sim_render-ego0_min0.003700459325635755
simulation-passed1
step_physics_max0.11278683299525988
step_physics_mean0.10243744726485728
step_physics_median0.10170638411578443
step_physics_min0.09355018783260036
survival_time_max59.99999999999873
survival_time_mean38.56249999999942
survival_time_min15.80000000000009
No reset possible
5717911012Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simabortedyesnogpu-prod-040:02:58
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5715611313Moustafa Elarabitemplate-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simhost-erroryesnogpu-prod-040:03:51
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5698611047Bhavya Patwa 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-040:28:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.384268564004464
survival_time_median56.84999999999891
deviation-center-line_median1.6714619089412763
in-drivable-lane_median13.099999999999785


other stats
agent_compute-ego0_max0.01276808073354322
agent_compute-ego0_mean0.012352274743181155
agent_compute-ego0_median0.01242892172414
agent_compute-ego0_min0.011783174790901407
complete-iteration_max0.197470470251039
complete-iteration_mean0.18582743766362217
complete-iteration_median0.1898849344433323
complete-iteration_min0.16606941151678511
deviation-center-line_max2.9837212370469395
deviation-center-line_mean1.698670008439766
deviation-center-line_min0.46803497882957223
deviation-heading_max13.362777940549266
deviation-heading_mean9.2835351537956
deviation-heading_median9.947838569369464
deviation-heading_min3.875685535894214
driven_any_max6.296564739392844
driven_any_mean4.38183524644512
driven_any_median4.909881723516703
driven_any_min1.411012799354233
driven_lanedir_consec_max5.952690535180785
driven_lanedir_consec_mean2.809817067748705
driven_lanedir_consec_min0.5180406078051067
driven_lanedir_max5.952690535180785
driven_lanedir_mean2.809817067748705
driven_lanedir_median2.384268564004464
driven_lanedir_min0.5180406078051067
get_duckie_state_max2.1269155103106833e-06
get_duckie_state_mean2.005173451164561e-06
get_duckie_state_median2.010882818727596e-06
get_duckie_state_min1.8720126568923685e-06
get_robot_state_max0.0037508445514750546
get_robot_state_mean0.003662071833514475
get_robot_state_median0.003669155191041428
get_robot_state_min0.0035591324004999904
get_state_dump_max0.004772864718769872
get_state_dump_mean0.0046202441382287305
get_state_dump_median0.004643485602330079
get_state_dump_min0.00442114062948489
get_ui_image_max0.03401057448216421
get_ui_image_mean0.029700189531614907
get_ui_image_median0.029441600686646557
get_ui_image_min0.025906982271002296
in-drivable-lane_max37.19999999999885
in-drivable-lane_mean16.07499999999961
in-drivable-lane_min0.9000000000000128
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 1.411012799354233, "get_ui_image": 0.027135266056009657, "step_physics": 0.1188898847186214, "survival_time": 18.60000000000013, "driven_lanedir": 0.5180406078051067, "get_state_dump": 0.0046111489109315435, "get_robot_state": 0.0037508445514750546, "sim_render-ego0": 0.003863670231509784, "get_duckie_state": 1.916297319427572e-06, "in-drivable-lane": 10.250000000000146, "deviation-heading": 3.875685535894214, "agent_compute-ego0": 0.011783174790901407, "complete-iteration": 0.18287181406813716, "set_robot_commands": 0.002168989692872076, "deviation-center-line": 0.46803497882957223, "driven_lanedir_consec": 0.5180406078051067, "sim_compute_sim_state": 0.008535560909608733, "sim_compute_performance-ego0": 0.002048341582351971}, "LF-norm-zigzag-000-ego0": {"driven_any": 6.296564739392844, "get_ui_image": 0.03401057448216421, "step_physics": 0.12352649992848316, "survival_time": 59.99999999999873, "driven_lanedir": 5.952690535180785, "get_state_dump": 0.00442114062948489, "get_robot_state": 0.0035591324004999904, "sim_render-ego0": 0.003712457582218065, "get_duckie_state": 1.8720126568923685e-06, "in-drivable-lane": 0.9000000000000128, "deviation-heading": 13.362777940549266, "agent_compute-ego0": 0.012133659272269346, "complete-iteration": 0.19689805481852737, "set_robot_commands": 0.002105607875280833, "deviation-center-line": 2.9837212370469395, "driven_lanedir_consec": 5.952690535180785, "sim_compute_sim_state": 0.011419623618717496, "sim_compute_performance-ego0": 0.0019259923304447424}, "LF-norm-techtrack-000-ego0": {"driven_any": 4.883083313052955, "get_ui_image": 0.031747935317283454, "step_physics": 0.12546417502469795, "survival_time": 53.699999999999086, "driven_lanedir": 3.1351823069814695, "get_state_dump": 0.004772864718769872, "get_robot_state": 0.0036142054269480152, "sim_render-ego0": 0.0038959946743277614, "get_duckie_state": 2.1269155103106833e-06, "in-drivable-lane": 15.94999999999942, "deviation-heading": 12.37852608133326, "agent_compute-ego0": 0.01276808073354322, "complete-iteration": 0.197470470251039, "set_robot_commands": 0.0021721664694852607, "deviation-center-line": 1.728179804650386, "driven_lanedir_consec": 3.1351823069814695, "sim_compute_sim_state": 0.010900196696436682, "sim_compute_performance-ego0": 0.002039274393126022}, "LF-norm-small_loop-000-ego0": {"driven_any": 4.93668013398045, "get_ui_image": 0.025906982271002296, "step_physics": 0.10471381890982216, "survival_time": 59.99999999999873, "driven_lanedir": 1.633354821027459, "get_state_dump": 0.004675822293728615, "get_robot_state": 0.00372410495513484, "sim_render-ego0": 0.003900774114832691, "get_duckie_state": 2.10546831802762e-06, "in-drivable-lane": 37.19999999999885, "deviation-heading": 7.517151057405664, "agent_compute-ego0": 0.012724184176010652, "complete-iteration": 0.16606941151678511, "set_robot_commands": 0.002210407828808228, "deviation-center-line": 1.6147440132321669, "driven_lanedir_consec": 1.633354821027459, "sim_compute_sim_state": 0.006126485597481835, "sim_compute_performance-ego0": 0.001999465154668473}}
set_robot_commands_max0.002210407828808228
set_robot_commands_mean0.0021642929666115993
set_robot_commands_median0.002170578081178668
set_robot_commands_min0.002105607875280833
sim_compute_performance-ego0_max0.002048341582351971
sim_compute_performance-ego0_mean0.002003268365147802
sim_compute_performance-ego0_median0.0020193697738972475
sim_compute_performance-ego0_min0.0019259923304447424
sim_compute_sim_state_max0.011419623618717496
sim_compute_sim_state_mean0.009245466705561188
sim_compute_sim_state_median0.009717878803022708
sim_compute_sim_state_min0.006126485597481835
sim_render-ego0_max0.003900774114832691
sim_render-ego0_mean0.003843224150722075
sim_render-ego0_median0.003879832452918773
sim_render-ego0_min0.003712457582218065
simulation-passed1
step_physics_max0.12546417502469795
step_physics_mean0.11814859464540615
step_physics_median0.12120819232355227
step_physics_min0.10471381890982216
survival_time_max59.99999999999873
survival_time_mean48.07499999999917
survival_time_min18.60000000000013
No reset possible
5696811312Moustafa Elarabitemplate-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-040:02:34
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5685711073Melisande Tengexercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-040:23:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median3.7046101482252727
survival_time_median45.47499999999952
deviation-center-line_median2.0467817105933737
in-drivable-lane_median14.99999999999951


other stats
agent_compute-ego0_max0.013280780388850355
agent_compute-ego0_mean0.012374875392667689
agent_compute-ego0_median0.012302337504054706
agent_compute-ego0_min0.011614046173710976
complete-iteration_max0.21882140646369927
complete-iteration_mean0.18141308597672692
complete-iteration_median0.17362587486857878
complete-iteration_min0.1595791877060508
deviation-center-line_max2.8503203702911883
deviation-center-line_mean1.7911237651038994
deviation-center-line_min0.220611268937662
deviation-heading_max16.951523277025846
deviation-heading_mean10.192268270417102
deviation-heading_median11.15565174238166
deviation-heading_min1.506246319879238
driven_any_max10.894633376328082
driven_any_mean7.008654685307506
driven_any_median7.807017699841321
driven_any_min1.5259499652193025
driven_lanedir_consec_max6.314579158652464
driven_lanedir_consec_mean3.597717657108941
driven_lanedir_consec_min0.667071173332755
driven_lanedir_max6.668328711269626
driven_lanedir_mean3.927031522619873
driven_lanedir_median4.17236056429863
driven_lanedir_min0.695076250612604
get_duckie_state_max1.60445972364776e-06
get_duckie_state_mean1.384666253616572e-06
get_duckie_state_median1.3435611518396129e-06
get_duckie_state_min1.247082987139302e-06
get_robot_state_max0.003803282367939851
get_robot_state_mean0.0035471933397609964
get_robot_state_median0.0035609341382384797
get_robot_state_min0.0032636227146271737
get_state_dump_max0.004718815258003889
get_state_dump_mean0.0044155002551075715
get_state_dump_median0.004426997824248694
get_state_dump_min0.00408919011392901
get_ui_image_max0.03594682675217907
get_ui_image_mean0.029089482522640386
get_ui_image_median0.02745594722884042
get_ui_image_min0.02549920888070163
in-drivable-lane_max25.599999999999156
in-drivable-lane_mean15.162499999999548
in-drivable-lane_min5.050000000000012
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 10.894633376328082, "get_ui_image": 0.02549920888070163, "step_physics": 0.09836832232320436, "survival_time": 59.99999999999873, "driven_lanedir": 6.668328711269626, "get_state_dump": 0.004176122163555009, "get_robot_state": 0.0033628702362213008, "sim_render-ego0": 0.0035657771521067244, "get_duckie_state": 1.2494642271983633e-06, "in-drivable-lane": 21.549999999998928, "deviation-heading": 11.905099240825722, "agent_compute-ego0": 0.011884493990603533, "complete-iteration": 0.1595791877060508, "set_robot_commands": 0.0019892874010198815, "deviation-center-line": 2.1859952585171594, "driven_lanedir_consec": 6.314579158652464, "sim_compute_sim_state": 0.008826115248503038, "sim_compute_performance-ego0": 0.0018261330610111692}, "LF-norm-zigzag-000-ego0": {"driven_any": 10.402723174692696, "get_ui_image": 0.03594682675217907, "step_physics": 0.14107673610874655, "survival_time": 59.99999999999873, "driven_lanedir": 5.022014577556571, "get_state_dump": 0.004718815258003889, "get_robot_state": 0.0037589980402556586, "sim_render-ego0": 0.004009860838382667, "get_duckie_state": 1.4376580764808625e-06, "in-drivable-lane": 25.599999999999156, "deviation-heading": 16.951523277025846, "agent_compute-ego0": 0.013280780388850355, "complete-iteration": 0.21882140646369927, "set_robot_commands": 0.0022870777647858555, "deviation-center-line": 2.8503203702911883, "driven_lanedir_consec": 4.131514167534067, "sim_compute_sim_state": 0.01156750547201012, "sim_compute_performance-ego0": 0.002082776467468618}, "LF-norm-techtrack-000-ego0": {"driven_any": 5.211312224989946, "get_ui_image": 0.02841758728027344, "step_physics": 0.11425096258040396, "survival_time": 30.950000000000305, "driven_lanedir": 3.3227065510406897, "get_state_dump": 0.00408919011392901, "get_robot_state": 0.0032636227146271737, "sim_render-ego0": 0.0034584087710226737, "get_duckie_state": 1.247082987139302e-06, "in-drivable-lane": 8.450000000000092, "deviation-heading": 10.4062042439376, "agent_compute-ego0": 0.011614046173710976, "complete-iteration": 0.17904838977321502, "set_robot_commands": 0.0018887085299338065, "deviation-center-line": 1.9075681626695875, "driven_lanedir_consec": 3.2777061289164786, "sim_compute_sim_state": 0.010273853425056705, "sim_compute_performance-ego0": 0.0017127610022021878}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.5259499652193025, "get_ui_image": 0.0264943071774074, "step_physics": 0.10648630103286431, "survival_time": 9.750000000000004, "driven_lanedir": 0.695076250612604, "get_state_dump": 0.004677873484942378, "get_robot_state": 0.003803282367939851, "sim_render-ego0": 0.004054619341480489, "get_duckie_state": 1.60445972364776e-06, "in-drivable-lane": 5.050000000000012, "deviation-heading": 1.506246319879238, "agent_compute-ego0": 0.01272018101750588, "complete-iteration": 0.16820335996394253, "set_robot_commands": 0.002348100652500075, "deviation-center-line": 0.220611268937662, "driven_lanedir_consec": 0.667071173332755, "sim_compute_sim_state": 0.005425251260095713, "sim_compute_performance-ego0": 0.0020935924685731226}}
set_robot_commands_max0.002348100652500075
set_robot_commands_mean0.0021282935870599044
set_robot_commands_median0.0021381825829028685
set_robot_commands_min0.0018887085299338065
sim_compute_performance-ego0_max0.0020935924685731226
sim_compute_performance-ego0_mean0.0019288157498137744
sim_compute_performance-ego0_median0.001954454764239893
sim_compute_performance-ego0_min0.0017127610022021878
sim_compute_sim_state_max0.01156750547201012
sim_compute_sim_state_mean0.009023181351416394
sim_compute_sim_state_median0.00954998433677987
sim_compute_sim_state_min0.005425251260095713
sim_render-ego0_max0.004054619341480489
sim_render-ego0_mean0.003772166525748138
sim_render-ego0_median0.003787818995244696
sim_render-ego0_min0.0034584087710226737
simulation-passed1
step_physics_max0.14107673610874655
step_physics_mean0.1150455805113048
step_physics_median0.11036863180663414
step_physics_min0.09836832232320436
survival_time_max59.99999999999873
survival_time_mean40.17499999999944
survival_time_min9.750000000000004
No reset possible
5669611092Melisande Tengexercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-040:26:50
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median3.5163090544043873
survival_time_median49.324999999999335
deviation-center-line_median2.260830047944907
in-drivable-lane_median12.174999999999793


other stats
agent_compute-ego0_max0.012734583797996976
agent_compute-ego0_mean0.012562457661128028
agent_compute-ego0_median0.012609018298916606
agent_compute-ego0_min0.012297210248681923
complete-iteration_max0.21428475946727032
complete-iteration_mean0.17946554407610715
complete-iteration_median0.1732812205990251
complete-iteration_min0.1570149756391082
deviation-center-line_max3.171786801030719
deviation-center-line_mean2.1455793172753093
deviation-center-line_min0.8888703721807046
deviation-heading_max15.175164341108848
deviation-heading_mean10.950989222431032
deviation-heading_median12.39953526808682
deviation-heading_min3.829722012441643
driven_any_max10.649531514443895
driven_any_mean8.117711556231505
driven_any_median8.447861580030835
driven_any_min4.925591550420453
driven_lanedir_consec_max6.909274380617803
driven_lanedir_consec_mean3.994840939613732
driven_lanedir_consec_min2.0374712690283507
driven_lanedir_max8.881616564734662
driven_lanedir_mean5.7177555082304306
driven_lanedir_median5.975967099579355
driven_lanedir_min2.0374712690283507
get_duckie_state_max2.135245825825484e-06
get_duckie_state_mean2.0244445175150736e-06
get_duckie_state_median2.0731150942774736e-06
get_duckie_state_min1.8163020556798645e-06
get_robot_state_max0.003646769988149727
get_robot_state_mean0.003620407356748673
get_robot_state_median0.003633818978896661
get_robot_state_min0.0035672214810516458
get_state_dump_max0.004610590848503803
get_state_dump_mean0.004510509292434127
get_state_dump_median0.004524003853905112
get_state_dump_min0.00438343861342248
get_ui_image_max0.03476066244357008
get_ui_image_mean0.02929463926297901
get_ui_image_median0.028580349821518705
get_ui_image_min0.025257194965308552
in-drivable-lane_max17.65000000000022
in-drivable-lane_mean12.174999999999857
in-drivable-lane_min6.699999999999619
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 10.615705283964576, "get_ui_image": 0.02769648026268647, "step_physics": 0.11094060507940312, "survival_time": 59.99999999999873, "driven_lanedir": 8.881616564734662, "get_state_dump": 0.004587305277015247, "get_robot_state": 0.003634068888490345, "sim_render-ego0": 0.003815587216074719, "get_duckie_state": 2.135245825825484e-06, "in-drivable-lane": 6.699999999999619, "deviation-heading": 15.021287745297885, "agent_compute-ego0": 0.012632582209489427, "complete-iteration": 0.17759125873905535, "set_robot_commands": 0.0021729876258589644, "deviation-center-line": 3.171786801030719, "driven_lanedir_consec": 6.909274380617803, "sim_compute_sim_state": 0.010055828650329234, "sim_compute_performance-ego0": 0.0019731158320850178}, "LF-norm-zigzag-000-ego0": {"driven_any": 6.280017876097097, "get_ui_image": 0.03476066244357008, "step_physics": 0.13934896158617596, "survival_time": 38.64999999999994, "driven_lanedir": 3.447668094325752, "get_state_dump": 0.004610590848503803, "get_robot_state": 0.0036335690693029754, "sim_render-ego0": 0.003858744019993824, "get_duckie_state": 2.035798952561016e-06, "in-drivable-lane": 15.399999999999718, "deviation-heading": 9.777782790875758, "agent_compute-ego0": 0.012734583797996976, "complete-iteration": 0.21428475946727032, "set_robot_commands": 0.00219653003899626, "deviation-center-line": 1.7486493960257792, "driven_lanedir_consec": 2.922854587051115, "sim_compute_sim_state": 0.011039803502479573, "sim_compute_performance-ego0": 0.002020471471840713}, "LF-norm-techtrack-000-ego0": {"driven_any": 4.925591550420453, "get_ui_image": 0.029464219380350945, "step_physics": 0.1008616655004063, "survival_time": 30.850000000000303, "driven_lanedir": 2.0374712690283507, "get_state_dump": 0.00438343861342248, "get_robot_state": 0.0035672214810516458, "sim_render-ego0": 0.003746547745269479, "get_duckie_state": 1.8163020556798645e-06, "in-drivable-lane": 17.65000000000022, "deviation-heading": 3.829722012441643, "agent_compute-ego0": 0.012585454388343789, "complete-iteration": 0.16897118245899484, "set_robot_commands": 0.0021160486060824595, "deviation-center-line": 0.8888703721807046, "driven_lanedir_consec": 2.0374712690283507, "sim_compute_sim_state": 0.01029242435319524, "sim_compute_performance-ego0": 0.001876415558231687}, "LF-norm-small_loop-000-ego0": {"driven_any": 10.649531514443895, "get_ui_image": 0.025257194965308552, "step_physics": 0.09774786189235717, "survival_time": 59.99999999999873, "driven_lanedir": 8.50426610483296, "get_state_dump": 0.004460702430794976, "get_robot_state": 0.003646769988149727, "sim_render-ego0": 0.003707149642194737, "get_duckie_state": 2.110431235993931e-06, "in-drivable-lane": 8.949999999999864, "deviation-heading": 15.175164341108848, "agent_compute-ego0": 0.012297210248681923, "complete-iteration": 0.1570149756391082, "set_robot_commands": 0.002106527206204912, "deviation-center-line": 2.7730106998640345, "driven_lanedir_consec": 4.109763521757659, "sim_compute_sim_state": 0.005820575105856102, "sim_compute_performance-ego0": 0.001883888721069031}}
set_robot_commands_max0.00219653003899626
set_robot_commands_mean0.0021480233692856487
set_robot_commands_median0.002144518115970712
set_robot_commands_min0.002106527206204912
sim_compute_performance-ego0_max0.002020471471840713
sim_compute_performance-ego0_mean0.001938472895806612
sim_compute_performance-ego0_median0.0019285022765770244
sim_compute_performance-ego0_min0.001876415558231687
sim_compute_sim_state_max0.011039803502479573
sim_compute_sim_state_mean0.009302157902965038
sim_compute_sim_state_median0.010174126501762238
sim_compute_sim_state_min0.005820575105856102
sim_render-ego0_max0.003858744019993824
sim_render-ego0_mean0.0037820071558831896
sim_render-ego0_median0.0037810674806720993
sim_render-ego0_min0.003707149642194737
simulation-passed1
step_physics_max0.13934896158617596
step_physics_mean0.11222477351458564
step_physics_median0.10590113528990472
step_physics_min0.09774786189235717
survival_time_max59.99999999999873
survival_time_mean47.374999999999424
survival_time_min30.850000000000303
No reset possible
5668911290Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:00:41
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5668711290Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:00:49
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5665611278Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:03:41
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5663411280Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:01:54
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5660811282Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:00:49
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5658911427Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simhost-erroryesnogpu-prod-040:02:00
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5649311290Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simtimeoutyesnogpu-prod-040:08:44
Timeout because eval [...]
Timeout because evaluator contacted us
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5645711281Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:01:51
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5641811284Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:02:28
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5637711285Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:01:06
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5632311287Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:03:03
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5630311283Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:00:42
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5623211425Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-040:03:40
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5616911286Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:02:43
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5611811310Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:01:14
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5610911310Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:00:43
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5606911296Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:01:57
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5606111296Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-040:00:43
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||