Duckietown Challenges Home Challenges Submissions

Evaluator 4897

ID4897
evaluatornogpu-prod-07
ownerI don't have one 😀
machinenogpu-prod_2b539ef57384
processnogpu-prod-07_2b539ef57384
version6.2.5
first heard
last heard
statusinactive
# evaluating
# success22 53952
# timeout1 57563
# failed2 54905
# error1 54250
# aborted1 57332
# host-error42 53938
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
5756311313Moustafa Elarabitemplate-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simtimeoutyesnogpu-prod-07----No reset possible
5749110912Dishank Bansal 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-070:23:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median3.1223850926988455
survival_time_median28.100000000000264
deviation-center-line_median0.9573343861341412
in-drivable-lane_median2.7750000000000377


other stats
agent_compute-ego0_max0.013140877416311217
agent_compute-ego0_mean0.012762576288175308
agent_compute-ego0_median0.012970492994193167
agent_compute-ego0_min0.011968441748003678
complete-iteration_max0.2067294027194504
complete-iteration_mean0.18226126493540337
complete-iteration_median0.18400771106334987
complete-iteration_min0.15430023489546318
deviation-center-line_max1.8608474495315033
deviation-center-line_mean1.0829030119620096
deviation-center-line_min0.5560958260482521
deviation-heading_max9.024849054340654
deviation-heading_mean5.2588186511519766
deviation-heading_median5.0487641049478995
deviation-heading_min1.912897340371456
driven_any_max9.277688686117006
driven_any_mean4.962696019171977
driven_any_median4.068392750954551
driven_any_min2.4363098886617984
driven_lanedir_consec_max9.12998798296876
driven_lanedir_consec_mean4.3412732562369065
driven_lanedir_consec_min1.9903348565811787
driven_lanedir_max9.12998798296876
driven_lanedir_mean4.347458241870845
driven_lanedir_median3.1347550639667205
driven_lanedir_min1.9903348565811787
get_duckie_state_max2.122364161204707e-06
get_duckie_state_mean1.972954764739824e-06
get_duckie_state_median1.978796403306952e-06
get_duckie_state_min1.8118620911406837e-06
get_robot_state_max0.00375200155352758
get_robot_state_mean0.0036571129720336017
get_robot_state_median0.003667394496740999
get_robot_state_min0.003541661341124828
get_state_dump_max0.004744638694575959
get_state_dump_mean0.0046549468176105184
get_state_dump_median0.004714132248411549
get_state_dump_min0.004446884079043018
get_ui_image_max0.03527752624070349
get_ui_image_mean0.030226325989699016
get_ui_image_median0.030225242468987376
get_ui_image_min0.025177292780117825
in-drivable-lane_max9.95000000000014
in-drivable-lane_mean3.875000000000054
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.4363098886617984, "get_ui_image": 0.0282423181767844, "step_physics": 0.10036449125208008, "survival_time": 16.250000000000096, "driven_lanedir": 2.1912391128237063, "get_state_dump": 0.004744638694575959, "get_robot_state": 0.003624774195665231, "sim_render-ego0": 0.003977595662778141, "get_duckie_state": 2.122364161204707e-06, "in-drivable-lane": 2.2000000000000313, "deviation-heading": 1.912897340371456, "agent_compute-ego0": 0.012921266760562829, "complete-iteration": 0.1683253371642411, "set_robot_commands": 0.002218531684641458, "deviation-center-line": 0.7470168778200689, "driven_lanedir_consec": 2.1912391128237063, "sim_compute_sim_state": 0.010120898668020049, "sim_compute_performance-ego0": 0.0020226079262107427}, "LF-norm-zigzag-000-ego0": {"driven_any": 3.4362841659318804, "get_ui_image": 0.03527752624070349, "step_physics": 0.12967051257771894, "survival_time": 24.150000000000208, "driven_lanedir": 1.9903348565811787, "get_state_dump": 0.00474039483661494, "get_robot_state": 0.00375200155352758, "sim_render-ego0": 0.003988611796670709, "get_duckie_state": 1.9945388983103855e-06, "in-drivable-lane": 9.95000000000014, "deviation-heading": 3.2508108054450817, "agent_compute-ego0": 0.013140877416311217, "complete-iteration": 0.2067294027194504, "set_robot_commands": 0.002220130163775988, "deviation-center-line": 0.5560958260482521, "driven_lanedir_consec": 1.9903348565811787, "sim_compute_sim_state": 0.011816195220001472, "sim_compute_performance-ego0": 0.002033553832818654}, "LF-norm-techtrack-000-ego0": {"driven_any": 4.700501335977223, "get_ui_image": 0.03220816676119035, "step_physics": 0.12594656335230556, "survival_time": 32.05000000000032, "driven_lanedir": 4.078271015109735, "get_state_dump": 0.004687869660208158, "get_robot_state": 0.003710014797816767, "sim_render-ego0": 0.003946800098240932, "get_duckie_state": 1.963053908303519e-06, "in-drivable-lane": 3.350000000000044, "deviation-heading": 6.846717404450717, "agent_compute-ego0": 0.013019719227823513, "complete-iteration": 0.19969008496245863, "set_robot_commands": 0.0022313093470635813, "deviation-center-line": 1.1676518944482135, "driven_lanedir_consec": 4.053531072573985, "sim_compute_sim_state": 0.011821857122617348, "sim_compute_performance-ego0": 0.0020266749034418123}, "LF-norm-small_loop-000-ego0": {"driven_any": 9.277688686117006, "get_ui_image": 0.025177292780117825, "step_physics": 0.09541280283518971, "survival_time": 59.99999999999873, "driven_lanedir": 9.12998798296876, "get_state_dump": 0.004446884079043018, "get_robot_state": 0.003541661341124828, "sim_render-ego0": 0.0037197897972214926, "get_duckie_state": 1.8118620911406837e-06, "in-drivable-lane": 0.0, "deviation-heading": 9.024849054340654, "agent_compute-ego0": 0.011968441748003678, "complete-iteration": 0.15430023489546318, "set_robot_commands": 0.0021340322931243617, "deviation-center-line": 1.8608474495315033, "driven_lanedir_consec": 9.12998798296876, "sim_compute_sim_state": 0.005917143960678012, "sim_compute_performance-ego0": 0.0018983766697130036}}
set_robot_commands_max0.0022313093470635813
set_robot_commands_mean0.0022010008721513475
set_robot_commands_median0.002219330924208723
set_robot_commands_min0.0021340322931243617
sim_compute_performance-ego0_max0.002033553832818654
sim_compute_performance-ego0_mean0.001995303333046053
sim_compute_performance-ego0_median0.0020246414148262773
sim_compute_performance-ego0_min0.0018983766697130036
sim_compute_sim_state_max0.011821857122617348
sim_compute_sim_state_mean0.00991902374282922
sim_compute_sim_state_median0.01096854694401076
sim_compute_sim_state_min0.005917143960678012
sim_render-ego0_max0.003988611796670709
sim_render-ego0_mean0.003908199338727819
sim_render-ego0_median0.003962197880509537
sim_render-ego0_min0.0037197897972214926
simulation-passed1
step_physics_max0.12967051257771894
step_physics_mean0.11284859250432357
step_physics_median0.1131555273021928
step_physics_min0.09541280283518971
survival_time_max59.99999999999873
survival_time_mean33.112499999999834
survival_time_min16.250000000000096
No reset possible
5735610959Melisande Tengexercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-070:24:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median3.3702010496974277
survival_time_median33.92500000000021
deviation-center-line_median1.728840377797431
in-drivable-lane_median7.424999999999967


other stats
agent_compute-ego0_max0.03527425041905156
agent_compute-ego0_mean0.028204709246360732
agent_compute-ego0_median0.032320651941556304
agent_compute-ego0_min0.012903282683278767
complete-iteration_max0.2353577251885927
complete-iteration_mean0.20120589144816745
complete-iteration_median0.19121323883067304
complete-iteration_min0.18703936294273096
deviation-center-line_max2.0153369046828327
deviation-center-line_mean1.4402362787195395
deviation-center-line_min0.2879274546004635
deviation-heading_max9.810512045351205
deviation-heading_mean6.564885709327446
deviation-heading_median7.756794342948893
deviation-heading_min0.9354421060607936
driven_any_max8.329673561958007
driven_any_mean4.751306566438373
driven_any_median4.529461159262945
driven_any_min1.6166303852695934
driven_lanedir_consec_max7.466378231716881
driven_lanedir_consec_mean3.725488652970663
driven_lanedir_consec_min0.6951742807709163
driven_lanedir_max7.466378231716881
driven_lanedir_mean3.725488652970663
driven_lanedir_median3.3702010496974277
driven_lanedir_min0.6951742807709163
get_duckie_state_max1.942883125451284e-06
get_duckie_state_mean1.5006486327319763e-06
get_duckie_state_median1.371997030772888e-06
get_duckie_state_min1.3157173439308448e-06
get_robot_state_max0.004033434103172487
get_robot_state_mean0.003769184045590738
get_robot_state_median0.003750199695053109
get_robot_state_min0.003542902689084248
get_state_dump_max0.005201112420036831
get_state_dump_mean0.0049115690653139385
get_state_dump_median0.004907805134019504
get_state_dump_min0.004629553573179915
get_ui_image_max0.036487786882767104
get_ui_image_mean0.03056317461429241
get_ui_image_median0.029720425673133342
get_ui_image_min0.02632406022813585
in-drivable-lane_max7.850000000000068
in-drivable-lane_mean6.737499999999985
in-drivable-lane_min4.249999999999936
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 8.329673561958007, "get_ui_image": 0.028769133192216426, "step_physics": 0.10655154633978622, "survival_time": 59.99999999999873, "driven_lanedir": 7.466378231716881, "get_state_dump": 0.005201112420036831, "get_robot_state": 0.004033434103172487, "sim_render-ego0": 0.004196181483907962, "get_duckie_state": 1.942883125451284e-06, "in-drivable-lane": 4.249999999999936, "deviation-heading": 9.810512045351205, "agent_compute-ego0": 0.029452283614680337, "complete-iteration": 0.19319194381580465, "set_robot_commands": 0.0023939613497922264, "deviation-center-line": 2.0153369046828327, "driven_lanedir_consec": 7.466378231716881, "sim_compute_sim_state": 0.01023979945345584, "sim_compute_performance-ego0": 0.00224326889679692}, "LF-norm-zigzag-000-ego0": {"driven_any": 4.797718224098764, "get_ui_image": 0.036487786882767104, "step_physics": 0.1344808947741155, "survival_time": 35.8500000000001, "driven_lanedir": 3.753621130249082, "get_state_dump": 0.005103562867741066, "get_robot_state": 0.003823683786525036, "sim_render-ego0": 0.004136516855287685, "get_duckie_state": 1.4139085094908818e-06, "in-drivable-lane": 7.099999999999827, "deviation-heading": 7.131430099340522, "agent_compute-ego0": 0.03518902026843228, "complete-iteration": 0.2353577251885927, "set_robot_commands": 0.0024005563146224595, "deviation-center-line": 1.6806221037651807, "driven_lanedir_consec": 3.753621130249082, "sim_compute_sim_state": 0.011496649147076194, "sim_compute_performance-ego0": 0.0021385346282491446}, "LF-norm-techtrack-000-ego0": {"driven_any": 4.261204094427125, "get_ui_image": 0.03067171815405026, "step_physics": 0.11898739847489712, "survival_time": 32.00000000000032, "driven_lanedir": 2.9867809691457734, "get_state_dump": 0.004629553573179915, "get_robot_state": 0.003542902689084248, "sim_render-ego0": 0.003873104238286963, "get_duckie_state": 1.3300855520548946e-06, "in-drivable-lane": 7.750000000000106, "deviation-heading": 8.382158586557264, "agent_compute-ego0": 0.012903282683278767, "complete-iteration": 0.18923453384554145, "set_robot_commands": 0.0021350766864842072, "deviation-center-line": 1.7770586518296816, "driven_lanedir_consec": 2.9867809691457734, "sim_compute_sim_state": 0.01042525184321887, "sim_compute_performance-ego0": 0.0019744495147847904}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.6166303852695934, "get_ui_image": 0.02632406022813585, "step_physics": 0.10328335673720748, "survival_time": 13.450000000000056, "driven_lanedir": 0.6951742807709163, "get_state_dump": 0.004712047400297942, "get_robot_state": 0.003676715603581181, "sim_render-ego0": 0.0040283538677074295, "get_duckie_state": 1.3157173439308448e-06, "in-drivable-lane": 7.850000000000068, "deviation-heading": 0.9354421060607936, "agent_compute-ego0": 0.03527425041905156, "complete-iteration": 0.18703936294273096, "set_robot_commands": 0.00235663166752568, "deviation-center-line": 0.2879274546004635, "driven_lanedir_consec": 0.6951742807709163, "sim_compute_sim_state": 0.005318312291745786, "sim_compute_performance-ego0": 0.001974304517110189}}
set_robot_commands_max0.0024005563146224595
set_robot_commands_mean0.0023215565046061436
set_robot_commands_median0.0023752965086589533
set_robot_commands_min0.0021350766864842072
sim_compute_performance-ego0_max0.00224326889679692
sim_compute_performance-ego0_mean0.002082639389235261
sim_compute_performance-ego0_median0.0020564920715169677
sim_compute_performance-ego0_min0.001974304517110189
sim_compute_sim_state_max0.011496649147076194
sim_compute_sim_state_mean0.009370003183874173
sim_compute_sim_state_median0.010332525648337356
sim_compute_sim_state_min0.005318312291745786
sim_render-ego0_max0.004196181483907962
sim_render-ego0_mean0.004058539111297509
sim_render-ego0_median0.004082435361497557
sim_render-ego0_min0.003873104238286963
simulation-passed1
step_physics_max0.1344808947741155
step_physics_mean0.1158257990815016
step_physics_median0.11276947240734168
step_physics_min0.10328335673720748
survival_time_max59.99999999999873
survival_time_mean35.3249999999998
survival_time_min13.450000000000056
No reset possible
5733211010Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simabortedyesnogpu-prod-070:01:57
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5731011312Moustafa Elarabitemplate-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-070:05:22
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5712811014Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-070:44:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.0
survival_time_median59.99999999999873
deviation-center-line_median1.2422730096440104
in-drivable-lane_median0.0


other stats
agent_compute-ego0_max0.014438563242840033
agent_compute-ego0_mean0.012687308653308192
agent_compute-ego0_median0.012204638230215004
agent_compute-ego0_min0.011901394909962726
complete-iteration_max0.3694212039642588
complete-iteration_mean0.2840975406366423
complete-iteration_median0.27652771864008846
complete-iteration_min0.21391352130213348
deviation-center-line_max4.053503393024394
deviation-center-line_mean1.731069028434027
deviation-center-line_min0.386226701423694
deviation-heading_max27.859809596736422
deviation-heading_mean14.925250437835436
deviation-heading_median14.309269950178932
deviation-heading_min3.22265225424745
driven_any_max2.6645352591003757e-13
driven_any_mean1.9984014443252818e-13
driven_any_median2.6645352591003757e-13
driven_any_min0.0
driven_lanedir_consec_max0.000286102294921875
driven_lanedir_consec_mean7.152557373046875e-05
driven_lanedir_consec_min0.0
driven_lanedir_max0.000286102294921875
driven_lanedir_mean7.152557373046875e-05
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max2.500913621583251e-06
get_duckie_state_mean1.9442231133021877e-06
get_duckie_state_median1.8414410822198948e-06
get_duckie_state_min1.593096667185711e-06
get_robot_state_max0.004143996600008924
get_robot_state_mean0.003705875462636066
get_robot_state_median0.0036080909906874886
get_robot_state_min0.0034633232691603636
get_state_dump_max0.004978089209500201
get_state_dump_mean0.004682371360276958
get_state_dump_median0.004640497633261446
get_state_dump_min0.00447040096508474
get_ui_image_max0.0368169771443795
get_ui_image_mean0.029545591931656737
get_ui_image_median0.028653009150248585
get_ui_image_min0.024059372281750275
in-drivable-lane_max0.0
in-drivable-lane_mean0.0
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.026488730155061823, "step_physics": 0.1951963548160016, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.00447040096508474, "get_robot_state": 0.003615696166179063, "sim_render-ego0": 0.003729136162852367, "get_duckie_state": 1.8678438058006671e-06, "in-drivable-lane": 0.0, "deviation-heading": 22.66279310353771, "agent_compute-ego0": 0.011901394909962726, "complete-iteration": 0.2575971758633629, "set_robot_commands": 0.002137594278607142, "deviation-center-line": 4.053503393024394, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008040816261806059, "sim_compute_performance-ego0": 0.0019381016517658217}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.0, "get_ui_image": 0.0368169771443795, "step_physics": 0.28739314611309474, "survival_time": 59.99999999999873, "driven_lanedir": 0.000286102294921875, "get_state_dump": 0.004978089209500201, "get_robot_state": 0.004143996600008924, "sim_render-ego0": 0.004317983600321062, "get_duckie_state": 2.500913621583251e-06, "in-drivable-lane": 0.0, "deviation-heading": 27.859809596736422, "agent_compute-ego0": 0.014438563242840033, "complete-iteration": 0.3694212039642588, "set_robot_commands": 0.0024576018394578208, "deviation-center-line": 1.0457540566888746, "driven_lanedir_consec": 0.000286102294921875, "sim_compute_sim_state": 0.012425198344564957, "sim_compute_performance-ego0": 0.0023453741843853268}, "LF-norm-techtrack-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.030817288145435343, "step_physics": 0.22792292097823805, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004551492265420194, "get_robot_state": 0.0036004858151959143, "sim_render-ego0": 0.003821317004125184, "get_duckie_state": 1.8150383586391224e-06, "in-drivable-lane": 0.0, "deviation-heading": 3.22265225424745, "agent_compute-ego0": 0.012059469207141123, "complete-iteration": 0.295458261416814, "set_robot_commands": 0.0021659966610949005, "deviation-center-line": 1.4387919625991463, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008523741332220098, "sim_compute_performance-ego0": 0.0019155005233472432}, "LF-norm-small_loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.024059372281750275, "step_physics": 0.15624232038073893, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004729503001102698, "get_robot_state": 0.0034633232691603636, "sim_render-ego0": 0.0035474677566286923, "get_duckie_state": 1.593096667185711e-06, "in-drivable-lane": 0.0, "deviation-heading": 5.955746796820156, "agent_compute-ego0": 0.012349807253288884, "complete-iteration": 0.21391352130213348, "set_robot_commands": 0.001998178964848324, "deviation-center-line": 0.386226701423694, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.005638684757940974, "sim_compute_performance-ego0": 0.0018113082294956432}}
set_robot_commands_max0.0024576018394578208
set_robot_commands_mean0.002189842936002047
set_robot_commands_median0.0021517954698510216
set_robot_commands_min0.001998178964848324
sim_compute_performance-ego0_max0.0023453741843853268
sim_compute_performance-ego0_mean0.002002571147248509
sim_compute_performance-ego0_median0.0019268010875565325
sim_compute_performance-ego0_min0.0018113082294956432
sim_compute_sim_state_max0.012425198344564957
sim_compute_sim_state_mean0.008657110174133021
sim_compute_sim_state_median0.008282278797013078
sim_compute_sim_state_min0.005638684757940974
sim_render-ego0_max0.004317983600321062
sim_render-ego0_mean0.0038539761309818264
sim_render-ego0_median0.003775226583488775
sim_render-ego0_min0.0035474677566286923
simulation-passed1
step_physics_max0.28739314611309474
step_physics_mean0.2166886855720183
step_physics_median0.21155963789711985
step_physics_min0.15624232038073893
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
5710611425Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-070:00:51
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5696011049Melisande Tengexercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-070:26:08
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median4.381157471276697
survival_time_median45.82499999999952
deviation-center-line_median1.5896444814096409
in-drivable-lane_median8.225000000000023


other stats
agent_compute-ego0_max0.013428329599314722
agent_compute-ego0_mean0.013057465305849005
agent_compute-ego0_median0.013172859214099598
agent_compute-ego0_min0.012455813195882091
complete-iteration_max0.21103585074088857
complete-iteration_mean0.18667855317582657
complete-iteration_median0.18584137014608296
complete-iteration_min0.1639956216702516
deviation-center-line_max2.748266384395304
deviation-center-line_mean1.5638487474174871
deviation-center-line_min0.32783964245536257
deviation-heading_max13.10814099247895
deviation-heading_mean7.936715170300672
deviation-heading_median7.662388379280076
deviation-heading_min3.313942930163587
driven_any_max7.73326538024463
driven_any_mean5.331369361305071
driven_any_median5.543406833334114
driven_any_min2.5053983983074226
driven_lanedir_consec_max7.586725152879582
driven_lanedir_consec_mean4.252365214583663
driven_lanedir_consec_min0.6604207629016774
driven_lanedir_max7.586725152879582
driven_lanedir_mean4.255566643478365
driven_lanedir_median4.381157471276697
driven_lanedir_min0.6732264784804844
get_duckie_state_max1.6384696358761954e-06
get_duckie_state_mean1.4901362645596298e-06
get_duckie_state_median1.4701002719113962e-06
get_duckie_state_min1.381874878539531e-06
get_robot_state_max0.004000282814074014
get_robot_state_mean0.003769837320708531
get_robot_state_median0.0037376303299807272
get_robot_state_min0.0036038058087986577
get_state_dump_max0.004878347980487234
get_state_dump_mean0.004687891051263143
get_state_dump_median0.0046782040030806105
get_state_dump_min0.0045168082184041175
get_ui_image_max0.03565927449114416
get_ui_image_mean0.03073824745600815
get_ui_image_median0.03009554491292032
get_ui_image_min0.027102625507047808
in-drivable-lane_max14.35000000000019
in-drivable-lane_mean7.700000000000059
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 7.73326538024463, "get_ui_image": 0.027529030219402836, "step_physics": 0.10289767640119388, "survival_time": 59.99999999999873, "driven_lanedir": 7.586725152879582, "get_state_dump": 0.0045168082184041175, "get_robot_state": 0.0036038058087986577, "sim_render-ego0": 0.0038121141660818945, "get_duckie_state": 1.381874878539531e-06, "in-drivable-lane": 0.0, "deviation-heading": 10.118195872664216, "agent_compute-ego0": 0.012455813195882091, "complete-iteration": 0.16841276261729066, "set_robot_commands": 0.0021314845295571767, "deviation-center-line": 2.0298325605554774, "driven_lanedir_consec": 7.586725152879582, "sim_compute_sim_state": 0.00938809126441822, "sim_compute_performance-ego0": 0.0019863449862160157}, "LF-norm-zigzag-000-ego0": {"driven_any": 7.332116345376601, "get_ui_image": 0.03565927449114416, "step_physics": 0.1336596645383811, "survival_time": 59.99999999999873, "driven_lanedir": 6.578985412110802, "get_state_dump": 0.004587243339799028, "get_robot_state": 0.0037314570218101328, "sim_render-ego0": 0.003914809643874855, "get_duckie_state": 1.52996835065424e-06, "in-drivable-lane": 3.899999999999867, "deviation-heading": 13.10814099247895, "agent_compute-ego0": 0.013026509257180805, "complete-iteration": 0.21103585074088857, "set_robot_commands": 0.002235199192183699, "deviation-center-line": 2.748266384395304, "driven_lanedir_consec": 6.578985412110802, "sim_compute_sim_state": 0.012053547056390284, "sim_compute_performance-ego0": 0.0020755256443198374}, "LF-norm-techtrack-000-ego0": {"driven_any": 3.7546973212916273, "get_ui_image": 0.0326620596064378, "step_physics": 0.128291724232093, "survival_time": 31.650000000000315, "driven_lanedir": 2.183329530442592, "get_state_dump": 0.004878347980487234, "get_robot_state": 0.004000282814074014, "sim_render-ego0": 0.004104338233779281, "get_duckie_state": 1.6384696358761954e-06, "in-drivable-lane": 12.550000000000178, "deviation-heading": 5.206580885895935, "agent_compute-ego0": 0.01331920917101839, "complete-iteration": 0.20326997767487537, "set_robot_commands": 0.002335683028404645, "deviation-center-line": 1.1494564022638043, "driven_lanedir_consec": 2.183329530442592, "sim_compute_sim_state": 0.01135916852800628, "sim_compute_performance-ego0": 0.002208128712530768}, "LF-norm-small_loop-000-ego0": {"driven_any": 2.5053983983074226, "get_ui_image": 0.027102625507047808, "step_physics": 0.1013868671724166, "survival_time": 21.700000000000173, "driven_lanedir": 0.6732264784804844, "get_state_dump": 0.004769164666362192, "get_robot_state": 0.003743803638151322, "sim_render-ego0": 0.003983592438971859, "get_duckie_state": 1.4102321931685524e-06, "in-drivable-lane": 14.35000000000019, "deviation-heading": 3.313942930163587, "agent_compute-ego0": 0.013428329599314722, "complete-iteration": 0.1639956216702516, "set_robot_commands": 0.0022573668381263468, "deviation-center-line": 0.32783964245536257, "driven_lanedir_consec": 0.6604207629016774, "sim_compute_sim_state": 0.005203440545619219, "sim_compute_performance-ego0": 0.002025934197436804}}
set_robot_commands_max0.002335683028404645
set_robot_commands_mean0.002239933397067967
set_robot_commands_median0.002246283015155023
set_robot_commands_min0.0021314845295571767
sim_compute_performance-ego0_max0.002208128712530768
sim_compute_performance-ego0_mean0.0020739833851258563
sim_compute_performance-ego0_median0.002050729920878321
sim_compute_performance-ego0_min0.0019863449862160157
sim_compute_sim_state_max0.012053547056390284
sim_compute_sim_state_mean0.0095010618486085
sim_compute_sim_state_median0.01037362989621225
sim_compute_sim_state_min0.005203440545619219
sim_render-ego0_max0.004104338233779281
sim_render-ego0_mean0.003953713620676973
sim_render-ego0_median0.003949201041423357
sim_render-ego0_min0.0038121141660818945
simulation-passed1
step_physics_max0.1336596645383811
step_physics_mean0.11655898308602115
step_physics_median0.11559470031664346
step_physics_min0.1013868671724166
survival_time_max59.99999999999873
survival_time_mean43.33749999999948
survival_time_min21.700000000000173
No reset possible
5693011427Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simhost-erroryesnogpu-prod-070:05:34
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5682611075Fernanda Custodio Pereira do Carmo 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-070:18:50
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.974637481526261
survival_time_median26.949999999999967
deviation-center-line_median1.460656626642476
in-drivable-lane_median5.1000000000000085


other stats
agent_compute-ego0_max0.01413726806640625
agent_compute-ego0_mean0.013209578675637944
agent_compute-ego0_median0.01314797181215457
agent_compute-ego0_min0.012405103011836383
complete-iteration_max0.24374302657874855
complete-iteration_mean0.19083275010929063
complete-iteration_median0.18032238102719503
complete-iteration_min0.15894321180402388
deviation-center-line_max2.787417108078256
deviation-center-line_mean1.464774902340205
deviation-center-line_min0.15036924799761245
deviation-heading_max7.401194599401272
deviation-heading_mean4.238478672744466
deviation-heading_median4.300474472949931
deviation-heading_min0.9517711456767292
driven_any_max10.88398168377621
driven_any_mean5.027825262666994
driven_any_median4.249528426547367
driven_any_min0.7282625137970306
driven_lanedir_consec_max9.565720700342991
driven_lanedir_consec_mean3.942641907375215
driven_lanedir_consec_min0.2555719661053466
driven_lanedir_max9.565720700342991
driven_lanedir_mean3.964233104975592
driven_lanedir_median3.017819876727015
driven_lanedir_min0.2555719661053466
get_duckie_state_max1.7634383193007462e-06
get_duckie_state_mean1.450734248692192e-06
get_duckie_state_median1.3957678810252222e-06
get_duckie_state_min1.247962913417577e-06
get_robot_state_max0.004213822854531778
get_robot_state_mean0.0038871400155585326
get_robot_state_median0.003868910635944808
get_robot_state_min0.003596915935812737
get_state_dump_max0.005116370347169068
get_state_dump_mean0.004792546203784275
get_state_dump_median0.004808814939420092
get_state_dump_min0.004436184589127849
get_ui_image_max0.03852548255576744
get_ui_image_mean0.03145984188997196
get_ui_image_median0.03046555022128466
get_ui_image_min0.026382784561551083
in-drivable-lane_max14.899999999999654
in-drivable-lane_mean6.912499999999927
in-drivable-lane_min2.550000000000036
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.407127450301493, "get_ui_image": 0.02992394799036337, "step_physics": 0.10507705194730284, "survival_time": 14.050000000000065, "driven_lanedir": 2.120622618809238, "get_state_dump": 0.004903905780602854, "get_robot_state": 0.003946416766930979, "sim_render-ego0": 0.004143111249233814, "get_duckie_state": 1.4989934069045047e-06, "in-drivable-lane": 2.550000000000036, "deviation-heading": 1.381315264631244, "agent_compute-ego0": 0.013717938822211949, "complete-iteration": 0.1766744260246872, "set_robot_commands": 0.002394310971523853, "deviation-center-line": 0.5432211838093643, "driven_lanedir_consec": 2.120622618809238, "sim_compute_sim_state": 0.010286686268258604, "sim_compute_performance-ego0": 0.0021769138092690324}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.7282625137970306, "get_ui_image": 0.03852548255576744, "step_physics": 0.16095535604803413, "survival_time": 5.4999999999999885, "driven_lanedir": 0.2555719661053466, "get_state_dump": 0.005116370347169068, "get_robot_state": 0.004213822854531778, "sim_render-ego0": 0.0043205312780431795, "get_duckie_state": 1.7634383193007462e-06, "in-drivable-lane": 3.399999999999988, "deviation-heading": 0.9517711456767292, "agent_compute-ego0": 0.01413726806640625, "complete-iteration": 0.24374302657874855, "set_robot_commands": 0.0024176567524403065, "deviation-center-line": 0.15036924799761245, "driven_lanedir_consec": 0.2555719661053466, "sim_compute_sim_state": 0.011517874829403987, "sim_compute_performance-ego0": 0.002433497626502235}, "LF-norm-techtrack-000-ego0": {"driven_any": 6.091929402793241, "get_ui_image": 0.031007152452205953, "step_physics": 0.11191941592328827, "survival_time": 39.84999999999987, "driven_lanedir": 3.915017134644791, "get_state_dump": 0.004436184589127849, "get_robot_state": 0.003596915935812737, "sim_render-ego0": 0.0038210019133144753, "get_duckie_state": 1.247962913417577e-06, "in-drivable-lane": 14.899999999999654, "deviation-heading": 7.401194599401272, "agent_compute-ego0": 0.012405103011836383, "complete-iteration": 0.1839703360297029, "set_robot_commands": 0.002144573326397659, "deviation-center-line": 2.378092069475587, "driven_lanedir_consec": 3.828652344243285, "sim_compute_sim_state": 0.012578367290640236, "sim_compute_performance-ego0": 0.00198265454524143}, "LF-norm-small_loop-000-ego0": {"driven_any": 10.88398168377621, "get_ui_image": 0.026382784561551083, "step_physics": 0.0968668971033914, "survival_time": 59.99999999999873, "driven_lanedir": 9.565720700342991, "get_state_dump": 0.004713724098237329, "get_robot_state": 0.003791404504958637, "sim_render-ego0": 0.003987487607157101, "get_duckie_state": 1.2925423551459397e-06, "in-drivable-lane": 6.80000000000003, "deviation-heading": 7.219633681268619, "agent_compute-ego0": 0.012578004802097193, "complete-iteration": 0.15894321180402388, "set_robot_commands": 0.002252296841611076, "deviation-center-line": 2.787417108078256, "driven_lanedir_consec": 9.565720700342991, "sim_compute_sim_state": 0.006208705663879547, "sim_compute_performance-ego0": 0.0020799132608354935}}
set_robot_commands_max0.0024176567524403065
set_robot_commands_mean0.002302209472993224
set_robot_commands_median0.0023233039065674644
set_robot_commands_min0.002144573326397659
sim_compute_performance-ego0_max0.002433497626502235
sim_compute_performance-ego0_mean0.0021682448104620475
sim_compute_performance-ego0_median0.002128413535052263
sim_compute_performance-ego0_min0.00198265454524143
sim_compute_sim_state_max0.012578367290640236
sim_compute_sim_state_mean0.010147908513045593
sim_compute_sim_state_median0.010902280548831297
sim_compute_sim_state_min0.006208705663879547
sim_render-ego0_max0.0043205312780431795
sim_render-ego0_mean0.004068033011937143
sim_render-ego0_median0.004065299428195458
sim_render-ego0_min0.0038210019133144753
simulation-passed1
step_physics_max0.16095535604803413
step_physics_mean0.11870468025550417
step_physics_median0.10849823393529556
step_physics_min0.0968668971033914
survival_time_max59.99999999999873
survival_time_mean29.84999999999966
survival_time_min5.4999999999999885
No reset possible
5660911102Melisande Tengexercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-070:36:09
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median6.3753086319111265
survival_time_median59.99999999999873
deviation-center-line_median2.6611282315632065
in-drivable-lane_median2.499999999999858


other stats
agent_compute-ego0_max0.035261472499338094
agent_compute-ego0_mean0.029428823629236857
agent_compute-ego0_median0.03380253362615936
agent_compute-ego0_min0.014848754765290602
complete-iteration_max0.2348931507020512
complete-iteration_mean0.19898901549287984
complete-iteration_median0.19154819947893076
complete-iteration_min0.17796651231160668
deviation-center-line_max3.1081060092717805
deviation-center-line_mean2.714677324599806
deviation-center-line_min2.428346826001031
deviation-heading_max13.777991935279948
deviation-heading_mean12.350648586667685
deviation-heading_median12.20922530516012
deviation-heading_min11.206151801070542
driven_any_max9.57854710357881
driven_any_mean9.002655148541704
driven_any_median9.236920255406917
driven_any_min7.958232979774177
driven_lanedir_consec_max8.064199363187171
driven_lanedir_consec_mean6.560874683428095
driven_lanedir_consec_min5.428682106702951
driven_lanedir_max9.196582016436832
driven_lanedir_mean8.054160625844027
driven_lanedir_median8.508796781186698
driven_lanedir_min6.00246692456588
get_duckie_state_max1.5013819431682908e-06
get_duckie_state_mean1.3930560487359468e-06
get_duckie_state_median1.3927232492505218e-06
get_duckie_state_min1.2853957532744522e-06
get_robot_state_max0.00378761065194847
get_robot_state_mean0.0037061859700102183
get_robot_state_median0.0037300591215297433
get_robot_state_min0.0035770149850329193
get_state_dump_max0.005045576159106405
get_state_dump_mean0.004756305422409871
get_state_dump_median0.004730832436194299
get_state_dump_min0.004517980658144479
get_ui_image_max0.03588641155426552
get_ui_image_mean0.0303403783773984
get_ui_image_median0.02992031218904341
get_ui_image_min0.025634477577241237
in-drivable-lane_max10.699999999999594
in-drivable-lane_mean4.062499999999827
in-drivable-lane_min0.549999999999998
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 9.53954530634224, "get_ui_image": 0.02789243115275031, "step_physics": 0.10400549358968236, "survival_time": 59.99999999999873, "driven_lanedir": 8.921205599586823, "get_state_dump": 0.004766612525387271, "get_robot_state": 0.0036857219461001126, "sim_render-ego0": 0.003850833660954738, "get_duckie_state": 1.2853957532744522e-06, "in-drivable-lane": 1.6999999999999034, "deviation-heading": 12.372828194054565, "agent_compute-ego0": 0.0339017555576677, "complete-iteration": 0.1924522292306282, "set_robot_commands": 0.0022654958212008384, "deviation-center-line": 3.1081060092717805, "driven_lanedir_consec": 6.22687349777281, "sim_compute_sim_state": 0.01001426024202701, "sim_compute_performance-ego0": 0.0019741824624143374}, "LF-norm-zigzag-000-ego0": {"driven_any": 7.958232979774177, "get_ui_image": 0.03588641155426552, "step_physics": 0.13540373467836023, "survival_time": 54.99999999999901, "driven_lanedir": 6.00246692456588, "get_state_dump": 0.004695052347001328, "get_robot_state": 0.003774396296959374, "sim_render-ego0": 0.003959860182371495, "get_duckie_state": 1.3739926288823015e-06, "in-drivable-lane": 10.699999999999594, "deviation-heading": 13.777991935279948, "agent_compute-ego0": 0.035261472499338094, "complete-iteration": 0.2348931507020512, "set_robot_commands": 0.0022274975772341417, "deviation-center-line": 2.53184494942641, "driven_lanedir_consec": 5.428682106702951, "sim_compute_sim_state": 0.01156704605546028, "sim_compute_performance-ego0": 0.002023422533982889}, "LF-norm-techtrack-000-ego0": {"driven_any": 9.57854710357881, "get_ui_image": 0.03194819322533651, "step_physics": 0.11381752941630266, "survival_time": 59.99999999999873, "driven_lanedir": 9.196582016436832, "get_state_dump": 0.005045576159106405, "get_robot_state": 0.00378761065194847, "sim_render-ego0": 0.004073838409436533, "get_duckie_state": 1.5013819431682908e-06, "in-drivable-lane": 0.549999999999998, "deviation-heading": 11.206151801070542, "agent_compute-ego0": 0.014848754765290602, "complete-iteration": 0.19064416972723333, "set_robot_commands": 0.002262411863976573, "deviation-center-line": 2.428346826001031, "driven_lanedir_consec": 8.064199363187171, "sim_compute_sim_state": 0.012602194262781708, "sim_compute_performance-ego0": 0.0021583627006791215}, "LF-norm-small_loop-000-ego0": {"driven_any": 8.934295204471592, "get_ui_image": 0.025634477577241237, "step_physics": 0.09636307060470392, "survival_time": 59.99999999999873, "driven_lanedir": 8.096387962786572, "get_state_dump": 0.004517980658144479, "get_robot_state": 0.0035770149850329193, "sim_render-ego0": 0.003785156985305926, "get_duckie_state": 1.411453869618742e-06, "in-drivable-lane": 3.2999999999998124, "deviation-heading": 12.045622416265678, "agent_compute-ego0": 0.033703311694651024, "complete-iteration": 0.17796651231160668, "set_robot_commands": 0.0021638651870867294, "deviation-center-line": 2.7904115137000027, "driven_lanedir_consec": 6.523743766049444, "sim_compute_sim_state": 0.006219074986161637, "sim_compute_performance-ego0": 0.0019153381366713855}}
set_robot_commands_max0.0022654958212008384
set_robot_commands_mean0.0022298176123745706
set_robot_commands_median0.0022449547206053575
set_robot_commands_min0.0021638651870867294
sim_compute_performance-ego0_max0.0021583627006791215
sim_compute_performance-ego0_mean0.0020178264584369333
sim_compute_performance-ego0_median0.0019988024981986135
sim_compute_performance-ego0_min0.0019153381366713855
sim_compute_sim_state_max0.012602194262781708
sim_compute_sim_state_mean0.010100643886607658
sim_compute_sim_state_median0.010790653148743643
sim_compute_sim_state_min0.006219074986161637
sim_render-ego0_max0.004073838409436533
sim_render-ego0_mean0.003917422309517173
sim_render-ego0_median0.003905346921663116
sim_render-ego0_min0.003785156985305926
simulation-passed1
step_physics_max0.13540373467836023
step_physics_mean0.11239745707226229
step_physics_median0.1089115115029925
step_physics_min0.09636307060470392
survival_time_max59.99999999999873
survival_time_mean58.7499999999988
survival_time_min54.99999999999901
No reset possible
5657911112Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:03:11
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5657611278Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:35
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5652211116Melisande Tengreal-exercise-1aido5-LF-sim-validationLFv-simfailedyesnogpu-prod-070:03:06
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 190, in main
    raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5650311429Frank (Chude) Qian 🇨🇦template-pytorchaido5-LF-sim-testingLFt-simhost-erroryesnogpu-prod-070:00:48
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5648311281Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:36
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5645611286Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:37
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5640911286Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:01:39
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5635711287Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:41
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5629111285Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:01:57
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5625211290Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:02:53
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5623511290Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:49
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5619711303Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:39
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5614211450Cliff Chandler 🇺🇸template-pytorchaido5-LF-sim-testingLFt-simhost-erroryesnogpu-prod-070:02:44
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5612011450Cliff Chandler 🇺🇸template-pytorchaido5-LF-sim-testingLFt-simhost-erroryesnogpu-prod-070:01:54
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5576811419Frank (Chude) Qian 🇨🇦challenge-aido_LF-baseline-behavior-cloningaido5-LFV-sim-testingLFVv-simsuccessyesnogpu-prod-070:32:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.1253887402315756
survival_time_median11.575000000000053
deviation-center-line_median0.5267204178062265
in-drivable-lane_median0.7750000000000012


other stats
agent_compute-ego0_max0.04713990432875497
agent_compute-ego0_mean0.04192387150832179
agent_compute-ego0_median0.04139478982778814
agent_compute-ego0_min0.0377660020489559
agent_compute-npc0_max0.023087386574063985
agent_compute-npc0_mean0.02061446081012313
agent_compute-npc0_median0.01987180378195221
agent_compute-npc0_min0.01962684910252409
agent_compute-npc1_max0.029141397096326016
agent_compute-npc1_mean0.02587362421013049
agent_compute-npc1_median0.02605174283468014
agent_compute-npc1_min0.022427732699385312
agent_compute-npc2_max0.03020934563762737
agent_compute-npc2_mean0.028541095753121193
agent_compute-npc2_median0.029956815398742107
agent_compute-npc2_min0.025457126222994102
agent_compute-npc3_max0.04055631494967737
agent_compute-npc3_mean0.03948278584108749
agent_compute-npc3_median0.03948278584108749
agent_compute-npc3_min0.03840925673249761
complete-iteration_max1.2261848537509907
complete-iteration_mean0.859561510387918
complete-iteration_median0.9123699897980696
complete-iteration_min0.3873212082045419
deviation-center-line_max1.9105102008993984
deviation-center-line_mean0.8065198207992316
deviation-center-line_min0.2621282466850749
deviation-heading_max6.290265571629366
deviation-heading_mean3.4034043590419496
deviation-heading_median2.90307150625943
deviation-heading_min1.517208852019572
driven_any_max11.398827339954666
driven_any_mean4.39832955407711
driven_any_median2.5763477020302243
driven_any_min1.041795472293323
driven_lanedir_consec_max11.089455557538962
driven_lanedir_consec_mean4.029473772368228
driven_lanedir_consec_min0.7776620514707999
driven_lanedir_max11.089455557538962
driven_lanedir_mean4.029473772368228
driven_lanedir_median2.1253887402315756
driven_lanedir_min0.7776620514707999
get_duckie_state_max2.2671052387782505e-06
get_duckie_state_mean2.040244775779371e-06
get_duckie_state_median2.080162845529357e-06
get_duckie_state_min1.73354817328052e-06
get_robot_state_max0.01658566139912808
get_robot_state_mean0.01335287207045062
get_robot_state_median0.014780473342642809
get_robot_state_min0.0072648801973887855
get_state_dump_max0.010376364921375308
get_state_dump_mean0.008884906179592355
get_state_dump_median0.009406413836768714
get_state_dump_min0.006350432123456683
get_ui_image_max0.047194489808840175
get_ui_image_mean0.04120137228427899
get_ui_image_median0.0435032449788041
get_ui_image_min0.030604509370667592
in-drivable-lane_max2.3000000000000096
in-drivable-lane_mean0.962500000000003
in-drivable-lane_min0.0
per-episodes
details{"LFV-norm-loop-000-ego0": {"driven_any": 11.398827339954666, "get_ui_image": 0.04029674375082212, "step_physics": 0.5549610540801874, "survival_time": 47.64999999999943, "driven_lanedir": 11.089455557538962, "get_state_dump": 0.008967049466738911, "get_robot_state": 0.013631392574910098, "sim_render-ego0": 0.003733936345802163, "sim_render-npc0": 0.003694982148816251, "sim_render-npc1": 0.003695425747325585, "sim_render-npc2": 0.0037277934436278273, "get_duckie_state": 2.0712926452764676e-06, "in-drivable-lane": 0.4500000000000064, "deviation-heading": 6.290265571629366, "agent_compute-ego0": 0.04273716233811289, "agent_compute-npc0": 0.01962684910252409, "agent_compute-npc1": 0.029141397096326016, "agent_compute-npc2": 0.03020934563762737, "complete-iteration": 0.80219862596044, "set_robot_commands": 0.002088555000113241, "deviation-center-line": 1.9105102008993984, "driven_lanedir_consec": 11.089455557538962, "sim_compute_sim_state": 0.03170471171413078, "sim_compute_performance-ego0": 0.001956549580492064, "sim_compute_performance-npc0": 0.0018624837043650245, "sim_compute_performance-npc1": 0.001836529817721129, "sim_compute_performance-npc2": 0.0018534590363252588}, "LFV-norm-zigzag-000-ego0": {"driven_any": 1.041795472293323, "get_ui_image": 0.047194489808840175, "step_physics": 0.7121543260378258, "survival_time": 5.299999999999989, "driven_lanedir": 0.9676818382990876, "get_state_dump": 0.009845778206798515, "get_robot_state": 0.01592955411037552, "sim_render-ego0": 0.0035378576439117717, "sim_render-npc0": 0.00334254157877414, "sim_render-npc1": 0.003387333076690959, "sim_render-npc2": 0.003436803817749023, "sim_render-npc3": 0.0034523210792898017, "get_duckie_state": 1.73354817328052e-06, "in-drivable-lane": 0.0, "deviation-heading": 2.039777560201971, "agent_compute-ego0": 0.0377660020489559, "agent_compute-npc0": 0.019915152932995943, "agent_compute-npc1": 0.022427732699385312, "agent_compute-npc2": 0.029956815398742107, "agent_compute-npc3": 0.04055631494967737, "complete-iteration": 1.0225413536356989, "set_robot_commands": 0.0019154013874374816, "deviation-center-line": 0.3565211169349438, "driven_lanedir_consec": 0.9676818382990876, "sim_compute_sim_state": 0.051280538612437025, "sim_compute_performance-ego0": 0.0018148645062312904, "sim_compute_performance-npc0": 0.0016699015537155009, "sim_compute_performance-npc1": 0.0016664366855799594, "sim_compute_performance-npc2": 0.0017016513325343622, "sim_compute_performance-npc3": 0.0016941489460312318}, "LFV-norm-techtrack-000-ego0": {"driven_any": 4.049784571756448, "get_ui_image": 0.04670974620678607, "step_physics": 0.9174160356224428, "survival_time": 17.600000000000115, "driven_lanedir": 3.283095642164064, "get_state_dump": 0.010376364921375308, "get_robot_state": 0.01658566139912808, "sim_render-ego0": 0.0035839567103399432, "sim_render-npc0": 0.0036252341932345384, "sim_render-npc1": 0.0035515365114293083, "sim_render-npc2": 0.003604283076170146, "sim_render-npc3": 0.003668579771565986, "get_duckie_state": 2.089033045782246e-06, "in-drivable-lane": 2.3000000000000096, "deviation-heading": 3.7663654523168897, "agent_compute-ego0": 0.04005241731746339, "agent_compute-npc0": 0.01982845463090848, "agent_compute-npc1": 0.02605174283468014, "agent_compute-npc2": 0.025457126222994102, "agent_compute-npc3": 0.03840925673249761, "complete-iteration": 1.2261848537509907, "set_robot_commands": 0.002025115928974098, "deviation-center-line": 0.6969197186775091, "driven_lanedir_consec": 3.283095642164064, "sim_compute_sim_state": 0.04781302176521453, "sim_compute_performance-ego0": 0.00186390782213076, "sim_compute_performance-npc0": 0.0017929191967583245, "sim_compute_performance-npc1": 0.0017632668146668343, "sim_compute_performance-npc2": 0.0018201252556387492, "sim_compute_performance-npc3": 0.0017792982690395106}, "LFV-norm-small_loop-000-ego0": {"driven_any": 1.1029108323040009, "get_ui_image": 0.030604509370667592, "step_physics": 0.2464234169040407, "survival_time": 5.549999999999988, "driven_lanedir": 0.7776620514707999, "get_state_dump": 0.006350432123456683, "get_robot_state": 0.0072648801973887855, "sim_render-ego0": 0.003874914986746652, "sim_render-npc0": 0.003830554229872567, "get_duckie_state": 2.2671052387782505e-06, "in-drivable-lane": 1.099999999999996, "deviation-heading": 1.517208852019572, "agent_compute-ego0": 0.04713990432875497, "agent_compute-npc0": 0.023087386574063985, "complete-iteration": 0.3873212082045419, "set_robot_commands": 0.002245215432984488, "deviation-center-line": 0.2621282466850749, "driven_lanedir_consec": 0.7776620514707999, "sim_compute_sim_state": 0.010058307221957616, "sim_compute_performance-ego0": 0.002027209315981184, "sim_compute_performance-npc0": 0.001985060317175729}}
set_robot_commands_max0.002245215432984488
set_robot_commands_mean0.002068571937377327
set_robot_commands_median0.002056835464543669
set_robot_commands_min0.0019154013874374816
sim_compute_performance-ego0_max0.002027209315981184
sim_compute_performance-ego0_mean0.0019156328062088243
sim_compute_performance-ego0_median0.001910228701311412
sim_compute_performance-ego0_min0.0018148645062312904
sim_compute_performance-npc0_max0.001985060317175729
sim_compute_performance-npc0_mean0.001827591193003645
sim_compute_performance-npc0_median0.0018277014505616745
sim_compute_performance-npc0_min0.0016699015537155009
sim_compute_performance-npc1_max0.001836529817721129
sim_compute_performance-npc1_mean0.0017554111059893075
sim_compute_performance-npc1_median0.0017632668146668343
sim_compute_performance-npc1_min0.0016664366855799594
sim_compute_performance-npc2_max0.0018534590363252588
sim_compute_performance-npc2_mean0.0017917452081661234
sim_compute_performance-npc2_median0.0018201252556387492
sim_compute_performance-npc2_min0.0017016513325343622
sim_compute_performance-npc3_max0.0017792982690395106
sim_compute_performance-npc3_mean0.001736723607535371
sim_compute_performance-npc3_median0.001736723607535371
sim_compute_performance-npc3_min0.0016941489460312318
sim_compute_sim_state_max0.051280538612437025
sim_compute_sim_state_mean0.03521414482843499
sim_compute_sim_state_median0.03975886673967265
sim_compute_sim_state_min0.010058307221957616
sim_render-ego0_max0.003874914986746652
sim_render-ego0_mean0.003682666421700132
sim_render-ego0_median0.003658946528071053
sim_render-ego0_min0.0035378576439117717
sim_render-npc0_max0.003830554229872567
sim_render-npc0_mean0.0036233280376743743
sim_render-npc0_median0.003660108171025395
sim_render-npc0_min0.00334254157877414
sim_render-npc1_max0.003695425747325585
sim_render-npc1_mean0.003544765111815284
sim_render-npc1_median0.0035515365114293083
sim_render-npc1_min0.003387333076690959
sim_render-npc2_max0.0037277934436278273
sim_render-npc2_mean0.003589626779182332
sim_render-npc2_median0.003604283076170146
sim_render-npc2_min0.003436803817749023
sim_render-npc3_max0.003668579771565986
sim_render-npc3_mean0.003560450425427894
sim_render-npc3_median0.003560450425427894
sim_render-npc3_min0.0034523210792898017
simulation-passed1
step_physics_max0.9174160356224428
step_physics_mean0.6077387081611242
step_physics_median0.6335576900590065
step_physics_min0.2464234169040407
survival_time_max47.64999999999943
survival_time_mean19.02499999999988
survival_time_min5.299999999999989
No reset possible
5573511296Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:01:55
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5571111284Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:34
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5567711284Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:00:33
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5561011280Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-070:02:10
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||