Duckietown Challenges Home Challenges Submissions

Evaluator 4896

ID4896
evaluatornogpu-prod-02
ownerI don't have one 😀
machinenogpu-prod_e7650f964a9f
processnogpu-prod-02_e7650f964a9f
version6.2.5
first heard
last heard
statusinactive
# evaluating
# success19 53948
# timeout
# failed3 54644
# error2 54128
# aborted
# host-error37 53933
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
5739510942Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-020:48:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.0
survival_time_median59.99999999999873
deviation-center-line_median1.2422730096440104
in-drivable-lane_median0.0


other stats
agent_compute-ego0_max0.013626745797315308
agent_compute-ego0_mean0.013274894070367232
agent_compute-ego0_median0.013342005525600106
agent_compute-ego0_min0.012788819432953414
complete-iteration_max0.34778029734050103
complete-iteration_mean0.295981762857858
complete-iteration_median0.30129087695868983
complete-iteration_min0.23356500017355125
deviation-center-line_max4.053503393024394
deviation-center-line_mean1.731069028434027
deviation-center-line_min0.386226701423694
deviation-heading_max27.859809596736422
deviation-heading_mean14.925250437835436
deviation-heading_median14.309269950178932
deviation-heading_min3.22265225424745
driven_any_max2.6645352591003757e-13
driven_any_mean1.9984014443252818e-13
driven_any_median2.6645352591003757e-13
driven_any_min0.0
driven_lanedir_consec_max0.000286102294921875
driven_lanedir_consec_mean7.152557373046875e-05
driven_lanedir_consec_min0.0
driven_lanedir_max0.000286102294921875
driven_lanedir_mean7.152557373046875e-05
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max1.4559216145968854e-06
get_duckie_state_mean1.3885748177940504e-06
get_duckie_state_median1.4093694440728918e-06
get_duckie_state_min1.279638768433532e-06
get_robot_state_max0.003909702205737366
get_robot_state_mean0.003737910254412547
get_robot_state_median0.003689635107658189
get_robot_state_min0.003662668596596444
get_state_dump_max0.004726419242395152
get_state_dump_mean0.004690059515756929
get_state_dump_median0.004691104110730478
get_state_dump_min0.0046516105991716095
get_ui_image_max0.035447697159054874
get_ui_image_mean0.030784701377922653
get_ui_image_median0.030857404701715704
get_ui_image_min0.02597629894920432
in-drivable-lane_max0.0
in-drivable-lane_mean0.0
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.02859741920833286, "step_physics": 0.2133099177199339, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004726419242395152, "get_robot_state": 0.003702394174199418, "sim_render-ego0": 0.00396382917869498, "get_duckie_state": 1.4416284108539109e-06, "in-drivable-lane": 0.0, "deviation-heading": 22.66279310353771, "agent_compute-ego0": 0.013174689679618283, "complete-iteration": 0.2803717679921832, "set_robot_commands": 0.0022210247014384783, "deviation-center-line": 4.053503393024394, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008561869445788076, "sim_compute_performance-ego0": 0.002026692715215246}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.0, "get_ui_image": 0.035447697159054874, "step_physics": 0.27048063437011616, "survival_time": 59.99999999999873, "driven_lanedir": 0.000286102294921875, "get_state_dump": 0.004683702216358804, "get_robot_state": 0.003676876041116961, "sim_render-ego0": 0.003944594695308028, "get_duckie_state": 1.3771104772918726e-06, "in-drivable-lane": 0.0, "deviation-heading": 27.859809596736422, "agent_compute-ego0": 0.013626745797315308, "complete-iteration": 0.34778029734050103, "set_robot_commands": 0.0022051558704995595, "deviation-center-line": 1.0457540566888746, "driven_lanedir_consec": 0.000286102294921875, "sim_compute_sim_state": 0.011618671171075595, "sim_compute_performance-ego0": 0.00200853717019417}, "LF-norm-techtrack-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.033117390195098545, "step_physics": 0.24911165832976911, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004698506005102153, "get_robot_state": 0.003909702205737366, "sim_render-ego0": 0.004094266176819305, "get_duckie_state": 1.4559216145968854e-06, "in-drivable-lane": 0.0, "deviation-heading": 3.22265225424745, "agent_compute-ego0": 0.01350932137158193, "complete-iteration": 0.3222099859251964, "set_robot_commands": 0.002317860759763694, "deviation-center-line": 1.4387919625991463, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.009206814928713884, "sim_compute_performance-ego0": 0.002150887752154983}, "LF-norm-small_loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.02597629894920432, "step_physics": 0.17228834019612513, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.0046516105991716095, "get_robot_state": 0.003662668596596444, "sim_render-ego0": 0.0038837743341476097, "get_duckie_state": 1.279638768433532e-06, "in-drivable-lane": 0.0, "deviation-heading": 5.955746796820156, "agent_compute-ego0": 0.012788819432953414, "complete-iteration": 0.23356500017355125, "set_robot_commands": 0.002183258880882041, "deviation-center-line": 0.386226701423694, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.006072114845993715, "sim_compute_performance-ego0": 0.001973174593033739}}
set_robot_commands_max0.002317860759763694
set_robot_commands_mean0.002231825053145943
set_robot_commands_median0.002213090285969019
set_robot_commands_min0.002183258880882041
sim_compute_performance-ego0_max0.002150887752154983
sim_compute_performance-ego0_mean0.0020398230576495345
sim_compute_performance-ego0_median0.0020176149427047077
sim_compute_performance-ego0_min0.001973174593033739
sim_compute_sim_state_max0.011618671171075595
sim_compute_sim_state_mean0.008864867597892819
sim_compute_sim_state_median0.00888434218725098
sim_compute_sim_state_min0.006072114845993715
sim_render-ego0_max0.004094266176819305
sim_render-ego0_mean0.003971616096242481
sim_render-ego0_median0.003954211937001504
sim_render-ego0_min0.0038837743341476097
simulation-passed1
step_physics_max0.27048063437011616
step_physics_mean0.22629763765398608
step_physics_median0.2312107880248515
step_physics_min0.17228834019612513
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
5728810969Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-020:23:59
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median3.0268520741740996
survival_time_median40.34999999999979
deviation-center-line_median1.1329678400294174
in-drivable-lane_median12.275000000000174


other stats
agent_compute-ego0_max0.013943648861791795
agent_compute-ego0_mean0.013073643266131791
agent_compute-ego0_median0.013176377784348544
agent_compute-ego0_min0.011998168634038284
complete-iteration_max0.20669024135483113
complete-iteration_mean0.18345788488525797
complete-iteration_median0.18845282709793457
complete-iteration_min0.15023564399033165
deviation-center-line_max3.0679705127710553
deviation-center-line_mean1.5293519379469522
deviation-center-line_min0.7835015589579193
deviation-heading_max10.427472857676666
deviation-heading_mean5.465622703932828
deviation-heading_median4.3060016161147265
deviation-heading_min2.823014725825194
driven_any_max7.699915957365383
driven_any_mean5.309662742768639
driven_any_median5.241531957532942
driven_any_min3.0556710986432885
driven_lanedir_consec_max7.554414419454948
driven_lanedir_consec_mean3.838851682518725
driven_lanedir_consec_min1.7472881622717544
driven_lanedir_max7.554414419454948
driven_lanedir_mean3.838851682518725
driven_lanedir_median3.0268520741740996
driven_lanedir_min1.7472881622717544
get_duckie_state_max1.5823189131989926e-06
get_duckie_state_mean1.4482296522968617e-06
get_duckie_state_median1.451981326909096e-06
get_duckie_state_min1.3066370421702617e-06
get_robot_state_max0.004021696939677773
get_robot_state_mean0.003779780347598936
get_robot_state_median0.003790789111843352
get_robot_state_min0.003515846227031266
get_state_dump_max0.005157907090025272
get_state_dump_mean0.004925082733904676
get_state_dump_median0.005004134320381655
get_state_dump_min0.004534155204830122
get_ui_image_max0.03640771264513138
get_ui_image_mean0.031158191546591097
get_ui_image_median0.03164534888962499
get_ui_image_min0.024934355761983017
in-drivable-lane_max25.049999999999795
in-drivable-lane_mean12.400000000000036
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 6.86903375187023, "get_ui_image": 0.02900673332138592, "step_physics": 0.10486034248412662, "survival_time": 50.29999999999928, "driven_lanedir": 4.00116007453239, "get_state_dump": 0.005063146678314663, "get_robot_state": 0.003832942087576903, "sim_render-ego0": 0.0041582967934329085, "get_duckie_state": 1.494200250953288e-06, "in-drivable-lane": 25.049999999999795, "deviation-heading": 3.7455752393808006, "agent_compute-ego0": 0.012950990027207012, "complete-iteration": 0.17576399791797082, "set_robot_commands": 0.002258744235071904, "deviation-center-line": 1.0006078262016331, "driven_lanedir_consec": 4.00116007453239, "sim_compute_sim_state": 0.011381241155357325, "sim_compute_performance-ego0": 0.002158763983990727}, "LF-norm-zigzag-000-ego0": {"driven_any": 3.614030163195654, "get_ui_image": 0.03640771264513138, "step_physics": 0.1276365461803618, "survival_time": 30.400000000000297, "driven_lanedir": 2.0525440738158096, "get_state_dump": 0.004945121962448646, "get_robot_state": 0.0037486361361098014, "sim_render-ego0": 0.004134309702906115, "get_duckie_state": 1.409762402864904e-06, "in-drivable-lane": 12.500000000000178, "deviation-heading": 4.866427992848652, "agent_compute-ego0": 0.013401765541490076, "complete-iteration": 0.20669024135483113, "set_robot_commands": 0.0022720491944862703, "deviation-center-line": 1.2653278538572017, "driven_lanedir_consec": 2.0525440738158096, "sim_compute_sim_state": 0.011976236193050892, "sim_compute_performance-ego0": 0.002078316286083904}, "LF-norm-techtrack-000-ego0": {"driven_any": 3.0556710986432885, "get_ui_image": 0.03428396445786406, "step_physics": 0.12354999673580697, "survival_time": 25.00000000000022, "driven_lanedir": 1.7472881622717544, "get_state_dump": 0.005157907090025272, "get_robot_state": 0.004021696939677773, "sim_render-ego0": 0.004368365643743032, "get_duckie_state": 1.5823189131989926e-06, "in-drivable-lane": 12.050000000000171, "deviation-heading": 2.823014725825194, "agent_compute-ego0": 0.013943648861791795, "complete-iteration": 0.20114165627789832, "set_robot_commands": 0.002391817564973812, "deviation-center-line": 0.7835015589579193, "driven_lanedir_consec": 1.7472881622717544, "sim_compute_sim_state": 0.011148595048519904, "sim_compute_performance-ego0": 0.0021809770199591053}, "LF-norm-small_loop-000-ego0": {"driven_any": 7.699915957365383, "get_ui_image": 0.024934355761983017, "step_physics": 0.09158621680031807, "survival_time": 59.99999999999873, "driven_lanedir": 7.554414419454948, "get_state_dump": 0.004534155204830122, "get_robot_state": 0.003515846227031266, "sim_render-ego0": 0.0037124827938413343, "get_duckie_state": 1.3066370421702617e-06, "in-drivable-lane": 0.0, "deviation-heading": 10.427472857676666, "agent_compute-ego0": 0.011998168634038284, "complete-iteration": 0.15023564399033165, "set_robot_commands": 0.0020710671573356226, "deviation-center-line": 3.0679705127710553, "driven_lanedir_consec": 7.554414419454948, "sim_compute_sim_state": 0.005940695785662217, "sim_compute_performance-ego0": 0.0018649589608452104}}
set_robot_commands_max0.002391817564973812
set_robot_commands_mean0.0022484195379669026
set_robot_commands_median0.002265396714779087
set_robot_commands_min0.0020710671573356226
sim_compute_performance-ego0_max0.0021809770199591053
sim_compute_performance-ego0_mean0.0020707540627197367
sim_compute_performance-ego0_median0.002118540135037316
sim_compute_performance-ego0_min0.0018649589608452104
sim_compute_sim_state_max0.011976236193050892
sim_compute_sim_state_mean0.010111692045647583
sim_compute_sim_state_median0.011264918101938613
sim_compute_sim_state_min0.005940695785662217
sim_render-ego0_max0.004368365643743032
sim_render-ego0_mean0.004093363733480848
sim_render-ego0_median0.0041463032481695115
sim_render-ego0_min0.0037124827938413343
simulation-passed1
step_physics_max0.1276365461803618
step_physics_mean0.11190827555015335
step_physics_median0.1142051696099668
step_physics_min0.09158621680031807
survival_time_max59.99999999999873
survival_time_mean41.42499999999963
survival_time_min25.00000000000022
No reset possible
5727611313Moustafa Elarabitemplate-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simhost-erroryesnogpu-prod-020:01:05
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5712111017Melisande Tengexercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-020:38:47
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median5.295215014769427
survival_time_median59.99999999999873
deviation-center-line_median2.837320356470493
in-drivable-lane_median4.274999999999892


other stats
agent_compute-ego0_max0.03565964512185788
agent_compute-ego0_mean0.030904525622638637
agent_compute-ego0_median0.03406660532971206
agent_compute-ego0_min0.019825246709272527
complete-iteration_max0.2417684381550099
complete-iteration_mean0.2121827340840698
complete-iteration_median0.21141290763931211
complete-iteration_min0.184136682902645
deviation-center-line_max3.821740626642996
deviation-center-line_mean2.9063145184532475
deviation-center-line_min2.1288767342290087
deviation-heading_max20.293331801522996
deviation-heading_mean14.790328017281617
deviation-heading_median14.336060700967735
deviation-heading_min10.195858865667995
driven_any_max9.699258585782962
driven_any_mean9.299283473998472
driven_any_median9.27439180192301
driven_any_min8.949091706364904
driven_lanedir_consec_max9.283751511714373
driven_lanedir_consec_mean5.881992521573319
driven_lanedir_consec_min3.653788545040048
driven_lanedir_max9.283751511714373
driven_lanedir_mean8.192043664586846
driven_lanedir_median8.128125426895073
driven_lanedir_min7.228172292842862
get_duckie_state_max1.670518187460157e-06
get_duckie_state_mean1.4210819304733848e-06
get_duckie_state_median1.3897162889263017e-06
get_duckie_state_min1.234376956580779e-06
get_robot_state_max0.0040435800941460934
get_robot_state_mean0.003810104184305539
get_robot_state_median0.003853355716606858
get_robot_state_min0.003490125209862346
get_state_dump_max0.005027233809853077
get_state_dump_mean0.004836675279047169
get_state_dump_median0.004914961488518886
get_state_dump_min0.00448954432929783
get_ui_image_max0.03726624052887852
get_ui_image_mean0.031049366150172328
get_ui_image_median0.03019513248106919
get_ui_image_min0.0265409591096724
in-drivable-lane_max5.600000000000014
in-drivable-lane_mean3.849999999999948
in-drivable-lane_min1.249999999999993
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 9.699258585782962, "get_ui_image": 0.0265409591096724, "step_physics": 0.10046932957353044, "survival_time": 59.99999999999873, "driven_lanedir": 9.283751511714373, "get_state_dump": 0.00448954432929783, "get_robot_state": 0.003490125209862346, "sim_render-ego0": 0.003647726838733632, "get_duckie_state": 1.234376956580779e-06, "in-drivable-lane": 1.249999999999993, "deviation-heading": 10.195858865667995, "agent_compute-ego0": 0.03294493792753831, "complete-iteration": 0.184136682902645, "set_robot_commands": 0.002074853665227199, "deviation-center-line": 2.1288767342290087, "driven_lanedir_consec": 9.283751511714373, "sim_compute_sim_state": 0.008577577279668168, "sim_compute_performance-ego0": 0.00181997666053232}, "LF-norm-zigzag-000-ego0": {"driven_any": 9.193799149849546, "get_ui_image": 0.03726624052887852, "step_physics": 0.13912650369585405, "survival_time": 59.99999999999873, "driven_lanedir": 8.376135453619224, "get_state_dump": 0.004913242929285511, "get_robot_state": 0.0039095439879125995, "sim_render-ego0": 0.00410074576251612, "get_duckie_state": 1.513094429568784e-06, "in-drivable-lane": 2.9999999999998757, "deviation-heading": 12.52467172698044, "agent_compute-ego0": 0.03565964512185788, "complete-iteration": 0.2417684381550099, "set_robot_commands": 0.002362675114932604, "deviation-center-line": 2.950672518505998, "driven_lanedir_consec": 5.618959469088649, "sim_compute_sim_state": 0.012190763797490027, "sim_compute_performance-ego0": 0.002143526355193914}, "LF-norm-techtrack-000-ego0": {"driven_any": 8.949091706364904, "get_ui_image": 0.0328771007944404, "step_physics": 0.13898134767562523, "survival_time": 59.99999999999873, "driven_lanedir": 7.228172292842862, "get_state_dump": 0.004916680047752259, "get_robot_state": 0.003797167445301117, "sim_render-ego0": 0.0039850722542412575, "get_duckie_state": 1.2663381482838195e-06, "in-drivable-lane": 5.549999999999909, "deviation-heading": 20.293331801522996, "agent_compute-ego0": 0.019825246709272527, "complete-iteration": 0.22076502171880896, "set_robot_commands": 0.0022565585985271064, "deviation-center-line": 3.821740626642996, "driven_lanedir_consec": 4.971470560450205, "sim_compute_sim_state": 0.011957525909195138, "sim_compute_performance-ego0": 0.0020777029756900173}, "LF-norm-small_loop-000-ego0": {"driven_any": 9.35498445399647, "get_ui_image": 0.02751316416769798, "step_physics": 0.1149933633955194, "survival_time": 59.99999999999873, "driven_lanedir": 7.880115400170923, "get_state_dump": 0.005027233809853077, "get_robot_state": 0.0040435800941460934, "sim_render-ego0": 0.004173626609884829, "get_duckie_state": 1.670518187460157e-06, "in-drivable-lane": 5.600000000000014, "deviation-heading": 16.147449674955034, "agent_compute-ego0": 0.03518827273188582, "complete-iteration": 0.2020607935598153, "set_robot_commands": 0.002378405778235341, "deviation-center-line": 2.7239681944349887, "driven_lanedir_consec": 3.653788545040048, "sim_compute_sim_state": 0.006450233014795206, "sim_compute_performance-ego0": 0.002193042578844107}}
set_robot_commands_max0.002378405778235341
set_robot_commands_mean0.0022681232892305626
set_robot_commands_median0.002309616856729855
set_robot_commands_min0.002074853665227199
sim_compute_performance-ego0_max0.002193042578844107
sim_compute_performance-ego0_mean0.0020585621425650894
sim_compute_performance-ego0_median0.0021106146654419657
sim_compute_performance-ego0_min0.00181997666053232
sim_compute_sim_state_max0.012190763797490027
sim_compute_sim_state_mean0.009794025000287136
sim_compute_sim_state_median0.010267551594431651
sim_compute_sim_state_min0.006450233014795206
sim_render-ego0_max0.004173626609884829
sim_render-ego0_mean0.00397679286634396
sim_render-ego0_median0.004042909008378689
sim_render-ego0_min0.003647726838733632
simulation-passed1
step_physics_max0.13912650369585405
step_physics_mean0.12339263608513228
step_physics_median0.1269873555355723
step_physics_min0.10046932957353044
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
5708911312Moustafa Elarabitemplate-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-020:00:45
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5706711283Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:51
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5690411063Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-020:34:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median7.471073770401649
survival_time_median59.99999999999873
deviation-center-line_median3.412372932990175
in-drivable-lane_median0.0


other stats
agent_compute-ego0_max0.012493553407781823
agent_compute-ego0_mean0.012127437659048418
agent_compute-ego0_median0.01216750101284818
agent_compute-ego0_min0.01168119520271549
complete-iteration_max0.1869495407330007
complete-iteration_mean0.1688414659726431
complete-iteration_median0.1695498208221448
complete-iteration_min0.1493166815132821
deviation-center-line_max3.997769033908204
deviation-center-line_mean3.4013647750752285
deviation-center-line_min2.782944200412361
deviation-heading_max10.93796726089042
deviation-heading_mean8.771131647614387
deviation-heading_median8.861775431061844
deviation-heading_min6.4230084674434424
driven_any_max8.481679065412935
driven_any_mean7.714378215090166
driven_any_median7.575401888252175
driven_any_min7.225030018443382
driven_lanedir_consec_max8.377128793537313
driven_lanedir_consec_mean7.592491936520633
driven_lanedir_consec_min7.050691411741925
driven_lanedir_max8.377128793537313
driven_lanedir_mean7.592491936520633
driven_lanedir_median7.471073770401649
driven_lanedir_min7.050691411741925
get_duckie_state_max1.4799421375538288e-06
get_duckie_state_mean1.3984013953673452e-06
get_duckie_state_median1.3951754986892433e-06
get_duckie_state_min1.3233124465370654e-06
get_robot_state_max0.003723388508297224
get_robot_state_mean0.003647327820129141
get_robot_state_median0.0036485790710068062
get_robot_state_min0.0035687646302057247
get_state_dump_max0.00469178303790827
get_state_dump_mean0.004604771472730803
get_state_dump_median0.004583164317522518
get_state_dump_min0.004560974217969908
get_ui_image_max0.03461554564603858
get_ui_image_mean0.029919145456658712
get_ui_image_median0.029639572227725775
get_ui_image_min0.025781891725144716
in-drivable-lane_max0.0
in-drivable-lane_mean0.0
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 8.481679065412935, "get_ui_image": 0.028378403653312383, "step_physics": 0.09511772758458475, "survival_time": 59.99999999999873, "driven_lanedir": 8.377128793537313, "get_state_dump": 0.00469178303790827, "get_robot_state": 0.003723388508297224, "sim_render-ego0": 0.003937690085316578, "get_duckie_state": 1.4799421375538288e-06, "in-drivable-lane": 0.0, "deviation-heading": 7.3527758495261235, "agent_compute-ego0": 0.012273299902503832, "complete-iteration": 0.16206278987570072, "set_robot_commands": 0.0022695259091062014, "deviation-center-line": 3.046179935310286, "driven_lanedir_consec": 8.377128793537313, "sim_compute_sim_state": 0.009541487515121575, "sim_compute_performance-ego0": 0.0020332098205718867}, "LF-norm-zigzag-000-ego0": {"driven_any": 7.420709318876207, "get_ui_image": 0.03461554564603858, "step_physics": 0.11182120340650624, "survival_time": 59.99999999999873, "driven_lanedir": 7.268361005900437, "get_state_dump": 0.004560974217969908, "get_robot_state": 0.003611830648633463, "sim_render-ego0": 0.00383885377253422, "get_duckie_state": 1.4027191339980354e-06, "in-drivable-lane": 0.0, "deviation-heading": 10.370775012597564, "agent_compute-ego0": 0.012493553407781823, "complete-iteration": 0.1869495407330007, "set_robot_commands": 0.0021757799620235295, "deviation-center-line": 3.997769033908204, "driven_lanedir_consec": 7.268361005900437, "sim_compute_sim_state": 0.011781083852623424, "sim_compute_performance-ego0": 0.0019580172460144703}, "LF-norm-techtrack-000-ego0": {"driven_any": 7.225030018443382, "get_ui_image": 0.030900740802139167, "step_physics": 0.1056625217720432, "survival_time": 59.99999999999873, "driven_lanedir": 7.050691411741925, "get_state_dump": 0.004573736460778636, "get_robot_state": 0.0035687646302057247, "sim_render-ego0": 0.0037506136866433735, "get_duckie_state": 1.3233124465370654e-06, "in-drivable-lane": 0.0, "deviation-heading": 10.93796726089042, "agent_compute-ego0": 0.012061702123192526, "complete-iteration": 0.17703685176858894, "set_robot_commands": 0.0021021485229415957, "deviation-center-line": 3.778565930670064, "driven_lanedir_consec": 7.050691411741925, "sim_compute_sim_state": 0.01238094737984358, "sim_compute_performance-ego0": 0.0019452194687131048}, "LF-norm-small_loop-000-ego0": {"driven_any": 7.730094457628143, "get_ui_image": 0.025781891725144716, "step_physics": 0.08922387777419015, "survival_time": 59.99999999999873, "driven_lanedir": 7.67378653490286, "get_state_dump": 0.004592592174266399, "get_robot_state": 0.0036853274933801505, "sim_render-ego0": 0.0038771899316233462, "get_duckie_state": 1.3876318633804511e-06, "in-drivable-lane": 0.0, "deviation-heading": 6.4230084674434424, "agent_compute-ego0": 0.01168119520271549, "complete-iteration": 0.1493166815132821, "set_robot_commands": 0.0022472128284463876, "deviation-center-line": 2.782944200412361, "driven_lanedir_consec": 7.67378653490286, "sim_compute_sim_state": 0.006143889161172656, "sim_compute_performance-ego0": 0.0019933919326947393}}
set_robot_commands_max0.0022695259091062014
set_robot_commands_mean0.0021986668056294288
set_robot_commands_median0.0022114963952349588
set_robot_commands_min0.0021021485229415957
sim_compute_performance-ego0_max0.0020332098205718867
sim_compute_performance-ego0_mean0.00198245961699855
sim_compute_performance-ego0_median0.001975704589354605
sim_compute_performance-ego0_min0.0019452194687131048
sim_compute_sim_state_max0.01238094737984358
sim_compute_sim_state_mean0.009961851977190309
sim_compute_sim_state_median0.0106612856838725
sim_compute_sim_state_min0.006143889161172656
sim_render-ego0_max0.003937690085316578
sim_render-ego0_mean0.0038510868690293794
sim_render-ego0_median0.003858021852078783
sim_render-ego0_min0.0037506136866433735
simulation-passed1
step_physics_max0.11182120340650624
step_physics_mean0.1004563326343311
step_physics_median0.100390124678314
step_physics_min0.08922387777419015
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
5688611425Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-020:01:25
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5682311077Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-020:09:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.3149021380472715
survival_time_median9.975000000000009
deviation-center-line_median0.12188253267766544
in-drivable-lane_median7.975000000000006


other stats
agent_compute-ego0_max0.013596708391919548
agent_compute-ego0_mean0.01309838720738422
agent_compute-ego0_median0.013108965481791432
agent_compute-ego0_min0.012578909474034464
complete-iteration_max0.2236579524146186
complete-iteration_mean0.18133061106496615
complete-iteration_median0.17384719452142322
complete-iteration_min0.15397010280239967
deviation-center-line_max1.1188749840889167
deviation-center-line_mean0.36127329964960586
deviation-center-line_min0.08245314915417581
deviation-heading_max4.524131338849149
deviation-heading_mean1.6331260530921736
deviation-heading_median0.722766321005672
deviation-heading_min0.5628402315082003
driven_any_max7.161388646499181
driven_any_mean2.807809807316765
driven_any_median1.7316424656440046
driven_any_min0.6065656514798712
driven_lanedir_consec_max1.9988457680440983
driven_lanedir_consec_mean0.6993296660964883
driven_lanedir_consec_min0.16866862024731155
driven_lanedir_max1.9988457680440983
driven_lanedir_mean0.6993296660964883
driven_lanedir_median0.3149021380472715
driven_lanedir_min0.16866862024731155
get_duckie_state_max1.3820627793908757e-06
get_duckie_state_mean1.3581731085006907e-06
get_duckie_state_median1.3556823983478518e-06
get_duckie_state_min1.3392648579161844e-06
get_robot_state_max0.003743806624795026
get_robot_state_mean0.003682958979972027
get_robot_state_median0.0036984952494417823
get_robot_state_min0.0035910387962095197
get_state_dump_max0.0048602111836805705
get_state_dump_mean0.004636162836836427
get_state_dump_median0.0046250102043261885
get_state_dump_min0.004434419755012758
get_ui_image_max0.03581649285775644
get_ui_image_mean0.0306954151068364
get_ui_image_median0.02988005105961423
get_ui_image_min0.027205065450360697
in-drivable-lane_max24.64999999999976
in-drivable-lane_mean10.799999999999942
in-drivable-lane_min2.5999999999999934
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 7.161388646499181, "get_ui_image": 0.027205065450360697, "step_physics": 0.0888515333975515, "survival_time": 38.69999999999994, "driven_lanedir": 1.9988457680440983, "get_state_dump": 0.004434419755012758, "get_robot_state": 0.0035910387962095197, "sim_render-ego0": 0.003831696048859627, "get_duckie_state": 1.3677535518523185e-06, "in-drivable-lane": 24.64999999999976, "deviation-heading": 4.524131338849149, "agent_compute-ego0": 0.012578909474034464, "complete-iteration": 0.15397010280239967, "set_robot_commands": 0.002193208509875882, "deviation-center-line": 1.1188749840889167, "driven_lanedir_consec": 1.9988457680440983, "sim_compute_sim_state": 0.00922894816244802, "sim_compute_performance-ego0": 0.0019704434179490613}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.6065656514798712, "get_ui_image": 0.03581649285775644, "step_physics": 0.14740624251189055, "survival_time": 3.999999999999994, "driven_lanedir": 0.16866862024731155, "get_state_dump": 0.004598329096664617, "get_robot_state": 0.003689144864494418, "sim_render-ego0": 0.00405388996924883, "get_duckie_state": 1.3392648579161844e-06, "in-drivable-lane": 2.5999999999999934, "deviation-heading": 0.8045328385580822, "agent_compute-ego0": 0.013596708391919548, "complete-iteration": 0.2236579524146186, "set_robot_commands": 0.002302266933299877, "deviation-center-line": 0.08245314915417581, "driven_lanedir_consec": 0.16866862024731155, "sim_compute_sim_state": 0.01005607475469142, "sim_compute_performance-ego0": 0.002050020076610424}, "LF-norm-techtrack-000-ego0": {"driven_any": 1.54645675618512, "get_ui_image": 0.03228139494829637, "step_physics": 0.1137640182984704, "survival_time": 9.299999999999995, "driven_lanedir": 0.32003220078866734, "get_state_dump": 0.0048602111836805705, "get_robot_state": 0.003743806624795026, "sim_render-ego0": 0.0041004642445773365, "get_duckie_state": 1.3820627793908757e-06, "in-drivable-lane": 7.2499999999999964, "deviation-heading": 0.6409998034532617, "agent_compute-ego0": 0.013404980062800931, "complete-iteration": 0.18536867814905503, "set_robot_commands": 0.002286854912252987, "deviation-center-line": 0.1176358625747966, "driven_lanedir_consec": 0.32003220078866734, "sim_compute_sim_state": 0.008752184118179077, "sim_compute_performance-ego0": 0.002082041878113772}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.9168281751028893, "get_ui_image": 0.027478707170932093, "step_physics": 0.10020480646151247, "survival_time": 10.650000000000016, "driven_lanedir": 0.3097720753058757, "get_state_dump": 0.004651691311987761, "get_robot_state": 0.0037078456343891462, "sim_render-ego0": 0.003883788518816511, "get_duckie_state": 1.3436112448433848e-06, "in-drivable-lane": 8.700000000000015, "deviation-heading": 0.5628402315082003, "agent_compute-ego0": 0.012812950900781935, "complete-iteration": 0.1623257108937914, "set_robot_commands": 0.002267493265811528, "deviation-center-line": 0.12612920278053427, "driven_lanedir_consec": 0.3097720753058757, "sim_compute_sim_state": 0.005268570418669799, "sim_compute_performance-ego0": 0.0019633357769975037}}
set_robot_commands_max0.002302266933299877
set_robot_commands_mean0.0022624559053100687
set_robot_commands_median0.0022771740890322576
set_robot_commands_min0.002193208509875882
sim_compute_performance-ego0_max0.002082041878113772
sim_compute_performance-ego0_mean0.00201646028741769
sim_compute_performance-ego0_median0.0020102317472797428
sim_compute_performance-ego0_min0.0019633357769975037
sim_compute_sim_state_max0.01005607475469142
sim_compute_sim_state_mean0.008326444363497078
sim_compute_sim_state_median0.00899056614031355
sim_compute_sim_state_min0.005268570418669799
sim_render-ego0_max0.0041004642445773365
sim_render-ego0_mean0.003967459695375576
sim_render-ego0_median0.003968839244032671
sim_render-ego0_min0.003831696048859627
simulation-passed1
step_physics_max0.14740624251189055
step_physics_mean0.11255665016735623
step_physics_median0.10698441237999144
step_physics_min0.0888515333975515
survival_time_max38.69999999999994
survival_time_mean15.662499999999987
survival_time_min3.999999999999994
No reset possible
5679511282Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:02:18
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5675611286Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:04:08
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5674511286Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:44
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5673211278Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:03:08
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5670411427Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simhost-erroryesnogpu-prod-020:02:27
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5668211289Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:04:40
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5578911422Frank (Chude) Qian 🇨🇦challenge-aido_LF-baseline-behavior-cloningaido5-LFVI_multi-sim-validationLFVIv-simsuccessyesnogpu-prod-021:19:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median8.00075320404907
survival_time_median59.99999999999873
deviation-center-line_median2.8239357280733985
in-drivable-lane_median0.2499999999999898


other stats
agent_compute-ego0_max0.04328814851155785
agent_compute-ego0_mean0.04088998376876488
agent_compute-ego0_median0.04088998376876488
agent_compute-ego0_min0.03849181902597191
agent_compute-ego1_max0.043693094031201314
agent_compute-ego1_mean0.041192149441009955
agent_compute-ego1_median0.041192149441009955
agent_compute-ego1_min0.03869120485081859
agent_compute-ego2_max0.04338253110175724
agent_compute-ego2_mean0.04084059201509728
agent_compute-ego2_median0.04084059201509728
agent_compute-ego2_min0.03829865292843732
agent_compute-ego3_max0.04303700798854145
agent_compute-ego3_mean0.04081960850016064
agent_compute-ego3_median0.04081960850016064
agent_compute-ego3_min0.03860220901177984
complete-iteration_max1.1046263337830124
complete-iteration_mean1.0426254363778629
complete-iteration_median1.0426254363778629
complete-iteration_min0.9806245389727132
deviation-center-line_max6.993703781443878
deviation-center-line_mean3.936741498504513
deviation-center-line_min1.4439738685565429
deviation-heading_max42.18086494270505
deviation-heading_mean21.323538483824755
deviation-heading_median13.496171805602964
deviation-heading_min8.910009242327769
driven_any_max14.590349051823852
driven_any_mean7.927626305348715
driven_any_median8.415174702140996
driven_any_min0.49155627944869673
driven_lanedir_consec_max14.286955591091672
driven_lanedir_consec_mean7.618726402833628
driven_lanedir_consec_min0.4720865370347065
driven_lanedir_max14.286955591091672
driven_lanedir_mean7.64089806951685
driven_lanedir_median8.000755545699063
driven_lanedir_min0.4720865370347065
get_duckie_state_max1.4011310002488162e-06
get_duckie_state_mean1.2663381482838195e-06
get_duckie_state_median1.2663381482838195e-06
get_duckie_state_min1.1315452963188229e-06
get_robot_state_max0.013816469217915022
get_robot_state_mean0.01316477257842128
get_robot_state_median0.01316477257842128
get_robot_state_min0.012513075938927542
get_state_dump_max0.00905766614172282
get_state_dump_mean0.008651897472505467
get_state_dump_median0.008651897472505467
get_state_dump_min0.008246128803288113
get_ui_image_max0.04908133347167461
get_ui_image_mean0.047140783612476006
get_ui_image_median0.047140783612476006
get_ui_image_min0.0452002337532774
in-drivable-lane_max1.2999999999999954
in-drivable-lane_mean0.4374999999999939
in-drivable-lane_min0.0
per-episodes
details{"LFVI_multi-norm-4way-000-ego0": {"driven_any": 14.590274250259988, "get_ui_image": 0.0452002337532774, "step_physics": 0.7106763957243577, "survival_time": 59.99999999999873, "driven_lanedir": 14.285870115338705, "get_state_dump": 0.008246128803288113, "get_robot_state": 0.012513075938927542, "sim_render-ego0": 0.0033630697455235463, "sim_render-ego1": 0.0033267646109829538, "sim_render-ego2": 0.0032966218324227696, "sim_render-ego3": 0.0032984630749882705, "get_duckie_state": 1.1315452963188229e-06, "in-drivable-lane": 0.0, "deviation-heading": 10.65141140471916, "agent_compute-ego0": 0.03849181902597191, "agent_compute-ego1": 0.03869120485081859, "agent_compute-ego2": 0.03829865292843732, "agent_compute-ego3": 0.03860220901177984, "complete-iteration": 0.9806245389727132, "set_robot_commands": 0.0020629726381325704, "deviation-center-line": 2.869488261734872, "driven_lanedir_consec": 14.285870115338705, "sim_compute_sim_state": 0.021974682311630567, "sim_compute_performance-ego0": 0.00177611538412966, "sim_compute_performance-ego1": 0.001628550760553441, "sim_compute_performance-ego2": 0.0016562168445317176, "sim_compute_performance-ego3": 0.001628711162062112}, "LFVI_multi-norm-4way-000-ego1": {"driven_any": 2.2400890986044524, "get_ui_image": 0.0452002337532774, "step_physics": 0.7106763957243577, "survival_time": 59.99999999999873, "driven_lanedir": 1.900791696912732, "get_state_dump": 0.008246128803288113, "get_robot_state": 0.012513075938927542, "sim_render-ego0": 0.0033630697455235463, "sim_render-ego1": 0.0033267646109829538, "sim_render-ego2": 0.0032966218324227696, "sim_render-ego3": 0.0032984630749882705, "get_duckie_state": 1.1315452963188229e-06, "in-drivable-lane": 0.8999999999999968, "deviation-heading": 38.71500900662983, "agent_compute-ego0": 0.03849181902597191, "agent_compute-ego1": 0.03869120485081859, "agent_compute-ego2": 0.03829865292843732, "agent_compute-ego3": 0.03860220901177984, "complete-iteration": 0.9806245389727132, "set_robot_commands": 0.0020629726381325704, "deviation-center-line": 6.000862773390881, "driven_lanedir_consec": 1.900791696912732, "sim_compute_sim_state": 0.021974682311630567, "sim_compute_performance-ego0": 0.00177611538412966, "sim_compute_performance-ego1": 0.001628550760553441, "sim_compute_performance-ego2": 0.0016562168445317176, "sim_compute_performance-ego3": 0.001628711162062112}, "LFVI_multi-norm-4way-000-ego2": {"driven_any": 14.59026030567754, "get_ui_image": 0.0452002337532774, "step_physics": 0.7106763957243577, "survival_time": 59.99999999999873, "driven_lanedir": 14.286955591091672, "get_state_dump": 0.008246128803288113, "get_robot_state": 0.012513075938927542, "sim_render-ego0": 0.0033630697455235463, "sim_render-ego1": 0.0033267646109829538, "sim_render-ego2": 0.0032966218324227696, "sim_render-ego3": 0.0032984630749882705, "get_duckie_state": 1.1315452963188229e-06, "in-drivable-lane": 0.0, "deviation-heading": 10.77473756366386, "agent_compute-ego0": 0.03849181902597191, "agent_compute-ego1": 0.03869120485081859, "agent_compute-ego2": 0.03829865292843732, "agent_compute-ego3": 0.03860220901177984, "complete-iteration": 0.9806245389727132, "set_robot_commands": 0.0020629726381325704, "deviation-center-line": 2.7783831944119246, "driven_lanedir_consec": 14.286955591091672, "sim_compute_sim_state": 0.021974682311630567, "sim_compute_performance-ego0": 0.00177611538412966, "sim_compute_performance-ego1": 0.001628550760553441, "sim_compute_performance-ego2": 0.0016562168445317176, "sim_compute_performance-ego3": 0.001628711162062112}, "LFVI_multi-norm-4way-000-ego3": {"driven_any": 0.9929039380973048, "get_ui_image": 0.0452002337532774, "step_physics": 0.7106763957243577, "survival_time": 59.99999999999873, "driven_lanedir": 0.8491677542940025, "get_state_dump": 0.008246128803288113, "get_robot_state": 0.012513075938927542, "sim_render-ego0": 0.0033630697455235463, "sim_render-ego1": 0.0033267646109829538, "sim_render-ego2": 0.0032966218324227696, "sim_render-ego3": 0.0032984630749882705, "get_duckie_state": 1.1315452963188229e-06, "in-drivable-lane": 0.14999999999999947, "deviation-heading": 42.18086494270505, "agent_compute-ego0": 0.03849181902597191, "agent_compute-ego1": 0.03869120485081859, "agent_compute-ego2": 0.03829865292843732, "agent_compute-ego3": 0.03860220901177984, "complete-iteration": 0.9806245389727132, "set_robot_commands": 0.0020629726381325704, "deviation-center-line": 6.993703781443878, "driven_lanedir_consec": 0.6900128436534608, "sim_compute_sim_state": 0.021974682311630567, "sim_compute_performance-ego0": 0.00177611538412966, "sim_compute_performance-ego1": 0.001628550760553441, "sim_compute_performance-ego2": 0.0016562168445317176, "sim_compute_performance-ego3": 0.001628711162062112}, "LFVI_multi-norm-udem1-000-ego0": {"driven_any": 14.590349051823852, "get_ui_image": 0.04908133347167461, "step_physics": 0.8043117660169895, "survival_time": 59.99999999999873, "driven_lanedir": 14.26626045879052, "get_state_dump": 0.00905766614172282, "get_robot_state": 0.013816469217915022, "sim_render-ego0": 0.003742545173130464, "sim_render-ego1": 0.0037882740948222064, "sim_render-ego2": 0.003741088060415555, "sim_render-ego3": 0.0036643553931548336, "get_duckie_state": 1.4011310002488162e-06, "in-drivable-lane": 0.3499999999999801, "deviation-heading": 8.910009242327769, "agent_compute-ego0": 0.04328814851155785, "agent_compute-ego1": 0.043693094031201314, "agent_compute-ego2": 0.04338253110175724, "agent_compute-ego3": 0.04303700798854145, "complete-iteration": 1.1046263337830124, "set_robot_commands": 0.002422880471298637, "deviation-center-line": 2.4063982353198834, "driven_lanedir_consec": 14.266258477145486, "sim_compute_sim_state": 0.02301007127086884, "sim_compute_performance-ego0": 0.001990572796773156, "sim_compute_performance-ego1": 0.001897744790997533, "sim_compute_performance-ego2": 0.0018679750253516969, "sim_compute_performance-ego3": 0.001911321349386172}, "LFVI_multi-norm-udem1-000-ego1": {"driven_any": 14.590327050880797, "get_ui_image": 0.04908133347167461, "step_physics": 0.8043117660169895, "survival_time": 59.99999999999873, "driven_lanedir": 14.100719394485392, "get_state_dump": 0.00905766614172282, "get_robot_state": 0.013816469217915022, "sim_render-ego0": 0.003742545173130464, "sim_render-ego1": 0.0037882740948222064, "sim_render-ego2": 0.003741088060415555, "sim_render-ego3": 0.0036643553931548336, "get_duckie_state": 1.4011310002488162e-06, "in-drivable-lane": 0.7999999999999794, "deviation-heading": 9.784630456195652, "agent_compute-ego0": 0.04328814851155785, "agent_compute-ego1": 0.043693094031201314, "agent_compute-ego2": 0.04338253110175724, "agent_compute-ego3": 0.04303700798854145, "complete-iteration": 1.1046263337830124, "set_robot_commands": 0.002422880471298637, "deviation-center-line": 2.7297678588323686, "driven_lanedir_consec": 14.100714711185407, "sim_compute_sim_state": 0.02301007127086884, "sim_compute_performance-ego0": 0.001990572796773156, "sim_compute_performance-ego1": 0.001897744790997533, "sim_compute_performance-ego2": 0.0018679750253516969, "sim_compute_performance-ego3": 0.001911321349386172}, "LFVI_multi-norm-udem1-000-ego2": {"driven_any": 0.49155627944869673, "get_ui_image": 0.04908133347167461, "step_physics": 0.8043117660169895, "survival_time": 59.99999999999873, "driven_lanedir": 0.4720865370347065, "get_state_dump": 0.00905766614172282, "get_robot_state": 0.013816469217915022, "sim_render-ego0": 0.003742545173130464, "sim_render-ego1": 0.0037882740948222064, "sim_render-ego2": 0.003741088060415555, "sim_render-ego3": 0.0036643553931548336, "get_duckie_state": 1.4011310002488162e-06, "in-drivable-lane": 0.0, "deviation-heading": 33.35403920681466, "agent_compute-ego0": 0.04328814851155785, "agent_compute-ego1": 0.043693094031201314, "agent_compute-ego2": 0.04338253110175724, "agent_compute-ego3": 0.04303700798854145, "complete-iteration": 1.1046263337830124, "set_robot_commands": 0.002422880471298637, "deviation-center-line": 6.271354014345749, "driven_lanedir_consec": 0.4720865370347065, "sim_compute_sim_state": 0.02301007127086884, "sim_compute_performance-ego0": 0.001990572796773156, "sim_compute_performance-ego1": 0.001897744790997533, "sim_compute_performance-ego2": 0.0018679750253516969, "sim_compute_performance-ego3": 0.001911321349386172}, "LFVI_multi-norm-udem1-000-ego3": {"driven_any": 1.3352504679970854, "get_ui_image": 0.04908133347167461, "step_physics": 0.8043117660169895, "survival_time": 59.99999999999873, "driven_lanedir": 0.9653330081870644, "get_state_dump": 0.00905766614172282, "get_robot_state": 0.013816469217915022, "sim_render-ego0": 0.003742545173130464, "sim_render-ego1": 0.0037882740948222064, "sim_render-ego2": 0.003741088060415555, "sim_render-ego3": 0.0036643553931548336, "get_duckie_state": 1.4011310002488162e-06, "in-drivable-lane": 1.2999999999999954, "deviation-heading": 16.21760604754207, "agent_compute-ego0": 0.04328814851155785, "agent_compute-ego1": 0.043693094031201314, "agent_compute-ego2": 0.04338253110175724, "agent_compute-ego3": 0.04303700798854145, "complete-iteration": 1.1046263337830124, "set_robot_commands": 0.002422880471298637, "deviation-center-line": 1.4439738685565429, "driven_lanedir_consec": 0.9471212503068523, "sim_compute_sim_state": 0.02301007127086884, "sim_compute_performance-ego0": 0.001990572796773156, "sim_compute_performance-ego1": 0.001897744790997533, "sim_compute_performance-ego2": 0.0018679750253516969, "sim_compute_performance-ego3": 0.001911321349386172}}
set_robot_commands_max0.002422880471298637
set_robot_commands_mean0.002242926554715603
set_robot_commands_median0.002242926554715603
set_robot_commands_min0.0020629726381325704
sim_compute_performance-ego0_max0.001990572796773156
sim_compute_performance-ego0_mean0.001883344090451408
sim_compute_performance-ego0_median0.001883344090451408
sim_compute_performance-ego0_min0.00177611538412966
sim_compute_performance-ego1_max0.001897744790997533
sim_compute_performance-ego1_mean0.0017631477757754869
sim_compute_performance-ego1_median0.0017631477757754869
sim_compute_performance-ego1_min0.001628550760553441
sim_compute_performance-ego2_max0.0018679750253516969
sim_compute_performance-ego2_mean0.0017620959349417072
sim_compute_performance-ego2_median0.0017620959349417072
sim_compute_performance-ego2_min0.0016562168445317176
sim_compute_performance-ego3_max0.001911321349386172
sim_compute_performance-ego3_mean0.001770016255724142
sim_compute_performance-ego3_median0.001770016255724142
sim_compute_performance-ego3_min0.001628711162062112
sim_compute_sim_state_max0.02301007127086884
sim_compute_sim_state_mean0.022492376791249703
sim_compute_sim_state_median0.022492376791249703
sim_compute_sim_state_min0.021974682311630567
sim_render-ego0_max0.003742545173130464
sim_render-ego0_mean0.003552807459327005
sim_render-ego0_median0.003552807459327005
sim_render-ego0_min0.0033630697455235463
sim_render-ego1_max0.0037882740948222064
sim_render-ego1_mean0.00355751935290258
sim_render-ego1_median0.00355751935290258
sim_render-ego1_min0.0033267646109829538
sim_render-ego2_max0.003741088060415555
sim_render-ego2_mean0.0035188549464191627
sim_render-ego2_median0.0035188549464191627
sim_render-ego2_min0.0032966218324227696
sim_render-ego3_max0.0036643553931548336
sim_render-ego3_mean0.003481409234071552
sim_render-ego3_median0.003481409234071552
sim_render-ego3_min0.0032984630749882705
simulation-passed1
step_physics_max0.8043117660169895
step_physics_mean0.7574940808706736
step_physics_median0.7574940808706736
step_physics_min0.7106763957243577
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
5574011290Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:02:14
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5565111281Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:02:48
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 49, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 49, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5561111284Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:33
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5557311284Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:45
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5549511285Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:41
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5546411291Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:54
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5537711306Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:03:35
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5535811299Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:32
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5530011307Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:02:06
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5527911310Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:43
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5526111310Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:35
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5524011287Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:44
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5522711301Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-020:00:33
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/