Duckietown Challenges Home Challenges Submissions

Evaluator 4879

ID4879
evaluatornogpu-prod-06
ownerI don't have one 😀
machinenogpu-prod_14918bb0249f
processnogpu-prod-06_14918bb0249f
version6.2.5
first heard
last heard
statusinactive
# evaluating
# success23 53903
# timeout1 57566
# failed6 54347
# error
# aborted2 57167
# host-error32 53992
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
5756610892Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simtimeoutyesnogpu-prod-06----No reset possible
5749010909Fernanda Custodio Pereira do Carmo 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:25:45
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median1.8171032566912029
survival_time_median24.40000000000021
deviation-center-line_median0.7041099970934674
in-drivable-lane_median10.924999999999876


other stats
agent_compute-ego0_max0.04894446944170457
agent_compute-ego0_mean0.03185885455922059
agent_compute-ego0_median0.032593442961316915
agent_compute-ego0_min0.013304062872543971
complete-iteration_max0.22633791309839477
complete-iteration_mean0.1950060576912377
complete-iteration_median0.18765519773041772
complete-iteration_min0.17837592220572043
deviation-center-line_max4.146709536868032
deviation-center-line_mean1.4656895098135565
deviation-center-line_min0.30782850819925905
deviation-heading_max6.6261262788775595
deviation-heading_mean2.846857486040311
deviation-heading_median1.9645850609499915
deviation-heading_min0.8321335433837017
driven_any_max9.341813977848751
driven_any_mean4.828220262935595
driven_any_median3.668207128944021
driven_any_min2.6346528160055893
driven_lanedir_consec_max7.049027758667621
driven_lanedir_consec_mean2.7747748667728276
driven_lanedir_consec_min0.41586519504128505
driven_lanedir_max7.063275101902191
driven_lanedir_mean2.7783367025814703
driven_lanedir_median1.8171032566912029
driven_lanedir_min0.41586519504128505
get_duckie_state_max1.7186537145930816e-06
get_duckie_state_mean1.4698392864783164e-06
get_duckie_state_median1.443410823863786e-06
get_duckie_state_min1.2738817835926117e-06
get_robot_state_max0.003973466189787349
get_robot_state_mean0.003852132359728619
get_robot_state_median0.00385516439873872
get_robot_state_min0.003724734451649687
get_state_dump_max0.005227075206290046
get_state_dump_mean0.0049543202916168755
get_state_dump_median0.0049508203398809115
get_state_dump_min0.004688565280415633
get_ui_image_max0.03676365148574911
get_ui_image_mean0.031173494786699337
get_ui_image_median0.030775394523181912
get_ui_image_min0.0263795386146844
in-drivable-lane_max24.20000000000027
in-drivable-lane_mean13.025000000000029
in-drivable-lane_min6.050000000000086
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 3.1028734066666983, "get_ui_image": 0.030025137985924735, "step_physics": 0.1011370966474501, "survival_time": 20.80000000000016, "driven_lanedir": 2.227622656975215, "get_state_dump": 0.005227075206290046, "get_robot_state": 0.003965081070824493, "sim_render-ego0": 0.004252807127771903, "get_duckie_state": 1.4722490196319506e-06, "in-drivable-lane": 6.050000000000086, "deviation-heading": 1.3761508615291107, "agent_compute-ego0": 0.024510286980681566, "complete-iteration": 0.18509960460434144, "set_robot_commands": 0.0025272535191451333, "deviation-center-line": 0.5105671432119578, "driven_lanedir_consec": 2.227622656975215, "sim_compute_sim_state": 0.01097516133059129, "sim_compute_performance-ego0": 0.002373169366118434}, "LF-norm-zigzag-000-ego0": {"driven_any": 4.233540851221343, "get_ui_image": 0.03676365148574911, "step_physics": 0.11098017029583772, "survival_time": 28.000000000000263, "driven_lanedir": 0.41586519504128505, "get_state_dump": 0.005137035978456657, "get_robot_state": 0.003973466189787349, "sim_render-ego0": 0.004316151035873223, "get_duckie_state": 1.7186537145930816e-06, "in-drivable-lane": 24.20000000000027, "deviation-heading": 2.553019260370873, "agent_compute-ego0": 0.04894446944170457, "complete-iteration": 0.22633791309839477, "set_robot_commands": 0.0024775103026745365, "deviation-center-line": 0.30782850819925905, "driven_lanedir_consec": 0.41586519504128505, "sim_compute_sim_state": 0.01141168386965937, "sim_compute_performance-ego0": 0.0022233981712191713}, "LF-norm-techtrack-000-ego0": {"driven_any": 2.6346528160055893, "get_ui_image": 0.031525651060439086, "step_physics": 0.1060122513837469, "survival_time": 17.90000000000012, "driven_lanedir": 1.4065838564071904, "get_state_dump": 0.004764604701305166, "get_robot_state": 0.003745247726652947, "sim_render-ego0": 0.004056494216068874, "get_duckie_state": 1.4145726280956215e-06, "in-drivable-lane": 8.45000000000012, "deviation-heading": 0.8321335433837017, "agent_compute-ego0": 0.013304062872543971, "complete-iteration": 0.17837592220572043, "set_robot_commands": 0.0022762246782733206, "deviation-center-line": 0.8976528509749772, "driven_lanedir_consec": 1.4065838564071904, "sim_compute_sim_state": 0.01053402483629317, "sim_compute_performance-ego0": 0.0020594357780095262}, "LF-norm-small_loop-000-ego0": {"driven_any": 9.341813977848751, "get_ui_image": 0.0263795386146844, "step_physics": 0.1001656434617372, "survival_time": 59.99999999999873, "driven_lanedir": 7.063275101902191, "get_state_dump": 0.004688565280415633, "get_robot_state": 0.003724734451649687, "sim_render-ego0": 0.003918957849227816, "get_duckie_state": 1.2738817835926117e-06, "in-drivable-lane": 13.399999999999633, "deviation-heading": 6.6261262788775595, "agent_compute-ego0": 0.04067659894195226, "complete-iteration": 0.190210790856494, "set_robot_commands": 0.0023180519313637563, "deviation-center-line": 4.146709536868032, "driven_lanedir_consec": 7.049027758667621, "sim_compute_sim_state": 0.006260473464152696, "sim_compute_performance-ego0": 0.001985861002455941}}
set_robot_commands_max0.0025272535191451333
set_robot_commands_mean0.0023997601078641867
set_robot_commands_median0.0023977811170191464
set_robot_commands_min0.0022762246782733206
sim_compute_performance-ego0_max0.002373169366118434
sim_compute_performance-ego0_mean0.002160466079450768
sim_compute_performance-ego0_median0.002141416974614349
sim_compute_performance-ego0_min0.001985861002455941
sim_compute_sim_state_max0.01141168386965937
sim_compute_sim_state_mean0.009795335875174132
sim_compute_sim_state_median0.010754593083442229
sim_compute_sim_state_min0.006260473464152696
sim_render-ego0_max0.004316151035873223
sim_render-ego0_mean0.004136102557235454
sim_render-ego0_median0.0041546506719203884
sim_render-ego0_min0.003918957849227816
simulation-passed1
step_physics_max0.11098017029583772
step_physics_mean0.10457379044719298
step_physics_median0.1035746740155985
step_physics_min0.1001656434617372
survival_time_max59.99999999999873
survival_time_mean31.67499999999982
survival_time_min17.90000000000012
No reset possible
5742410925Philippe Reddy 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:11:15
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median1.397169406218369
survival_time_median13.850000000000062
deviation-center-line_median0.494063000130184
in-drivable-lane_median6.675000000000059


other stats
agent_compute-ego0_max0.013855758260508052
agent_compute-ego0_mean0.013427991984223423
agent_compute-ego0_median0.013471802856483433
agent_compute-ego0_min0.012912603963418786
complete-iteration_max0.2688659799510035
complete-iteration_mean0.19642762222081137
complete-iteration_median0.17818807453241342
complete-iteration_min0.16046835986741512
deviation-center-line_max1.0729963650784728
deviation-center-line_mean0.5535080306803141
deviation-center-line_min0.15290975738241577
deviation-heading_max5.195457861135293
deviation-heading_mean2.275521381638133
deviation-heading_median1.569646592351191
deviation-heading_min0.7673344807148553
driven_any_max6.741819457035245
driven_any_mean3.275042935619368
driven_any_median2.685516633454039
driven_any_min0.9873190185341488
driven_lanedir_consec_max2.475258595560404
driven_lanedir_consec_mean1.4658362027788523
driven_lanedir_consec_min0.593747403118267
driven_lanedir_max3.4166138835034428
driven_lanedir_mean1.701175024764612
driven_lanedir_median1.397169406218369
driven_lanedir_min0.593747403118267
get_duckie_state_max1.38940482304014e-06
get_duckie_state_mean1.3102349356532944e-06
get_duckie_state_median1.3040743919698948e-06
get_duckie_state_min1.2433861356332482e-06
get_robot_state_max0.00398971911134391
get_robot_state_mean0.0038351742600307897
get_robot_state_median0.003815538135077479
get_robot_state_min0.003719901658624292
get_state_dump_max0.004988645685130152
get_state_dump_mean0.004859882768794401
get_state_dump_median0.004859395689270107
get_state_dump_min0.0047320940115072345
get_ui_image_max0.0383415098848014
get_ui_image_mean0.03175118563100791
get_ui_image_median0.030223829992845775
get_ui_image_min0.028215572653538676
in-drivable-lane_max15.20000000000008
in-drivable-lane_mean7.687500000000049
in-drivable-lane_min2.199999999999995
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.4025058319112764, "get_ui_image": 0.028215572653538676, "step_physics": 0.09272925027338157, "survival_time": 12.500000000000044, "driven_lanedir": 2.03915493951993, "get_state_dump": 0.0048191480902561624, "get_robot_state": 0.003719901658624292, "sim_render-ego0": 0.00393573886369804, "get_duckie_state": 1.2433861356332482e-06, "in-drivable-lane": 2.250000000000032, "deviation-heading": 1.4489225324600572, "agent_compute-ego0": 0.012912603963418786, "complete-iteration": 0.16046835986741512, "set_robot_commands": 0.002227106892255198, "deviation-center-line": 0.6668262751642334, "driven_lanedir_consec": 2.03915493951993, "sim_compute_sim_state": 0.009791626873244329, "sim_compute_performance-ego0": 0.002033799767969139}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.9873190185341488, "get_ui_image": 0.0383415098848014, "step_physics": 0.18801129480888104, "survival_time": 5.749999999999988, "driven_lanedir": 0.593747403118267, "get_state_dump": 0.004988645685130152, "get_robot_state": 0.00398971911134391, "sim_render-ego0": 0.004165965935279583, "get_duckie_state": 1.38940482304014e-06, "in-drivable-lane": 2.199999999999995, "deviation-heading": 1.690370652242325, "agent_compute-ego0": 0.013817760451086635, "complete-iteration": 0.2688659799510035, "set_robot_commands": 0.0023361958306411215, "deviation-center-line": 0.32129972509613464, "driven_lanedir_consec": 0.593747403118267, "sim_compute_sim_state": 0.011008087931008177, "sim_compute_performance-ego0": 0.0021128818906586745}, "LF-norm-techtrack-000-ego0": {"driven_any": 6.741819457035245, "get_ui_image": 0.032123472597505955, "step_physics": 0.11756533438020998, "survival_time": 33.25000000000025, "driven_lanedir": 3.4166138835034428, "get_state_dump": 0.0047320940115072345, "get_robot_state": 0.003742048690268943, "sim_render-ego0": 0.003946192629702456, "get_duckie_state": 1.2644060381181965e-06, "in-drivable-lane": 15.20000000000008, "deviation-heading": 5.195457861135293, "agent_compute-ego0": 0.013125845261880228, "complete-iteration": 0.1926355719924331, "set_robot_commands": 0.0022334025070831942, "deviation-center-line": 1.0729963650784728, "driven_lanedir_consec": 2.475258595560404, "sim_compute_sim_state": 0.01307177651035893, "sim_compute_performance-ego0": 0.0020066166783238317}, "LF-norm-small_loop-000-ego0": {"driven_any": 2.968527434996801, "get_ui_image": 0.028324187388185595, "step_physics": 0.0982655900423644, "survival_time": 15.20000000000008, "driven_lanedir": 0.7551838729168079, "get_state_dump": 0.0048996432882840515, "get_robot_state": 0.0038890275798860144, "sim_render-ego0": 0.004130827012609263, "get_duckie_state": 1.3437427458215933e-06, "in-drivable-lane": 11.100000000000088, "deviation-heading": 0.7673344807148553, "agent_compute-ego0": 0.013855758260508052, "complete-iteration": 0.1637405770723937, "set_robot_commands": 0.0022993822566798475, "deviation-center-line": 0.15290975738241577, "driven_lanedir_consec": 0.7551838729168079, "sim_compute_sim_state": 0.0058958921276155065, "sim_compute_performance-ego0": 0.0020882645591360624}}
set_robot_commands_max0.0023361958306411215
set_robot_commands_mean0.0022740218716648403
set_robot_commands_median0.002266392381881521
set_robot_commands_min0.002227106892255198
sim_compute_performance-ego0_max0.0021128818906586745
sim_compute_performance-ego0_mean0.002060390724021927
sim_compute_performance-ego0_median0.0020610321635526007
sim_compute_performance-ego0_min0.0020066166783238317
sim_compute_sim_state_max0.01307177651035893
sim_compute_sim_state_mean0.009941845860556736
sim_compute_sim_state_median0.010399857402126253
sim_compute_sim_state_min0.0058958921276155065
sim_render-ego0_max0.004165965935279583
sim_render-ego0_mean0.004044681110322336
sim_render-ego0_median0.0040385098211558595
sim_render-ego0_min0.00393573886369804
simulation-passed1
step_physics_max0.18801129480888104
step_physics_mean0.12414286737620923
step_physics_median0.1079154622112872
step_physics_min0.09272925027338157
survival_time_max33.25000000000025
survival_time_mean16.67500000000009
survival_time_min5.749999999999988
No reset possible
5739710943Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simfailedyesnogpu-prod-060:03:53
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 218, in read_one
    msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 304, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 341, in read_until_over
    raise TimeoutError(msg) from None
TimeoutError: Timeout while reading output of node "ego0".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 563, in run_episode
    r: MsgReceived = await loop.run_in_executor(executor, f)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 135, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 281, in read_one
    raise TimeoutError(msg) from e
TimeoutError: Timeout detected on /fifos/ego0-out after 37 messages. Expected topic "commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 286, in main
    length_s = await run_episode(
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 567, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5735510954Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:08:20
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.45148801110635506
survival_time_median10.800000000000018
deviation-center-line_median0.25363504663675707
in-drivable-lane_median8.075000000000003


other stats
agent_compute-ego0_max0.012946488094978592
agent_compute-ego0_mean0.01236836630938753
agent_compute-ego0_median0.01233658609372792
agent_compute-ego0_min0.011853804955115683
complete-iteration_max0.19606345433455247
complete-iteration_mean0.1646771489529008
complete-iteration_median0.15971753812160622
complete-iteration_min0.14321006523383842
deviation-center-line_max0.7446003464316293
deviation-center-line_mean0.33415576478734504
deviation-center-line_min0.08475261944423669
deviation-heading_max2.9691882334099753
deviation-heading_mean1.4821401359021589
deviation-heading_median1.19328721505287
deviation-heading_min0.5727978800929198
driven_any_max5.924436070784932
driven_any_mean2.651436650841013
driven_any_median2.0461076068088055
driven_any_min0.5890953189615077
driven_lanedir_consec_max1.347959781105445
driven_lanedir_consec_mean0.6054990160615574
driven_lanedir_consec_min0.17106026092807447
driven_lanedir_max1.347959781105445
driven_lanedir_mean0.6054990160615574
driven_lanedir_median0.45148801110635506
driven_lanedir_min0.17106026092807447
get_duckie_state_max1.3218445031821322e-06
get_duckie_state_mean1.1765705276986972e-06
get_duckie_state_median1.1680035946278926e-06
get_duckie_state_min1.048430418356871e-06
get_robot_state_max0.003698870843770553
get_robot_state_mean0.003407456943777954
get_robot_state_median0.003340539696452382
get_robot_state_min0.0032498775384364985
get_state_dump_max0.004602325611374005
get_state_dump_mean0.00433947935053218
get_state_dump_median0.004297316729367434
get_state_dump_min0.004160958332019848
get_ui_image_max0.033025912749461636
get_ui_image_mean0.02828374585358027
get_ui_image_median0.02792467572251145
get_ui_image_min0.02425971921983656
in-drivable-lane_max21.800000000000203
in-drivable-lane_mean10.112500000000052
in-drivable-lane_min2.499999999999994
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.4129922755770425, "get_ui_image": 0.025906009333474297, "step_physics": 0.0821535521083408, "survival_time": 12.550000000000043, "driven_lanedir": 0.6095674127943356, "get_state_dump": 0.004358075913928804, "get_robot_state": 0.0033773694719587055, "sim_render-ego0": 0.003607729124644446, "get_duckie_state": 1.1845240517268103e-06, "in-drivable-lane": 8.950000000000014, "deviation-heading": 1.5991811598286385, "agent_compute-ego0": 0.01211006584621611, "complete-iteration": 0.14416184027989706, "set_robot_commands": 0.0019813444879319933, "deviation-center-line": 0.38598928837835295, "driven_lanedir_consec": 0.6095674127943356, "sim_compute_sim_state": 0.008799216103932214, "sim_compute_performance-ego0": 0.0017889397484915598}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.5890953189615077, "get_ui_image": 0.033025912749461636, "step_physics": 0.12673252056806517, "survival_time": 3.8499999999999943, "driven_lanedir": 0.17106026092807447, "get_state_dump": 0.004236557544806065, "get_robot_state": 0.0032498775384364985, "sim_render-ego0": 0.003471224735944699, "get_duckie_state": 1.048430418356871e-06, "in-drivable-lane": 2.499999999999994, "deviation-heading": 0.7873932702771017, "agent_compute-ego0": 0.012563106341239734, "complete-iteration": 0.19606345433455247, "set_robot_commands": 0.0019122851200592825, "deviation-center-line": 0.08475261944423669, "driven_lanedir_consec": 0.17106026092807447, "sim_compute_sim_state": 0.009090833174876675, "sim_compute_performance-ego0": 0.001702577639848758}, "LF-norm-techtrack-000-ego0": {"driven_any": 5.924436070784932, "get_ui_image": 0.0299433421115486, "step_physics": 0.1041435339823872, "survival_time": 29.35000000000028, "driven_lanedir": 1.347959781105445, "get_state_dump": 0.004602325611374005, "get_robot_state": 0.003698870843770553, "sim_render-ego0": 0.003923815934836459, "get_duckie_state": 1.3218445031821322e-06, "in-drivable-lane": 21.800000000000203, "deviation-heading": 2.9691882334099753, "agent_compute-ego0": 0.012946488094978592, "complete-iteration": 0.17527323596331537, "set_robot_commands": 0.0022281120423556997, "deviation-center-line": 0.7446003464316293, "driven_lanedir_consec": 1.347959781105445, "sim_compute_sim_state": 0.01164015701838902, "sim_compute_performance-ego0": 0.002054778491558672}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.6792229380405683, "get_ui_image": 0.02425971921983656, "step_physics": 0.0874166842345353, "survival_time": 9.049999999999994, "driven_lanedir": 0.29340860941837454, "get_state_dump": 0.004160958332019848, "get_robot_state": 0.0033037099209460583, "sim_render-ego0": 0.003549191977951553, "get_duckie_state": 1.1514831375289749e-06, "in-drivable-lane": 7.199999999999993, "deviation-heading": 0.5727978800929198, "agent_compute-ego0": 0.011853804955115683, "complete-iteration": 0.14321006523383842, "set_robot_commands": 0.001950575755192683, "deviation-center-line": 0.1212808048951612, "driven_lanedir_consec": 0.29340860941837454, "sim_compute_sim_state": 0.004880278975098997, "sim_compute_performance-ego0": 0.0017569589090871288}}
set_robot_commands_max0.0022281120423556997
set_robot_commands_mean0.0020180793513849145
set_robot_commands_median0.0019659601215623384
set_robot_commands_min0.0019122851200592825
sim_compute_performance-ego0_max0.002054778491558672
sim_compute_performance-ego0_mean0.0018258136972465295
sim_compute_performance-ego0_median0.0017729493287893445
sim_compute_performance-ego0_min0.001702577639848758
sim_compute_sim_state_max0.01164015701838902
sim_compute_sim_state_mean0.008602621318074226
sim_compute_sim_state_median0.008945024639404445
sim_compute_sim_state_min0.004880278975098997
sim_render-ego0_max0.003923815934836459
sim_render-ego0_mean0.0036379904433442895
sim_render-ego0_median0.0035784605512979993
sim_render-ego0_min0.003471224735944699
simulation-passed1
step_physics_max0.12673252056806517
step_physics_mean0.10011157272333213
step_physics_median0.09578010910846126
step_physics_min0.0821535521083408
survival_time_max29.35000000000028
survival_time_mean13.700000000000076
survival_time_min3.8499999999999943
No reset possible
5733811426Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFP-sim-testingLFP-simhost-erroryesnogpu-prod-060:02:16
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5731411313Moustafa Elarabitemplate-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simhost-erroryesnogpu-prod-060:07:02
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5724210986Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:12:52
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.4375362976228532
survival_time_median21.02500000000016
deviation-center-line_median0.23863994287192283
in-drivable-lane_median16.300000000000026


other stats
agent_compute-ego0_max0.05151975668218147
agent_compute-ego0_mean0.040643743669541085
agent_compute-ego0_median0.048743796041888415
agent_compute-ego0_min0.01356762591220603
complete-iteration_max0.2284927206734816
complete-iteration_mean0.1930872744032904
complete-iteration_median0.19005536197483489
complete-iteration_min0.16374565299001026
deviation-center-line_max0.7108817711987756
deviation-center-line_mean0.32608637118647665
deviation-center-line_min0.11618382780328544
deviation-heading_max3.4302304458832125
deviation-heading_mean2.0007511255623975
deviation-heading_median1.999380641812215
deviation-heading_min0.574012772741948
driven_any_max6.528954048925543
driven_any_mean4.011999951243906
driven_any_median4.147368908801504
driven_any_min1.2243079384470723
driven_lanedir_consec_max1.5446949074739402
driven_lanedir_consec_mean0.672890587694576
driven_lanedir_consec_min0.2717948480586574
driven_lanedir_max1.5446949074739402
driven_lanedir_mean0.6840168798980043
driven_lanedir_median0.4597888820297098
driven_lanedir_min0.2717948480586574
get_duckie_state_max1.3575029145017195e-06
get_duckie_state_mean1.2528697343405083e-06
get_duckie_state_median1.2467358544053763e-06
get_duckie_state_min1.1605043140495613e-06
get_robot_state_max0.003591050704320272
get_robot_state_mean0.0035517386279954044
get_robot_state_median0.003574805317426263
get_robot_state_min0.0034662931728088203
get_state_dump_max0.005446220045962168
get_state_dump_mean0.004774580411780624
get_state_dump_median0.004614497753279061
get_state_dump_min0.004423106094602206
get_ui_image_max0.035089162488778435
get_ui_image_mean0.029748357701786535
get_ui_image_median0.028812874857309222
get_ui_image_min0.026278518603749253
in-drivable-lane_max29.650000000000315
in-drivable-lane_mean16.350000000000094
in-drivable-lane_min3.1500000000000155
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 6.528954048925543, "get_ui_image": 0.026744351606891203, "step_physics": 0.08574188751858318, "survival_time": 34.65000000000017, "driven_lanedir": 1.5446949074739402, "get_state_dump": 0.004423106094602206, "get_robot_state": 0.0034662931728088203, "sim_render-ego0": 0.0037243191034718273, "get_duckie_state": 1.2405324050947294e-06, "in-drivable-lane": 23.950000000000035, "deviation-heading": 3.4302304458832125, "agent_compute-ego0": 0.048598788313632055, "complete-iteration": 0.18628986184122923, "set_robot_commands": 0.002113922185100808, "deviation-center-line": 0.7108817711987756, "driven_lanedir_consec": 1.5446949074739402, "sim_compute_sim_state": 0.009544424089986929, "sim_compute_performance-ego0": 0.0018478810615429617}, "LF-norm-zigzag-000-ego0": {"driven_any": 1.2243079384470723, "get_ui_image": 0.035089162488778435, "step_physics": 0.11832894881566364, "survival_time": 9.55, "driven_lanedir": 0.6094373087504588, "get_state_dump": 0.004547588527202606, "get_robot_state": 0.003591050704320272, "sim_render-ego0": 0.003771464029947917, "get_duckie_state": 1.2529393037160237e-06, "in-drivable-lane": 3.1500000000000155, "deviation-heading": 3.3617234855377363, "agent_compute-ego0": 0.04888880377014478, "complete-iteration": 0.2284927206734816, "set_robot_commands": 0.002210532625516256, "deviation-center-line": 0.3596342859110875, "driven_lanedir_consec": 0.5649321399367455, "sim_compute_sim_state": 0.010062155624230703, "sim_compute_performance-ego0": 0.001914167155822118}, "LF-norm-techtrack-000-ego0": {"driven_any": 6.346476652354482, "get_ui_image": 0.030881398107727245, "step_physics": 0.0896681611094189, "survival_time": 31.650000000000315, "driven_lanedir": 0.31014045530896084, "get_state_dump": 0.005446220045962168, "get_robot_state": 0.0035659386908594368, "sim_render-ego0": 0.004050049119943324, "get_duckie_state": 1.1605043140495613e-06, "in-drivable-lane": 29.650000000000315, "deviation-heading": 0.6370377980866938, "agent_compute-ego0": 0.01356762591220603, "complete-iteration": 0.16374565299001026, "set_robot_commands": 0.002354834358022792, "deviation-center-line": 0.11764559983275816, "driven_lanedir_consec": 0.31014045530896084, "sim_compute_sim_state": 0.012120042689591177, "sim_compute_performance-ego0": 0.00201208320701912}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.948261165248526, "get_ui_image": 0.026278518603749253, "step_physics": 0.0946325594158264, "survival_time": 10.400000000000013, "driven_lanedir": 0.2717948480586574, "get_state_dump": 0.004681406979355516, "get_robot_state": 0.0035836719439930895, "sim_render-ego0": 0.0038149938629004378, "get_duckie_state": 1.3575029145017195e-06, "in-drivable-lane": 8.650000000000013, "deviation-heading": 0.574012772741948, "agent_compute-ego0": 0.05151975668218147, "complete-iteration": 0.1938208621084405, "set_robot_commands": 0.002297147039021031, "deviation-center-line": 0.11618382780328544, "driven_lanedir_consec": 0.2717948480586574, "sim_compute_sim_state": 0.005027314692593077, "sim_compute_performance-ego0": 0.001897329348696476}}
set_robot_commands_max0.002354834358022792
set_robot_commands_mean0.002244109051915222
set_robot_commands_median0.002253839832268643
set_robot_commands_min0.002113922185100808
sim_compute_performance-ego0_max0.00201208320701912
sim_compute_performance-ego0_mean0.001917865193270169
sim_compute_performance-ego0_median0.0019057482522592969
sim_compute_performance-ego0_min0.0018478810615429617
sim_compute_sim_state_max0.012120042689591177
sim_compute_sim_state_mean0.009188484274100472
sim_compute_sim_state_median0.009803289857108814
sim_compute_sim_state_min0.005027314692593077
sim_render-ego0_max0.004050049119943324
sim_render-ego0_mean0.003840206529065876
sim_render-ego0_median0.003793228946424177
sim_render-ego0_min0.0037243191034718273
simulation-passed1
step_physics_max0.11832894881566364
step_physics_mean0.09709288921487302
step_physics_median0.09215036026262265
step_physics_min0.08574188751858318
survival_time_max34.65000000000017
survival_time_mean21.56250000000012
survival_time_min9.55
No reset possible
5722010992Ayman Shams 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:10:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.47012717615738536
survival_time_median14.025000000000064
deviation-center-line_median0.2727085289044093
in-drivable-lane_median9.950000000000031


other stats
agent_compute-ego0_max0.013540368466763883
agent_compute-ego0_mean0.013229338990414464
agent_compute-ego0_median0.013300125737977285
agent_compute-ego0_min0.012776736018939411
complete-iteration_max0.23841011233446072
complete-iteration_mean0.2000849174917302
complete-iteration_median0.19581589788458437
complete-iteration_min0.17029776186329135
deviation-center-line_max0.6453530003710608
deviation-center-line_mean0.32076586223137094
deviation-center-line_min0.09229339074560426
deviation-heading_max2.9237916475906136
deviation-heading_mean1.8091895827393685
deviation-heading_median1.8679424683179957
deviation-heading_min0.5770817467308698
driven_any_max4.769097562595553
driven_any_mean2.581098673863047
driven_any_median2.468862341590068
driven_any_min0.6175724496764992
driven_lanedir_consec_max0.687451862564556
driven_lanedir_consec_mean0.4532562823381533
driven_lanedir_consec_min0.18531891447328652
driven_lanedir_max0.687451862564556
driven_lanedir_mean0.4532562823381533
driven_lanedir_median0.47012717615738536
driven_lanedir_min0.18531891447328652
get_duckie_state_max1.814307236089939e-06
get_duckie_state_mean1.7590023437520153e-06
get_duckie_state_median1.7749143229795087e-06
get_duckie_state_min1.671873492959105e-06
get_robot_state_max0.00430651118115681
get_robot_state_mean0.004191965243110657
get_robot_state_median0.004203008768402693
get_robot_state_min0.004055332254480432
get_state_dump_max0.005307104780867293
get_state_dump_mean0.0051246121947381845
get_state_dump_median0.005119574794555144
get_state_dump_min0.00495219440897516
get_ui_image_max0.03765245181758229
get_ui_image_mean0.03225621735670503
get_ui_image_median0.03144914258724798
get_ui_image_min0.02847413243474187
in-drivable-lane_max21.550000000000193
in-drivable-lane_mean11.012500000000063
in-drivable-lane_min2.599999999999993
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 4.769097562595553, "get_ui_image": 0.028506019128062735, "step_physics": 0.09924564874571676, "survival_time": 28.300000000000267, "driven_lanedir": 0.672814765240359, "get_state_dump": 0.00495219440897516, "get_robot_state": 0.004055332254480432, "sim_render-ego0": 0.004235364772655346, "get_duckie_state": 1.671873492959105e-06, "in-drivable-lane": 21.550000000000193, "deviation-heading": 2.9237916475906136, "agent_compute-ego0": 0.012776736018939411, "complete-iteration": 0.17029776186329135, "set_robot_commands": 0.002442356560385837, "deviation-center-line": 0.6453530003710608, "driven_lanedir_consec": 0.672814765240359, "sim_compute_sim_state": 0.011708720861498852, "sim_compute_performance-ego0": 0.0022643512308702686}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.6175724496764992, "get_ui_image": 0.03765245181758229, "step_physics": 0.15720823915993296, "survival_time": 4.049999999999994, "driven_lanedir": 0.18531891447328652, "get_state_dump": 0.005154877174191359, "get_robot_state": 0.00430651118115681, "sim_render-ego0": 0.004402675279756871, "get_duckie_state": 1.814307236089939e-06, "in-drivable-lane": 2.599999999999993, "deviation-heading": 0.8141684389306508, "agent_compute-ego0": 0.013154971890333222, "complete-iteration": 0.23841011233446072, "set_robot_commands": 0.002565590346731791, "deviation-center-line": 0.09229339074560426, "driven_lanedir_consec": 0.18531891447328652, "sim_compute_sim_state": 0.011463464760198824, "sim_compute_performance-ego0": 0.0023806240500473396}, "LF-norm-techtrack-000-ego0": {"driven_any": 3.248024284975392, "get_ui_image": 0.03439226604643322, "step_physics": 0.13017822258056155, "survival_time": 18.850000000000133, "driven_lanedir": 0.687451862564556, "get_state_dump": 0.00508427241491893, "get_robot_state": 0.004123198922979769, "sim_render-ego0": 0.004310016278867368, "get_duckie_state": 1.7597561790829614e-06, "in-drivable-lane": 12.50000000000007, "deviation-heading": 2.92171649770534, "agent_compute-ego0": 0.013445279585621344, "complete-iteration": 0.20807514241132788, "set_robot_commands": 0.002476208424442029, "deviation-center-line": 0.4363057666971769, "driven_lanedir_consec": 0.687451862564556, "sim_compute_sim_state": 0.01162243646288675, "sim_compute_performance-ego0": 0.002330873378370174}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.6897003982047436, "get_ui_image": 0.02847413243474187, "step_physics": 0.1164667541916306, "survival_time": 9.199999999999996, "driven_lanedir": 0.26743958707441173, "get_state_dump": 0.005307104780867293, "get_robot_state": 0.004282818613825618, "sim_render-ego0": 0.004359856167355099, "get_duckie_state": 1.7900724668760558e-06, "in-drivable-lane": 7.399999999999995, "deviation-heading": 0.5770817467308698, "agent_compute-ego0": 0.013540368466763883, "complete-iteration": 0.18355665335784088, "set_robot_commands": 0.0024768932445629224, "deviation-center-line": 0.1091112911116416, "driven_lanedir_consec": 0.26743958707441173, "sim_compute_sim_state": 0.00617553221212851, "sim_compute_performance-ego0": 0.0023571465466473555}}
set_robot_commands_max0.002565590346731791
set_robot_commands_mean0.0024902621440306446
set_robot_commands_median0.002476550834502476
set_robot_commands_min0.002442356560385837
sim_compute_performance-ego0_max0.0023806240500473396
sim_compute_performance-ego0_mean0.0023332488014837843
sim_compute_performance-ego0_median0.0023440099625087647
sim_compute_performance-ego0_min0.0022643512308702686
sim_compute_sim_state_max0.011708720861498852
sim_compute_sim_state_mean0.010242538574178235
sim_compute_sim_state_median0.011542950611542788
sim_compute_sim_state_min0.00617553221212851
sim_render-ego0_max0.004402675279756871
sim_render-ego0_mean0.004326978124658671
sim_render-ego0_median0.004334936223111234
sim_render-ego0_min0.004235364772655346
simulation-passed1
step_physics_max0.15720823915993296
step_physics_mean0.12577471616946043
step_physics_median0.12332248838609608
step_physics_min0.09924564874571676
survival_time_max28.300000000000267
survival_time_mean15.1000000000001
survival_time_min4.049999999999994
No reset possible
5719011010Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simabortedyesnogpu-prod-060:07:20
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 45, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 45, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5716711012Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simabortedyesnogpu-prod-060:03:07
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5715911312Moustafa Elarabitemplate-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-060:00:57
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5705911030Mo Kleit 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:15:47
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median1.272282011431431
survival_time_median21.62500000000017
deviation-center-line_median0.7780625807688288
in-drivable-lane_median10.125000000000115


other stats
agent_compute-ego0_max0.013586632379396694
agent_compute-ego0_mean0.012838949863196572
agent_compute-ego0_median0.012808712853642537
agent_compute-ego0_min0.01215174136610452
complete-iteration_max0.18727133910693425
complete-iteration_mean0.16922111063784484
complete-iteration_median0.17049314855373565
complete-iteration_min0.14862680633697384
deviation-center-line_max4.090417640673199
deviation-center-line_mean1.4818197188408493
deviation-center-line_min0.28073607315254007
deviation-heading_max13.10281462040622
deviation-heading_mean4.691103974146976
deviation-heading_median1.9866596086654629
deviation-heading_min1.6882820588507574
driven_any_max8.617610743868251
driven_any_mean3.433407725211655
driven_any_median2.250184654371409
driven_any_min0.6156508482355505
driven_lanedir_consec_max7.774821502288197
driven_lanedir_consec_mean2.6383945684434265
driven_lanedir_consec_min0.23419274862264627
driven_lanedir_max7.774821502288197
driven_lanedir_mean2.6383945684434265
driven_lanedir_median1.272282011431431
driven_lanedir_min0.23419274862264627
get_duckie_state_max1.3281537600212697e-06
get_duckie_state_mean1.2637223959643458e-06
get_duckie_state_median1.2569962501085714e-06
get_duckie_state_min1.2127433236189714e-06
get_robot_state_max0.003611892929876038
get_robot_state_mean0.003566451809504125
get_robot_state_median0.0035801590292312297
get_robot_state_min0.0034935962496780034
get_state_dump_max0.004544824829900452
get_state_dump_mean0.004485849234751847
get_state_dump_median0.004491272585713479
get_state_dump_min0.004416026937679981
get_ui_image_max0.03552261946713113
get_ui_image_mean0.0297224630551117
get_ui_image_median0.028804645638104743
get_ui_image_min0.025757941477106177
in-drivable-lane_max15.300000000000216
in-drivable-lane_mean10.312500000000114
in-drivable-lane_min5.7000000000000055
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 1.8425914314558616, "get_ui_image": 0.026994549132323197, "step_physics": 0.10829683930760972, "survival_time": 17.90000000000012, "driven_lanedir": 0.9508895746612598, "get_state_dump": 0.004494595328413344, "get_robot_state": 0.003586348716927106, "sim_render-ego0": 0.003858476628168047, "get_duckie_state": 1.29569539784721e-06, "in-drivable-lane": 12.90000000000013, "deviation-heading": 1.6882820588507574, "agent_compute-ego0": 0.012393449342350442, "complete-iteration": 0.1728689790104093, "set_robot_commands": 0.0021500740210658, "deviation-center-line": 0.516448377668857, "driven_lanedir_consec": 0.9508895746612598, "sim_compute_sim_state": 0.00901327558214618, "sim_compute_performance-ego0": 0.002002925925932223}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.6156508482355505, "get_ui_image": 0.03552261946713113, "step_physics": 0.11110260598946615, "survival_time": 9.5, "driven_lanedir": 0.23419274862264627, "get_state_dump": 0.004544824829900452, "get_robot_state": 0.003611892929876038, "sim_render-ego0": 0.003982144500572644, "get_duckie_state": 1.3281537600212697e-06, "in-drivable-lane": 5.7000000000000055, "deviation-heading": 1.8605658808771288, "agent_compute-ego0": 0.013223976364934631, "complete-iteration": 0.18727133910693425, "set_robot_commands": 0.0022359730685568603, "deviation-center-line": 0.28073607315254007, "driven_lanedir_consec": 0.23419274862264627, "sim_compute_sim_state": 0.010918157887084322, "sim_compute_performance-ego0": 0.0020492214182908624}, "LF-norm-techtrack-000-ego0": {"driven_any": 2.657777877286956, "get_ui_image": 0.030614742143886296, "step_physics": 0.09802575655809538, "survival_time": 25.350000000000225, "driven_lanedir": 1.593674448201602, "get_state_dump": 0.004416026937679981, "get_robot_state": 0.0034935962496780034, "sim_render-ego0": 0.003826959865299735, "get_duckie_state": 1.2127433236189714e-06, "in-drivable-lane": 15.300000000000216, "deviation-heading": 2.112753336453797, "agent_compute-ego0": 0.013586632379396694, "complete-iteration": 0.168117318097062, "set_robot_commands": 0.0020836840464374213, "deviation-center-line": 1.0396767838688008, "driven_lanedir_consec": 1.593674448201602, "sim_compute_sim_state": 0.010103275926094356, "sim_compute_performance-ego0": 0.0018887318025423783}, "LF-norm-small_loop-000-ego0": {"driven_any": 8.617610743868251, "get_ui_image": 0.025757941477106177, "step_physics": 0.08894221550419765, "survival_time": 59.99999999999873, "driven_lanedir": 7.774821502288197, "get_state_dump": 0.004487949843013614, "get_robot_state": 0.003573969341535354, "sim_render-ego0": 0.0037658540136510385, "get_duckie_state": 1.2182971023699329e-06, "in-drivable-lane": 7.350000000000104, "deviation-heading": 13.10281462040622, "agent_compute-ego0": 0.01215174136610452, "complete-iteration": 0.14862680633697384, "set_robot_commands": 0.0021263629967326627, "deviation-center-line": 4.090417640673199, "driven_lanedir_consec": 7.774821502288197, "sim_compute_sim_state": 0.005837902836160398, "sim_compute_performance-ego0": 0.001904314701801335}}
set_robot_commands_max0.0022359730685568603
set_robot_commands_mean0.002149023533198186
set_robot_commands_median0.0021382185088992312
set_robot_commands_min0.0020836840464374213
sim_compute_performance-ego0_max0.0020492214182908624
sim_compute_performance-ego0_mean0.0019612984621416997
sim_compute_performance-ego0_median0.001953620313866779
sim_compute_performance-ego0_min0.0018887318025423783
sim_compute_sim_state_max0.010918157887084322
sim_compute_sim_state_mean0.008968153057871313
sim_compute_sim_state_median0.009558275754120268
sim_compute_sim_state_min0.005837902836160398
sim_render-ego0_max0.003982144500572644
sim_render-ego0_mean0.003858358751922866
sim_render-ego0_median0.0038427182467338913
sim_render-ego0_min0.0037658540136510385
simulation-passed1
step_physics_max0.11110260598946615
step_physics_mean0.10159185433984222
step_physics_median0.10316129793285254
step_physics_min0.08894221550419765
survival_time_max59.99999999999873
survival_time_mean28.18749999999977
survival_time_min9.5
No reset possible
5704211032Yishu Malhotra 🇨🇦exercise_ros_templateaido5-LF-sim-validationLFv-simfailedyesnogpu-prod-060:03:03
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 218, in read_one
    msgs = read_reply(self.fpout, timeout=timeout, waiting_for=waiting_for, nickname=self.nickname,)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 304, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 341, in read_until_over
    raise TimeoutError(msg) from None
TimeoutError: Timeout while reading output of node "ego0".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 563, in run_episode
    r: MsgReceived = await loop.run_in_executor(executor, f)
  File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 135, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 281, in read_one
    raise TimeoutError(msg) from e
TimeoutError: Timeout detected on /fifos/ego0-out after 3 messages. Expected topic "commands".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 286, in main
    length_s = await run_episode(
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 567, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5686911069Francois Hebertsim-exercise-1aido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:35:19
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median9.921791298933035
survival_time_median59.99999999999873
deviation-center-line_median3.1795103597540875
in-drivable-lane_median1.8499999999999748


other stats
agent_compute-ego0_max0.01312515281022935
agent_compute-ego0_mean0.01251454789076717
agent_compute-ego0_median0.01239769524281269
agent_compute-ego0_min0.012137648267213946
complete-iteration_max0.2164072591398876
complete-iteration_mean0.18151396656909852
complete-iteration_median0.17545234650795308
complete-iteration_min0.1587439141206003
deviation-center-line_max3.8117953590097273
deviation-center-line_mean3.20820565067313
deviation-center-line_min2.6620065241746196
deviation-heading_max10.553992582708297
deviation-heading_mean8.535801968889084
deviation-heading_median8.441054353237817
deviation-heading_min6.707106586372407
driven_any_max10.422618532586828
driven_any_mean10.420334839027626
driven_any_median10.422466276886738
driven_any_min10.413788269750205
driven_lanedir_consec_max10.295579784077065
driven_lanedir_consec_mean9.840352648141357
driven_lanedir_consec_min9.222248210622288
driven_lanedir_max10.295579784077065
driven_lanedir_mean9.840352648141357
driven_lanedir_median9.921791298933035
driven_lanedir_min9.222248210622288
get_duckie_state_max1.671510771053419e-06
get_duckie_state_mean1.4657978213498435e-06
get_duckie_state_median1.4101635109475013e-06
get_duckie_state_min1.3713534924509525e-06
get_robot_state_max0.0038886874244175385
get_robot_state_mean0.003801688762826784
get_robot_state_median0.00383501475697056
get_robot_state_min0.003648038112948479
get_state_dump_max0.005057103826441832
get_state_dump_mean0.00485578474653055
get_state_dump_median0.004868116307318161
get_state_dump_min0.004629802545044047
get_ui_image_max0.0359688047366178
get_ui_image_mean0.030027796237494526
get_ui_image_median0.029374373743278004
get_ui_image_min0.025393632726804304
in-drivable-lane_max5.449999999999911
in-drivable-lane_mean2.287499999999965
in-drivable-lane_min0.0
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 10.422618532586828, "get_ui_image": 0.0287430566316044, "step_physics": 0.10461140195098546, "survival_time": 59.99999999999873, "driven_lanedir": 10.295579784077065, "get_state_dump": 0.004971862534102949, "get_robot_state": 0.0038191045352958023, "sim_render-ego0": 0.00404610403570704, "get_duckie_state": 1.4322981250772468e-06, "in-drivable-lane": 0.0, "deviation-heading": 6.707106586372407, "agent_compute-ego0": 0.012614910648228426, "complete-iteration": 0.1726168004003393, "set_robot_commands": 0.002245967334553562, "deviation-center-line": 2.9044424182154924, "driven_lanedir_consec": 10.295579784077065, "sim_compute_sim_state": 0.009375565256504691, "sim_compute_performance-ego0": 0.002098046770500799}, "LF-norm-zigzag-000-ego0": {"driven_any": 10.422458673138308, "get_ui_image": 0.0359688047366178, "step_physics": 0.13615560174286118, "survival_time": 59.99999999999873, "driven_lanedir": 9.222248210622288, "get_state_dump": 0.005057103826441832, "get_robot_state": 0.0038886874244175385, "sim_render-ego0": 0.0040897503185033995, "get_duckie_state": 1.671510771053419e-06, "in-drivable-lane": 5.449999999999911, "deviation-heading": 10.553992582708297, "agent_compute-ego0": 0.01312515281022935, "complete-iteration": 0.2164072591398876, "set_robot_commands": 0.002296558923268695, "deviation-center-line": 3.8117953590097273, "driven_lanedir_consec": 9.222248210622288, "sim_compute_sim_state": 0.01353406866424586, "sim_compute_performance-ego0": 0.002189492702881164}, "LF-norm-techtrack-000-ego0": {"driven_any": 10.413788269750205, "get_ui_image": 0.030005690854951604, "step_physics": 0.1074929731672352, "survival_time": 59.99999999999873, "driven_lanedir": 9.826912012562968, "get_state_dump": 0.004629802545044047, "get_robot_state": 0.003648038112948479, "sim_render-ego0": 0.003862780397083241, "get_duckie_state": 1.388028896817756e-06, "in-drivable-lane": 2.250000000000032, "deviation-heading": 8.319909771906287, "agent_compute-ego0": 0.012180479837396957, "complete-iteration": 0.17828789261556685, "set_robot_commands": 0.002209459117409788, "deviation-center-line": 2.6620065241746196, "driven_lanedir_consec": 9.826912012562968, "sim_compute_sim_state": 0.012140864635089552, "sim_compute_performance-ego0": 0.002029930324379749}, "LF-norm-small_loop-000-ego0": {"driven_any": 10.422473880635168, "get_ui_image": 0.025393632726804304, "step_physics": 0.09785557190246327, "survival_time": 59.99999999999873, "driven_lanedir": 10.016670585303103, "get_state_dump": 0.004764370080533373, "get_robot_state": 0.0038509249786453184, "sim_render-ego0": 0.004003770941004566, "get_duckie_state": 1.3713534924509525e-06, "in-drivable-lane": 1.4499999999999176, "deviation-heading": 8.562198934569345, "agent_compute-ego0": 0.012137648267213946, "complete-iteration": 0.1587439141206003, "set_robot_commands": 0.0022585330458108235, "deviation-center-line": 3.454578301292682, "driven_lanedir_consec": 10.016670585303103, "sim_compute_sim_state": 0.006250444796559813, "sim_compute_performance-ego0": 0.0021427037019118183}}
set_robot_commands_max0.002296558923268695
set_robot_commands_mean0.0022526296052607173
set_robot_commands_median0.002252250190182193
set_robot_commands_min0.002209459117409788
sim_compute_performance-ego0_max0.002189492702881164
sim_compute_performance-ego0_mean0.0021150433749183824
sim_compute_performance-ego0_median0.0021203752362063087
sim_compute_performance-ego0_min0.002029930324379749
sim_compute_sim_state_max0.01353406866424586
sim_compute_sim_state_mean0.01032523583809998
sim_compute_sim_state_median0.010758214945797123
sim_compute_sim_state_min0.006250444796559813
sim_render-ego0_max0.0040897503185033995
sim_render-ego0_mean0.004000601423074562
sim_render-ego0_median0.004024937488355803
sim_render-ego0_min0.003862780397083241
simulation-passed1
step_physics_max0.13615560174286118
step_physics_mean0.11152888719088629
step_physics_median0.10605218755911032
step_physics_min0.09785557190246327
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
5684611283Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:01:23
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5672911089Fernanda Custodio Pereira do Carmo 🇨🇦sim-exercise-1aido5-LF-sim-validationLFv-simsuccessyesnogpu-prod-060:20:16
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median2.064517503538326
survival_time_median32.29999999999981
deviation-center-line_median0.9032964929224688
in-drivable-lane_median13.849999999999843


other stats
agent_compute-ego0_max0.014002426640018002
agent_compute-ego0_mean0.012105908212823311
agent_compute-ego0_median0.01168544271397538
agent_compute-ego0_min0.011050320783324484
complete-iteration_max0.2026783110021235
complete-iteration_mean0.1739597578210451
complete-iteration_median0.1738956928499351
complete-iteration_min0.14536933458218665
deviation-center-line_max4.096814664524914
deviation-center-line_mean1.585522320923597
deviation-center-line_min0.438681633324535
deviation-heading_max8.086859783030274
deviation-heading_mean4.143434710843369
deviation-heading_median3.562072649425774
deviation-heading_min1.3627337614916537
driven_any_max10.908307109528767
driven_any_mean6.099956446061222
driven_any_median5.539976185482861
driven_any_min2.411566303750398
driven_lanedir_consec_max7.92280136661639
driven_lanedir_consec_mean3.3230493866085173
driven_lanedir_consec_min1.2403611727410266
driven_lanedir_max7.92280136661639
driven_lanedir_mean3.418052470240132
driven_lanedir_median2.2545236708015555
driven_lanedir_min1.2403611727410266
get_duckie_state_max1.3885917244376716e-06
get_duckie_state_mean1.291790949008724e-06
get_duckie_state_median1.270807732484075e-06
get_duckie_state_min1.2369566066290743e-06
get_robot_state_max0.003986500121734954
get_robot_state_mean0.0036779476106899582
get_robot_state_median0.0035930577009005873
get_robot_state_min0.003539174919223706
get_state_dump_max0.00513960633959089
get_state_dump_mean0.00472743262251068
get_state_dump_median0.004647727041666234
get_state_dump_min0.0044746700671193605
get_ui_image_max0.0340642450929998
get_ui_image_mean0.03019272277432359
get_ui_image_median0.0309600862463188
get_ui_image_min0.024786473511656953
in-drivable-lane_max31.49999999999942
in-drivable-lane_mean15.737499999999793
in-drivable-lane_min3.7500000000000537
per-episodes
details{"LF-norm-loop-000-ego0": {"driven_any": 2.411566303750398, "get_ui_image": 0.027982674687113983, "step_physics": 0.09603185843158242, "survival_time": 15.05000000000008, "driven_lanedir": 2.060192231651288, "get_state_dump": 0.004768212899467014, "get_robot_state": 0.00364200168887511, "sim_render-ego0": 0.0038555378945458016, "get_duckie_state": 1.259197462473484e-06, "in-drivable-lane": 3.7500000000000537, "deviation-heading": 1.3627337614916537, "agent_compute-ego0": 0.01169330236927563, "complete-iteration": 0.16182709924432614, "set_robot_commands": 0.0021905172739597346, "deviation-center-line": 0.5853081876666081, "driven_lanedir_consec": 2.060192231651288, "sim_compute_sim_state": 0.009626037237659984, "sim_compute_performance-ego0": 0.0019474542693586536}, "LF-norm-zigzag-000-ego0": {"driven_any": 8.042961794183986, "get_ui_image": 0.03393749780552362, "step_physics": 0.11182193756103516, "survival_time": 46.4499999999995, "driven_lanedir": 2.4488551099518223, "get_state_dump": 0.0045272411838654545, "get_robot_state": 0.0035441137129260647, "sim_render-ego0": 0.003802656358288181, "get_duckie_state": 1.2369566066290743e-06, "in-drivable-lane": 31.49999999999942, "deviation-heading": 5.189174120137809, "agent_compute-ego0": 0.01167758305867513, "complete-iteration": 0.18596428645554408, "set_robot_commands": 0.0021003948744907175, "deviation-center-line": 1.2212847981783297, "driven_lanedir_consec": 2.0688427754253635, "sim_compute_sim_state": 0.012538322069311656, "sim_compute_performance-ego0": 0.001932210563331522}, "LF-norm-techtrack-000-ego0": {"driven_any": 3.036990576781737, "get_ui_image": 0.0340642450929998, "step_physics": 0.12503711404381218, "survival_time": 18.150000000000123, "driven_lanedir": 1.2403611727410266, "get_state_dump": 0.00513960633959089, "get_robot_state": 0.003986500121734954, "sim_render-ego0": 0.00432832162458818, "get_duckie_state": 1.3885917244376716e-06, "in-drivable-lane": 10.750000000000092, "deviation-heading": 1.9349711787137391, "agent_compute-ego0": 0.014002426640018002, "complete-iteration": 0.2026783110021235, "set_robot_commands": 0.0023247979499481535, "deviation-center-line": 0.438681633324535, "driven_lanedir_consec": 1.2403611727410266, "sim_compute_sim_state": 0.01151105990776649, "sim_compute_performance-ego0": 0.0021864044797289504}, "LF-norm-small_loop-000-ego0": {"driven_any": 10.908307109528767, "get_ui_image": 0.024786473511656953, "step_physics": 0.0877255200347138, "survival_time": 59.99999999999873, "driven_lanedir": 7.92280136661639, "get_state_dump": 0.0044746700671193605, "get_robot_state": 0.003539174919223706, "sim_render-ego0": 0.003672311943238423, "get_duckie_state": 1.2824180024946658e-06, "in-drivable-lane": 16.949999999999598, "deviation-heading": 8.086859783030274, "agent_compute-ego0": 0.011050320783324484, "complete-iteration": 0.14536933458218665, "set_robot_commands": 0.0020729532646795393, "deviation-center-line": 4.096814664524914, "driven_lanedir_consec": 7.92280136661639, "sim_compute_sim_state": 0.006102437480700999, "sim_compute_performance-ego0": 0.0018634285954611189}}
set_robot_commands_max0.0023247979499481535
set_robot_commands_mean0.002172165840769536
set_robot_commands_median0.0021454560742252262
set_robot_commands_min0.0020729532646795393
sim_compute_performance-ego0_max0.0021864044797289504
sim_compute_performance-ego0_mean0.0019823744769700614
sim_compute_performance-ego0_median0.001939832416345088
sim_compute_performance-ego0_min0.0018634285954611189
sim_compute_sim_state_max0.012538322069311656
sim_compute_sim_state_mean0.00994446417385978
sim_compute_sim_state_median0.010568548572713237
sim_compute_sim_state_min0.006102437480700999
sim_render-ego0_max0.00432832162458818
sim_render-ego0_mean0.003914706955165146
sim_render-ego0_median0.003829097126416991
sim_render-ego0_min0.003672311943238423
simulation-passed1
step_physics_max0.12503711404381218
step_physics_mean0.10515410751778588
step_physics_median0.1039268979963088
step_physics_min0.0877255200347138
survival_time_max59.99999999999873
survival_time_mean34.9124999999996
survival_time_min15.05000000000008
No reset possible
5671811278Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:00:47
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 48, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 48, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5670111284Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:01:48
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5666111286Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:03:50
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5665511285Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:00:38
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5664511281Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:00:50
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5663211287Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:02:27
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5660711289Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:01:16
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5659711290Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:01:16
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5657411291Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:00:45
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5655811425Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFP-sim-validationLFP-simhost-errornonogpu-prod-060:00:47
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5650611303Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:02:16
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5649711296Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:00:40
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 50, in init
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5643911427Frank (Chude) Qian 🇨🇦template-pytorchaido5-LFV_multi-sim-validationLFVmultibodyv-simhost-erroryesnogpu-prod-060:02:30
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5638211429Frank (Chude) Qian 🇨🇦template-pytorchaido5-LF-sim-testingLFt-simhost-erroryesnogpu-prod-060:02:18
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 248, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 21, in init
              ||     self.check_gpu_available(context)
              ||   File "solution.py", line 48, in check_gpu_available
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
              ||     raise Exception(msg) from e
              || Exception: Exception while calling the node's init() function.
              ||
              || | Traceback (most recent call last):
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              || |     call_if_fun_exists(node, "init", context=context_data)
              || |   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              || |     f(**kwargs)
              || |   File "solution.py", line 21, in init
              || |     self.check_gpu_available(context)
              || |   File "solution.py", line 48, in check_gpu_available
              || |     raise Exception(msg)
              || | Exception: I need a GPU; bailing.
              || |
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5628911299Moustafa Elarabitemplate-pytorchaido5-LF-sim-validationLFv-simhost-erroryesnogpu-prod-060:06:13
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 242, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 143, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 309, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego0" aborted with the following error:

error in ego0 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 314, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "solution.py", line 50, in init
              ||     raise Exception(msg)
              || Exception: I need a GPU; bailing.
              ||
              || The above exception was the direct cause of the following exception:
              ||
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper