Duckietown Challenges Home Challenges Submissions

Evaluator 572

ID572
evaluatoridsc-rudolf-5533
ownerAndrea Censi 🇨🇭
machineidsc-rudolf
processidsc-rudolf-5533
versiond-c:4.0.11;d-c-r:4.0.10;d-s:4.0.12
first heard
last heard
statusinactive
# evaluating
# success2 17200
# timeout
# failed1 17214
# error
# aborted
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
172142350Andrea Censi 🇨🇭Baseline solution using imitation learning from logsaido2_LFVI_r3-v4-z2step1-simulationfailedyesidsc-rudolf-55330:01:58
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "experiment_manager.py", line 285, in run_episode
    r: MsgReceived = agent.write_topic_and_expect('get_commands', expect='commands')
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 92, in write_topic_and_expect
    ob: MsgReceived = self.read_one(expect_topic=expect, timeout=timeout)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 173, in read_one
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 254, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 282, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "agent" aborted:

error in agent |Exception while handling a message on topic "get_commands".
               |
               || Traceback (most recent call last):
               ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
               ||     handle_message_node(parsed, receiver0, context0)
               ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
               ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
               ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
               ||     f(**kwargs)
               ||   File "imitation_agent.py", line 66, in on_received_get_commands
               ||     pwm_left, pwm_right = self.compute_action(self.config.current_image)
               ||   File "imitation_agent.py", line 41, in compute_action
               ||     graph = load_graph(frozen_model_filename)
               ||   File "/project/graph_utils.py", line 8, in load_graph
               ||     graph_def.ParseFromString(f.read())
               ||   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 125, in read
               ||     self._preread_check()
               ||   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/lib/io/file_io.py", line 85, in _preread_check
               ||     compat.as_bytes(self.__name), 1024 * 512, status)
               ||   File "/usr/local/lib/python3.7/dist-packages/tensorflow/python/framework/errors_impl.py", line 528, in __exit__
               ||     c_api.TF_GetCode(self.status.status))
               || tensorflow.python.framework.errors_impl.NotFoundError: frozen_graph.pb; No such file or directory
               ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 426, in <module>
    wrap(cie)
  File "experiment_manager.py", line 415, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 128, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 289, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
172112345Andrea Censi 🇨🇭Baseline solution using reinforcement learningaido2_LF_r3-v4-z2step1-simulationsuccessyesidsc-rudolf-55330:10:47
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.05113776419817784
survival_time_median0.49999999999999994
deviation-center-line_median0.008774826897947714
in-drivable-lane_median0.19999999999999996


other stats
agent_compute-ego_max0.2044907808303833
agent_compute-ego_mean0.1892151304880778
agent_compute-ego_median0.1872672017415365
agent_compute-ego_min0.17243916988372804
deviation-center-line_max0.6640097263157637
deviation-center-line_mean0.241989195058862
deviation-center-line_min0.007560060087080871
deviation-heading_max6.201252762569833
deviation-heading_mean2.1120161618128583
deviation-heading_median0.25256854734000034
deviation-heading_min0.20738866508883577
driven_any_max4.230112183161788
driven_any_mean1.766504980467755
driven_any_median0.1510632818157028
driven_any_min0.13435273427237915
driven_lanedir_consec_max1.2754744103387052
driven_lanedir_consec_mean0.4080386223046074
driven_lanedir_consec_min0.0343207258437741
driven_lanedir_max1.2754744103387052
driven_lanedir_mean0.4080386223046074
driven_lanedir_median0.05113776419817784
driven_lanedir_min0.0343207258437741
in-drivable-lane_max10.800000000000043
in-drivable-lane_mean3.760000000000016
in-drivable-lane_min0.19999999999999996
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 0.13435273427237915, "sim_physics": 0.012355446815490724, "survival_time": 0.49999999999999994, "driven_lanedir": 0.0343207258437741, "sim_render-ego": 0.06217975616455078, "in-drivable-lane": 0.19999999999999996, "agent_compute-ego": 0.2044907808303833, "deviation-heading": 0.25256854734000034, "set_robot_commands": 0.07336192131042481, "deviation-center-line": 0.008774826897947714, "driven_lanedir_consec": 0.0343207258437741, "sim_compute_sim_state": 0.03755407333374024, "sim_compute_performance-ego": 0.058626532554626465, "sim_compute_robot_state-ego": 0.060285425186157225}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.1473833435403592, "sim_physics": 0.01220390796661377, "survival_time": 0.49999999999999994, "driven_lanedir": 0.05113776419817784, "sim_render-ego": 0.05926222801208496, "in-drivable-lane": 0.19999999999999996, "agent_compute-ego": 0.18432047367095947, "deviation-heading": 0.20738866508883577, "set_robot_commands": 0.08080313205718995, "deviation-center-line": 0.007560060087080871, "driven_lanedir_consec": 0.05113776419817784, "sim_compute_sim_state": 0.0358931303024292, "sim_compute_performance-ego": 0.06259937286376953, "sim_compute_robot_state-ego": 0.08154175281524659}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 4.230112183161788, "sim_physics": 0.01332601229349772, "survival_time": 14.950000000000076, "driven_lanedir": 1.2754744103387052, "sim_render-ego": 0.06462034543355306, "in-drivable-lane": 7.400000000000036, "agent_compute-ego": 0.19755802631378175, "deviation-heading": 6.201252762569833, "set_robot_commands": 0.07518365542093913, "deviation-center-line": 0.6640097263157637, "driven_lanedir_consec": 1.2754744103387052, "sim_compute_sim_state": 0.040791343053181967, "sim_compute_performance-ego": 0.06335133949915568, "sim_compute_robot_state-ego": 0.06471392472585043}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.1510632818157028, "sim_physics": 0.0146561861038208, "survival_time": 0.49999999999999994, "driven_lanedir": 0.04574468100814766, "sim_render-ego": 0.067840313911438, "in-drivable-lane": 0.19999999999999996, "agent_compute-ego": 0.17243916988372804, "deviation-heading": 0.2178508621650775, "set_robot_commands": 0.07118029594421386, "deviation-center-line": 0.008482663873330566, "driven_lanedir_consec": 0.04574468100814766, "sim_compute_sim_state": 0.04154102802276611, "sim_compute_performance-ego": 0.07152070999145507, "sim_compute_robot_state-ego": 0.06416006088256836}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 4.169613359548547, "sim_physics": 0.01295148213704427, "survival_time": 14.950000000000076, "driven_lanedir": 0.6335155301342321, "sim_render-ego": 0.06491060892740885, "in-drivable-lane": 10.800000000000043, "agent_compute-ego": 0.1872672017415365, "deviation-heading": 3.681019971900545, "set_robot_commands": 0.07347466468811036, "deviation-center-line": 0.5211186981201873, "driven_lanedir_consec": 0.6335155301342321, "sim_compute_sim_state": 0.04132832765579224, "sim_compute_performance-ego": 0.06281874020894368, "sim_compute_robot_state-ego": 0.0647518801689148}}
set_robot_commands_max0.08080313205718995
set_robot_commands_mean0.07480073388417563
set_robot_commands_median0.07347466468811036
set_robot_commands_min0.07118029594421386
sim_compute_performance-ego_max0.07152070999145507
sim_compute_performance-ego_mean0.0637833390235901
sim_compute_performance-ego_median0.06281874020894368
sim_compute_performance-ego_min0.058626532554626465
sim_compute_robot_state-ego_max0.08154175281524659
sim_compute_robot_state-ego_mean0.06709060875574749
sim_compute_robot_state-ego_median0.06471392472585043
sim_compute_robot_state-ego_min0.060285425186157225
sim_compute_sim_state_max0.04154102802276611
sim_compute_sim_state_mean0.03942158047358195
sim_compute_sim_state_median0.040791343053181967
sim_compute_sim_state_min0.0358931303024292
sim_physics_max0.0146561861038208
sim_physics_mean0.013098607063293456
sim_physics_median0.01295148213704427
sim_physics_min0.01220390796661377
sim_render-ego_max0.067840313911438
sim_render-ego_mean0.06376265048980714
sim_render-ego_median0.06462034543355306
sim_render-ego_min0.05926222801208496
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean6.280000000000031
survival_time_min0.49999999999999994
No reset possible
172002339Andrea Censi 🇨🇭Minimal agentaido2_LFVI_r3-v4-z2step1-simulationsuccessyesidsc-rudolf-55330:15:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.72155331423772
survival_time_median4.249999999999993
deviation-center-line_median0.15741823007542524
in-drivable-lane_median1.7499999999999938


other stats
agent_compute-ego_max0.15585036137524774
agent_compute-ego_mean0.14403429416533928
agent_compute-ego_median0.14144430738506894
agent_compute-ego_min0.13882970316656704
deviation-center-line_max0.32861340027620084
deviation-center-line_mean0.16720563088123774
deviation-center-line_min0.08372346350452832
deviation-heading_max1.6095487863912878
deviation-heading_mean0.7416120746207572
deviation-heading_median0.6111580236941071
deviation-heading_min0.3663194255812494
driven_any_max2.174849274549989
driven_any_mean1.370290316170529
driven_any_median1.2632550932005688
driven_any_min0.9890108043245008
driven_lanedir_consec_max1.0300199271670714
driven_lanedir_consec_mean0.7679244191832472
driven_lanedir_consec_min0.523101221541693
driven_lanedir_max1.0586376106182156
driven_lanedir_mean0.773647955873476
driven_lanedir_median0.72155331423772
driven_lanedir_min0.523101221541693
in-drivable-lane_max3.149999999999989
in-drivable-lane_mean1.7699999999999938
in-drivable-lane_min0.1999999999999993
per-episodes
details{"udem1-0-0": {"driven_any": 2.174849274549989, "sim_physics": 0.027912639749461208, "survival_time": 7.249999999999982, "driven_lanedir": 1.0586376106182156, "sim_render-ego": 0.06296855827857709, "in-drivable-lane": 3.149999999999989, "agent_compute-ego": 0.13882970316656704, "deviation-heading": 1.6095487863912878, "set_robot_commands": 0.0751145313526022, "deviation-center-line": 0.32861340027620084, "driven_lanedir_consec": 1.0300199271670714, "sim_compute_sim_state": 0.03564939992181186, "sim_compute_performance-ego": 0.06044099741968615, "sim_compute_robot_state-ego": 0.06219741393779886, "sim_compute_robot_state-npc0": 0.06789039743357692, "sim_compute_robot_state-npc1": 0.06263807230982288, "sim_compute_robot_state-npc2": 0.061368837027714174, "sim_compute_robot_state-npc3": 0.06133571986494393}, "udem1-1-0": {"driven_any": 0.9890108043245008, "sim_physics": 0.027432542858701763, "survival_time": 3.2999999999999963, "driven_lanedir": 0.6673090295116273, "sim_render-ego": 0.06161834615649599, "in-drivable-lane": 0.849999999999997, "agent_compute-ego": 0.14144430738506894, "deviation-heading": 0.6111580236941071, "set_robot_commands": 0.07651883905584161, "deviation-center-line": 0.08372346350452832, "driven_lanedir_consec": 0.6673090295116273, "sim_compute_sim_state": 0.0352868889317368, "sim_compute_performance-ego": 0.061380993236194954, "sim_compute_robot_state-ego": 0.06299265586968625, "sim_compute_robot_state-npc0": 0.06769753586162221, "sim_compute_robot_state-npc1": 0.06096386909484863, "sim_compute_robot_state-npc2": 0.06205280621846517, "sim_compute_robot_state-npc3": 0.06139900828852798}, "udem1-2-0": {"driven_any": 1.4132589628801058, "sim_physics": 0.030182884094562937, "survival_time": 4.699999999999991, "driven_lanedir": 0.523101221541693, "sim_render-ego": 0.06270567406999304, "in-drivable-lane": 2.8999999999999906, "agent_compute-ego": 0.1436931275306864, "deviation-heading": 0.3663194255812494, "set_robot_commands": 0.07908861941479622, "deviation-center-line": 0.10412585673382428, "driven_lanedir_consec": 0.523101221541693, "sim_compute_sim_state": 0.035506930757076186, "sim_compute_performance-ego": 0.05980611354746717, "sim_compute_robot_state-ego": 0.06292126787469741, "sim_compute_robot_state-npc0": 0.06776781792336321, "sim_compute_robot_state-npc1": 0.06480332891991798, "sim_compute_robot_state-npc2": 0.06346646521953826, "sim_compute_robot_state-npc3": 0.06186198173685277}, "udem1-3-0": {"driven_any": 1.2632550932005688, "sim_physics": 0.02896913079654469, "survival_time": 4.249999999999993, "driven_lanedir": 0.72155331423772, "sim_render-ego": 0.06474916794720818, "in-drivable-lane": 1.7499999999999938, "agent_compute-ego": 0.14035397136912625, "deviation-heading": 0.37503744954470697, "set_robot_commands": 0.07522085414213293, "deviation-center-line": 0.15741823007542524, "driven_lanedir_consec": 0.72155331423772, "sim_compute_sim_state": 0.035288423650404986, "sim_compute_performance-ego": 0.0629699286292581, "sim_compute_robot_state-ego": 0.06116440436419319, "sim_compute_robot_state-npc0": 0.06817139737746294, "sim_compute_robot_state-npc1": 0.06274326268364401, "sim_compute_robot_state-npc2": 0.05957463769351735, "sim_compute_robot_state-npc3": 0.0637991232030532}, "udem1-4-0": {"driven_any": 1.0110774458974832, "sim_physics": 0.02642823317471673, "survival_time": 3.399999999999996, "driven_lanedir": 0.8976386034581241, "sim_render-ego": 0.06440585150438197, "in-drivable-lane": 0.1999999999999993, "agent_compute-ego": 0.15585036137524774, "deviation-heading": 0.7459966878924339, "set_robot_commands": 0.08175129750195671, "deviation-center-line": 0.16214720381621003, "driven_lanedir_consec": 0.8976386034581241, "sim_compute_sim_state": 0.03529106168185964, "sim_compute_performance-ego": 0.06468088837230906, "sim_compute_robot_state-ego": 0.06529782449497896, "sim_compute_robot_state-npc0": 0.07378783646751852, "sim_compute_robot_state-npc1": 0.06796142984839047, "sim_compute_robot_state-npc2": 0.06555228724199183, "sim_compute_robot_state-npc3": 0.06457521284327787}}
set_robot_commands_max0.08175129750195671
set_robot_commands_mean0.07753882829346594
set_robot_commands_median0.07651883905584161
set_robot_commands_min0.0751145313526022
sim_compute_performance-ego_max0.06468088837230906
sim_compute_performance-ego_mean0.06185578424098308
sim_compute_performance-ego_median0.061380993236194954
sim_compute_performance-ego_min0.05980611354746717
sim_compute_robot_state-ego_max0.06529782449497896
sim_compute_robot_state-ego_mean0.06291471330827093
sim_compute_robot_state-ego_median0.06292126787469741
sim_compute_robot_state-ego_min0.06116440436419319
sim_compute_robot_state-npc0_max0.07378783646751852
sim_compute_robot_state-npc0_mean0.06906299701270877
sim_compute_robot_state-npc0_median0.06789039743357692
sim_compute_robot_state-npc0_min0.06769753586162221
sim_compute_robot_state-npc1_max0.06796142984839047
sim_compute_robot_state-npc1_mean0.06382199257132479
sim_compute_robot_state-npc1_median0.06274326268364401
sim_compute_robot_state-npc1_min0.06096386909484863
sim_compute_robot_state-npc2_max0.06555228724199183
sim_compute_robot_state-npc2_mean0.06240300668024536
sim_compute_robot_state-npc2_median0.06205280621846517
sim_compute_robot_state-npc2_min0.05957463769351735
sim_compute_robot_state-npc3_max0.06457521284327787
sim_compute_robot_state-npc3_mean0.06259420918733115
sim_compute_robot_state-npc3_median0.06186198173685277
sim_compute_robot_state-npc3_min0.06133571986494393
sim_compute_sim_state_max0.03564939992181186
sim_compute_sim_state_mean0.03540454098857789
sim_compute_sim_state_median0.03529106168185964
sim_compute_sim_state_min0.0352868889317368
sim_physics_max0.030182884094562937
sim_physics_mean0.028185086134797463
sim_physics_median0.027912639749461208
sim_physics_min0.02642823317471673
sim_render-ego_max0.06474916794720818
sim_render-ego_mean0.06328951959133125
sim_render-ego_median0.06296855827857709
sim_render-ego_min0.06161834615649599
simulation-passed1
survival_time_max7.249999999999982
survival_time_mean4.579999999999992
survival_time_min3.2999999999999963
No reset possible