Duckietown Challenges Home Challenges Submissions

Evaluator 890

ID890
evaluatorip-172-31-40-253-31940
ownerI don't have one 😀
machineip-172-31-40-253
processip-172-31-40-253-31940
versiond-c:4.0.25;d-c-r:4.0.31;d-s:4.0.21
first heard
last heard
statusinactive
# evaluating
# success1 21686
# timeout1 21693
# failed
# error2 21650
# aborted
# host-error1 21662
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
216932799Rami Al-Naim 🇷🇺challenge-aido_LF-baseline-duckietownaido2-LFVI-sim-testingstep1-simulationtimeoutnoip-172-31-40-253-319401:05:47
I can see how the jo [...]
I can see how the job 21693 is timeout because passed 3947 seconds and the timeout is 3600.0.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
216862952Bhairav MehtaBaseline solution using reinforcement learningaido2-LFVI-sim-testingstep1-simulationsuccessnoip-172-31-40-253-319400:18:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.15087872457908125
survival_time_median1.2500000000000004
deviation-center-line_median0.08911468879468738
in-drivable-lane_median0


other stats
agent_compute-ego_max0.2064178466796875
agent_compute-ego_mean0.18043592638625716
agent_compute-ego_median0.17653359413146974
agent_compute-ego_min0.16484275731173428
deviation-center-line_max0.10816664621085528
deviation-center-line_mean0.08443835561916067
deviation-center-line_min0.05681003309128412
deviation-heading_max0.9240053802117336
deviation-heading_mean0.6398005063803905
deviation-heading_median0.6273577861347631
deviation-heading_min0.3910109377109483
driven_any_max0.23674769153127512
driven_any_mean0.20076254612162153
driven_any_median0.20543637794990943
driven_any_min0.15512755837130163
driven_lanedir_consec_max0.19687653993463392
driven_lanedir_consec_mean0.14760879007453892
driven_lanedir_consec_min0.08798099666234993
driven_lanedir_max0.19687653993463392
driven_lanedir_mean0.14760879007453892
driven_lanedir_median0.15087872457908125
driven_lanedir_min0.08798099666234993
in-drivable-lane_max0.15000000000000013
in-drivable-lane_mean0.023333333333333355
in-drivable-lane_min0
per-episodes
details{"udem1-0-0": {"driven_any": 0.21547480321068987, "sim_physics": 0.24272774827891383, "survival_time": 1.4500000000000006, "driven_lanedir": 0.12187857862626528, "sim_render-ego": 0.06836330479589002, "in-drivable-lane": 0.15000000000000013, "agent_compute-ego": 0.20155491500065245, "deviation-heading": 0.7848566389534851, "set_robot_commands": 0.11322406242633688, "deviation-center-line": 0.07405982534611899, "driven_lanedir_consec": 0.12187857862626528, "sim_compute_sim_state": 0.04167227909482758, "sim_compute_performance-ego": 0.07425968400363264, "sim_compute_robot_state-ego": 0.08596188446571087, "sim_compute_robot_state-npc0": 0.07984938292667784, "sim_compute_robot_state-npc1": 0.07546396913199589, "sim_compute_robot_state-npc2": 0.07240991756833833, "sim_compute_robot_state-npc3": 0.07448032806659567}, "udem1-1-0": {"driven_any": 0.18759322019388175, "sim_physics": 0.22909112329836245, "survival_time": 1.3500000000000003, "driven_lanedir": 0.08798099666234993, "sim_render-ego": 0.06686004002888997, "in-drivable-lane": 0.15000000000000013, "agent_compute-ego": 0.18246716040152092, "deviation-heading": 0.9240053802117336, "set_robot_commands": 0.09895200199551052, "deviation-center-line": 0.07069072185407953, "driven_lanedir_consec": 0.08798099666234993, "sim_compute_sim_state": 0.04262893288223832, "sim_compute_performance-ego": 0.0720626778072781, "sim_compute_robot_state-ego": 0.07418297838281702, "sim_compute_robot_state-npc0": 0.07323205912554706, "sim_compute_robot_state-npc1": 0.07149454399391457, "sim_compute_robot_state-npc2": 0.07498707594694914, "sim_compute_robot_state-npc3": 0.07127535784686054}, "udem1-2-0": {"driven_any": 0.16440017200874574, "sim_physics": 0.2656980390134065, "survival_time": 1.1500000000000004, "driven_lanedir": 0.13843055665086457, "sim_render-ego": 0.06462480710900348, "in-drivable-lane": 0, "agent_compute-ego": 0.17051005363464355, "deviation-heading": 0.40341714494760905, "set_robot_commands": 0.10023918359175972, "deviation-center-line": 0.0993469494740469, "driven_lanedir_consec": 0.13843055665086457, "sim_compute_sim_state": 0.04254598202912704, "sim_compute_performance-ego": 0.07223617512246837, "sim_compute_robot_state-ego": 0.07306445163229237, "sim_compute_robot_state-npc0": 0.07057571411132812, "sim_compute_robot_state-npc1": 0.065344354380732, "sim_compute_robot_state-npc2": 0.06891852876414424, "sim_compute_robot_state-npc3": 0.07344926958498747}, "udem1-3-0": {"driven_any": 0.17965168019122532, "sim_physics": 0.22961548100347104, "survival_time": 1.1500000000000004, "driven_lanedir": 0.15242300912021078, "sim_render-ego": 0.06727818820787512, "in-drivable-lane": 0, "agent_compute-ego": 0.17283971413322116, "deviation-heading": 0.3910109377109483, "set_robot_commands": 0.092799072680266, "deviation-center-line": 0.09688825904470004, "driven_lanedir_consec": 0.15242300912021078, "sim_compute_sim_state": 0.0394427258035411, "sim_compute_performance-ego": 0.0741470482038415, "sim_compute_robot_state-ego": 0.07624282007632048, "sim_compute_robot_state-npc0": 0.06927008214204208, "sim_compute_robot_state-npc1": 0.07064585063768469, "sim_compute_robot_state-npc2": 0.07323498311250107, "sim_compute_robot_state-npc3": 0.07140191741611647}, "udem1-4-0": {"driven_any": 0.17073561523261965, "sim_physics": 0.21788969906893643, "survival_time": 1.1000000000000003, "driven_lanedir": 0.12551552799869525, "sim_render-ego": 0.06392546133561568, "in-drivable-lane": 0, "agent_compute-ego": 0.18503065542741257, "deviation-heading": 0.6273577861347631, "set_robot_commands": 0.10046652230349454, "deviation-center-line": 0.05681003309128412, "driven_lanedir_consec": 0.12551552799869525, "sim_compute_sim_state": 0.04029303247278387, "sim_compute_performance-ego": 0.07079674980857155, "sim_compute_robot_state-ego": 0.07406152378429066, "sim_compute_robot_state-npc0": 0.07176481593738902, "sim_compute_robot_state-npc1": 0.06825419989499179, "sim_compute_robot_state-npc2": 0.06895995140075684, "sim_compute_robot_state-npc3": 0.06627300652590665}, "udem1-5-0": {"driven_any": 0.23674769153127512, "sim_physics": 0.2093401878110824, "survival_time": 1.5500000000000007, "driven_lanedir": 0.15479867407139647, "sim_render-ego": 0.06356764608813871, "in-drivable-lane": 0, "agent_compute-ego": 0.17505506546266617, "deviation-heading": 0.9049637428581206, "set_robot_commands": 0.09852495501118322, "deviation-center-line": 0.08824938982796826, "driven_lanedir_consec": 0.15479867407139647, "sim_compute_sim_state": 0.03907079081381521, "sim_compute_performance-ego": 0.07394834487668929, "sim_compute_robot_state-ego": 0.07293203569227649, "sim_compute_robot_state-npc0": 0.07078238456479964, "sim_compute_robot_state-npc1": 0.07129976057237195, "sim_compute_robot_state-npc2": 0.06746309034286006, "sim_compute_robot_state-npc3": 0.06692917116226689}, "udem1-6-0": {"driven_any": 0.20436480927156184, "sim_physics": 0.25384930610656736, "survival_time": 1.2500000000000004, "driven_lanedir": 0.17307260071531472, "sim_render-ego": 0.06287874221801758, "in-drivable-lane": 0, "agent_compute-ego": 0.1667983913421631, "deviation-heading": 0.4913579924760839, "set_robot_commands": 0.09534509658813475, "deviation-center-line": 0.10128678756318246, "driven_lanedir_consec": 0.17307260071531472, "sim_compute_sim_state": 0.03996796607971191, "sim_compute_performance-ego": 0.06692344665527344, "sim_compute_robot_state-ego": 0.07689661979675293, "sim_compute_robot_state-npc0": 0.0724455451965332, "sim_compute_robot_state-npc1": 0.07007987022399903, "sim_compute_robot_state-npc2": 0.07152935028076172, "sim_compute_robot_state-npc3": 0.09871230125427248}, "udem1-7-0": {"driven_any": 0.22635733789116064, "sim_physics": 0.25912662063326153, "survival_time": 1.4000000000000006, "driven_lanedir": 0.19687653993463392, "sim_render-ego": 0.07346793583461217, "in-drivable-lane": 0, "agent_compute-ego": 0.19463820116860525, "deviation-heading": 0.51147815379253, "set_robot_commands": 0.1102514352117266, "deviation-center-line": 0.10816664621085528, "driven_lanedir_consec": 0.19687653993463392, "sim_compute_sim_state": 0.04301676579884121, "sim_compute_performance-ego": 0.07892794268471855, "sim_compute_robot_state-ego": 0.09280276298522948, "sim_compute_robot_state-npc0": 0.076524589742933, "sim_compute_robot_state-npc1": 0.07858706372124809, "sim_compute_robot_state-npc2": 0.07436127322060722, "sim_compute_robot_state-npc3": 0.07508867979049683}, "udem1-8-0": {"driven_any": 0.2190785765488754, "sim_physics": 0.20681300656548865, "survival_time": 1.4500000000000006, "driven_lanedir": 0.15087872457908125, "sim_render-ego": 0.06416102113394902, "in-drivable-lane": 0, "agent_compute-ego": 0.17257195505602607, "deviation-heading": 0.7936006689747942, "set_robot_commands": 0.09777245028265592, "deviation-center-line": 0.06424072540742048, "driven_lanedir_consec": 0.15087872457908125, "sim_compute_sim_state": 0.03829674063057735, "sim_compute_performance-ego": 0.0732797178728827, "sim_compute_robot_state-ego": 0.07623330478010507, "sim_compute_robot_state-npc0": 0.07082537124896872, "sim_compute_robot_state-npc1": 0.06885226019497576, "sim_compute_robot_state-npc2": 0.07191655553620437, "sim_compute_robot_state-npc3": 0.06596096630754142}, "udem1-9-0": {"driven_any": 0.20543637794990943, "sim_physics": 0.2760410213470459, "survival_time": 1.2500000000000004, "driven_lanedir": 0.15139335955829547, "sim_render-ego": 0.07149924278259277, "in-drivable-lane": 0, "agent_compute-ego": 0.2064178466796875, "deviation-heading": 0.7206597741277969, "set_robot_commands": 0.10766206741333008, "deviation-center-line": 0.06488027478418378, "driven_lanedir_consec": 0.15139335955829547, "sim_compute_sim_state": 0.04340471267700195, "sim_compute_performance-ego": 0.07562179565429687, "sim_compute_robot_state-ego": 0.08695178985595703, "sim_compute_robot_state-npc0": 0.07215569496154785, "sim_compute_robot_state-npc1": 0.07367435455322266, "sim_compute_robot_state-npc2": 0.06984254837036133, "sim_compute_robot_state-npc3": 0.07393942832946777}, "udem1-10-0": {"driven_any": 0.22703981014727237, "sim_physics": 0.2015069675445557, "survival_time": 1.2500000000000004, "driven_lanedir": 0.178911196648472, "sim_render-ego": 0.0673088264465332, "in-drivable-lane": 0, "agent_compute-ego": 0.17653359413146974, "deviation-heading": 0.6951844138429675, "set_robot_commands": 0.09446273803710938, "deviation-center-line": 0.07425581059807138, "driven_lanedir_consec": 0.178911196648472, "sim_compute_sim_state": 0.038349885940551755, "sim_compute_performance-ego": 0.07054581642150878, "sim_compute_robot_state-ego": 0.07460909843444824, "sim_compute_robot_state-npc0": 0.06817362785339355, "sim_compute_robot_state-npc1": 0.06730252265930176, "sim_compute_robot_state-npc2": 0.06872591018676757, "sim_compute_robot_state-npc3": 0.07286240577697754}, "udem1-11-0": {"driven_any": 0.17844578156371568, "sim_physics": 0.2120149569077925, "survival_time": 1.1000000000000003, "driven_lanedir": 0.14257447627234532, "sim_render-ego": 0.06336078860542992, "in-drivable-lane": 0.05000000000000005, "agent_compute-ego": 0.17392043633894486, "deviation-heading": 0.40493999345571224, "set_robot_commands": 0.09992853077975186, "deviation-center-line": 0.09478014709514518, "driven_lanedir_consec": 0.14257447627234532, "sim_compute_sim_state": 0.04076273874803023, "sim_compute_performance-ego": 0.0747024796225808, "sim_compute_robot_state-ego": 0.07807949456301602, "sim_compute_robot_state-npc0": 0.07059124383059415, "sim_compute_robot_state-npc1": 0.06747145002538507, "sim_compute_robot_state-npc2": 0.07078016888011586, "sim_compute_robot_state-npc3": 0.07577728141437877}, "udem1-12-0": {"driven_any": 0.231221137113261, "sim_physics": 0.24610655307769777, "survival_time": 1.5000000000000009, "driven_lanedir": 0.14547556956982843, "sim_render-ego": 0.06634092330932617, "in-drivable-lane": 0, "agent_compute-ego": 0.18186882336934407, "deviation-heading": 0.8898942490245054, "set_robot_commands": 0.10612837473551431, "deviation-center-line": 0.09337423432175009, "driven_lanedir_consec": 0.14547556956982843, "sim_compute_sim_state": 0.042909765243530275, "sim_compute_performance-ego": 0.07063059012095134, "sim_compute_robot_state-ego": 0.08051412105560303, "sim_compute_robot_state-npc0": 0.07380648454030354, "sim_compute_robot_state-npc1": 0.07439757188161214, "sim_compute_robot_state-npc2": 0.07391850153605144, "sim_compute_robot_state-npc3": 0.0757316509882609}, "udem1-13-0": {"driven_any": 0.15512755837130163, "sim_physics": 0.2049555778503418, "survival_time": 1.1000000000000003, "driven_lanedir": 0.12357384525578974, "sim_render-ego": 0.06012014909224077, "in-drivable-lane": 0, "agent_compute-ego": 0.16484275731173428, "deviation-heading": 0.4598679191409758, "set_robot_commands": 0.09812300855463202, "deviation-center-line": 0.08911468879468738, "driven_lanedir_consec": 0.12357384525578974, "sim_compute_sim_state": 0.036649552258578216, "sim_compute_performance-ego": 0.07071767070076683, "sim_compute_robot_state-ego": 0.07086370208046654, "sim_compute_robot_state-npc0": 0.07217723673040216, "sim_compute_robot_state-npc1": 0.07054055820811879, "sim_compute_robot_state-npc2": 0.07089303840290416, "sim_compute_robot_state-npc3": 0.0692342844876376}, "udem1-14-0": {"driven_any": 0.2097636205988276, "sim_physics": 0.2089957042976662, "survival_time": 1.3500000000000003, "driven_lanedir": 0.17034819545454027, "sim_render-ego": 0.06439517162464282, "in-drivable-lane": 0, "agent_compute-ego": 0.1814893263357657, "deviation-heading": 0.5944128000538305, "set_robot_commands": 0.10325439771016438, "deviation-center-line": 0.09043084087391608, "driven_lanedir_consec": 0.17034819545454027, "sim_compute_sim_state": 0.04010921937448007, "sim_compute_performance-ego": 0.07475524478488499, "sim_compute_robot_state-ego": 0.07772375036168981, "sim_compute_robot_state-npc0": 0.0717289889300311, "sim_compute_robot_state-npc1": 0.0713455941942003, "sim_compute_robot_state-npc2": 0.07610825256065086, "sim_compute_robot_state-npc3": 0.06750804406625253}}
set_robot_commands_max0.11322406242633688
set_robot_commands_mean0.10114225982143804
set_robot_commands_median0.09992853077975186
set_robot_commands_min0.092799072680266
sim_compute_performance-ego_max0.07892794268471855
sim_compute_performance-ego_mean0.07290369228935638
sim_compute_performance-ego_median0.0732797178728827
sim_compute_performance-ego_min0.06692344665527344
sim_compute_robot_state-ego_max0.09280276298522948
sim_compute_robot_state-ego_mean0.07807468919646507
sim_compute_robot_state-ego_median0.07624282007632048
sim_compute_robot_state-ego_min0.07086370208046654
sim_compute_robot_state-npc0_max0.07984938292667784
sim_compute_robot_state-npc0_mean0.07226021478949939
sim_compute_robot_state-npc0_median0.07176481593738902
sim_compute_robot_state-npc0_min0.06817362785339355
sim_compute_robot_state-npc1_max0.07858706372124809
sim_compute_robot_state-npc1_mean0.07098359495158363
sim_compute_robot_state-npc1_median0.07064585063768469
sim_compute_robot_state-npc1_min0.065344354380732
sim_compute_robot_state-npc2_max0.07610825256065086
sim_compute_robot_state-npc2_mean0.0716032764073316
sim_compute_robot_state-npc2_median0.07152935028076172
sim_compute_robot_state-npc2_min0.06746309034286006
sim_compute_robot_state-npc3_max0.09871230125427248
sim_compute_robot_state-npc3_mean0.07324160620120132
sim_compute_robot_state-npc3_median0.07286240577697754
sim_compute_robot_state-npc3_min0.06596096630754142
sim_compute_sim_state_max0.04340471267700195
sim_compute_sim_state_mean0.04060807265650908
sim_compute_sim_state_median0.04029303247278387
sim_compute_sim_state_min0.036649552258578216
sim_physics_max0.2760410213470459
sim_physics_mean0.2309181328536393
sim_physics_median0.22909112329836245
sim_physics_min0.2015069675445557
sim_render-ego_max0.07346793583461217
sim_render-ego_mean0.06587681657418383
sim_render-ego_median0.06462480710900348
sim_render-ego_min0.06012014909224077
simulation-passed1
survival_time_max1.5500000000000007
survival_time_mean1.293333333333334
survival_time_min1.1000000000000003
No reset possible
216622964Julian ZillyBaseline-IL-logs-tensorflowaido2-LFVI-sim-testingstep1-simulationhost-errornoip-172-31-40-253-319400:31:43
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 484, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 332, in convert
    data = open(log).read().strip()
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2964/step1-simulation-ip-172-31-40-253-31940-job21662/logs/challenges-runner/stdout.log'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
216613264Liu Sam 🇹🇼challenge-aido_LF-template-randomaido2-PREDstep1-simulationerrornoip-172-31-40-253-319400:01:16
Unexpected exception [...]
Unexpected exception:
Traceback (most recent call last):
  File "/project/src/zuper-nodes/src/zuper_nodes/compatibility.py", line 24, in check_compatible_protocol
    raise IncompatibleProtocol(msg)
zuper_nodes.compatibility.IncompatibleProtocol: For input "observations", cannot use type <class 'aido_schemas.schemas.Duckiebot1Observations'> as <class 'float'>: Type <class 'aido_schemas.schemas.Duckiebot1Observations'>
 is not a subclass of <class 'float'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 75, in <module>
    main(cie)
  File "experiment_manager.py", line 26, in main
    agent_ci._get_node_protocol()
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 84, in _get_node_protocol
    check_compatible_protocol(self.node_protocol, self.expect_protocol)
  File "/project/src/zuper-nodes/src/zuper_nodes/compatibility.py", line 42, in check_compatible_protocol
    raise IncompatibleProtocol(msg) from e
zuper_nodes.compatibility.IncompatibleProtocol: Cannot say that p1 is a sub-protocol of p2
p1: |InteractionProtocol(description='Particularization for Duckiebot1 observations and commands.', inputs={'observations': <class 'aido_schemas.schemas.Duckiebot1Observations'>, 'seed': <class 'int'>, 'get_commands': <class 'NoneType'>, 'episode_start': <class 'aido_schemas.protocol_agent.EpisodeStart'>}, outputs={'commands': <class 'aido_schemas.schemas.Duckiebot1Commands'>}, language='(in:seed)? ; ((in:episode_start ; ((in:observations | (in:get_commands ; out:commands))*))*)')
p2: |InteractionProtocol(description='An estimator receives a stream of values and must predict the next value.', inputs={'observations': <class 'float'>, 'seed': <class 'int'>, 'get_prediction': <class 'NoneType'>}, outputs={'prediction': <class 'float'>}, language='(in:seed)? ; ((in:observations | (in:get_prediction ; out:prediction))*)')
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
216503038Manfred Diazchallenge-aido_LF-template-pytorchaido2-LFVI-sim-testingstep1-simulationerrornoip-172-31-40-253-319400:02:10
The container "solut [...]
The container "solution" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible