Duckietown Challenges Home Challenges Submissions

Evaluator 898

ID898
evaluatorip-172-31-38-104-5317
ownerI don't have one 😀
machineip-172-31-38-104
processip-172-31-38-104-5317
versiond-c:4.0.25;d-c-r:4.0.31;d-s:4.0.21
first heard
last heard
statusinactive
# evaluating
# success1 21647
# timeout1 21678
# failed
# error1 21670
# aborted
# host-error1 21671
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
216783058Andrea Censi 🇨🇭rotationaido2-LFV-sim-testingstep1-simulationtimeoutnoip-172-31-38-104-53171:05:05
I can see how the jo [...]
I can see how the job 21678 is timeout because passed 3904 seconds and the timeout is 3600.0.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
216712812Andrea Censi 🇨🇭challenge-aido_LF-template-random - random_agentaido2-LFVI-sim-testingstep1-simulationhost-errornoip-172-31-38-104-53170:08:47
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 484, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 332, in convert
    data = open(log).read().strip()
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2812/step1-simulation-ip-172-31-38-104-5317-job21671/logs/challenges-runner/stdout.log'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
216703265Liu Sam 🇹🇼challenge-aido_LF-template-randomaido2-PREDstep1-simulationerrornoip-172-31-38-104-53170:00:41
Unexpected exception [...]
Unexpected exception:
Traceback (most recent call last):
  File "/project/src/zuper-nodes/src/zuper_nodes/compatibility.py", line 24, in check_compatible_protocol
    raise IncompatibleProtocol(msg)
zuper_nodes.compatibility.IncompatibleProtocol: For input "observations", cannot use type <class 'aido_schemas.schemas.Duckiebot1Observations'> as <class 'float'>: Type <class 'aido_schemas.schemas.Duckiebot1Observations'>
 is not a subclass of <class 'float'>

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 75, in <module>
    main(cie)
  File "experiment_manager.py", line 26, in main
    agent_ci._get_node_protocol()
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 84, in _get_node_protocol
    check_compatible_protocol(self.node_protocol, self.expect_protocol)
  File "/project/src/zuper-nodes/src/zuper_nodes/compatibility.py", line 42, in check_compatible_protocol
    raise IncompatibleProtocol(msg) from e
zuper_nodes.compatibility.IncompatibleProtocol: Cannot say that p1 is a sub-protocol of p2
p1: |InteractionProtocol(description='Particularization for Duckiebot1 observations and commands.', inputs={'observations': <class 'aido_schemas.schemas.Duckiebot1Observations'>, 'seed': <class 'int'>, 'get_commands': <class 'NoneType'>, 'episode_start': <class 'aido_schemas.protocol_agent.EpisodeStart'>}, outputs={'commands': <class 'aido_schemas.schemas.Duckiebot1Commands'>}, language='(in:seed)? ; ((in:episode_start ; ((in:observations | (in:get_commands ; out:commands))*))*)')
p2: |InteractionProtocol(description='An estimator receives a stream of values and must predict the next value.', inputs={'observations': <class 'float'>, 'seed': <class 'int'>, 'get_prediction': <class 'NoneType'>}, outputs={'prediction': <class 'float'>}, language='(in:seed)? ; ((in:observations | (in:get_prediction ; out:prediction))*)')
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
216472928Bhairav Mehtachallenge-aido_LF-template-pytorchaido2-LFVI-sim-testingstep1-simulationsuccessnoip-172-31-38-104-53170:21:15
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.09613152749156262
survival_time_median1.4500000000000006
deviation-center-line_median0.09610590151575148
in-drivable-lane_median0


other stats
agent_compute-ego_max0.21263515658494905
agent_compute-ego_mean0.18123852992808284
agent_compute-ego_median0.17479983243075284
agent_compute-ego_min0.1696744806626264
deviation-center-line_max0.1179275412358482
deviation-center-line_mean0.09392921973123272
deviation-center-line_min0.04562900143561815
deviation-heading_max1.1654846503269083
deviation-heading_mean0.8730410074352928
deviation-heading_median0.9436020892271848
deviation-heading_min0.5678447688031131
driven_any_max0.2651390141800698
driven_any_mean0.17367925893120437
driven_any_median0.16716546460691847
driven_any_min0.11308137601172406
driven_lanedir_consec_max0.19518780013747516
driven_lanedir_consec_mean0.11280779260062984
driven_lanedir_consec_min0.060213634950749606
driven_lanedir_max0.19518780013747516
driven_lanedir_mean0.11280779260062984
driven_lanedir_median0.09613152749156262
driven_lanedir_min0.060213634950749606
in-drivable-lane_max0.15000000000000013
in-drivable-lane_mean0.04000000000000002
in-drivable-lane_min0
per-episodes
details{"udem1-0-0": {"driven_any": 0.2317309200517844, "sim_physics": 0.20720551354544503, "survival_time": 1.7500000000000009, "driven_lanedir": 0.14475532173437133, "sim_render-ego": 0.06016817774091448, "in-drivable-lane": 0.10000000000000007, "agent_compute-ego": 0.1803779397691999, "deviation-heading": 1.0382589355284852, "set_robot_commands": 0.1110705988747733, "deviation-center-line": 0.10175612777376604, "driven_lanedir_consec": 0.14475532173437133, "sim_compute_sim_state": 0.035630655288696286, "sim_compute_performance-ego": 0.06768918037414551, "sim_compute_robot_state-ego": 0.07341056551252093, "sim_compute_robot_state-npc0": 0.06511012486049107, "sim_compute_robot_state-npc1": 0.06646130425589425, "sim_compute_robot_state-npc2": 0.07057131358555385, "sim_compute_robot_state-npc3": 0.0676192079271589}, "udem1-1-0": {"driven_any": 0.18633631803060724, "sim_physics": 0.2190211931864421, "survival_time": 1.5000000000000009, "driven_lanedir": 0.08429895437560253, "sim_render-ego": 0.0671254316965739, "in-drivable-lane": 0.10000000000000007, "agent_compute-ego": 0.175505526860555, "deviation-heading": 1.1654846503269083, "set_robot_commands": 0.09912832578023274, "deviation-center-line": 0.08709330096082696, "driven_lanedir_consec": 0.08429895437560253, "sim_compute_sim_state": 0.03776322205861409, "sim_compute_performance-ego": 0.06923322677612305, "sim_compute_robot_state-ego": 0.0726245403289795, "sim_compute_robot_state-npc0": 0.07063500881195069, "sim_compute_robot_state-npc1": 0.0712285041809082, "sim_compute_robot_state-npc2": 0.0708048423131307, "sim_compute_robot_state-npc3": 0.06622961362202963}, "udem1-2-0": {"driven_any": 0.12741277318039604, "sim_physics": 0.2472943250949566, "survival_time": 1.3000000000000005, "driven_lanedir": 0.09613152749156262, "sim_render-ego": 0.0621641782613901, "in-drivable-lane": 0, "agent_compute-ego": 0.17328906976259673, "deviation-heading": 0.5678447688031131, "set_robot_commands": 0.09714548404400164, "deviation-center-line": 0.11342293581882544, "driven_lanedir_consec": 0.09613152749156262, "sim_compute_sim_state": 0.03657911374018742, "sim_compute_performance-ego": 0.0719151680286114, "sim_compute_robot_state-ego": 0.07439176852886493, "sim_compute_robot_state-npc0": 0.07233922298137958, "sim_compute_robot_state-npc1": 0.06755540921137883, "sim_compute_robot_state-npc2": 0.0687945347565871, "sim_compute_robot_state-npc3": 0.06733851249401386}, "udem1-3-0": {"driven_any": 0.12978562006817648, "sim_physics": 0.26453351974487305, "survival_time": 1.4000000000000006, "driven_lanedir": 0.09366368841161864, "sim_render-ego": 0.06567295960017613, "in-drivable-lane": 0, "agent_compute-ego": 0.18886303901672363, "deviation-heading": 0.6996653701894028, "set_robot_commands": 0.10263370616095406, "deviation-center-line": 0.1179275412358482, "driven_lanedir_consec": 0.09366368841161864, "sim_compute_sim_state": 0.03986798013959613, "sim_compute_performance-ego": 0.07208750077656337, "sim_compute_robot_state-ego": 0.08700088092258998, "sim_compute_robot_state-npc0": 0.07037637914930071, "sim_compute_robot_state-npc1": 0.07336914539337158, "sim_compute_robot_state-npc2": 0.07269853353500366, "sim_compute_robot_state-npc3": 0.07386149678911481}, "udem1-4-0": {"driven_any": 0.1390257283515965, "sim_physics": 0.21485584515791672, "survival_time": 1.3000000000000005, "driven_lanedir": 0.07570014261508695, "sim_render-ego": 0.06258147496443528, "in-drivable-lane": 0, "agent_compute-ego": 0.17438483238220215, "deviation-heading": 0.997037173963243, "set_robot_commands": 0.10268209530757023, "deviation-center-line": 0.07001643293455236, "driven_lanedir_consec": 0.07570014261508695, "sim_compute_sim_state": 0.04115039568680983, "sim_compute_performance-ego": 0.06844746149503268, "sim_compute_robot_state-ego": 0.07081066645108737, "sim_compute_robot_state-npc0": 0.06788748044234055, "sim_compute_robot_state-npc1": 0.07272420479701115, "sim_compute_robot_state-npc2": 0.07127936069781964, "sim_compute_robot_state-npc3": 0.0687927191074078}, "udem1-5-0": {"driven_any": 0.2651390141800698, "sim_physics": 0.22206916460176793, "survival_time": 2.0500000000000007, "driven_lanedir": 0.18219939454894665, "sim_render-ego": 0.06667406384537859, "in-drivable-lane": 0.04999999999999982, "agent_compute-ego": 0.21263515658494905, "deviation-heading": 1.15973752915049, "set_robot_commands": 0.10445573271774664, "deviation-center-line": 0.11110489775716537, "driven_lanedir_consec": 0.18219939454894665, "sim_compute_sim_state": 0.04201136565789944, "sim_compute_performance-ego": 0.0775255866167022, "sim_compute_robot_state-ego": 0.08734312290098609, "sim_compute_robot_state-npc0": 0.07648444757228945, "sim_compute_robot_state-npc1": 0.07411107202855552, "sim_compute_robot_state-npc2": 0.0756465923495409, "sim_compute_robot_state-npc3": 0.07576319647998345}, "udem1-6-0": {"driven_any": 0.12515051429313342, "sim_physics": 0.21636761559380424, "survival_time": 1.3500000000000003, "driven_lanedir": 0.0759397376742541, "sim_render-ego": 0.0643780408082185, "in-drivable-lane": 0.05000000000000005, "agent_compute-ego": 0.1727215537318477, "deviation-heading": 0.7779159730296502, "set_robot_commands": 0.09609149120472096, "deviation-center-line": 0.09361590406919652, "driven_lanedir_consec": 0.0759397376742541, "sim_compute_sim_state": 0.03896631134880914, "sim_compute_performance-ego": 0.06854717819779008, "sim_compute_robot_state-ego": 0.07027203948409469, "sim_compute_robot_state-npc0": 0.06679938457630298, "sim_compute_robot_state-npc1": 0.06784126493665907, "sim_compute_robot_state-npc2": 0.06855735955414949, "sim_compute_robot_state-npc3": 0.06889862484402126}, "udem1-7-0": {"driven_any": 0.16716546460691847, "sim_physics": 0.20217802606779953, "survival_time": 1.4500000000000006, "driven_lanedir": 0.1321056806488976, "sim_render-ego": 0.06262547394324994, "in-drivable-lane": 0, "agent_compute-ego": 0.1706758449817526, "deviation-heading": 0.6406241954942626, "set_robot_commands": 0.09864197928329994, "deviation-center-line": 0.11407403079285318, "driven_lanedir_consec": 0.1321056806488976, "sim_compute_sim_state": 0.04046157310748923, "sim_compute_performance-ego": 0.06716641886480923, "sim_compute_robot_state-ego": 0.06797643365531132, "sim_compute_robot_state-npc0": 0.06848285115998366, "sim_compute_robot_state-npc1": 0.07640654465247845, "sim_compute_robot_state-npc2": 0.06702442004762847, "sim_compute_robot_state-npc3": 0.06988431667459422}, "udem1-8-0": {"driven_any": 0.22360088550299773, "sim_physics": 0.2072869055979961, "survival_time": 1.850000000000001, "driven_lanedir": 0.15472414715656635, "sim_render-ego": 0.06205195349615973, "in-drivable-lane": 0.05000000000000005, "agent_compute-ego": 0.17147619659836227, "deviation-heading": 1.013248892050086, "set_robot_commands": 0.09644811862223857, "deviation-center-line": 0.08010871779244341, "driven_lanedir_consec": 0.15472414715656635, "sim_compute_sim_state": 0.039812403756218986, "sim_compute_performance-ego": 0.0674201540044836, "sim_compute_robot_state-ego": 0.07752048002706992, "sim_compute_robot_state-npc0": 0.06987298501504434, "sim_compute_robot_state-npc1": 0.07128410725980192, "sim_compute_robot_state-npc2": 0.07074030025585278, "sim_compute_robot_state-npc3": 0.07160626231013117}, "udem1-9-0": {"driven_any": 0.1786155239299018, "sim_physics": 0.26890310747870083, "survival_time": 1.4500000000000006, "driven_lanedir": 0.10745030893037066, "sim_render-ego": 0.07313320554535964, "in-drivable-lane": 0, "agent_compute-ego": 0.2008025892849626, "deviation-heading": 1.0531869012241095, "set_robot_commands": 0.1004252680416765, "deviation-center-line": 0.07942420579286685, "driven_lanedir_consec": 0.10745030893037066, "sim_compute_sim_state": 0.04277633798533473, "sim_compute_performance-ego": 0.07828646692736395, "sim_compute_robot_state-ego": 0.09163559716323326, "sim_compute_robot_state-npc0": 0.07258506478934452, "sim_compute_robot_state-npc1": 0.07424197525813661, "sim_compute_robot_state-npc2": 0.07165253573450549, "sim_compute_robot_state-npc3": 0.07279313021692736}, "udem1-10-0": {"driven_any": 0.13823742285611357, "sim_physics": 0.22362091908088097, "survival_time": 1.3000000000000005, "driven_lanedir": 0.060213634950749606, "sim_render-ego": 0.06179614250476544, "in-drivable-lane": 0.10000000000000007, "agent_compute-ego": 0.18686910775991583, "deviation-heading": 0.9436020892271848, "set_robot_commands": 0.09448412748483512, "deviation-center-line": 0.04562900143561815, "driven_lanedir_consec": 0.060213634950749606, "sim_compute_sim_state": 0.040065618661733776, "sim_compute_performance-ego": 0.07253593664902908, "sim_compute_robot_state-ego": 0.08001536589402419, "sim_compute_robot_state-npc0": 0.06602340478163499, "sim_compute_robot_state-npc1": 0.07118623073284443, "sim_compute_robot_state-npc2": 0.06795921692481408, "sim_compute_robot_state-npc3": 0.06848968909336971}, "udem1-11-0": {"driven_any": 0.11308137601172406, "sim_physics": 0.21850404956124045, "survival_time": 1.1000000000000003, "driven_lanedir": 0.07916519634871833, "sim_render-ego": 0.06282641670920631, "in-drivable-lane": 0, "agent_compute-ego": 0.17479983243075284, "deviation-heading": 0.5903958685153334, "set_robot_commands": 0.1110805489800193, "deviation-center-line": 0.09610590151575148, "driven_lanedir_consec": 0.07916519634871833, "sim_compute_sim_state": 0.03957229310815984, "sim_compute_performance-ego": 0.07260031049901788, "sim_compute_robot_state-ego": 0.07771366292780096, "sim_compute_robot_state-npc0": 0.06849362633445046, "sim_compute_robot_state-npc1": 0.07167624343525279, "sim_compute_robot_state-npc2": 0.06778148087588223, "sim_compute_robot_state-npc3": 0.07183499769731001}, "udem1-12-0": {"driven_any": 0.228550961747096, "sim_physics": 0.2555698407323737, "survival_time": 1.900000000000001, "driven_lanedir": 0.1354122780914595, "sim_render-ego": 0.07000953900186639, "in-drivable-lane": 0.15000000000000013, "agent_compute-ego": 0.19378029045305753, "deviation-heading": 1.0476010423513582, "set_robot_commands": 0.10529834345767372, "deviation-center-line": 0.1013555944208479, "driven_lanedir_consec": 0.1354122780914595, "sim_compute_sim_state": 0.04354979489978991, "sim_compute_performance-ego": 0.07179917159833406, "sim_compute_robot_state-ego": 0.08532487091265227, "sim_compute_robot_state-npc0": 0.07513662388450221, "sim_compute_robot_state-npc1": 0.07068616465518349, "sim_compute_robot_state-npc2": 0.07594274847131026, "sim_compute_robot_state-npc3": 0.07552864677027653}, "udem1-13-0": {"driven_any": 0.11491889952044622, "sim_physics": 0.22254350781440735, "survival_time": 1.2000000000000004, "driven_lanedir": 0.07516907589376753, "sim_render-ego": 0.06214448809623718, "in-drivable-lane": 0, "agent_compute-ego": 0.1727224886417389, "deviation-heading": 0.6633956396267795, "set_robot_commands": 0.0923206905523936, "deviation-center-line": 0.094209622945068, "driven_lanedir_consec": 0.07516907589376753, "sim_compute_sim_state": 0.04013113180796305, "sim_compute_performance-ego": 0.07220701376597087, "sim_compute_robot_state-ego": 0.07539464036623637, "sim_compute_robot_state-npc0": 0.06739758451779683, "sim_compute_robot_state-npc1": 0.07114443182945251, "sim_compute_robot_state-npc2": 0.06959767142931621, "sim_compute_robot_state-npc3": 0.07261658708254497}, "udem1-14-0": {"driven_any": 0.2364374616371038, "sim_physics": 0.2075013833887437, "survival_time": 1.7000000000000008, "driven_lanedir": 0.19518780013747516, "sim_render-ego": 0.06321934391470517, "in-drivable-lane": 0, "agent_compute-ego": 0.1696744806626264, "deviation-heading": 0.7376160820489865, "set_robot_commands": 0.09239527057198917, "deviation-center-line": 0.10309408072286076, "driven_lanedir_consec": 0.19518780013747516, "sim_compute_sim_state": 0.04021878803477568, "sim_compute_performance-ego": 0.06974042163175695, "sim_compute_robot_state-ego": 0.07217055208542768, "sim_compute_robot_state-npc0": 0.06884741081910975, "sim_compute_robot_state-npc1": 0.06897312052109662, "sim_compute_robot_state-npc2": 0.07092907849480123, "sim_compute_robot_state-npc3": 0.06808355275322409}}
set_robot_commands_max0.1110805489800193
set_robot_commands_mean0.10028678540560838
set_robot_commands_median0.09912832578023274
set_robot_commands_min0.0923206905523936
sim_compute_performance-ego_max0.07828646692736395
sim_compute_performance-ego_mean0.07114674641371559
sim_compute_performance-ego_median0.07179917159833406
sim_compute_performance-ego_min0.06716641886480923
sim_compute_robot_state-ego_max0.09163559716323326
sim_compute_robot_state-ego_mean0.07757367914405863
sim_compute_robot_state-ego_median0.07539464036623637
sim_compute_robot_state-ego_min0.06797643365531132
sim_compute_robot_state-npc0_max0.07648444757228945
sim_compute_robot_state-npc0_mean0.06976477331306147
sim_compute_robot_state-npc0_median0.06884741081910975
sim_compute_robot_state-npc0_min0.06511012486049107
sim_compute_robot_state-npc1_max0.07640654465247845
sim_compute_robot_state-npc1_mean0.07125931487653503
sim_compute_robot_state-npc1_median0.0712285041809082
sim_compute_robot_state-npc1_min0.06646130425589425
sim_compute_robot_state-npc2_max0.07594274847131026
sim_compute_robot_state-npc2_mean0.0706653326017264
sim_compute_robot_state-npc2_median0.07074030025585278
sim_compute_robot_state-npc2_min0.06702442004762847
sim_compute_robot_state-npc3_max0.07576319647998345
sim_compute_robot_state-npc3_mean0.07062270359080718
sim_compute_robot_state-npc3_median0.06988431667459422
sim_compute_robot_state-npc3_min0.06622961362202963
sim_compute_sim_state_max0.04354979489978991
sim_compute_sim_state_mean0.03990379901880515
sim_compute_sim_state_median0.040065618661733776
sim_compute_sim_state_min0.035630655288696286
sim_physics_max0.26890310747870083
sim_physics_mean0.22649699444315655
sim_physics_median0.2190211931864421
sim_physics_min0.20217802606779953
sim_render-ego_max0.07313320554535964
sim_render-ego_mean0.06443805934190912
sim_render-ego_median0.06282641670920631
sim_render-ego_min0.06016817774091448
simulation-passed1
survival_time_max2.0500000000000007
survival_time_mean1.5066666666666673
survival_time_min1.1000000000000003
No reset possible