Duckietown Challenges Home Challenges Submissions

Evaluator 1858

ID1858
evaluatorreg01
ownerI don't have one πŸ˜€
machinearchimede
processreg01
versiond-c:6.0.31;d-c-r:6.0.15
first heard
last heard
statusinactive
# evaluating
# success5 33320
# timeout
# failed1 33356
# error
# aborted2 33314
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
333856817Anthony CourchesneΒ πŸ‡¨πŸ‡¦exercise_ros_templateaido5-LF-sim-validationLFv-simsuccessnoreg010:14:37
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.9108257388504304
survival_time_median7.299999999999987
deviation-center-line_median0.416995561612648
in-drivable-lane_median0.0


other stats
agent_compute-ego_max0.03108164302089758
agent_compute-ego_mean0.030672209001956347
agent_compute-ego_median0.030759567645463073
agent_compute-ego_min0.030200268771197344
complete-iteration_max0.28917584084627923
complete-iteration_mean0.2679982573798404
complete-iteration_median0.2658069110881778
complete-iteration_min0.2554517053067684
deviation-center-line_max1.0005393760093493
deviation-center-line_mean0.47299276284864344
deviation-center-line_min0.15018943264297369
deviation-heading_max3.458435103483227
deviation-heading_mean1.5789535708996485
deviation-heading_median1.127002610574158
deviation-heading_min1.0431063466706216
driven_any_max1.9399377525437296
driven_any_mean1.03582796197342
driven_any_median0.9242754956696432
driven_any_min0.37989546253513634
driven_lanedir_consec_max1.8442816995038704
driven_lanedir_consec_mean0.9993305773093976
driven_lanedir_consec_min0.3480555274901094
driven_lanedir_max1.8442816995038704
driven_lanedir_mean0.9993305773093976
driven_lanedir_median0.9108257388504304
driven_lanedir_min0.3480555274901094
get_state_dump_max0.0213118737882322
get_state_dump_mean0.020867912403701783
get_state_dump_median0.020890079308478134
get_state_dump_min0.020453360703614382
get_ui_image_max0.046559195769460576
get_ui_image_mean0.04603124938893234
get_ui_image_median0.04620958380692139
get_ui_image_min0.04489747683207194
in-drivable-lane_max0.0
in-drivable-lane_mean0.0
in-drivable-lane_min0.0
per-episodes
details{"ETHZ_autolab_technical_track-0-0-ego": {"driven_any": 1.8871756896102352, "get_ui_image": 0.046295507313453985, "step_physics": 0.09155991306043652, "survival_time": 14.600000000000072, "driven_lanedir": 1.8017291112612652, "get_state_dump": 0.020703374523006073, "sim_render-ego": 0.007636943908586894, "in-drivable-lane": 0.0, "agent_compute-ego": 0.030238052753553, "deviation-heading": 3.4151689642096064, "complete-iteration": 0.266363798755489, "set_robot_commands": 0.006548038900715031, "deviation-center-line": 0.9423607958044778, "driven_lanedir_consec": 1.8017291112612652, "sim_compute_sim_state": 0.03503922566975633, "sim_compute_performance-ego": 0.006247673132648207, "sim_compute_robot_state-ego": 0.010430254348336832, "sim_compute_robot_state-parked0": 0.01141723541364278}, "ETHZ_autolab_technical_track-0-1-ego": {"driven_any": 1.9399377525437296, "get_ui_image": 0.04607408920923869, "step_physics": 0.0914242990811666, "survival_time": 14.950000000000076, "driven_lanedir": 1.8442816995038704, "get_state_dump": 0.02062805414199829, "sim_render-ego": 0.007657119433085124, "in-drivable-lane": 0.0, "agent_compute-ego": 0.03069087743759155, "deviation-heading": 3.458435103483227, "complete-iteration": 0.267750727335612, "set_robot_commands": 0.007065747578938802, "deviation-center-line": 1.0005393760093493, "driven_lanedir_consec": 1.8442816995038704, "sim_compute_sim_state": 0.03497854471206665, "sim_compute_performance-ego": 0.006340161959330241, "sim_compute_robot_state-ego": 0.010987353324890136, "sim_compute_robot_state-parked0": 0.01165531317392985}, "ETHZ_autolab_technical_track-1-0-ego": {"driven_any": 0.37989546253513634, "get_ui_image": 0.04525473341345787, "step_physics": 0.08903070539236069, "survival_time": 3.1999999999999966, "driven_lanedir": 0.3480555274901094, "get_state_dump": 0.021088857203722, "sim_render-ego": 0.007390856742858887, "in-drivable-lane": 0.0, "agent_compute-ego": 0.0303792841732502, "deviation-heading": 1.1491939014250685, "complete-iteration": 0.2554517053067684, "set_robot_commands": 0.006378848105669022, "deviation-center-line": 0.18151689914565472, "driven_lanedir_consec": 0.3480555274901094, "sim_compute_sim_state": 0.027828935533761975, "sim_compute_performance-ego": 0.0062357112765312195, "sim_compute_robot_state-ego": 0.010366827249526978, "sim_compute_robot_state-parked0": 0.011251747608184814}, "ETHZ_autolab_technical_track-1-1-ego": {"driven_any": 0.6919192084489386, "get_ui_image": 0.045820835474375134, "step_physics": 0.08969211148786115, "survival_time": 5.549999999999988, "driven_lanedir": 0.6777733962352692, "get_state_dump": 0.020736980008649396, "sim_render-ego": 0.008344517097816811, "in-drivable-lane": 0.0, "agent_compute-ego": 0.03040108809600005, "deviation-heading": 1.0681468421570182, "complete-iteration": 0.26453605858055323, "set_robot_commands": 0.008367516972997167, "deviation-center-line": 0.34034936872778193, "driven_lanedir_consec": 0.6777733962352692, "sim_compute_sim_state": 0.028434014535165048, "sim_compute_performance-ego": 0.007110522674010681, "sim_compute_robot_state-ego": 0.012697196221566415, "sim_compute_robot_state-parked0": 0.01268044463149062}, "ETHZ_autolab_technical_track-2-0-ego": {"driven_any": 0.6919212031132335, "get_ui_image": 0.046500304797748186, "step_physics": 0.09080038199553618, "survival_time": 5.549999999999988, "driven_lanedir": 0.6478918010181687, "get_state_dump": 0.020453360703614382, "sim_render-ego": 0.0076588510393022415, "in-drivable-lane": 0.0, "agent_compute-ego": 0.030200268771197344, "deviation-heading": 1.1005356311779118, "complete-iteration": 0.26525002342086657, "set_robot_commands": 0.006764390447118261, "deviation-center-line": 0.15711486252821372, "driven_lanedir_consec": 0.6478918010181687, "sim_compute_sim_state": 0.034565693623310814, "sim_compute_performance-ego": 0.006069943711564348, "sim_compute_robot_state-ego": 0.0105622910164498, "sim_compute_robot_state-parked0": 0.01142400449460691}, "ETHZ_autolab_technical_track-2-1-ego": {"driven_any": 0.691921107823358, "get_ui_image": 0.04489747683207194, "step_physics": 0.0886901584831444, "survival_time": 5.549999999999988, "driven_lanedir": 0.6493044804218419, "get_state_dump": 0.0213118737882322, "sim_render-ego": 0.008328850204880172, "in-drivable-lane": 0.0, "agent_compute-ego": 0.030997398737314587, "deviation-heading": 1.1048113197232476, "complete-iteration": 0.2709892870069624, "set_robot_commands": 0.008544307571273667, "deviation-center-line": 0.15018943264297369, "driven_lanedir_consec": 0.6493044804218419, "sim_compute_sim_state": 0.034503893809275585, "sim_compute_performance-ego": 0.007165111936964431, "sim_compute_robot_state-ego": 0.01318845018610224, "sim_compute_robot_state-parked0": 0.01310801935625506}, "ETHZ_autolab_technical_track-3-0-ego": {"driven_any": 1.0902440411577237, "get_ui_image": 0.046559195769460576, "step_physics": 0.11293773902089972, "survival_time": 8.549999999999986, "driven_lanedir": 1.07959644242594, "get_state_dump": 0.020946115081073247, "sim_render-ego": 0.008353441082246123, "in-drivable-lane": 0.0, "agent_compute-ego": 0.03108164302089758, "deviation-heading": 1.0431063466706216, "complete-iteration": 0.28917584084627923, "set_robot_commands": 0.008012442560920938, "deviation-center-line": 0.48544332664247675, "driven_lanedir_consec": 1.07959644242594, "sim_compute_sim_state": 0.028813661887631774, "sim_compute_performance-ego": 0.007156669047840855, "sim_compute_robot_state-ego": 0.012391091787327104, "sim_compute_robot_state-parked0": 0.012665826674790413}, "ETHZ_autolab_technical_track-3-1-ego": {"driven_any": 1.1035214072648611, "get_ui_image": 0.0463481376625899, "step_physics": 0.11194273226522986, "survival_time": 8.649999999999988, "driven_lanedir": 1.0914431577156762, "get_state_dump": 0.020834043535883025, "sim_render-ego": 0.007642752862390067, "in-drivable-lane": 0.0, "agent_compute-ego": 0.030980067446052684, "deviation-heading": 1.0450504528857063, "complete-iteration": 0.28187305803243823, "set_robot_commands": 0.0065558618203752994, "deviation-center-line": 0.4513140921881332, "driven_lanedir_consec": 1.0914431577156762, "sim_compute_sim_state": 0.029053143683196492, "sim_compute_performance-ego": 0.006279016505776113, "sim_compute_robot_state-ego": 0.01059828190445211, "sim_compute_robot_state-parked0": 0.011383055262482924}, "ETHZ_autolab_technical_track-4-0-ego": {"driven_any": 0.758306950181563, "get_ui_image": 0.04612366030038881, "step_physics": 0.08834989602900734, "survival_time": 6.0499999999999865, "driven_lanedir": 0.7420550352749209, "get_state_dump": 0.020985853573507515, "sim_render-ego": 0.007733969649007497, "in-drivable-lane": 0.0, "agent_compute-ego": 0.0309251517303719, "deviation-heading": 1.1603210959043766, "complete-iteration": 0.2581192363392223, "set_robot_commands": 0.006452532839183965, "deviation-center-line": 0.3826770310371628, "driven_lanedir_consec": 0.7420550352749209, "sim_compute_sim_state": 0.028779575647401415, "sim_compute_performance-ego": 0.006425891040770476, "sim_compute_robot_state-ego": 0.01044188846241344, "sim_compute_robot_state-parked0": 0.011648091402920807}, "ETHZ_autolab_technical_track-4-1-ego": {"driven_any": 1.123436797055423, "get_ui_image": 0.04643855311653831, "step_physics": 0.08771346509456635, "survival_time": 8.79999999999999, "driven_lanedir": 1.111175121746914, "get_state_dump": 0.02099061147733168, "sim_render-ego": 0.007947313514622774, "in-drivable-lane": 0.0, "agent_compute-ego": 0.0308282578533346, "deviation-heading": 1.2447660513597003, "complete-iteration": 0.2604728381742131, "set_robot_commands": 0.0074145252054387874, "deviation-center-line": 0.6384224437602106, "driven_lanedir_consec": 1.111175121746914, "sim_compute_sim_state": 0.028758216987956657, "sim_compute_performance-ego": 0.006514583121646534, "sim_compute_robot_state-ego": 0.011598066850142046, "sim_compute_robot_state-parked0": 0.012013588439334522}}
set_robot_commands_max0.008544307571273667
set_robot_commands_mean0.007210421200263094
set_robot_commands_median0.006915069013028532
set_robot_commands_min0.006378848105669022
sim_compute_performance-ego_max0.007165111936964431
sim_compute_performance-ego_mean0.00655452844070831
sim_compute_performance-ego_median0.006383026500050359
sim_compute_performance-ego_min0.006069943711564348
sim_compute_robot_state-ego_max0.01318845018610224
sim_compute_robot_state-ego_mean0.01132617013512071
sim_compute_robot_state-ego_median0.010792817614671124
sim_compute_robot_state-ego_min0.010366827249526978
sim_compute_robot_state-parked0_max0.01310801935625506
sim_compute_robot_state-parked0_mean0.01192473264576387
sim_compute_robot_state-parked0_median0.011651702288425329
sim_compute_robot_state-parked0_min0.011251747608184814
sim_compute_sim_state_max0.03503922566975633
sim_compute_sim_state_mean0.03107549060895227
sim_compute_sim_state_median0.028933402785414133
sim_compute_sim_state_min0.027828935533761975
sim_render-ego_max0.008353441082246123
sim_render-ego_mean0.007869461553479657
sim_render-ego_median0.007696410344154869
sim_render-ego_min0.007390856742858887
simulation-passed1
step_physics_max0.11293773902089972
step_physics_mean0.09421414019102088
step_physics_median0.09024624674169866
step_physics_min0.08771346509456635
survival_time_max14.950000000000076
survival_time_mean8.145000000000007
survival_time_min3.1999999999999966
No reset possible
333836815Bence HadlaczkyPredictorLastaido2-PREDstep1-simulationsuccessyesreg010:00:28
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
error_L10.11017473382982
error_L20.02175711920110779


No reset possible
333726810Caleb BGΒ πŸ‡ΊπŸ‡ΈPredictorLastaido2-PREDstep1-simulationsuccessyesreg010:00:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
error_L10.10344491912496398
error_L20.017661495596809625


No reset possible
333566800Anthony CourchesneΒ πŸ‡¨πŸ‡¦baseline-duckietownaido5-LF-sim-validationLFv-simfailednoreg010:01:05
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "experiment_manager.py", line 437, in run_episode
    agent_ci.write_topic_and_expect_zero("observations", obs_plus)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 141, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 299, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/usr/local/lib/python3.8/site-packages/zuper_nodes_wrapper/wrapper_outside.py", line 327, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "ego" aborted:

error in ego |Exception while handling a message on topic "observations".
             |
             || Traceback (most recent call last):
             ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
             ||     f(**kwargs)
             ||   File "solution.py", line 39, in on_received_observations
             ||     jpg_data = data['camera']['jpg_data']
             || TypeError: 'Duckiebot1Observations' object is not subscriptable
             ||
             || The above exception was the direct cause of the following exception:
             ||
             || Traceback (most recent call last):
             ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 319, in loop
             ||     handle_message_node(parsed, receiver0, context0)
             ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 474, in handle_message_node
             ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
             ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 24, in call_if_fun_exists
             ||     raise ZTypeError(msg, f=f, args=kwargs, argspec=a) from e
             || zuper_commons.types.exceptions.ZTypeError: Cannot call function <bound method ROSBaselineAgent.on_received_observations of <__main__.ROSBaselineAgent object at 0x7f0f6babe040>>.
             || β”‚       f: <bound method ROSBaselineAgent.on_received_observations of <__main__.ROSBaselineAgent object at 0x7f0f6babe040>>
             || β”‚    args: dict[2]
             || β”‚          β”‚ data:
             || β”‚          β”‚ Duckiebot1Observations
             || β”‚          β”‚ β”‚ camera: JPGImage(jpg_data=42809 bytes b'\xff\xd8\xff\xe0\x00\x10JFIF')
             || β”‚          β”‚ context: <zuper_nodes_wrapper.wrapper.ConcreteContext object at 0x7f0f50275940>
             || β”‚ argspec: <class 'inspect.FullArgSpec'>[7]
             || β”‚          #0 [self, context, data]
             || β”‚          #1 None
             || β”‚          #2 None
             || β”‚          #3 None
             || β”‚          #4 []
             || β”‚          #5 None
             || β”‚          #6 dict[2]
             || β”‚             β”‚ context: <class 'zuper_nodes_wrapper.interface.Context'>
             || β”‚             β”‚ data:
             || β”‚             β”‚ dataclass aido_schemas.schemas.Duckiebot1Observations
             || β”‚             β”‚  field  camera : dataclass aido_schemas.protocol_simulator.JPGImage
             || β”‚             β”‚                   field jpg_data : bytes
             || β”‚             β”‚                          __doc__
             || β”‚             β”‚                                            An image in JPG format.
             || β”‚             β”‚
             || β”‚             β”‚                                            jpg_data
             || β”‚             β”‚        __doc__   Duckiebot1Observations(camera: aido_schemas.protoc
             ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 666, in scoring_context
    yield cie
  File "experiment_manager.py", line 607, in <module>
    wrap(cie)
  File "experiment_manager.py", line 595, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "experiment_manager.py", line 213, in main
    length_s = await run_episode(
  File "experiment_manager.py", line 447, in run_episode
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: Trouble with communication to the agent.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
333296786Dejan MilojevicΒ πŸ‡¨πŸ‡­PredictorLastaido2-PREDstep1-simulationabortedyesreg010:00:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
error_L10.11959038661934109
error_L20.022791485555247733


No reset possible
333276785David DubachΒ πŸ‡¨πŸ‡­PredictorLastaido2-PREDstep1-simulationsuccessyesreg010:00:26
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
error_L10.1152679163692428
error_L20.02148878939166793


No reset possible
333206781Silvan LΓΆwΒ πŸ‡¨πŸ‡­PredictorLastaido2-PREDstep1-simulationsuccessyesreg010:00:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
error_L10.12137416414455288
error_L20.023241781602475572


No reset possible
333146777Frederic LetschPredictorLastaido2-PREDstep1-simulationabortedyesreg010:00:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
error_L10.10931251740857088
error_L20.018665621472175885


No reset possible