Duckietown Challenges Home Challenges Submissions

Submission 2856

Submission2856
Competingyes
Challengeaido2-LF-sim-testing
UserAlexander Karavaev
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 22737
Next
User labelchallenge-aido_LF-baseline-duckietown
Admin priority50
Blessingn/a
User priority50

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
22737step1-simulationsuccessyes0:34:59
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.2506382951605066
survival_time_median4.799999999999991
deviation-center-line_median0.2084327403075257
in-drivable-lane_median0.1999999999999993


other stats
agent_compute-ego_max0.11953585530504768
agent_compute-ego_mean0.1073600371497808
agent_compute-ego_median0.10810404045637265
agent_compute-ego_min0.09325301079522996
deviation-center-line_max0.34032468935256244
deviation-center-line_mean0.20945039143379293
deviation-center-line_min0.07549656914488843
deviation-heading_max2.2702669924732906
deviation-heading_mean1.666058580625534
deviation-heading_median1.6898654544228242
deviation-heading_min1.0089346537032806
driven_any_max1.174538089426396
driven_any_mean0.508103765596656
driven_any_median0.358881432506834
driven_any_min0.14594095312198974
driven_lanedir_consec_max0.5145104630569763
driven_lanedir_consec_mean0.31666408287668574
driven_lanedir_consec_min0.10339599919686993
driven_lanedir_max0.5145104630569763
driven_lanedir_mean0.31666408287668574
driven_lanedir_median0.2506382951605066
driven_lanedir_min0.10339599919686993
in-drivable-lane_max8.800000000000091
in-drivable-lane_mean1.8266666666666849
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 1.1707120947128424, "sim_physics": 0.1267444626490275, "survival_time": 14.950000000000076, "driven_lanedir": 0.5145104630569763, "sim_render-ego": 0.06865200519561768, "in-drivable-lane": 7.600000000000096, "agent_compute-ego": 0.11372513771057129, "deviation-heading": 2.0224655390890365, "set_robot_commands": 0.10873793681462604, "deviation-center-line": 0.34032468935256244, "driven_lanedir_consec": 0.5145104630569763, "sim_compute_sim_state": 0.04407287359237671, "sim_compute_performance-ego": 0.07576136509577433, "sim_compute_robot_state-ego": 0.08394577344258626}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.19815766139498991, "sim_physics": 0.13110584446362086, "survival_time": 2.799999999999998, "driven_lanedir": 0.10339599919686993, "sim_render-ego": 0.0653134116104671, "in-drivable-lane": 0.6499999999999977, "agent_compute-ego": 0.10455174105507987, "deviation-heading": 1.2604517984795416, "set_robot_commands": 0.10567751952580044, "deviation-center-line": 0.07549656914488843, "driven_lanedir_consec": 0.10339599919686993, "sim_compute_sim_state": 0.04115228567804609, "sim_compute_performance-ego": 0.07448898042951312, "sim_compute_robot_state-ego": 0.07938354781695775}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.14594095312198974, "sim_physics": 0.12636830085931822, "survival_time": 2.1500000000000004, "driven_lanedir": 0.12180489336724332, "sim_render-ego": 0.06778058894844942, "in-drivable-lane": 0, "agent_compute-ego": 0.10447256509647812, "deviation-heading": 1.0089346537032806, "set_robot_commands": 0.11214071650837742, "deviation-center-line": 0.20509776381830788, "driven_lanedir_consec": 0.12180489336724332, "sim_compute_sim_state": 0.04452327794806902, "sim_compute_performance-ego": 0.07436132985492085, "sim_compute_robot_state-ego": 0.08156596782595613}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.5477429518756478, "sim_physics": 0.12734883148353415, "survival_time": 7.149999999999983, "driven_lanedir": 0.4807143964063374, "sim_render-ego": 0.06393375596800051, "in-drivable-lane": 0.1999999999999993, "agent_compute-ego": 0.10689336603338068, "deviation-heading": 2.039336227550392, "set_robot_commands": 0.10442412983287463, "deviation-center-line": 0.22455249635290225, "driven_lanedir_consec": 0.4807143964063374, "sim_compute_sim_state": 0.04026778094418399, "sim_compute_performance-ego": 0.07307632819755927, "sim_compute_robot_state-ego": 0.0807385878129439}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 0.539700175382884, "sim_physics": 0.13639997759609357, "survival_time": 7.049999999999983, "driven_lanedir": 0.4430532861735075, "sim_render-ego": 0.07008454647470028, "in-drivable-lane": 0.5999999999999979, "agent_compute-ego": 0.11250782351121837, "deviation-heading": 1.6898654544228242, "set_robot_commands": 0.11280946021384382, "deviation-center-line": 0.2009608928383967, "driven_lanedir_consec": 0.4430532861735075, "sim_compute_sim_state": 0.04202535304617375, "sim_compute_performance-ego": 0.07588424242980091, "sim_compute_robot_state-ego": 0.09255181982162151}, "ETHZ_autolab_technical_track-5-0": {"driven_any": 0.358881432506834, "sim_physics": 0.12563508252302805, "survival_time": 4.799999999999991, "driven_lanedir": 0.24996581983127353, "sim_render-ego": 0.06692049652338028, "in-drivable-lane": 0.7499999999999973, "agent_compute-ego": 0.11225420236587524, "deviation-heading": 1.726371760301949, "set_robot_commands": 0.10355283071597415, "deviation-center-line": 0.2084327403075257, "driven_lanedir_consec": 0.24996581983127353, "sim_compute_sim_state": 0.04189896583557129, "sim_compute_performance-ego": 0.07695648322502772, "sim_compute_robot_state-ego": 0.08144701768954594}, "ETHZ_autolab_technical_track-6-0": {"driven_any": 1.174538089426396, "sim_physics": 0.12492018461227417, "survival_time": 14.950000000000076, "driven_lanedir": 0.4834346220808072, "sim_render-ego": 0.06658458789189657, "in-drivable-lane": 7.750000000000095, "agent_compute-ego": 0.114366131623586, "deviation-heading": 2.2702669924732906, "set_robot_commands": 0.11149310906728108, "deviation-center-line": 0.2743815030349422, "driven_lanedir_consec": 0.4834346220808072, "sim_compute_sim_state": 0.043011268774668376, "sim_compute_performance-ego": 0.07786723613739013, "sim_compute_robot_state-ego": 0.08638458410898844}, "ETHZ_autolab_technical_track-7-0": {"driven_any": 0.22630502192271323, "sim_physics": 0.11194834255036852, "survival_time": 3.149999999999997, "driven_lanedir": 0.20617849289485157, "sim_render-ego": 0.058874330823383635, "in-drivable-lane": 0, "agent_compute-ego": 0.09551143267798046, "deviation-heading": 1.087255853236586, "set_robot_commands": 0.09287117776416597, "deviation-center-line": 0.3065823317153645, "driven_lanedir_consec": 0.20617849289485157, "sim_compute_sim_state": 0.037994134993780224, "sim_compute_performance-ego": 0.06479831725832015, "sim_compute_robot_state-ego": 0.07105127970377605}, "ETHZ_autolab_technical_track-8-0": {"driven_any": 1.1745254111500234, "sim_physics": 0.11504963954289754, "survival_time": 14.950000000000076, "driven_lanedir": 0.4238096941710188, "sim_render-ego": 0.06144462108612061, "in-drivable-lane": 8.800000000000091, "agent_compute-ego": 0.10136522928873698, "deviation-heading": 1.798260683612272, "set_robot_commands": 0.09529559373855592, "deviation-center-line": 0.2257290346540354, "driven_lanedir_consec": 0.4238096941710188, "sim_compute_sim_state": 0.03769586165746053, "sim_compute_performance-ego": 0.07238796234130859, "sim_compute_robot_state-ego": 0.0757396936416626}, "ETHZ_autolab_technical_track-9-0": {"driven_any": 0.2986203233908063, "sim_physics": 0.129703692447992, "survival_time": 4.049999999999994, "driven_lanedir": 0.2448157841802927, "sim_render-ego": 0.06892095966103637, "in-drivable-lane": 0.09999999999999964, "agent_compute-ego": 0.11953585530504768, "deviation-heading": 1.566234589636264, "set_robot_commands": 0.11034879566710672, "deviation-center-line": 0.16339096533809297, "driven_lanedir_consec": 0.2448157841802927, "sim_compute_sim_state": 0.041597999172446165, "sim_compute_performance-ego": 0.07827583654427234, "sim_compute_robot_state-ego": 0.08336916970617977}, "ETHZ_autolab_technical_track-10-0": {"driven_any": 0.5155985329149106, "sim_physics": 0.13794118210121437, "survival_time": 6.749999999999984, "driven_lanedir": 0.4491283199122199, "sim_render-ego": 0.06592200950339988, "in-drivable-lane": 0.1999999999999993, "agent_compute-ego": 0.1107838895585802, "deviation-heading": 1.993315826634416, "set_robot_commands": 0.10637820031907824, "deviation-center-line": 0.2274651292636858, "driven_lanedir_consec": 0.4491283199122199, "sim_compute_sim_state": 0.04021088105660898, "sim_compute_performance-ego": 0.07363427303455494, "sim_compute_robot_state-ego": 0.08256285455491807}, "ETHZ_autolab_technical_track-11-0": {"driven_any": 0.4191766131754861, "sim_physics": 0.13334951314840232, "survival_time": 5.549999999999988, "driven_lanedir": 0.3844029845816549, "sim_render-ego": 0.06612631866523812, "in-drivable-lane": 0, "agent_compute-ego": 0.11098115079037778, "deviation-heading": 1.6242636432739648, "set_robot_commands": 0.10500492491163647, "deviation-center-line": 0.2473430511466462, "driven_lanedir_consec": 0.3844029845816549, "sim_compute_sim_state": 0.04124582780374063, "sim_compute_performance-ego": 0.07233163043185398, "sim_compute_robot_state-ego": 0.07826708028982352}, "ETHZ_autolab_technical_track-12-0": {"driven_any": 0.22227672727917808, "sim_physics": 0.11454647971737768, "survival_time": 3.099999999999997, "driven_lanedir": 0.1437249104087437, "sim_render-ego": 0.0626580253724129, "in-drivable-lane": 0.34999999999999876, "agent_compute-ego": 0.10209498097819666, "deviation-heading": 1.559514169617989, "set_robot_commands": 0.09887941037454911, "deviation-center-line": 0.1233476366123096, "driven_lanedir_consec": 0.1437249104087437, "sim_compute_sim_state": 0.03905685870878158, "sim_compute_performance-ego": 0.06966256710790819, "sim_compute_robot_state-ego": 0.07346145952901532}, "ETHZ_autolab_technical_track-13-0": {"driven_any": 0.3106729758488854, "sim_physics": 0.1021024017106919, "survival_time": 4.199999999999993, "driven_lanedir": 0.2506382951605066, "sim_render-ego": 0.058990810598645894, "in-drivable-lane": 0.1999999999999993, "agent_compute-ego": 0.09325301079522996, "deviation-heading": 1.4826504230190012, "set_robot_commands": 0.08735046500251406, "deviation-center-line": 0.11769296843007047, "driven_lanedir_consec": 0.2506382951605066, "sim_compute_sim_state": 0.03624081895464942, "sim_compute_performance-ego": 0.0640290720122201, "sim_compute_robot_state-ego": 0.06922773803983416}, "ETHZ_autolab_technical_track-14-0": {"driven_any": 0.31870751984625395, "sim_physics": 0.12498325802559077, "survival_time": 4.299999999999993, "driven_lanedir": 0.25038328172798296, "sim_render-ego": 0.06476710563482241, "in-drivable-lane": 0.1999999999999993, "agent_compute-ego": 0.10810404045637265, "deviation-heading": 1.8616910943321976, "set_robot_commands": 0.1033698320388794, "deviation-center-line": 0.2009580994971636, "driven_lanedir_consec": 0.25038328172798296, "sim_compute_sim_state": 0.038896419281183287, "sim_compute_performance-ego": 0.07343665943589321, "sim_compute_robot_state-ego": 0.08240098177000534}}
set_robot_commands_max0.11280946021384382
set_robot_commands_mean0.1038889401663509
set_robot_commands_median0.10500492491163647
set_robot_commands_min0.08735046500251406
sim_compute_performance-ego_max0.07827583654427234
sim_compute_performance-ego_mean0.07313015223575454
sim_compute_performance-ego_median0.07363427303455494
sim_compute_performance-ego_min0.0640290720122201
sim_compute_robot_state-ego_max0.09255181982162151
sim_compute_robot_state-ego_mean0.08013983705025432
sim_compute_robot_state-ego_median0.08144701768954594
sim_compute_robot_state-ego_min0.06922773803983416
sim_compute_sim_state_max0.04452327794806902
sim_compute_sim_state_mean0.040659373829849334
sim_compute_sim_state_median0.04115228567804609
sim_compute_sim_state_min0.03624081895464942
sim_physics_max0.13794118210121437
sim_physics_mean0.12454314622876211
sim_physics_median0.12636830085931822
sim_physics_min0.1021024017106919
sim_render-ego_max0.07008454647470028
sim_render-ego_mean0.06513157159717144
sim_render-ego_median0.06592200950339988
sim_render-ego_min0.058874330823383635
simulation-passed1
survival_time_max14.950000000000076
survival_time_mean6.660000000000008
survival_time_min2.1500000000000004
No reset possible
20974step1-simulationsuccessno0:35:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20287step1-simulationsuccessno0:11:49
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19339step1-simulationsuccessno0:14:14
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19307step1-simulationhost-errorno0:05:46
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 990, in run_docker
    return subprocess.check_output(cmd0, cwd=cwd, stderr=sys.stderr)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', '-p', 'job19307-662888', 'ps', '-q', 'solution']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 478, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/project/src/duckietown_challenges_runner/runner.py", line 573, in run_single
    write_logs(wd, project, services=config['services'])
  File "/project/src/duckietown_challenges_runner/runner.py", line 899, in write_logs
    container_id = get_id_for_service(wd, project, service)
  File "/project/src/duckietown_challenges_runner/runner.py", line 585, in get_id_for_service
    o = run_docker(wd, project, cmd, get_output=True)
  File "/project/src/duckietown_challenges_runner/runner.py", line 998, in run_docker
    raise DockerComposeFail(msg) from e
duckietown_challenges_runner.runner.DockerComposeFail: Could not run ['docker-compose', '-p', 'job19307-662888', 'ps', '-q', 'solution']:

   >  Command '['docker-compose', '-p', 'job19307-662888', 'ps', '-q', 'solution']' returned non-zero exit status 1.

stdout | 

stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19189step1-simulationerrorno0:05:26
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 173, in main
    make_video1(fn, out_video)
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
    pg('video_aido', dict(filename=fn, output=output))
  File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
    model.init()
  File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
    block.init()
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
    self.log = read_simulator_log_cbor(self.get_config('filename'))
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
    render_time = read_perfomance(ld)
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
    for i, ob in enumerate(read_topic2(ld, 'timing_information')):
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
    for ob in ld.objects:
AttributeError: 'str' object has no attribute 'objects'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 435, in <module>
    wrap(cie)
  File "experiment_manager.py", line 423, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 173, in main
 >     make_video1(fn, out_video)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
 >     pg('video_aido', dict(filename=fn, output=output))
 >   File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
 >     model.init()
 >   File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
 >     block.init()
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
 >     self.log = read_simulator_log_cbor(self.get_config('filename'))
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
 >     render_time = read_perfomance(ld)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
 >     for i, ob in enumerate(read_topic2(ld, 'timing_information')):
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
 >     for ob in ld.objects:
 > AttributeError: 'str' object has no attribute 'objects'
 > 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18595step1-simulationsuccessno0:22:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18594step1-simulationsuccessno0:11:23
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18561step1-simulationsuccessno0:11:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible