Duckietown Challenges Home Challenges Submissions

Submission 2826

Submission2826
Competingyes
Challengeaido2-LFVI-sim-testing
UserLiam Paull 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 22753
Next
User labelchallenge-aido_LF-template-random
Admin priority50
Blessingn/a
User priority50

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
22753step1-simulationsuccessyes0:32:35
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.6537162971975778
survival_time_median2.5999999999999988
deviation-center-line_median0.14884708769804722
in-drivable-lane_median0.549999999999998


other stats
agent_compute-ego_max0.17503853142261505
agent_compute-ego_mean0.1616996486587731
agent_compute-ego_median0.16267921707846902
agent_compute-ego_min0.14329748153686522
deviation-center-line_max0.5141120095458062
deviation-center-line_mean0.1769000227005324
deviation-center-line_min0.06216029860587694
deviation-heading_max1.9929437250081896
deviation-heading_mean0.692977823395608
deviation-heading_median0.4928440244444201
deviation-heading_min0.24234723632033853
driven_any_max3.5343244022418343
driven_any_mean1.3470385992234175
driven_any_median1.001633421251194
driven_any_min0.3412551960104244
driven_lanedir_consec_max2.494308053441959
driven_lanedir_consec_mean0.881391085527715
driven_lanedir_consec_min0.3199180625488627
driven_lanedir_max2.494308053441959
driven_lanedir_mean0.8962528018383226
driven_lanedir_median0.7206363324756615
driven_lanedir_min0.3199180625488627
in-drivable-lane_max3.549999999999991
in-drivable-lane_mean0.839999999999998
in-drivable-lane_min0
per-episodes
details{"udem1-0-0": {"driven_any": 1.5811782993597916, "sim_physics": 0.24190238155896152, "survival_time": 3.949999999999994, "driven_lanedir": 0.9217908232230632, "sim_render-ego": 0.06837659847887256, "in-drivable-lane": 1.3499999999999952, "agent_compute-ego": 0.16933427883099905, "deviation-heading": 0.5694288623982039, "set_robot_commands": 0.1078747163845014, "deviation-center-line": 0.2348196785686717, "driven_lanedir_consec": 0.9213701538661684, "sim_compute_sim_state": 0.04059939746615253, "sim_compute_performance-ego": 0.07509958593151238, "sim_compute_robot_state-ego": 0.08658451973637447, "sim_compute_robot_state-npc0": 0.07452456558806987, "sim_compute_robot_state-npc1": 0.07428433019903642, "sim_compute_robot_state-npc2": 0.07502128504499604, "sim_compute_robot_state-npc3": 0.07359993306896355}, "udem1-1-0": {"driven_any": 0.47798123795353353, "sim_physics": 0.2213175126484462, "survival_time": 1.4000000000000006, "driven_lanedir": 0.4500753586658095, "sim_render-ego": 0.06724798679351807, "in-drivable-lane": 0, "agent_compute-ego": 0.17040954317365373, "deviation-heading": 0.46504862964833654, "set_robot_commands": 0.10447372709001812, "deviation-center-line": 0.09791133281082, "driven_lanedir_consec": 0.4500753586658095, "sim_compute_sim_state": 0.04100292069571359, "sim_compute_performance-ego": 0.07221246617180961, "sim_compute_robot_state-ego": 0.08044728210994176, "sim_compute_robot_state-npc0": 0.07470400844301496, "sim_compute_robot_state-npc1": 0.07312654597418648, "sim_compute_robot_state-npc2": 0.0715107832636152, "sim_compute_robot_state-npc3": 0.07234995705740792}, "udem1-2-0": {"driven_any": 2.1864003300000694, "sim_physics": 0.2089520658104165, "survival_time": 5.14999999999999, "driven_lanedir": 1.826231945363564, "sim_render-ego": 0.06708352311143598, "in-drivable-lane": 0.549999999999998, "agent_compute-ego": 0.16503314138616174, "deviation-heading": 0.8115272458314897, "set_robot_commands": 0.10455103522365532, "deviation-center-line": 0.1927316041609401, "driven_lanedir_consec": 1.826231945363564, "sim_compute_sim_state": 0.04099220905489134, "sim_compute_performance-ego": 0.0739899028852148, "sim_compute_robot_state-ego": 0.08472587992844072, "sim_compute_robot_state-npc0": 0.07468117788000014, "sim_compute_robot_state-npc1": 0.07381924842167827, "sim_compute_robot_state-npc2": 0.07324514574217565, "sim_compute_robot_state-npc3": 0.07409904998483009}, "udem1-3-0": {"driven_any": 0.9410513078477724, "sim_physics": 0.24297440548737845, "survival_time": 2.3999999999999995, "driven_lanedir": 0.493517638549009, "sim_render-ego": 0.0673045168320338, "in-drivable-lane": 0.8999999999999988, "agent_compute-ego": 0.17503853142261505, "deviation-heading": 0.35154027506548424, "set_robot_commands": 0.10869709153970082, "deviation-center-line": 0.08975118546699407, "driven_lanedir_consec": 0.493517638549009, "sim_compute_sim_state": 0.04334000746409098, "sim_compute_performance-ego": 0.07461315393447876, "sim_compute_robot_state-ego": 0.08880828817685445, "sim_compute_robot_state-npc0": 0.07520978649457295, "sim_compute_robot_state-npc1": 0.07805013656616211, "sim_compute_robot_state-npc2": 0.07355680068333943, "sim_compute_robot_state-npc3": 0.07453201214472453}, "udem1-4-0": {"driven_any": 0.34749292376309954, "sim_physics": 0.23331678481329057, "survival_time": 1.0500000000000005, "driven_lanedir": 0.3325249708009643, "sim_render-ego": 0.06787588482811338, "in-drivable-lane": 0, "agent_compute-ego": 0.174367280233474, "deviation-heading": 0.30044835691402494, "set_robot_commands": 0.11362730889093307, "deviation-center-line": 0.06216029860587694, "driven_lanedir_consec": 0.3325249708009643, "sim_compute_sim_state": 0.043800456183297296, "sim_compute_performance-ego": 0.07624313944862002, "sim_compute_robot_state-ego": 0.08888275282723564, "sim_compute_robot_state-npc0": 0.07780563263666063, "sim_compute_robot_state-npc1": 0.07789668582734607, "sim_compute_robot_state-npc2": 0.08009172621227446, "sim_compute_robot_state-npc3": 0.07440700985136486}, "udem1-5-0": {"driven_any": 3.5343244022418343, "sim_physics": 0.2296221427574843, "survival_time": 8.349999999999984, "driven_lanedir": 2.494308053441959, "sim_render-ego": 0.06651998993879307, "in-drivable-lane": 2, "agent_compute-ego": 0.1661674176861426, "deviation-heading": 1.9929437250081896, "set_robot_commands": 0.10541931455006856, "deviation-center-line": 0.5141120095458062, "driven_lanedir_consec": 2.494308053441959, "sim_compute_sim_state": 0.041540832576637494, "sim_compute_performance-ego": 0.07415972641128266, "sim_compute_robot_state-ego": 0.08228771700830517, "sim_compute_robot_state-npc0": 0.07411146449471662, "sim_compute_robot_state-npc1": 0.07392770658710046, "sim_compute_robot_state-npc2": 0.07286546330252093, "sim_compute_robot_state-npc3": 0.0739904663519945}, "udem1-6-0": {"driven_any": 2.176040550989443, "sim_physics": 0.211460134358082, "survival_time": 5.14999999999999, "driven_lanedir": 1.9748003483573369, "sim_render-ego": 0.0629593307532153, "in-drivable-lane": 0.29999999999999893, "agent_compute-ego": 0.15849983807906365, "deviation-heading": 0.7877172510198454, "set_robot_commands": 0.09656974644336884, "deviation-center-line": 0.30661477465861414, "driven_lanedir_consec": 1.9748003483573369, "sim_compute_sim_state": 0.042041077197176736, "sim_compute_performance-ego": 0.07118459812645773, "sim_compute_robot_state-ego": 0.07727286422136918, "sim_compute_robot_state-npc0": 0.06916949124012178, "sim_compute_robot_state-npc1": 0.06975884576445644, "sim_compute_robot_state-npc2": 0.06833722521957841, "sim_compute_robot_state-npc3": 0.07080699170677407}, "udem1-7-0": {"driven_any": 1.0050236108346422, "sim_physics": 0.21209826946258545, "survival_time": 2.499999999999999, "driven_lanedir": 0.8701796426684156, "sim_render-ego": 0.06312098026275635, "in-drivable-lane": 0, "agent_compute-ego": 0.14329748153686522, "deviation-heading": 0.9227446658975216, "set_robot_commands": 0.09266141891479492, "deviation-center-line": 0.14884708769804722, "driven_lanedir_consec": 0.8546116490381102, "sim_compute_sim_state": 0.03849190711975098, "sim_compute_performance-ego": 0.06596234798431397, "sim_compute_robot_state-ego": 0.06990399837493896, "sim_compute_robot_state-npc0": 0.0662151288986206, "sim_compute_robot_state-npc1": 0.0689169979095459, "sim_compute_robot_state-npc2": 0.0704302453994751, "sim_compute_robot_state-npc3": 0.06821877479553223}, "udem1-8-0": {"driven_any": 0.9970416603019576, "sim_physics": 0.1861679508135869, "survival_time": 2.5999999999999988, "driven_lanedir": 0.8041152574540316, "sim_render-ego": 0.06302736355708195, "in-drivable-lane": 0.29999999999999893, "agent_compute-ego": 0.15932984077013457, "deviation-heading": 0.6817155913229235, "set_robot_commands": 0.0985938677420983, "deviation-center-line": 0.1898768080806288, "driven_lanedir_consec": 0.8041152574540316, "sim_compute_sim_state": 0.03962328342291025, "sim_compute_performance-ego": 0.06916679785801814, "sim_compute_robot_state-ego": 0.07313522467246422, "sim_compute_robot_state-npc0": 0.07074390924893893, "sim_compute_robot_state-npc1": 0.07074678402680618, "sim_compute_robot_state-npc2": 0.07216124351208027, "sim_compute_robot_state-npc3": 0.06717104178208572}, "udem1-9-0": {"driven_any": 0.4374998710472299, "sim_physics": 0.22933509716620812, "survival_time": 1.3000000000000005, "driven_lanedir": 0.4177527119659043, "sim_render-ego": 0.06367901655343863, "in-drivable-lane": 0, "agent_compute-ego": 0.15621323768909162, "deviation-heading": 0.3796137446285097, "set_robot_commands": 0.09923061957726112, "deviation-center-line": 0.08207813392324871, "driven_lanedir_consec": 0.4177527119659043, "sim_compute_sim_state": 0.04063542072589581, "sim_compute_performance-ego": 0.06412511605482835, "sim_compute_robot_state-ego": 0.0736994926746075, "sim_compute_robot_state-npc0": 0.06700860537015475, "sim_compute_robot_state-npc1": 0.07145221416766827, "sim_compute_robot_state-npc2": 0.06968254309434158, "sim_compute_robot_state-npc3": 0.07190390733572152}, "udem1-10-0": {"driven_any": 0.3412551960104244, "sim_physics": 0.20011992888017135, "survival_time": 1.1000000000000003, "driven_lanedir": 0.3199180625488627, "sim_render-ego": 0.06140307946638627, "in-drivable-lane": 0, "agent_compute-ego": 0.16267921707846902, "deviation-heading": 0.3796986873122644, "set_robot_commands": 0.09430048682472925, "deviation-center-line": 0.06276575300040536, "driven_lanedir_consec": 0.3199180625488627, "sim_compute_sim_state": 0.03961896896362305, "sim_compute_performance-ego": 0.06663214076649059, "sim_compute_robot_state-ego": 0.0740671157836914, "sim_compute_robot_state-npc0": 0.06950640678405762, "sim_compute_robot_state-npc1": 0.06467446413907138, "sim_compute_robot_state-npc2": 0.0688620155507868, "sim_compute_robot_state-npc3": 0.07256188175894997}, "udem1-11-0": {"driven_any": 0.9951985237696338, "sim_physics": 0.2167334890365601, "survival_time": 2.499999999999999, "driven_lanedir": 0.7206363324756615, "sim_render-ego": 0.06541077613830566, "in-drivable-lane": 0.549999999999998, "agent_compute-ego": 0.16791627883911134, "deviation-heading": 0.30074636349085415, "set_robot_commands": 0.10590073585510254, "deviation-center-line": 0.16567670185555422, "driven_lanedir_consec": 0.7185075497455284, "sim_compute_sim_state": 0.04037092685699463, "sim_compute_performance-ego": 0.07113199710845947, "sim_compute_robot_state-ego": 0.08040196895599365, "sim_compute_robot_state-npc0": 0.07548116207122803, "sim_compute_robot_state-npc1": 0.07278655052185058, "sim_compute_robot_state-npc2": 0.07131700992584228, "sim_compute_robot_state-npc3": 0.07201198577880859}, "udem1-12-0": {"driven_any": 1.964687870253437, "sim_physics": 0.2003512940508254, "survival_time": 4.699999999999991, "driven_lanedir": 0.35162821871931205, "sim_render-ego": 0.0630185325094994, "in-drivable-lane": 3.549999999999991, "agent_compute-ego": 0.15175351183465186, "deviation-heading": 0.24234723632033853, "set_robot_commands": 0.098077581283894, "deviation-center-line": 0.13126670634798973, "driven_lanedir_consec": 0.35162821871931205, "sim_compute_sim_state": 0.039169882206206624, "sim_compute_performance-ego": 0.0710801418791426, "sim_compute_robot_state-ego": 0.07393046642871613, "sim_compute_robot_state-npc0": 0.06866304164237165, "sim_compute_robot_state-npc1": 0.07016004400050387, "sim_compute_robot_state-npc2": 0.06939416996976162, "sim_compute_robot_state-npc3": 0.07023952108748416}, "udem1-13-0": {"driven_any": 1.001633421251194, "sim_physics": 0.1976577639579773, "survival_time": 2.5999999999999988, "driven_lanedir": 0.6537162971975778, "sim_render-ego": 0.0623513597708482, "in-drivable-lane": 0.5999999999999979, "agent_compute-ego": 0.14942414943988508, "deviation-heading": 0.4928440244444201, "set_robot_commands": 0.0986908353292025, "deviation-center-line": 0.13361893417935136, "driven_lanedir_consec": 0.6537162971975778, "sim_compute_sim_state": 0.03967227844091562, "sim_compute_performance-ego": 0.06678574818831223, "sim_compute_robot_state-ego": 0.0733109712600708, "sim_compute_robot_state-npc0": 0.06788832866228543, "sim_compute_robot_state-npc1": 0.07110820366786076, "sim_compute_robot_state-npc2": 0.0679028584406926, "sim_compute_robot_state-npc3": 0.06762800308374259}, "udem1-14-0": {"driven_any": 2.218769782727203, "sim_physics": 0.18841309320359004, "survival_time": 5.249999999999989, "driven_lanedir": 0.8125963661433656, "sim_render-ego": 0.06030330657958984, "in-drivable-lane": 2.499999999999993, "agent_compute-ego": 0.1560309818812779, "deviation-heading": 1.7163026916317128, "set_robot_commands": 0.09567794572739374, "deviation-center-line": 0.24126933160503777, "driven_lanedir_consec": 0.6077880672015845, "sim_compute_sim_state": 0.03802407128470285, "sim_compute_performance-ego": 0.07011439232599168, "sim_compute_robot_state-ego": 0.07565754027593703, "sim_compute_robot_state-npc0": 0.0696953432900565, "sim_compute_robot_state-npc1": 0.06885545594351632, "sim_compute_robot_state-npc2": 0.06692738305954705, "sim_compute_robot_state-npc3": 0.06964860870724633}}
set_robot_commands_max0.11362730889093307
set_robot_commands_mean0.1016230954251148
set_robot_commands_median0.09923061957726112
set_robot_commands_min0.09266141891479492
sim_compute_performance-ego_max0.07624313944862002
sim_compute_performance-ego_mean0.07083341700499554
sim_compute_performance-ego_median0.07113199710845947
sim_compute_performance-ego_min0.06412511605482835
sim_compute_robot_state-ego_max0.08888275282723564
sim_compute_robot_state-ego_mean0.07887440549566273
sim_compute_robot_state-ego_median0.07727286422136918
sim_compute_robot_state-ego_min0.06990399837493896
sim_compute_robot_state-npc0_max0.07780563263666063
sim_compute_robot_state-npc0_mean0.07169387018299136
sim_compute_robot_state-npc0_median0.07074390924893893
sim_compute_robot_state-npc0_min0.0662151288986206
sim_compute_robot_state-npc1_max0.07805013656616211
sim_compute_robot_state-npc1_mean0.0719709475811193
sim_compute_robot_state-npc1_median0.07145221416766827
sim_compute_robot_state-npc1_min0.06467446413907138
sim_compute_robot_state-npc2_max0.08009172621227446
sim_compute_robot_state-npc2_mean0.07142039322806851
sim_compute_robot_state-npc2_median0.07131700992584228
sim_compute_robot_state-npc2_min0.06692738305954705
sim_compute_robot_state-npc3_max0.07453201214472453
sim_compute_robot_state-npc3_mean0.07154460963304204
sim_compute_robot_state-npc3_median0.07201198577880859
sim_compute_robot_state-npc3_min0.06717104178208572
sim_compute_sim_state_max0.043800456183297296
sim_compute_sim_state_mean0.04059490931059732
sim_compute_sim_state_median0.04059939746615253
sim_compute_sim_state_min0.03802407128470285
sim_physics_max0.24297440548737845
sim_physics_mean0.2146948209337043
sim_physics_median0.21209826946258545
sim_physics_min0.1861679508135869
sim_render-ego_max0.06837659847887256
sim_render-ego_mean0.06464548303825922
sim_render-ego_median0.06367901655343863
sim_render-ego_min0.06030330657958984
simulation-passed1
survival_time_max8.349999999999984
survival_time_mean3.3333333333333286
survival_time_min1.0500000000000005
No reset possible
21753step1-simulationsuccessno0:31:43
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
21739step1-simulationhost-errorno0:27:21
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 347, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in convert
    with open(fn, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2826/step1-simulation-ip-172-31-25-98-2348-job21739/logs/challenges-runner/stdout.html'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
21684step1-simulationtimeoutno1:05:10
I can see how the jo [...]
I can see how the job 21684 is timeout because passed 3910 seconds and the timeout is 3600.0.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
21682step1-simulationtimeoutno1:05:29
I can see how the jo [...]
I can see how the job 21682 is timeout because passed 3929 seconds and the timeout is 3600.0.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
21668step1-simulationhost-errorno0:25:05
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 484, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 332, in convert
    data = open(log).read().strip()
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2826/step1-simulation-ip-172-31-42-167-7194-job21668/logs/challenges-runner/stdout.log'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20989step1-simulationsuccessno0:21:40
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20309step1-simulationsuccessno0:09:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20307step1-simulationsuccessno0:08:39
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20055step1-simulationsuccessno0:09:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19818step1-simulationsuccessno0:06:36
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19747step1-simulationsuccessno0:06:52
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19321step1-simulationsuccessno0:07:33
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18616step1-simulationsuccessno0:11:32
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18615step1-simulationhost-errorno0:01:41
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 990, in run_docker
    return subprocess.check_output(cmd0, cwd=cwd, stderr=sys.stderr)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', '-p', 'job18615-918310', 'ps', '-q', 'solution']' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 478, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/project/src/duckietown_challenges_runner/runner.py", line 573, in run_single
    write_logs(wd, project, services=config['services'])
  File "/project/src/duckietown_challenges_runner/runner.py", line 899, in write_logs
    container_id = get_id_for_service(wd, project, service)
  File "/project/src/duckietown_challenges_runner/runner.py", line 585, in get_id_for_service
    o = run_docker(wd, project, cmd, get_output=True)
  File "/project/src/duckietown_challenges_runner/runner.py", line 998, in run_docker
    raise DockerComposeFail(msg) from e
duckietown_challenges_runner.runner.DockerComposeFail: Could not run ['docker-compose', '-p', 'job18615-918310', 'ps', '-q', 'solution']:

   >  Command '['docker-compose', '-p', 'job18615-918310', 'ps', '-q', 'solution']' returned non-zero exit status 1.

stdout | 

stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18283step1-simulationsuccessno0:07:45
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18282step1-simulationsuccessno0:13:35
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18162step1-simulationsuccessno0:05:53
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18158step1-simulationsuccessno0:06:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible