Duckietown Challenges Home Challenges Submissions

Submission 2399

Submission2399
Competingyes
Challengeaido2-LFVI-sim-testing
UserLiam Paull 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 23006
Next
User labelrandom_agent
Admin priority50
Blessingn/a
User priority50

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
23006step1-simulationsuccessyes0:31:35
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.6537162971975778
survival_time_median2.5999999999999988
deviation-center-line_median0.14884708769804722
in-drivable-lane_median0.549999999999998


other stats
agent_compute-ego_max0.15876758098602295
agent_compute-ego_mean0.14293282439578114
agent_compute-ego_median0.1440584109379695
agent_compute-ego_min0.1151271915435791
deviation-center-line_max0.5141120095458062
deviation-center-line_mean0.1769000227005324
deviation-center-line_min0.06216029860587694
deviation-heading_max1.9929437250081896
deviation-heading_mean0.692977823395608
deviation-heading_median0.4928440244444201
deviation-heading_min0.24234723632033853
driven_any_max3.5343244022418343
driven_any_mean1.3470385992234175
driven_any_median1.001633421251194
driven_any_min0.3412551960104244
driven_lanedir_consec_max2.494308053441959
driven_lanedir_consec_mean0.881391085527715
driven_lanedir_consec_min0.3199180625488627
driven_lanedir_max2.494308053441959
driven_lanedir_mean0.8962528018383226
driven_lanedir_median0.7206363324756615
driven_lanedir_min0.3199180625488627
in-drivable-lane_max3.549999999999991
in-drivable-lane_mean0.839999999999998
in-drivable-lane_min0
per-episodes
details{"udem1-0-0": {"driven_any": 1.5811782993597916, "sim_physics": 0.20626541330844544, "survival_time": 3.949999999999994, "driven_lanedir": 0.9217908232230632, "sim_render-ego": 0.06594670573367348, "in-drivable-lane": 1.3499999999999952, "agent_compute-ego": 0.14196757123440126, "deviation-heading": 0.5694288623982039, "set_robot_commands": 0.0992590294608587, "deviation-center-line": 0.2348196785686717, "driven_lanedir_consec": 0.9213701538661684, "sim_compute_sim_state": 0.041479747506636605, "sim_compute_performance-ego": 0.0735602107229112, "sim_compute_robot_state-ego": 0.077212246158455, "sim_compute_robot_state-npc0": 0.07337225841570504, "sim_compute_robot_state-npc1": 0.07369668272477162, "sim_compute_robot_state-npc2": 0.07147967060909996, "sim_compute_robot_state-npc3": 0.07265432273285298}, "udem1-1-0": {"driven_any": 0.47798123795353353, "sim_physics": 0.22035236018044607, "survival_time": 1.4000000000000006, "driven_lanedir": 0.4500753586658095, "sim_render-ego": 0.06974882738930839, "in-drivable-lane": 0, "agent_compute-ego": 0.145516642502376, "deviation-heading": 0.46504862964833654, "set_robot_commands": 0.10073697566986084, "deviation-center-line": 0.09791133281082, "driven_lanedir_consec": 0.4500753586658095, "sim_compute_sim_state": 0.04100521121706281, "sim_compute_performance-ego": 0.07501309258597237, "sim_compute_robot_state-ego": 0.08231098311288017, "sim_compute_robot_state-npc0": 0.0733947583607265, "sim_compute_robot_state-npc1": 0.07568772350038801, "sim_compute_robot_state-npc2": 0.07574936321803502, "sim_compute_robot_state-npc3": 0.07293503625052315}, "udem1-2-0": {"driven_any": 2.1864003300000694, "sim_physics": 0.20772280970823415, "survival_time": 5.14999999999999, "driven_lanedir": 1.826231945363564, "sim_render-ego": 0.0656303040032248, "in-drivable-lane": 0.549999999999998, "agent_compute-ego": 0.14655368999370094, "deviation-heading": 0.8115272458314897, "set_robot_commands": 0.10222257688207534, "deviation-center-line": 0.1927316041609401, "driven_lanedir_consec": 1.826231945363564, "sim_compute_sim_state": 0.04199059032699437, "sim_compute_performance-ego": 0.07207248974772333, "sim_compute_robot_state-ego": 0.07805474522044357, "sim_compute_robot_state-npc0": 0.07522412874166248, "sim_compute_robot_state-npc1": 0.07403288767175767, "sim_compute_robot_state-npc2": 0.07594718979400339, "sim_compute_robot_state-npc3": 0.07651922772231612}, "udem1-3-0": {"driven_any": 0.9410513078477724, "sim_physics": 0.22813271482785544, "survival_time": 2.3999999999999995, "driven_lanedir": 0.493517638549009, "sim_render-ego": 0.06932537754376729, "in-drivable-lane": 0.8999999999999988, "agent_compute-ego": 0.14405137300491333, "deviation-heading": 0.35154027506548424, "set_robot_commands": 0.1019674688577652, "deviation-center-line": 0.08975118546699407, "driven_lanedir_consec": 0.493517638549009, "sim_compute_sim_state": 0.0439109206199646, "sim_compute_performance-ego": 0.07799484829107921, "sim_compute_robot_state-ego": 0.0845315009355545, "sim_compute_robot_state-npc0": 0.07607050736745198, "sim_compute_robot_state-npc1": 0.0799860308567683, "sim_compute_robot_state-npc2": 0.07520255446434021, "sim_compute_robot_state-npc3": 0.07673073808352153}, "udem1-4-0": {"driven_any": 0.34749292376309954, "sim_physics": 0.2287416003999256, "survival_time": 1.0500000000000005, "driven_lanedir": 0.3325249708009643, "sim_render-ego": 0.06448833147684734, "in-drivable-lane": 0, "agent_compute-ego": 0.14127262433369955, "deviation-heading": 0.30044835691402494, "set_robot_commands": 0.09856789452689034, "deviation-center-line": 0.06216029860587694, "driven_lanedir_consec": 0.3325249708009643, "sim_compute_sim_state": 0.03860307875133696, "sim_compute_performance-ego": 0.07259783290681385, "sim_compute_robot_state-ego": 0.07726267405918666, "sim_compute_robot_state-npc0": 0.06956986018589564, "sim_compute_robot_state-npc1": 0.06954545066470191, "sim_compute_robot_state-npc2": 0.06976372855050224, "sim_compute_robot_state-npc3": 0.07365500359308153}, "udem1-5-0": {"driven_any": 3.5343244022418343, "sim_physics": 0.18436753535698988, "survival_time": 8.349999999999984, "driven_lanedir": 2.494308053441959, "sim_render-ego": 0.05926522523343206, "in-drivable-lane": 2, "agent_compute-ego": 0.13025375326236566, "deviation-heading": 1.9929437250081896, "set_robot_commands": 0.10806677726928346, "deviation-center-line": 0.5141120095458062, "driven_lanedir_consec": 2.494308053441959, "sim_compute_sim_state": 0.03711533118150905, "sim_compute_performance-ego": 0.06542824271196376, "sim_compute_robot_state-ego": 0.06983515316854694, "sim_compute_robot_state-npc0": 0.07241665246243963, "sim_compute_robot_state-npc1": 0.06629533967572059, "sim_compute_robot_state-npc2": 0.06493125561468616, "sim_compute_robot_state-npc3": 0.06582438017793758}, "udem1-6-0": {"driven_any": 2.176040550989443, "sim_physics": 0.23764921855000615, "survival_time": 5.14999999999999, "driven_lanedir": 1.9748003483573369, "sim_render-ego": 0.06820001648467722, "in-drivable-lane": 0.29999999999999893, "agent_compute-ego": 0.14662952793454662, "deviation-heading": 0.7877172510198454, "set_robot_commands": 0.10414145293745022, "deviation-center-line": 0.30661477465861414, "driven_lanedir_consec": 1.9748003483573369, "sim_compute_sim_state": 0.045311680117857106, "sim_compute_performance-ego": 0.07453934891710004, "sim_compute_robot_state-ego": 0.08282894532657364, "sim_compute_robot_state-npc0": 0.07616148874597642, "sim_compute_robot_state-npc1": 0.07561011916225396, "sim_compute_robot_state-npc2": 0.0754710285408983, "sim_compute_robot_state-npc3": 0.07262817169856099}, "udem1-7-0": {"driven_any": 1.0050236108346422, "sim_physics": 0.179863600730896, "survival_time": 2.499999999999999, "driven_lanedir": 0.8701796426684156, "sim_render-ego": 0.05415925979614258, "in-drivable-lane": 0, "agent_compute-ego": 0.1151271915435791, "deviation-heading": 0.9227446658975216, "set_robot_commands": 0.07557839870452882, "deviation-center-line": 0.14884708769804722, "driven_lanedir_consec": 0.8546116490381102, "sim_compute_sim_state": 0.03163711547851562, "sim_compute_performance-ego": 0.05862809658050537, "sim_compute_robot_state-ego": 0.06090038776397705, "sim_compute_robot_state-npc0": 0.058478655815124514, "sim_compute_robot_state-npc1": 0.056223664283752445, "sim_compute_robot_state-npc2": 0.05600606918334961, "sim_compute_robot_state-npc3": 0.05435097694396973}, "udem1-8-0": {"driven_any": 0.9970416603019576, "sim_physics": 0.2190061578383813, "survival_time": 2.5999999999999988, "driven_lanedir": 0.8041152574540316, "sim_render-ego": 0.06777447920579177, "in-drivable-lane": 0.29999999999999893, "agent_compute-ego": 0.1440584109379695, "deviation-heading": 0.6817155913229235, "set_robot_commands": 0.09895107837823722, "deviation-center-line": 0.1898768080806288, "driven_lanedir_consec": 0.8041152574540316, "sim_compute_sim_state": 0.04265966782203087, "sim_compute_performance-ego": 0.06933384675246018, "sim_compute_robot_state-ego": 0.0777853222993704, "sim_compute_robot_state-npc0": 0.07461893558502197, "sim_compute_robot_state-npc1": 0.07395915343211247, "sim_compute_robot_state-npc2": 0.07393068533677322, "sim_compute_robot_state-npc3": 0.07510991279895489}, "udem1-9-0": {"driven_any": 0.4374998710472299, "sim_physics": 0.2494822465456449, "survival_time": 1.3000000000000005, "driven_lanedir": 0.4177527119659043, "sim_render-ego": 0.06347837814917931, "in-drivable-lane": 0, "agent_compute-ego": 0.13733465854938215, "deviation-heading": 0.3796137446285097, "set_robot_commands": 0.09392273426055908, "deviation-center-line": 0.08207813392324871, "driven_lanedir_consec": 0.4177527119659043, "sim_compute_sim_state": 0.0423206457724938, "sim_compute_performance-ego": 0.07274389266967773, "sim_compute_robot_state-ego": 0.07532570912287785, "sim_compute_robot_state-npc0": 0.07314827808967003, "sim_compute_robot_state-npc1": 0.0700080944941594, "sim_compute_robot_state-npc2": 0.07061399863316463, "sim_compute_robot_state-npc3": 0.07735259716327374}, "udem1-10-0": {"driven_any": 0.3412551960104244, "sim_physics": 0.2679245580326427, "survival_time": 1.1000000000000003, "driven_lanedir": 0.3199180625488627, "sim_render-ego": 0.06800353527069092, "in-drivable-lane": 0, "agent_compute-ego": 0.15876758098602295, "deviation-heading": 0.3796986873122644, "set_robot_commands": 0.10603215477683328, "deviation-center-line": 0.06276575300040536, "driven_lanedir_consec": 0.3199180625488627, "sim_compute_sim_state": 0.04313578388907693, "sim_compute_performance-ego": 0.07728669860146263, "sim_compute_robot_state-ego": 0.0904348763552579, "sim_compute_robot_state-npc0": 0.08036056431857022, "sim_compute_robot_state-npc1": 0.07386553287506104, "sim_compute_robot_state-npc2": 0.07739199291576039, "sim_compute_robot_state-npc3": 0.07732458548112349}, "udem1-11-0": {"driven_any": 0.9951985237696338, "sim_physics": 0.1980842113494873, "survival_time": 2.499999999999999, "driven_lanedir": 0.7206363324756615, "sim_render-ego": 0.06905413150787354, "in-drivable-lane": 0.549999999999998, "agent_compute-ego": 0.15138526916503903, "deviation-heading": 0.30074636349085415, "set_robot_commands": 0.1003742742538452, "deviation-center-line": 0.16567670185555422, "driven_lanedir_consec": 0.7185075497455284, "sim_compute_sim_state": 0.04120028018951416, "sim_compute_performance-ego": 0.07182169437408448, "sim_compute_robot_state-ego": 0.07537510395050048, "sim_compute_robot_state-npc0": 0.0726310920715332, "sim_compute_robot_state-npc1": 0.0705941915512085, "sim_compute_robot_state-npc2": 0.07229021072387695, "sim_compute_robot_state-npc3": 0.07254035949707031}, "udem1-12-0": {"driven_any": 1.964687870253437, "sim_physics": 0.2251405436941918, "survival_time": 4.699999999999991, "driven_lanedir": 0.35162821871931205, "sim_render-ego": 0.06546889213805503, "in-drivable-lane": 3.549999999999991, "agent_compute-ego": 0.1431552197070832, "deviation-heading": 0.24234723632033853, "set_robot_commands": 0.1028529684594337, "deviation-center-line": 0.13126670634798973, "driven_lanedir_consec": 0.35162821871931205, "sim_compute_sim_state": 0.04235585699690149, "sim_compute_performance-ego": 0.07649892949043437, "sim_compute_robot_state-ego": 0.08129870891571045, "sim_compute_robot_state-npc0": 0.07577622444071669, "sim_compute_robot_state-npc1": 0.07417083547470417, "sim_compute_robot_state-npc2": 0.07365939718611697, "sim_compute_robot_state-npc3": 0.07655607385838285}, "udem1-13-0": {"driven_any": 1.001633421251194, "sim_physics": 0.23385420670876136, "survival_time": 2.5999999999999988, "driven_lanedir": 0.6537162971975778, "sim_render-ego": 0.06769793308698215, "in-drivable-lane": 0.5999999999999979, "agent_compute-ego": 0.1496557639195369, "deviation-heading": 0.4928440244444201, "set_robot_commands": 0.10504039434286264, "deviation-center-line": 0.13361893417935136, "driven_lanedir_consec": 0.6537162971975778, "sim_compute_sim_state": 0.04383929417683528, "sim_compute_performance-ego": 0.07194500244580783, "sim_compute_robot_state-ego": 0.08143355296208309, "sim_compute_robot_state-npc0": 0.07207691210966843, "sim_compute_robot_state-npc1": 0.07369835560138409, "sim_compute_robot_state-npc2": 0.0762828909433805, "sim_compute_robot_state-npc3": 0.07908960030629085}, "udem1-14-0": {"driven_any": 2.218769782727203, "sim_physics": 0.2164903618040539, "survival_time": 5.249999999999989, "driven_lanedir": 0.8125963661433656, "sim_render-ego": 0.06617939585731143, "in-drivable-lane": 2.499999999999993, "agent_compute-ego": 0.14826308886210124, "deviation-heading": 1.7163026916317128, "set_robot_commands": 0.099482942762829, "deviation-center-line": 0.24126933160503777, "driven_lanedir_consec": 0.6077880672015845, "sim_compute_sim_state": 0.04352556637355259, "sim_compute_performance-ego": 0.07354088964916411, "sim_compute_robot_state-ego": 0.0787614073072161, "sim_compute_robot_state-npc0": 0.07414175215221587, "sim_compute_robot_state-npc1": 0.07405749502636137, "sim_compute_robot_state-npc2": 0.07363930883861723, "sim_compute_robot_state-npc3": 0.07504286538986933}}
set_robot_commands_max0.10806677726928346
set_robot_commands_mean0.09981314143622086
set_robot_commands_median0.10073697566986084
set_robot_commands_min0.07557839870452882
sim_compute_performance-ego_max0.07799484829107921
sim_compute_performance-ego_mean0.07220034109647736
sim_compute_performance-ego_median0.07274389266967773
sim_compute_performance-ego_min0.05862809658050537
sim_compute_robot_state-ego_max0.0904348763552579
sim_compute_robot_state-ego_mean0.07822342111057559
sim_compute_robot_state-ego_median0.07805474522044357
sim_compute_robot_state-ego_min0.06090038776397705
sim_compute_robot_state-npc0_max0.08036056431857022
sim_compute_robot_state-npc0_mean0.07316280459082525
sim_compute_robot_state-npc0_median0.0733947583607265
sim_compute_robot_state-npc0_min0.058478655815124514
sim_compute_robot_state-npc1_max0.0799860308567683
sim_compute_robot_state-npc1_mean0.07209543713300702
sim_compute_robot_state-npc1_median0.07386553287506104
sim_compute_robot_state-npc1_min0.056223664283752445
sim_compute_robot_state-npc2_max0.07739199291576039
sim_compute_robot_state-npc2_mean0.07215728963684032
sim_compute_robot_state-npc2_median0.07365939718611697
sim_compute_robot_state-npc2_min0.05600606918334961
sim_compute_robot_state-npc3_max0.07908960030629085
sim_compute_robot_state-npc3_mean0.07322092344651526
sim_compute_robot_state-npc3_median0.07504286538986933
sim_compute_robot_state-npc3_min0.05435097694396973
sim_compute_sim_state_max0.045311680117857106
sim_compute_sim_state_mean0.04133938469468548
sim_compute_sim_state_median0.0423206457724938
sim_compute_sim_state_min0.03163711547851562
sim_physics_max0.2679245580326427
sim_physics_mean0.2202051692690641
sim_physics_median0.22035236018044607
sim_physics_min0.179863600730896
sim_render-ego_max0.06974882738930839
sim_render-ego_mean0.06562805285846383
sim_render-ego_median0.06617939585731143
sim_render-ego_min0.05415925979614258
simulation-passed1
survival_time_max8.349999999999984
survival_time_mean3.3333333333333286
survival_time_min1.0500000000000005
No reset possible
21775step1-simulationsuccessno0:23:28
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
21767step1-simulationhost-errorno0:25:42
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 347, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in convert
    with open(fn, 'w') as f:
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2399/step1-simulation-ip-172-31-40-253-31940-job21767/logs/challenges-runner/stdout.html'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
21334step1-simulationsuccessno0:29:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
21326step1-simulationhost-errorno0:20:46
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 483, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__
    next(self.gen)
  File "/project/src/duckietown_challenges_runner/runner.py", line 343, in setup_logging
    convert(f_stdout)
  File "/project/src/duckietown_challenges_runner/runner.py", line 332, in convert
    data = open(log).read().strip()
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2399/step1-simulation-ip-172-31-38-104-5317-job21326/logs/challenges-runner/stdout.log'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20602step1-simulationsuccessno0:07:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20598step1-simulationsuccessno0:07:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20018step1-simulationsuccessno0:06:11
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19646step1-simulationsuccessno0:08:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18897step1-simulationsuccessno0:11:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18548step1-simulationsuccessno0:11:44
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18531step1-simulationhost-errorno0:14:25
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 481, in get_cr
    uploaded = upload_files(wd, aws_config)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1016, in upload_files
    uploaded = upload(aws_config, toupload)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1171, in upload
    sha256hex = compute_sha256hex(realfile)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1217, in compute_sha256hex
    res: bytes = subprocess.check_output(cmd)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['shasum', '-a', '256', '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2399/step1-simulation-ip-172-31-25-98-9205-job18531/tmp/tmpy52_xpmm/episodes/udem1-1-0/camera.mp4']' returned non-zero exit status 1.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18530step1-simulationhost-errorno0:14:49
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 488, in get_cr
    shutil.rmtree(wd)
  File "/usr/lib/python3.6/shutil.py", line 471, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 469, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/duckietown/DT18/evaluator/executions/aido2-LFVI-sim-testing/submission2399/step1-simulation-ip-172-31-25-98-9766-job18530'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17862step1-simulationsuccessno0:06:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17572step1-simulationsuccessno0:05:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17570step1-simulationsuccessno0:05:31
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible