Submission 11067

Submission

11067

Competing

yes

Challenge

aido5-LF-sim-validation

User

Ayman Shams 🇨🇦

Date submitted

2020-11-04 23:54:13+00:00

Last status update

2021-12-04 20:52:39+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 56874

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

56874

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

56874

LFv-sim

success

yes

2020-12-03 07:43:48+00:00

2020-12-03 07:51:57+00:00

0:08:09

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.33125336066698896
survival_time_median	8.574999999999994
deviation-center-line_median	0.13020002713835033
in-drivable-lane_median	6.499999999999994

other stats

agent_compute-ego0_max	0.013785411076373363
agent_compute-ego0_mean	0.012903093547535051
agent_compute-ego0_median	0.012696175246403136
agent_compute-ego0_min	0.012434612620960584
complete-iteration_max	0.21607649469950113
complete-iteration_mean	0.17694737276438388
complete-iteration_median	0.17316009908634294
complete-iteration_min	0.1453927981853485
deviation-center-line_max	0.3582480361712454
deviation-center-line_mean	0.17821800375088356
deviation-center-line_min	0.0942239245555882
deviation-heading_max	1.8360971496363152
deviation-heading_mean	0.9714685263236874
deviation-heading_median	0.73896553523825
deviation-heading_min	0.5718458851819346
driven_any_max	3.076857727560073
driven_any_mean	1.6441554910044405
driven_any_median	1.439611654932459
driven_any_min	0.6205409265927703
driven_lanedir_consec_max	0.638406755711705
driven_lanedir_consec_mean	0.3737138870438259
driven_lanedir_consec_min	0.1939420711296207
driven_lanedir_max	0.638406755711705
driven_lanedir_mean	0.3737138870438259
driven_lanedir_median	0.33125336066698896
driven_lanedir_min	0.1939420711296207
get_duckie_state_max	1.53679445565465e-06
get_duckie_state_mean	1.4255848709338216e-06
get_duckie_state_median	1.4480858138113312e-06
get_duckie_state_min	1.269373400457974e-06
get_robot_state_max	0.003915338631135872
get_robot_state_mean	0.003679878698073262
get_robot_state_median	0.0036425637351806383
get_robot_state_min	0.0035190486907958982
get_state_dump_max	0.005013184375073536
get_state_dump_mean	0.004801337480307919
get_state_dump_median	0.004753782358040007
get_state_dump_min	0.004684600830078125
get_ui_image_max	0.03471582768911339
get_ui_image_mean	0.02963666031430068
get_ui_image_median	0.02939765473269121
get_ui_image_min	0.02503550410270691
in-drivable-lane_max	12.000000000000052
in-drivable-lane_mean	6.887500000000009
in-drivable-lane_min	2.5499999999999927
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 3.076857727560073, "get_ui_image": 0.027791592569062204, "step_physics": 0.09570239457217128, "survival_time": 16.4500000000001, "driven_lanedir": 0.638406755711705, "get_state_dump": 0.004796380707711884, "get_robot_state": 0.003635910785559452, "sim_render-ego0": 0.0038190357612841058, "get_duckie_state": 1.440626202207623e-06, "in-drivable-lane": 12.000000000000052, "deviation-heading": 1.8360971496363152, "agent_compute-ego0": 0.012434612620960584, "complete-iteration": 0.16205417026172986, "set_robot_commands": 0.0021420442696773643, "deviation-center-line": 0.3582480361712454, "driven_lanedir_consec": 0.638406755711705, "sim_compute_sim_state": 0.009673798445499303, "sim_compute_performance-ego0": 0.0019704370787649444}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.6205409265927703, "get_ui_image": 0.03471582768911339, "step_physics": 0.1399893013827772, "survival_time": 4.099999999999993, "driven_lanedir": 0.1939420711296207, "get_state_dump": 0.005013184375073536, "get_robot_state": 0.003915338631135872, "sim_render-ego0": 0.004134965230183429, "get_duckie_state": 1.53679445565465e-06, "in-drivable-lane": 2.5499999999999927, "deviation-heading": 0.8343713740745141, "agent_compute-ego0": 0.013785411076373363, "complete-iteration": 0.21607649469950113, "set_robot_commands": 0.0023011443126632505, "deviation-center-line": 0.0942239245555882, "driven_lanedir_consec": 0.1939420711296207, "sim_compute_sim_state": 0.009983384465596762, "sim_compute_performance-ego0": 0.002143316958324019}, "LF-norm-techtrack-000-ego0": {"driven_any": 1.1062167831137593, "get_ui_image": 0.031003716896320213, "step_physics": 0.11597394449957486, "survival_time": 7.199999999999982, "driven_lanedir": 0.33091066285740367, "get_state_dump": 0.00471118400836813, "get_robot_state": 0.0036492166848018258, "sim_render-ego0": 0.003860572288776266, "get_duckie_state": 1.269373400457974e-06, "in-drivable-lane": 5.099999999999982, "deviation-heading": 0.6435596964019858, "agent_compute-ego0": 0.012798718748421504, "complete-iteration": 0.18426602791095603, "set_robot_commands": 0.0021700250691380992, "deviation-center-line": 0.12133515795264206, "driven_lanedir_consec": 0.33091066285740367, "sim_compute_sim_state": 0.008021316857173525, "sim_compute_performance-ego0": 0.0019917800508696456}, "LF-norm-small_loop-000-ego0": {"driven_any": 1.773006526751158, "get_ui_image": 0.02503550410270691, "step_physics": 0.08650865077972412, "survival_time": 9.950000000000006, "driven_lanedir": 0.33159605847657425, "get_state_dump": 0.004684600830078125, "get_robot_state": 0.0035190486907958982, "sim_render-ego0": 0.003783861398696899, "get_duckie_state": 1.4555454254150392e-06, "in-drivable-lane": 7.900000000000006, "deviation-heading": 0.5718458851819346, "agent_compute-ego0": 0.012593631744384765, "complete-iteration": 0.1453927981853485, "set_robot_commands": 0.0021311962604522706, "deviation-center-line": 0.1390648963240586, "driven_lanedir_consec": 0.33159605847657425, "sim_compute_sim_state": 0.0051013314723968506, "sim_compute_performance-ego0": 0.0019422757625579832}}
set_robot_commands_max	0.0023011443126632505
set_robot_commands_mean	0.0021861024779827464
set_robot_commands_median	0.002156034669407732
set_robot_commands_min	0.0021311962604522706
sim_compute_performance-ego0_max	0.002143316958324019
sim_compute_performance-ego0_mean	0.002011952462629148
sim_compute_performance-ego0_median	0.001981108564817295
sim_compute_performance-ego0_min	0.0019422757625579832
sim_compute_sim_state_max	0.009983384465596762
sim_compute_sim_state_mean	0.00819495781016661
sim_compute_sim_state_median	0.008847557651336414
sim_compute_sim_state_min	0.0051013314723968506
sim_render-ego0_max	0.004134965230183429
sim_render-ego0_mean	0.003899608669735175
sim_render-ego0_median	0.003839804025030186
sim_render-ego0_min	0.003783861398696899
simulation-passed	1
step_physics_max	0.1399893013827772
step_physics_mean	0.10954357280856188
step_physics_median	0.10583816953587308
step_physics_min	0.08650865077972412
survival_time_max	16.4500000000001
survival_time_mean	9.42500000000002
survival_time_min	4.099999999999993

No reset possible

56871

LFv-sim

success

yes

2020-12-03 07:42:27+00:00

2020-12-03 07:52:17+00:00

0:09:50

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50787

LFv-sim

error

2020-12-02 08:19:19+00:00

2020-12-02 08:21:21+00:00

0:02:02

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140050566817920
- M:video_aido:cmdline(in:/;out:/) 140050566818400
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50769

LFv-sim

error

2020-12-02 08:17:32+00:00

2020-12-02 08:19:37+00:00

0:02:05

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140487351213936
- M:video_aido:cmdline(in:/;out:/) 140487351212688
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50767

LFv-sim

error

2020-12-02 08:17:18+00:00

2020-12-02 08:21:09+00:00

0:03:51

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140441287113696
- M:video_aido:cmdline(in:/;out:/) 140441287112160
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40325

LFv-sim

success

2020-11-07 10:52:56+00:00

2020-11-07 11:02:13+00:00

0:09:17

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40324

LFv-sim

success

2020-11-07 10:50:59+00:00

2020-11-07 11:00:15+00:00

0:09:16

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39702

LFv-sim

success

2020-11-06 16:47:36+00:00

2020-11-06 16:56:50+00:00

0:09:14

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39701

LFv-sim

success

2020-11-06 16:46:55+00:00

2020-11-06 16:58:54+00:00

0:11:59

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39689

LFv-sim

success

2020-11-06 16:34:07+00:00

2020-11-06 16:46:34+00:00

0:12:27

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39687

LFv-sim

success

2020-11-06 16:33:23+00:00

2020-11-06 16:45:59+00:00

0:12:36

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39685

LFv-sim

success

2020-11-06 16:32:48+00:00

2020-11-06 16:45:33+00:00

0:12:45

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38844

LFv-sim

success

2020-11-04 23:55:56+00:00

2020-11-05 00:05:41+00:00

0:09:45

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38841

LFv-sim

success

2020-11-04 23:55:25+00:00

2020-11-05 00:05:29+00:00

0:10:04

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible