Submission 11037

Submission

11037

Competing

yes

Challenge

aido5-LF-sim-validation

User

Melisande Teng

Date submitted

2020-11-04 20:54:38+00:00

Last status update

2021-12-04 20:52:39+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 57032

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

57032

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

57032

LFv-sim

success

yes

2020-12-03 08:17:20+00:00

2020-12-03 08:52:10+00:00

0:34:50

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	6.279183743632279
survival_time_median	59.99999999999873
deviation-center-line_median	2.8420585260761992
in-drivable-lane_median	4.949999999999953

other stats

agent_compute-ego0_max	0.02817475011604811
agent_compute-ego0_mean	0.016148654497434058
agent_compute-ego0_median	0.012327780433737367
agent_compute-ego0_min	0.011764307006213387
complete-iteration_max	0.19974552066399592
complete-iteration_mean	0.17922958545541884
complete-iteration_median	0.1804613970797028
complete-iteration_min	0.15625002699827373
deviation-center-line_max	3.0473256631723644
deviation-center-line_mean	2.8012024754484712
deviation-center-line_min	2.4733671864691233
deviation-heading_max	17.927973437118307
deviation-heading_mean	13.698380560160262
deviation-heading_median	12.689074397887245
deviation-heading_min	11.487400007748244
driven_any_max	9.641960777387547
driven_any_mean	9.498308405331956
driven_any_median	9.559404714309242
driven_any_min	9.232463415321794
driven_lanedir_consec_max	7.609601397184802
driven_lanedir_consec_mean	6.326676620087721
driven_lanedir_consec_min	5.138737595901524
driven_lanedir_max	8.695244872355788
driven_lanedir_mean	8.059137352844587
driven_lanedir_median	8.4393790966729
driven_lanedir_min	6.66254634567676
get_duckie_state_max	1.288572020772891e-06
get_duckie_state_mean	1.1906536492975824e-06
get_duckie_state_median	1.169561247940762e-06
get_duckie_state_min	1.134920080535914e-06
get_robot_state_max	0.003715194730734845
get_robot_state_mean	0.003610034221217198
get_robot_state_median	0.003631938307807408
get_robot_state_min	0.0034610655385191296
get_state_dump_max	0.004705441782218431
get_state_dump_mean	0.004504454076339759
get_state_dump_median	0.004485918123656566
get_state_dump_min	0.004340538275827476
get_ui_image_max	0.034217769954722685
get_ui_image_mean	0.029644059648521733
get_ui_image_median	0.02920055389404297
get_ui_image_min	0.02595736085127831
in-drivable-lane_max	11.6999999999997
in-drivable-lane_mean	6.224999999999855
in-drivable-lane_min	3.2999999999998124
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 9.575301793676523, "get_ui_image": 0.02595736085127831, "step_physics": 0.0937461898686189, "survival_time": 59.99999999999873, "driven_lanedir": 8.695244872355788, "get_state_dump": 0.004340538275827476, "get_robot_state": 0.0034610655385191296, "sim_render-ego0": 0.00362023088358324, "get_duckie_state": 1.134920080535914e-06, "in-drivable-lane": 3.2999999999998124, "deviation-heading": 12.441951509618765, "agent_compute-ego0": 0.011764307006213387, "complete-iteration": 0.15625002699827373, "set_robot_commands": 0.00207009085211329, "deviation-center-line": 2.8113474096911477, "driven_lanedir_consec": 6.372940559164191, "sim_compute_sim_state": 0.009340455589643824, "sim_compute_performance-ego0": 0.0018724877074001037}, "LF-norm-zigzag-000-ego0": {"driven_any": 9.232463415321794, "get_ui_image": 0.034217769954722685, "step_physics": 0.12613784065850073, "survival_time": 59.99999999999873, "driven_lanedir": 6.66254634567676, "get_state_dump": 0.004421257357315457, "get_robot_state": 0.003583055650264794, "sim_render-ego0": 0.0037876259377358063, "get_duckie_state": 1.1678738558322166e-06, "in-drivable-lane": 11.6999999999997, "deviation-heading": 17.927973437118307, "agent_compute-ego0": 0.012160130285601334, "complete-iteration": 0.19974552066399592, "set_robot_commands": 0.002200094488240797, "deviation-center-line": 2.8727696424612508, "driven_lanedir_consec": 5.138737595901524, "sim_compute_sim_state": 0.011205303778160026, "sim_compute_performance-ego0": 0.001952576101272926}, "LF-norm-techtrack-000-ego0": {"driven_any": 9.641960777387547, "get_ui_image": 0.03178983762996778, "step_physics": 0.11058967734852204, "survival_time": 59.99999999999873, "driven_lanedir": 8.493976134856625, "get_state_dump": 0.004550578889997674, "get_robot_state": 0.0036808209653500217, "sim_render-ego0": 0.003881122150786413, "get_duckie_state": 1.171248640049308e-06, "in-drivable-lane": 4.949999999999914, "deviation-heading": 11.487400007748244, "agent_compute-ego0": 0.012495430581873402, "complete-iteration": 0.18389179049483148, "set_robot_commands": 0.0022188856837949984, "deviation-center-line": 2.4733671864691233, "driven_lanedir_consec": 7.609601397184802, "sim_compute_sim_state": 0.012551552449336754, "sim_compute_performance-ego0": 0.002050153818059027}, "LF-norm-small_loop-000-ego0": {"driven_any": 9.54350763494196, "get_ui_image": 0.026611270158118153, "step_physics": 0.09933516306245851, "survival_time": 59.99999999999873, "driven_lanedir": 8.384782058489174, "get_state_dump": 0.004705441782218431, "get_robot_state": 0.003715194730734845, "sim_render-ego0": 0.003895454065289525, "get_duckie_state": 1.288572020772891e-06, "in-drivable-lane": 4.949999999999993, "deviation-heading": 12.936197286155725, "agent_compute-ego0": 0.02817475011604811, "complete-iteration": 0.17703100366457417, "set_robot_commands": 0.002249970622701907, "deviation-center-line": 3.0473256631723644, "driven_lanedir_consec": 6.185426928100368, "sim_compute_sim_state": 0.006302411113551614, "sim_compute_performance-ego0": 0.001955053192888271}}
set_robot_commands_max	0.002249970622701907
set_robot_commands_mean	0.002184760411712748
set_robot_commands_median	0.0022094900860178974
set_robot_commands_min	0.00207009085211329
sim_compute_performance-ego0_max	0.002050153818059027
sim_compute_performance-ego0_mean	0.0019575677049050817
sim_compute_performance-ego0_median	0.001953814647080598
sim_compute_performance-ego0_min	0.0018724877074001037
sim_compute_sim_state_max	0.012551552449336754
sim_compute_sim_state_mean	0.009849930732673054
sim_compute_sim_state_median	0.010272879683901923
sim_compute_sim_state_min	0.006302411113551614
sim_render-ego0_max	0.003895454065289525
sim_render-ego0_mean	0.0037961082593487463
sim_render-ego0_median	0.0038343740442611097
sim_render-ego0_min	0.00362023088358324
simulation-passed	1
step_physics_max	0.12613784065850073
step_physics_mean	0.10745221773452504
step_physics_median	0.10496242020549028
step_physics_min	0.0937461898686189
survival_time_max	59.99999999999873
survival_time_mean	59.99999999999873
survival_time_min	59.99999999999873

No reset possible

57030

LFv-sim

success

yes

2020-12-03 08:16:43+00:00

2020-12-03 08:34:53+00:00

0:18:10

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

57020

LFv-sim

success

yes

2020-12-03 08:15:16+00:00

2020-12-03 08:43:42+00:00

0:28:26

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

57014

LFv-sim

success

yes

2020-12-03 08:14:49+00:00

2020-12-03 08:41:40+00:00

0:26:51

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50839

LFv-sim

error

2020-12-02 08:25:11+00:00

2020-12-02 08:32:00+00:00

0:06:49

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140663823195104
- M:video_aido:cmdline(in:/;out:/) 140663823208992
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50827

LFv-sim

error

2020-12-02 08:24:00+00:00

2020-12-02 08:35:47+00:00

0:11:47

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140131627433936
- M:video_aido:cmdline(in:/;out:/) 140131627461216
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40395

LFv-sim

success

2020-11-07 11:57:01+00:00

2020-11-07 12:09:31+00:00

0:12:30

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40393

LFv-sim

success

2020-11-07 11:55:35+00:00

2020-11-07 12:07:54+00:00

0:12:19

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40392

LFv-sim

success

2020-11-07 11:55:10+00:00

2020-11-07 12:07:44+00:00

0:12:34

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40391

LFv-sim

success

2020-11-07 11:54:30+00:00

2020-11-07 12:06:50+00:00

0:12:20

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39726

LFv-sim

success

2020-11-06 17:13:53+00:00

2020-11-06 17:23:01+00:00

0:09:08

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39725

LFv-sim

success

2020-11-06 17:13:15+00:00

2020-11-06 17:22:23+00:00

0:09:08

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38785

LFv-sim

success

2020-11-04 20:55:43+00:00

2020-11-04 21:07:04+00:00

0:11:21

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible