Submission 9296

Submission

9296

Competing

yes

Challenge

aido5-LF-sim-validation

User

Yishu Malhotra 🇨🇦

Date submitted

2020-10-25 13:17:21+00:00

Last status update

2021-12-04 20:52:33+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 58367

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

58367

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

58367

LFv-sim

success

yes

2020-12-03 22:55:52+00:00

2020-12-03 23:40:17+00:00

0:44:25

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.0
survival_time_median	59.99999999999873
deviation-center-line_median	1.2422730096440104
in-drivable-lane_median	0.0

other stats

agent_compute-ego0_max	0.01392882590885464
agent_compute-ego0_mean	0.012127407732653082
agent_compute-ego0_median	0.01165105301970546
agent_compute-ego0_min	0.011278698982346763
complete-iteration_max	0.39523695370835327
complete-iteration_mean	0.3092348971136603
complete-iteration_median	0.29044198940238986
complete-iteration_min	0.2608186559415082
deviation-center-line_max	4.053503393024394
deviation-center-line_mean	1.731069028434027
deviation-center-line_min	0.386226701423694
deviation-heading_max	27.859809596736422
deviation-heading_mean	14.925250437835436
deviation-heading_median	14.309269950178932
deviation-heading_min	3.22265225424745
driven_any_max	2.6645352591003757e-13
driven_any_mean	1.9984014443252818e-13
driven_any_median	2.6645352591003757e-13
driven_any_min	0.0
driven_lanedir_consec_max	0.000286102294921875
driven_lanedir_consec_mean	7.152557373046875e-05
driven_lanedir_consec_min	0.0
driven_lanedir_max	0.000286102294921875
driven_lanedir_mean	7.152557373046875e-05
driven_lanedir_median	0.0
driven_lanedir_min	0.0
get_duckie_state_max	1.391800714472152e-06
get_duckie_state_mean	1.340930805317468e-06
get_duckie_state_median	1.3360175165308207e-06
get_duckie_state_min	1.2998874737360793e-06
get_robot_state_max	0.003965658311740643
get_robot_state_mean	0.0037751228088741
get_robot_state_median	0.0037544226269241574
get_robot_state_min	0.0036259876699074422
get_state_dump_max	0.004926426623087937
get_state_dump_mean	0.00480497171241576
get_state_dump_median	0.00480489220646994
get_state_dump_min	0.004683675813635223
get_ui_image_max	0.040934931130135285
get_ui_image_mean	0.03181946754058533
get_ui_image_median	0.029725850372886185
get_ui_image_min	0.02689123828643367
in-drivable-lane_max	0.0
in-drivable-lane_mean	0.0
in-drivable-lane_min	0.0
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.02689123828643367, "step_physics": 0.1982336147540217, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004683675813635223, "get_robot_state": 0.0036325476548753114, "sim_render-ego0": 0.003727990125835587, "get_duckie_state": 1.303857808109128e-06, "in-drivable-lane": 0.0, "deviation-heading": 22.66279310353771, "agent_compute-ego0": 0.011278698982346763, "complete-iteration": 0.2608186559415082, "set_robot_commands": 0.0021887643450404287, "deviation-center-line": 4.053503393024394, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008128760160752677, "sim_compute_performance-ego0": 0.001970375904334177}, "LF-norm-zigzag-000-ego0": {"driven_any": 0.0, "get_ui_image": 0.040934931130135285, "step_physics": 0.3102697901285062, "survival_time": 59.99999999999873, "driven_lanedir": 0.000286102294921875, "get_state_dump": 0.004926426623087937, "get_robot_state": 0.003965658311740643, "sim_render-ego0": 0.004090078268122613, "get_duckie_state": 1.3681772249525136e-06, "in-drivable-lane": 0.0, "deviation-heading": 27.859809596736422, "agent_compute-ego0": 0.01392882590885464, "complete-iteration": 0.39523695370835327, "set_robot_commands": 0.0025092679991710197, "deviation-center-line": 1.0457540566888746, "driven_lanedir_consec": 0.000286102294921875, "sim_compute_sim_state": 0.012299736770960214, "sim_compute_performance-ego0": 0.002224748875080398}, "LF-norm-techtrack-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.031589467162196586, "step_physics": 0.2355539943653777, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004858707607437629, "get_robot_state": 0.0036259876699074422, "sim_render-ego0": 0.00377329759653363, "get_duckie_state": 1.2998874737360793e-06, "in-drivable-lane": 0.0, "deviation-heading": 3.22265225424745, "agent_compute-ego0": 0.011280330591257368, "complete-iteration": 0.3038682816526872, "set_robot_commands": 0.002291569602578804, "deviation-center-line": 1.4387919625991463, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008832492598089747, "sim_compute_performance-ego0": 0.0019784493807650525}, "LF-norm-small_loop-000-ego0": {"driven_any": 2.6645352591003757e-13, "get_ui_image": 0.02786223358357578, "step_physics": 0.2134539580762039, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.004751076805502251, "get_robot_state": 0.003876297598973003, "sim_render-ego0": 0.0039575622044038415, "get_duckie_state": 1.391800714472152e-06, "in-drivable-lane": 0.0, "deviation-heading": 5.955746796820156, "agent_compute-ego0": 0.012021775448153556, "complete-iteration": 0.2770156971520925, "set_robot_commands": 0.002423300731192024, "deviation-center-line": 0.386226701423694, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.006446417324945988, "sim_compute_performance-ego0": 0.0021389533240630367}}
set_robot_commands_max	0.0025092679991710197
set_robot_commands_mean	0.002353225669495569
set_robot_commands_median	0.002357435166885414
set_robot_commands_min	0.0021887643450404287
sim_compute_performance-ego0_max	0.002224748875080398
sim_compute_performance-ego0_mean	0.002078131871060666
sim_compute_performance-ego0_median	0.002058701352414045
sim_compute_performance-ego0_min	0.001970375904334177
sim_compute_sim_state_max	0.012299736770960214
sim_compute_sim_state_mean	0.008926851713687156
sim_compute_sim_state_median	0.008480626379421211
sim_compute_sim_state_min	0.006446417324945988
sim_render-ego0_max	0.004090078268122613
sim_render-ego0_mean	0.003887232048723918
sim_render-ego0_median	0.003865429900468736
sim_render-ego0_min	0.003727990125835587
simulation-passed	1
step_physics_max	0.3102697901285062
step_physics_mean	0.23937783933102735
step_physics_median	0.2245039762207908
step_physics_min	0.1982336147540217
survival_time_max	59.99999999999873
survival_time_mean	59.99999999999873
survival_time_min	59.99999999999873

No reset possible

58361

LFv-sim

success

yes

2020-12-03 22:53:44+00:00

2020-12-03 23:36:28+00:00

0:42:44

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52391

LFv-sim

error

2020-12-02 11:32:49+00:00

2020-12-02 11:43:43+00:00

0:10:54

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139682944054032
- M:video_aido:cmdline(in:/;out:/) 139682943658352
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52378

LFv-sim

error

2020-12-02 11:31:41+00:00

2020-12-02 11:42:11+00:00

0:10:30

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139817091112912
- M:video_aido:cmdline(in:/;out:/) 139817091108976
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

52368

LFv-sim

error

2020-12-02 11:31:16+00:00

2020-12-02 11:41:54+00:00

0:10:38

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140130745433440
- M:video_aido:cmdline(in:/;out:/) 140130745431472
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41751

LFv-sim

success

2020-11-08 13:46:30+00:00

2020-11-08 13:55:36+00:00

0:09:06

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41750

LFv-sim

success

2020-11-08 13:45:50+00:00

2020-11-08 13:55:00+00:00

0:09:10

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

41749

LFv-sim

success

2020-11-08 13:45:21+00:00

2020-11-08 13:54:28+00:00

0:09:07

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38228

LFv-sim

success

2020-11-04 00:50:03+00:00

2020-11-04 00:58:37+00:00

0:08:34

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

36352

LFv-sim

success

2020-11-02 04:46:02+00:00

2020-11-02 04:55:55+00:00

0:09:53

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35777

LFv-sim

error

2020-11-01 04:12:03+00:00

2020-11-01 04:12:51+00:00

0:00:48

The container "solut [...]

The container "solution" exited with code 1.


Look at the logs for the container to know more about the error.

No results found! Something very wrong. 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1063, in run_one
    cr = read_challenge_results(wd)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/challenge_results.py", line 90, in read_challenge_results
    raise NoResultsFound(msg)
duckietown_challenges.challenge_results.NoResultsFound: File '/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9296/LFv-sim-noname-sandy-1-job35777-a-wd/challenge-results/challenge_results.yaml' does not exist.

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35381

LFv-sim

error

2020-10-31 03:33:45+00:00

2020-10-31 03:55:48+00:00

0:22:03

The result file is n [...]

The result file is not found in working dir /tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9296/LFv-sim-reg05-b2dee9d94ee0-1-job35381:

File '/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9296/LFv-sim-reg05-b2dee9d94ee0-1-job35381/challenge-results/challenge_results.yaml' does not exist.

This usually means that the evaluator did not finish and some times that there was an import error.
Check the evaluator log to see what happened.

List of all files:

 -/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9296/LFv-sim-reg05-b2dee9d94ee0-1-job35381/docker-compose.original.yaml
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9296/LFv-sim-reg05-b2dee9d94ee0-1-job35381/docker-compose.yaml
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9296/LFv-sim-reg05-b2dee9d94ee0-1-job35381/logs/challenges-runner/stdout.log
-/tmp/duckietown/DT18/evaluator/executions/aido5-LF-sim-validation/submission9296/LFv-sim-reg05-b2dee9d94ee0-1-job35381/logs/challenges-runner/stderr.log

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

35014

LFv-sim

success

2020-10-29 04:23:04+00:00

2020-10-29 04:45:33+00:00

0:22:29

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

34631

LFv-sim

success

2020-10-25 13:18:30+00:00

2020-10-25 13:43:18+00:00

0:24:48

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

34630

LFv-sim

success

2020-10-25 13:17:58+00:00

2020-10-25 13:41:37+00:00

0:23:39

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible