Submission 11576

Submission

11576

Competing

yes

Challenge

aido5-LF-sim-validation

User

Anastasiya Nikolskaya 🇷🇺

Date submitted

2020-11-25 13:28:01+00:00

Last status update

2021-12-04 20:52:45+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 53884

User label

JetBrains Research

Admin priority

Blessing

n/a

User priority

53884

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

53884

LFv-sim

success

yes

2020-12-02 23:48:45+00:00

2020-12-03 00:42:24+00:00

0:53:39

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	3.1437909603001533
survival_time_median	59.99999999999873
deviation-center-line_median	1.2258068304677303
in-drivable-lane_median	46.67499999999863

other stats

agent_compute-ego0_max	0.1335890846982983
agent_compute-ego0_mean	0.1101761651376602
agent_compute-ego0_median	0.12048159332497728
agent_compute-ego0_min	0.0661523892023879
complete-iteration_max	0.503204750677231
complete-iteration_mean	0.4112902456378064
complete-iteration_median	0.3908893310656456
complete-iteration_min	0.36017756974270304
deviation-center-line_max	2.257853774645673
deviation-center-line_mean	1.316408252363411
deviation-center-line_min	0.5561655738725109
deviation-heading_max	3.841895465273165
deviation-heading_mean	2.790936109244121
deviation-heading_median	3.0851610850194926
deviation-heading_min	1.1515268016643334
driven_any_max	5.583649117161323
driven_any_mean	4.292981642358845
driven_any_median	4.777769225618016
driven_any_min	2.0327390010380255
driven_lanedir_consec_max	4.7142372388402745
driven_lanedir_consec_mean	2.927033267665281
driven_lanedir_consec_min	0.7063139112205437
driven_lanedir_max	4.7142372388402745
driven_lanedir_mean	2.927033267665281
driven_lanedir_median	3.1437909603001533
driven_lanedir_min	0.7063139112205437
get_duckie_state_max	1.4708103684958174e-06
get_duckie_state_mean	1.3756216019019797e-06
get_duckie_state_median	1.3776067690885036e-06
get_duckie_state_min	1.2764625009350931e-06
get_robot_state_max	0.004137746301122153
get_robot_state_mean	0.004030096788986041
get_robot_state_median	0.004077743828842582
get_robot_state_min	0.0038271531971368466
get_state_dump_max	0.005436766058281002
get_state_dump_mean	0.005096245169341812
get_state_dump_median	0.005081036108717334
get_state_dump_min	0.004786142401651578
get_ui_image_max	0.04155734853879498
get_ui_image_mean	0.03491960457222944
get_ui_image_median	0.03351670220730009
get_ui_image_min	0.031087665335522603
in-drivable-lane_max	55.74999999999871
in-drivable-lane_mean	48.52499999999864
in-drivable-lane_min	44.999999999998586
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 5.583649117161323, "get_ui_image": 0.033123437808415576, "step_physics": 0.22809633942666796, "survival_time": 59.99999999999873, "driven_lanedir": 4.7142372388402745, "get_state_dump": 0.005436766058281002, "get_robot_state": 0.004079593012076829, "sim_render-ego0": 0.004200797394650068, "get_duckie_state": 1.4708103684958174e-06, "in-drivable-lane": 46.299999999998654, "deviation-heading": 3.0542668868364298, "agent_compute-ego0": 0.1216405535816253, "complete-iteration": 0.4118257869193198, "set_robot_commands": 0.002538827535611803, "deviation-center-line": 1.130960102418779, "driven_lanedir_consec": 4.7142372388402745, "sim_compute_sim_state": 0.010352352874463644, "sim_compute_performance-ego0": 0.0022660977238917928}, "LF-norm-zigzag-000-ego0": {"driven_any": 2.0327390010380255, "get_ui_image": 0.04155734853879498, "step_physics": 0.2981166551750367, "survival_time": 59.99999999999873, "driven_lanedir": 0.7063139112205437, "get_state_dump": 0.005050886282813638, "get_robot_state": 0.004075894645608335, "sim_render-ego0": 0.004198704432885315, "get_duckie_state": 1.375919376979958e-06, "in-drivable-lane": 55.74999999999871, "deviation-heading": 1.1515268016643334, "agent_compute-ego0": 0.1335890846982983, "complete-iteration": 0.503204750677231, "set_robot_commands": 0.0024747713519373505, "deviation-center-line": 0.5561655738725109, "driven_lanedir_consec": 0.7063139112205437, "sim_compute_sim_state": 0.011728413396830563, "sim_compute_performance-ego0": 0.002310519810024646}, "LF-norm-techtrack-000-ego0": {"driven_any": 4.644606654931128, "get_ui_image": 0.0339099666061846, "step_physics": 0.22961988020300567, "survival_time": 59.99999999999873, "driven_lanedir": 3.284700845295121, "get_state_dump": 0.004786142401651578, "get_robot_state": 0.0038271531971368466, "sim_render-ego0": 0.003904669012852652, "get_duckie_state": 1.2764625009350931e-06, "in-drivable-lane": 47.04999999999862, "deviation-heading": 3.841895465273165, "agent_compute-ego0": 0.0661523892023879, "complete-iteration": 0.36017756974270304, "set_robot_commands": 0.002259417834825857, "deviation-center-line": 1.320653558516682, "driven_lanedir_consec": 3.284700845295121, "sim_compute_sim_state": 0.013557806300878723, "sim_compute_performance-ego0": 0.0020768543167177782}, "LF-norm-small_loop-000-ego0": {"driven_any": 4.910931796304905, "get_ui_image": 0.031087665335522603, "step_physics": 0.19480579520740873, "survival_time": 59.99999999999873, "driven_lanedir": 3.0028810753051856, "get_state_dump": 0.0051111859346210315, "get_robot_state": 0.004137746301122153, "sim_render-ego0": 0.004166317224304047, "get_duckie_state": 1.3792941611970493e-06, "in-drivable-lane": 44.999999999998586, "deviation-heading": 3.116055283202555, "agent_compute-ego0": 0.11932263306832928, "complete-iteration": 0.3699528752119714, "set_robot_commands": 0.0025566616820653807, "deviation-center-line": 2.257853774645673, "driven_lanedir_consec": 3.0028810753051856, "sim_compute_sim_state": 0.0064047472760838135, "sim_compute_performance-ego0": 0.00226973673386141}}
set_robot_commands_max	0.0025566616820653807
set_robot_commands_mean	0.002457419601110098
set_robot_commands_median	0.0025067994437745766
set_robot_commands_min	0.002259417834825857
sim_compute_performance-ego0_max	0.002310519810024646
sim_compute_performance-ego0_mean	0.002230802146123907
sim_compute_performance-ego0_median	0.002267917228876601
sim_compute_performance-ego0_min	0.0020768543167177782
sim_compute_sim_state_max	0.013557806300878723
sim_compute_sim_state_mean	0.010510829962064185
sim_compute_sim_state_median	0.011040383135647102
sim_compute_sim_state_min	0.0064047472760838135
sim_render-ego0_max	0.004200797394650068
sim_render-ego0_mean	0.004117622016173021
sim_render-ego0_median	0.004182510828594681
sim_render-ego0_min	0.003904669012852652
simulation-passed	1
step_physics_max	0.2981166551750367
step_physics_mean	0.23765966750302975
step_physics_median	0.2288581098148368
step_physics_min	0.19480579520740873
survival_time_max	59.99999999999873
survival_time_mean	59.99999999999873
survival_time_min	59.99999999999873

No reset possible

49486

LFv-sim

error

2020-12-02 05:41:55+00:00

2020-12-02 05:53:48+00:00

0:11:53

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140267944762768
- M:video_aido:cmdline(in:/;out:/) 140268265025936
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

49476

LFv-sim

error

2020-12-02 05:40:47+00:00

2020-12-02 05:45:36+00:00

0:04:49

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139938697588064
- M:video_aido:cmdline(in:/;out:/) 139938697585280
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

49464

LFv-sim

error

2020-12-02 05:39:47+00:00

2020-12-02 05:48:31+00:00

0:08:44

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 139636474685760
- M:video_aido:cmdline(in:/;out:/) 139636474557584
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42984

LFv-sim

success

2020-11-25 13:40:16+00:00

2020-11-25 13:54:05+00:00

0:13:49

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42983

LFv-sim

success

2020-11-25 13:39:53+00:00

2020-11-25 13:53:31+00:00

0:13:38

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42982

LFv-sim

success

2020-11-25 13:39:05+00:00

2020-11-25 13:49:35+00:00

0:10:30

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42981

LFv-sim

success

2020-11-25 13:36:03+00:00

2020-11-25 13:54:04+00:00

0:18:01

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42980

LFv-sim

success

2020-11-25 13:35:34+00:00

2020-11-25 13:53:32+00:00

0:17:58

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42979

LFv-sim

host-error

2020-11-25 13:34:29+00:00

2020-11-25 13:34:43+00:00

0:00:14

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 687, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 888, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1388, in write_logs
    services2id: Dict[str, str] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 937, in get_services_id
    container = client.containers.get(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/models/containers.py", line 880, in get
    resp = self.client.api.inspect_container(container_id)
  File "/usr/local/lib/python3.8/dist-packages/docker/utils/decorators.py", line 16, in wrapped
    raise errors.NullResource(
docker.errors.NullResource: Resource ID was not provided

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42978

LFv-sim

success

2020-11-25 13:33:33+00:00

2020-11-25 13:49:33+00:00

0:16:00

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42977

LFv-sim

success

2020-11-25 13:30:44+00:00

2020-11-25 13:54:04+00:00

0:23:20

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

42976

LFv-sim

success

2020-11-25 13:30:10+00:00

2020-11-25 13:54:06+00:00

0:23:56

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible