Submission 11080

Submission

11080

Competing

yes

Challenge

aido5-LF-sim-validation

User

Melisande Teng

Date submitted

2020-11-05 00:37:58+00:00

Last status update

2021-12-04 20:52:39+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

LFv-sim: 56825

User label

exercise_ros_template

Admin priority

Blessing

n/a

User priority

56825

Click the images to see detailed statistics about the episode.

LF-norm-loop-000

LF-norm-small_loop-000

LF-norm-techtrack-000

LF-norm-zigzag-000

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

56825

LFv-sim

success

yes

2020-12-03 07:35:36+00:00

2020-12-03 08:12:25+00:00

0:36:49

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	5.991005218153541
survival_time_median	59.99999999999873
deviation-center-line_median	3.0938258468329725
in-drivable-lane_median	9.9249999999998

other stats

agent_compute-ego0_max	0.03554844300415395
agent_compute-ego0_mean	0.02423357452381461
agent_compute-ego0_median	0.024801596515283898
agent_compute-ego0_min	0.01178266206053671
complete-iteration_max	0.23211371114510085
complete-iteration_mean	0.1859780056192714
complete-iteration_median	0.17882701836458154
complete-iteration_min	0.1541442746028217
deviation-center-line_max	3.4009170443706394
deviation-center-line_mean	3.0666856192420457
deviation-center-line_min	2.6781737389315983
deviation-heading_max	19.30997198270497
deviation-heading_mean	15.292406526220924
deviation-heading_median	15.136184969322578
deviation-heading_min	11.587284183533578
driven_any_max	9.6687639204586
driven_any_mean	9.334333358764985
driven_any_median	9.320929362481513
driven_any_min	9.026710789638312
driven_lanedir_consec_max	8.63935556617444
driven_lanedir_consec_mean	6.263253913323066
driven_lanedir_consec_min	4.431649650810741
driven_lanedir_max	8.63935556617444
driven_lanedir_mean	7.292986090627949
driven_lanedir_median	7.322108201888423
driven_lanedir_min	5.888372392560509
get_duckie_state_max	2.1668099840912195e-06
get_duckie_state_mean	1.9289373259659513e-06
get_duckie_state_median	1.9527097030245793e-06
get_duckie_state_min	1.643519913723427e-06
get_robot_state_max	0.003657733868004182
get_robot_state_mean	0.003497746564466491
get_robot_state_median	0.0035258825375178177
get_robot_state_min	0.0032814873148261457
get_state_dump_max	0.004644909865850215
get_state_dump_mean	0.004394388665366828
get_state_dump_median	0.004413376144326597
get_state_dump_min	0.004105892506963903
get_ui_image_max	0.03538676007006389
get_ui_image_mean	0.02899328318364813
get_ui_image_median	0.027363549660484956
get_ui_image_min	0.02585927334355871
in-drivable-lane_max	15.299999999999564
in-drivable-lane_mean	9.674999999999796
in-drivable-lane_min	3.5500000000000167
per-episodes	details {"LF-norm-loop-000-ego0": {"driven_any": 9.6687639204586, "get_ui_image": 0.02585927334355871, "step_physics": 0.08974301606590404, "survival_time": 59.99999999999873, "driven_lanedir": 8.63935556617444, "get_state_dump": 0.004260956893654092, "get_robot_state": 0.003432340566363561, "sim_render-ego0": 0.003653760158846916, "get_duckie_state": 1.8072962066116777e-06, "in-drivable-lane": 4.69999999999985, "deviation-heading": 11.587284183533578, "agent_compute-ego0": 0.014757132550064072, "complete-iteration": 0.1541442746028217, "set_robot_commands": 0.002057221410276491, "deviation-center-line": 2.6781737389315983, "driven_lanedir_consec": 8.63935556617444, "sim_compute_sim_state": 0.008434456254322265, "sim_compute_performance-ego0": 0.0018729933294825112}, "LF-norm-zigzag-000-ego0": {"driven_any": 9.156956227846743, "get_ui_image": 0.03538676007006389, "step_physics": 0.1335066176771026, "survival_time": 59.99999999999873, "driven_lanedir": 5.888372392560509, "get_state_dump": 0.004565795394999101, "get_robot_state": 0.003657733868004182, "sim_render-ego0": 0.003948925933075586, "get_duckie_state": 2.1668099840912195e-06, "in-drivable-lane": 15.299999999999564, "deviation-heading": 19.30997198270497, "agent_compute-ego0": 0.03554844300415395, "complete-iteration": 0.23211371114510085, "set_robot_commands": 0.0023095901562311966, "deviation-center-line": 2.9701959003866047, "driven_lanedir_consec": 4.431649650810741, "sim_compute_sim_state": 0.01107021017336627, "sim_compute_performance-ego0": 0.0020337162367211692}, "LF-norm-techtrack-000-ego0": {"driven_any": 9.026710789638312, "get_ui_image": 0.0286853772019665, "step_physics": 0.11547254086731872, "survival_time": 59.99999999999873, "driven_lanedir": 6.019738226244154, "get_state_dump": 0.004105892506963903, "get_robot_state": 0.0032814873148261457, "sim_render-ego0": 0.0035134181690454286, "get_duckie_state": 1.643519913723427e-06, "in-drivable-lane": 15.14999999999975, "deviation-heading": 18.25154393459369, "agent_compute-ego0": 0.01178266206053671, "complete-iteration": 0.1813210150681368, "set_robot_commands": 0.0019139113970144304, "deviation-center-line": 3.4009170443706394, "driven_lanedir_consec": 5.5564976827775645, "sim_compute_sim_state": 0.010771805797389504, "sim_compute_performance-ego0": 0.0017224592928286892}, "LF-norm-small_loop-000-ego0": {"driven_any": 9.484902497116275, "get_ui_image": 0.02604172211900341, "step_physics": 0.09309331801015074, "survival_time": 59.99999999999873, "driven_lanedir": 8.624478177532692, "get_state_dump": 0.004644909865850215, "get_robot_state": 0.003619424508672074, "sim_render-ego0": 0.0038240350950369726, "get_duckie_state": 2.0981231994374805e-06, "in-drivable-lane": 3.5500000000000167, "deviation-heading": 12.020826004051465, "agent_compute-ego0": 0.03484606048050371, "complete-iteration": 0.1763330216610263, "set_robot_commands": 0.0021894148843274526, "deviation-center-line": 3.2174557932793397, "driven_lanedir_consec": 6.425512753529518, "sim_compute_sim_state": 0.0060929666450875285, "sim_compute_performance-ego0": 0.001897464882424233}}
set_robot_commands_max	0.0023095901562311966
set_robot_commands_mean	0.0021175344619623927
set_robot_commands_median	0.0021233181473019717
set_robot_commands_min	0.0019139113970144304
sim_compute_performance-ego0_max	0.0020337162367211692
sim_compute_performance-ego0_mean	0.0018816584353641508
sim_compute_performance-ego0_median	0.001885229105953372
sim_compute_performance-ego0_min	0.0017224592928286892
sim_compute_sim_state_max	0.01107021017336627
sim_compute_sim_state_mean	0.009092359717541392
sim_compute_sim_state_median	0.009603131025855884
sim_compute_sim_state_min	0.0060929666450875285
sim_render-ego0_max	0.003948925933075586
sim_render-ego0_mean	0.003735034839001226
sim_render-ego0_median	0.003738897626941944
sim_render-ego0_min	0.0035134181690454286
simulation-passed	1
step_physics_max	0.1335066176771026
step_physics_mean	0.10795387315511902
step_physics_median	0.10428292943873473
step_physics_min	0.08974301606590404
survival_time_max	59.99999999999873
survival_time_mean	59.99999999999873
survival_time_min	59.99999999999873

No reset possible

56813

LFv-sim

host-error

yes

2020-12-03 07:33:49+00:00

2020-12-03 08:00:47+00:00

0:26:58

Uncaught exception: [...]

Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 59, in get_services_id
    raise ZValueError(container_ids=container_ids, services=services, res=res, names=names)
zuper_commons.types.exceptions.ZValueError: 

│ container_ids: [16cc04c9ec40b7232c3b6acc4be1c41996899eb63c6c7cdd4ea1ab2af3e67f29]
│      services: dict[3]
│                │ evaluator:
│                │ dict[7]
│                │ │ image: docker.io/andreacensi/aido5-lf-sim-validation-lfv-sim-evaluator@sha256:6d0af9441525e1ed05be582f41e00fc178083c86797d28cd1a255c7025d0fd50
│                │ │ environment:
│                │ │ dict[10]
│                │ │ │ experiment_manager_parameters:
│                │ │ │ |episodes_per_scenario: 1
│                │ │ │ |episode_length_s: 60.0
│                │ │ │ |min_episode_length_s: 0.0
│                │ │ │ |seed: 888
│                │ │ │ |physics_dt: 0.05
│                │ │ │ |max_failures: 2
│                │ │ │ |fifo_dir: /fifos
│                │ │ │ |sim_in: /fifos/simulator-in
│                │ │ │ |sim_out: /fifos/simulator-out
│                │ │ │ |sm_in: /fifos/scenario_maker-in
│                │ │ │ |sm_out: /fifos/scenario_maker-out
│                │ │ │ |timeout_initialization: 120
│                │ │ │ |timeout_regular: 120
│                │ │ │ |port: 10123
│                │ │ │ |scenarios:
│                │ │ │ |- /scenarios
│                │ │ │ |
│                │ │ │ challenge_name: aido5-LF-sim-validation
│                │ │ │ challenge_step_name: LFv-sim
│                │ │ │ submission_id: 11080
│                │ │ │ submitter_name: melisande
│                │ │ │ SUBMISSION_CONTAINER: docker.io/melisande/aido-submissions:2020_11_05_00_37_10@sha256:c0b95c731a100a8d48b3183eedd5f223893e8fca50df37bb2857ee25f657c67e
│                │ │ │ username: ubuntu
│                │ │ │ uid: 0
│                │ │ │ USER: ubuntu
│                │ │ │ HOME: /fake-home/ubuntu
│                │ │ ports: [10123]
│                │ │ labels: {org.duckietown.created_by_runner: true, org.duckietown.runner_name: nogpu-prod-02_67f7c0c46c90}
│                │ │ user: 0:0
│                │ │ volumes:
│                │ │ [
│                │ │  /tmp/duckietown/aido5-LF-sim-validation/submission11080/LFv-sim-nogpu-prod-02_67f7c0c46c90-job56813-a-wd:/challenges:rw,
│                │ │  /tmp/duckietown/aido5-LF-sim-validation/submission11080/LFv-sim-nogpu-prod-02_67f7c0c46c90-job56813-a-fifos:/fifos:rw,
│                │ │  /tmp/duckietown/dt-challenges-runner/20_12_03_07_33_49-70945/fake-ubuntu-home:/fake-home/ubuntu:rw]
│                │ │ networks: {evaluation: {aliases: [evaluation]} }
│                │ simulator:
│                │ dict[6]
│                │ │ image: docker.io/duckietown/challenge-aido_lf-simulator-gym@sha256:c0096866077db3574e425d40603d8f5fc8ebbd164da7c0578df94ff4ede58d95
│                │ │ environment:
│                │ │ dict[12]
│                │ │ │ AIDONODE_CONFIG:
│                │ │ │ |env_constructor: Simulator
│                │ │ │ |env_parameters:
│                │ │ │ |  max_steps: 500001 # we don't want the gym to reset itself
│                │ │ │ |  domain_rand: 0
│                │ │ │ |  camera_width: 640
│                │ │ │ |  camera_height: 480
│                │ │ │ |  distortion: true
│                │ │ │ |  num_tris_distractors: 0
│                │ │ │ |  color_ground: [0, 0.3, 0] # green
│                │ │ │ |  enable_leds: true
│                │ │ │ |
│                │ │ │ AIDONODE_DATA_IN: /fifos/simulator-in
│                │ │ │ AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
│                │ │ │ challenge_name: aido5-LF-sim-validation
│                │ │ │ challenge_step_name: LFv-sim
│                │ │ │ submission_id: 11080
│                │ │ │ submitter_name: melisande
│                │ │ │ SUBMISSION_CONTAINER: docker.io/melisande/aido-submissions:2020_11_05_00_37_10@sha256:c0b95c731a100a8d48b3183eedd5f223893e8fca50df37bb2857ee25f657c67e
│                │ │ │ username: ubuntu
│                │ │ │ uid: 0
│                │ │ │ USER: ubuntu
│                │ │ │ HOME: /fake-home/ubuntu
│                │ │ labels: {org.duckietown.created_by_runner: true, org.duckietown.runner_name: nogpu-prod-02_67f7c0c46c90}
│                │ │ user: 0:0
│                │ │ volumes:
│                │ │ [
│                │ │  /tmp/duckietown/aido5-LF-sim-validation/submission11080/LFv-sim-nogpu-prod-02_67f7c0c46c90-job56813-a-wd:/challenges:rw,
│                │ │  /tmp/duckietown/aido5-LF-sim-validation/submission11080/LFv-sim-nogpu-prod-02_67f7c0c46c90-job56813-a-fifos:/fifos:rw,
│                │ │  /tmp/duckietown/dt-challenges-runner/20_12_03_07_33_49-70945/fake-ubuntu-home:/fake-home/ubuntu:rw]
│                │ │ networks: {evaluation: {aliases: [evaluation]} }
│                │ solution-ego0:
│                │ dict[6]
│                │ │ image: docker.io/melisande/aido-submissions@sha256:c0b95c731a100a8d48b3183eedd5f223893e8fca50df37bb2857ee25f657c67e
│                │ │ environment:
│                │ │ dict[13]
│                │ │ │ AIDONODE_NAME: ego0
│                │ │ │ AIDONODE_DATA_IN: /fifos/ego0-in
│                │ │ │ AIDO_REQUIRE_GPU: 1
│                │ │ │ AIDONODE_DATA_OUT: fifo:/fifos/ego0-out
│                │ │ │ challenge_name: aido5-LF-sim-validation
│                │ │ │ challenge_step_name: LFv-sim
│                │ │ │ submission_id: 11080
│                │ │ │ submitter_name: melisande
│                │ │ │ SUBMISSION_CONTAINER: docker.io/melisande/aido-submissions:2020_11_05_00_37_10@sha256:c0b95c731a100a8d48b3183eedd5f223893e8fca50df37bb2857ee25f657c67e
│                │ │ │ username: ubuntu
│                │ │ │ uid: 0
│                │ │ │ USER: ubuntu
│                │ │ │ HOME: /fake-home/ubuntu
│                │ │ labels: {org.duckietown.created_by_runner: true, org.duckietown.runner_name: nogpu-prod-02_67f7c0c46c90}
│                │ │ user: 0:0
│                │ │ volumes:
│                │ │ [
│                │ │  /tmp/duckietown/aido5-LF-sim-validation/submission11080/LFv-sim-nogpu-prod-02_67f7c0c46c90-job56813-a-wd:/challenges:rw,
│                │ │  /tmp/duckietown/aido5-LF-sim-validation/submission11080/LFv-sim-nogpu-prod-02_67f7c0c46c90-job56813-a-fifos:/fifos:rw,
│                │ │  /tmp/duckietown/dt-challenges-runner/20_12_03_07_33_49-70945/fake-ubuntu-home:/fake-home/ubuntu:rw]
│                │ │ networks: {evaluation: {aliases: [evaluation]} }
│           res: {solution-ego0: 16cc04c9ec40b7232c3b6acc4be1c41996899eb63c6c7cdd4ea1ab2af3e67f29}
│         names: dict[1]
│                │ 16cc04c9ec40b7232c3b6acc4be1c41996899eb63c6c7cdd4ea1ab2af3e67f29: nogpu-prod-02_67f7c0c46c90-job56813-888896_solution-ego0_1

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 777, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 991, in run_single
    write_logs(wd, project, services=config["services"])
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 120, in write_logs
    services2id: Dict[ServiceName, ContainerID] = get_services_id(wd, project, services)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/docker_compose.py", line 63, in get_services_id
    raise DockerComposeFail(msg, output=output.decode(), names=names) from e
duckietown_challenges_runner.docker_compose.DockerComposeFail: Cannot get process ids
│ output: |16cc04c9ec40b7232c3b6acc4be1c41996899eb63c6c7cdd4ea1ab2af3e67f29
│         |
│  names: dict[1]
│         │ 16cc04c9ec40b7232c3b6acc4be1c41996899eb63c6c7cdd4ea1ab2af3e67f29: nogpu-prod-02_67f7c0c46c90-job56813-888896_solution-ego0_1

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56807

LFv-sim

success

yes

2020-12-03 07:33:08+00:00

2020-12-03 08:05:07+00:00

0:31:59

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

56801

LFv-sim

success

yes

2020-12-03 07:30:55+00:00

2020-12-03 07:59:35+00:00

0:28:40

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

50704

LFv-sim

error

2020-12-02 08:12:21+00:00

2020-12-02 08:16:53+00:00

0:04:32

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
    im = Image.open(filename)
  File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
    raise UnidentifiedImageError(
PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
    result = block.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
    image = imread(self.config.file)
  File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
    raise ValueError(msg) from e
ValueError: Could not open filename "banner1.png".

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 329, in main
    make_video2(
  File "/usr/local/lib/python3.8/site-packages/aido_analyze/utils_video.py", line 149, in make_video2
    pg("video_aido", params)
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 280, in pg
    raise e
  File "/usr/local/lib/python3.8/site-packages/procgraph/scripts/pgmain.py", line 277, in pg
    model.update()
  File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 321, in update
    raise BadMethodCall("update", block, traceback.format_exc())
procgraph.core.exceptions.BadMethodCall: User-thrown exception while calling update() in block 'static_image'.
- B:StaticImage:static_image(in:/;out:rgb) 140068763263184
- M:video_aido:cmdline(in:/;out:/) 140068763266784
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 51, in imread
>     im = Image.open(filename)
>   File "/usr/local/lib/python3.8/site-packages/PIL/Image.py", line 2943, in open
>     raise UnidentifiedImageError(
> PIL.UnidentifiedImageError: cannot identify image file 'banner1.png'
> 
> The above exception was the direct cause of the following exception:
> 
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.8/site-packages/procgraph/core/model.py", line 316, in update
>     result = block.update()
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 31, in update
>     image = imread(self.config.file)
>   File "/usr/local/lib/python3.8/site-packages/procgraph_pil/imread_imp.py", line 54, in imread
>     raise ValueError(msg) from e
> ValueError: Could not open filename "banner1.png".

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 68, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 34, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 383, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40287

LFv-sim

success

2020-11-07 10:22:24+00:00

2020-11-07 10:33:19+00:00

0:10:55

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40286

LFv-sim

success

2020-11-07 10:21:40+00:00

2020-11-07 10:30:56+00:00

0:09:16

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

40285

LFv-sim

success

2020-11-07 10:20:29+00:00

2020-11-07 10:32:27+00:00

0:11:58

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39670

LFv-sim

success

2020-11-06 16:14:25+00:00

2020-11-06 16:23:26+00:00

0:09:01

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

39669

LFv-sim

success

2020-11-06 16:13:46+00:00

2020-11-06 16:22:51+00:00

0:09:05

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38869

LFv-sim

success

2020-11-05 00:39:15+00:00

2020-11-05 00:49:34+00:00

0:10:19

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

38868

LFv-sim

success

2020-11-05 00:38:43+00:00

2020-11-05 00:47:24+00:00

0:08:41

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible