Duckietown Challenges Home Challenges Submissions

Submission 13637

Submission13637
Competingyes
Challengeaido-LFVI_multi-sim-validation
UserRaphael Jean
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobssim-0of4: 104856 sim-1of4: 104894 sim-2of4: 104861 sim-3of4: 104896
Next
User labelmobile-segmentation
Admin priority50
Blessingn/a
User priority50

104894

Click the images to see detailed statistics about the episode.

4way-000

104856

Click the images to see detailed statistics about the episode.

udem-000

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
104896sim-3of4successyes0:01:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
skipped1
No reset possible
104894sim-1of4successyes0:03:33
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median5.349999999999989
in-drivable-lane_median0.9749999999999964
driven_lanedir_consec_median1.0145378137304215
deviation-center-line_median0.33707202315893514


other stats
deviation-center-line_max0.6483181953289276
deviation-center-line_mean0.3576731979096809
deviation-center-line_min0.10823054999192566
deviation-heading_max1.1254387391116698
deviation-heading_mean0.9358589963304024
deviation-heading_median0.9678746393479044
deviation-heading_min0.6822479675141305
distance-from-start_max1.6057929908455528
distance-from-start_mean1.2510953007601937
distance-from-start_median1.2057147090459095
distance-from-start_min0.987158794103402
driven_any_max1.937917444328892
driven_any_mean1.469929267818153
driven_any_median1.4655340428403516
driven_any_min1.010731541263017
driven_lanedir_consec_max1.5872458144222448
driven_lanedir_consec_mean1.0803518362793876
driven_lanedir_consec_min0.7050859032344629
driven_lanedir_max1.888144666826664
driven_lanedir_mean1.2411381729126956
driven_lanedir_median1.1200744708364665
driven_lanedir_min0.8362590831511851
in-drivable-lane_max2.6999999999999904
in-drivable-lane_mean1.162499999999996
in-drivable-lane_min0.0
per-episodes
details{"4way-000-ego0": {"driven_any": 1.8850465898239896, "survival_time": 5.349999999999989, "driven_lanedir": 1.295709232484239, "in-drivable-lane": 1.8999999999999932, "deviation-heading": 0.8700776985592218, "distance-from-start": 1.6057929908455528, "deviation-center-line": 0.10823054999192566, "driven_lanedir_consec": 1.295709232484239}, "4way-000-ego1": {"driven_any": 1.0460214958567138, "survival_time": 5.349999999999989, "driven_lanedir": 0.944439709188694, "in-drivable-lane": 2.6999999999999904, "deviation-heading": 0.6822479675141305, "distance-from-start": 0.987158794103402, "deviation-center-line": 0.1911311112336549, "driven_lanedir_consec": 0.733366394976604}, "4way-000-ego2": {"driven_any": 1.937917444328892, "survival_time": 5.349999999999989, "driven_lanedir": 1.888144666826664, "in-drivable-lane": 0.0, "deviation-heading": 1.065671580136587, "distance-from-start": 1.4112263604471482, "deviation-center-line": 0.48301293508421533, "driven_lanedir_consec": 1.5872458144222448}, "4way-000-ego3": {"driven_any": 1.010731541263017, "survival_time": 5.349999999999989, "driven_lanedir": 0.8362590831511851, "in-drivable-lane": 0.04999999999999982, "deviation-heading": 1.1254387391116698, "distance-from-start": 1.0002030576446712, "deviation-center-line": 0.6483181953289276, "driven_lanedir_consec": 0.7050859032344629}}
simulation-passed1
survival_time_max5.349999999999989
survival_time_mean5.349999999999989
survival_time_min5.349999999999989
No reset possible
104893sim-3of4successyes0:01:08
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104890sim-3of4successyes0:01:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104888sim-1of4successyes0:03:14
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104885sim-3of4successyes0:01:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104882sim-3of4successyes0:01:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104880sim-1of4successyes0:03:15
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104878sim-3of4successyes0:03:00
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104875sim-1of4successyes0:03:15
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104872sim-3of4successyes0:01:09
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104871sim-3of4timeoutyes0:00:10
Timeout because eval [...]
Timeout because evaluator contacted us
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104870sim-1of4successyes0:03:14
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104868sim-3of4successyes0:01:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104864sim-3of4successyes0:01:22
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104863sim-1of4timeoutyes0:00:10
Timeout because eval [...]
Timeout because evaluator contacted us
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104862sim-1of4successyes0:04:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104861sim-2of4successyes0:01:15
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
skipped1
No reset possible
104859sim-2of4successyes0:01:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104858sim-3of4successyes0:01:09
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104857sim-2of4successyes0:01:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
104856sim-0of4successyes0:05:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median12.550000000000043
in-drivable-lane_median0.6999999999999982
driven_lanedir_consec_median0.7037464582389903
deviation-center-line_median1.5041174749956825


other stats
deviation-center-line_max1.644438324951205
deviation-center-line_mean1.3212122877994932
deviation-center-line_min0.6321758762554034
deviation-heading_max14.564868662132817
deviation-heading_mean7.95844681832123
deviation-heading_median7.951386500411211
deviation-heading_min1.366145610329682
distance-from-start_max3.4526215628421757
distance-from-start_mean1.651874574286553
distance-from-start_median1.088845915025075
distance-from-start_min0.9771849042538864
driven_any_max5.132011522014099
driven_any_mean2.110162766575158
driven_any_median1.153041601432658
driven_any_min1.0025563414212155
driven_lanedir_consec_max4.293008292626892
driven_lanedir_consec_mean1.5894209822358063
driven_lanedir_consec_min0.657182719838354
driven_lanedir_max4.29305171243689
driven_lanedir_mean1.614633962943412
driven_lanedir_median0.7381518079514795
driven_lanedir_min0.6891805234338002
in-drivable-lane_max2.250000000000032
in-drivable-lane_mean0.9625000000000068
in-drivable-lane_min0.1999999999999993
per-episodes
details{"udem-000-ego0": {"driven_any": 5.132011522014099, "survival_time": 12.550000000000043, "driven_lanedir": 4.29305171243689, "in-drivable-lane": 2.250000000000032, "deviation-heading": 2.248039967650166, "distance-from-start": 3.4526215628421757, "deviation-center-line": 0.6321758762554034, "driven_lanedir_consec": 4.293008292626892}, "udem-000-ego1": {"driven_any": 1.303169591453934, "survival_time": 12.550000000000043, "driven_lanedir": 0.6891805234338002, "in-drivable-lane": 1.0499999999999976, "deviation-heading": 1.366145610329682, "distance-from-start": 1.188111904600963, "deviation-center-line": 1.644438324951205, "driven_lanedir_consec": 0.6891805234338002}, "udem-000-ego2": {"driven_any": 1.0025563414212155, "survival_time": 12.550000000000043, "driven_lanedir": 0.7491199120506384, "in-drivable-lane": 0.1999999999999993, "deviation-heading": 13.654733033172256, "distance-from-start": 0.9771849042538864, "deviation-center-line": 1.374292826582619, "driven_lanedir_consec": 0.7183123930441804}, "udem-000-ego3": {"driven_any": 1.002913611411382, "survival_time": 12.550000000000043, "driven_lanedir": 0.7271837038523206, "in-drivable-lane": 0.34999999999999876, "deviation-heading": 14.564868662132817, "distance-from-start": 0.9895799254491876, "deviation-center-line": 1.6339421234087452, "driven_lanedir_consec": 0.657182719838354}}
simulation-passed1
survival_time_max12.550000000000043
survival_time_mean12.550000000000043
survival_time_min12.550000000000043
No reset possible
99788sim-0of4failedno0:01:21
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 293, in main
    sim_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 423, in main
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: external protocol violation
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99786sim-1of4failedno0:01:27
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 293, in main
    sim_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 423, in main
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: external protocol violation
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99785sim-3of4successno0:01:09
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99782sim-2of4successno0:01:06
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99779sim-0of4host-errorno0:01:16
The container "simul [...]
The container "simulator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99777sim-1of4host-errorno0:01:19
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99775sim-3of4successno0:01:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99773sim-2of4successno0:01:06
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99769sim-0of4failedno0:01:25
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 293, in main
    sim_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 423, in main
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: external protocol violation
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99767sim-1of4failedno0:01:25
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 293, in main
    sim_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 423, in main
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: external protocol violation
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99766sim-3of4successno0:01:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99762sim-2of4successno0:01:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99758sim-0of4failedno0:01:03
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 293, in main
    sim_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 328, in read_reply
    raise RemoteNodeAborted(msg) from None
zuper_nodes.structures.RemoteNodeAborted: Remote node closed communication (None)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 423, in main
    raise dc.InvalidSubmission(msg) from e
duckietown_challenges.exceptions.InvalidSubmission: external protocol violation
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99756sim-3of4successno0:01:21
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99755sim-1of4host-errorno0:01:23
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
99754sim-2of4successno0:01:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97711sim-0of4successno0:03:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97710sim-1of4successno0:04:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97700sim-0of4successno0:03:59
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97698sim-1of4successno0:05:09
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97697sim-2of4successno0:01:08
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97696sim-3of4successno0:01:09
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97695sim-2of4successno0:01:08
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97694sim-3of4successno0:01:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97692sim-2of4successno0:01:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
97691sim-3of4successno0:01:13
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
73429sim-1of4failedno0:03:27
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 219, in main
    raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego1: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
73412sim-1of4host-errorno0:01:06
The container "evalu [...]
The container "evaluator" exited with code 1.


Look at the logs for the container to know more about the error.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible