Duckietown Challenges Home Challenges Submissions

Submission 13593

Submission13593
Competingyes
Challengeaido-LFV_multi-sim-validation
UserAndras Beres
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobssim-0of4: 105375 sim-1of4: 105383 sim-2of4: 105374 sim-3of4: 105370
Next
User label202-1
Admin priority50
Blessingn/a
User priority50

105383

Click the images to see detailed statistics about the episode.

autolab-000

105375

Click the images to see detailed statistics about the episode.

loop2-000

105374

Click the images to see detailed statistics about the episode.

loop-000

105370

Click the images to see detailed statistics about the episode.

zigzag-000

Evaluation jobs for this submission

See previous jobs for previous versions of challenges
Job IDstepstatusup to datedate starteddate completeddurationmessage
105383sim-1of4successyes0:04:40
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median12.950000000000047
in-drivable-lane_median0.825000000000006
driven_lanedir_consec_median2.6523854140626213
deviation-center-line_median0.8147060649575999


other stats
deviation-center-line_max1.2228993407770044
deviation-center-line_mean0.7983954063396792
deviation-center-line_min0.3412701546665129
deviation-heading_max3.841756181853065
deviation-heading_mean2.5619969388991537
deviation-heading_median2.5649760300143862
deviation-heading_min1.276279513714777
distance-from-start_max2.8237463815705772
distance-from-start_mean2.097568620095059
distance-from-start_median1.9334611317136177
distance-from-start_min1.6996058353824242
driven_any_max5.860492196189636
driven_any_mean3.5549316400238036
driven_any_median2.928560538932264
driven_any_min2.502113286041049
driven_lanedir_consec_max5.6743146411174745
driven_lanedir_consec_mean3.29972578041653
driven_lanedir_consec_min2.2198176524234037
driven_lanedir_max5.6743146411174745
driven_lanedir_mean3.29972578041653
driven_lanedir_median2.6523854140626213
driven_lanedir_min2.2198176524234037
in-drivable-lane_max7.60000000000006
in-drivable-lane_mean2.4625000000000172
in-drivable-lane_min0.5999999999999979
per-episodes
details{"autolab-000-ego0": {"driven_any": 2.5333534496840824, "survival_time": 12.950000000000047, "driven_lanedir": 2.2198176524234037, "in-drivable-lane": 0.6499999999999977, "deviation-heading": 3.510833829640038, "distance-from-start": 1.8979559802097097, "deviation-center-line": 0.8285005616631292, "driven_lanedir_consec": 2.2198176524234037}, "autolab-000-ego1": {"driven_any": 2.502113286041049, "survival_time": 12.950000000000047, "driven_lanedir": 2.265180567716465, "in-drivable-lane": 7.60000000000006, "deviation-heading": 1.276279513714777, "distance-from-start": 1.9689662832175256, "deviation-center-line": 0.3412701546665129, "driven_lanedir_consec": 2.265180567716465}, "autolab-000-ego2": {"driven_any": 5.860492196189636, "survival_time": 12.950000000000047, "driven_lanedir": 5.6743146411174745, "in-drivable-lane": 1.0000000000000142, "deviation-heading": 1.6191182303887346, "distance-from-start": 2.8237463815705772, "deviation-center-line": 0.8009115682520707, "driven_lanedir_consec": 5.6743146411174745}, "autolab-000-ego3": {"driven_any": 3.3237676281804465, "survival_time": 12.950000000000047, "driven_lanedir": 3.0395902604087777, "in-drivable-lane": 0.5999999999999979, "deviation-heading": 3.841756181853065, "distance-from-start": 1.6996058353824242, "deviation-center-line": 1.2228993407770044, "driven_lanedir_consec": 3.0395902604087777}}
simulation-passed1
survival_time_max12.950000000000047
survival_time_mean12.950000000000047
survival_time_min12.950000000000047
No reset possible
105380sim-1of4host-erroryes0:01:19
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 298, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 338, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego3" aborted with the following error:

error in ego3 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "submission.py", line 60, in init
              ||     self.agent = self.create_agent(self.env)
              ||   File "submission.py", line 162, in create_agent_dagger
              ||     agent = DaggerAgent.load_from_checkpoint(
              ||   File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/device_dtype_mixin.py", line 124, in cuda
              ||     return super().cuda(device=device)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
              ||     return self._apply(lambda t: t.cuda(device))
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
              ||     param_applied = fn(param)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
              ||     return self._apply(lambda t: t.cuda(device))
              || RuntimeError: CUDA error: out of memory
              ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges/cie_concrete.py", line 691, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 306, in main
    raise InvalidEnvironment(msg) from e
duckietown_challenges.exceptions.InvalidEnvironment: Detected out of CUDA memory:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_experiment_manager/code.py", line 298, in main
    robot_ci.write_topic_and_expect_zero("seed", config.seed)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 156, in write_topic_and_expect_zero
    msgs = read_reply(self.fpout, timeout=timeout, nickname=self.nickname)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper_outside.py", line 338, in read_reply
    raise RemoteNodeAborted(msg)
zuper_nodes.structures.RemoteNodeAborted: The remote node "ego3" aborted with the following error:

error in ego3 |Unexpected error:
              |
              || Traceback (most recent call last):
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 322, in loop
              ||     call_if_fun_exists(node, "init", context=context_data)
              ||   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
              ||     f(**kwargs)
              ||   File "submission.py", line 60, in init
              ||     self.agent = self.create_agent(self.env)
              ||   File "submission.py", line 162, in create_agent_dagger
              ||     agent = DaggerAgent.load_from_checkpoint(
              ||   File "/usr/local/lib/python3.8/dist-packages/pytorch_lightning/utilities/device_dtype_mixin.py", line 124, in cuda
              ||     return super().cuda(device=device)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in cuda
              ||     return self._apply(lambda t: t.cuda(device))
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 359, in _apply
              ||     module._apply(fn)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 381, in _apply
              ||     param_applied = fn(param)
              ||   File "/usr/local/lib/python3.8/dist-packages/torch/nn/modules/module.py", line 463, in <lambda>
              ||     return self._apply(lambda t: t.cuda(device))
              || RuntimeError: CUDA error: out of memory
              ||
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
105375sim-0of4successyes0:07:48
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median59.99999999999873
in-drivable-lane_median0.0
driven_lanedir_consec_median29.561988136996668
deviation-center-line_median4.401976470780635


other stats
deviation-center-line_max4.5072542627184315
deviation-center-line_mean4.401976470780635
deviation-center-line_min4.296698678842838
deviation-heading_max9.781818520990235
deviation-heading_mean9.704702204504509
deviation-heading_median9.704702204504509
deviation-heading_min9.627585888018777
distance-from-start_max1.5452584168864982
distance-from-start_mean1.5346388047888606
distance-from-start_median1.5346388047888606
distance-from-start_min1.524019192691223
driven_any_max30.185254721535156
driven_any_mean30.13535199384767
driven_any_median30.13535199384767
driven_any_min30.085449266160182
driven_lanedir_consec_max29.617081076703677
driven_lanedir_consec_mean29.561988136996668
driven_lanedir_consec_min29.506895197289655
driven_lanedir_max29.617081076703677
driven_lanedir_mean29.561988136996668
driven_lanedir_median29.561988136996668
driven_lanedir_min29.506895197289655
in-drivable-lane_max0.0
in-drivable-lane_mean0.0
in-drivable-lane_min0.0
per-episodes
details{"loop2-000-ego0": {"driven_any": 30.185254721535156, "survival_time": 59.99999999999873, "driven_lanedir": 29.617081076703677, "in-drivable-lane": 0.0, "deviation-heading": 9.627585888018777, "distance-from-start": 1.524019192691223, "deviation-center-line": 4.296698678842838, "driven_lanedir_consec": 29.617081076703677}, "loop2-000-ego1": {"driven_any": 30.085449266160182, "survival_time": 59.99999999999873, "driven_lanedir": 29.506895197289655, "in-drivable-lane": 0.0, "deviation-heading": 9.781818520990235, "distance-from-start": 1.5452584168864982, "deviation-center-line": 4.5072542627184315, "driven_lanedir_consec": 29.506895197289655}}
simulation-passed1
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_min59.99999999999873
No reset possible
105374sim-2of4successyes0:05:25
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median20.10000000000015
in-drivable-lane_median0.04999999999999982
driven_lanedir_consec_median3.2856474994125424
deviation-center-line_median1.4665314505700766


other stats
deviation-center-line_max2.312727721625923
deviation-center-line_mean1.6249008803803635
deviation-center-line_min1.2538128987553785
deviation-heading_max2.579550795886515
deviation-heading_mean1.3853615246802988
deviation-heading_median1.0763305366819296
deviation-heading_min0.8092342294708216
distance-from-start_max3.176288349014952
distance-from-start_mean2.296819690111778
distance-from-start_median2.309609215856738
distance-from-start_min1.3917719797186845
driven_any_max9.899279831992786
driven_any_mean4.5855057895929985
driven_any_median3.314929835275091
driven_any_min1.8128836558290269
driven_lanedir_consec_max9.577855702599049
driven_lanedir_consec_mean4.472622872075474
driven_lanedir_consec_min1.741340786877766
driven_lanedir_max9.577855702599049
driven_lanedir_mean4.472622872075474
driven_lanedir_median3.2856474994125424
driven_lanedir_min1.741340786877766
in-drivable-lane_max0.4999999999999982
in-drivable-lane_mean0.14999999999999947
in-drivable-lane_min0.0
per-episodes
details{"loop-000-ego0": {"driven_any": 1.8128836558290269, "survival_time": 20.10000000000015, "driven_lanedir": 1.741340786877766, "in-drivable-lane": 0.09999999999999964, "deviation-heading": 0.8092342294708216, "distance-from-start": 1.3917719797186845, "deviation-center-line": 2.312727721625923, "driven_lanedir_consec": 1.741340786877766}, "loop-000-ego1": {"driven_any": 9.899279831992786, "survival_time": 20.10000000000015, "driven_lanedir": 9.577855702599049, "in-drivable-lane": 0.4999999999999982, "deviation-heading": 2.579550795886515, "distance-from-start": 3.176288349014952, "deviation-center-line": 1.3012522961492006, "driven_lanedir_consec": 9.577855702599049}, "loop-000-ego2": {"driven_any": 4.749996879430239, "survival_time": 20.10000000000015, "driven_lanedir": 4.707808504503678, "in-drivable-lane": 0.0, "deviation-heading": 1.0986795159473453, "distance-from-start": 3.1496860658373236, "deviation-center-line": 1.2538128987553785, "driven_lanedir_consec": 4.707808504503678}, "loop-000-ego3": {"driven_any": 1.879862791119943, "survival_time": 20.10000000000015, "driven_lanedir": 1.863486494321407, "in-drivable-lane": 0.0, "deviation-heading": 1.0539815574165137, "distance-from-start": 1.4695323658761523, "deviation-center-line": 1.6318106049909529, "driven_lanedir_consec": 1.863486494321407}}
simulation-passed1
survival_time_max20.10000000000015
survival_time_mean20.10000000000015
survival_time_min20.10000000000015
No reset possible
105370sim-3of4successyes0:13:19
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median48.69999999999937
in-drivable-lane_median2.1499999999999964
driven_lanedir_consec_median12.322743070479795
deviation-center-line_median3.596401485581716


other stats
deviation-center-line_max4.316224314377097
deviation-center-line_mean3.184080204965642
deviation-center-line_min1.2272935343220397
deviation-heading_max9.256244103646258
deviation-heading_mean5.965304786719054
deviation-heading_median5.816868851598063
deviation-heading_min2.971237340033831
distance-from-start_max4.542798343894436
distance-from-start_mean3.8390801594514583
distance-from-start_median3.7403849820716744
distance-from-start_min3.332752329768047
driven_any_max22.58986938727228
driven_any_mean14.184108232372845
driven_any_median12.98835312210083
driven_any_min8.169857298017433
driven_lanedir_consec_max21.29553013873238
driven_lanedir_consec_mean13.410013698119794
driven_lanedir_consec_min7.699038512787202
driven_lanedir_max21.29553013873238
driven_lanedir_mean13.410013698119794
driven_lanedir_median12.322743070479795
driven_lanedir_min7.699038512787202
in-drivable-lane_max30.849999999999252
in-drivable-lane_mean8.999999999999815
in-drivable-lane_min0.8500000000000121
per-episodes
details{"zigzag-000-ego0": {"driven_any": 8.467021358004626, "survival_time": 48.69999999999937, "driven_lanedir": 8.303049153354266, "in-drivable-lane": 30.849999999999252, "deviation-heading": 2.971237340033831, "distance-from-start": 3.745813445273361, "deviation-center-line": 1.2272935343220397, "driven_lanedir_consec": 8.303049153354266}, "zigzag-000-ego1": {"driven_any": 8.169857298017433, "survival_time": 48.69999999999937, "driven_lanedir": 7.699038512787202, "in-drivable-lane": 0.8500000000000121, "deviation-heading": 3.903201020561347, "distance-from-start": 3.332752329768047, "deviation-center-line": 4.316224314377097, "driven_lanedir_consec": 7.699038512787202}, "zigzag-000-ego2": {"driven_any": 22.58986938727228, "survival_time": 48.69999999999937, "driven_lanedir": 21.29553013873238, "in-drivable-lane": 2.2999999999999936, "deviation-heading": 7.730536682634778, "distance-from-start": 3.734956518869988, "deviation-center-line": 3.8310792488572134, "driven_lanedir_consec": 21.29553013873238}, "zigzag-000-ego3": {"driven_any": 17.509684886197036, "survival_time": 48.69999999999937, "driven_lanedir": 16.342436987605325, "in-drivable-lane": 1.9999999999999991, "deviation-heading": 9.256244103646258, "distance-from-start": 4.542798343894436, "deviation-center-line": 3.3617237223062184, "driven_lanedir_consec": 16.342436987605325}}
simulation-passed1
survival_time_max48.69999999999937
survival_time_mean48.69999999999937
survival_time_min48.69999999999937
No reset possible
90640402successyes0:29:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
90633402timeoutyes0:00:27
Timeout because eval [...]
Timeout because evaluator contacted us
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
90626403failedyes0:06:20
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/duckietown_challenges/cie_concrete.py", line 681, in scoring_context
    yield cie
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 60, in go
    wrap(cie)
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/experiment_manager.py", line 33, in wrap
    asyncio.run(main(cie, logdir, attempts), debug=True)
  File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
  File "/usr/local/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.8/site-packages/duckietown_experiment_manager/code.py", line 219, in main
    raise InvalidSubmission(msg)
duckietown_challenges.exceptions.InvalidSubmission: Timeout during connection to ego0: <SignalTimeout in state: 2>
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
74118402successyes0:13:05
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
74096403successyes0:13:33
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
62335356successyes1:34:36
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible