Submission 2871

Submission

2871

Competing

yes

Challenge

aido2-LF-sim-validation

User

Peter Almasi 🇭🇺

Date submitted

2019-04-20 17:14:43+00:00

Last status update

2021-12-04 20:58:44+00:00

Complete

complete

Details

Evaluation is complete.

Sisters

Result

💚

Jobs

step1-simulation: 22729

User label

Baseline solution using reinforcement learning

Admin priority

Blessing

n/a

User priority

22729

Click the images to see detailed statistics about the episode.

ETHZ_autolab_technical_track-0-0

ETHZ_autolab_technical_track-1-0

ETHZ_autolab_technical_track-2-0

ETHZ_autolab_technical_track-3-0

ETHZ_autolab_technical_track-4-0

Evaluation jobs for this submission

Show only up-to-date jobs

Job ID

step

status

up to date

date started

date completed

duration

message

22729

step1-simulation

success

yes

2019-05-17 01:08:44+00:00

2019-05-17 01:13:43+00:00

0:04:59

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

driven_lanedir_consec_median	0.4904896000634382
survival_time_median	2.549999999999999
deviation-center-line_median	0.09541691214490904
in-drivable-lane_median	0.7000000000000005

other stats

agent_compute-ego_max	0.1892401883096406
agent_compute-ego_mean	0.1782978719993496
agent_compute-ego_median	0.17596090224481398
agent_compute-ego_min	0.1739170481176937
deviation-center-line_max	0.16706879016331402
deviation-center-line_mean	0.11592128594329898
deviation-center-line_min	0.08423996792039261
deviation-heading_max	1.083109621003245
deviation-heading_mean	0.5856919368998621
deviation-heading_median	0.5492589823469466
deviation-heading_min	0.17756428089562137
driven_any_max	1.5514963210370207
driven_any_mean	1.0785795125277358
driven_any_median	1.1506069130399446
driven_any_min	0.5472645876238714
driven_lanedir_consec_max	0.9474636443237155
driven_lanedir_consec_mean	0.5278843998977987
driven_lanedir_consec_min	0.16532270128573723
driven_lanedir_max	0.9474636443237155
driven_lanedir_mean	0.5557552569621684
driven_lanedir_median	0.5276479243681105
driven_lanedir_min	0.2412522269178341
in-drivable-lane_max	1.4999999999999998
in-drivable-lane_mean	0.849999999999999
in-drivable-lane_min	0.2500000000000001
per-episodes	details {"ETHZ_autolab_technical_track-0-0": {"driven_any": 0.5991875632412206, "sim_physics": 0.06959721349900769, "survival_time": 1.5500000000000007, "driven_lanedir": 0.2412522269178341, "sim_render-ego": 0.06003359825380387, "in-drivable-lane": 0.7000000000000005, "agent_compute-ego": 0.17596090224481398, "deviation-heading": 0.1846359612340746, "set_robot_commands": 0.07947229569958102, "deviation-center-line": 0.08423996792039261, "driven_lanedir_consec": 0.2412522269178341, "sim_compute_sim_state": 0.04314797155318722, "sim_compute_performance-ego": 0.06688147206460276, "sim_compute_robot_state-ego": 0.06895706730504189}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 1.5514963210370207, "sim_physics": 0.06020353471531587, "survival_time": 3.399999999999996, "driven_lanedir": 0.5276479243681105, "sim_render-ego": 0.060863266972934496, "in-drivable-lane": 1.4999999999999998, "agent_compute-ego": 0.1739170481176937, "deviation-heading": 1.083109621003245, "set_robot_commands": 0.07779080026289996, "deviation-center-line": 0.13747104025917367, "driven_lanedir_consec": 0.4904896000634382, "sim_compute_sim_state": 0.04094335261513205, "sim_compute_performance-ego": 0.06403706003637875, "sim_compute_robot_state-ego": 0.06304815937491025}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.5472645876238714, "sim_physics": 0.06490089676596901, "survival_time": 1.6500000000000008, "driven_lanedir": 0.2646071874193092, "sim_render-ego": 0.06453294465036104, "in-drivable-lane": 0.2500000000000001, "agent_compute-ego": 0.1892401883096406, "deviation-heading": 0.9338908390194228, "set_robot_commands": 0.0768728256225586, "deviation-center-line": 0.09541691214490904, "driven_lanedir_consec": 0.16532270128573723, "sim_compute_sim_state": 0.040650830124363754, "sim_compute_performance-ego": 0.06348383065425989, "sim_compute_robot_state-ego": 0.06318543174050072}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 1.1506069130399446, "sim_physics": 0.061640505697212966, "survival_time": 2.549999999999999, "driven_lanedir": 0.7978053017818727, "sim_render-ego": 0.06220032187069163, "in-drivable-lane": 0.6499999999999981, "agent_compute-ego": 0.17525816898719937, "deviation-heading": 0.5492589823469466, "set_robot_commands": 0.07734928879083372, "deviation-center-line": 0.09540971922870548, "driven_lanedir_consec": 0.7948938268982682, "sim_compute_sim_state": 0.04230743296006147, "sim_compute_performance-ego": 0.06523799428752824, "sim_compute_robot_state-ego": 0.06790478556763892}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 1.544342177696622, "sim_physics": 0.060788146911128875, "survival_time": 3.099999999999997, "driven_lanedir": 0.9474636443237155, "sim_render-ego": 0.06420958811236967, "in-drivable-lane": 1.149999999999996, "agent_compute-ego": 0.17711305233740038, "deviation-heading": 0.17756428089562137, "set_robot_commands": 0.07899979622133317, "deviation-center-line": 0.16706879016331402, "driven_lanedir_consec": 0.9474636443237155, "sim_compute_sim_state": 0.04189070963090466, "sim_compute_performance-ego": 0.06715198486082015, "sim_compute_robot_state-ego": 0.06479328678500268}}
set_robot_commands_max	0.07947229569958102
set_robot_commands_mean	0.07809700131944129
set_robot_commands_median	0.07779080026289996
set_robot_commands_min	0.0768728256225586
sim_compute_performance-ego_max	0.06715198486082015
sim_compute_performance-ego_mean	0.06535846838071796
sim_compute_performance-ego_median	0.06523799428752824
sim_compute_performance-ego_min	0.06348383065425989
sim_compute_robot_state-ego_max	0.06895706730504189
sim_compute_robot_state-ego_mean	0.0655777461546189
sim_compute_robot_state-ego_median	0.06479328678500268
sim_compute_robot_state-ego_min	0.06304815937491025
sim_compute_sim_state_max	0.04314797155318722
sim_compute_sim_state_mean	0.04178805937672983
sim_compute_sim_state_median	0.04189070963090466
sim_compute_sim_state_min	0.040650830124363754
sim_physics_max	0.06959721349900769
sim_physics_mean	0.06342605951772688
sim_physics_median	0.061640505697212966
sim_physics_min	0.06020353471531587
sim_render-ego_max	0.06453294465036104
sim_render-ego_mean	0.06236794397203214
sim_render-ego_median	0.06220032187069163
sim_render-ego_min	0.06003359825380387
simulation-passed	1
survival_time_max	3.399999999999996
survival_time_mean	2.4499999999999984
survival_time_min	1.5500000000000007

No reset possible

20955

step1-simulation

success

2019-04-27 16:00:17+00:00

2019-04-27 16:10:05+00:00

0:09:48

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

20277

step1-simulation

success

2019-04-25 11:00:24+00:00

2019-04-25 11:10:34+00:00

0:10:10

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

19935

step1-simulation

error

2019-04-25 08:39:02+00:00

2019-04-25 08:40:00+00:00

0:00:58

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 135, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 250, in run_episode
    sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:

error in simulator |Exception while handling a message on topic "episode_start".
                   |
                   || Traceback (most recent call last):
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
                   ||     handle_message_node(parsed, receiver0, context0)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
                   ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
                   ||     f(**kwargs)
                   ||   File "gym_bridge.py", line 209, in on_received_episode_start
                   ||     self.update_observations(context)
                   ||   File "gym_bridge.py", line 281, in update_observations
                   ||     jpg_data = rgb2jpg(obs)
                   ||   File "gym_bridge.py", line 359, in rgb2jpg
                   ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
                   || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3, 4>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
                   || > Unsupported depth of input image:
                   || >     'VDepth::contains(depth)'
                   || > where
                   || >     'depth' is 6 (CV_64F)
                   ||
                   ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 135, in main
 >     physics_dt=config.physics_dt)
 >   File "experiment_manager.py", line 250, in run_episode
 >     sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
 >     others = read_until_over(fpout, timeout=timeout, nickname=nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
 >     raise RemoteNodeAborted(m)
 > zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:
 > 
 > error in simulator |Exception while handling a message on topic "episode_start".
 >                    |
 >                    || Traceback (most recent call last):
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
 >                    ||     handle_message_node(parsed, receiver0, context0)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
 >                    ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
 >                    ||     f(**kwargs)
 >                    ||   File "gym_bridge.py", line 209, in on_received_episode_start
 >                    ||     self.update_observations(context)
 >                    ||   File "gym_bridge.py", line 281, in update_observations
 >                    ||     jpg_data = rgb2jpg(obs)
 >                    ||   File "gym_bridge.py", line 359, in rgb2jpg
 >                    ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
 >                    || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3, 4>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
 >                    || > Unsupported depth of input image:
 >                    || >     'VDepth::contains(depth)'
 >                    || > where
 >                    || >     'depth' is 6 (CV_64F)
 >                    ||
 >                    ||
 >

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

19874

step1-simulation

error

2019-04-25 08:33:58+00:00

2019-04-25 08:34:59+00:00

0:01:01

InvalidEvaluator: Tr [...]

InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 135, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 250, in run_episode
    sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:

error in simulator |Exception while handling a message on topic "episode_start".
                   |
                   || Traceback (most recent call last):
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
                   ||     handle_message_node(parsed, receiver0, context0)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
                   ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
                   ||     f(**kwargs)
                   ||   File "gym_bridge.py", line 209, in on_received_episode_start
                   ||     self.update_observations(context)
                   ||   File "gym_bridge.py", line 274, in update_observations
                   ||     jpg_data = rgb2jpg(obs)
                   ||   File "gym_bridge.py", line 352, in rgb2jpg
                   ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
                   || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
                   || > Unsupported depth of input image:
                   || >     'VDepth::contains(depth)'
                   || > where
                   || >     'depth' is 6 (CV_64F)
                   ||
                   ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 135, in main
 >     physics_dt=config.physics_dt)
 >   File "experiment_manager.py", line 250, in run_episode
 >     sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
 >     others = read_until_over(fpout, timeout=timeout, nickname=nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
 >     raise RemoteNodeAborted(m)
 > zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:
 > 
 > error in simulator |Exception while handling a message on topic "episode_start".
 >                    |
 >                    || Traceback (most recent call last):
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
 >                    ||     handle_message_node(parsed, receiver0, context0)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
 >                    ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
 >                    ||     f(**kwargs)
 >                    ||   File "gym_bridge.py", line 209, in on_received_episode_start
 >                    ||     self.update_observations(context)
 >                    ||   File "gym_bridge.py", line 274, in update_observations
 >                    ||     jpg_data = rgb2jpg(obs)
 >                    ||   File "gym_bridge.py", line 352, in rgb2jpg
 >                    ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
 >                    || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<1>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
 >                    || > Unsupported depth of input image:
 >                    || >     'VDepth::contains(depth)'
 >                    || > where
 >                    || >     'depth' is 6 (CV_64F)
 >                    ||
 >                    ||
 >

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

19298

step1-simulation

success

2019-04-24 13:51:14+00:00

2019-04-24 13:59:10+00:00

0:07:56

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible

18930

step1-simulation

success

2019-04-20 17:15:06+00:00

2019-04-20 17:21:44+00:00

0:06:38

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.

No reset possible