Duckietown Challenges Home Challenges Submissions

Submission 2403

Submission2403
Competingyes
Challengeaido2-LF-sim-validation
UserVictor Guerra 🇫🇷
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 23001
Next
User labelBaseline solution using reinforcement learning
Admin priority50
Blessingn/a
User priority50

23001

Click the images to see detailed statistics about the episode.

ETHZ_autolab_technical_track-0-0

ETHZ_autolab_technical_track-1-0

ETHZ_autolab_technical_track-2-0

ETHZ_autolab_technical_track-3-0

ETHZ_autolab_technical_track-4-0

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
23001step1-simulationsuccessyes0:05:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.0879707357496411
survival_time_median1.6000000000000008
deviation-center-line_median0.061421608772326886
in-drivable-lane_median0.2500000000000002


other stats
agent_compute-ego_max0.18957674843924385
agent_compute-ego_mean0.17229409782659438
agent_compute-ego_median0.17428878375462123
agent_compute-ego_min0.1575377214522589
deviation-center-line_max0.09613914017329872
deviation-center-line_mean0.05992299300298358
deviation-center-line_min0.0334463189102957
deviation-heading_max0.9070426694003584
deviation-heading_mean0.7425432243241079
deviation-heading_median0.7516893356055382
deviation-heading_min0.6253597766950494
driven_any_max0.2151225960209842
driven_any_mean0.17651487499882995
driven_any_median0.18657001193419032
driven_any_min0.10724970951735524
driven_lanedir_consec_max0.10384855163725804
driven_lanedir_consec_mean0.0831559759195335
driven_lanedir_consec_min0.05746618586638652
driven_lanedir_max0.10384855163725804
driven_lanedir_mean0.0831559759195335
driven_lanedir_median0.0879707357496411
driven_lanedir_min0.05746618586638652
in-drivable-lane_max0.6000000000000005
in-drivable-lane_mean0.3100000000000003
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 0.2151225960209842, "sim_physics": 0.12717093058994838, "survival_time": 1.7500000000000009, "driven_lanedir": 0.09144368554025784, "sim_render-ego": 0.06494336809430803, "in-drivable-lane": 0.5000000000000004, "agent_compute-ego": 0.18957674843924385, "deviation-heading": 0.7764592681912098, "set_robot_commands": 0.10278299195425852, "deviation-center-line": 0.06818860347594843, "driven_lanedir_consec": 0.09144368554025784, "sim_compute_sim_state": 0.04189400672912598, "sim_compute_performance-ego": 0.07558866909572057, "sim_compute_robot_state-ego": 0.0838160446711949}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.19467619321899263, "sim_physics": 0.11166834086179732, "survival_time": 1.6000000000000008, "driven_lanedir": 0.10384855163725804, "sim_render-ego": 0.06000310182571411, "in-drivable-lane": 0.2500000000000002, "agent_compute-ego": 0.17453084886074066, "deviation-heading": 0.7516893356055382, "set_robot_commands": 0.09388358891010284, "deviation-center-line": 0.04041929368304813, "driven_lanedir_consec": 0.10384855163725804, "sim_compute_sim_state": 0.035697028040885925, "sim_compute_performance-ego": 0.0657435730099678, "sim_compute_robot_state-ego": 0.07521426677703857}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.10724970951735524, "sim_physics": 0.10795231092543828, "survival_time": 1.0500000000000005, "driven_lanedir": 0.075050720804124, "sim_render-ego": 0.05739810353233701, "in-drivable-lane": 0, "agent_compute-ego": 0.1575377214522589, "deviation-heading": 0.6253597766950494, "set_robot_commands": 0.08951079277765184, "deviation-center-line": 0.09613914017329872, "driven_lanedir_consec": 0.075050720804124, "sim_compute_sim_state": 0.038053069795881, "sim_compute_performance-ego": 0.06615343548002697, "sim_compute_robot_state-ego": 0.06795352981204078}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.18657001193419032, "sim_physics": 0.10810890368052892, "survival_time": 1.4000000000000006, "driven_lanedir": 0.0879707357496411, "sim_render-ego": 0.05760937077658517, "in-drivable-lane": 0.20000000000000015, "agent_compute-ego": 0.16553638662610734, "deviation-heading": 0.9070426694003584, "set_robot_commands": 0.09236527340752738, "deviation-center-line": 0.061421608772326886, "driven_lanedir_consec": 0.0879707357496411, "sim_compute_sim_state": 0.04067489930561611, "sim_compute_performance-ego": 0.06430785145078387, "sim_compute_robot_state-ego": 0.07048079797199794}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 0.17895586430262742, "sim_physics": 0.10930899211338588, "survival_time": 1.7500000000000009, "driven_lanedir": 0.05746618586638652, "sim_render-ego": 0.06895688601902553, "in-drivable-lane": 0.6000000000000005, "agent_compute-ego": 0.17428878375462123, "deviation-heading": 0.6521650717283833, "set_robot_commands": 0.09396939277648926, "deviation-center-line": 0.0334463189102957, "driven_lanedir_consec": 0.05746618586638652, "sim_compute_sim_state": 0.03510379110063825, "sim_compute_performance-ego": 0.06838094166346959, "sim_compute_robot_state-ego": 0.07191990443638392}}
set_robot_commands_max0.10278299195425852
set_robot_commands_mean0.09450240796520595
set_robot_commands_median0.09388358891010284
set_robot_commands_min0.08951079277765184
sim_compute_performance-ego_max0.07558866909572057
sim_compute_performance-ego_mean0.06803489413999377
sim_compute_performance-ego_median0.06615343548002697
sim_compute_performance-ego_min0.06430785145078387
sim_compute_robot_state-ego_max0.0838160446711949
sim_compute_robot_state-ego_mean0.07387690873373122
sim_compute_robot_state-ego_median0.07191990443638392
sim_compute_robot_state-ego_min0.06795352981204078
sim_compute_sim_state_max0.04189400672912598
sim_compute_sim_state_mean0.03828455899442945
sim_compute_sim_state_median0.038053069795881
sim_compute_sim_state_min0.03510379110063825
sim_physics_max0.12717093058994838
sim_physics_mean0.11284189563421976
sim_physics_median0.10930899211338588
sim_physics_min0.10795231092543828
sim_render-ego_max0.06895688601902553
sim_render-ego_mean0.06178216604959397
sim_render-ego_median0.06000310182571411
sim_render-ego_min0.05739810353233701
simulation-passed1
survival_time_max1.7500000000000009
survival_time_mean1.510000000000001
survival_time_min1.0500000000000005
No reset possible
21305step1-simulationerrorno0:03:24
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 87, in main
    sim_ci.write_topic_and_expect_zero('seed', config.seed)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 247, in read_reply
    wm: WireMessage = read_next_cbor(fpout, timeout=timeout, waiting_for=waiting_for)
  File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 161, in read_next_cbor
    wait_for_data(f, timeout, waiting_for)
  File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 146, in wait_for_data
    raise TimeoutError(msg)
TimeoutError: Timeout after 30.0 s.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 87, in main
 >     sim_ci.write_topic_and_expect_zero('seed', config.seed)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 247, in read_reply
 >     wm: WireMessage = read_next_cbor(fpout, timeout=timeout, waiting_for=waiting_for)
 >   File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 161, in read_next_cbor
 >     wait_for_data(f, timeout, waiting_for)
 >   File "/project/src/zuper-utils/src/zuper_json/json2cbor.py", line 146, in wait_for_data
 >     raise TimeoutError(msg)
 > TimeoutError: Timeout after 30.0 s.
 > 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20632step1-simulationsuccessno0:07:56
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20594step1-simulationhost-errorno0:12:35
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 990, in run_docker
    return subprocess.check_output(cmd0, cwd=cwd, stderr=sys.stderr)
  File "/usr/lib/python3.6/subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "/usr/lib/python3.6/subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['docker-compose', '-p', 'job20594-774297', 'ps', '-q', 'solution']' returned non-zero exit status 255.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 478, in get_cr
    submission_id=submission_id, timeout_sec=timeout_sec)
  File "/project/src/duckietown_challenges_runner/runner.py", line 573, in run_single
    write_logs(wd, project, services=config['services'])
  File "/project/src/duckietown_challenges_runner/runner.py", line 899, in write_logs
    container_id = get_id_for_service(wd, project, service)
  File "/project/src/duckietown_challenges_runner/runner.py", line 585, in get_id_for_service
    o = run_docker(wd, project, cmd, get_output=True)
  File "/project/src/duckietown_challenges_runner/runner.py", line 998, in run_docker
    raise DockerComposeFail(msg) from e
duckietown_challenges_runner.runner.DockerComposeFail: Could not run ['docker-compose', '-p', 'job20594-774297', 'ps', '-q', 'solution']:

   >  Command '['docker-compose', '-p', 'job20594-774297', 'ps', '-q', 'solution']' returned non-zero exit status 255.

stdout | 

stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20019step1-simulationerrorno0:00:59
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 135, in main
    physics_dt=config.physics_dt)
  File "experiment_manager.py", line 250, in run_episode
    sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
    nickname=self.nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
    others = read_until_over(fpout, timeout=timeout, nickname=nickname)
  File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
    raise RemoteNodeAborted(m)
zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:

error in simulator |Exception while handling a message on topic "episode_start".
                   |
                   || Traceback (most recent call last):
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
                   ||     handle_message_node(parsed, receiver0, context0)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
                   ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
                   ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
                   ||     f(**kwargs)
                   ||   File "gym_bridge.py", line 209, in on_received_episode_start
                   ||     self.update_observations(context)
                   ||   File "gym_bridge.py", line 281, in update_observations
                   ||     jpg_data = rgb2jpg(obs)
                   ||   File "gym_bridge.py", line 359, in rgb2jpg
                   ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
                   || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3, 4>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
                   || > Unsupported depth of input image:
                   || >     'VDepth::contains(depth)'
                   || > where
                   || >     'depth' is 6 (CV_64F)
                   ||
                   ||

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 439, in <module>
    wrap(cie)
  File "experiment_manager.py", line 427, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 135, in main
 >     physics_dt=config.physics_dt)
 >   File "experiment_manager.py", line 250, in run_episode
 >     sim_ci.write_topic_and_expect_zero('episode_start', EpisodeStart(episode_name))
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 102, in write_topic_and_expect_zero
 >     nickname=self.nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 255, in read_reply
 >     others = read_until_over(fpout, timeout=timeout, nickname=nickname)
 >   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper_outside.py", line 283, in read_until_over
 >     raise RemoteNodeAborted(m)
 > zuper_nodes.structures.RemoteNodeAborted: External node "simulator" aborted:
 > 
 > error in simulator |Exception while handling a message on topic "episode_start".
 >                    |
 >                    || Traceback (most recent call last):
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 274, in loop
 >                    ||     handle_message_node(parsed, receiver0, context0)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/wrapper.py", line 428, in handle_message_node
 >                    ||     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
 >                    ||   File "/project/src/zuper-nodes/src/zuper_nodes_wrapper/utils.py", line 18, in call_if_fun_exists
 >                    ||     f(**kwargs)
 >                    ||   File "gym_bridge.py", line 209, in on_received_episode_start
 >                    ||     self.update_observations(context)
 >                    ||   File "gym_bridge.py", line 281, in update_observations
 >                    ||     jpg_data = rgb2jpg(obs)
 >                    ||   File "gym_bridge.py", line 359, in rgb2jpg
 >                    ||     bgr = cv2.cvtColor(rgb, cv2.COLOR_RGB2BGR)
 >                    || cv2.error: OpenCV(4.1.0) /io/opencv/modules/imgproc/src/color.simd_helpers.hpp:94: error: (-2:Unspecified error) in function 'cv::impl::{anonymous}::CvtHelper<VScn, VDcn, VDepth, sizePolicy>::CvtHelper(cv::InputArray, cv::OutputArray, int) [with VScn = cv::impl::{anonymous}::Set<3, 4>; VDcn = cv::impl::{anonymous}::Set<3, 4>; VDepth = cv::impl::{anonymous}::Set<0, 2, 5>; cv::impl::{anonymous}::SizePolicy sizePolicy = (cv::impl::<unnamed>::SizePolicy)2u; cv::InputArray = const cv::_InputArray&; cv::OutputArray = const cv::_OutputArray&]'
 >                    || > Unsupported depth of input image:
 >                    || >     'VDepth::contains(depth)'
 >                    || > where
 >                    || >     'depth' is 6 (CV_64F)
 >                    ||
 >                    ||
 > 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19642step1-simulationsuccessno0:03:34
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18891step1-simulationsuccessno0:04:59
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18523step1-simulationsuccessno0:04:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17907step1-simulationsuccessno0:04:31
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17763step1-simulationsuccessno0:08:26
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
17593step1-simulationerrorno0:12:40
The result file is n [...]
The result file is not found. This usually means that the evaluator did not finish
and some times that there was an import error.

Check the evaluator log to see what happened.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible