Duckietown Challenges Home Challenges Submissions

Evaluator 5408

ID5408
evaluatorgpu-production-spot-0-02
ownerI don't have one 😀
machinegpu-production-spot-0_1457a4236b24
processgpu-production-spot-0-02_1457a4236b24
version6.2.7
first heard
last heard
statusinactive
# evaluating
# success9 80231
# timeout
# failed
# error
# aborted2 80338
# host-error
arm0
x86_641
Mac0
gpu available1
Number of processors64
Processor frequency (MHz)0.0 GHz
Free % of processors100%
RAM total (MB)249.0 GB
RAM free (MB)245.9 GB
Disk (MB)969.3 GB
Disk available (MB)881.8 GB
Docker Hub
P11
P2
Cloud simulations1
PI Camera0
# Duckiebots0
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
8034316193Maximilian van Amerongenexercises_braitenbergmooc-BV1sim-1of5successnogpu-production-spot-0-020:07:44
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
distance-from-start_mean2.0510711573736655


other stats
agent_compute-ego0_max0.011396103690796683
agent_compute-ego0_mean0.011396103690796683
agent_compute-ego0_median0.011396103690796683
agent_compute-ego0_min0.011396103690796683
complete-iteration_max0.2766901787030037
complete-iteration_mean0.2766901787030037
complete-iteration_median0.2766901787030037
complete-iteration_min0.2766901787030037
deviation-center-line_max0.0
deviation-center-line_mean0.0
deviation-center-line_median0.0
deviation-center-line_min0.0
deviation-heading_max0.0
deviation-heading_mean0.0
deviation-heading_median0.0
deviation-heading_min0.0
distance-from-start_max2.0510711573736655
distance-from-start_median2.0510711573736655
distance-from-start_min2.0510711573736655
driven_any_max2.7444522682632484
driven_any_mean2.7444522682632484
driven_any_median2.7444522682632484
driven_any_min2.7444522682632484
driven_lanedir_consec_max0.0
driven_lanedir_consec_mean0.0
driven_lanedir_consec_median0.0
driven_lanedir_consec_min0.0
driven_lanedir_max0.0
driven_lanedir_mean0.0
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max0.1268453567337959
get_duckie_state_mean0.1268453567337959
get_duckie_state_median0.1268453567337959
get_duckie_state_min0.1268453567337959
get_robot_state_max0.004101598584974134
get_robot_state_mean0.004101598584974134
get_robot_state_median0.004101598584974134
get_robot_state_min0.004101598584974134
get_state_dump_max0.024687332843107913
get_state_dump_mean0.024687332843107913
get_state_dump_median0.024687332843107913
get_state_dump_min0.024687332843107913
get_ui_image_max0.01631783303760347
get_ui_image_mean0.01631783303760347
get_ui_image_median0.01631783303760347
get_ui_image_min0.01631783303760347
in-drivable-lane_max38.79999999999993
in-drivable-lane_mean38.79999999999993
in-drivable-lane_median38.79999999999993
in-drivable-lane_min38.79999999999993
per-episodes
details{"d60-ego0": {"driven_any": 2.7444522682632484, "get_ui_image": 0.01631783303760347, "step_physics": 0.07508751753827933, "survival_time": 38.79999999999993, "driven_lanedir": 0.0, "get_state_dump": 0.024687332843107913, "get_robot_state": 0.004101598584974134, "sim_render-ego0": 0.003893833516340611, "get_duckie_state": 0.1268453567337959, "in-drivable-lane": 38.79999999999993, "deviation-heading": 0.0, "agent_compute-ego0": 0.011396103690796683, "complete-iteration": 0.2766901787030037, "set_robot_commands": 0.002352353995677894, "distance-from-start": 2.0510711573736655, "deviation-center-line": 0.0, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.009861327199555426, "sim_compute_performance-ego0": 0.002035796105324685}}
set_robot_commands_max0.002352353995677894
set_robot_commands_mean0.002352353995677894
set_robot_commands_median0.002352353995677894
set_robot_commands_min0.002352353995677894
sim_compute_performance-ego0_max0.002035796105324685
sim_compute_performance-ego0_mean0.002035796105324685
sim_compute_performance-ego0_median0.002035796105324685
sim_compute_performance-ego0_min0.002035796105324685
sim_compute_sim_state_max0.009861327199555426
sim_compute_sim_state_mean0.009861327199555426
sim_compute_sim_state_median0.009861327199555426
sim_compute_sim_state_min0.009861327199555426
sim_render-ego0_max0.003893833516340611
sim_render-ego0_mean0.003893833516340611
sim_render-ego0_median0.003893833516340611
sim_render-ego0_min0.003893833516340611
simulation-passed1
step_physics_max0.07508751753827933
step_physics_mean0.07508751753827933
step_physics_median0.07508751753827933
step_physics_min0.07508751753827933
survival_time_max38.79999999999993
survival_time_mean38.79999999999993
survival_time_median38.79999999999993
survival_time_min38.79999999999993
No reset possible
8034113798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortednogpu-production-spot-0-020:00:22
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3Ab13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 745, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 944, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
8033813798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortednogpu-production-spot-0-020:00:22
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3Ab13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 745, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 944, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
8033616026Ethan Frankcollision-check-exercisemooc-collision-check-teststep1successnogpu-production-spot-0-020:00:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.08
false_positive_rate0.02


other stats
nsamples200
No reset possible
8033316026Ethan Frankcollision-check-exercisemooc-collision-check-teststep1successnogpu-production-spot-0-020:00:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.08
false_positive_rate0.02


other stats
nsamples200
No reset possible
8033116192Maximilian van Amerongenexercises_braitenbergmooc-BV1sim-2of5successnogpu-production-spot-0-020:01:37
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
distance-from-start_mean0.7642667267988466


other stats
agent_compute-ego0_max0.0121732493664356
agent_compute-ego0_mean0.0121732493664356
agent_compute-ego0_median0.0121732493664356
agent_compute-ego0_min0.0121732493664356
complete-iteration_max0.22095074552170776
complete-iteration_mean0.22095074552170776
complete-iteration_median0.22095074552170776
complete-iteration_min0.22095074552170776
deviation-center-line_max0.0
deviation-center-line_mean0.0
deviation-center-line_median0.0
deviation-center-line_min0.0
deviation-heading_max0.0
deviation-heading_mean0.0
deviation-heading_median0.0
deviation-heading_min0.0
distance-from-start_max0.7642667267988466
distance-from-start_median0.7642667267988466
distance-from-start_min0.7642667267988466
driven_any_max0.7735715762755211
driven_any_mean0.7735715762755211
driven_any_median0.7735715762755211
driven_any_min0.7735715762755211
driven_lanedir_consec_max0.0
driven_lanedir_consec_mean0.0
driven_lanedir_consec_median0.0
driven_lanedir_consec_min0.0
driven_lanedir_max0.0
driven_lanedir_mean0.0
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max0.0799894485067814
get_duckie_state_mean0.0799894485067814
get_duckie_state_median0.0799894485067814
get_duckie_state_min0.0799894485067814
get_robot_state_max0.0037871852834173974
get_robot_state_mean0.0037871852834173974
get_robot_state_median0.0037871852834173974
get_robot_state_min0.0037871852834173974
get_state_dump_max0.017419353444525536
get_state_dump_mean0.017419353444525536
get_state_dump_median0.017419353444525536
get_state_dump_min0.017419353444525536
get_ui_image_max0.014739477888066718
get_ui_image_mean0.014739477888066718
get_ui_image_median0.014739477888066718
get_ui_image_min0.014739477888066718
in-drivable-lane_max4.6499999999999915
in-drivable-lane_mean4.6499999999999915
in-drivable-lane_median4.6499999999999915
in-drivable-lane_min4.6499999999999915
per-episodes
details{"d40-ego0": {"driven_any": 0.7735715762755211, "get_ui_image": 0.014739477888066718, "step_physics": 0.07530237005112018, "survival_time": 4.6499999999999915, "driven_lanedir": 0.0, "get_state_dump": 0.017419353444525536, "get_robot_state": 0.0037871852834173974, "sim_render-ego0": 0.0036543658439149246, "get_duckie_state": 0.0799894485067814, "in-drivable-lane": 4.6499999999999915, "deviation-heading": 0.0, "agent_compute-ego0": 0.0121732493664356, "complete-iteration": 0.22095074552170776, "set_robot_commands": 0.0022919786737320273, "distance-from-start": 0.7642667267988466, "deviation-center-line": 0.0, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.009566337504285448, "sim_compute_performance-ego0": 0.001927624357507584}}
set_robot_commands_max0.0022919786737320273
set_robot_commands_mean0.0022919786737320273
set_robot_commands_median0.0022919786737320273
set_robot_commands_min0.0022919786737320273
sim_compute_performance-ego0_max0.001927624357507584
sim_compute_performance-ego0_mean0.001927624357507584
sim_compute_performance-ego0_median0.001927624357507584
sim_compute_performance-ego0_min0.001927624357507584
sim_compute_sim_state_max0.009566337504285448
sim_compute_sim_state_mean0.009566337504285448
sim_compute_sim_state_median0.009566337504285448
sim_compute_sim_state_min0.009566337504285448
sim_render-ego0_max0.0036543658439149246
sim_render-ego0_mean0.0036543658439149246
sim_render-ego0_median0.0036543658439149246
sim_render-ego0_min0.0036543658439149246
simulation-passed1
step_physics_max0.07530237005112018
step_physics_mean0.07530237005112018
step_physics_median0.07530237005112018
step_physics_min0.07530237005112018
survival_time_max4.6499999999999915
survival_time_mean4.6499999999999915
survival_time_median4.6499999999999915
survival_time_min4.6499999999999915
No reset possible
8029115871Bea Baselines 🐤baseline-solutionmooc-collision-check-teststep1successnogpu-production-spot-0-020:01:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.01
false_positive_rate0.0


other stats
nsamples200
No reset possible
8026615870Bea Baselines 🐤baseline-solutionmooc-collision-check-valistep1successnogpu-production-spot-0-020:01:11
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.015
false_positive_rate0.0


other stats
nsamples400
No reset possible
8025515872Bea Baselines 🐤dummy_collision_checkermooc-collision-check-valistep1successnogpu-production-spot-0-020:00:32
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.485
false_positive_rate0.51


other stats
nsamples400
No reset possible
8024015874Bea Baselines 🐤collision-check-exercisemooc-collision-check-valistep1successnogpu-production-spot-0-020:00:28
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.605
false_positive_rate0.475


other stats
nsamples400
No reset possible
8023116019Martin Cote 🇨🇦collision-check-exercisemooc-collision-check-valistep1successnogpu-production-spot-0-020:01:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.04
false_positive_rate0.035


other stats
nsamples400
No reset possible