Duckietown Challenges Home Challenges Submissions

Evaluator 5312

ID5312
evaluatornogpu-production-b-spot-0-04
ownerI don't have one 😀
machinenogpu-production-b-spot-0_4f456800b9a9
processnogpu-production-b-spot-0-04_4f456800b9a9
version6.2.7
first heard
last heard
statusinactive
# evaluating
# success5 79230
# timeout
# failed
# error
# aborted2 79183
# host-error
arm0
x86_641
Mac0
gpu available0
Number of processors96
Processor frequency (MHz)0.0 GHz
Free % of processors99%
RAM total (MB)184.7 GB
RAM free (MB)175.8 GB
Disk (MB)969.3 GB
Disk available (MB)922.4 GB
Docker Hub
P11
P2
Cloud simulations1
PI Camera0
# Duckiebots0
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
7959116076Awni Altabaacollision-check-exercisemooc-collision-check-teststep1successnonogpu-production-b-spot-0-040:00:23
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.41
false_positive_rate0.56


other stats
nsamples200
No reset possible
7942616045Zoe Nolanexercises_braitenbergmooc-BV1sim-1of5successnonogpu-production-b-spot-0-040:09:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
distance-from-start_mean2.8425645610808967


other stats
agent_compute-ego0_max0.009926340562120068
agent_compute-ego0_mean0.009926340562120068
agent_compute-ego0_median0.009926340562120068
agent_compute-ego0_min0.009926340562120068
complete-iteration_max0.24273738932550004
complete-iteration_mean0.24273738932550004
complete-iteration_median0.24273738932550004
complete-iteration_min0.24273738932550004
deviation-center-line_max0.0
deviation-center-line_mean0.0
deviation-center-line_median0.0
deviation-center-line_min0.0
deviation-heading_max0.0
deviation-heading_mean0.0
deviation-heading_median0.0
deviation-heading_min0.0
distance-from-start_max2.8425645610808967
distance-from-start_median2.8425645610808967
distance-from-start_min2.8425645610808967
driven_any_max4.09651514989727
driven_any_mean4.09651514989727
driven_any_median4.09651514989727
driven_any_min4.09651514989727
driven_lanedir_consec_max0.0
driven_lanedir_consec_mean0.0
driven_lanedir_consec_median0.0
driven_lanedir_consec_min0.0
driven_lanedir_max0.0
driven_lanedir_mean0.0
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max0.08866990615088775
get_duckie_state_mean0.08866990615088775
get_duckie_state_median0.08866990615088775
get_duckie_state_min0.08866990615088775
get_robot_state_max0.0032948947369704933
get_robot_state_mean0.0032948947369704933
get_robot_state_median0.0032948947369704933
get_robot_state_min0.0032948947369704933
get_state_dump_max0.01711178004592781
get_state_dump_mean0.01711178004592781
get_state_dump_median0.01711178004592781
get_state_dump_min0.01711178004592781
get_ui_image_max0.03875637372070109
get_ui_image_mean0.03875637372070109
get_ui_image_median0.03875637372070109
get_ui_image_min0.03875637372070109
in-drivable-lane_max59.99999999999873
in-drivable-lane_mean59.99999999999873
in-drivable-lane_median59.99999999999873
in-drivable-lane_min59.99999999999873
per-episodes
details{"d50-ego0": {"driven_any": 4.09651514989727, "get_ui_image": 0.03875637372070109, "step_physics": 0.07000130320667328, "survival_time": 59.99999999999873, "driven_lanedir": 0.0, "get_state_dump": 0.01711178004592781, "get_robot_state": 0.0032948947369704933, "sim_render-ego0": 0.0031669380067290117, "get_duckie_state": 0.08866990615088775, "in-drivable-lane": 59.99999999999873, "deviation-heading": 0.0, "agent_compute-ego0": 0.009926340562120068, "complete-iteration": 0.24273738932550004, "set_robot_commands": 0.0019378197580253351, "distance-from-start": 2.8425645610808967, "deviation-center-line": 0.0, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008126154430303645, "sim_compute_performance-ego0": 0.0016578575852113798}}
set_robot_commands_max0.0019378197580253351
set_robot_commands_mean0.0019378197580253351
set_robot_commands_median0.0019378197580253351
set_robot_commands_min0.0019378197580253351
sim_compute_performance-ego0_max0.0016578575852113798
sim_compute_performance-ego0_mean0.0016578575852113798
sim_compute_performance-ego0_median0.0016578575852113798
sim_compute_performance-ego0_min0.0016578575852113798
sim_compute_sim_state_max0.008126154430303645
sim_compute_sim_state_mean0.008126154430303645
sim_compute_sim_state_median0.008126154430303645
sim_compute_sim_state_min0.008126154430303645
sim_render-ego0_max0.0031669380067290117
sim_render-ego0_mean0.0031669380067290117
sim_render-ego0_median0.0031669380067290117
sim_render-ego0_min0.0031669380067290117
simulation-passed1
step_physics_max0.07000130320667328
step_physics_mean0.07000130320667328
step_physics_median0.07000130320667328
step_physics_min0.07000130320667328
survival_time_max59.99999999999873
survival_time_mean59.99999999999873
survival_time_median59.99999999999873
survival_time_min59.99999999999873
No reset possible
7939216039Zoe Nolanexercises_braitenbergmooc-BV1sim-2of5successnonogpu-production-b-spot-0-040:02:19
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
distance-from-start_mean0.8150899336796846


other stats
agent_compute-ego0_max0.009929229001529882
agent_compute-ego0_mean0.009929229001529882
agent_compute-ego0_median0.009929229001529882
agent_compute-ego0_min0.009929229001529882
complete-iteration_max0.2166726638059147
complete-iteration_mean0.2166726638059147
complete-iteration_median0.2166726638059147
complete-iteration_min0.2166726638059147
deviation-center-line_max0.0
deviation-center-line_mean0.0
deviation-center-line_median0.0
deviation-center-line_min0.0
deviation-heading_max0.0
deviation-heading_mean0.0
deviation-heading_median0.0
deviation-heading_min0.0
distance-from-start_max0.8150899336796846
distance-from-start_median0.8150899336796846
distance-from-start_min0.8150899336796846
driven_any_max0.8259389749387058
driven_any_mean0.8259389749387058
driven_any_median0.8259389749387058
driven_any_min0.8259389749387058
driven_lanedir_consec_max0.0
driven_lanedir_consec_mean0.0
driven_lanedir_consec_median0.0
driven_lanedir_consec_min0.0
driven_lanedir_max0.0
driven_lanedir_mean0.0
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max0.0676792459409745
get_duckie_state_mean0.0676792459409745
get_duckie_state_median0.0676792459409745
get_duckie_state_min0.0676792459409745
get_robot_state_max0.003122066865201856
get_robot_state_mean0.003122066865201856
get_robot_state_median0.003122066865201856
get_robot_state_min0.003122066865201856
get_state_dump_max0.01411472285380129
get_state_dump_mean0.01411472285380129
get_state_dump_median0.01411472285380129
get_state_dump_min0.01411472285380129
get_ui_image_max0.034994531850345796
get_ui_image_mean0.034994531850345796
get_ui_image_median0.034994531850345796
get_ui_image_min0.034994531850345796
in-drivable-lane_max12.150000000000038
in-drivable-lane_mean12.150000000000038
in-drivable-lane_median12.150000000000038
in-drivable-lane_min12.150000000000038
per-episodes
details{"d40-ego0": {"driven_any": 0.8259389749387058, "get_ui_image": 0.034994531850345796, "step_physics": 0.07137599538584224, "survival_time": 12.150000000000038, "driven_lanedir": 0.0, "get_state_dump": 0.01411472285380129, "get_robot_state": 0.003122066865201856, "sim_render-ego0": 0.003085233148981313, "get_duckie_state": 0.0676792459409745, "in-drivable-lane": 12.150000000000038, "deviation-heading": 0.0, "agent_compute-ego0": 0.009929229001529882, "complete-iteration": 0.2166726638059147, "set_robot_commands": 0.001825356092609343, "distance-from-start": 0.8150899336796846, "deviation-center-line": 0.0, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008842530797739497, "sim_compute_performance-ego0": 0.001620651268568195}}
set_robot_commands_max0.001825356092609343
set_robot_commands_mean0.001825356092609343
set_robot_commands_median0.001825356092609343
set_robot_commands_min0.001825356092609343
sim_compute_performance-ego0_max0.001620651268568195
sim_compute_performance-ego0_mean0.001620651268568195
sim_compute_performance-ego0_median0.001620651268568195
sim_compute_performance-ego0_min0.001620651268568195
sim_compute_sim_state_max0.008842530797739497
sim_compute_sim_state_mean0.008842530797739497
sim_compute_sim_state_median0.008842530797739497
sim_compute_sim_state_min0.008842530797739497
sim_render-ego0_max0.003085233148981313
sim_render-ego0_mean0.003085233148981313
sim_render-ego0_median0.003085233148981313
sim_render-ego0_min0.003085233148981313
simulation-passed1
step_physics_max0.07137599538584224
step_physics_mean0.07137599538584224
step_physics_median0.07137599538584224
step_physics_min0.07137599538584224
survival_time_max12.150000000000038
survival_time_mean12.150000000000038
survival_time_median12.150000000000038
survival_time_min12.150000000000038
No reset possible
7937716037Jerry Kuchexercises_braitenbergmooc-BV1sim-0of5successnonogpu-production-b-spot-0-040:05:27
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
distance-from-start_mean1.748441980985996


other stats
agent_compute-ego0_max0.010209898033527412
agent_compute-ego0_mean0.010209898033527412
agent_compute-ego0_median0.010209898033527412
agent_compute-ego0_min0.010209898033527412
complete-iteration_max0.23342792193094888
complete-iteration_mean0.23342792193094888
complete-iteration_median0.23342792193094888
complete-iteration_min0.23342792193094888
deviation-center-line_max0.0
deviation-center-line_mean0.0
deviation-center-line_median0.0
deviation-center-line_min0.0
deviation-heading_max0.0
deviation-heading_mean0.0
deviation-heading_median0.0
deviation-heading_min0.0
distance-from-start_max1.748441980985996
distance-from-start_median1.748441980985996
distance-from-start_min1.748441980985996
driven_any_max1.8409397934211753
driven_any_mean1.8409397934211753
driven_any_median1.8409397934211753
driven_any_min1.8409397934211753
driven_lanedir_consec_max0.0
driven_lanedir_consec_mean0.0
driven_lanedir_consec_median0.0
driven_lanedir_consec_min0.0
driven_lanedir_max0.0
driven_lanedir_mean0.0
driven_lanedir_median0.0
driven_lanedir_min0.0
get_duckie_state_max0.0801826501133466
get_duckie_state_mean0.0801826501133466
get_duckie_state_median0.0801826501133466
get_duckie_state_min0.0801826501133466
get_robot_state_max0.0033319415468158146
get_robot_state_mean0.0033319415468158146
get_robot_state_median0.0033319415468158146
get_robot_state_min0.0033319415468158146
get_state_dump_max0.016182370619340376
get_state_dump_mean0.016182370619340376
get_state_dump_median0.016182370619340376
get_state_dump_min0.016182370619340376
get_ui_image_max0.037351699790569265
get_ui_image_mean0.037351699790569265
get_ui_image_median0.037351699790569265
get_ui_image_min0.037351699790569265
in-drivable-lane_max24.700000000000216
in-drivable-lane_mean24.700000000000216
in-drivable-lane_median24.700000000000216
in-drivable-lane_min24.700000000000216
per-episodes
details{"d45-ego0": {"driven_any": 1.8409397934211753, "get_ui_image": 0.037351699790569265, "step_physics": 0.07062652953947433, "survival_time": 24.700000000000216, "driven_lanedir": 0.0, "get_state_dump": 0.016182370619340376, "get_robot_state": 0.0033319415468158146, "sim_render-ego0": 0.003247585200300121, "get_duckie_state": 0.0801826501133466, "in-drivable-lane": 24.700000000000216, "deviation-heading": 0.0, "agent_compute-ego0": 0.010209898033527412, "complete-iteration": 0.23342792193094888, "set_robot_commands": 0.0019454537015972716, "distance-from-start": 1.748441980985996, "deviation-center-line": 0.0, "driven_lanedir_consec": 0.0, "sim_compute_sim_state": 0.008521119994346541, "sim_compute_performance-ego0": 0.00173409924362645}}
set_robot_commands_max0.0019454537015972716
set_robot_commands_mean0.0019454537015972716
set_robot_commands_median0.0019454537015972716
set_robot_commands_min0.0019454537015972716
sim_compute_performance-ego0_max0.00173409924362645
sim_compute_performance-ego0_mean0.00173409924362645
sim_compute_performance-ego0_median0.00173409924362645
sim_compute_performance-ego0_min0.00173409924362645
sim_compute_sim_state_max0.008521119994346541
sim_compute_sim_state_mean0.008521119994346541
sim_compute_sim_state_median0.008521119994346541
sim_compute_sim_state_min0.008521119994346541
sim_render-ego0_max0.003247585200300121
sim_render-ego0_mean0.003247585200300121
sim_render-ego0_median0.003247585200300121
sim_render-ego0_min0.003247585200300121
simulation-passed1
step_physics_max0.07062652953947433
step_physics_mean0.07062652953947433
step_physics_median0.07062652953947433
step_physics_min0.07062652953947433
survival_time_max24.700000000000216
survival_time_mean24.700000000000216
survival_time_median24.700000000000216
survival_time_min24.700000000000216
No reset possible
7923015957Alessandro Zanardi 🇨🇭minimal-agent-fullaido-LFVI_multi_full-sim-validationsim-3of4successyesnogpu-production-b-spot-0-040:01:45
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
skipped1
No reset possible
7918513798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortednonogpu-production-b-spot-0-040:00:22
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3Ab13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 745, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 944, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
7918313798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortednonogpu-production-b-spot-0-040:00:53
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 261, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 941, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.35/images/create?tag=sha256%3Ab13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 65, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 415, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 263, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 38, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 84, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  None

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 745, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 944, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 42, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible