Duckietown Challenges Home Challenges Submissions

Evaluator 5449

ID5449
evaluatornogpu-production-b-spot-1-03
ownerI don't have one 😀
machinenogpu-production-b-spot-1
processnogpu-production-b-spot-1-03
version6.2.32
first heard
last heard
statusinactive
# evaluating
# success10 84105
# timeout
# failed
# error
# aborted3 84254
# host-error
arm0
x86_641
Mac0
gpu available0
Number of processors96
Processor frequency (MHz)0.0 GHz
Free % of processors100%
RAM total (MB)184.7 GB
RAM free (MB)177.2 GB
Disk (MB)969.3 GB
Disk available (MB)946.0 GB
Docker Hub
P11
P2
Cloud simulations1
PI Camera0
# Duckiebots0
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
8458516820Luigi Cappellabase-image-mlmooc-visservoingsimsuccessnonogpu-production-b-spot-1-030:08:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
in-drivable-lane_median6.5500000000000735
deviation-center-line_median0.6810259083111982
driven_lanedir_consec_median1.5041068411139604
survival_time_median22.450000000000166


other stats
agent_compute-ego0_max0.005013229385499031
agent_compute-ego0_mean0.004960180107447362
agent_compute-ego0_median0.004960180107447362
agent_compute-ego0_min0.004907130829395692
complete-iteration_max0.11249165573427754
complete-iteration_mean0.11182022358954556
complete-iteration_median0.11182022358954556
complete-iteration_min0.11114879144481354
deviation-center-line_max1.0767352899015146
deviation-center-line_mean0.6810259083111982
deviation-center-line_min0.28531652672088187
deviation-heading_max6.966692128279194
deviation-heading_mean4.045888493289607
deviation-heading_median4.045888493289607
deviation-heading_min1.1250848583000206
distance-from-start_max1.1750524915159009
distance-from-start_mean1.1084472174420823
distance-from-start_median1.1084472174420823
distance-from-start_min1.0418419433682635
driven_any_max3.2976548364310427
driven_any_mean2.236353663973472
driven_any_median2.236353663973472
driven_any_min1.1750524915159009
driven_lanedir_consec_max2.122127037678123
driven_lanedir_consec_mean1.5041068411139604
driven_lanedir_consec_min0.8860866445497975
driven_lanedir_max2.122127037678123
driven_lanedir_mean1.5041068411139604
driven_lanedir_median1.5041068411139604
driven_lanedir_min0.8860866445497975
get_duckie_state_max1.468966084141885e-06
get_duckie_state_mean1.445167871033959e-06
get_duckie_state_median1.445167871033959e-06
get_duckie_state_min1.421369657926033e-06
get_robot_state_max0.003190172295416555
get_robot_state_mean0.0031118120842585817
get_robot_state_median0.0031118120842585817
get_robot_state_min0.003033451873100608
get_state_dump_max0.004167747113012498
get_state_dump_mean0.00400143597542638
get_state_dump_median0.00400143597542638
get_state_dump_min0.0038351248378402617
get_ui_image_max0.0373255895690684
get_ui_image_mean0.03690530728366666
get_ui_image_median0.03690530728366666
get_ui_image_min0.03648502499826493
in-drivable-lane_max9.900000000000102
in-drivable-lane_mean6.5500000000000735
in-drivable-lane_min3.2000000000000455
per-episodes
details{"LF-small-loop-000-ego0": {"driven_any": 3.2976548364310427, "get_ui_image": 0.0373255895690684, "step_physics": 0.05245554483741339, "survival_time": 32.55000000000029, "driven_lanedir": 2.122127037678123, "get_state_dump": 0.0038351248378402617, "get_robot_state": 0.003033451873100608, "sim_render-ego0": 0.002969568126772079, "get_duckie_state": 1.421369657926033e-06, "in-drivable-lane": 9.900000000000102, "deviation-heading": 6.966692128279194, "agent_compute-ego0": 0.004907130829395692, "complete-iteration": 0.11114879144481354, "set_robot_commands": 0.001802410084777083, "distance-from-start": 1.0418419433682635, "deviation-center-line": 1.0767352899015146, "driven_lanedir_consec": 2.122127037678123, "sim_compute_sim_state": 0.003219659708760267, "sim_compute_performance-ego0": 0.0015264478929203711}, "LF-small-loop-001-ego0": {"driven_any": 1.1750524915159009, "get_ui_image": 0.03648502499826493, "step_physics": 0.05297797341500559, "survival_time": 12.35000000000004, "driven_lanedir": 0.8860866445497975, "get_state_dump": 0.004167747113012498, "get_robot_state": 0.003190172295416555, "sim_render-ego0": 0.003103931103983233, "get_duckie_state": 1.468966084141885e-06, "in-drivable-lane": 3.2000000000000455, "deviation-heading": 1.1250848583000206, "agent_compute-ego0": 0.005013229385499031, "complete-iteration": 0.11249165573427754, "set_robot_commands": 0.0018551888004426033, "distance-from-start": 1.1750524915159009, "deviation-center-line": 0.28531652672088187, "driven_lanedir_consec": 0.8860866445497975, "sim_compute_sim_state": 0.004029544130448372, "sim_compute_performance-ego0": 0.0015949174281089536}}
set_robot_commands_max0.0018551888004426033
set_robot_commands_mean0.0018287994426098432
set_robot_commands_median0.0018287994426098432
set_robot_commands_min0.001802410084777083
sim_compute_performance-ego0_max0.0015949174281089536
sim_compute_performance-ego0_mean0.0015606826605146624
sim_compute_performance-ego0_median0.0015606826605146624
sim_compute_performance-ego0_min0.0015264478929203711
sim_compute_sim_state_max0.004029544130448372
sim_compute_sim_state_mean0.00362460191960432
sim_compute_sim_state_median0.00362460191960432
sim_compute_sim_state_min0.003219659708760267
sim_render-ego0_max0.003103931103983233
sim_render-ego0_mean0.003036749615377656
sim_render-ego0_median0.003036749615377656
sim_render-ego0_min0.002969568126772079
simulation-passed1
step_physics_max0.05297797341500559
step_physics_mean0.05271675912620949
step_physics_median0.05271675912620949
step_physics_min0.05245554483741339
survival_time_max32.55000000000029
survival_time_mean22.450000000000166
survival_time_min12.35000000000004
No reset possible
8431916659Bea Baselines 🐤baseline-solutionmooc-planning-cc-static-teststep1-3of4successyesnogpu-production-b-spot-1-030:04:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
mistakes1.0
success_ratio1.0
duration30.507126661541704
complexity93.33333333333331
avg_min_distance0.210701289869716


other stats
nfeasible6
nsamples6
No reset possible
8430716661Bea Baselines 🐤baseline-solutionmooc-planning-cc-dynamic-teststep1-2of4successyesnogpu-production-b-spot-1-030:05:12
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
mistakes1.0
success_ratio1.0
duration29.747882934684437
complexity85.0
avg_min_distance0.02541947948548058


other stats
nfeasible6
nsamples6
No reset possible
8428816655Bea Baselines 🐤baseline-solutionmooc-planning-dd-dynamic-teststep1-2of4successyesnogpu-production-b-spot-1-030:05:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
mistakes0.3333333333333333
success_ratio1.0
duration29.889309079566573
complexity85.5
avg_min_distance0.055014608025610395


other stats
nfeasible6
nsamples6
No reset possible
8426713798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortedyesnogpu-production-b-spot-1-030:00:22
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/create?tag=2021_03_25_17_23_41&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 93, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 430, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=2021_03_25_17_23_41&fromImage=docker.io%2Fnitaigao%2Faido-submissions: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 47, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet, credentials=credentials)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 112, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  2021_03_25_17_23_41

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 836, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1038, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5, credentials=credentials, quiet=False)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 51, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull nitaigao/aido-submissions:2021_03_25_17_23_41@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
8426513798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortedyesnogpu-production-b-spot-1-030:00:23
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/create?tag=2021_03_25_17_23_41&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 93, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 430, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=2021_03_25_17_23_41&fromImage=docker.io%2Fnitaigao%2Faido-submissions: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 47, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet, credentials=credentials)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 112, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  2021_03_25_17_23_41

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 836, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1038, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5, credentials=credentials, quiet=False)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 51, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull nitaigao/aido-submissions:2021_03_25_17_23_41@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
8425413798Nicholas Kostelniktemplate-randomaido-hello-sim-validation370abortedyesnogpu-production-b-spot-1-030:00:23
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 268, in _raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.8/dist-packages/requests/models.py", line 953, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: http+docker://localhost/v1.41/images/create?tag=2021_03_25_17_23_41&fromImage=docker.io%2Fnitaigao%2Faido-submissions

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 93, in docker_pull
    pulling = client.api.pull(repository=repository, tag=br.tag, stream=True, decode=True)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/image.py", line 430, in pull
    self._raise_for_status(response)
  File "/usr/local/lib/python3.8/dist-packages/docker/api/client.py", line 270, in _raise_for_status
    raise create_api_error_from_http_exception(e)
  File "/usr/local/lib/python3.8/dist-packages/docker/errors.py", line 31, in create_api_error_from_http_exception
    raise cls(e, response=response, explanation=explanation)
docker.errors.ImageNotFound: 404 Client Error for http+docker://localhost/v1.41/images/create?tag=2021_03_25_17_23_41&fromImage=docker.io%2Fnitaigao%2Faido-submissions: Not Found ("pull access denied for nitaigao/aido-submissions, repository does not exist or may require 'docker login': denied: requested access to the resource is denied")

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 47, in docker_pull_retry
    return docker_pull(client, image_name, quiet=quiet, credentials=credentials)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 112, in docker_pull
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: Cannot pull repo  docker.io/nitaigao/aido-submissions@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691  tag  2021_03_25_17_23_41

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 836, in get_cr
    cr = run_single(
  File "/usr/local/lib/python3.8/dist-packages/duckietown_challenges_runner/runner.py", line 1038, in run_single
    docker_pull_retry(client, image, ntimes=4, wait=5, credentials=credentials, quiet=False)
  File "/usr/local/lib/python3.8/dist-packages/duckietown_build_utils/docker_pulling.py", line 51, in docker_pull_retry
    raise PullError(msg) from e
duckietown_build_utils.docker_pulling.PullError: After trying 4 I still could not pull nitaigao/aido-submissions:2021_03_25_17_23_41@sha256:b13078d04947eb3a802ebc4e9db985f6a60c2a3cae145d65fbd44ef0177e1691
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
8421116583Bea Baselines 🐤planning-exercisemooc-planning-cc-empty-valistep1successyesnogpu-production-b-spot-1-030:09:36
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
mistakes1.0
success_ratio1.0
duration22.0
complexity8.0
avg_min_distance0.2626749063171028


other stats
nfeasible27
nsamples27
No reset possible
8420216586Bea Baselines 🐤planning-exercisemooc-planning-cc-static-teststep1-0of4successyesnogpu-production-b-spot-1-030:02:37
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
mistakes1.0
success_ratio1.0
duration22.0
complexity8.0
avg_min_distance0.0


other stats
nfeasible6
nsamples6
No reset possible
8418616591Bea Baselines 🐤baseline-solutionmooc-planning-dd-static-valistep1-3of4successyesnogpu-production-b-spot-1-030:05:26
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
mistakes0.0
success_ratio1.0
duration38.54601697471739
complexity112.16666666666669
avg_min_distance0.23461567781683448


other stats
nfeasible6
nsamples6
No reset possible
8416616593Bea Baselines 🐤baseline-solutionmooc-planning-dd-dynamic-valistep1-1of4successyesnogpu-production-b-spot-1-030:07:33
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
mistakes1.0
success_ratio1.0
duration43.259052643710525
complexity118.16666666666669
avg_min_distance0.0


other stats
nfeasible6
nsamples6
No reset possible
8414215871Bea Baselines 🐤baseline-solutionmooc-collision-check-teststep1successyesnogpu-production-b-spot-1-030:00:42
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.04
false_positive_rate0.0


other stats
nsamples200
No reset possible
8410515870Bea Baselines 🐤baseline-solutionmooc-collision-check-valistep1successyesnogpu-production-b-spot-1-030:01:50
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
false_negative_rate0.03
false_positive_rate0.0


other stats
nsamples200
No reset possible