Duckietown Challenges Home Challenges Submissions

Evaluator 117

ID117
evaluatoridsc-rudolf-16939
ownerAndrea Censi 🇨🇭
machineidsc-rudolf
processidsc-rudolf-16939
versionDC:3.1.42;DCR:3.2.4;DTS:3.0.29
first heard
last heard
statusinactive
# evaluating
# success17 511
# timeout1 674
# failed3 555
# error5 523
# aborted
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
6741Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationtimeoutnoidsc-rudolf-169390:11:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6681Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:08
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6641Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:55
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6581Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:57
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6521Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:56
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6471Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:52
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6421Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:54
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6371Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6311Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6261Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:00
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6191Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6131Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6091Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:54
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6051Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:47
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
6001Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:05
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
5941Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:00:59
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
5901Andrea Censi 🇨🇭nullaido1_LF1-v3step1-simulationsuccessnoidsc-rudolf-169390:01:53
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation"1.0"
No reset possible
585298Manfred DiazTensorflow templateaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-169390:00:41
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 586, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 581, in wrap_solution
    solution.run(cis)
  File "solution.py", line 61, in run
    solve(params, cis)  # let's try to solve the challenge, exciting ah?
  File "solution.py", line 35, in solve
    action = model.predict(observation)
  File "/workspace/model.py", line 64, in predict
    self._observation: [state],
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 887, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1086, in _run
    str(subfeed_t.get_shape())))
ValueError: Cannot feed value of shape (1, 120, 160, 3) for Tensor u'state:0', which has shape '(1, 480, 640, 3)'
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
574213Andrea Censi 🇨🇭Java templateaido1_amod_service_quality_r1-v3step1-simulationfailednoidsc-rudolf-169390:01:05
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/amod/target/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Uncaught exception in solution:
Traceback (most recent call last):
  File "/aidamod/target/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 485, in wrap_solution
    solution.run(cis)
  File "/project/solution.py", line 25, in run
    subprocess.check_call(cmd, cwd=cwd, stdout=sys.stdout, stderr=sys.stderr)
  File "/usr/lib/python2.7/subprocess.py", line 186, in check_call
    raise CalledProcessError(retcode, cmd)
CalledProcessError: Command '['java', '-Xmx10000m', '-cp', 'aidamod-1.4.7.jar', 'aidamod.demo.AidoGuest', 'aido-host']' returned non-zero exit status 1
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
573144Andrea Censi 🇨🇭Random executionaido1_LF1_r3-v3step4-vizerrornoidsc-rudolf-169390:00:14
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/visualizer/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 530, in wrap_scorer
    evaluator.score(cie)
  File "/visualizer/videos.py", line 12, in score
    logdir = cie.get_completed_step_evaluation_file('step1-simulation', 'episodes')
  File "/visualizer/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 398, in get_completed_step_evaluation_file
    return get_completed_step_evaluation_file(self.root, step_name, basename)
  File "/visualizer/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 427, in get_completed_step_evaluation_file
    available = get_completed_step_evaluation_files(root, step_name)
  File "/visualizer/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 421, in get_completed_step_evaluation_files
    raise InvalidEnvironment(msg)
InvalidEnvironment: Could not find dir /previous-steps/step1-simulation/challenge-evaluation-output
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
567280Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-169390:00:15
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 488, in wrap_evaluator
    evaluator.score(cie)
  File "eval.py", line 96, in score
    raise dc.InvalidEvaluator(msg)
InvalidEvaluator: Gym exited with code 2
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
562275Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-169390:00:26
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 412, in run
    run_docker(wd, project, cmd)
  File "/project/src/duckietown_challenges_runner/runner.py", line 560, in run_docker
    raise DockerComposeFail(msg)
DockerComposeFail: Could not run ['docker-compose', '-p', 'job562-3789', 'up', '--abort-on-container-exit']:

   >  Command '['docker-compose', '-p', 'job562-3789', 'up', '--abort-on-container-exit']' returned non-zero exit status 1

 docker-compose stdout  | None
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
555224Julian ZillyBaseline solution using imitation learning from logsaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-169390:00:47
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 581, in wrap_solution
    solution.run(cis)
  File "solution.py", line 88, in run
    raise InvalidSubmission(str(e))
InvalidSubmission: Can't connect to the gym-duckietown-server
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
525282Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-169390:00:40
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 414, in run
    cr = read_challenge_results(wd)
  File "/project/src/duckietown-challenges/src/duckietown_challenges/challenge_results.py", line 63, in read_challenge_results
    raise Exception(msg)
Exception: File u'/tmp/duckietown/DT18/evaluator/executions/aido1_LF1_r3-v3/submission282/step1-simulation-idsc-rudolf-16939-job525/challenge-results/challenge_results.yaml' does not exist.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
523280Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-169390:00:55
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 414, in run
    cr = read_challenge_results(wd)
  File "/project/src/duckietown-challenges/src/duckietown_challenges/challenge_results.py", line 63, in read_challenge_results
    raise Exception(msg)
Exception: File u'/tmp/duckietown/DT18/evaluator/executions/aido1_LF1_r3-v3/submission280/step1-simulation-idsc-rudolf-16939-job523/challenge-results/challenge_results.yaml' does not exist.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
511269Anton Mashikhin 🇷🇺PyTorch DDPG templateaido1_LF1_r2-v3step2-scoringsuccessyesidsc-rudolf-169390:01:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
traveled_tiles_median2
survival_time_mean3.853333333333327
good_angle_mean5.442991034664965
valid_direction_mean4.213333333333321


other stats
episodes
details{"ep000": {"nsteps": 324, "reward": -4.1844171646890445, "good_angle": 15.73415153219411, "survival_time": 10.799999999999972, "traveled_tiles": 3, "valid_direction": 9.999999999999972}, "ep001": {"nsteps": 47, "reward": -21.796466509712506, "good_angle": 0.10180976303776018, "survival_time": 1.5666666666666682, "traveled_tiles": 1, "valid_direction": 0}, "ep002": {"nsteps": 161, "reward": -7.318896681115159, "good_angle": 5.644748726153944, "survival_time": 1, "traveled_tiles": 2, "valid_direction": 3.699999999999994}, "ep003": {"nsteps": 500, "reward": -1.8001624366045, "good_angle": 3.745197824892488, "survival_time": 0.9666666666666666, "traveled_tiles": 5, "valid_direction": 4.466666666666651}, "ep004": {"nsteps": 148, "reward": -9.42853342922958, "good_angle": 1.9890473270465228, "survival_time": 4.933333333333326, "traveled_tiles": 1, "valid_direction": 2.8999999999999897}}
good_angle_max15.73415153219411
good_angle_median3.745197824892488
good_angle_min0.10180976303776018
reward_max-1.8001624366045
reward_mean-8.905695244270158
reward_median-7.318896681115159
reward_min-21.796466509712506
survival_time_max10.799999999999972
survival_time_median1.5666666666666682
survival_time_min0.9666666666666666
traveled_tiles_max5
traveled_tiles_mean2.4
traveled_tiles_min1
valid_direction_max9.999999999999972
valid_direction_median3.699999999999994
valid_direction_min0
No reset possible