Duckietown Challenges Home Challenges Submissions

Evaluator 234

ID234
evaluatoridsc-rudolf-11443
ownerAndrea Censi 🇨🇭
machineidsc-rudolf
processidsc-rudolf-11443
versionDC:3.1.52;DCR:3.2.12;DTS:3.0.30
first heard
last heard
statusinactive
# evaluating
# success7 2902
# timeout4 2961
# failed5 2897
# error6 2916
# aborted
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
3034337Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-114430:11:00
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3031344Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-114430:00:26
Uncaught exception w [...]
Uncaught exception while running Docker Compose:
Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 526, in run_one
    cr = read_challenge_results(wd)
  File "/project/src/duckietown-challenges/src/duckietown_challenges/challenge_results.py", line 63, in read_challenge_results
    raise Exception(msg)
Exception: File u'/tmp/duckietown/DT18/evaluator/executions/aido1_LF1_r3-v3/submission344/step1-simulation-idsc-rudolf-11443-job3031/challenge-results/challenge_results.yaml' does not exist.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3029346Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-114430:11:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3027348Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-114430:11:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3008305Andrea Censi 🇨🇭Random executionaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-114430:02:19
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 586, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 581, in wrap_solution
    solution.run(cis)
  File "solution.py", line 64, in run
    solve(gym_environment, cis)  # let's try to solve the challenge, exciting ah?
  File "solution.py", line 39, in solve
    observation, reward, done, info = env.step(action)
  File "/usr/local/lib/python2.7/dist-packages/gym/wrappers/time_limit.py", line 31, in step
    observation, reward, done, info = self.env.step(action)
  File "/notebooks/src/gym-duckietown-agent/gym_duckietown_agent/envs/simplesimagent_env.py", line 107, in step
    obs, rew, done, misc = self.sim.step(action, with_observation=True)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 55, in step
    return self._failsafe_observe(msg)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 86, in _failsafe_observe
    raise Exception(msg)
Exception: Giving up to connect to the gym duckietown server at host: evaluator


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
3005336Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step2-scoringsuccessnoidsc-rudolf-114430:00:16
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median3.766666666666663


other stats
episodes
details{"ep000": {"nsteps": 48, "reward": -21.42554907328061, "good_angle": 2.8207892741930283, "survival_time": 1.6000000000000016, "traveled_tiles": 3, "valid_direction": 1.2000000000000015}, "ep001": {"nsteps": 113, "reward": -9.126021487457034, "good_angle": 0.4336047256271916, "survival_time": 3.766666666666663, "traveled_tiles": 2, "valid_direction": 0.6333333333333311}, "ep002": {"nsteps": 243, "reward": -4.33631397868864, "good_angle": 0.5664875769269541, "survival_time": 8.099999999999982, "traveled_tiles": 5, "valid_direction": 0.5999999999999999}, "ep003": {"nsteps": 39, "reward": -27.6859427125026, "good_angle": 0.3331729932997127, "survival_time": 1.3000000000000007, "traveled_tiles": 1, "valid_direction": 0.8333333333333334}, "ep004": {"nsteps": 144, "reward": -7.162975964742752, "good_angle": 0.3325536370126206, "survival_time": 4.799999999999993, "traveled_tiles": 4, "valid_direction": 0.9999999999999992}}
good_angle_max2.8207892741930283
good_angle_mean0.8973216414119014
good_angle_median0.4336047256271916
good_angle_min0.3325536370126206
reward_max-4.33631397868864
reward_mean-13.947360643334326
reward_median-9.126021487457034
reward_min-27.6859427125026
survival_time_max8.099999999999982
survival_time_mean3.913333333333328
survival_time_min1.3000000000000007
traveled_tiles_max5
traveled_tiles_mean3
traveled_tiles_median3
traveled_tiles_min1
valid_direction_max1.2000000000000015
valid_direction_mean0.8533333333333332
valid_direction_median0.8333333333333334
valid_direction_min0.5999999999999999
No reset possible
2992336Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step1-simulationsuccessnoidsc-rudolf-114430:02:39
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
2990320Andrea Censi 🇨🇭PyTorch DDPG templateaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-114430:00:23
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 586, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 581, in wrap_solution
    solution.run(cis)
  File "solution.py", line 87, in run
    solve(params, cis)
  File "solution.py", line 60, in solve
    action = model.predict(observation)
  File "/workspace/model.py", line 176, in predict
    return self.actor(state).cpu().data.numpy().flatten()
  File "/opt/conda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/workspace/model.py", line 72, in forward
    x = self.lr(self.lin1(x))
  File "/opt/conda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 477, in __call__
    result = self.forward(*input, **kwargs)
  File "/opt/conda/lib/python2.7/site-packages/torch/nn/modules/linear.py", line 55, in forward
    return F.linear(input, self.weight, self.bias)
  File "/opt/conda/lib/python2.7/site-packages/torch/nn/functional.py", line 1024, in linear
    return torch.addmm(bias, input, weight.t())
RuntimeError: size mismatch, m1: [1 x 127872], m2: [4032 x 512] at /tmp/pip-req-build-vRRdPa/aten/src/TH/generic/THTensorMath.cpp:2070


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2980349Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step4-vizsuccessnoidsc-rudolf-114430:01:16
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2975348Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-114430:00:34
InvalidEnvironment: [...]
InvalidEnvironment:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 482, in wrap_evaluator
    raise InvalidEnvironment(out[SPECIAL_INVALID_ENVIRONMENT])
InvalidEnvironment: InvalidEnvironment:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 585, in wrap_solution
    solution.run(cis)
  File "solution.py", line 117, in run
    solve(params, cis)
  File "solution.py", line 74, in solve
    check_valid_observations(observation)
  File "solution.py", line 21, in check_valid_observations
    raise InvalidEnvironment(msg)
InvalidEnvironment: I expected size (3, 480, 640), while I got size (3, 120, 160)

Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2971347Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-114430:10:31
Timeout: Waited 602 [...]
Timeout:

Waited 602.973622084 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2966346Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationsuccessnoidsc-rudolf-114430:02:40
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
2965345Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-114430:00:41
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 590, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 585, in wrap_solution
    solution.run(cis)
  File "solution.py", line 116, in run
    solve(params, cis)
  File "solution.py", line 65, in solve
    model.load("model", "models")
  File "/workspace/model.py", line 229, in load
    self.actor.load_state_dict(torch.load('{}/{}_actor.pth'.format(directory, filename), map_location=device))
  File "/opt/conda/lib/python2.7/site-packages/torch/nn/modules/module.py", line 719, in load_state_dict
    self.__class__.__name__, "\n\t".join(error_msgs)))
RuntimeError: Error(s) in loading state_dict for ActorCNN:
	size mismatch for lin1.weight: copying a param of torch.Size([512, 127872]) from checkpoint, where the shape is torch.Size([512, 4032]) in current model.


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2961343Bhairav MehtaPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-114430:11:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2948195Florian GolemoPyTorch DDPG templateaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-114430:10:28
Timeout: Waited 603 [...]
Timeout:

Waited 603.209515095 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2944219Bhairav MehtaROS-based Lane Followingaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-114430:00:25
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Env Duckietown-Lf-Lfv-Navv-Silent-v1 not found (valid versions include ['Duckietown-Lf-Lfv-Navv-Silent-v0'])
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2925280Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-114430:10:27
Timeout: Waited 602 [...]
Timeout:

Waited 602.973664999 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2916282Anton Mashikhin 🇷🇺AI DL RL MML XXXL 2k18 yooaido1_LF1_r3-v3step1-simulationerrornoidsc-rudolf-114430:10:29
Timeout: Waited 602 [...]
Timeout:

Waited 602.914832115 for container to finish. Giving up. 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
2915334Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step4-vizsuccessnoidsc-rudolf-114430:01:08
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2912334Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step3-videossuccessnoidsc-rudolf-114430:00:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
2902334Andrea Censi 🇨🇭Tensorflow templateaido1_LFV_r1-v3step2-scoringsuccessnoidsc-rudolf-114430:01:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median3.766666666666663


other stats
episodes
details{"ep000": {"nsteps": 48, "reward": -21.42554907328061, "good_angle": 2.8207892741930283, "survival_time": 1.6000000000000016, "traveled_tiles": 3, "valid_direction": 1.2000000000000015}, "ep001": {"nsteps": 113, "reward": -9.126021487457034, "good_angle": 0.4336047256271916, "survival_time": 3.766666666666663, "traveled_tiles": 2, "valid_direction": 0.6333333333333311}, "ep002": {"nsteps": 243, "reward": -4.33631397868864, "good_angle": 0.5664875769269541, "survival_time": 8.099999999999982, "traveled_tiles": 5, "valid_direction": 0.5999999999999999}, "ep003": {"nsteps": 41, "reward": -26.565000374142716, "good_angle": 0.35119644914517983, "survival_time": 1.3666666666666676, "traveled_tiles": 1, "valid_direction": 1.0000000000000004}, "ep004": {"nsteps": 144, "reward": -7.17783817061637, "good_angle": 0.320953404745754, "survival_time": 4.799999999999993, "traveled_tiles": 4, "valid_direction": 1.0333333333333328}}
good_angle_max2.8207892741930283
good_angle_mean0.8986062861276215
good_angle_median0.4336047256271916
good_angle_min0.320953404745754
reward_max-4.33631397868864
reward_mean-13.726144616837072
reward_median-9.126021487457034
reward_min-26.565000374142716
survival_time_max8.099999999999982
survival_time_mean3.9266666666666614
survival_time_min1.3666666666666676
traveled_tiles_max5
traveled_tiles_mean3
traveled_tiles_median3
traveled_tiles_min1
valid_direction_max1.2000000000000015
valid_direction_mean0.8933333333333332
valid_direction_median1.0000000000000004
valid_direction_min0.5999999999999999
No reset possible
2897332Andrea Censi 🇨🇭Random executionaido1_LFV_r1-v3step1-simulationfailednoidsc-rudolf-114430:00:40
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 586, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 581, in wrap_solution
    solution.run(cis)
  File "solution.py", line 73, in run
    solve(gym_environment, cis)  # let's try to solve the challenge, exciting ah?
  File "solution.py", line 46, in solve
    observation, reward, done, info = env.step(action)
  File "/usr/local/lib/python2.7/dist-packages/gym/wrappers/time_limit.py", line 31, in step
    observation, reward, done, info = self.env.step(action)
  File "/notebooks/src/gym-duckietown-agent/gym_duckietown_agent/envs/simplesimagent_env.py", line 109, in step
    obs, rew, done, misc = self.sim.step(action, with_observation=True)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 55, in step
    return self._failsafe_observe(msg)
  File "/notebooks/src/duckietown-slimremote/duckietown_slimremote/pc/robot.py", line 86, in _failsafe_observe
    raise Exception(msg)
Exception: Giving up to connect to the gym duckietown server at host: evaluator


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible