Duckietown Challenges Home Challenges Submissions

Evaluator 324

ID324
evaluatoridsc-rudolf-4185
ownerAndrea Censi 🇨🇭
machineidsc-rudolf
processidsc-rudolf-4185
versionDC:3.1.56;DCR:3.2.14;DTS:3.0.30
first heard
last heard
statusinactive
# evaluating
# success16 6000
# timeout5 6178
# failed4 6083
# error
# aborted4 5973
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job IDsubmissionuseruser labelchallengestepstatusup to dateevaluatordate starteddate completeddurationmessage
6206829Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-41850:33:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6199833Orlando Marquez 🇨🇦Baseline solution using imitation learning from logsaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-41850:39:30
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6195838Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step2-scoringtimeoutyesidsc-rudolf-41850:46:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6180832Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-41850:33:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6178833Orlando Marquez 🇨🇦Baseline solution using imitation learning from logsaido1_LF1_r3-v3step1-simulationtimeoutnoidsc-rudolf-41850:33:16
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6173838Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step2-scoringsuccessyesidsc-rudolf-41850:01:11
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median1.8000000000000025


other stats
episodes
details{"ep000": {"nsteps": 62, "reward": -16.476209455477676, "good_angle": 0.5005522135713053, "survival_time": 2.066666666666669, "traveled_tiles": 1, "valid_direction": 1.3666666666666691}, "ep001": {"nsteps": 37, "reward": -27.530120182681728, "good_angle": 0.09259917590374704, "survival_time": 1.2333333333333338, "traveled_tiles": 1, "valid_direction": 0.23333333333333328}, "ep002": {"nsteps": 52, "reward": -19.591462997862926, "good_angle": 0.16945723347049738, "survival_time": 1.7333333333333354, "traveled_tiles": 2, "valid_direction": 0.7333333333333352}, "ep003": {"nsteps": 67, "reward": -15.205426388378465, "good_angle": 0.3164877561361908, "survival_time": 2.233333333333335, "traveled_tiles": 1, "valid_direction": 1.3666666666666687}, "ep004": {"nsteps": 54, "reward": -18.903807604202516, "good_angle": 0.13643665976135383, "survival_time": 1.8000000000000025, "traveled_tiles": 1, "valid_direction": 0.533333333333335}}
good_angle_max0.5005522135713053
good_angle_mean0.2431066077686189
good_angle_median0.16945723347049738
good_angle_min0.09259917590374704
reward_max-15.205426388378465
reward_mean-19.54140532572066
reward_median-18.903807604202516
reward_min-27.530120182681728
survival_time_max2.233333333333335
survival_time_mean1.813333333333335
survival_time_min1.2333333333333338
traveled_tiles_max2
traveled_tiles_mean1.2
traveled_tiles_median1
traveled_tiles_min1
valid_direction_max1.3666666666666691
valid_direction_mean0.8466666666666682
valid_direction_median0.7333333333333352
valid_direction_min0.23333333333333328
No reset possible
6165836Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step1-simulationabortednoidsc-rudolf-41850:00:28
The result file is n [...]
The result file is not found. This usually means that the evaluator did not finish
and some times that there was an import error.

Check the evaluator log to see what happened.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6156839Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step1-simulationsuccessnoidsc-rudolf-41850:01:22
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
6148839Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step2-scoringsuccessnoidsc-rudolf-41850:02:08
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median6.899999999999985


other stats
episodes
details{"ep000": {"nsteps": 164, "reward": -6.705390841313979, "good_angle": 4.762022619145479, "survival_time": 5.466666666666657, "traveled_tiles": 1, "valid_direction": 4.5666666666666575}, "ep001": {"nsteps": 207, "reward": -5.820953307106443, "good_angle": 10.028977853210447, "survival_time": 6.899999999999985, "traveled_tiles": 1, "valid_direction": 5.999999999999986}, "ep002": {"nsteps": 90, "reward": -11.26712071566532, "good_angle": 0.855300465475497, "survival_time": 2.999999999999999, "traveled_tiles": 2, "valid_direction": 2.0333333333333323}, "ep003": {"nsteps": 330, "reward": -3.767540347289336, "good_angle": 7.093201338622533, "survival_time": 10.999999999999972, "traveled_tiles": 1, "valid_direction": 8.13333333333331}, "ep004": {"nsteps": 302, "reward": -4.224530833164255, "good_angle": 6.723103235107182, "survival_time": 10.066666666666642, "traveled_tiles": 1, "valid_direction": 7.366666666666647}}
good_angle_max10.028977853210447
good_angle_mean5.8925211023122275
good_angle_median6.723103235107182
good_angle_min0.855300465475497
reward_max-3.767540347289336
reward_mean-6.357107208907867
reward_median-5.820953307106443
reward_min-11.26712071566532
survival_time_max10.999999999999972
survival_time_mean7.28666666666665
survival_time_min2.999999999999999
traveled_tiles_max2
traveled_tiles_mean1.2
traveled_tiles_median1
traveled_tiles_min1
valid_direction_max8.13333333333331
valid_direction_mean5.619999999999986
valid_direction_median5.999999999999986
valid_direction_min2.0333333333333323
No reset possible
6144838Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step4-vizsuccessnoidsc-rudolf-41850:01:02
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
6141838Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step2-scoringsuccessnoidsc-rudolf-41850:00:20
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median5.966666666666655


other stats
episodes
details{"ep000": {"nsteps": 148, "reward": -7.326902210097713, "good_angle": 3.8513875373206266, "survival_time": 4.933333333333326, "traveled_tiles": 1, "valid_direction": 3.999999999999992}, "ep001": {"nsteps": 179, "reward": -6.233594004814019, "good_angle": 8.24345295673754, "survival_time": 5.966666666666655, "traveled_tiles": 1, "valid_direction": 5.199999999999989}, "ep002": {"nsteps": 88, "reward": -11.529293883570723, "good_angle": 0.7052056161783535, "survival_time": 2.9333333333333327, "traveled_tiles": 2, "valid_direction": 1.8}, "ep003": {"nsteps": 327, "reward": -3.7954779892493415, "good_angle": 7.017368372635797, "survival_time": 10.899999999999972, "traveled_tiles": 1, "valid_direction": 7.933333333333311}, "ep004": {"nsteps": 309, "reward": -4.145339975521212, "good_angle": 6.583917946858259, "survival_time": 10.299999999999974, "traveled_tiles": 1, "valid_direction": 7.59999999999998}}
good_angle_max8.24345295673754
good_angle_mean5.280266485946115
good_angle_median6.583917946858259
good_angle_min0.7052056161783535
reward_max-3.7954779892493415
reward_mean-6.606121612650601
reward_median-6.233594004814019
reward_min-11.529293883570723
survival_time_max10.899999999999972
survival_time_mean7.006666666666652
survival_time_min2.9333333333333327
traveled_tiles_max2
traveled_tiles_mean1.2
traveled_tiles_median1
traveled_tiles_min1
valid_direction_max7.933333333333311
valid_direction_mean5.306666666666655
valid_direction_median5.199999999999989
valid_direction_min1.8
No reset possible
6137838Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step1-simulationsuccessnoidsc-rudolf-41850:02:38
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
6135836Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step1-simulationabortednoidsc-rudolf-41850:00:32
The result file is n [...]
The result file is not found. This usually means that the evaluator did not finish
and some times that there was an import error.

Check the evaluator log to see what happened.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6133835Mikita Sazanovich 🇷🇺RL solutionaido1_LF1_r3-v3step1-simulationabortednoidsc-rudolf-41850:00:33
The result file is n [...]
The result file is not found. This usually means that the evaluator did not finish
and some times that there was an import error.

Check the evaluator log to see what happened.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6125834Samuel LavoieYoung Dukeaido1_LF1_r3-v3step3-videossuccessnoidsc-rudolf-41850:01:14
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
6119833Orlando Marquez 🇨🇦Baseline solution using imitation learning from logsaido1_LF1_r3-v3step4-vizsuccessnoidsc-rudolf-41850:01:19
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
6112833Orlando Marquez 🇨🇦Baseline solution using imitation learning from logsaido1_LF1_r3-v3step1-simulationsuccessnoidsc-rudolf-41850:02:48
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
6108830Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-41850:10:34
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 477, in wrap_evaluator
    cie.wait_for_solution()
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 270, in wait_for_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Time out: Timeout of 600 while waiting for /challenge-solution-output/output-solution.yaml.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6095827Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-41850:10:34
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 477, in wrap_evaluator
    cie.wait_for_solution()
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 270, in wait_for_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Time out: Timeout of 600 while waiting for /challenge-solution-output/output-solution.yaml.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6094826Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-41850:10:32
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 477, in wrap_evaluator
    cie.wait_for_solution()
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 270, in wait_for_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Time out: Timeout of 600 while waiting for /challenge-solution-output/output-solution.yaml.
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6083822Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step1-simulationfailednoidsc-rudolf-41850:00:34
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 590, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 585, in wrap_solution
    solution.run(cis)
  File "solution.py", line 117, in run
    solve(params, cis)
  File "solution.py", line 64, in solve
    model = Model(config_name=config_name, config=config)
  File "/workspace/model.py", line 15, in __init__
    self.init_model(config_name)
  File "/workspace/model.py", line 45, in init_model
    map_location=self.device))
  File "/opt/conda/lib/python2.7/site-packages/torch/serialization.py", line 356, in load
    f = open(f, 'rb')
IOError: [Errno 2] No such file or directory: '/workspace/models/weights/mml2_d_28_178000.pth.pth'


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
6074821Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step3-videossuccessnoidsc-rudolf-41850:02:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
6069821Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step2-scoringsuccessnoidsc-rudolf-41850:00:21
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median8.566666666666647


other stats
episodes
details{"ep000": {"nsteps": 257, "reward": -4.378372870136509, "good_angle": 23.922353620219457, "survival_time": 8.566666666666647, "traveled_tiles": 4, "valid_direction": 3.7999999999999936}, "ep001": {"nsteps": 339, "reward": -3.4165047393585652, "good_angle": 29.73258777467919, "survival_time": 11.29999999999997, "traveled_tiles": 6, "valid_direction": 4.66666666666665}, "ep002": {"nsteps": 99, "reward": -10.13124891556329, "good_angle": 0.07467375246629761, "survival_time": 3.299999999999998, "traveled_tiles": 3, "valid_direction": 0}, "ep003": {"nsteps": 470, "reward": -2.4095897665413273, "good_angle": 15.688016218271525, "survival_time": 15.666666666666622, "traveled_tiles": 8, "valid_direction": 5.59999999999998}, "ep004": {"nsteps": 157, "reward": -7.765255263526064, "good_angle": 1.1790963587830048, "survival_time": 5.2333333333333245, "traveled_tiles": 4, "valid_direction": 1.8999999999999932}}
good_angle_max29.73258777467919
good_angle_mean14.119345544883894
good_angle_median15.688016218271525
good_angle_min0.07467375246629761
reward_max-2.4095897665413273
reward_mean-5.620194311025151
reward_median-4.378372870136509
reward_min-10.13124891556329
survival_time_max15.666666666666622
survival_time_mean8.813333333333313
survival_time_min3.299999999999998
traveled_tiles_max8
traveled_tiles_mean5
traveled_tiles_median4
traveled_tiles_min3
valid_direction_max5.59999999999998
valid_direction_mean3.1933333333333236
valid_direction_median3.7999999999999936
valid_direction_min0
No reset possible
6043814Benjamin Ramtoula 🇨🇦My modified ROS-based Lane Followingaido1_LF1_r3-v3step1-simulationsuccessnoidsc-rudolf-41850:02:32
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
6036813Benjamin Ramtoula 🇨🇦My modified ROS-based Lane Followingaido1_LF1_r3-v3step4-vizsuccessnoidsc-rudolf-41850:01:29
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
6031813Benjamin Ramtoula 🇨🇦My modified ROS-based Lane Followingaido1_LF1_r3-v3step3-videossuccessnoidsc-rudolf-41850:01:54
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
6027813Benjamin Ramtoula 🇨🇦My modified ROS-based Lane Followingaido1_LF1_r3-v3step1-simulationsuccessnoidsc-rudolf-41850:06:04
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
6000808Samuel LavoiePyTorch DDPG templateaido1_LF1_r3-v3step4-vizsuccessnoidsc-rudolf-41850:00:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5973805Anton Mashikhin 🇷🇺SAIC MOSCOW MMLaido1_LF1_r3-v3step1-simulationabortednoidsc-rudolf-41850:00:13
Error while running [...]

Creating job5973-7962_evaluator_1 ... error
stderr | ERROR: for job5973-7962_evaluator_1  Cannot start service evaluator: network job5973-7962_evaluation not found
stderr | 
stderr | ERROR: for solution  Cannot start service solution: network job5973-7962_evaluation not found
stderr | 
stderr | ERROR: for evaluator  Cannot start service evaluator: network job5973-7962_evaluation not found
stderr | Encountered errors while bringing up the project.
stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible