Duckietown Challenges Home Challenges Submissions

Evaluator 294

ID294
evaluatorrudolf-3
ownerAndrea Censi ðŸ‡¨ðŸ‡­
machineidsc-rudolf
processrudolf-3
versionDC:3.1.54;DCR:3.2.14;DTS:3.0.30
first heard
last heard
statusinactive
# evaluating
# success21 5749
# timeout
# failed1 5780
# error
# aborted7 5790
# host-error
arm
x86_64
Mac
gpu available
Number of processors
Processor frequency (MHz)
Free % of processors
RAM total (MB)
RAM free (MB)
Disk (MB)
Disk available (MB)
Docker Hub
P1
P2
Cloud simulations
PI Camera
# Duckiebots
Map 3x3 avaiable
Number of duckies
gpu cores
AIDO 2 Map LF public
AIDO 2 Map LF private
AIDO 2 Map LFV public
AIDO 2 Map LFV private
AIDO 2 Map LFVI public
AIDO 2 Map LFVI private
AIDO 3 Map LF public
AIDO 3 Map LF private
AIDO 3 Map LFV public
AIDO 3 Map LFV private
AIDO 3 Map LFVI public
AIDO 3 Map LFVI private
AIDO 5 Map large loop
ETU track
for 2021, map is ETH_small_inter
IPFS mountpoint /ipfs available
IPNS mountpoint /ipns available

Evaluator jobs

Job ID
submission
user
user label
challenge
step
status
up to date
evaluator
date started
date completed
duration
message
5947799Julian ZillyBaseline solution using imitation learning from logsaido1_LF1_r3-v3step4-vizabortednorudolf-30:01:48
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5945799Julian ZillyBaseline solution using imitation learning from logsaido1_LF1_r3-v3step3-videosabortednorudolf-30:00:53
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5941786Marta Tintore ðŸ‡¨ðŸ‡­AMOD18-AIDO not that random executionaido1_LF1_r3-v3step4-vizsuccessnorudolf-30:01:09
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5936786Marta Tintore ðŸ‡¨ðŸ‡­AMOD18-AIDO not that random executionaido1_LF1_r3-v3step2-scoringsuccessnorudolf-30:00:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median9.033333333333312


other stats
episodes
details{"ep000": {"nsteps": 170, "reward": -7.790757534083198, "good_angle": 24.085888249464556, "survival_time": 5.666666666666656, "traveled_tiles": 2, "valid_direction": 3.899999999999987}, "ep001": {"nsteps": 376, "reward": -3.5683142356970845, "good_angle": 22.87213440078576, "survival_time": 12.5333333333333, "traveled_tiles": 3, "valid_direction": 3.899999999999986}, "ep002": {"nsteps": 318, "reward": -3.1791894810119388, "good_angle": 0.13315919092022602, "survival_time": 10.599999999999971, "traveled_tiles": 4, "valid_direction": 0}, "ep003": {"nsteps": 258, "reward": -6.154849209064661, "good_angle": 26.1922502546216, "survival_time": 8.59999999999998, "traveled_tiles": 3, "valid_direction": 3.93333333333332}, "ep004": {"nsteps": 271, "reward": -5.561203524712611, "good_angle": 5.142374861417086, "survival_time": 9.033333333333312, "traveled_tiles": 3, "valid_direction": 3.033333333333323}}
good_angle_max26.1922502546216
good_angle_mean15.685161391441843
good_angle_median22.87213440078576
good_angle_min0.13315919092022602
reward_max-3.1791894810119388
reward_mean-5.250862796913898
reward_median-5.561203524712611
reward_min-7.790757534083198
survival_time_max12.5333333333333
survival_time_mean9.286666666666644
survival_time_min5.666666666666656
traveled_tiles_max4
traveled_tiles_mean3
traveled_tiles_median3
traveled_tiles_min2
valid_direction_max3.93333333333332
valid_direction_mean2.953333333333324
valid_direction_median3.899999999999986
valid_direction_min0
No reset possible
5923799Julian ZillyBaseline solution using imitation learning from logsaido1_LF1_r3-v3step1-simulationabortednorudolf-30:02:45
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
5914798Luca BonaminiRandom executionaido1_LF1_r3-v3step1-simulationsuccessnorudolf-30:02:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
5910795Andrea Censi ðŸ‡¨ðŸ‡­Random executionaido1_LF1_r3-v3step4-vizsuccessnorudolf-30:01:13
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5904795Andrea Censi ðŸ‡¨ðŸ‡­Random executionaido1_LF1_r3-v3step2-scoringsuccessnorudolf-30:02:03
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median9.066666666666643


other stats
episodes
details{"ep000": {"nsteps": 170, "reward": -7.919950384252211, "good_angle": 24.718891823637797, "survival_time": 5.666666666666656, "traveled_tiles": 2, "valid_direction": 3.933333333333321}, "ep001": {"nsteps": 380, "reward": -4.437672361104112, "good_angle": 26.30556823220511, "survival_time": 12.666666666666632, "traveled_tiles": 3, "valid_direction": 3.966666666666653}, "ep002": {"nsteps": 427, "reward": -2.442277737524152, "good_angle": 0.10019474324583716, "survival_time": 14.233333333333292, "traveled_tiles": 5, "valid_direction": 0}, "ep003": {"nsteps": 258, "reward": -6.166539188041243, "good_angle": 25.99665214275984, "survival_time": 8.59999999999998, "traveled_tiles": 3, "valid_direction": 3.899999999999987}, "ep004": {"nsteps": 272, "reward": -5.383548286089482, "good_angle": 4.178591015961236, "survival_time": 9.066666666666643, "traveled_tiles": 3, "valid_direction": 3.1666666666666563}}
good_angle_max26.30556823220511
good_angle_mean16.259979591561965
good_angle_median24.718891823637797
good_angle_min0.10019474324583716
reward_max-2.442277737524152
reward_mean-5.26999759140224
reward_median-5.383548286089482
reward_min-7.919950384252211
survival_time_max14.233333333333292
survival_time_mean10.046666666666642
survival_time_min5.666666666666656
traveled_tiles_max5
traveled_tiles_mean3.2
traveled_tiles_median3
traveled_tiles_min2
valid_direction_max3.966666666666653
valid_direction_mean2.993333333333324
valid_direction_median3.899999999999987
valid_direction_min0
No reset possible
5894794Manish PrajapatAMOD18-AIDOaido1_LF1_r3-v3step1-simulationsuccessnorudolf-30:02:06
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
5882789Mikita Sazanovich ðŸ‡·ðŸ‡ºROS-based Lane Followingaido1_LF1_r3-v3step3-videossuccessnorudolf-30:02:00
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5878789Mikita Sazanovich ðŸ‡·ðŸ‡ºROS-based Lane Followingaido1_LF1_r3-v3step2-scoringsuccessnorudolf-30:00:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median16.666666666666654


other stats
episodes
details{"ep000": {"nsteps": 147, "reward": -7.483778158215438, "good_angle": 11.5501667838303, "survival_time": 4.899999999999992, "traveled_tiles": 3, "valid_direction": 2.866666666666658}, "ep001": {"nsteps": 500, "reward": -0.32564113597408867, "good_angle": 18.040583723581097, "survival_time": 16.666666666666654, "traveled_tiles": 4, "valid_direction": 3.1666666666666607}, "ep002": {"nsteps": 500, "reward": -0.3706761087477207, "good_angle": 0.04195710814195461, "survival_time": 16.666666666666654, "traveled_tiles": 4, "valid_direction": 0}, "ep003": {"nsteps": 500, "reward": -0.3631334312947947, "good_angle": 18.964130352810113, "survival_time": 16.666666666666654, "traveled_tiles": 5, "valid_direction": 3.09999999999999}, "ep004": {"nsteps": 500, "reward": -0.7984821920012182, "good_angle": 2.844816453757069, "survival_time": 16.666666666666654, "traveled_tiles": 5, "valid_direction": 6.266666666666653}}
good_angle_max18.964130352810113
good_angle_mean10.288330884424106
good_angle_median11.5501667838303
good_angle_min0.04195710814195461
reward_max-0.32564113597408867
reward_mean-1.8683422052466525
reward_median-0.3706761087477207
reward_min-7.483778158215438
survival_time_max16.666666666666654
survival_time_mean14.313333333333322
survival_time_min4.899999999999992
traveled_tiles_max5
traveled_tiles_mean4.2
traveled_tiles_median4
traveled_tiles_min3
valid_direction_max6.266666666666653
valid_direction_mean3.079999999999992
valid_direction_median3.09999999999999
valid_direction_min0
No reset possible
5859786Marta Tintore ðŸ‡¨ðŸ‡­AMOD18-AIDO not that random executionaido1_LF1_r3-v3step3-videosabortednorudolf-30:00:50
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5849786Marta Tintore ðŸ‡¨ðŸ‡­AMOD18-AIDO not that random executionaido1_LF1_r3-v3step1-simulationabortednorudolf-30:02:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
5846781Aneri MuniAMOD18-AIDO not that random executionaido1_LF1_r3-v3step4-vizsuccessnorudolf-30:01:00
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5840781Aneri MuniAMOD18-AIDO not that random executionaido1_LF1_r3-v3step2-scoringsuccessnorudolf-30:00:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median5.833333333333322


other stats
episodes
details{"ep000": {"nsteps": 144, "reward": -9.06813865651687, "good_angle": 20.52817005800521, "survival_time": 4.799999999999993, "traveled_tiles": 2, "valid_direction": 3.3666666666666583}, "ep001": {"nsteps": 175, "reward": -7.296941792624337, "good_angle": 0.7101056477194247, "survival_time": 5.833333333333322, "traveled_tiles": 2, "valid_direction": 2.299999999999992}, "ep002": {"nsteps": 475, "reward": -2.0803148428691354, "good_angle": 1.457937384196954, "survival_time": 15.833333333333288, "traveled_tiles": 5, "valid_direction": 1.86666666666666}, "ep003": {"nsteps": 231, "reward": -6.718039963152502, "good_angle": 24.418321537832004, "survival_time": 7.699999999999982, "traveled_tiles": 3, "valid_direction": 3.533333333333321}, "ep004": {"nsteps": 123, "reward": -10.564881026260252, "good_angle": 0.2059180554482576, "survival_time": 4.099999999999995, "traveled_tiles": 2, "valid_direction": 0.93333333333333}}
good_angle_max24.418321537832004
good_angle_mean9.464090536640368
good_angle_median1.457937384196954
good_angle_min0.2059180554482576
reward_max-2.0803148428691354
reward_mean-7.145663256284619
reward_median-7.296941792624337
reward_min-10.564881026260252
survival_time_max15.833333333333288
survival_time_mean7.6533333333333164
survival_time_min4.099999999999995
traveled_tiles_max5
traveled_tiles_mean2.8
traveled_tiles_median2
traveled_tiles_min2
valid_direction_max3.533333333333321
valid_direction_mean2.3999999999999924
valid_direction_median2.299999999999992
valid_direction_min0.93333333333333
No reset possible
5829780Amaury Camus ðŸ‡¨ðŸ‡­AMOD18-AIDO not that random executionaido1_LF1_r3-v3step3-videossuccessnorudolf-30:00:37
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5817778Alessandro MorraAMOD18-AIDO not that random executionaido1_LF1_r3-v3step2-scoringsuccessnorudolf-30:00:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median4.266666666666661


other stats
episodes
details{"ep000": {"nsteps": 81, "reward": -14.283126489615734, "good_angle": 11.54529418232562, "survival_time": 2.7, "traveled_tiles": 2, "valid_direction": 1.9}, "ep001": {"nsteps": 178, "reward": -6.58459713209546, "good_angle": 11.172940062450415, "survival_time": 5.933333333333322, "traveled_tiles": 3, "valid_direction": 1.86666666666666}, "ep002": {"nsteps": 130, "reward": -7.712395287763614, "good_angle": 0.0776753162616688, "survival_time": 4.333333333333328, "traveled_tiles": 3, "valid_direction": 0}, "ep003": {"nsteps": 122, "reward": -10.274274486987316, "good_angle": 12.420306624567775, "survival_time": 4.066666666666662, "traveled_tiles": 3, "valid_direction": 1.86666666666666}, "ep004": {"nsteps": 128, "reward": -9.712881646148162, "good_angle": 2.503320872424451, "survival_time": 4.266666666666661, "traveled_tiles": 3, "valid_direction": 1.4333333333333282}}
good_angle_max12.420306624567775
good_angle_mean7.543907411605986
good_angle_median11.172940062450415
good_angle_min0.0776753162616688
reward_max-6.58459713209546
reward_mean-9.713455008522056
reward_median-9.712881646148162
reward_min-14.283126489615734
survival_time_max5.933333333333322
survival_time_mean4.259999999999995
survival_time_min2.7
traveled_tiles_max3
traveled_tiles_mean2.8
traveled_tiles_median3
traveled_tiles_min2
valid_direction_max1.9
valid_direction_mean1.4133333333333298
valid_direction_median1.86666666666666
valid_direction_min0
No reset possible
5812776Alessandro MorraAMOD18-AIDO not that random executionaido1_LF1_r3-v3step3-videossuccessnorudolf-30:00:41
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5804777Yannick BerdouAMOD18-AIDO not that random executionaido1_LF1_r3-v3step2-scoringsuccessnorudolf-30:00:17
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median3.166666666666665


other stats
episodes
details{"ep000": {"nsteps": 60, "reward": -18.55895467996597, "good_angle": 8.631361811097454, "survival_time": 2.0000000000000027, "traveled_tiles": 2, "valid_direction": 1.4000000000000028}, "ep001": {"nsteps": 132, "reward": -8.560128116697976, "good_angle": 8.371777309085797, "survival_time": 4.399999999999994, "traveled_tiles": 3, "valid_direction": 1.399999999999995}, "ep002": {"nsteps": 140, "reward": -7.20356833904516, "good_angle": 0.034675126335285154, "survival_time": 4.66666666666666, "traveled_tiles": 4, "valid_direction": 0}, "ep003": {"nsteps": 91, "reward": -13.248195889231924, "good_angle": 9.188896207380877, "survival_time": 3.0333333333333323, "traveled_tiles": 3, "valid_direction": 1.3999999999999972}, "ep004": {"nsteps": 95, "reward": -12.396050412678404, "good_angle": 1.806465789696306, "survival_time": 3.166666666666665, "traveled_tiles": 3, "valid_direction": 1.0666666666666629}}
good_angle_max9.188896207380877
good_angle_mean5.606635248719142
good_angle_median8.371777309085797
good_angle_min0.034675126335285154
reward_max-7.20356833904516
reward_mean-11.993379487523889
reward_median-12.396050412678404
reward_min-18.55895467996597
survival_time_max4.66666666666666
survival_time_mean3.4533333333333305
survival_time_min2.0000000000000027
traveled_tiles_max4
traveled_tiles_mean3
traveled_tiles_median3
traveled_tiles_min2
valid_direction_max1.4000000000000028
valid_direction_mean1.0533333333333317
valid_direction_median1.399999999999995
valid_direction_min0
No reset possible
5796773Amaury Camus ðŸ‡¨ðŸ‡­AMOD18-AIDO not that random executionaido1_LF1_r3-v3step4-vizabortednorudolf-30:00:52
Error while running [...]

Creating job5796-6102_videos_1 ... error
stderr | ERROR: for job5796-6102_videos_1  Cannot start service videos: network job5796-6102_evaluation not found
stderr | 
stderr | ERROR: for videos  Cannot start service videos: network job5796-6102_evaluation not found
stderr | Encountered errors while bringing up the project.
stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5792774Gianmarco BernasconiNot Random executionaido1_LF1_r3-v3step3-videossuccessnorudolf-30:00:46
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5790774Gianmarco BernasconiNot Random executionaido1_LF1_r3-v3step2-scoringabortednorudolf-30:00:12
Error while running [...]

Creating job5790-2378_scorer_1 ... error
stderr | ERROR: for job5790-2378_scorer_1  Cannot start service scorer: network job5790-2378_evaluation not found
stderr | 
stderr | ERROR: for scorer  Cannot start service scorer: network job5790-2378_evaluation not found
stderr | Encountered errors while bringing up the project.
stderr | 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5784774Gianmarco BernasconiNot Random executionaido1_LF1_r3-v3step1-simulationsuccessnorudolf-30:02:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
5780771Maximilien PicquetAMOD18-AIDO not that random executionaido1_LF1_r3-v3step1-simulationfailednorudolf-30:00:38
InvalidSubmission: T [...]
InvalidSubmission:
Traceback (most recent call last):
  File "/workspace/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 486, in wrap_evaluator
    raise InvalidSubmission(out[SPECIAL_INVALID_SUBMISSION])
InvalidSubmission: Invalid solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 590, in wrap_solution
    raise InvalidSubmission(msg)
InvalidSubmission: Uncaught exception in solution:
Traceback (most recent call last):
  File "/notebooks/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 585, in wrap_solution
    solution.run(cis)
  File "solution.py", line 64, in run
    solve(gym_environment, cis)  # let's try to solve the challenge, exciting ah?
  File "solution.py", line 39, in solve
    observation, reward, done, info = env.step(action)
  File "/usr/local/lib/python2.7/dist-packages/gym/wrappers/time_limit.py", line 31, in step
    observation, reward, done, info = self.env.step(action)
  File "/notebooks/src/gym-duckietown-agent/gym_duckietown_agent/envs/simplesimagent_env.py", line 105, in step
    assert len(action) == 2
AssertionError


Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
5777770Simon Schaefer ðŸ‡¨ðŸ‡­AMOD18-AIDO not that random executionaido1_LF1_r3-v3step3-videossuccessnorudolf-30:00:42
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5766768Dzenan LapandicRandom executionaido1_LF1_r3-v3step1-simulationsuccessnorudolf-30:02:07
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
simulation-passed1
No reset possible
5763765Andrea Censi ðŸ‡¨ðŸ‡­Random executionaido1_LF1_r3-v3step4-vizsuccessnorudolf-30:01:50
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5752766Andrea Censi ðŸ‡¨ðŸ‡­Random executionaido1_LFV_r1-v3step3-videossuccessnorudolf-30:00:52
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
other stats
videos1
No reset possible
5749765Andrea Censi ðŸ‡¨ðŸ‡­Random executionaido1_LF1_r3-v3step2-scoringsuccessnorudolf-30:00:18
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
survival_time_median8.633333333333313


other stats
episodes
details{"ep000": {"nsteps": 168, "reward": -7.73651718135391, "good_angle": 23.848374609970847, "survival_time": 5.59999999999999, "traveled_tiles": 2, "valid_direction": 3.8666666666666543}, "ep001": {"nsteps": 379, "reward": -3.182353427777396, "good_angle": 20.844525634495124, "survival_time": 12.6333333333333, "traveled_tiles": 3, "valid_direction": 3.966666666666653}, "ep002": {"nsteps": 255, "reward": -3.954567064091052, "good_angle": 0.1750858106248234, "survival_time": 8.49999999999998, "traveled_tiles": 3, "valid_direction": 0}, "ep003": {"nsteps": 259, "reward": -6.242146121949303, "good_angle": 26.09013709887037, "survival_time": 8.633333333333313, "traveled_tiles": 3, "valid_direction": 3.93333333333332}, "ep004": {"nsteps": 271, "reward": -5.5840711386159505, "good_angle": 5.0010487282363805, "survival_time": 9.033333333333312, "traveled_tiles": 3, "valid_direction": 2.9999999999999902}}
good_angle_max26.09013709887037
good_angle_mean15.19183437643951
good_angle_median20.844525634495124
good_angle_min0.1750858106248234
reward_max-3.182353427777396
reward_mean-5.339930986757522
reward_median-5.5840711386159505
reward_min-7.73651718135391
survival_time_max12.6333333333333
survival_time_mean8.879999999999978
survival_time_min5.59999999999999
traveled_tiles_max3
traveled_tiles_mean2.8
traveled_tiles_median3
traveled_tiles_min2
valid_direction_max3.966666666666653
valid_direction_mean2.9533333333333234
valid_direction_median3.8666666666666543
valid_direction_min0
No reset possible