Duckietown Challenges Home Challenges Submissions

Submission 2818

Submission2818
Competingyes
Challengeaido2-LF-sim-testing
UserLiam Paull 🇨🇦
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobsstep1-simulation: 22759
Next
User labelchallenge-aido_LF-template-random
Admin priority50
Blessingn/a
User priority50

The highlights are available only to the owner and the admins.

Evaluation jobs for this submission

Show only up-to-date jobs
Job IDstepstatusup to datedate starteddate completeddurationmessage
22759step1-simulationsuccessyes0:15:01
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec_median0.5723310224856362
survival_time_median2.549999999999999
deviation-center-line_median0.10099507669212504
in-drivable-lane_median0.4499999999999984


other stats
agent_compute-ego_max0.15117651341008206
agent_compute-ego_mean0.14068361804031848
agent_compute-ego_median0.14280658108847483
agent_compute-ego_min0.11937679076681332
deviation-center-line_max0.2838635700333795
deviation-center-line_mean0.12786165368054708
deviation-center-line_min0.07112529346797256
deviation-heading_max0.5289022616823588
deviation-heading_mean0.3391682685652287
deviation-heading_median0.32928385103533875
deviation-heading_min0.19401296908406307
driven_any_max2.1706243860917107
driven_any_mean0.9759490378276146
driven_any_median1.0117660515775693
driven_any_min0.23626560261560617
driven_lanedir_consec_max1.919400421063832
driven_lanedir_consec_mean0.6896171461819246
driven_lanedir_consec_min0.2264965947033466
driven_lanedir_max1.919400421063832
driven_lanedir_mean0.6896171461819246
driven_lanedir_median0.5723310224856362
driven_lanedir_min0.2264965947033466
in-drivable-lane_max1.7999999999999936
in-drivable-lane_mean0.5899999999999982
in-drivable-lane_min0
per-episodes
details{"ETHZ_autolab_technical_track-0-0": {"driven_any": 1.0243289631429708, "sim_physics": 0.11176469149412932, "survival_time": 2.6999999999999984, "driven_lanedir": 0.5723310224856362, "sim_render-ego": 0.05705622831980387, "in-drivable-lane": 0.9999999999999976, "agent_compute-ego": 0.14148211479187012, "deviation-heading": 0.19401296908406307, "set_robot_commands": 0.0924577624709518, "deviation-center-line": 0.15907928712407893, "driven_lanedir_consec": 0.5723310224856362, "sim_compute_sim_state": 0.03756400832423457, "sim_compute_performance-ego": 0.06091920093253807, "sim_compute_robot_state-ego": 0.06821718039336028}, "ETHZ_autolab_technical_track-1-0": {"driven_any": 0.9933867744747804, "sim_physics": 0.10657774700837976, "survival_time": 2.549999999999999, "driven_lanedir": 0.8010406009604816, "sim_render-ego": 0.05898066595488904, "in-drivable-lane": 0.29999999999999893, "agent_compute-ego": 0.1476132542479272, "deviation-heading": 0.5289022616823588, "set_robot_commands": 0.08589900241178625, "deviation-center-line": 0.13005513204193506, "driven_lanedir_consec": 0.8010406009604816, "sim_compute_sim_state": 0.03690014165990493, "sim_compute_performance-ego": 0.06324121063830805, "sim_compute_robot_state-ego": 0.06883692741394043}, "ETHZ_autolab_technical_track-2-0": {"driven_any": 0.23626560261560617, "sim_physics": 0.12251743148354922, "survival_time": 0.8500000000000002, "driven_lanedir": 0.2264965947033466, "sim_render-ego": 0.0591580447028665, "in-drivable-lane": 0, "agent_compute-ego": 0.1357131284825942, "deviation-heading": 0.23048898560984332, "set_robot_commands": 0.08997802173390108, "deviation-center-line": 0.0788839760023766, "driven_lanedir_consec": 0.2264965947033466, "sim_compute_sim_state": 0.042992143069996554, "sim_compute_performance-ego": 0.061269409516278434, "sim_compute_robot_state-ego": 0.06797632049111758}, "ETHZ_autolab_technical_track-3-0": {"driven_any": 0.4346971317320432, "sim_physics": 0.11990645408630372, "survival_time": 1.2500000000000004, "driven_lanedir": 0.40985475638935265, "sim_render-ego": 0.06403008460998535, "in-drivable-lane": 0, "agent_compute-ego": 0.14908554077148437, "deviation-heading": 0.4138862827398061, "set_robot_commands": 0.10303014755249024, "deviation-center-line": 0.08421324241865966, "driven_lanedir_consec": 0.40985475638935265, "sim_compute_sim_state": 0.03944601058959961, "sim_compute_performance-ego": 0.06810245513916016, "sim_compute_robot_state-ego": 0.0667952060699463}, "ETHZ_autolab_technical_track-4-0": {"driven_any": 1.618214457700942, "sim_physics": 0.11308641071561017, "survival_time": 3.949999999999994, "driven_lanedir": 0.7890999076769956, "sim_render-ego": 0.06059138684333125, "in-drivable-lane": 1.7999999999999936, "agent_compute-ego": 0.14953582498091686, "deviation-heading": 0.23606825392187256, "set_robot_commands": 0.0956459166128424, "deviation-center-line": 0.2072381278624381, "driven_lanedir_consec": 0.7890999076769956, "sim_compute_sim_state": 0.03727719451807722, "sim_compute_performance-ego": 0.06267236456086364, "sim_compute_robot_state-ego": 0.0696328259721587}, "ETHZ_autolab_technical_track-5-0": {"driven_any": 1.0038514956833002, "sim_physics": 0.08870693615504674, "survival_time": 2.4499999999999993, "driven_lanedir": 0.7531703123735944, "sim_render-ego": 0.049352096051585914, "in-drivable-lane": 0.4499999999999984, "agent_compute-ego": 0.11937679076681332, "deviation-heading": 0.4596437246381204, "set_robot_commands": 0.07476212540451362, "deviation-center-line": 0.08968283443958727, "driven_lanedir_consec": 0.7531703123735944, "sim_compute_sim_state": 0.029975949501504704, "sim_compute_performance-ego": 0.0529802283462213, "sim_compute_robot_state-ego": 0.0599993540316212}, "ETHZ_autolab_technical_track-6-0": {"driven_any": 1.0542898001504375, "sim_physics": 0.07600830282483782, "survival_time": 2.799999999999998, "driven_lanedir": 0.5389398655375865, "sim_render-ego": 0.04772914733205523, "in-drivable-lane": 1.1499999999999972, "agent_compute-ego": 0.11945236580712457, "deviation-heading": 0.4109275195546752, "set_robot_commands": 0.0758951987539019, "deviation-center-line": 0.10786249131209884, "driven_lanedir_consec": 0.5389398655375865, "sim_compute_sim_state": 0.032044129712241035, "sim_compute_performance-ego": 0.05164019976343427, "sim_compute_robot_state-ego": 0.053339366401944845}, "ETHZ_autolab_technical_track-7-0": {"driven_any": 0.3164475681403097, "sim_physics": 0.11845144771394275, "survival_time": 1.0500000000000005, "driven_lanedir": 0.3093025166124813, "sim_render-ego": 0.05931589716956729, "in-drivable-lane": 0, "agent_compute-ego": 0.14458686964852469, "deviation-heading": 0.2242052814634115, "set_robot_commands": 0.0844211010705857, "deviation-center-line": 0.09758615464743424, "driven_lanedir_consec": 0.3093025166124813, "sim_compute_sim_state": 0.03566882723853702, "sim_compute_performance-ego": 0.06128812971569243, "sim_compute_robot_state-ego": 0.07168849309285481}, "ETHZ_autolab_technical_track-8-0": {"driven_any": 1.2018528489510765, "sim_physics": 0.10380313760143216, "survival_time": 2.9499999999999975, "driven_lanedir": 0.6922532395203, "sim_render-ego": 0.056251024795790854, "in-drivable-lane": 1.0499999999999965, "agent_compute-ego": 0.14532009625838974, "deviation-heading": 0.32928385103533875, "set_robot_commands": 0.08636815265073615, "deviation-center-line": 0.07112529346797256, "driven_lanedir_consec": 0.6922532395203, "sim_compute_sim_state": 0.032322358276884434, "sim_compute_performance-ego": 0.06191707465608241, "sim_compute_robot_state-ego": 0.06369139784473483}, "ETHZ_autolab_technical_track-9-0": {"driven_any": 1.0435481553257333, "sim_physics": 0.1008928125554865, "survival_time": 2.7499999999999982, "driven_lanedir": 0.5204405548974873, "sim_render-ego": 0.05921255891973322, "in-drivable-lane": 1.1499999999999977, "agent_compute-ego": 0.1456624984741211, "deviation-heading": 0.33296447766287096, "set_robot_commands": 0.08346153172579679, "deviation-center-line": 0.0950267943453044, "driven_lanedir_consec": 0.5204405548974873, "sim_compute_sim_state": 0.03369640437039462, "sim_compute_performance-ego": 0.06498578244989568, "sim_compute_robot_state-ego": 0.06657355915416371}, "ETHZ_autolab_technical_track-10-0": {"driven_any": 1.5954548672193265, "sim_physics": 0.10572023825211958, "survival_time": 3.8499999999999943, "driven_lanedir": 1.1042463557848754, "sim_render-ego": 0.057809030854856815, "in-drivable-lane": 1.0499999999999965, "agent_compute-ego": 0.13627429441972214, "deviation-heading": 0.3566810710676767, "set_robot_commands": 0.09021450946857404, "deviation-center-line": 0.22513914436082189, "driven_lanedir_consec": 1.1042463557848754, "sim_compute_sim_state": 0.03597989949313077, "sim_compute_performance-ego": 0.0629664489201137, "sim_compute_robot_state-ego": 0.06540992662504122}, "ETHZ_autolab_technical_track-11-0": {"driven_any": 0.49030923107489305, "sim_physics": 0.11284977197647096, "survival_time": 1.4000000000000006, "driven_lanedir": 0.4826393189617785, "sim_render-ego": 0.057041517325810025, "in-drivable-lane": 0, "agent_compute-ego": 0.14280658108847483, "deviation-heading": 0.23322336656472228, "set_robot_commands": 0.08812700850622994, "deviation-center-line": 0.10099507669212504, "driven_lanedir_consec": 0.4826393189617785, "sim_compute_sim_state": 0.03591845716748919, "sim_compute_performance-ego": 0.06432822772434779, "sim_compute_robot_state-ego": 0.06650879553386144}, "ETHZ_autolab_technical_track-12-0": {"driven_any": 1.0117660515775693, "sim_physics": 0.10891577309253168, "survival_time": 2.549999999999999, "driven_lanedir": 0.791717039833255, "sim_render-ego": 0.06148292971592323, "in-drivable-lane": 0.3999999999999986, "agent_compute-ego": 0.15117651341008206, "deviation-heading": 0.320449356593834, "set_robot_commands": 0.0952617561115938, "deviation-center-line": 0.1017166571540705, "driven_lanedir_consec": 0.791717039833255, "sim_compute_sim_state": 0.03877721580804563, "sim_compute_performance-ego": 0.07104898434059292, "sim_compute_robot_state-ego": 0.07849649822010714}, "ETHZ_autolab_technical_track-13-0": {"driven_any": 0.44419823353351984, "sim_physics": 0.09960473908318414, "survival_time": 1.3500000000000003, "driven_lanedir": 0.4333246859278659, "sim_render-ego": 0.05808208606861256, "in-drivable-lane": 0, "agent_compute-ego": 0.1395866517667417, "deviation-heading": 0.2988926743481924, "set_robot_commands": 0.0896724506660744, "deviation-center-line": 0.08545702330592322, "driven_lanedir_consec": 0.4333246859278659, "sim_compute_sim_state": 0.03840106504934805, "sim_compute_performance-ego": 0.061319986979166664, "sim_compute_robot_state-ego": 0.06418256406430844}, "ETHZ_autolab_technical_track-14-0": {"driven_any": 2.1706243860917107, "sim_physics": 0.09305192442501292, "survival_time": 5.09999999999999, "driven_lanedir": 1.919400421063832, "sim_render-ego": 0.05624539711896111, "in-drivable-lane": 0.4999999999999982, "agent_compute-ego": 0.14258174568999046, "deviation-heading": 0.5178939525116442, "set_robot_commands": 0.0884536247627408, "deviation-center-line": 0.2838635700333795, "driven_lanedir_consec": 1.919400421063832, "sim_compute_sim_state": 0.0359801451365153, "sim_compute_performance-ego": 0.06106500765856575, "sim_compute_robot_state-ego": 0.06379492843852323}}
set_robot_commands_max0.10303014755249024
set_robot_commands_mean0.08824322066018127
set_robot_commands_median0.0884536247627408
set_robot_commands_min0.07476212540451362
sim_compute_performance-ego_max0.07104898434059292
sim_compute_performance-ego_mean0.06198298075608408
sim_compute_performance-ego_median0.06191707465608241
sim_compute_performance-ego_min0.05164019976343427
sim_compute_robot_state-ego_max0.07849649822010714
sim_compute_robot_state-ego_mean0.06634288958317895
sim_compute_robot_state-ego_median0.06657355915416371
sim_compute_robot_state-ego_min0.053339366401944845
sim_compute_sim_state_max0.042992143069996554
sim_compute_sim_state_mean0.036196263327726914
sim_compute_sim_state_median0.0359801451365153
sim_compute_sim_state_min0.029975949501504704
sim_physics_max0.12251743148354922
sim_physics_mean0.10545718789786918
sim_physics_median0.10657774700837976
sim_physics_min0.07600830282483782
sim_render-ego_max0.06403008460998535
sim_render-ego_mean0.057489206385584815
sim_render-ego_median0.05808208606861256
sim_render-ego_min0.04772914733205523
simulation-passed1
survival_time_max5.09999999999999
survival_time_mean2.503333333333331
survival_time_min0.8500000000000002
No reset possible
22757step1-simulationhost-erroryes0:00:10
Uncaught exception: [...]
Uncaught exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/ipfshttpclient/http.py", line 255, in _do_raise_for_status
    response.raise_for_status()
  File "/usr/local/lib/python3.6/dist-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error for url: http+ip4://127.0.0.1:5001/api/v0/swarm/connect?stream-channels=true&arg=%2Fdns4%2Fipfs.duckietown.org%2Ftcp%2F4001%2Fipfs%2FQmPyoL4ZwaTYtGsvFG8A5fG85tQH5sCHWtexkNVjqa52iK

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown_challenges_runner/runner.py", line 541, in get_cr
    allow_host_network=allow_host_network, use_ipfs=use_ipfs)
  File "/project/src/duckietown_challenges_runner/runner.py", line 605, in run_single
    prepare_dir(wd, aws_config, steps2artefacts, use_ipfs=use_ipfs)
  File "/project/src/duckietown_challenges_runner/runner.py", line 877, in prepare_dir
    download_artefacts(aws_config, steps2artefacts, previous_steps_dir, use_ipfs=use_ipfs)
  File "/project/src/duckietown_challenges_runner/runner.py", line 1151, in download_artefacts
    res = client.swarm_connect(peer)
  File "/usr/local/lib/python3.6/dist-packages/ipfsapi/client/__init__.py", line 205, in wrapper
    return value(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ipfshttpclient/client/base.py", line 15, in wrapper
    result = func(*args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ipfshttpclient/client/swarm.py", line 137, in connect
    return self._client.request('/swarm/connect', args, decoder='json', **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/ipfshttpclient/http.py", line 44, in wrapper
    return func(self, *args, **merged)
  File "/usr/local/lib/python3.6/dist-packages/ipfshttpclient/http.py", line 364, in request
    files, headers, data, timeout=timeout)
  File "/usr/local/lib/python3.6/dist-packages/ipfshttpclient/http.py", line 286, in _request
    self._do_raise_for_status(res)
  File "/usr/local/lib/python3.6/dist-packages/ipfshttpclient/http.py", line 273, in _do_raise_for_status
    six.raise_from(exceptions.ErrorResponse(msg, error), error)
  File "<string>", line 3, in raise_from
ipfshttpclient.exceptions.ErrorResponse: connect QmPyoL4ZwaTYtGsvFG8A5fG85tQH5sCHWtexkNVjqa52iK failure: dial attempt failed: <peer.ID T35iPP> --> <peer.ID PyoL4Z> dial attempt failed: context deadline exceeded
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20992step1-simulationsuccessno0:12:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
20313step1-simulationsuccessno0:05:57
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19331step1-simulationsuccessno0:05:10
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
19193step1-simulationerrorno0:02:45
InvalidEvaluator: Tr [...]
InvalidEvaluator:
Traceback (most recent call last):
  File "experiment_manager.py", line 173, in main
    make_video1(fn, out_video)
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
    pg('video_aido', dict(filename=fn, output=output))
  File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
    model.init()
  File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
    block.init()
  File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
    self.log = read_simulator_log_cbor(self.get_config('filename'))
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
    render_time = read_perfomance(ld)
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
    for i, ob in enumerate(read_topic2(ld, 'timing_information')):
  File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
    for ob in ld.objects:
AttributeError: 'str' object has no attribute 'objects'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/project/src/duckietown-challenges/src/duckietown_challenges/cie_concrete.py", line 610, in scoring_context
    yield cie
  File "experiment_manager.py", line 435, in <module>
    wrap(cie)
  File "experiment_manager.py", line 423, in wrap
    main(cie, logdir, attempts)
  File "experiment_manager.py", line 186, in main
    raise dc.InvalidEvaluator(msg) from e
duckietown_challenges.exceptions.InvalidEvaluator: Anomalous error while running episodes:

 > Traceback (most recent call last):
 >   File "experiment_manager.py", line 173, in main
 >     make_video1(fn, out_video)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 56, in make_video1
 >     pg('video_aido', dict(filename=fn, output=output))
 >   File "/project/src/procgraph/src/procgraph/scripts/pgmain.py", line 244, in pg
 >     model.init()
 >   File "/project/src/procgraph/src/procgraph/core/model.py", line 218, in init
 >     block.init()
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_video.py", line 16, in init
 >     self.log = read_simulator_log_cbor(self.get_config('filename'))
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 179, in read_simulator_log_cbor
 >     render_time = read_perfomance(ld)
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 74, in read_perfomance
 >     for i, ob in enumerate(read_topic2(ld, 'timing_information')):
 >   File "/project/src/aido-protocols/src/aido_schemas/utils_drawing.py", line 49, in read_topic2
 >     for ob in ld.objects:
 > AttributeError: 'str' object has no attribute 'objects'
 > 
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18613step1-simulationsuccessno0:05:51
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18279step1-simulationsuccessno0:07:25
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18187step1-simulationsuccessno0:05:59
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
18157step1-simulationsuccessno0:07:42
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible