Challenge "aido1_LFV_r1-v3"

These are the metrics defined:

Traveled distance - driven_lanedir_consec_median

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Survival time - survival_time_median

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle.

Lateral deviation - deviation-center-line_median

This is the median lateral deviation from the center line.

Major infractions - in-drivable-lane_median

This is the median of the time spent outside of the drivable zones. For example this penalizes driving in the wrong lane.

Evaluation steps details

• At the beginning execute step step1-simulation.

• If step step1-simulation has result success, then execute step step2-scoring.

• If step step1-simulation has result failed, then declare the submission FAILED.

• If step step1-simulation has result error, then declare the submission ERROR.

• If step step2-scoring has result success, then execute step step3-videos.

• If step step2-scoring has result failed, then declare the submission FAILED.

• If step step2-scoring has result error, then declare the submission ERROR.

• If step step3-videos has result success, then execute step step4-viz.

• If step step3-videos has result failed, then declare the submission FAILED.

• If step step3-videos has result error, then declare the submission ERROR.

• If step step4-viz has result success, then declare the submission SUCCESS.

• If step step4-viz has result failed, then declare the submission FAILED.

• If step step4-viz has result error, then declare the submission ERROR.

Evaluation step step1-simulation

Timeout None

This is the Docker Compose configuration skeleton:

version: '3'
services:
    solution:
        image: SUBMISSION_CONTAINER
        environment: {}
    evaluator:
        image: andreacensi/aido1_lfv_r1-v3-step1-simulation-evaluator:2018_11_22_15_24_56
        environment:
            launcher_parameters: "challenge: LF\nepisodes: 5\nsteps_per_episode: 500\n\
                environment-constructor: Simulator\ninclude_map: true\nenvironment-parameters:\
                \ {seed: 123, map_name: loop_obstacles, max_steps: 500000, domain_rand:\
                \ 0,\n  camera_width: 640, camera_height: 480, accept_start_angle_deg:\
                \ 6, full_transparency: true,\n  distortion: true}\nagent-info: {challenge:\
                \ LF}\n"
            solution_parameters: '{env: Duckietown-Lf-Lfv-Navv-Silent-v1}

                '

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

No particular resources required.

Evaluation step step3-videos

Timeout None

This is the Docker Compose configuration skeleton:

version: '3'
services:
    videos:
        image: andreacensi/aido1_lfv_r1-v3-step3-videos-videos:2018_11_08_16_21_39
        environment: {}

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

No particular resources required.

Evaluation step step2-scoring

Timeout None

This is the Docker Compose configuration skeleton:

version: '3'
services:
    scorer:
        image: andreacensi/aido1_lfv_r1-v3-step2-scoring-scorer:2018_11_08_16_21_45
        environment: {}

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

No particular resources required.

Evaluation step step4-viz

Timeout None

This is the Docker Compose configuration skeleton:

version: '3'
services:
    visualization:
        image: andreacensi/aido1_lfv_r1-v3-step4-viz-visualization:2018_12_06_13_08_48
        environment: {}

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

No particular resources required.