AI Driving Olympics Home Challenges Submissions Jobs

Challenge "LFV 🚗🚗 - Lane following + Vehicles (simulation 👾, validation 🏋)"

Challenge description

Lane following challenge.

Leaderboard

Submissions

Challenge logistics

Scoring

Scoring criteria

These are the metrics defined:

Traveled distance - driven_lanedir_consec_median

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Survival time - survival_time_median

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle.

Lateral deviation - deviation-center-line_median

This is the median lateral deviation from the center line.

Major infractions - in-drivable-lane_median

This is the median of the time spent outside of the drivable zones. For example this penalizes driving in the wrong lane.

Dependencies

Dependencies

Depends on successful evaluation on LF 🚗 - Lane following (simulation 👾, validation 🏋)

The submission must first pass the LF validation.

The sum of the following tests should be at least 2.0.

Test on absolute scores:

good_enough(1.0 points)
Obtain at least 0.2 for score driven_lanedir_consec_median.

Test on relative performance:

better-than-bea-straight(1.0 points)
Do at least as good as a submission ofBea Baselines labeled straight.

Details

Technical details

Evaluation steps details

  • At the beginning execute step step1-simulation.

  • If step step1-simulation finishes with status success, then declare the submission SUCCESS.

  • If step step1-simulation finishes with status failed, then declare the submission FAILED.

  • If step step1-simulation finishes with status error, then declare the submission ERROR.

Evaluation step step1-simulation

Timeout 3600.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    solution:
        image: SUBMISSION_CONTAINER
        environment:
            AIDONODE_DATA_IN: /fifos/agent-in
            AIDONODE_DATA_OUT: fifo:/fifos/agent-out
    evaluator:
        image: docker.io/andreacensi/aido3-lfv-sim-validation-step1-simulation-evaluator:2019_12_02_17_31_55@sha256:e88f8176fbc592df2d3a373bb2d5d92d36dc2500d334e79c4ecbec58c96f1462
        environment:
            experiment_manager_parameters: 'episodes_per_scenario: 1

                episode_length_s: 15.0

                min_episode_length_s: 0.0

                seed: 43

                physics_dt: 0.05

                max_failures: 2

                agent_in: /fifos/agent-in

                agent_out: /fifos/agent-out

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                '
    simulator:
        image: docker.io/andreacensi/aido3-lfv-sim-validation-step1-simulation-simulator:2019_12_02_17_32_07@sha256:c580bc85b33c69f06773487b22417d32e78f5ccdc660efcdae5cf03c152565c4
        environment:
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    scenario_maker:
        image: docker.io/andreacensi/aido3-lfv-sim-validation-step1-simulation-scenario_maker:2019_12_02_17_32_24@sha256:334aefbd8e2447e6bf8df442acd9a55cfad7ac178aff72f7cfb498b47fc06007
        environment:
            AIDONODE_CONFIG: 'maps:

                - ETHZ_autolab_technical_track

                scenarios_per_map: 5

                robots_npcs: 4

                '
            AIDONODE_DATA_IN: /fifos/scenario_maker-in
            AIDONODE_DATA_OUT: fifo:/fifos/scenario_maker-out
            AIDONODE_ENCODING: cbor
            DISABLE_CONTRACTS: '1'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Cloud simulations1