AI Driving Olympics Home Challenges Submissions

Challenge "aido-LFV_multi-sim-testing"

Challenge description

In this variation of the lane following challenge, your agent is embodied in multiple robots. Avoid the crashes with different versions of yourself.

See the documentation for more details.

Leaderboard

Submissions

Challenge logistics

Scoring

Scoring criteria

These are the metrics defined:

Survival time - survival_time_median

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle or pedestrian.

Major infractions - in-drivable-lane_median

This is the median of the time spent outside of the drivable zones. For example this penalizes driving in the wrong lane.

Traveled distance - driven_lanedir_consec_median

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Lateral deviation - deviation-center-line_median

This is the median lateral deviation from the center line.

Dependencies

Dependencies

Depends on successful evaluation on LFV_multi 🚗🚗🚗 - Lane following multibody (simulation 👾, validation 🏋)

The submission must first pass the validation step.

The sum of the following tests should be at least 2.0.

Test on absolute scores:

good_enough(1.0 points)
Obtain at least 0.2 for score driven_lanedir_consec_median.

Test on relative performance:

better-than-bea-straight(1.0 points)
Do at least as good as a submission of 1639 labeled straight.

Details

Technical details

Evaluation steps details

Evaluation step sim-0of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:da47d1c450129b467f49d306b49b1706a1d69010b8e95a9b840f993cbcdc9091
        environment:
            replica: '{"index": 0, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 3

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                scenarios:

                - /scenarios

                port: 10123

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\ntopdown_resolution: 600\ndebug_no_video: false\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out
    solution-ego1:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: ego1
            AIDONODE_DATA_IN: /fifos/ego1-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego1-out
    solution-ego2:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: ego2
            AIDONODE_DATA_IN: /fifos/ego2-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego2-out
    solution-ego3:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: ego3
            AIDONODE_DATA_IN: /fifos/ego3-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego3-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Cloud simulations1

Evaluation step sim-1of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:da47d1c450129b467f49d306b49b1706a1d69010b8e95a9b840f993cbcdc9091
        environment:
            replica: '{"index": 1, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 3

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                scenarios:

                - /scenarios

                port: 10123

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\ntopdown_resolution: 600\ndebug_no_video: false\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out
    solution-ego1:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: ego1
            AIDONODE_DATA_IN: /fifos/ego1-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego1-out
    solution-ego2:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: ego2
            AIDONODE_DATA_IN: /fifos/ego2-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego2-out
    solution-ego3:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: ego3
            AIDONODE_DATA_IN: /fifos/ego3-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego3-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Cloud simulations1

Evaluation step sim-2of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:da47d1c450129b467f49d306b49b1706a1d69010b8e95a9b840f993cbcdc9091
        environment:
            replica: '{"index": 2, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 3

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                scenarios:

                - /scenarios

                port: 10123

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\ntopdown_resolution: 600\ndebug_no_video: false\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out
    solution-ego1:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: ego1
            AIDONODE_DATA_IN: /fifos/ego1-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego1-out
    solution-ego2:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: ego2
            AIDONODE_DATA_IN: /fifos/ego2-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego2-out
    solution-ego3:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: ego3
            AIDONODE_DATA_IN: /fifos/ego3-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego3-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Cloud simulations1

Evaluation step sim-3of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:da47d1c450129b467f49d306b49b1706a1d69010b8e95a9b840f993cbcdc9091
        environment:
            replica: '{"index": 3, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 3

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                scenarios:

                - /scenarios

                port: 10123

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\ntopdown_resolution: 600\ndebug_no_video: false\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out
    solution-ego1:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: ego1
            AIDONODE_DATA_IN: /fifos/ego1-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego1-out
    solution-ego2:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: ego2
            AIDONODE_DATA_IN: /fifos/ego2-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego2-out
    solution-ego3:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: ego3
            AIDONODE_DATA_IN: /fifos/ego3-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego3-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Cloud simulations1