Challenge "aido-LFI-state-sim-validation"

These are the metrics defined:

Survival time - survival_time_median

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle or pedestrian.

Major infractions - in-drivable-lane_median

This is the median of the time spent outside of the drivable zones. For example this penalizes driving in the wrong lane.

Traveled distance - driven_lanedir_consec_median

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Lateral deviation - deviation-center-line_median

This is the median lateral deviation from the center line.

Evaluation steps details

• At the beginning execute step sim-0of4.

• If step sim-0of4 has result failed, then declare the submission FAILED.

• If step sim-0of4 has result error, then declare the submission ERROR.

• At the beginning execute step sim-1of4.

• If step sim-1of4 has result failed, then declare the submission FAILED.

• If step sim-1of4 has result error, then declare the submission ERROR.

• At the beginning execute step sim-2of4.

• If step sim-2of4 has result failed, then declare the submission FAILED.

• If step sim-2of4 has result error, then declare the submission ERROR.

• At the beginning execute step sim-3of4.

• If step sim-3of4 has result failed, then declare the submission FAILED.

• If step sim-3of4 has result error, then declare the submission ERROR.

• If (step sim-0of4 has result success) and (step sim-1of4 has result success) and (step sim-2of4 has result success) and (step sim-3of4 has result success), then declare the submission SUCCESS.

Evaluation step sim-0of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    npc0:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: npc0
            AIDONODE_DATA_IN: /fifos/npc0-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc0-out
    npc1:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: npc1
            AIDONODE_DATA_IN: /fifos/npc1-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc1-out
    npc2:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: npc2
            AIDONODE_DATA_IN: /fifos/npc2-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc2-out
    npc3:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: npc3
            AIDONODE_DATA_IN: /fifos/npc3-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc3-out
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:7dc9cd4ad7e9d52c83deef562e9d6b3ef214fd8e6e5b51dd7c1ec658f65064b4
        environment:
            replica: '{"index": 0, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 1

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                port: 10123

                scenarios:

                - /scenarios

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 0, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step sim-1of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    npc0:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: npc0
            AIDONODE_DATA_IN: /fifos/npc0-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc0-out
    npc1:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: npc1
            AIDONODE_DATA_IN: /fifos/npc1-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc1-out
    npc2:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: npc2
            AIDONODE_DATA_IN: /fifos/npc2-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc2-out
    npc3:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: npc3
            AIDONODE_DATA_IN: /fifos/npc3-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc3-out
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:7dc9cd4ad7e9d52c83deef562e9d6b3ef214fd8e6e5b51dd7c1ec658f65064b4
        environment:
            replica: '{"index": 1, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 1

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                port: 10123

                scenarios:

                - /scenarios

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 1, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step sim-2of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    npc0:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: npc0
            AIDONODE_DATA_IN: /fifos/npc0-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc0-out
    npc1:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: npc1
            AIDONODE_DATA_IN: /fifos/npc1-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc1-out
    npc2:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: npc2
            AIDONODE_DATA_IN: /fifos/npc2-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc2-out
    npc3:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: npc3
            AIDONODE_DATA_IN: /fifos/npc3-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc3-out
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:7dc9cd4ad7e9d52c83deef562e9d6b3ef214fd8e6e5b51dd7c1ec658f65064b4
        environment:
            replica: '{"index": 2, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 1

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                port: 10123

                scenarios:

                - /scenarios

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 2, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step sim-3of4

Timeout 10000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    npc0:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: npc0
            AIDONODE_DATA_IN: /fifos/npc0-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc0-out
    npc1:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: npc1
            AIDONODE_DATA_IN: /fifos/npc1-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc1-out
    npc2:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: npc2
            AIDONODE_DATA_IN: /fifos/npc2-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc2-out
    npc3:
        image: docker.io/andreacensi/duckietown-challenges@sha256:5ff981d800f1a3ac674ce4b10ebfe9075f61792c1e944f7e6e9075d6ecaa1e08
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: npc3
            AIDONODE_DATA_IN: /fifos/npc3-in
            AIDONODE_DATA_OUT: fifo:/fifos/npc3-out
    evaluator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:7dc9cd4ad7e9d52c83deef562e9d6b3ef214fd8e6e5b51dd7c1ec658f65064b4
        environment:
            replica: '{"index": 3, "total": 4}'
            experiment_manager_parameters: 'episodes_per_scenario: 1

                episode_length_s: 60.0

                min_episode_length_s: 0.0

                seed: 20200922

                physics_dt: 0.05

                max_failures: 2

                fifo_dir: /fifos

                sim_in: /fifos/simulator-in

                sim_out: /fifos/simulator-out

                sm_in: /fifos/scenario_maker-in

                sm_out: /fifos/scenario_maker-out

                timeout_initialization: 120

                timeout_regular: 120

                port: 10123

                scenarios:

                - /scenarios

                '
        ports:
        - '10123'
    simulator:
        image: docker.io/andreacensi/duckietown-challenges@sha256:94bdf625b89d4faf0b11a72aaebe4db6c4ef5fc855a333a26d7d585d4ea6cb07
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_CONFIG: "env_constructor: Simulator\nenv_parameters:\n  max_steps:\
                \ 500001 # we don't want the gym to reset itself\n  domain_rand: 0\n\
                \  camera_width: 640\n  camera_height: 480\n  distortion: true\n \
                \ num_tris_distractors: 0\n  color_ground: [0, 0.3, 0] # green\n \
                \ enable_leds: true\n"
            AIDONODE_DATA_IN: /fifos/simulator-in
            AIDONODE_DATA_OUT: fifo:/fifos/simulator-out
    solution-ego0:
        image: SUBMISSION_CONTAINER
        environment:
            replica: '{"index": 3, "total": 4}'
            AIDONODE_NAME: ego0
            AIDONODE_DATA_IN: /fifos/ego0-in
            AIDONODE_DATA_OUT: fifo:/fifos/ego0-out

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step