Challenge "aido3-off-LFVI-real-validation"

These are the metrics defined:

Traveled distance - driven_lanedir_consec

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Survival time - survival_time

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle.

Lateral deviation - deviation-center-line

This is the median lateral deviation from the center line.

Major infractions - in-drivable-lane

This is the median of the time spent outside of the drivable zones. For example this penalizes driving in the wrong lane.

Evaluation steps details

• At the beginning execute step eval0.

• If step eval0 has result success, then execute step eval0-visualize.

• If step eval0 has result failed, then declare the submission FAILED.

• If step eval0 has result error, then declare the submission ERROR.

• If step eval0 has result success, then execute step eval0-videos.

• If step eval1 has result success, then execute step eval1-videos.

• If step eval2 has result success, then execute step eval2-videos.

• If step eval0 has result success, then execute step eval1.

• If step eval0-visualize has result failed, then declare the submission FAILED.

• If step eval0-visualize has result error, then declare the submission ERROR.

• If step eval1 has result success, then execute step eval1-visualize.

• If step eval1 has result failed, then declare the submission FAILED.

• If step eval1 has result error, then declare the submission ERROR.

• If step eval1 has result success, then execute step eval2.

• If step eval1-visualize has result failed, then declare the submission FAILED.

• If step eval1-visualize has result error, then declare the submission ERROR.

• If step eval2 has result success, then execute step eval2-visualize.

• If step eval2 has result failed, then declare the submission FAILED.

• If step eval2 has result error, then declare the submission ERROR.

• If step eval2-visualize has result success, then declare the submission SUCCESS.

• If step eval2-visualize has result failed, then declare the submission FAILED.

• If step eval2-visualize has result error, then declare the submission ERROR.

Evaluation step eval0

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval0-evaluator:2020_02_28_11_20_14@sha256:14937e5e9a768219ba7097107df82d268b8c35a5f7fe87dc71269d4d164f6e9e
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval1

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval1-evaluator:2020_02_28_11_20_25@sha256:14937e5e9a768219ba7097107df82d268b8c35a5f7fe87dc71269d4d164f6e9e
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval2

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval2-evaluator:2020_02_28_11_20_38@sha256:14937e5e9a768219ba7097107df82d268b8c35a5f7fe87dc71269d4d164f6e9e
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval0-videos-evaluator:2020_02_28_11_21_22@sha256:3b828381f8de914eedb4ad4431e7c50cd3aa4a19be6498f6abb11dd4d9fb8c60
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/logs_raw
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '1'
            BAG_NAME_FILTER: autobot,watchtower
            OUTPUT_FRAMERATE: '7'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval1-videos

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval1-videos-evaluator:2020_02_28_11_21_40@sha256:3b828381f8de914eedb4ad4431e7c50cd3aa4a19be6498f6abb11dd4d9fb8c60
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval1/logs_raw
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '1'
            BAG_NAME_FILTER: autobot,watchtower
            OUTPUT_FRAMERATE: '7'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval2-videos

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval2-videos-evaluator:2020_02_28_11_21_56@sha256:3b828381f8de914eedb4ad4431e7c50cd3aa4a19be6498f6abb11dd4d9fb8c60
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval2/logs_raw
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '1'
            BAG_NAME_FILTER: autobot,watchtower
            OUTPUT_FRAMERATE: '7'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-visualize

Timeout 1080.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval0-visualize-evaluator:2020_02_28_11_20_50@sha256:d95babe778e197cc8f534d425915a334466289b2238c4e27bcde97c86a4a0674
        environment:
            STEP_NAME: eval0

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval1-visualize

Timeout 1080.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval1-visualize-evaluator:2020_02_28_11_21_00@sha256:d95babe778e197cc8f534d425915a334466289b2238c4e27bcde97c86a4a0674
        environment:
            STEP_NAME: eval1

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval2-visualize

Timeout 1080.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido3-off-lfvi-real-validation-eval2-visualize-evaluator:2020_02_28_11_21_11@sha256:d95babe778e197cc8f534d425915a334466289b2238c4e27bcde97c86a4a0674
        environment:
            STEP_NAME: eval2

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step