Challenge "aido2-LF-real-validation"

These are the metrics defined:

Traveled distance - driven_lanedir_consec_median

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Survival time - survival_time_median

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle.

Evaluation steps details

• At the beginning execute step eval0.

• If step eval0 has result success, then execute step eval0-videos-autobots.

• If step eval0 has result failed, then declare the submission FAILED.

• If step eval0 has result error, then declare the submission ERROR.

• If step eval0-videos-autobots has result success, then execute step eval0-videos-watchtowers.

• If step eval0-videos-autobots has result failed, then declare the submission FAILED.

• If step eval0-videos-autobots has result error, then declare the submission ERROR.

• If step eval0-videos-watchtowers has result success, then execute step eval1.

• If step eval0-videos-watchtowers has result failed, then declare the submission FAILED.

• If step eval0-videos-watchtowers has result error, then declare the submission ERROR.

• If step eval1 has result success, then execute step eval1-videos-autobots.

• If step eval1 has result failed, then declare the submission FAILED.

• If step eval1 has result error, then declare the submission ERROR.

• If step eval1-videos-autobots has result success, then execute step eval1-videos-watchtowers.

• If step eval1-videos-autobots has result failed, then declare the submission FAILED.

• If step eval1-videos-autobots has result error, then declare the submission ERROR.

• If step eval1-videos-watchtowers has result success, then declare the submission SUCCESS.

• If step eval1-videos-watchtowers has result failed, then declare the submission FAILED.

• If step eval1-videos-watchtowers has result error, then declare the submission ERROR.

Evaluation step hw-evaluation

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-hw-evaluation-evaluator:2019_05_13_17_48_37@sha256:1168d196612a40db2eedce3f412838293600898ed3f6a96dccc2679c82acee56
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-validation-eval0-evaluator:2019_06_11_19_00_11@sha256:04dad2af97033eef32f70bfea1f9fad34663a5ad17f7e998c7984bb68012de88
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos

Timeout 1800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/aleksandarpetrov/aido2-lf-real-validation-eval0-videos-evaluator:2019_05_17_13_23_57@sha256:90e7c45aff3c2936cca59c2eb16668e882cc06a377ce9315900496a8efaaeb61
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-extractATs

Timeout 3600.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/aleksandarpetrov/aido2-lf-real-validation-eval0-extractats-evaluator:2019_05_17_14_45_03@sha256:a12b33e0fb36d6abb88b542b1d0a751467ff24824ac209b75d09db985d931d64
        environment:
            BAGS_PATH: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            ACQ_OUTPUT_BAG: /challenges/challenge-evaluation-output/processed.bag
            ACQ_TEST_STREAM: '0'
            ACQ_DEVICE_THREADS: '6'
            ACQ_OBSERVATIONS_STATISTICS_OUTPUT: /challenges/challenge-evaluation-output/statistics.yaml

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-WesQuackerson

Timeout 3600.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/aleksandarpetrov/aido2-lf-real-validation-eval0-wesquackerson-evaluator:2019_05_17_14_45_11@sha256:285263e1b8a043e4ef9f7dc25b3d5796c6a0bb09a4d629bd8bba5ec1b07159e5
        environment:
            ATMSGS_BAG: /challenges/previous-steps/eval0-extractATs/challenge-evaluation-output/processed.bag
            TITLE_CARD: '1'
            VIDEO_BAGS: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_FILE: /challenges/challenge-evaluation-output/overheadVideo.mp4
            POSES_TOPIC: /poses_acquisition/poses
            VIDEO_TOPIC: /camera_node/image/compressed
            MIN_SHOT_LENGTH: '12'
            OUTPUT_FRAMERATE: '12'
            PREVIOUS_RESULTS: /challenges/previous-steps/eval0/challenge-evaluation-output/challenge-results/challenge_results.yaml
            CAMERA_RESOLUTION_WIDTH: '972'
            CAMERA_RESOLUTION_HEIGHT: '1296'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-0

Timeout 1800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-eval0-videos-0-evaluator:2019_05_16_13_56_53@sha256:45494b8ddc5c6d02c900845a72601852f9b63253b18d19108d7f5ee51ebbeaa5
        environment:
            WORKER_I: '0'
            WORKER_N: '2'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-1

Timeout 1800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-eval0-videos-1-evaluator:2019_05_16_13_57_12@sha256:45494b8ddc5c6d02c900845a72601852f9b63253b18d19108d7f5ee51ebbeaa5
        environment:
            WORKER_I: '1'
            WORKER_N: '2'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-autobots

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-validation-eval0-videos-autobots-evaluator:2019_06_11_19_00_51@sha256:5876da21ebc74f8c77acec009b39b23c52b9f374eef76e182936e1ad8d05ae86
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '1'
            BAG_NAME_FILTER: autobot
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-watchtowers

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-validation-eval0-videos-watchtowers-evaluator:2019_06_11_19_03_32@sha256:5876da21ebc74f8c77acec009b39b23c52b9f374eef76e182936e1ad8d05ae86
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '0'
            BAG_NAME_FILTER: watchtower
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval1

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-validation-eval1-evaluator:2019_06_11_19_00_39@sha256:04dad2af97033eef32f70bfea1f9fad34663a5ad17f7e998c7984bb68012de88
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval1-videos-autobots

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-validation-eval1-videos-autobots-evaluator:2019_06_11_19_03_47@sha256:5876da21ebc74f8c77acec009b39b23c52b9f374eef76e182936e1ad8d05ae86
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '1'
            BAG_NAME_FILTER: autobot
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval1-videos-watchtowers

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-validation-eval1-videos-watchtowers-evaluator:2019_06_11_19_04_02@sha256:5876da21ebc74f8c77acec009b39b23c52b9f374eef76e182936e1ad8d05ae86
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '0'
            BAG_NAME_FILTER: watchtower
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step