Challenge "aido2-LF-real-validation-offseason"

These are the metrics defined:

Traveled distance - driven_lanedir_consec_median

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Survival time - survival_time_median

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle.

Evaluation steps details

• At the beginning execute step eval0.

• If step eval0 has result success, then execute step eval0-videos-autobots.

• If step eval0 has result failed, then declare the submission FAILED.

• If step eval0 has result error, then declare the submission ERROR.

• If step eval0-videos-autobots has result success, then execute step eval0-videos-watchtowers.

• If step eval0-videos-autobots has result failed, then execute step eval0-extractATs.

• If step eval0-videos-autobots has result error, then execute step eval0-extractATs.

• If step eval0-videos-watchtowers has result success, then execute step eval0-extractATs.

• If step eval0-videos-watchtowers has result failed, then execute step eval0-extractATs.

• If step eval0-videos-watchtowers has result error, then execute step eval0-extractATs.

• If step eval0-extractATs has result success, then execute step eval0-WesQuackerson.

• If step eval0-extractATs has result failed, then declare the submission FAILED.

• If step eval0-extractATs has result error, then declare the submission ERROR.

• If step eval0-WesQuackerson has result success, then execute step eval0-localization.

• If step eval0-WesQuackerson has result failed, then execute step eval0-localization.

• If step eval0-WesQuackerson has result error, then execute step eval0-localization.

• If step eval0-localization has result success, then declare the submission SUCCESS.

• If step eval0-localization has result failed, then declare the submission FAILED.

• If step eval0-localization has result error, then declare the submission ERROR.

Evaluation step eval0

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-offseason-eval0-evaluator:2019_06_19_16_12_11@sha256:1cc1f34ccf9060b7ef286d8dfcf0e1815d5a220d5850a342090537ed313f7ea3
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval1

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-offseason-eval1-evaluator:2019_06_19_16_13_07@sha256:1cc1f34ccf9060b7ef286d8dfcf0e1815d5a220d5850a342090537ed313f7ea3
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-autobots

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-offseason-eval0-videos-autobots-evaluator:2019_06_19_16_13_19@sha256:a7425ff06a130654f37ddf2d5ad75760d65fc29c60bab935e96a88db7413c7f8
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '1'
            BAG_NAME_FILTER: autobot
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-watchtowers

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-offseason-eval0-videos-watchtowers-evaluator:2019_06_19_16_27_08@sha256:a7425ff06a130654f37ddf2d5ad75760d65fc29c60bab935e96a88db7413c7f8
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '0'
            BAG_NAME_FILTER: watchtower
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-extractATs

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-offseason-eval0-extractats-evaluator:2019_06_19_16_27_28@sha256:e8754d9dda791175a48761aed021863c57c3fdd503e06bbf7ad5da3edcae8140
        environment:
            BAGS_PATH: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            ACQ_OUTPUT_BAG: /challenges/challenge-evaluation-output/processed.bag
            ACQ_TEST_STREAM: '0'
            ACQ_DEVICE_THREADS: '6'
            ACQ_OBSERVATIONS_STATISTICS_OUTPUT: /challenges/challenge-evaluation-output/statistics.yaml

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-WesQuackerson

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-offseason-eval0-wesquackerson-evaluator:2019_06_19_16_29_15@sha256:f08d21880f67096f78a3948136327d0067f5df3e037b15c391b51aa6f4827acb
        environment:
            ATMSGS_BAG: /challenges/previous-steps/eval0-extractATs/challenge-evaluation-output/processed.bag
            TITLE_CARD: '1'
            VIDEO_BAGS: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_FILE: /challenges/challenge-evaluation-output/overheadVideo.mp4
            POSES_TOPIC: /poses_acquisition/poses
            VIDEO_TOPIC: /camera_node/image/compressed
            MIN_SHOT_LENGTH: '12'
            OUTPUT_FRAMERATE: '12'
            PREVIOUS_RESULTS: /challenges/previous-steps/eval0/challenge-evaluation-output/challenge-results/challenge_results.yaml
            CAMERA_RESOLUTION_WIDTH: '972'
            CAMERA_RESOLUTION_HEIGHT: '1296'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-localization

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-validation-offseason-eval0-localization-evaluator:2019_06_19_16_31_04@sha256:ee4b5676af8bfd33fe2f3bfad1f866414295f5f03ffa9e468f54ac7eaa14bad1
        environment:
            ATMSGS_BAG: /challenges/previous-steps/eval0-extractATs/challenge-evaluation-output/processed.bag
            OUTPUT_DIR: /challenges/challenge-evaluation-output

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step