Challenge "aido2-LF-real-testing"

These are the metrics defined:

Traveled distance - driven_lanedir_consec_median

This is the median distance traveled, along a lane. (That is, going in circles will not make this metric increase.)

This is discretized to tiles.

Survival time - survival_time_median

This is the median survival time. The simulation is terminated when the car goes outside of the road or it crashes with an obstacle.

Evaluation steps details

• At the beginning execute step eval0.

• If step eval0 has result success, then execute step eval0-videos-autobots.

• If step eval0 has result failed, then declare the submission FAILED.

• If step eval0 has result error, then declare the submission ERROR.

• If step eval0-videos-autobots has result success, then execute step eval0-videos-watchtowers.

• If step eval0-videos-autobots has result failed, then execute step eval0-extractATs.

• If step eval0-videos-autobots has result error, then execute step eval0-extractATs.

• If step eval0-videos-watchtowers has result success, then execute step eval0-extractATs.

• If step eval0-videos-watchtowers has result failed, then execute step eval0-extractATs.

• If step eval0-videos-watchtowers has result error, then execute step eval0-extractATs.

• If step eval0-extractATs has result success, then execute step eval0-WesQuackerson.

• If step eval0-extractATs has result failed, then declare the submission FAILED.

• If step eval0-extractATs has result error, then declare the submission ERROR.

• If step eval0-WesQuackerson has result success, then declare the submission SUCCESS.

• If step eval0-WesQuackerson has result failed, then declare the submission FAILED.

• If step eval0-WesQuackerson has result error, then declare the submission ERROR.

Evaluation step hw-evaluation

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/amaurx/aido2-lf-real-testing-hw-evaluation-evaluator:2019_05_13_17_48_19@sha256:1168d196612a40db2eedce3f412838293600898ed3f6a96dccc2679c82acee56
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0

Timeout 18000.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-testing-eval0-evaluator:2019_06_11_19_07_10@sha256:04dad2af97033eef32f70bfea1f9fad34663a5ad17f7e998c7984bb68012de88
        environment: {}
        ports:
        - 8005:8005

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos

Timeout 1800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/andreacensi/aido2-lf-real-testing-eval0-videos-evaluator:2019_05_15_16_22_04@sha256:440645bb2e89727086a010c45c71b3d069b60489343501e05eca8f5c9108ee3c
        environment:
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-0

Timeout 1800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/aleksandarpetrov/aido2-lf-real-testing-eval0-videos-0-evaluator:2019_05_16_07_22_45@sha256:90e7c45aff3c2936cca59c2eb16668e882cc06a377ce9315900496a8efaaeb61
        environment:
            WORKER_I: '0'
            WORKER_N: '2'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-1

Timeout 1800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/aleksandarpetrov/aido2-lf-real-testing-eval0-videos-1-evaluator:2019_05_16_07_33_00@sha256:90e7c45aff3c2936cca59c2eb16668e882cc06a377ce9315900496a8efaaeb61
        environment:
            WORKER_I: '1'
            WORKER_N: '2'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-extractATs

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-testing-eval0-extractats-evaluator:2019_05_20_11_23_06@sha256:419dbf15fc26c5a361678e984c0a9c852298fc27e6b6e2444633f561f60f006a
        environment:
            BAGS_PATH: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            ACQ_OUTPUT_BAG: /challenges/challenge-evaluation-output/processed.bag
            ACQ_TEST_STREAM: '0'
            ACQ_DEVICE_THREADS: '6'
            ACQ_OBSERVATIONS_STATISTICS_OUTPUT: /challenges/challenge-evaluation-output/statistics.yaml

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-WesQuackerson

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-testing-eval0-wesquackerson-evaluator:2019_05_20_11_23_38@sha256:410e1c87425a36638404166049a50dada92d2a819e18f859858bd3eed0a59978
        environment:
            ATMSGS_BAG: /challenges/previous-steps/eval0-extractATs/challenge-evaluation-output/processed.bag
            TITLE_CARD: '1'
            VIDEO_BAGS: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_FILE: /challenges/challenge-evaluation-output/overheadVideo.mp4
            POSES_TOPIC: /poses_acquisition/poses
            VIDEO_TOPIC: /camera_node/image/compressed
            MIN_SHOT_LENGTH: '12'
            OUTPUT_FRAMERATE: '12'
            PREVIOUS_RESULTS: /challenges/previous-steps/eval0/challenge-evaluation-output/challenge-results/challenge_results.yaml
            CAMERA_RESOLUTION_WIDTH: '972'
            CAMERA_RESOLUTION_HEIGHT: '1296'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-autobots

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-testing-eval0-videos-autobots-evaluator:2019_06_11_19_07_22@sha256:5876da21ebc74f8c77acec009b39b23c52b9f374eef76e182936e1ad8d05ae86
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '1'
            BAG_NAME_FILTER: autobot
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step

Evaluation step eval0-videos-watchtowers

Timeout 10800.0

This is the Docker Compose configuration skeleton:

version: '3'
services:
    evaluator:
        image: docker.io/gian1717/aido2-lf-real-testing-eval0-videos-watchtowers-evaluator:2019_06_11_19_07_37@sha256:5876da21ebc74f8c77acec009b39b23c52b9f374eef76e182936e1ad8d05ae86
        environment:
            WORKER_I: '0'
            WORKER_N: '1'
            INPUT_DIR: /challenges/previous-steps/eval0/challenge-evaluation-output/raw_logs/bags
            OUTPUT_DIR: /challenges/challenge-evaluation-output
            DEBUG_OVERLAY: '0'
            BAG_NAME_FILTER: watchtower
            OUTPUT_FRAMERATE: '12'

The text SUBMISSION_CONTAINER will be replaced with the user containter.

Resources required for evaluating this step