AI Driving Olympics Home Challenges Submissions

Submission 13284

Submission13284
Competingyes
Challengeaido5-LF-real-validation
UserAndrás Kalapos 🇭🇺
Date submitted
Last status update
Completecomplete
DetailsEvaluation is complete.
Sisters
Result💚
Jobseval0: 60547 eval0-videos: 61564 eval0-visualize: 61567 eval1: 60551 eval1-videos: 61570 eval1-visualize: 61565 eval2: 60578 eval2-videos: 61568 eval2-visualize: 61569
Next
User labelreal-v1.0-3092-363
Admin priority150
Blessingn/a
User priority150

61570

robot

watchtower

61569

Click the images to see detailed statistics about the episode.

sync

visualization

61568

robot

watchtower

61567

Click the images to see detailed statistics about the episode.

sync

visualization

61565

Click the images to see detailed statistics about the episode.

sync

visualization

61564

robot

watchtower

60578

scenario/drawing.html

60551

scenario/drawing.html

60547

scenario/drawing.html

Evaluation jobs for this submission

See previous jobs for previous versions of challenges
Job IDstepstatusup to datedate starteddate completeddurationmessage
61570eval1-videossuccessyes0:02:30
all ok, 8 bags proce [...]
all ok, 8 bags processed
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
61569eval2-visualizesuccessyes0:01:13
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec1.538375804350982
survival_time12.901892185211182
deviation-center-line0.5271305418699314
in-drivable-lane7.605623722076416


other stats
deviation-heading1.874646830288386
driven_any2.3729383598200204
driven_lanedir1.538375804350982
visualized-eval2-passed1
No reset possible
61568eval2-videossuccessyes0:02:36
all ok, 8 bags proce [...]
all ok, 8 bags processed
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
61567eval0-visualizesuccessyes0:01:28
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec0.5095739644743909
survival_time5.999385833740234
deviation-center-line0.18664647817264143
in-drivable-lane1.9999778270721436


other stats
deviation-heading1.093488464947737
driven_any0.6315599174515889
driven_lanedir0.5095739644743909
visualized-eval0-passed1
No reset possible
61565eval1-visualizesuccessyes0:01:51
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
driven_lanedir_consec0.6129078664816039
survival_time8.403920650482178
deviation-center-line0.10089285772150086
in-drivable-lane6.505793333053589


other stats
deviation-heading1.0347678980734258
driven_any1.3836701285671344
driven_lanedir0.6129078664816039
visualized-eval1-passed1
No reset possible
61564eval0-videossuccessyes0:01:29
all ok, 8 bags proce [...]
all ok, 8 bags processed
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60578eval2successyes0:09:14
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60564eval2abortedyes0:03:04
Job shutdown [...]
Job shutdown
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60563eval2abortedyes0:08:23
Operator message: '' [...]
Operator message: ''
Logs:
DEBUG:commons:version: 6.1.7 *
INFO:typing:version: 6.1.8
DEBUG:aido_schemas:aido-protocols version 6.0.33 path /usr/local/lib/python3.8/dist-packages
INFO:nodes:version 6.1.1 path /usr/local/lib/python3.8/dist-packages pyparsing 2.4.6
2020-12-11 22:39:20.295126: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-11 22:39:23.583160: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:
2020-12-11 22:39:23.583197: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-12-11 22:39:23.583241: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
DEBUG:ipce:version 6.0.36 path /usr/local/lib/python3.8/dist-packages
INFO:nodes_wrapper:checking implementation
INFO:nodes_wrapper:checking implementation OK
DEBUG:nodes_wrapper:run_loop
  fin: /fifos/ego0-in
 fout: fifo:/fifos/ego0-out
INFO:nodes_wrapper:Fifo /fifos/ego0-out created. I will block until a reader appears.
INFO:nodes_wrapper:Fifo reader appeared for /fifos/ego0-out.
INFO:nodes_wrapper:Node RLlibAgent starting reading
 fi_desc: /fifos/ego0-in
 fo_desc: fifo:/fifos/ego0-out
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: init()
WARNING:config.config:Found paths with seed 3092:
WARNING:config.config:0: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/config_dump_3092.yml
WARNING:config.config:Found checkpoints in ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47:
WARNING:config.config:0: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
WARNING:config.config:Config loaded from ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/config_dump_3092.yml
WARNING:config.config:Model checkpoint loaded from ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
WARNING:config.config:Updating default config values by: 
 env_config:
  mode: inference

WARNING:config.config:Env_config.mode is 'inference', some hyperparameters will be overwritten by: 
 rllib_config:
  num_workers: 0
  num_gpus: 0
  callbacks: {}
ray_init_config:
  num_cpus: 1
  memory: 2097152000
  object_store_memory: 209715200
  redis_max_memory: 209715200
  local_mode: true

INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: === Wrappers ===================================
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: Observation wrappers
 <ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>
<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>
<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>
<NormalizeWrapper<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>>
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: Action wrappers
 <Heading2WheelVelsWrapper<NormalizeWrapper<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>>>
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: Reward wrappers
 
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: === Config ===================================
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: seed: 3092
experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
algo: PPO
algo_config_files:
  PPO: config/algo/ppo.yml
  general: config/algo/general.yml
env_config:
  mode: inference
  episode_max_steps: 500
  resized_input_shape: (84, 84)
  crop_image_top: true
  top_crop_divider: 3
  grayscale_image: false
  frame_stacking: true
  frame_stacking_depth: 3
  motion_blur: false
  action_type: heading
  reward_function: posangle
  distortion: true
  accepted_start_angle_deg: 30
  simulation_framerate: 30
  frame_skip: 3
  action_delay_ratio: 0.0
  training_map: multimap_aido5
  domain_rand: true
  dynamics_rand: true
  camera_rand: true
  frame_repeating: 0.0
  spawn_obstacles: false
  obstacles:
    duckie:
      density: 0.5
      static: true
    duckiebot:
      density: 0
      static: false
  spawn_forward_obstacle: false
  aido_wrapper: true
  wandb:
    project: duckietown-rllib
  experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
  seed: 3092
ray_init_config:
  num_cpus: 1
  webui_host: 127.0.0.1
  memory: 2097152000
  object_store_memory: 209715200
  redis_max_memory: 209715200
  local_mode: true
restore_seed: 3091
restore_experiment_idx: 0
restore_checkpoint_idx: 0
debug_hparams:
  rllib_config:
    num_workers: 1
    num_gpus: 0
  ray_init_config:
    num_cpus: 1
    memory: 2097152000
    object_store_memory: 209715200
    redis_max_memory: 209715200
    local_mode: true
inference_hparams:
  rllib_config:
    num_workers: 0
    num_gpus: 0
    callbacks: {}
  ray_init_config:
    num_cpus: 1
    memory: 2097152000
    object_store_memory: 209715200
    redis_max_memory: 209715200
    local_mode: true
timesteps_total: 4000000.0
rllib_config:
  num_workers: 0
  sample_batch_size: 265
  num_gpus: 0
  train_batch_size: 4096
  gamma: 0.99
  lr: 5.0e-05
  monitor: false
  evaluation_interval: 25
  evaluation_num_episodes: 2
  evaluation_config:
    monitor: false
    explore: false
  seed: 1234
  lambda: 0.95
  sgd_minibatch_size: 128
  vf_loss_coeff: 0.5
  entropy_coeff: 0.0
  clip_param: 0.2
  vf_clip_param: 0.2
  grad_clip: 0.5
  env: Duckietown
  callbacks: {}
  env_config:
    mode: inference
    episode_max_steps: 500
    resized_input_shape: (84, 84)
    crop_image_top: true
    top_crop_divider: 3
    grayscale_image: false
    frame_stacking: true
    frame_stacking_depth: 3
    motion_blur: false
    action_type: heading
    reward_function: posangle
    distortion: true
    accepted_start_angle_deg: 30
    simulation_framerate: 30
    frame_skip: 3
    action_delay_ratio: 0.0
    training_map: multimap_aido5
    domain_rand: true
    dynamics_rand: true
    camera_rand: true
    frame_repeating: 0.0
    spawn_obstacles: false
    obstacles:
      duckie:
        density: 0.5
        static: true
      duckiebot:
        density: 0
        static: false
    spawn_forward_obstacle: false
    aido_wrapper: true
    wandb:
      project: duckietown-rllib
    experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
    seed: 3092

2020-12-11 22:39:24,141	INFO trainer.py:428 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2020-12-11 22:39:24,162	ERROR syncer.py:39 -- Log sync requires rsync to be installed.
2020-12-11 22:39:24,162	WARNING deprecation.py:29 -- DeprecationWarning: `sample_batch_size` has been deprecated. Use `rollout_fragment_length` instead. This will raise an error in the future!
2020-12-11 22:39:24,162	INFO trainer.py:583 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
2020-12-11 22:39:24.177929: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-11 22:39:24.188468: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2599935000 Hz
2020-12-11 22:39:24.188993: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x676c000 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-11 22:39:24.189021: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-11 22:39:29,457	INFO trainable.py:217 -- Getting current IP.
2020-12-11 22:39:29,457	WARNING util.py:37 -- Install gputil for GPU system monitoring.
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: Restoring checkpoint from: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
2020-12-11 22:39:29,562	INFO trainable.py:217 -- Getting current IP.
2020-12-11 22:39:29,562	INFO trainable.py:422 -- Restored on 172.17.0.2 from checkpoint: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
2020-12-11 22:39:29,562	INFO trainable.py:430 -- Current state after restoring: {'_iteration': 363, '_timesteps_total': 1539120, '_time_total': 110224.9016327858, '_episodes_total': 8614}
INFO:nodes_wrapper:c70e6aa61a1f:RLlibAgent: Starting episode "episode".
ERROR:nodes_wrapper:Error in node RLlibAgent: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
    handle_message_node(parsed, receiver0, context0)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
    call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
    f(**kwargs)
  File "solution.py", line 80, in on_received_get_commands
    pwm_left, pwm_right = self.compute_action(self.current_image)
  File "solution.py", line 73, in compute_action
    action = self.model.predict(observation)
  File "/submission/model.py", line 63, in predict
    action = self.model.compute_action(observation, explore=False)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
    result = self.get_policy(policy_id).compute_single_action(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
    [action], state_out, info = self.compute_actions(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
    return builder.get(fetches)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
    self._executed = run_timeline(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
    fetches = sess.run(ops, feed_dict=feed_dict)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
  File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
    sys.exit(signal.SIGTERM)
SystemExit: Signals.SIGTERM

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
    loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 378, in loop
    raise InternalProblem(msg) from e  # XXX
zuper_nodes.structures.InternalProblem: Exception while handling a message on topic "get_commands".

| Traceback (most recent call last):
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
|     handle_message_node(parsed, receiver0, context0)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
|     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|     f(**kwargs)
|   File "solution.py", line 80, in on_received_get_commands
|     pwm_left, pwm_right = self.compute_action(self.current_image)
|   File "solution.py", line 73, in compute_action
|     action = self.model.predict(observation)
|   File "/submission/model.py", line 63, in predict
|     action = self.model.compute_action(observation, explore=False)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
|     result = self.get_policy(policy_id).compute_single_action(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
|     [action], state_out, info = self.compute_actions(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
|     return builder.get(fetches)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
|     self._executed = run_timeline(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
|     fetches = sess.run(ops, feed_dict=feed_dict)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|     result = self._run(None, fetches, feed_dict, options_ptr,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|     results = self._do_run(handle, final_targets, final_fetches,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|     return self._do_call(_run_fn, feeds, fetches, targets, options,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|     return fn(*args)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|     return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|     return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|   File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
|     sys.exit(signal.SIGTERM)
| SystemExit: Signals.SIGTERM
| 

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
    handle_message_node(parsed, receiver0, context0)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
    call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
    f(**kwargs)
  File "solution.py", line 80, in on_received_get_commands
    pwm_left, pwm_right = self.compute_action(self.current_image)
  File "solution.py", line 73, in compute_action
    action = self.model.predict(observation)
  File "/submission/model.py", line 63, in predict
    action = self.model.compute_action(observation, explore=False)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
    result = self.get_policy(policy_id).compute_single_action(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
    [action], state_out, info = self.compute_actions(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
    return builder.get(fetches)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
    self._executed = run_timeline(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
    fetches = sess.run(ops, feed_dict=feed_dict)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
  File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
    sys.exit(signal.SIGTERM)
SystemExit: Signals.SIGTERM

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
    loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 378, in loop
    raise InternalProblem(msg) from e  # XXX
zuper_nodes.structures.InternalProblem: Exception while handling a message on topic "get_commands".

| Traceback (most recent call last):
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
|     handle_message_node(parsed, receiver0, context0)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
|     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|     f(**kwargs)
|   File "solution.py", line 80, in on_received_get_commands
|     pwm_left, pwm_right = self.compute_action(self.current_image)
|   File "solution.py", line 73, in compute_action
|     action = self.model.predict(observation)
|   File "/submission/model.py", line 63, in predict
|     action = self.model.compute_action(observation, explore=False)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
|     result = self.get_policy(policy_id).compute_single_action(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
|     [action], state_out, info = self.compute_actions(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
|     return builder.get(fetches)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
|     self._executed = run_timeline(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
|     fetches = sess.run(ops, feed_dict=feed_dict)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|     result = self._run(None, fetches, feed_dict, options_ptr,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|     results = self._do_run(handle, final_targets, final_fetches,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|     return self._do_call(_run_fn, feeds, fetches, targets, options,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|     return fn(*args)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|     return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|     return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|   File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
|     sys.exit(signal.SIGTERM)
| SystemExit: Signals.SIGTERM
| 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "solution.py", line 127, in <module>
    main()
  File "solution.py", line 123, in main
    wrap_direct(node=node, protocol=protocol)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
    run_loop(node, protocol, args)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 251, in run_loop
    raise Exception(msg) from e
Exception: Error in node RLlibAgent
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60560eval2abortedyes0:07:04
Operator message: '' [...]
Operator message: ''
Logs:
DEBUG:commons:version: 6.1.7 *
INFO:typing:version: 6.1.8
DEBUG:aido_schemas:aido-protocols version 6.0.33 path /usr/local/lib/python3.8/dist-packages
INFO:nodes:version 6.1.1 path /usr/local/lib/python3.8/dist-packages pyparsing 2.4.6
2020-12-11 22:29:03.403854: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-11 22:29:06.557963: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:
2020-12-11 22:29:06.557996: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-12-11 22:29:06.558026: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
DEBUG:ipce:version 6.0.36 path /usr/local/lib/python3.8/dist-packages
INFO:nodes_wrapper:checking implementation
INFO:nodes_wrapper:checking implementation OK
DEBUG:nodes_wrapper:run_loop
  fin: /fifos/ego0-in
 fout: fifo:/fifos/ego0-out
INFO:nodes_wrapper:Fifo /fifos/ego0-out created. I will block until a reader appears.
INFO:nodes_wrapper:Fifo reader appeared for /fifos/ego0-out.
INFO:nodes_wrapper:Node RLlibAgent starting reading
 fi_desc: /fifos/ego0-in
 fo_desc: fifo:/fifos/ego0-out
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: init()
WARNING:config.config:Found paths with seed 3092:
WARNING:config.config:0: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/config_dump_3092.yml
WARNING:config.config:Found checkpoints in ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47:
WARNING:config.config:0: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
WARNING:config.config:Config loaded from ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/config_dump_3092.yml
WARNING:config.config:Model checkpoint loaded from ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
WARNING:config.config:Updating default config values by: 
 env_config:
  mode: inference

WARNING:config.config:Env_config.mode is 'inference', some hyperparameters will be overwritten by: 
 rllib_config:
  num_workers: 0
  num_gpus: 0
  callbacks: {}
ray_init_config:
  num_cpus: 1
  memory: 2097152000
  object_store_memory: 209715200
  redis_max_memory: 209715200
  local_mode: true

INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: === Wrappers ===================================
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: Observation wrappers
 <ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>
<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>
<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>
<NormalizeWrapper<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>>
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: Action wrappers
 <Heading2WheelVelsWrapper<NormalizeWrapper<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>>>
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: Reward wrappers
 
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: === Config ===================================
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: seed: 3092
experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
algo: PPO
algo_config_files:
  PPO: config/algo/ppo.yml
  general: config/algo/general.yml
env_config:
  mode: inference
  episode_max_steps: 500
  resized_input_shape: (84, 84)
  crop_image_top: true
  top_crop_divider: 3
  grayscale_image: false
  frame_stacking: true
  frame_stacking_depth: 3
  motion_blur: false
  action_type: heading
  reward_function: posangle
  distortion: true
  accepted_start_angle_deg: 30
  simulation_framerate: 30
  frame_skip: 3
  action_delay_ratio: 0.0
  training_map: multimap_aido5
  domain_rand: true
  dynamics_rand: true
  camera_rand: true
  frame_repeating: 0.0
  spawn_obstacles: false
  obstacles:
    duckie:
      density: 0.5
      static: true
    duckiebot:
      density: 0
      static: false
  spawn_forward_obstacle: false
  aido_wrapper: true
  wandb:
    project: duckietown-rllib
  experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
  seed: 3092
ray_init_config:
  num_cpus: 1
  webui_host: 127.0.0.1
  memory: 2097152000
  object_store_memory: 209715200
  redis_max_memory: 209715200
  local_mode: true
restore_seed: 3091
restore_experiment_idx: 0
restore_checkpoint_idx: 0
debug_hparams:
  rllib_config:
    num_workers: 1
    num_gpus: 0
  ray_init_config:
    num_cpus: 1
    memory: 2097152000
    object_store_memory: 209715200
    redis_max_memory: 209715200
    local_mode: true
inference_hparams:
  rllib_config:
    num_workers: 0
    num_gpus: 0
    callbacks: {}
  ray_init_config:
    num_cpus: 1
    memory: 2097152000
    object_store_memory: 209715200
    redis_max_memory: 209715200
    local_mode: true
timesteps_total: 4000000.0
rllib_config:
  num_workers: 0
  sample_batch_size: 265
  num_gpus: 0
  train_batch_size: 4096
  gamma: 0.99
  lr: 5.0e-05
  monitor: false
  evaluation_interval: 25
  evaluation_num_episodes: 2
  evaluation_config:
    monitor: false
    explore: false
  seed: 1234
  lambda: 0.95
  sgd_minibatch_size: 128
  vf_loss_coeff: 0.5
  entropy_coeff: 0.0
  clip_param: 0.2
  vf_clip_param: 0.2
  grad_clip: 0.5
  env: Duckietown
  callbacks: {}
  env_config:
    mode: inference
    episode_max_steps: 500
    resized_input_shape: (84, 84)
    crop_image_top: true
    top_crop_divider: 3
    grayscale_image: false
    frame_stacking: true
    frame_stacking_depth: 3
    motion_blur: false
    action_type: heading
    reward_function: posangle
    distortion: true
    accepted_start_angle_deg: 30
    simulation_framerate: 30
    frame_skip: 3
    action_delay_ratio: 0.0
    training_map: multimap_aido5
    domain_rand: true
    dynamics_rand: true
    camera_rand: true
    frame_repeating: 0.0
    spawn_obstacles: false
    obstacles:
      duckie:
        density: 0.5
        static: true
      duckiebot:
        density: 0
        static: false
    spawn_forward_obstacle: false
    aido_wrapper: true
    wandb:
      project: duckietown-rllib
    experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
    seed: 3092

2020-12-11 22:29:07,019	INFO trainer.py:428 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2020-12-11 22:29:07,036	ERROR syncer.py:39 -- Log sync requires rsync to be installed.
2020-12-11 22:29:07,037	WARNING deprecation.py:29 -- DeprecationWarning: `sample_batch_size` has been deprecated. Use `rollout_fragment_length` instead. This will raise an error in the future!
2020-12-11 22:29:07,037	INFO trainer.py:583 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
2020-12-11 22:29:07.050348: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-11 22:29:07.061066: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2599935000 Hz
2020-12-11 22:29:07.061705: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x7972150 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-11 22:29:07.061754: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-11 22:29:12,040	INFO trainable.py:217 -- Getting current IP.
2020-12-11 22:29:12,041	WARNING util.py:37 -- Install gputil for GPU system monitoring.
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: Restoring checkpoint from: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
2020-12-11 22:29:12,106	INFO trainable.py:217 -- Getting current IP.
2020-12-11 22:29:12,106	INFO trainable.py:422 -- Restored on 172.17.0.2 from checkpoint: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
2020-12-11 22:29:12,106	INFO trainable.py:430 -- Current state after restoring: {'_iteration': 363, '_timesteps_total': 1539120, '_time_total': 110224.9016327858, '_episodes_total': 8614}
INFO:nodes_wrapper:4dd9d5a4eb8d:RLlibAgent: Starting episode "episode".
ERROR:nodes_wrapper:Error in node RLlibAgent: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
    handle_message_node(parsed, receiver0, context0)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
    call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
    f(**kwargs)
  File "solution.py", line 80, in on_received_get_commands
    pwm_left, pwm_right = self.compute_action(self.current_image)
  File "solution.py", line 73, in compute_action
    action = self.model.predict(observation)
  File "/submission/model.py", line 63, in predict
    action = self.model.compute_action(observation, explore=False)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
    result = self.get_policy(policy_id).compute_single_action(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
    [action], state_out, info = self.compute_actions(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
    return builder.get(fetches)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
    self._executed = run_timeline(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
    fetches = sess.run(ops, feed_dict=feed_dict)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
  File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
    sys.exit(signal.SIGTERM)
SystemExit: Signals.SIGTERM

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
    loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 378, in loop
    raise InternalProblem(msg) from e  # XXX
zuper_nodes.structures.InternalProblem: Exception while handling a message on topic "get_commands".

| Traceback (most recent call last):
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
|     handle_message_node(parsed, receiver0, context0)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
|     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|     f(**kwargs)
|   File "solution.py", line 80, in on_received_get_commands
|     pwm_left, pwm_right = self.compute_action(self.current_image)
|   File "solution.py", line 73, in compute_action
|     action = self.model.predict(observation)
|   File "/submission/model.py", line 63, in predict
|     action = self.model.compute_action(observation, explore=False)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
|     result = self.get_policy(policy_id).compute_single_action(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
|     [action], state_out, info = self.compute_actions(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
|     return builder.get(fetches)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
|     self._executed = run_timeline(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
|     fetches = sess.run(ops, feed_dict=feed_dict)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|     result = self._run(None, fetches, feed_dict, options_ptr,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|     results = self._do_run(handle, final_targets, final_fetches,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|     return self._do_call(_run_fn, feeds, fetches, targets, options,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|     return fn(*args)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|     return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|     return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|   File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
|     sys.exit(signal.SIGTERM)
| SystemExit: Signals.SIGTERM
| 

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
    handle_message_node(parsed, receiver0, context0)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
    call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
    f(**kwargs)
  File "solution.py", line 80, in on_received_get_commands
    pwm_left, pwm_right = self.compute_action(self.current_image)
  File "solution.py", line 73, in compute_action
    action = self.model.predict(observation)
  File "/submission/model.py", line 63, in predict
    action = self.model.compute_action(observation, explore=False)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
    result = self.get_policy(policy_id).compute_single_action(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
    [action], state_out, info = self.compute_actions(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
    return builder.get(fetches)
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
    self._executed = run_timeline(
  File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
    fetches = sess.run(ops, feed_dict=feed_dict)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
    result = self._run(None, fetches, feed_dict, options_ptr,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
    results = self._do_run(handle, final_targets, final_fetches,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
    return self._do_call(_run_fn, feeds, fetches, targets, options,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
    return self._call_tf_sessionrun(options, feed_dict, fetch_list,
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
    return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
  File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
    sys.exit(signal.SIGTERM)
SystemExit: Signals.SIGTERM

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
    loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 378, in loop
    raise InternalProblem(msg) from e  # XXX
zuper_nodes.structures.InternalProblem: Exception while handling a message on topic "get_commands".

| Traceback (most recent call last):
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
|     handle_message_node(parsed, receiver0, context0)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
|     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|     f(**kwargs)
|   File "solution.py", line 80, in on_received_get_commands
|     pwm_left, pwm_right = self.compute_action(self.current_image)
|   File "solution.py", line 73, in compute_action
|     action = self.model.predict(observation)
|   File "/submission/model.py", line 63, in predict
|     action = self.model.compute_action(observation, explore=False)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/agents/trainer.py", line 781, in compute_action
|     result = self.get_policy(policy_id).compute_single_action(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/policy.py", line 150, in compute_single_action
|     [action], state_out, info = self.compute_actions(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/policy/tf_policy.py", line 268, in compute_actions
|     return builder.get(fetches)
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 42, in get
|     self._executed = run_timeline(
|   File "/usr/local/lib/python3.8/dist-packages/ray/rllib/utils/tf_run_builder.py", line 89, in run_timeline
|     fetches = sess.run(ops, feed_dict=feed_dict)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 957, in run
|     result = self._run(None, fetches, feed_dict, options_ptr,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1180, in _run
|     results = self._do_run(handle, final_targets, final_fetches,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1358, in _do_run
|     return self._do_call(_run_fn, feeds, fetches, targets, options,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1365, in _do_call
|     return fn(*args)
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1349, in _run_fn
|     return self._call_tf_sessionrun(options, feed_dict, fetch_list,
|   File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/client/session.py", line 1441, in _call_tf_sessionrun
|     return tf_session.TF_SessionRun_wrapper(self._session, options, feed_dict,
|   File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
|     sys.exit(signal.SIGTERM)
| SystemExit: Signals.SIGTERM
| 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "solution.py", line 127, in <module>
    main()
  File "solution.py", line 123, in main
    wrap_direct(node=node, protocol=protocol)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
    run_loop(node, protocol, args)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 251, in run_loop
    raise Exception(msg) from e
Exception: Error in node RLlibAgent
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60557eval2abortedyes0:06:42
Job shutdown [...]
Job shutdown
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60554eval2abortedyes0:04:13
Job shutdown [...]
Job shutdown
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60551eval1successyes0:06:24
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60550eval1abortedyes0:00:33
Job shutdown [...]
Job shutdown
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60547eval0successyes0:07:28
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60546eval0timeoutyes----No reset possible
60545eval0abortedyes0:06:15
Operator message: '' [...]
Operator message: ''
Logs:
DEBUG:commons:version: 6.1.7 *
INFO:typing:version: 6.1.8
DEBUG:aido_schemas:aido-protocols version 6.0.33 path /usr/local/lib/python3.8/dist-packages
INFO:nodes:version 6.1.1 path /usr/local/lib/python3.8/dist-packages pyparsing 2.4.6
2020-12-11 21:10:57.484773: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.10.1
2020-12-11 21:11:00.240024: W tensorflow/stream_executor/platform/default/dso_loader.cc:59] Could not load dynamic library 'libcuda.so.1'; dlerror: libcuda.so.1: cannot open shared object file: No such file or directory; LD_LIBRARY_PATH: /usr/local/cuda/lib64:/usr/local/cuda/extras/CUPTI/lib64:
2020-12-11 21:11:00.240064: W tensorflow/stream_executor/cuda/cuda_driver.cc:312] failed call to cuInit: UNKNOWN ERROR (303)
2020-12-11 21:11:00.240112: I tensorflow/stream_executor/cuda/cuda_diagnostics.cc:163] no NVIDIA GPU device is present: /dev/nvidia0 does not exist
DEBUG:ipce:version 6.0.36 path /usr/local/lib/python3.8/dist-packages
INFO:nodes_wrapper:checking implementation
INFO:nodes_wrapper:checking implementation OK
DEBUG:nodes_wrapper:run_loop
  fin: /fifos/ego0-in
 fout: fifo:/fifos/ego0-out
INFO:nodes_wrapper:Fifo /fifos/ego0-out created. I will block until a reader appears.
INFO:nodes_wrapper:Fifo reader appeared for /fifos/ego0-out.
INFO:nodes_wrapper:Node RLlibAgent starting reading
 fi_desc: /fifos/ego0-in
 fo_desc: fifo:/fifos/ego0-out
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: init()
WARNING:config.config:Found paths with seed 3092:
WARNING:config.config:0: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/config_dump_3092.yml
WARNING:config.config:Found checkpoints in ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47:
WARNING:config.config:0: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
WARNING:config.config:Config loaded from ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/config_dump_3092.yml
WARNING:config.config:Model checkpoint loaded from ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
WARNING:config.config:Updating default config values by: 
 env_config:
  mode: inference

WARNING:config.config:Env_config.mode is 'inference', some hyperparameters will be overwritten by: 
 rllib_config:
  num_workers: 0
  num_gpus: 0
  callbacks: {}
ray_init_config:
  num_cpus: 1
  memory: 2097152000
  object_store_memory: 209715200
  redis_max_memory: 209715200
  local_mode: true

INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: === Wrappers ===================================
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: Observation wrappers
 <ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>
<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>
<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>
<NormalizeWrapper<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>>
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: Action wrappers
 <Heading2WheelVelsWrapper<NormalizeWrapper<ObservationBufferWrapper<ResizeWrapper<ClipImageWrapper<DummyDuckietownGymLikeEnv instance>>>>>>
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: Reward wrappers
 
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: === Config ===================================
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: seed: 3092
experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
algo: PPO
algo_config_files:
  PPO: config/algo/ppo.yml
  general: config/algo/general.yml
env_config:
  mode: inference
  episode_max_steps: 500
  resized_input_shape: (84, 84)
  crop_image_top: true
  top_crop_divider: 3
  grayscale_image: false
  frame_stacking: true
  frame_stacking_depth: 3
  motion_blur: false
  action_type: heading
  reward_function: posangle
  distortion: true
  accepted_start_angle_deg: 30
  simulation_framerate: 30
  frame_skip: 3
  action_delay_ratio: 0.0
  training_map: multimap_aido5
  domain_rand: true
  dynamics_rand: true
  camera_rand: true
  frame_repeating: 0.0
  spawn_obstacles: false
  obstacles:
    duckie:
      density: 0.5
      static: true
    duckiebot:
      density: 0
      static: false
  spawn_forward_obstacle: false
  aido_wrapper: true
  wandb:
    project: duckietown-rllib
  experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
  seed: 3092
ray_init_config:
  num_cpus: 1
  webui_host: 127.0.0.1
  memory: 2097152000
  object_store_memory: 209715200
  redis_max_memory: 209715200
  local_mode: true
restore_seed: 3091
restore_experiment_idx: 0
restore_checkpoint_idx: 0
debug_hparams:
  rllib_config:
    num_workers: 1
    num_gpus: 0
  ray_init_config:
    num_cpus: 1
    memory: 2097152000
    object_store_memory: 209715200
    redis_max_memory: 209715200
    local_mode: true
inference_hparams:
  rllib_config:
    num_workers: 0
    num_gpus: 0
    callbacks: {}
  ray_init_config:
    num_cpus: 1
    memory: 2097152000
    object_store_memory: 209715200
    redis_max_memory: 209715200
    local_mode: true
timesteps_total: 4000000.0
rllib_config:
  num_workers: 0
  sample_batch_size: 265
  num_gpus: 0
  train_batch_size: 4096
  gamma: 0.99
  lr: 5.0e-05
  monitor: false
  evaluation_interval: 25
  evaluation_num_episodes: 2
  evaluation_config:
    monitor: false
    explore: false
  seed: 1234
  lambda: 0.95
  sgd_minibatch_size: 128
  vf_loss_coeff: 0.5
  entropy_coeff: 0.0
  clip_param: 0.2
  vf_clip_param: 0.2
  grad_clip: 0.5
  env: Duckietown
  callbacks: {}
  env_config:
    mode: inference
    episode_max_steps: 500
    resized_input_shape: (84, 84)
    crop_image_top: true
    top_crop_divider: 3
    grayscale_image: false
    frame_stacking: true
    frame_stacking_depth: 3
    motion_blur: false
    action_type: heading
    reward_function: posangle
    distortion: true
    accepted_start_angle_deg: 30
    simulation_framerate: 30
    frame_skip: 3
    action_delay_ratio: 0.0
    training_map: multimap_aido5
    domain_rand: true
    dynamics_rand: true
    camera_rand: true
    frame_repeating: 0.0
    spawn_obstacles: false
    obstacles:
      duckie:
        density: 0.5
        static: true
      duckiebot:
        density: 0
        static: false
    spawn_forward_obstacle: false
    aido_wrapper: true
    wandb:
      project: duckietown-rllib
    experiment_name: PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand
    seed: 3092

2020-12-11 21:11:03,758	INFO trainer.py:428 -- Tip: set 'eager': true or the --eager flag to enable TensorFlow eager execution
2020-12-11 21:11:03,777	ERROR syncer.py:39 -- Log sync requires rsync to be installed.
2020-12-11 21:11:03,778	WARNING deprecation.py:29 -- DeprecationWarning: `sample_batch_size` has been deprecated. Use `rollout_fragment_length` instead. This will raise an error in the future!
2020-12-11 21:11:03,778	INFO trainer.py:583 -- Current log_level is WARN. For more information, set 'log_level': 'INFO' / 'DEBUG' or use the -v and -vv flags.
2020-12-11 21:11:03.793170: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations:  AVX2 FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2020-12-11 21:11:03.800627: I tensorflow/core/platform/profile_utils/cpu_utils.cc:104] CPU Frequency: 2599935000 Hz
2020-12-11 21:11:03.801289: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x90734d0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-12-11 21:11:03.801341: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
2020-12-11 21:11:08,963	INFO trainable.py:217 -- Getting current IP.
2020-12-11 21:11:08,964	WARNING util.py:37 -- Install gputil for GPU system monitoring.
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: Restoring checkpoint from: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
2020-12-11 21:11:09,040	INFO trainable.py:217 -- Getting current IP.
2020-12-11 21:11:09,040	INFO trainable.py:422 -- Restored on 172.17.0.2 from checkpoint: ./models/PPO-RLlib-AIDO5_FrameSkip3_NewMaps_StartAngle30_AIDOWrapper_DomainRand_3092/Dec10_00-31-47/PPO_0_2020-12-10_00-31-48u8cipgyq/checkpoint_363/checkpoint-363
2020-12-11 21:11:09,040	INFO trainable.py:430 -- Current state after restoring: {'_iteration': 363, '_timesteps_total': 1539120, '_time_total': 110224.9016327858, '_episodes_total': 8614}
INFO:nodes_wrapper:5f97768fd25d:RLlibAgent: Starting episode "episode".
ERROR:nodes_wrapper:Error in node RLlibAgent: 
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
    handle_message_node(parsed, receiver0, context0)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
    call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
    f(**kwargs)
  File "solution.py", line 61, in on_received_observations
    new_image = jpg2rgb(camera.jpg_data)
  File "solution.py", line 113, in jpg2rgb
    im = im.convert('RGB')
  File "/usr/local/lib/python3.8/dist-packages/PIL/Image.py", line 902, in convert
    self.load()
  File "/usr/local/lib/python3.8/dist-packages/PIL/ImageFile.py", line 261, in load
    n, err_code = decoder.decode(b)
  File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
    sys.exit(signal.SIGTERM)
SystemExit: Signals.SIGTERM

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
    loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 378, in loop
    raise InternalProblem(msg) from e  # XXX
zuper_nodes.structures.InternalProblem: Exception while handling a message on topic "observations".

| Traceback (most recent call last):
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
|     handle_message_node(parsed, receiver0, context0)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
|     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|     f(**kwargs)
|   File "solution.py", line 61, in on_received_observations
|     new_image = jpg2rgb(camera.jpg_data)
|   File "solution.py", line 113, in jpg2rgb
|     im = im.convert('RGB')
|   File "/usr/local/lib/python3.8/dist-packages/PIL/Image.py", line 902, in convert
|     self.load()
|   File "/usr/local/lib/python3.8/dist-packages/PIL/ImageFile.py", line 261, in load
|     n, err_code = decoder.decode(b)
|   File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
|     sys.exit(signal.SIGTERM)
| SystemExit: Signals.SIGTERM
| 

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
    handle_message_node(parsed, receiver0, context0)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
    call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
    f(**kwargs)
  File "solution.py", line 61, in on_received_observations
    new_image = jpg2rgb(camera.jpg_data)
  File "solution.py", line 113, in jpg2rgb
    im = im.convert('RGB')
  File "/usr/local/lib/python3.8/dist-packages/PIL/Image.py", line 902, in convert
    self.load()
  File "/usr/local/lib/python3.8/dist-packages/PIL/ImageFile.py", line 261, in load
    n, err_code = decoder.decode(b)
  File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
    sys.exit(signal.SIGTERM)
SystemExit: Signals.SIGTERM

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 243, in run_loop
    loop(node_name, fi, fo, node, protocol, tin, tout, config=config, fi_desc=fin, fo_desc=fout)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 378, in loop
    raise InternalProblem(msg) from e  # XXX
zuper_nodes.structures.InternalProblem: Exception while handling a message on topic "observations".

| Traceback (most recent call last):
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 355, in loop
|     handle_message_node(parsed, receiver0, context0)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 531, in handle_message_node
|     call_if_fun_exists(agent, expect_fn, data=ob, context=context, timing=timing)
|   File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/utils.py", line 21, in call_if_fun_exists
|     f(**kwargs)
|   File "solution.py", line 61, in on_received_observations
|     new_image = jpg2rgb(camera.jpg_data)
|   File "solution.py", line 113, in jpg2rgb
|     im = im.convert('RGB')
|   File "/usr/local/lib/python3.8/dist-packages/PIL/Image.py", line 902, in convert
|     self.load()
|   File "/usr/local/lib/python3.8/dist-packages/PIL/ImageFile.py", line 261, in load
|     n, err_code = decoder.decode(b)
|   File "/usr/local/lib/python3.8/dist-packages/ray/worker.py", line 881, in sigterm_handler
|     sys.exit(signal.SIGTERM)
| SystemExit: Signals.SIGTERM
| 

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "solution.py", line 127, in <module>
    main()
  File "solution.py", line 123, in main
    wrap_direct(node=node, protocol=protocol)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/interface.py", line 24, in wrap_direct
    run_loop(node, protocol, args)
  File "/usr/local/lib/python3.8/dist-packages/zuper_nodes_wrapper/wrapper.py", line 251, in run_loop
    raise Exception(msg) from e
Exception: Error in node RLlibAgent
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60543eval0abortedyes0:03:49
Job shutdown [...]
Job shutdown
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60542eval0abortedyes0:02:04
Job shutdown [...]
Job shutdown
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible
60541eval0abortedyes0:06:49
Job shutdown [...]
Job shutdown
Artefacts hidden. If you are the author, please login using the top-right link or use the dashboard.
No reset possible