Skip to content

Use different gce client per process

Aaron Graubert requested to merge github/fork/Marlin-Na/patch-gceforkprocess into master

Created by: Marlin-Na

If the python process is forked, we use a new gce client. Otherwise it will lead to error:

[20210518-12:43:48] [0138500e-e992-4036-ace4-ddedcd3e7785_ea550049-0c6d-4f04-92e4-ecb9d092661c:LocalizeToLocalDisk] Unexpected error: SSLError(1, '[SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2622)')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine/runner.py", line 48, in inner
    new_state = method(self, state, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/prefect/engine/task_runner.py", line 865, in get_task_run_state
    value = prefect.utilities.executors.run_task_with_timeout(
  File "/usr/local/lib/python3.8/dist-packages/prefect/utilities/executors.py", line 323, in run_task_with_timeout
    return task.run(*args, **kwargs)  # type: ignore
  File "/usr/local/lib/python3.8/dist-packages/wolf/localization.py", line 448, in run
    wolf_out = super(BatchLocalDisk, self_copy).run()
  File "/usr/local/lib/python3.8/dist-packages/wolf/task.py", line 598, in run
    task = self._get_runnable_copy(bound_inputs=task_inputs)
  File "/usr/local/lib/python3.8/dist-packages/wolf/task.py", line 523, in _get_runnable_copy
    task.backend = canine.orchestrator.BACKENDS[conf['type']](**conf)
  File "/usr/local/lib/python3.8/dist-packages/canine/backends/dockerTransient.py", line 63, in __init__
    self.config["image"] = self.get_latest_image(self.config["image_family"])["name"] if image is None else image
  File "/usr/local/lib/python3.8/dist-packages/canine/backends/dockerTransient.py", line 257, in get_latest_image
    ans = gce.images().getFromFamily(family = image_family, project = self.config["project"]).execute()
  File "/usr/local/lib/python3.8/dist-packages/googleapiclient/_helpers.py", line 134, in positional_wrapper
    return wrapped(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/googleapiclient/http.py", line 920, in execute
    resp, content = _retry_request(
  File "/usr/local/lib/python3.8/dist-packages/googleapiclient/http.py", line 222, in _retry_request
    raise exception
  File "/usr/local/lib/python3.8/dist-packages/googleapiclient/http.py", line 191, in _retry_request
    resp, content = http.request(uri, method, *args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/google_auth_httplib2.py", line 218, in request
    response, content = self.http.request(
  File "/usr/local/lib/python3.8/dist-packages/httplib2/__init__.py", line 1708, in request
    (response, content) = self._request(
  File "/usr/local/lib/python3.8/dist-packages/httplib2/__init__.py", line 1424, in _request
    (response, content) = self._conn_request(conn, request_uri, method, body, headers)
  File "/usr/local/lib/python3.8/dist-packages/httplib2/__init__.py", line 1376, in _conn_request
    response = conn.getresponse()
  File "/usr/lib/python3.8/http/client.py", line 1347, in getresponse
    response.begin()
  File "/usr/lib/python3.8/http/client.py", line 307, in begin
    version, status, reason = self._read_status()

Merge request reports