cfn-lint -u is very flaky in GitHub Actions

[roger-zhangg]

CloudFormation Lint Version

latest

What operating system are you using?

Linux

Describe the bug

https://github.com/aws/serverless-application-model/actions/runs/19517345578/job/55875798766

We are running this script: https://github.com/aws/serverless-application-model/blame/develop/bin/run_cfn_lint.sh
Bascially we install with pip install cfn-lint --upgrade --quiet
Then cfn-lint -u and retry for 3 times. But recently it seems it's very likely to fail for 3 times in a row. Is this a new issue?

Full log:

bin/run_cfn_lint.sh
+ VENV=.venv_cfn_lint
+ [ ! -d .venv_cfn_lint ]
+ python3 -m venv .venv_cfn_lint
+ .venv_cfn_lint/bin/python -m pip install cfn-lint --upgrade --quiet

Notice:  A new release of pip is available: 23.0.1 -> 25.3
Notice:  To update, run: python -m pip install --upgrade pip
+ MAX_RETRIES=3
+ RETRY_COUNT=0
+ [ 0 -lt 3 ]
+ .venv_cfn_lint/bin/cfn-lint -u
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1447, in connect
    super().connect()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/socket.py", line 835, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/socket.py", line 966, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/schema/manager.py", line 314, in _update_provider_schema
    if not (url_has_newer_version(url) or force):
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/helpers.py", line 437, in url_has_newer_version
    with urlopen(req) as res:
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/bin/cfn-lint", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/runner/cli.py", line 338, in main
    runner.cli()
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/runner/cli.py", line 268, in cli
    cfnlint.maintenance.update_resource_specs(self.config.force)
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/maintenance.py", line 29, in update_resource_specs
    PROVIDER_SCHEMA_MANAGER.update(force)
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/schema/manager.py", line 246, in update
    pool.starmap(self._update_provider_schema, provider_pool_tuple)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
+ RETRY_COUNT=1
cfn-lint schema update failed, retrying... (attempt 1 of 3)
+ [ 1 -lt 3 ]
+ echo cfn-lint schema update failed, retrying... (attempt 1 of 3)
+ sleep 2
+ [ 1 -lt 3 ]
+ .venv_cfn_lint/bin/cfn-lint -u
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1447, in connect
    super().connect()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/socket.py", line 835, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/socket.py", line 966, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/schema/manager.py", line 314, in _update_provider_schema
    if not (url_has_newer_version(url) or force):
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/helpers.py", line 437, in url_has_newer_version
    with urlopen(req) as res:
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/bin/cfn-lint", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/runner/cli.py", line 338, in main
    runner.cli()
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/runner/cli.py", line 268, in cli
    cfnlint.maintenance.update_resource_specs(self.config.force)
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/maintenance.py", line 29, in update_resource_specs
    PROVIDER_SCHEMA_MANAGER.update(force)
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/schema/manager.py", line 246, in update
    pool.starmap(self._update_provider_schema, provider_pool_tuple)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
+ RETRY_COUNT=2
+ [ 2 -lt 3 ]
cfn-lint schema update failed, retrying... (attempt 2 of 3)
+ echo cfn-lint schema update failed, retrying... (attempt 2 of 3)
+ sleep 2
+ [ 2 -lt 3 ]
+ .venv_cfn_lint/bin/cfn-lint -u
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1346, in do_open
    h.request(req.get_method(), req.selector, req.data, headers,
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1285, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1331, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1280, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1040, in _send_output
    self.send(msg)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 980, in send
    self.connect()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 1447, in connect
    super().connect()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/http/client.py", line 946, in connect
    self.sock = self._create_connection(
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/socket.py", line 835, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/socket.py", line 966, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/schema/manager.py", line 314, in _update_provider_schema
    if not (url_has_newer_version(url) or force):
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/helpers.py", line 437, in url_has_newer_version
    with urlopen(req) as res:
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 214, in urlopen
    return opener.open(url, data, timeout)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 517, in open
    response = self._open(req, data)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 534, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 494, in _call_chain
    result = func(*args)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1389, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/urllib/request.py", line 1349, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/bin/cfn-lint", line 8, in <module>
    sys.exit(main())
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/runner/cli.py", line 338, in main
    runner.cli()
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/runner/cli.py", line 268, in cli
    cfnlint.maintenance.update_resource_specs(self.config.force)
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/maintenance.py", line 29, in update_resource_specs
    PROVIDER_SCHEMA_MANAGER.update(force)
  File "/home/runner/work/serverless-application-model/serverless-application-model/.venv_cfn_lint/lib/python3.9/site-packages/cfnlint/schema/manager.py", line 246, in update
    pool.starmap(self._update_provider_schema, provider_pool_tuple)
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 372, in starmap
    return self._map_async(func, iterable, starmapstar, chunksize).get()
  File "/opt/hostedtoolcache/Python/3.9.25/x64/lib/python3.9/multiprocessing/pool.py", line 771, in get
    raise self._value
urllib.error.URLError: <urlopen error [Errno -3] Temporary failure in name resolution>
+ RETRY_COUNT=3
cfn-lint schema update failed after 3 attempts
+ [ 3 -lt 3 ]
+ echo cfn-lint schema update failed after 3 attempts
+ exit 1
make: *** [Makefile:57: lint] Error 1

Expected behavior

cfn-lint -u should finish successfully

Reproduction template

https://github.com/aws/serverless-application-model/blob/develop/bin/run_cfn_lint.sh

Remember, it's flaky so it won't always fail

[roger-zhangg]

May I get an eye on this? This is still happening.

[kddejong]

This will sometimes happen when the cfn-lint has issues communicating with remote endpoints to get the schemas. Each region has its own endpoint and we pull schemas for every region by default. I have, previously, debated coupling the region parameter with --update-specs as most people running this just want to update a region or two. This would increase the likely hood of success, speed up the process, but will not guarantee success.

This #4319 may also help with making that feature easier. Additionally, this PR will more gracefully handle those failures but the exit code will still not be 0 which may break your scripts or you will have to find a way to ignore the failures. If it fails to update it will just keep what it has.

If you are interested in the capability to couple the region parameter let me know and I can take a look at implementing it.