We have recently migrated our application from Python 2 to Python 3 on Google App Engine and are encountering a recurring issue related to logging. Despite the application’s functionality appearing unaffected, our error console consistently reports problems with submitting logs. This issue seems to be happening in the background but it is causing concerns regarding the reliability and integrity of our logging process, which is critical to us for analytics and security.

Error Details:

The error we are seeing is as follows:

“Failed to submit 10 logs”

And the traceback includes:

File “/layers/google.python.pip/pip/lib/python3.11/site-packages/google/api_core/grpc_helpers.py”, line 79, in error_remapped_callable return callable_(*args, **kwargs)

File “/layers/google.python.pip/pip/lib/python3.11/site-packages/grpc/_channel.py”, line 1160, in call
return _end_unary_response_blocking(state, call, False, None)

(See Image for more extensive trace)

FYI: We now have over 4,500+ occurrences of the issue in less than 4 days!!!

Steps to reproduce: Unknown

Other information (workarounds we have tried, documentation consulted, etc.):

  1. Try to identify if these errors occur when hitting particular endpoint – could not identify as errors keep happening all around, every minute and are not associated with any request log
  2. Use latest cloud SDK CLI version while deploying (459.0.0)
  3. Use the gcloud beta command
  4. There is a Github thread for a similar error which says the issue got fixed back in 2017, however other people have faced this issue again as mentioned in comments
    https://github.com/googleapis/google-cloud-python/pull/3407

We are seeking assistance in diagnosing and resolving this issue.

Is there are any known issues with logging in GAE applications post Python 3 migration?

Additional Information:

Environment: Google App Engine
Runtime: Python 3.11
appengine-python-standard>=0.2.3
google-cloud-logging==3.8.0

Thank you for your assistance.
Regards

2