Failure using method (v2_runner_on_ok) in callback plugin ... 'utf-8' codec can't encode character '\udc80' in position 1886: surrogates not allowed

[laidbackware]

SUMMARY

When attempting to use the example from community.crypto.get_certificate contact google.com, it triggers the exception below.

[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x7f37ab1034f0>): 'utf-8' codec can't encode character '\udc80' in position 1886: surrogates not allowed
Callback Exception: 
  File "/home/matt/.local/lib/python3.10/site-packages/ansible/executor/task_queue_manager.py", line 450, in send_callback
    method(*new_args, **kwargs)
   File "/home/matt/.local/lib/python3.10/site-packages/ansible/plugins/callback/default.py", line 106, in v2_runner_on_ok
    self._display.display(msg, color=color)
   File "/home/matt/.local/lib/python3.10/site-packages/ansible/utils/display.py", line 289, in display
    fileobj.write(msg2)

ISSUE TYPE

• Bug Report

COMPONENT NAME

community.crypto.get_certificate

ANSIBLE VERSION

ansible [core 2.14.4]
  config file = None
  configured module search path = ['/home/matt/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /home/matt/.local/lib/python3.10/site-packages/ansible
  ansible collection location = /home/matt/.ansible/collections:/usr/share/ansible/collections
  executable location = /home/matt/.local/bin/ansible
  python version = 3.10.6 (main, Mar 10 2023, 10:55:28) [GCC 11.3.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

COLLECTION VERSION

# /home/matt/.ansible/collections/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.11.1 

# /home/matt/.local/lib/python3.10/site-packages/ansible_collections
Collection       Version
---------------- -------
community.crypto 2.11.1

CONFIGURATION

CONFIG_FILE() = None

OS / ENVIRONMENT

Running locally on Ubuntu 22.04 Desktop
Kernel Linux derek 5.19.0-38-generic

STEPS TO REPRODUCE

Running the playbook below with -vvv, generates the error.

---
- name: Debugging
  hosts: localhost
  gather_facts: false
  tasks:
    - name: Get a cert from an https port
      community.crypto.get_certificate:
        host: "www.google.com"
        port: 443
      delegate_to: localhost
      run_once: true
      register: cert

I've updated Ansible and have tried installing older versions of the collection with no success. I have not other issues using community or buildin modules, only this crypto module.

EXPECTED RESULTS

I expect it to not trigger an exception.

ACTUAL RESULTS

[WARNING]: Failure using method (v2_runner_on_ok) in callback plugin (<ansible.plugins.callback.default.CallbackModule object at 0x7f37ab1034f0>): 'utf-8' codec can't encode character '\udc80' in position 1886: surrogates not allowed
Callback Exception: 
  File "/home/matt/.local/lib/python3.10/site-packages/ansible/executor/task_queue_manager.py", line 450, in send_callback
    method(*new_args, **kwargs)
   File "/home/matt/.local/lib/python3.10/site-packages/ansible/plugins/callback/default.py", line 106, in v2_runner_on_ok
    self._display.display(msg, color=color)
   File "/home/matt/.local/lib/python3.10/site-packages/ansible/utils/display.py", line 289, in display
    fileobj.write(msg2)

[felixfontein]

This is (partially) a problem in ansible-core's default callback, see ansible/ansible#80258 for an issue tracking that. The value that is tripping the callback up is something returned as part of the certificate, but fixing that on our side is not so simple as modifying the return values is a breaking change.

[felixfontein]

Hmm, I looked at the module, the documentation claims that the extension value is Base64 encoded, but it actually isn't - that's what's breaking the callback.

I traced back the incorrect documentation to ansible/ansible@054285c#diff-dac138d1d6f28769493a6ef20f3b41996c9a792fb948a3a072d7f808eb6f4c07. Maybe the base64.b64decode confused me and I wrote "Base64 encoded"? Hard to say now...

Anyway, this is something that we can fix, in two ways:

1. Add compatibility code to turn this value into valid UTF-8. In case anyone depends on the proper ASN.1 content, this will be a problem for them.
2. At the same time add an option that changes the return value to be Base64 encoded.

[felixfontein]

Hmm, thinking about it while working on it, turning these values to valid UTF-8 is a breaking change similar to just changing them to Base64 encoded... This is something we shouldn't really do.

[felixfontein]

#592 adds an asn1_base64 option which allows to Base64 encode the ASN.1 value.

[felixfontein]

Next: deprecate the current default value of asn1_base64. Should happen in a few months.

[felixfontein]

#600 does the deprecation.