Avro schema evolution not working with Union type and null default

[jrkinley]

Version & Environment

Redpanda v2.11.10

What went wrong?

Adding a new field to an Avro schema with a default value of null should be backwards compatible (note that None in Python is correctly translated to null by json.dumps):

{
  "name": "value2",
  "type": ["null", "int"],
  "default": None
}

But the schema registry rejects the new version of the schema with:

{
  "error_code": 409,
  "message": "Schema being registered is incompatible with an earlier schema for subject \"{sensor-value}\""
}

The subject's compatibility mode is set to BACKWARD and Python's Avro schema compatibility checks pass ok:

from avro.schema import parse
from avro.compatibility import ReaderWriterCompatibilityChecker

is_compatible = ReaderWriterCompatibilityChecker().get_compatibility(
  parse(json.dumps(sensor_schema_v2)),
  parse(json.dumps(sensor_schema))
)
print(f"Schemas compatible: {is_compatible.compatibility}")

What should have happened instead?

The schema registry should accept the new schema version because the additional field is backward compatible.

How to reproduce the issue?

schema-registry.txt

[andrewhsu]

/backport v21.11.x