Error 500 for CycloneDX or SPDX for project

[pombredanne]

In the branch https:/nexB/scancode.io/compare/debug-populate-purldb
requesting a CycloneDX in the UI at https://<...>/project/<...>/results/cyclonedx/1.6/ gives a 500 Error

The logs have this:

web-1  | ERROR Internal Server Error: /project/.../results/cyclonedx/1.6/
web-1  | Traceback (most recent call last):
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1  |     response = get_response(request)
web-1  |                ^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
web-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
web-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/usr/local/lib/python3.12/contextlib.py", line 81, in inner
web-1  |     return func(*args, **kwds)
web-1  |            ^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/views/generic/base.py", line 104, in view
web-1  |     return self.dispatch(request, *args, **kwargs)
web-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/contrib/auth/mixins.py", line 135, in dispatch
web-1  |     return super().dispatch(request, *args, **kwargs)
web-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/views/generic/base.py", line 143, in dispatch
web-1  |     return handler(request, *args, **kwargs)
web-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/scanpipe/views.py", line 1322, in get
web-1  |     output_file = output.to_cyclonedx(project, **output_kwargs)
web-1  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/scanpipe/pipes/output.py", line 780, in to_cyclonedx
web-1  |     bom = get_cyclonedx_bom(project)
web-1  |           ^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/scanpipe/pipes/output.py", line 728, in get_cyclonedx_bom
web-1  |     component = package.as_cyclonedx()
web-1  |                 ^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/scanpipe/models.py", line 3390, in as_cyclonedx
web-1  |     if expression_spdx := self.get_declared_license_expression_spdx():
web-1  |                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/scanpipe/models.py", line 3333, in get_declared_license_expression_spdx
web-1  |     return build_spdx_license_expression(self.declared_license_expression)
web-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/licensedcode/cache.py", line 513, in build_spdx_license_expression
web-1  |     validate_spdx_license_keys(license_expression=license_expression, licensing=licensing)
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/licensedcode/cache.py", line 553, in validate_spdx_license_keys
web-1  |     raise InvalidLicenseKeyError(f"ERROR in parsing license_expression: {license_expression}: type: {type(license_expression)} :{messages}")
web-1  | licensedcode.cache.InvalidLicenseKeyError: ERROR in parsing license_expression: lgplk-2.1: type: <class 'str'> :['License key: lgplk-2.1 is not a valid license key from LicenseDB', 'Error rendering SPDX license key for: lgplk-2.1']

The lgplk-2.1 comes from a faulty ABOUT file and the code was about-mapped

The problem is that I have no way to fix this short of a full new project rescan:

1. I cannot edit things to correct
2. I cannot add a corrected ABOUT file to just rerun the step (which would not yield anything anyway since resource status are now set)

[pombredanne]

Note that I fixed by hand for now and SPDX works again:

DiscoveredPackage.objects.filter(declared_license_expression="lgplk-2.1").update(declared_license_expression="lgpl-2.1")

[pombredanne]

But the CycloneDX stills fails:

 WARNING Not Found: /project/.../results/spdx/
web-1  | ERROR Internal Server Error: /project/.../results/cyclonedx/1.6/
web-1  | Traceback (most recent call last):
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/core/handlers/exception.py", line 55, in inner
web-1  |     response = get_response(request)
web-1  |                ^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/core/handlers/base.py", line 197, in _get_response
web-1  |     response = wrapped_callback(request, *callback_args, **callback_kwargs)
web-1  |                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/usr/local/lib/python3.12/contextlib.py", line 81, in inner
web-1  |     return func(*args, **kwds)
web-1  |            ^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/views/generic/base.py", line 104, in view
web-1  |     return self.dispatch(request, *args, **kwargs)
web-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/contrib/auth/mixins.py", line 135, in dispatch
web-1  |     return super().dispatch(request, *args, **kwargs)
web-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/django/views/generic/base.py", line 143, in dispatch
web-1  |     return handler(request, *args, **kwargs)
web-1  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/scanpipe/views.py", line 1322, in get
web-1  |     output_file = output.to_cyclonedx(project, **output_kwargs)
web-1  |                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
web-1  |   File "/opt/scancodeio/scanpipe/pipes/output.py", line 785, in to_cyclonedx
web-1  |     json_outputter.generate()
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/cyclonedx/output/json.py", line 71, in generate
web-1  |     bom.validate()
web-1  |   File "/opt/scancodeio/venv/lib/python3.12/site-packages/cyclonedx/model/bom.py", line 658, in validate
web-1  |     raise UnknownComponentDependencyException(
web-1  | cyclonedx.exception.model.UnknownComponentDependencyException: One or more Components have Dependency references to Components/Services that are not known in this BOM. They are: {<BomRef 'pkg:maven/oro/[email protected]?uuid=7c146662-c889-482f-8ffc-678693e5008a' id=140678358973952>, <BomRef 'pkg:maven/org.springframework/[email protected]?uuid=a77f25b3-c06d-447e-ae9a-faf387d2a063' id=140678358407200>, <BomRef 'pkg:maven/org.springframework/[email protected]?uuid=bd3d0a73-c428-4254-bf4c-542130c22312' id=140678358682752>, <BomRef 'pkg:maven/org.springframework/[email protected]?uuid=c50dc80d-849e-451d-a92e-6c2793b0ed12' id=140678358497904>..... 
..........................(truncated)
}

[tdruez]

The source of the problem is duplicated packages, where the purl and all attributes of the packages are is identical, except for the package_uid (and m2m such as the related resources).

There's an identity issue, when we add the second entry to the bom.components which is a set, the second entry is not added as a different entry of the set.
Eventually causing the missing "Dependency references" later in the processing.

The cyclonedx.Component.hash does not involve the bom-ref, making our package_uid approach incompatible:
https:/CycloneDX/cyclonedx-python-lib/blob/main/cyclonedx/model/component.py#L1770

An idea to workaround the fact the bom-ref is not considered in the comparison, is to include the package_uid value in the attribute of the CycloneDX.Component. Using the property seems to be the most logical place for this value.

[tdruez]

Fix merged in #1317 and new property added in https:/nexB/aboutcode-cyclonedx-taxonomy