Unable to serialize AtomicBoolean on Java 17

[basil]

Problem

See jenkinsci/docker-plugin#905. A few plugins, most notably the Docker plugin, attempt to serialize an AtomicBoolean via XStream. Under Java 11, this resulted in an "illegal reflective access operation has occurred" warning, while on Java 17 this results in "java.lang.reflect.InaccessibleObjectException: Unable to make field private static final long java.util.concurrent.atomic.AtomicBoolean.serialVersionUID accessible: module java.base does not opens java.util.concurrent.atomic to unnamed module", as shown when running the new test added in this PR without the changes to core/src/main:

Java 11

[INFO] Running jenkins.util.xstream.AtomicBooleanFieldsTest
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.thoughtworks.xstream.converters.reflection.FieldDictionary (file:/home/basil/.m2/repository/com/thoughtworks/xstream/xstream/1.4.19/xstream-1.4.19.jar) to field java.util.concurrent.atomic.AtomicBoolean.serialVersionUID
WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.converters.reflection.FieldDictionary
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.242 s - in jenkins.util.xstream.AtomicBooleanFieldsTest

Java 17

[INFO] Running jenkins.util.xstream.AtomicBooleanFieldsTest
[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.206 s <<< FAILURE! - in jenkins.util.xstream.AtomicBooleanFieldsTest
[ERROR] jenkins.util.xstream.AtomicBooleanFieldsTest.testAtomicBooleanFields  Time elapsed: 0.197 s  <<< ERROR!
java.lang.RuntimeException: Failed to serialize jenkins.util.xstream.AtomicBooleanFieldsTest$Musican#alive for class jenkins.util.xstream.AtomicBooleanFieldsTest$Musican
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:276)
        at hudson.util.RobustReflectionConverter$2.visit(RobustReflectionConverter.java:243)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:174)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:228)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:165)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:87)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeBareItem(AbstractCollectionConverter.java:94)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeItem(AbstractCollectionConverter.java:66)
        at com.thoughtworks.xstream.converters.collections.AbstractCollectionConverter.writeCompleteItem(AbstractCollectionConverter.java:81)
        at com.thoughtworks.xstream.converters.collections.CollectionConverter.marshal(CollectionConverter.java:75)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:44)
        at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:83)
        at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1266)
        at com.thoughtworks.xstream.XStream.marshal(XStream.java:1255)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:1228)
        at com.thoughtworks.xstream.XStream.toXML(XStream.java:1215)
        at jenkins.util.xstream.AtomicBooleanFieldsTest.testAtomicBooleanFields(AtomicBooleanFieldsTest.java:39)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
        at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
        at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
        at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
        at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
        at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
        at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
        at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
        at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
        at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
        at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
        at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
        at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
        at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
        at org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:42)
        at org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
        at org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:72)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:147)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:127)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:90)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.lambda$execute$0(EngineExecutionOrchestrator.java:55)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.withInterceptedStreams(EngineExecutionOrchestrator.java:102)
        at org.junit.platform.launcher.core.EngineExecutionOrchestrator.execute(EngineExecutionOrchestrator.java:54)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:114)
        at org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:86)
        at org.junit.platform.launcher.core.DefaultLauncherSession$DelegatingLauncher.execute(DefaultLauncherSession.java:86)
        at org.apache.maven.surefire.junitplatform.LazyLauncher.execute(LazyLauncher.java:55)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.execute(JUnitPlatformProvider.java:223)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invokeAllTests(JUnitPlatformProvider.java:175)
        at org.apache.maven.surefire.junitplatform.JUnitPlatformProvider.invoke(JUnitPlatformProvider.java:135)
        at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:456)
        at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:169)
        at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:595)
        at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:581)
Caused by: java.lang.reflect.InaccessibleObjectException: Unable to make field private static final long java.util.concurrent.atomic.AtomicBoolean.serialVersionUID accessible: module java.base does not "opens java.util.concurrent.atomic" to unnamed module @11758f2a
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:354)
        at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(AccessibleObject.java:297)
        at java.base/java.lang.reflect.Field.checkCanSetAccessible(Field.java:178)
        at java.base/java.lang.reflect.Field.setAccessible(Field.java:172)
        at com.thoughtworks.xstream.converters.reflection.FieldDictionary.buildDictionaryEntryForClass(FieldDictionary.java:176)
        at com.thoughtworks.xstream.converters.reflection.FieldDictionary.buildMap(FieldDictionary.java:142)
        at com.thoughtworks.xstream.converters.reflection.FieldDictionary.fieldsFor(FieldDictionary.java:80)
        at com.thoughtworks.xstream.converters.reflection.PureJavaReflectionProvider.visitSerializableFields(PureJavaReflectionProvider.java:167)
        at hudson.util.RobustReflectionConverter.doMarshal(RobustReflectionConverter.java:208)
        at hudson.util.RobustReflectionConverter.marshal(RobustReflectionConverter.java:165)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:68)
        at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:59)
        at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:83)
        at hudson.util.RobustReflectionConverter.marshallField(RobustReflectionConverter.java:285)
        at hudson.util.RobustReflectionConverter$2.writeField(RobustReflectionConverter.java:272)
        ... 64 more

Evaluation

Filed upstream as x-stream/xstream#308. XStream has no built-in awareness of the AtomicBoolean type, so it uses reflection to serialize each of its fields. This breaks down on Java 17, where the serialVersionUID field is not accessible. It is arguably preferable for plugins to serialize a plain boolean rather than an AtomicBoolean, but given the presence of

jenkins/core/src/main/resources/jenkins/security/whitelisted-classes.txt

Line 131 in 6448700

java.util.concurrent.atomic.AtomicBoolean

a handful of plugins have unfortunately come to rely on this behavior.

Solution

While we could open the java.util.concurrent module, we would rather not, since it generally indicates a code smell whereas usages should be ideally rewritten to use a non-concurrent data type. This change gets the affected plugins working again without precluding the long-term improvement of migrating those plugins away from AtomicBoolean.

Implementation

This PR implements a custom converter for AtomicBoolean that serializes to the same on-disk format as that implicitly used by the reflection-based serializer that works in Java 11. This same converter was submitted to XStream in x-stream/xstream#309. Since that pull request has remained unacknowledged for over a month, I am adding the converter to Jenkins in the meantime.

Future work

When the upstream change is merged and released, and when we upgrade to that release, this custom converter can be removed. If for some reason the custom converter is found to be problematic, this PR can be reverted at any time without compatibility issues, because the on-disk format remains unchanged.

Testing done

Integration testing

With the changes to core/src/main/java, the new test now passes without any warnings on both Java 11 and Java 17:

[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running jenkins.util.xstream.AtomicBooleanFieldsTest
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.224 s - in jenkins.util.xstream.AtomicBooleanFieldsTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

Functional testing

I ran a Docker build on Java 17 before and after this PR. Before this PR, the build failed as in jenkinsci/docker-plugin#905. After this PR, the build passed.

Proposed changelog entries

Allow serialization of atomic boolean data types on Java 17.

Proposed upgrade guidelines

N/A

Submitter checklist

• The Jira issue, if it exists, is well-described.
• The changelog entries and upgrade guidelines are appropriate for the audience affected by the change (users or developers, depending on the change) and are in the imperative mood (see examples).
  • Fill in the Proposed upgrade guidelines section only if there are breaking changes or changes that may require extra steps from users during upgrade.
• There is automated testing or an explanation as to why this change has no tests.
• New public classes, fields, and methods are annotated with @Restricted or have @since TODO Javadocs, as appropriate.
• New deprecations are annotated with @Deprecated(since = "TODO") or @Deprecated(forRemoval = true, since = "TODO"), if applicable.
• New or substantially changed JavaScript is not defined inline and does not call eval to ease future introduction of Content Security Policy (CSP) directives (see documentation).
• For dependency updates, there are links to external changelogs and, if possible, full differentials.
• For new APIs and extension points, there is a link to at least one consumer.

Desired reviewers

@mention

Maintainer checklist

Before the changes are marked as ready-for-merge:

• There are at least two (2) approvals for the pull request and no outstanding requests for change.
• Conversations in the pull request are over, or it is explicit that a reviewer is not blocking the change.
• Changelog entries in the pull request title and/or Proposed changelog entries are accurate, human-readable, and in the imperative mood.
• Proper changelog labels are set so that the changelog can be generated automatically.
• If the change needs additional upgrade steps from users, the upgrade-guide-needed label is set and there is a Proposed upgrade guidelines section in the pull request title (see example).
• If it would make sense to backport the change to LTS, a Jira issue must exist, be a Bug or Improvement, and be labeled as lts-candidate to be considered (see query).

[timja]

/label ready-for-merge

---

This PR is now ready for merge, after ~24 hours, we will merge it if there's no negative feedback.

Thanks!