[SPARK-49541][BUILD] Upgrade log4j2 to 2.24.1

[panbingkun]

What changes were proposed in this pull request?

The pr aims to upgrade log4j2 from 2.22.1 to 2.24.1.

Why are the changes needed?

• The full release notes:
  https:/apache/logging-log4j2/releases/tag/rel%2F2.24.1
  https:/apache/logging-log4j2/releases/tag/rel%2F2.24.0
  https:/apache/logging-log4j2/releases/tag/rel%2F2.23.1
  https:/apache/logging-log4j2/releases/tag/rel%2F2.23.0

• The new version contains some bug fixes:
  Fix regression in JdkMapAdapterStringMap performance. (Avoid a slow exception catch in JdkMapAdapterStringMap constructor logging-log4j2#2238)
  Fix NPE in PatternProcessor for a UNIX_MILLIS pattern (NPE in PatternProcessor when using a UNIX_MILLIS date pattern logging-log4j2#2346)
  Fix that parameterized message formatting throws an exception when there are insufficient number of parameters. It previously simply
  didn't replace the '{}' sequence. The behavior changed in 2.21.0 and should be restored for backward compatibility. (Log messages with partially missing parameters logging-log4j2#2380)
  Fix putAll() in the default thread context map implementation (Update newMap.numEntries after copying the old map in UnmodifiableArrayBackedMap#copyAndPutAll logging-log4j2#2942)

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Pass GA.

Was this patch authored or co-authored using generative AI tooling?

No.

[LuciferYang]

#48057 (comment)

Upon further analysis, it appears that there are some issues with the method copyAndPutAll in org.apache.logging.log4j.internal.map.UnmodifiableArrayBackedMap. From the debugging process, it seems that the intention here is to put executor_id -> 1 into an UnmodifiableArrayBackedMap that already contains data -> some-data. However, the resulting map only contains executor_id -> 1. I believe this is not as expected.

[LuciferYang]

try to fix this issue: apache/logging-log4j2#2942

[LuciferYang]

try to fix this issue: apache/logging-log4j2#2942

Fixed, let's wait 2.24.1

[LuciferYang]

Testing log4j 2.24.1 #48310

[LuciferYang]

@panbingkun 2.24.1 should be able to pass the tests(https:/LuciferYang/spark/runs/30898064859). Could you please update the PR, including the PR title and description? We can continue to move forward with this PR.

[LuciferYang]

+1, LGTM

But it's best to retrigger the Docker integrated tests :)

[panbingkun]

+1, LGTM

But it's best to retrigger the Docker integrated tests :)

Done.

[LuciferYang]

Merged into master for Spark 4.0. Thanks @panbingkun

[dongjoon-hyun]

+1, LGTM.
Thank you, @panbingkun and @LuciferYang .