HADOOP-19299. HttpReferrerAuditHeader resilience #7095

steveloughran · 2024-10-02T18:59:00Z

HADOOP-19299

Catch all exceptions raised when building logger header log message at info; stack at debug

How was this patch tested?

new unit test

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

Catch all exceptions raised when building logger header log message at info; stack at debug Change-Id: I119abd167978dbc515554a36189462599870d1f9

hadoop-yetus · 2024-10-02T21:31:49Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 44s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 26s		trunk passed
+1 💚	compile	9m 0s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	8m 31s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 10s		trunk passed
+1 💚	mvnsite	1m 37s		trunk passed
+1 💚	javadoc	1m 18s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 7s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 25s		trunk passed
+1 💚	shadedclient	20m 56s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 52s		the patch passed
+1 💚	compile	8m 44s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	8m 44s		the patch passed
+1 💚	compile	8m 31s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	8m 31s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	2m 2s	/results-checkstyle-root.txt	root: The patch generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
+1 💚	mvnsite	1m 38s		the patch passed
+1 💚	javadoc	1m 15s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 10s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 34s		the patch passed
+1 💚	shadedclient	21m 22s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 9s		hadoop-common in the patch passed.
+1 💚	unit	2m 14s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 43s		The patch does not generate ASF License warnings.
		151m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/1/artifact/out/Dockerfile
GITHUB PR	#7095
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux d9f312ed6e40 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `99cda77`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/1/testReport/
Max. process+thread count	1276 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

style checks + extra javadocs Change-Id: I1fbe381c49f51f9f4df62fd674a693c89ce9e9ed

steveloughran · 2024-10-03T14:47:47Z

tested s3 london. two failures from new tests which came with #7067 ; reopened that JIRA

hadoop-yetus · 2024-10-03T15:29:20Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 22s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 17s		Maven dependency ordering for branch
+1 💚	mvninstall	22m 31s		trunk passed
+1 💚	compile	10m 44s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	9m 44s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 26s		trunk passed
+1 💚	mvnsite	1m 33s		trunk passed
+1 💚	javadoc	1m 18s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 57s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 19s		trunk passed
+1 💚	shadedclient	25m 2s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 54s		the patch passed
+1 💚	compile	10m 23s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	10m 23s		the patch passed
+1 💚	compile	9m 42s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	9m 42s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 29s		the patch passed
+1 💚	mvnsite	1m 19s		the patch passed
+1 💚	javadoc	1m 7s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 7s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 40s		the patch passed
+1 💚	shadedclient	24m 53s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 14s		hadoop-common in the patch passed.
+1 💚	unit	2m 9s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		167m 32s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/2/artifact/out/Dockerfile
GITHUB PR	#7095
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux a9ec9e0b8b3c 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0be63c1`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/2/testReport/
Max. process+thread count	1276 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

...ct/hadoop-common/src/main/java/org/apache/hadoop/fs/store/audit/HttpReferrerAuditHeader.java

zeekling · 2024-10-04T06:38:21Z

...ct/hadoop-common/src/main/java/org/apache/hadoop/fs/store/audit/HttpReferrerAuditHeader.java

@@ -166,17 +179,18 @@ private HttpReferrerAuditHeader(
 * per entry, and "" returned.
 * @return a referrer string or ""
 */
- public String buildHttpReferrer() {
+ public synchronized String buildHttpReferrer() {


Why need synchronized?

That is the bug fix that has been found. Please see the jira for details. https://issues.apache.org/jira/browse/HADOOP-19299

mukund-thakur

+1 pending one log comment.

steveloughran

thanks; testing

...ct/hadoop-common/src/main/java/org/apache/hadoop/fs/store/audit/HttpReferrerAuditHeader.java

Change-Id: If7a1ff24de449be395667f550341389a5083097c

steveloughran · 2024-10-04T14:22:17Z

tested s3 london; -Dparallel-tests -DtestsThreadCount=8

only failures were those of #7098

hadoop-yetus · 2024-10-04T16:42:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 19s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 11s		Maven dependency ordering for branch
+1 💚	mvninstall	21m 47s		trunk passed
+1 💚	compile	10m 45s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	9m 6s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 12s		trunk passed
+1 💚	mvnsite	1m 36s		trunk passed
+1 💚	javadoc	1m 26s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 8s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 20s		trunk passed
+1 💚	shadedclient	21m 8s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 23s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 55s		the patch passed
+1 💚	compile	8m 46s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	8m 46s		the patch passed
+1 💚	compile	8m 45s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	8m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 4s		the patch passed
+1 💚	mvnsite	1m 23s		the patch passed
+1 💚	javadoc	1m 7s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 54s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 24s		the patch passed
+1 💚	shadedclient	23m 4s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	15m 27s		hadoop-common in the patch passed.
+1 💚	unit	2m 0s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 36s		The patch does not generate ASF License warnings.
		156m 12s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/3/artifact/out/Dockerfile
GITHUB PR	#7095
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 8b77f6da62ab 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `fd28453`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/3/testReport/
Max. process+thread count	1276 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

steveloughran · 2024-10-04T17:03:35Z

Build happy; merging and backporting

Let's no flood the logs if there is a recurrent problem, ok? Change-Id: I189a5b9cb7c6d264bc4835a631e3da5ec3bb22de

hadoop-yetus · 2024-10-04T20:03:58Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 20s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 2 new or modified test files.
			_ trunk Compile Tests _
+0 🆗	mvndep	14m 34s		Maven dependency ordering for branch
+1 💚	mvninstall	19m 20s		trunk passed
+1 💚	compile	9m 1s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	8m 23s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	2m 9s		trunk passed
+1 💚	mvnsite	1m 32s		trunk passed
+1 💚	javadoc	1m 20s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 10s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 23s		trunk passed
+1 💚	shadedclient	21m 3s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+0 🆗	mvndep	0m 22s		Maven dependency ordering for patch
+1 💚	mvninstall	0m 56s		the patch passed
+1 💚	compile	8m 45s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	8m 45s		the patch passed
+1 💚	compile	8m 35s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	8m 35s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	2m 3s		the patch passed
+1 💚	mvnsite	1m 35s		the patch passed
+1 💚	javadoc	1m 19s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	1m 10s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 32s		the patch passed
+1 💚	shadedclient	21m 11s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	16m 7s		hadoop-common in the patch passed.
+1 💚	unit	2m 16s		hadoop-aws in the patch passed.
+1 💚	asflicense	0m 44s		The patch does not generate ASF License warnings.
		151m 20s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/4/artifact/out/Dockerfile
GITHUB PR	#7095
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 6fbae0e8103e 5.15.0-117-generic #127-Ubuntu SMP Fri Jul 5 20:13:28 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `0883106`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/4/testReport/
Max. process+thread count	1276 (vs. ulimit of 5500)
modules	C: hadoop-common-project/hadoop-common hadoop-tools/hadoop-aws U: .
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7095/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

* HttpReferrerAuditHeader is thread safe, copying the lists/maps passed in and using synchronized methods when necessary. * All exceptions raised when building referrer header are caught and swallowed. * The first such error is logged at warn, * all errors plus stack are logged at debug Contributed by Steve Loughran

HADOOP-19299. HttpReferrerAuditHeader resilience

99cda77

Catch all exceptions raised when building logger header log message at info; stack at debug Change-Id: I119abd167978dbc515554a36189462599870d1f9

github-actions bot added trunk Common TOOLS AWS labels Oct 2, 2024

HADOOP-19299. HttpReferrerAuditHeader resilience

0be63c1

style checks + extra javadocs Change-Id: I1fbe381c49f51f9f4df62fd674a693c89ce9e9ed

zeekling reviewed Oct 4, 2024

View reviewed changes

mukund-thakur approved these changes Oct 4, 2024

View reviewed changes

steveloughran commented Oct 4, 2024

View reviewed changes

...ct/hadoop-common/src/main/java/org/apache/hadoop/fs/store/audit/HttpReferrerAuditHeader.java Show resolved Hide resolved

HADOOP-19299. review feedback

fd28453

Change-Id: If7a1ff24de449be395667f550341389a5083097c

HADOOP-19299. log only once

0883106

Let's no flood the logs if there is a recurrent problem, ok? Change-Id: I189a5b9cb7c6d264bc4835a631e3da5ec3bb22de

mukund-thakur approved these changes Oct 7, 2024

View reviewed changes

steveloughran merged commit 50e6b49 into apache:trunk Oct 7, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

HADOOP-19299. HttpReferrerAuditHeader resilience #7095

HADOOP-19299. HttpReferrerAuditHeader resilience #7095

steveloughran commented Oct 2, 2024

hadoop-yetus commented Oct 2, 2024

steveloughran commented Oct 3, 2024

hadoop-yetus commented Oct 3, 2024

zeekling Oct 4, 2024

mukund-thakur Oct 4, 2024

mukund-thakur left a comment

steveloughran left a comment

steveloughran commented Oct 4, 2024

hadoop-yetus commented Oct 4, 2024

steveloughran commented Oct 4, 2024

hadoop-yetus commented Oct 4, 2024

HADOOP-19299. HttpReferrerAuditHeader resilience #7095

HADOOP-19299. HttpReferrerAuditHeader resilience #7095

Conversation

steveloughran commented Oct 2, 2024

How was this patch tested?

For code changes:

hadoop-yetus commented Oct 2, 2024

steveloughran commented Oct 3, 2024

hadoop-yetus commented Oct 3, 2024

zeekling Oct 4, 2024

Choose a reason for hiding this comment

mukund-thakur Oct 4, 2024

Choose a reason for hiding this comment

mukund-thakur left a comment

Choose a reason for hiding this comment

steveloughran left a comment

Choose a reason for hiding this comment

steveloughran commented Oct 4, 2024

hadoop-yetus commented Oct 4, 2024

steveloughran commented Oct 4, 2024

hadoop-yetus commented Oct 4, 2024