Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add --write-log-init-timeout option to allow specifying a timeout to retry writing logs to stdout #698

Merged
merged 6 commits into from
Aug 6, 2023

Conversation

romanisb
Copy link
Contributor

This contribution proposes a new PipelineRunOptions which provides an option to set a specific timeout within the runner is periodically trying to initialize the log output stream.

This option will replace the hardcoded retrycount with a more dynamical approach.

Without increasing the duration of this retry, we occasionally ran into issues as shown below. Especially in a Kubernetes environment in which the nodes is under high utilization.

Cloning into '.'...
Checking out pipeline from revision master
Your branch is up to date with 'origin/master'.
Already on 'master'
Delete pipeline git clone credentials
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by com.thoughtworks.xstream.core.util.Fields (file:/app/repo/org/jvnet/hudson/xstream/1.4.7-jenkins-1/xstream-1.4.7-jenkins-1.jar) to field java.util.TreeMap.comparator
WARNING: Please consider reporting this to the maintainers of com.thoughtworks.xstream.core.util.Fields
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
2021-01-15 07:50:03.002+0000 [id=1]	SEVERE	h.i.i.InstallUncaughtExceptionHandler$DefaultUncaughtExceptionHandler#uncaughtException: A thread (main/1) died unexpectedly due to an uncaught exception, this may leave your Jenkins in a bad way and is usually indicative of a bug in the code.
java.io.FileNotFoundException: /jenkins_home/jobs/job/builds/1/log (No such file or directory)
	at java.base/java.io.RandomAccessFile.open0(Native Method)
	at java.base/java.io.RandomAccessFile.open(RandomAccessFile.java:345)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:259)
	at java.base/java.io.RandomAccessFile.<init>(RandomAccessFile.java:214)
	at org.kohsuke.stapler.framework.io.LargeText$FileSession.<init>(LargeText.java:399)
	at org.kohsuke.stapler.framework.io.LargeText$2.open(LargeText.java:121)
	at org.kohsuke.stapler.framework.io.LargeText.writeLogTo(LargeText.java:212)
	at hudson.console.AnnotatedLargeText.writeLogTo(AnnotatedLargeText.java:165)
	at hudson.model.Run.writeWholeLogTo(Run.java:1556)
	at io.jenkins.jenkinsfile.runner.Runner.writeLogTo(Runner.java:134)
	at io.jenkins.jenkinsfile.runner.Runner.run(Runner.java:94)
Caused: java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at io.jenkins.jenkinsfile.runner.JenkinsfileRunnerLauncher.doLaunch(JenkinsfileRunnerLauncher.java:32)
	at io.jenkins.jenkinsfile.runner.JenkinsLauncher.launch(JenkinsLauncher.java:83)
	at io.jenkins.jenkinsfile.runner.App.run(App.java:13)
	at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.run(Bootstrap.java:312)
	at io.jenkins.jenkinsfile.runner.bootstrap.Bootstrap.main(Bootstrap.java:135)

Testing done

Submitter checklist

romanisb and others added 6 commits June 28, 2023 13:20
This option allows the consumer to define how often the write loop
will retry writing to the provided output stream instead of having
a hardcoded default which in some cases is not sufficient enough.
@romanisb romanisb requested a review from a team as a code owner June 30, 2023 07:14
@romanisb romanisb changed the title Args/retry log write Add --write-log-init-timeout option to allow specifying a timeout to retry writing logs to stdout Jun 30, 2023
Copy link
Member

@oleg-nenashev oleg-nenashev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks! It is a good patch

@oleg-nenashev oleg-nenashev added the enhancement New feature or request label Aug 6, 2023
@oleg-nenashev oleg-nenashev merged commit 58b0a62 into jenkinsci:main Aug 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants