Skip to content
This repository has been archived by the owner on Jun 20, 2024. It is now read-only.

Jc allow memory override eib #392

Merged
merged 2 commits into from
Jun 23, 2021
Merged

Jc allow memory override eib #392

merged 2 commits into from
Jun 23, 2021

Conversation

jacarey
Copy link
Contributor

@jacarey jacarey commented Jun 22, 2021 •
edited
Loading

This PR allows the setting of memory used by ExtractIlluminaBarcodes by using this.resources.memory. Previously pickResources as always setting memory to 4G.

This sets the default value to 4G, but if the user sets a larger size by using .requires() than the new value with be used.

Additionally, this PR adds a LinearMemoryRetry with a retry count of 3.

Tested with a basic pipeline:

val eib   = new ExtractIlluminaBarcodes(basecallsDir=basecallsDir, lane=1, readStructure="151T8B8B151T", barcodeFile=barcodeFile).requires(Cores(8), Memory("64M"))
    root ==> eib

[2021/06/22 14:01:53 | DagrCoreMain | Info] Executing dagr from dagr version null as j@work on JRE 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10
[2021/06/22 14:01:53 | TaskManager | Info] 'ExtractIlluminaBarcodes' has been started on attempt #1 with 16.00 cores and 64m memory
[2021/06/22 14:01:54 | TaskManager | Info] 'ExtractIlluminaBarcodes' has failed (execution) on attempt #1 with 16.00 cores and 64m memory
[2021/06/22 14:01:54 | TaskManager | Info] 'ExtractIlluminaBarcodes' has been started on attempt #2 with 16.00 cores and 128m memory
[2021/06/22 14:01:55 | TaskManager | Info] 'ExtractIlluminaBarcodes' has failed (execution) on attempt #2 with 16.00 cores and 128m memory
[2021/06/22 14:01:55 | TaskManager | Info] 'ExtractIlluminaBarcodes' has been started on attempt #3 with 16.00 cores and 256m memory
[2021/06/22 14:02:34 | TaskManager | Info] 'ExtractIlluminaBarcodes' has succeeded on attempt #3 with 16.00 cores and 256m memory
[2021/06/22 14:02:34 | TaskManager | Info] ################################################################################
[2021/06/22 14:02:34 | TaskManager | Info] Completed execution successfully.
[2021/06/22 14:02:34 | DagrCoreMain | Info] dagr completed. Elapsed time: 0.69 minutes.

[feature] Allow override of memory and MemoryRedoubleStrategy for Ext…
8406263 
…ractIlluminaBarcodes

Use LinearMemoryRetry for ExtractIlluminaBarcodes.
Add maxNumIterations to `MemoryRetry`
@jacarey jacarey force-pushed the jc_allow_memory_override_eib branch from ba59502 to 8406263 Compare June 22, 2021 17:53
@jacarey jacarey marked this pull request as ready for review June 22, 2021 17:53
@jacarey jacarey requested a review from nh13 June 22, 2021 17:53
nh13
nh13 approved these changes Jun 23, 2021
View reviewed changes
Copy link
Member

@nh13 nh13 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jacarey jacarey merged commit 0f011b9 into master Jun 23, 2021
@jacarey jacarey deleted the jc_allow_memory_override_eib branch June 23, 2021 13:25
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@nh13 nh13 nh13 approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jacarey @nh13