YARN-11736. Enhance MultiNodeLookupPolicy to allow configuration of extended comparators for better usability. #7121
Conversation
…xtended comparators for better usability.
|
💔 -1 overall
This message was automatically generated. |
|
@yangwwei @sunilgovind @shameersss1 Could you please help to review this PR? Thanks! |
…th the same policy class and improve UT.
|
💔 -1 overall
This message was automatically generated. |
|
Hi, @yangjiandan It seems that you were using and contributing for multi-node mechanism recently, could you please help to review this PR? Thanks. |
|
Sure, will review this week. |
There was a problem hiding this comment.
Please add the modification about the configs and new class here : https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/CapacityScheduler.html (you can find the changes to be done for this page in the hadoop code base itself) so that the end users know about its existence.
| @@ -2915,11 +2915,20 @@ private void updateResourceValuesFromConfig(Set<String> resourceTypes, | |||
| } | |||
| } | |||
|
|
|||
| public static final String MULTI_NODE_SORTING_POLICY_SUFFIX = | |||
There was a problem hiding this comment.
Can we have javadoc explaining the use of this config
| PREFIX + MULTI_NODE_SORTING_POLICY_SUFFIX; | ||
|
|
||
| public static final String MULTI_NODE_SORTING_POLICY_CURRENT_NAME = | ||
| MULTI_NODE_SORTING_POLICY_NAME + ".current-name"; |
There was a problem hiding this comment.
Can we have javadoc explaining the use of this config
| private String policyClassName; | ||
| private long sortingInterval; | ||
|
|
||
| public MultiNodePolicySpec(String policyClassName, long timeout) { | ||
| public MultiNodePolicySpec(String policyName, String policyClassName, |
There was a problem hiding this comment.
why both policyName and policyClassName is required here ?
There was a problem hiding this comment.
In MultiNodeSorter#initPolicy, Policy instance will be created based on policyClassName, and policyName will be used to get conf belong to this policy instance in MultiComparatorPolicy#setConf. This is the only way for every policy instance to know which configuration belong to it, another way is to update the policy interface that I prefer not to use. If there are better approaches, feel free to propose them.
.../java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/placement/MultiNodeSorter.java
Outdated
Show resolved
Hide resolved
| } | ||
| Configuration policyConf = new Configuration(this.getConfig()); |
There was a problem hiding this comment.
Can we reuse config object instead of creating new one ?
There was a problem hiding this comment.
In MultiNodeSortingManager#createAllPolicies, we can see all the MultiNodeSorter instances owns a shared config, policyName will be set in policyConf, which is a instance-level configuration, so that policyInstance can get the configurations belong to itself.
| // conf keys and default values | ||
| public static final String COMPARATORS_CONF_KEY = "comparators"; | ||
| protected static final List<Comparator> DEFAULT_COMPARATORS = Collections | ||
| .unmodifiableList(Arrays.asList( |
There was a problem hiding this comment.
So this means the default sorting policy is based on resource utilization, the node having less resource utilization will be given priority
There was a problem hiding this comment.
Yes, default comparators will only be used when there's no comparators configured for this policy or configured incorrectly. Because I believe optimizing the workload distribution among nodes is the primary use case.
| } | ||
| this.conf = conf; | ||
| String policyName = conf.get( | ||
| CapacitySchedulerConfiguration.MULTI_NODE_SORTING_POLICY_CURRENT_NAME); |
There was a problem hiding this comment.
what is the difference between MULTI_NODE_SORTING_POLICY_CURRENT_NAME and ``MULTI_NODE_SORTING_POLICY`
There was a problem hiding this comment.
CurrentlyMULTI_NODE_SORTING_POLICY_NAME is the prefix of all the configurations of multi-node policy: yarn.scheduler.capacity.multi-node-sorting.policy, it's not a proper name but used in many places so I prefer not to update it. MULTI_NODE_SORTING_POLICY_CURRENT_NAME is used to transfer the policyName to policy instance.
@shameersss1 There is no introduce about multi-node policy in the document yet, therefore I am unable to update or add any further information. |
|
@TaoYang526 @shameersss1 I am eager to help, even though I’m not very familiar with this part of YARN. I will do my best. If we can confirm together that the modified code is fine, we can proceed with merging it. I've reviewed @shameersss1 code, and the quality is quite good. |
|
@slfan1989 Thank you for joining us, it's great to have you help out. |
|
@shameersss1 Thanks for the review. I have added javadoc for key fields in the last commit. |
|
💔 -1 overall
This message was automatically generated. |
Description of PR
Please refer to JIRA: YARN-11736
How was this patch tested?
UT
For code changes:
LICENSE,LICENSE-binary,NOTICE-binaryfiles?