@InterfaceAudience.Private @InterfaceStability.Unstable public class DominantResourceCalculator extends ResourceCalculator
ResourceCalculator which uses the concept of
dominant resource to compare multi-dimensional resources.
Essentially the idea is that the in a multi-resource environment,
the resource allocation should be determined by the dominant share
of an entity (user or queue), which is the maximum share that the
entity has been allocated of any resource.
In a nutshell, it seeks to maximize the minimum dominant share across
all entities.
For example, if user A runs CPU-heavy tasks and user B runs
memory-heavy tasks, it attempts to equalize CPU share of user A
with Memory-share of user B.
In the single resource case, it reduces to max-min fairness for that resource.
See the Dominant Resource Fairness paper for more details:
www.cs.berkeley.edu/~matei/papers/2011/nsdi_drf.pdf| Constructor and Description |
|---|
DominantResourceCalculator() |
| Modifier and Type | Method and Description |
|---|---|
int |
compare(org.apache.hadoop.yarn.api.records.Resource clusterResource,
org.apache.hadoop.yarn.api.records.Resource lhs,
org.apache.hadoop.yarn.api.records.Resource rhs,
boolean singleType)
On a cluster with capacity
clusterResource, compare lhs
and rhs. |
long |
computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available,
org.apache.hadoop.yarn.api.records.Resource required)
Compute the number of containers which can be allocated given
available and required resources. |
float |
divide(org.apache.hadoop.yarn.api.records.Resource clusterResource,
org.apache.hadoop.yarn.api.records.Resource numerator,
org.apache.hadoop.yarn.api.records.Resource denominator)
Divide resource
numerator by resource denominator
using specified policy (domination, average, fairness etc.); hence overall
clusterResource is provided for context. |
org.apache.hadoop.yarn.api.records.Resource |
divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
float denominator)
Divide-and-ceil
numerator by denominator. |
org.apache.hadoop.yarn.api.records.Resource |
divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
int denominator)
Divide-and-ceil
numerator by denominator. |
org.apache.hadoop.yarn.api.records.Resource |
divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
long denominator) |
boolean |
fitsIn(org.apache.hadoop.yarn.api.records.Resource cluster,
org.apache.hadoop.yarn.api.records.Resource smaller,
org.apache.hadoop.yarn.api.records.Resource bigger)
Check if a smaller resource can be contained by bigger resource.
|
boolean |
isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
Determine if all resources are zero.
|
boolean |
isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
Check if resource has any major resource types (which are all NodeManagers
included) has a > 0 value.
|
boolean |
isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
Check if resource has any major resource types (which are all NodeManagers
included) a zero value or negative value.
|
boolean |
isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
Determine if a resource is not suitable for use as a divisor
(will result in divide by 0, etc)
|
org.apache.hadoop.yarn.api.records.Resource |
multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
double by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Multiply resource
r by factor by
and normalize down using step-factor stepFactor. |
org.apache.hadoop.yarn.api.records.Resource |
multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
double[] by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Multiply resource
r by factor by
and normalize up using step-factor stepFactor. |
org.apache.hadoop.yarn.api.records.Resource |
multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
double by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Multiply resource
r by factor by
and normalize up using step-factor stepFactor. |
org.apache.hadoop.yarn.api.records.Resource |
normalize(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource minimumResource,
org.apache.hadoop.yarn.api.records.Resource maximumResource,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Normalize resource
r given the base
minimumResource and verify against max allowed
maximumResource using a step factor for the normalization. |
float |
ratio(org.apache.hadoop.yarn.api.records.Resource a,
org.apache.hadoop.yarn.api.records.Resource b)
Ratio of resource
a to resource b. |
org.apache.hadoop.yarn.api.records.Resource |
roundDown(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Round-down resource
r given factor stepFactor. |
org.apache.hadoop.yarn.api.records.Resource |
roundUp(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
Round-up resource
r given factor stepFactor. |
compare, divideAndCeil, divideAndCeil, divideAndCeil, divideAndCeil, roundDown, roundDown, roundUp, roundUppublic int compare(org.apache.hadoop.yarn.api.records.Resource clusterResource,
org.apache.hadoop.yarn.api.records.Resource lhs,
org.apache.hadoop.yarn.api.records.Resource rhs,
boolean singleType)
ResourceCalculatorclusterResource, compare lhs
and rhs. Consider all resources unless singleType is set
to true. When singleType is set to true, consider only one
resource as per the ResourceCalculator implementation; the
DefaultResourceCalculator considers memory and
DominantResourceCalculator considers the dominant resource.compare in class ResourceCalculatorclusterResource - cluster capacitylhs - First Resource to comparerhs - Second Resource to comparesingleType - Whether to consider a single resource type or all
resource typeslhs is smaller, 0 if equal and 1 if it is largerpublic long computeAvailableContainers(org.apache.hadoop.yarn.api.records.Resource available,
org.apache.hadoop.yarn.api.records.Resource required)
ResourceCalculatoravailable and required resources.computeAvailableContainers in class ResourceCalculatoravailable - available resourcesrequired - required resourcespublic float divide(org.apache.hadoop.yarn.api.records.Resource clusterResource,
org.apache.hadoop.yarn.api.records.Resource numerator,
org.apache.hadoop.yarn.api.records.Resource denominator)
ResourceCalculatornumerator by resource denominator
using specified policy (domination, average, fairness etc.); hence overall
clusterResource is provided for context.divide in class ResourceCalculatorclusterResource - cluster resourcesnumerator - numeratordenominator - denominatornumerator/denominator
using specific policypublic boolean isInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
ResourceCalculatorisInvalidDivisor in class ResourceCalculatorr - resourcepublic boolean isAllInvalidDivisor(org.apache.hadoop.yarn.api.records.Resource r)
ResourceCalculatorisAllInvalidDivisor in class ResourceCalculatorr - resourcepublic float ratio(org.apache.hadoop.yarn.api.records.Resource a,
org.apache.hadoop.yarn.api.records.Resource b)
ResourceCalculatora to resource b.ratio in class ResourceCalculatora - resourceb - resourcea to resource bpublic org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
int denominator)
ResourceCalculatornumerator by denominator.divideAndCeil in class ResourceCalculatornumerator - numerator resourcedenominator - denominatorpublic org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
long denominator)
public org.apache.hadoop.yarn.api.records.Resource divideAndCeil(org.apache.hadoop.yarn.api.records.Resource numerator,
float denominator)
ResourceCalculatornumerator by denominator.divideAndCeil in class ResourceCalculatornumerator - numerator resourcedenominator - denominatorpublic org.apache.hadoop.yarn.api.records.Resource normalize(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource minimumResource,
org.apache.hadoop.yarn.api.records.Resource maximumResource,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculatorr given the base
minimumResource and verify against max allowed
maximumResource using a step factor for the normalization.normalize in class ResourceCalculatorr - resourceminimumResource - minimum valuemaximumResource - the upper bound of the resource to be allocatedstepFactor - the increment for resources to be allocatedpublic org.apache.hadoop.yarn.api.records.Resource roundUp(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculatorr given factor stepFactor.roundUp in class ResourceCalculatorr - resourcestepFactor - step-factorpublic org.apache.hadoop.yarn.api.records.Resource roundDown(org.apache.hadoop.yarn.api.records.Resource r,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculatorr given factor stepFactor.roundDown in class ResourceCalculatorr - resourcestepFactor - step-factorpublic org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
double[] by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculatorr by factor by
and normalize up using step-factor stepFactor.multiplyAndNormalizeUp in class ResourceCalculatorr - resource to be multipliedby - multiplier array for all resource typesstepFactor - factor by which to normalize uppublic org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeUp(org.apache.hadoop.yarn.api.records.Resource r,
double by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculatorr by factor by
and normalize up using step-factor stepFactor.multiplyAndNormalizeUp in class ResourceCalculatorr - resource to be multipliedby - multiplierstepFactor - factor by which to normalize uppublic org.apache.hadoop.yarn.api.records.Resource multiplyAndNormalizeDown(org.apache.hadoop.yarn.api.records.Resource r,
double by,
org.apache.hadoop.yarn.api.records.Resource stepFactor)
ResourceCalculatorr by factor by
and normalize down using step-factor stepFactor.multiplyAndNormalizeDown in class ResourceCalculatorr - resource to be multipliedby - multiplierstepFactor - factor by which to normalize downpublic boolean fitsIn(org.apache.hadoop.yarn.api.records.Resource cluster,
org.apache.hadoop.yarn.api.records.Resource smaller,
org.apache.hadoop.yarn.api.records.Resource bigger)
ResourceCalculatorfitsIn in class ResourceCalculatorpublic boolean isAnyMajorResourceZeroOrNegative(org.apache.hadoop.yarn.api.records.Resource resource)
ResourceCalculatorisAnyMajorResourceZeroOrNegative in class ResourceCalculatorresource - resourcepublic boolean isAnyMajorResourceAboveZero(org.apache.hadoop.yarn.api.records.Resource resource)
ResourceCalculatorisAnyMajorResourceAboveZero in class ResourceCalculatorresource - resourceCopyright © 2022 Apache Software Foundation. All rights reserved.