diff --git a/README.md b/README.md index 5b0b595ff7..d5f27f3595 100644 --- a/README.md +++ b/README.md @@ -18,8 +18,12 @@ Download the latest version of the [yaml](./config/) and apply it the cluster. kubectl apply -f aws-k8s-cni.yaml ``` -Launch kubelet with network plugins set to cni (`--network-plugin=cni`), the cni directories configured (`--cni-config-dir` and `--cni-bin-dir`) and node ip set to the primary IPv4 address of the primary ENI for the instance (`--node-ip=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)`). -It is also recommended to set `--max-pods` equal to _(the number of ENIs for the instance type × (the number of IPs per ENI - 1)) + 2_; for details, see [vpc_ip_resource_limit.go][]. Setting `--max-pods` will prevent scheduling that exceeds the IP address resources available to the kubelet. +Launch kubelet with network plugins set to cni (`--network-plugin=cni`), the cni directories configured (`--cni-config-dir` +and `--cni-bin-dir`) and node ip set to the primary IPv4 address of the primary ENI for the instance +(`--node-ip=$(curl http://169.254.169.254/latest/meta-data/local-ipv4)`). +It is also recommended to set `--max-pods` equal to _(the number of ENIs for the instance type × +(the number of IPs per ENI - 1)) + 2_; for details, see [vpc_ip_resource_limit.go][]. Setting `--max-pods` will prevent +scheduling that exceeds the IP address resources available to the kubelet. [vpc_ip_resource_limit.go]: ./pkg/awsutils/vpc_ip_resource_limit.go @@ -73,83 +77,128 @@ The details can be found in [Proposal: CNI plugin for Kubernetes networking over ## ENI Allocation -When a worker node first joins the cluster, there is only 1 ENI along with all of its addresses in the ENI. Without any configuration, ipamD always try to keep one extra ENI. +When a worker node first joins the cluster, there is only 1 ENI along with all of its addresses in the ENI. Without any +configuration, ipamD always try to keep one extra ENI. -When number of pods running on the node exceeds the number of addresses on a single ENI, the CNI backend start allocating a new ENI and start using following allocation scheme: +When number of pods running on the node exceeds the number of addresses on a single ENI, the CNI backend start allocating +a new ENI and start using following allocation scheme: -For example, a m4.4xlarge node can have up to 8 ENIs, and each ENI can have up to 30 IP addresses. ( https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html ). +For example, a m4.4xlarge node can have up to 8 ENIs, and each ENI can have up to 30 IP addresses. +(https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html). * If the number of current running Pods is between 0 to 29, ipamD will allocate one more eni. And Warm-Pool size is 2 eni * (30 -1) = 58 * If the number of current running Pods is between 30 and 58, ipamD will allocate 2 more eni. And Warm-Pool size is 3 eni * (30 -1) = 87 ### CNI Configuration Variables -The Amazon VPC CNI plugin for Kubernetes supports a number of configuration options, which are set through environment variables\. The following environment variables are available, and all of them are optional\. - -`AWS_VPC_CNI_NODE_PORT_SUPPORT` -Type: Boolean -Default: `true` -Specifies whether `NodePort` services are enabled on a worker node's primary network interface\. This requires additional `iptables` rules and that the kernel's reverse path filter on the primary interface is set to `loose`\. - -`AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG` -Type: Boolean -Default: `false` -Specifies that your pods may use subnets and security groups that are independent of your worker node's VPC configuration\. By default, pods share the same subnet and security groups as the worker node's primary interface\. Setting this variable to `true` causes `ipamD` to use the security groups and VPC subnet in a worker node's `ENIConfig` for elastic network interface allocation\. You must create an `ENIConfig` custom resource for each subnet that your pods will reside in, and then annotate or label each worker node to use a specific `ENIConfig` (multiple worker nodes can be annotated or labelled with the same `ENIConfig`)\. Worker nodes can only be annotated with a single `ENIConfig` at a time, and the subnet in the `ENIConfig` must belong to the same Availability Zone that the worker node resides in\. -For more information, see [*CNI Custom Networking*](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) in the Amazon EKS User Guide\. - -`ENI_CONFIG_ANNOTATION_DEF` -Type: String -Default: `k8s.amazonaws.com/eniConfig` -Specifies node annotation key name\. This should be used when `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`. Annotation value will be used to set `ENIConfig` name\. Note that annotations take precedence over labels\. - -`ENI_CONFIG_LABEL_DEF` -Type: String -Default: `k8s.amazonaws.com/eniConfig` -Specifies node label key name\. This should be used when `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`\. Label value will be used to set `ENIConfig` name\. Note that annotations will take precedence over labels\. To use labels ensure annotation with key _k8s.amazonaws.com/eniConfig_ or defined key (in `ENI_CONFIG_ANNOTATION_DEF`) is not set on the node. -To select an `ENIConfig` based upon availability zone set this to _failure-domain.beta.kubernetes.io/zone_ and create an `ENIConfig` custom resource for each availability zone (e.g. `us-east-1a`). - -`AWS_VPC_K8S_CNI_EXTERNALSNAT` -Type: Boolean -Default: `false` -Specifies whether an external NAT gateway should be used to provide SNAT of secondary ENI IP addresses\. If set to `true`, the SNAT `iptables` rule and off\-VPC IP rule are not applied, and these rules are removed if they have already been applied\. -Disable SNAT if you need to allow inbound communication to your pods from external VPNs, direct connections, and external VPCs, and your pods do not need to access the Internet directly via an Internet Gateway\. However, your nodes must be running in a private subnet and connected to the internet through an AWS NAT Gateway or another external NAT device\. - -`AWS_VPC_K8S_CNI_RANDOMIZESNAT` -Type: String -Default: `hashrandom` +The Amazon VPC CNI plugin for Kubernetes supports a number of configuration options, which are set through environment variables. +The following environment variables are available, and all of them are optional. + +`AWS_VPC_CNI_NODE_PORT_SUPPORT` +Type: Boolean +Default: `true` +Specifies whether `NodePort` services are enabled on a worker node's primary network interface\. This requires additional +`iptables` rules and that the kernel's reverse path filter on the primary interface is set to `loose`. + +`AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG` +Type: Boolean +Default: `false` +Specifies that your pods may use subnets and security groups that are independent of your worker node's VPC configuration. +By default, pods share the same subnet and security groups as the worker node's primary interface\. Setting this variable +to `true` causes `ipamD` to use the security groups and VPC subnet in a worker node's `ENIConfig` for elastic network interface +allocation\. You must create an `ENIConfig` custom resource for each subnet that your pods will reside in, and then annotate or +label each worker node to use a specific `ENIConfig` (multiple worker nodes can be annotated or labelled with the same `ENIConfig`). +Worker nodes can only be annotated with a single `ENIConfig` at a time, and the subnet in the `ENIConfig` must belong to the +same Availability Zone that the worker node resides in. +For more information, see [*CNI Custom Networking*](https://docs.aws.amazon.com/eks/latest/userguide/cni-custom-network.html) +in the Amazon EKS User Guide. + +`ENI_CONFIG_ANNOTATION_DEF` +Type: String +Default: `k8s.amazonaws.com/eniConfig` +Specifies node annotation key name. This should be used when `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`. Annotation value +will be used to set `ENIConfig` name. Note that annotations take precedence over labels. + +`ENI_CONFIG_LABEL_DEF` +Type: String +Default: `k8s.amazonaws.com/eniConfig` +Specifies node label key name\. This should be used when `AWS_VPC_K8S_CNI_CUSTOM_NETWORK_CFG=true`. Label value will be used +to set `ENIConfig` name\. Note that annotations will take precedence over labels. To use labels ensure annotation with key +`k8s.amazonaws.com/eniConfig` or defined key (in `ENI_CONFIG_ANNOTATION_DEF`) is not set on the node. +To select an `ENIConfig` based upon availability zone set this to `failure-domain.beta.kubernetes.io/zone` and create an +`ENIConfig` custom resource for each availability zone (e.g. `us-east-1a`). + +`AWS_VPC_K8S_CNI_EXTERNALSNAT` +Type: Boolean +Default: `false` +Specifies whether an external NAT gateway should be used to provide SNAT of secondary ENI IP addresses. If set to `true`, the +SNAT `iptables` rule and off\-VPC IP rule are not applied, and these rules are removed if they have already been applied. +Disable SNAT if you need to allow inbound communication to your pods from external VPNs, direct connections, and external VPCs, +and your pods do not need to access the Internet directly via an Internet Gateway. However, your nodes must be running in a +private subnet and connected to the internet through an AWS NAT Gateway or another external NAT device. + +`AWS_VPC_K8S_CNI_RANDOMIZESNAT` +Type: String +Default: `hashrandom` Valid Values: `hashrandom`, `prng`, `none` -Specifies weather the SNAT `iptables` rule should randomize the outgoing ports for connections\. When enabled (`hashrandom`) the `--random` flag will be added to the SNAT `iptables` rule\. This should be used when `AWS_VPC_K8S_CNI_EXTERNALSNAT=true`\. -To use pseudo random number generation rather than hash based (i.e. `--random-fully`) use `prng` for the environment variable\. For old versions of `iptables` that do not support `--random-fully` this option will fall back to `--random`\. -Disable (`none`) this functionality if you rely on sequential port allocation for outgoing connections\. - -`WARM_ENI_TARGET` -Type: Integer -Default: `1` -Specifies the number of free elastic network interfaces \(and all of their available IP addresses\) that the `ipamD` daemon should attempt to keep available for pod assignment on the node\. By default, `ipamD` attempts to keep 1 elastic network interface and all of its IP addresses available for pod assignment\. -The number of IP addresses per network interface varies by instance type\. For more information, see [IP Addresses Per Network Interface Per Instance Type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI) in the *Amazon EC2 User Guide for Linux Instances*\. -For example, an `m4.4xlarge` launches with 1 network interface and 30 IP addresses\. If 5 pods are placed on the node and 5 free IP addresses are removed from the IP address warm pool, then `ipamD` attempts to allocate more interfaces until `WARM_ENI_TARGET` free interfaces are available on the node\. -If `WARM_IP_TARGET` is set, then this environment variable is ignored and the `WARM_IP_TARGET` behavior is used instead\. - -`WARM_IP_TARGET` -Type: Integer -Default: None -Specifies the number of free IP addresses that the `ipamD` daemon should attempt to keep available for pod assignment on the node\. For example, if `WARM_IP_TARGET` is set to 10, then `ipamD` attempts to keep 10 free IP addresses available at all times\. If the elastic network interfaces on the node are unable to provide these free addresses, `ipamD` attempts to allocate more interfaces until `WARM_IP_TARGET` free IP addresses are available\. -This environment variable overrides `WARM_ENI_TARGET` behavior\. - -`MAX_ENI` -Type: Integer -Default: None -Specifies the maximum number of ENIs that will be attached to the node. When MAX_ENI is unset or 0 (or lower), the setting is not used, and the maximum number of ENIs is always equal to the maximum number for the instance type in question. Even when MAX_ENI is a positive number, it is limited by the maximum number for the instance type. +Specifies weather the SNAT `iptables` rule should randomize the outgoing ports for connections\. When enabled (`hashrandom`) +the `--random` flag will be added to the SNAT `iptables` rule\. This should be used when `AWS_VPC_K8S_CNI_EXTERNALSNAT=true`. +To use pseudo random number generation rather than hash based (i.e. `--random-fully`) use `prng` for the environment variable. +For old versions of `iptables` that do not support `--random-fully` this option will fall back to `--random`. +Disable (`none`) this functionality if you rely on sequential port allocation for outgoing connections. + +`WARM_ENI_TARGET` +Type: Integer +Default: `1` +Specifies the number of free elastic network interfaces \(and all of their available IP addresses\) that the `ipamD` daemon should +attempt to keep available for pod assignment on the node\. By default, `ipamD` attempts to keep 1 elastic network interface and all +of its IP addresses available for pod assignment. The number of IP addresses per network interface varies by instance type. For more +information, see [IP Addresses Per Network Interface Per Instance Type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI) +in the *Amazon EC2 User Guide for Linux Instances*. + +For example, an `m4.4xlarge` launches with 1 network interface and 30 IP addresses\. If 5 pods are placed on the node and 5 free IP +addresses are removed from the IP address warm pool, then `ipamD` attempts to allocate more interfaces until `WARM_ENI_TARGET` free +interfaces are available on the node. +If `WARM_IP_TARGET` is set, then this environment variable is ignored and the `WARM_IP_TARGET` behavior is used instead. + +`WARM_IP_TARGET` +Type: Integer +Default: None +Specifies the number of free IP addresses that the `ipamD` daemon should attempt to keep available for pod assignment on the node. +For example, if `WARM_IP_TARGET` is set to 10, then `ipamD` attempts to keep 10 free IP addresses available at all times. If the +elastic network interfaces on the node are unable to provide these free addresses, `ipamD` attempts to allocate more interfaces +until `WARM_IP_TARGET` free IP addresses are available. +This environment variable overrides `WARM_ENI_TARGET` behavior. + +`MAX_ENI` +Type: Integer +Default: None +Specifies the maximum number of ENIs that will be attached to the node. When `MAX_ENI` is unset or 0 (or lower), the setting +is not used, and the maximum number of ENIs is always equal to the maximum number for the instance type in question. Even when +`MAX_ENI` is a positive number, it is limited by the maximum number for the instance type. `AWS_VPC_K8S_CNI_LOG_FILE` Type: String Default: Unset -Valid Values: stdout or a file path +Valid Values: `stdout` or a file path Specifies where to write the logging output. Either to stdout or to override the default file. +`DISABLE_INTROSPECTION` +Type: Boolean +Default: `false` +Specifies whether introspection endpoints are disabled on a worker node. Setting this to `true` will reduce the debugging +information we can get from the node when running the `aws-cni-support.sh` script. + +`DISABLE_METRICS` +Type: Boolean +Default: `false` +Specifies whether prometeus metrics endpoints are enabled on a worker node. + ### Notes -`L-IPAMD`(aws-node daemonSet) running on every worker node requires access to kubernetes API server. If it can **not** reach kubernetes API server, ipamD will exit and CNI will not be able to get any IP address for Pods. Here is a way to confirm if `L-IPAMD` has access to the kubernetes API server. +`L-IPAMD`(aws-node daemonSet) running on every worker node requires access to kubernetes API server. If it can **not** reach +kubernetes API server, ipamD will exit and CNI will not be able to get any IP address for Pods. Here is a way to confirm if +`L-IPAMD` has access to the kubernetes API server. ``` diff --git a/ipamd/introspect.go b/ipamd/introspect.go index ca8099f2a0..df723d6694 100644 --- a/ipamd/introspect.go +++ b/ipamd/introspect.go @@ -16,20 +16,22 @@ package ipamd import ( "encoding/json" "net/http" + "os" "strconv" "sync" "time" - log "github.com/cihub/seelog" - "github.com/prometheus/client_golang/prometheus/promhttp" - "github.com/aws/amazon-vpc-cni-k8s/pkg/networkutils" "github.com/aws/amazon-vpc-cni-k8s/pkg/utils" + log "github.com/cihub/seelog" ) const ( - // IntrospectionPort is the port for ipamd introspection - IntrospectionPort = 61678 + // introspectionAddress is listening on localhost 61679 for ipamd introspection + introspectionAddress = "127.0.0.1:61679" + + // Environment variable to disable the introspection endpoints + envDisableIntrospection = "DISABLE_INTROSPECTION" ) type rootResponse struct { @@ -42,29 +44,33 @@ type LoggingHandler struct { } func (lh LoggingHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) { - log.Info("Handling http request", "method", r.Method, "from", r.RemoteAddr, "uri", r.RequestURI) + log.Info("Handling http request: ", ", method: ", r.Method, ", from: ", r.RemoteAddr, ", URI: ", r.RequestURI) lh.h.ServeHTTP(w, r) } -// SetupHTTP sets up ipamd introspection service endpoint -func (c *IPAMContext) SetupHTTP() { - server := c.setupServer() +// ServeIntrospection sets up ipamd introspection endpoints +func (c *IPAMContext) ServeIntrospection() { + if disableIntrospection() { + log.Info("Introspection endpoints disabled") + return + } + log.Info("Serving introspection endpoints on ", introspectionAddress) + server := c.setupIntrospectionServer() for { once := sync.Once{} - utils.RetryWithBackoff(utils.NewSimpleBackoff(time.Second, time.Minute, 0.2, 2), func() error { - // TODO, make this cancellable and use the passed in context; for - // now, not critical if this gets interrupted + _ = utils.RetryWithBackoff(utils.NewSimpleBackoff(time.Second, time.Minute, 0.2, 2), func() error { err := server.ListenAndServe() once.Do(func() { - log.Error("Error running http api", "err", err) + log.Error("Error running http API: ", err) }) return err }) } } -func (c *IPAMContext) setupServer() *http.Server { +func (c *IPAMContext) setupIntrospectionServer() *http.Server { + // If enabled, add introspection endpoints serverFunctions := map[string]func(w http.ResponseWriter, r *http.Request){ "/v1/enis": eniV1RequestHandler(c), "/v1/eni-configs": eniConfigRequestHandler(c), @@ -81,26 +87,24 @@ func (c *IPAMContext) setupServer() *http.Server { availableCommandResponse, err := json.Marshal(&availableCommands) if err != nil { - log.Error("Failed to Marshal: %v", err) + log.Error("Failed to marshal: %v", err) } defaultHandler := func(w http.ResponseWriter, r *http.Request) { logErr(w.Write(availableCommandResponse)) } - serveMux := http.NewServeMux() serveMux.HandleFunc("/", defaultHandler) for key, fn := range serverFunctions { serveMux.HandleFunc(key, fn) } - serveMux.Handle("/metrics", promhttp.Handler()) // Log all requests and then pass through to serveMux loggingServeMux := http.NewServeMux() loggingServeMux.Handle("/", LoggingHandler{serveMux}) server := &http.Server{ - Addr: ":" + strconv.Itoa(IntrospectionPort), + Addr: introspectionAddress, Handler: loggingServeMux, ReadTimeout: 5 * time.Second, WriteTimeout: 5 * time.Second, @@ -112,7 +116,7 @@ func eniV1RequestHandler(ipam *IPAMContext) func(http.ResponseWriter, *http.Requ return func(w http.ResponseWriter, r *http.Request) { responseJSON, err := json.Marshal(ipam.dataStore.GetENIInfos()) if err != nil { - log.Error("Failed to marshal ENI data: %v", err) + log.Errorf("Failed to marshal ENI data: %v", err) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) return } @@ -124,7 +128,7 @@ func podV1RequestHandler(ipam *IPAMContext) func(http.ResponseWriter, *http.Requ return func(w http.ResponseWriter, r *http.Request) { responseJSON, err := json.Marshal(ipam.dataStore.GetPodInfos()) if err != nil { - log.Error("Failed to marshal pod data: %v", err) + log.Errorf("Failed to marshal pod data: %v", err) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) return } @@ -136,7 +140,7 @@ func eniConfigRequestHandler(ipam *IPAMContext) func(http.ResponseWriter, *http. return func(w http.ResponseWriter, r *http.Request) { responseJSON, err := json.Marshal(ipam.eniConfig.Getter()) if err != nil { - log.Error("Failed to marshal ENI config: %v", err) + log.Errorf("Failed to marshal ENI config: %v", err) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) return } @@ -148,7 +152,7 @@ func networkEnvV1RequestHandler() func(http.ResponseWriter, *http.Request) { return func(w http.ResponseWriter, r *http.Request) { responseJSON, err := json.Marshal(networkutils.GetConfigForDebug()) if err != nil { - log.Error("Failed to marshal network env var data: %v", err) + log.Errorf("Failed to marshal network env var data: %v", err) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) return } @@ -160,7 +164,7 @@ func ipamdEnvV1RequestHandler() func(http.ResponseWriter, *http.Request) { return func(w http.ResponseWriter, r *http.Request) { responseJSON, err := json.Marshal(GetConfigForDebug()) if err != nil { - log.Error("Failed to marshal ipamd env var data: %v", err) + log.Errorf("Failed to marshal ipamd env var data: %v", err) http.Error(w, http.StatusText(http.StatusInternalServerError), http.StatusInternalServerError) return } @@ -173,3 +177,19 @@ func logErr(_ int, err error) { log.Errorf("Write failed: %v", err) } } + +// disableIntrospection returns true if we should disable the introspection +func disableIntrospection() bool { + return getEnvBoolWithDefault(envDisableIntrospection, false) +} + +func getEnvBoolWithDefault(envName string, def bool) bool { + if strValue := os.Getenv(envName); strValue != "" { + parsedValue, err := strconv.ParseBool(strValue) + if err == nil { + return parsedValue + } + log.Errorf("Failed to parse %s, using default `%t`: %v", envName, def, err.Error()) + } + return def +} diff --git a/ipamd/ipamd_test.go b/ipamd/ipamd_test.go index 263b5858c1..757146264b 100644 --- a/ipamd/ipamd_test.go +++ b/ipamd/ipamd_test.go @@ -42,7 +42,6 @@ const ( primaryENIid = "eni-00000000" secENIid = "eni-00000001" testAttachmentID = "eni-00000000-attach" - eniID = "eni-5731da78" primaryMAC = "12:ef:2a:98:e5:5a" secMAC = "12:ef:2a:98:e5:5b" primaryDevice = 0 @@ -51,10 +50,8 @@ const ( secSubnet = "10.10.20.0/24" ipaddr01 = "10.10.10.11" ipaddr02 = "10.10.10.12" - ipaddr03 = "10.10.10.13" ipaddr11 = "10.10.20.11" ipaddr12 = "10.10.20.12" - ipaddr13 = "10.10.20.13" vpcCIDR = "10.10.0.0/16" ) @@ -163,12 +160,12 @@ func TestNodeInit(t *testing.T) { } func TestIncreaseIPPoolDefault(t *testing.T) { - os.Unsetenv(envCustomNetworkCfg) + _ = os.Unsetenv(envCustomNetworkCfg) testIncreaseIPPool(t, false) } func TestIncreaseIPPoolCustomENI(t *testing.T) { - os.Setenv(envCustomNetworkCfg, "true") + _ = os.Setenv(envCustomNetworkCfg, "true") testIncreaseIPPool(t, true) } @@ -337,15 +334,15 @@ func TestGetWarmENITarget(t *testing.T) { ctrl, _, _, _, _, _ := setup(t) defer ctrl.Finish() - os.Setenv("WARM_IP_TARGET", "5") + _ = os.Setenv("WARM_IP_TARGET", "5") warmIPTarget := getWarmIPTarget() assert.Equal(t, warmIPTarget, 5) - os.Unsetenv("WARM_IP_TARGET") + _ = os.Unsetenv("WARM_IP_TARGET") warmIPTarget = getWarmIPTarget() assert.Equal(t, warmIPTarget, noWarmIPTarget) - os.Setenv("WARM_IP_TARGET", "non-integer-string") + _ = os.Setenv("WARM_IP_TARGET", "non-integer-string") warmIPTarget = getWarmIPTarget() assert.Equal(t, warmIPTarget, noWarmIPTarget) } @@ -355,32 +352,32 @@ func TestGetMaxENI(t *testing.T) { defer ctrl.Finish() // MaxENI 5 is less than upper bound of 10, so 5 - os.Setenv("MAX_ENI", "5") + _ = os.Setenv("MAX_ENI", "5") maxENI := getMaxENI(10) assert.Equal(t, maxENI, 5) // MaxENI 5 is greater than upper bound of 4, so 4 - os.Setenv("MAX_ENI", "5") + _ = os.Setenv("MAX_ENI", "5") maxENI = getMaxENI(4) assert.Equal(t, maxENI, 4) // MaxENI 0 is 0, which means disabled; so use upper bound - os.Setenv("MAX_ENI", "0") + _ = os.Setenv("MAX_ENI", "0") maxENI = getMaxENI(4) assert.Equal(t, maxENI, 4) // MaxENI 1 is less than upper bound of 4, so 1. - os.Setenv("MAX_ENI", "1") + _ = os.Setenv("MAX_ENI", "1") maxENI = getMaxENI(4) assert.Equal(t, maxENI, 1) // Empty MaxENI means disabled, so use upper bound - os.Unsetenv("MAX_ENI") + _ = os.Unsetenv("MAX_ENI") maxENI = getMaxENI(10) assert.Equal(t, maxENI, 10) // Invalid MaxENI means disabled, so use upper bound - os.Setenv("MAX_ENI", "non-integer-string") + _ = os.Setenv("MAX_ENI", "non-integer-string") maxENI = getMaxENI(10) assert.Equal(t, maxENI, 10) } @@ -398,20 +395,20 @@ func TestGetWarmIPTargetState(t *testing.T) { mockContext.dataStore = datastore.NewDataStore() - os.Unsetenv("WARM_IP_TARGET") + _ = os.Unsetenv("WARM_IP_TARGET") _, _, warmIPTargetDefined := mockContext.ipTargetState() assert.False(t, warmIPTargetDefined) - os.Setenv("WARM_IP_TARGET", "5") + _ = os.Setenv("WARM_IP_TARGET", "5") short, over, warmIPTargetDefined := mockContext.ipTargetState() assert.True(t, warmIPTargetDefined) assert.Equal(t, 5, short) assert.Equal(t, 0, over) // add 2 addresses to datastore - mockContext.dataStore.AddENI("eni-1", 1, true) - mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.1") - mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.2") + _ = mockContext.dataStore.AddENI("eni-1", 1, true) + _ = mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.1") + _ = mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.2") short, over, warmIPTargetDefined = mockContext.ipTargetState() assert.True(t, warmIPTargetDefined) @@ -419,9 +416,9 @@ func TestGetWarmIPTargetState(t *testing.T) { assert.Equal(t, 0, over) // add 3 more addresses to datastore - mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.3") - mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.4") - mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.5") + _ = mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.3") + _ = mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.4") + _ = mockContext.dataStore.AddIPv4AddressFromStore("eni-1", "1.1.1.5") short, over, warmIPTargetDefined = mockContext.ipTargetState() assert.True(t, warmIPTargetDefined) diff --git a/ipamd/metrics.go b/ipamd/metrics.go new file mode 100644 index 0000000000..795db72e95 --- /dev/null +++ b/ipamd/metrics.go @@ -0,0 +1,73 @@ +// Copyright 2014-2017 Amazon.com, Inc. or its affiliates. All Rights Reserved. +// +// Licensed under the Apache License, Version 2.0 (the "License"). You may +// not use this file except in compliance with the License. A copy of the +// License is located at +// +// http://aws.amazon.com/apache2.0/ +// +// or in the "license" file accompanying this file. This file is distributed +// on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either +// express or implied. See the License for the specific language governing +// permissions and limitations under the License. + +package ipamd + +import ( + "net/http" + "strconv" + "sync" + "time" + + log "github.com/cihub/seelog" + "github.com/prometheus/client_golang/prometheus/promhttp" + + "github.com/aws/amazon-vpc-cni-k8s/pkg/utils" +) + +const ( + // metricsPort is the port for prometheus metrics + metricsPort = 61678 + + // Environment variable to disable the metrics endpoint on 61678 + envDisableMetrics = "DISABLE_METRICS" +) + +// ServeMetrics sets up ipamd metrics and introspection endpoints +func (c *IPAMContext) ServeMetrics() { + if disableMetrics() { + log.Info("Metrics endpoint disabled") + return + } + + log.Info("Serving metrics on port ", metricsPort) + server := c.setupMetricsServer() + for { + once := sync.Once{} + _ = utils.RetryWithBackoff(utils.NewSimpleBackoff(time.Second, time.Minute, 0.2, 2), func() error { + err := server.ListenAndServe() + once.Do(func() { + log.Error("Error running http API: ", err) + }) + return err + }) + } +} + +func (c *IPAMContext) setupMetricsServer() *http.Server { + // Always add the metrics endpoint + serveMux := http.NewServeMux() + serveMux.Handle("/metrics", promhttp.Handler()) + server := &http.Server{ + Addr: ":" + strconv.Itoa(metricsPort), + Handler: serveMux, + ReadTimeout: 5 * time.Second, + WriteTimeout: 5 * time.Second, + } + return server +} + +// disableMetrics returns true if we should disable metrics +func disableMetrics() bool { + return getEnvBoolWithDefault(envDisableMetrics, false) +} diff --git a/ipamd/rpc_handler.go b/ipamd/rpc_handler.go index df75cd6281..4f4ede580b 100644 --- a/ipamd/rpc_handler.go +++ b/ipamd/rpc_handler.go @@ -31,7 +31,7 @@ import ( ) const ( - port = "127.0.0.1:50051" + ipamdgRPCaddress = "127.0.0.1:50051" ) type server struct { @@ -91,7 +91,7 @@ func (s *server) DelNetwork(ctx context.Context, in *pb.DelNetworkRequest) (*pb. // RunRPCHandler handles request from gRPC func (c *IPAMContext) RunRPCHandler() error { - lis, err := net.Listen("tcp", port) + lis, err := net.Listen("tcp", ipamdgRPCaddress) if err != nil { log.Errorf("Failed to listen gRPC port: %v", err) return errors.Wrap(err, "ipamd: failed to listen to gRPC port") diff --git a/main.go b/main.go index 3f2d0d45e2..ee284947a3 100644 --- a/main.go +++ b/main.go @@ -60,13 +60,24 @@ func _main() int { awsK8sAgent, err := ipamd.New(discoverController, eniConfigController) if err != nil { - log.Error("initialization failure", err) + log.Error("Initialization failure ", err) return 1 } + // Pool manager go awsK8sAgent.StartNodeIPPoolManager() - go awsK8sAgent.SetupHTTP() - awsK8sAgent.RunRPCHandler() + + // Prometheus metrics + go awsK8sAgent.ServeMetrics() + + // CNI introspection endpoints + go awsK8sAgent.ServeIntrospection() + + err = awsK8sAgent.RunRPCHandler() + if err != nil { + log.Error("Failed to set up gRPC handler ", err) + return 1 + } return 0 } diff --git a/scripts/aws-cni-support.sh b/scripts/aws-cni-support.sh index 17a98fc026..8b374373b3 100755 --- a/scripts/aws-cni-support.sh +++ b/scripts/aws-cni-support.sh @@ -17,18 +17,18 @@ # Set language to C to make sorting consistent among different environments. export LANG=C -set -euo pipefail +set -uo pipefail LOG_DIR="/var/log/aws-routed-eni" mkdir -p ${LOG_DIR} -# collecting L-IPAMD introspection data -curl http://localhost:61678/v1/enis > ${LOG_DIR}/eni.out -curl http://localhost:61678/v1/pods > ${LOG_DIR}/pod.out -curl http://localhost:61678/v1/networkutils-env-settings > ${LOG_DIR}/networkutils-env.out -curl http://localhost:61678/v1/ipamd-env-settings > ${LOG_DIR}/ipamd-env.out -curl http://localhost:61678/v1/eni-configs > ${LOG_DIR}/eni-configs.out +# Collecting L-IPAMD introspection data +curl http://localhost:61679/v1/enis > ${LOG_DIR}/eni.out +curl http://localhost:61679/v1/pods > ${LOG_DIR}/pod.out +curl http://localhost:61679/v1/networkutils-env-settings > ${LOG_DIR}/networkutils-env.out +curl http://localhost:61679/v1/ipamd-env-settings > ${LOG_DIR}/ipamd-env.out +curl http://localhost:61679/v1/eni-configs > ${LOG_DIR}/eni-configs.out -# metrics +# Metrics curl http://localhost:61678/metrics 2>&1 > ${LOG_DIR}/metrics.out # Collecting kubelet introspection data diff --git a/scripts/install-aws.sh b/scripts/install-aws.sh index 89549e08d8..5814284a4a 100755 --- a/scripts/install-aws.sh +++ b/scripts/install-aws.sh @@ -1,5 +1,5 @@ #!/usr/bin/env bash -echo "=====Starting installing AWS-CNI =========" +echo "===== Starting installing AWS-CNI =========" sed -i s/__VETHPREFIX__/${AWS_VPC_K8S_CNI_VETHPREFIX:-"eni"}/g /app/10-aws.conflist cp /app/aws-cni /host/opt/cni/bin/ cp /app/portmap /host/opt/cni/bin/ @@ -10,5 +10,5 @@ if [[ -f /host/etc/cni/net.d/aws.conf ]]; then rm /host/etc/cni/net.d/aws.conf fi -echo "=====Starting amazon-k8s-agent ===========" +echo "===== Starting amazon-k8s-agent ===========" /app/aws-k8s-agent