Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Set the kubelet --resolv-conf flag conditionally at kubeadm init time #845

Closed
luxas opened this issue May 22, 2018 · 5 comments · Fixed by kubernetes/kubernetes#64665
Closed

Set the kubelet --resolv-conf flag conditionally at kubeadm init time #845

luxas opened this issue May 22, 2018 · 5 comments · Fixed by kubernetes/kubernetes#64665
Assignees
@stealthybox
Labels
area/releasing area/upgrades lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor.
Milestone
v1.11

Comments

@luxas
Copy link
Member

luxas commented May 22, 2018

When kubernetes/kubernetes#63887 merges, we should set the kubelet's --resolv-conf flag to /run/systemd/resolve/resolv.conf conditionally at kubeadm init time, in cmd/kubeadm/app/phases/kubelet/flags.go.

This instead of doing it statically in the deb/rpm packages.
cc @randomvariable @detiber @stealthybox @kad
xref related issue: #822
xref related PR: kubernetes/kubernetes#63632
@luxas luxas added this to the v1.11 milestone May 22, 2018
This was referenced May 22, 2018
Closed
Closed
@luxas
Copy link
Member Author

luxas commented May 23, 2018

@randomvariable has volunteered to start working on this in a day or two. Please let us know here when you do 👍

@randomvariable
Copy link
Member

Was going to pick this up, but notice you have #847, but we also want run time detection, or sensible defaulting for newer Ubuntu's don't we?

@luxas
Copy link
Member Author

luxas commented May 28, 2018

@randomvariable This should be done automatically. If systemd-resolved is used, the kubelet's --resolv-conf argument should be set by kubeadm so we don't break anyone. #847 basically means the user could do it him/herself, and even override kubeadm's defaulting for --resolv-conf, but doesn't mean we should automatically set this flag. Does that make sense to you?

@luxas luxas mentioned this issue May 29, 2018
Closed
@stealthybox
Copy link
Member

/assign

@stealthybox
Copy link
Member

We plan to get this into 1.11 before the weekend.
-- discussed in Kubeadm Office Hours - Wed May 30

@luxas luxas added the lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor. label May 30, 2018
stealthybox added a commit to stealthybox/kubernetes that referenced this issue Jun 3, 2018
@stealthybox
kubeadm: Set the kubelet --resolv-conf flag conditionally on init
b7df5cd 
Fixes kubernetes/kubeadm#845
@stealthybox stealthybox mentioned this issue Jun 3, 2018
Merged
k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Jun 4, 2018
Merge pull request #64665 from stealthybox/feature/kubeadm_845-system…
86ae84b 
…d-resolved

Automatic merge from submit-queue. If you want to cherry-pick this change to another branch, please follow the instructions <a href="https:/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>.

kubeadm: Set the kubelet `--resolv-conf` flag conditionally on init

**What this PR does / why we need it**:
`kubeadm init` detects if systemd-resolved is running and configures the kubelet to use a working resolv.conf.
This patch also removes the warning message prompting manual user action for this configuration.

/area kubeadm
/area kubelet
/area dns
/kind bug
/priority important-soon

/sig cluster-lifecycle
/assign @timothysc

**Which issue(s) this PR fixes** 
Fixes kubernetes/kubeadm#845

**Special notes for your reviewer**:
See the difference in `KUBELET_KUBEADM_ARGS` when running with this build and enabling the resolved daemon on Ubuntu 17.10:
```bash
root@vagrant:/vagrant/bin# bash << EOF
systemctl start systemd-resolved
./845_kubeadm init |& tail -n5
cat /var/lib/kubelet/kubeadm-flags.env

./845_kubeadm reset --force |& tail -n2
systemctl stop systemd-resolved
echo nameserver 8.8.8.8 > /etc/resolv.conf

./845_kubeadm init |& tail -n5
cat /var/lib/kubelet/kubeadm-flags.env

EOF
You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.0.2.15:6443 --token 77q84j.0342evur7rrfrwwx --discovery-token-ca-cert-hash sha256:190040f9c3adf8410bc6766dac79f8679870190564e15e8f8d1704fafa03f678

KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni --resolv-conf=/run/systemd/resolve/resolv.conf
I0603 06:49:16.592482   14106 reset.go:276] [reset] deleting contents of config directories: [/etc/kubernetes/manifests /etc/kubernetes/pki]
I0603 06:49:16.592858   14106 reset.go:290] [reset] deleting files: [/etc/kubernetes/admin.conf /etc/kubernetes/kubelet.conf /etc/kubernetes/bootstrap-kubelet.conf /etc/kubernetes/controller-manager.conf /etc/kubernetes/scheduler.conf]
You can now join any number of machines by running the following on each node
as root:

  kubeadm join 10.0.2.15:6443 --token 8mdart.gp67vq3nh9urq4z5 --discovery-token-ca-cert-hash sha256:da6b2e5841546eae134524b045e782f0dd91a6b53becc8d69c15d9eab9c88758

KUBELET_KUBEADM_ARGS=--cgroup-driver=cgroupfs --cni-bin-dir=/opt/cni/bin --cni-conf-dir=/etc/cni/net.d --network-plugin=cni
```

**Release note**:
```release-note
`kubeadm init` detects if systemd-resolved is running and configures the kubelet to use a working resolv.conf.
```
wenjiaswe pushed a commit to wenjiaswe/kubernetes that referenced this issue Jun 19, 2018
@stealthybox @wenjiaswe
kubeadm: Set the kubelet --resolv-conf flag conditionally on init
c219587 
Fixes kubernetes/kubeadm#845
This was referenced Sep 29, 2018
Closed
Closed
@bhack bhack mentioned this issue Oct 16, 2018
Closed
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
4490add 
This problem occurs because systems using systemd-resolved copy
127.0.0.53 from the host's /etc/resolv.conf.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
bf2d802 
This problem occurs because systems using systemd-resolved copy
127.0.0.53 from the host's /etc/resolv.conf.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
3b39821 
This problem occurs because kube-dns on systems using systemd-resolved
copy 127.0.0.53 from the host's /etc/resolv.conf.

Since 127.0.0.53 is a loopback address, dns queries never get past
kube-dns causing our conformance tests to fail on DNS related issues.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.
@vannrt vannrt mentioned this issue Dec 7, 2018
Open
vannrt added a commit to platform9/nodeadm that referenced this issue Dec 7, 2018
@vannrt
Fix for systemd-resolved DNS incompatibility
f48df82 
This problem occurs because kube-dns on systems using systemd-resolved
copy 127.0.0.53 from the host's /etc/resolv.conf.

Since 127.0.0.53 is a loopback address, dns queries never get past
kube-dns causing our conformance tests to fail on DNS related issues.

More discussion here: kubernetes/kubernetes#45828

Related issues:
kubernetes/kubeadm#787
kubernetes/kubeadm#273
kubernetes/kubeadm#845

The upstream fix is now in v1.11.

Without the fix, the kubedns and dnsmasq containers would copy the host's `/etc/resolv.conf`:
```
\# This file is managed by man:systemd-resolved(8). Do not edit.
\#
\# This is a dynamic resolv.conf file for connecting local clients to the
\# internal DNS stub resolver of systemd-resolved. This file lists all
\# configured search domains.
\#
\# Run "systemd-resolve --status" to see details about the uplink DNS servers
\# currently in use.
\#
\# Third party programs must not access this file directly, but only through the
\# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
\# replace this symlink by a static file or a different symlink.
\#
\# See man:systemd-resolved.service(8) for details about the supported modes of
\# operation for /etc/resolv.conf.

nameserver 127.0.0.53
search platform9.sys
```

After the fix:
```
\# This file is managed by man:systemd-resolved(8). Do not edit.
\#
\# This is a dynamic resolv.conf file for connecting local clients directly to
\# all known uplink DNS servers. This file lists all configured search domains.
\#
\# Third party programs must not access this file directly, but only through the
\# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way,
\# replace this symlink by a static file or a different symlink.
\#
\# See man:systemd-resolved.service(8) for details about the supported modes of
\# operation for /etc/resolv.conf.

nameserver 10.105.16.2
nameserver 10.105.16.4
search platform9.sys
```
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@stealthybox stealthybox

Labels
area/releasing area/upgrades lifecycle/active Indicates that an issue or PR is actively being worked on by a contributor.
Projects
None yet
Milestone
v1.11
Development

Successfully merging a pull request may close this issue.

kubeadm: Set the kubelet --resolv-conf flag conditionally on init stealthybox/kubernetes
3 participants
@randomvariable @stealthybox @luxas