Kubeadm DNS setup isn't compatible with stock ubuntu 18.04 #787
Assignees
Labels
priority/important-soon
Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Comments
|
Happens to me on Ubuntu 18.04 as well. I tried with both coredns and kube-dns and they both fail. Are there any known workarounds either in Kubernetes or the OS? |
|
Answering my own question. Workaround is to use the legacy format resolv.conf that systemd-resolved provides: sudo rm /etc/resolv.conf
sudo ln -s /run/systemd/resolve/resolv.conf /etc/resolv.conf |
|
/assign @detiber - This overlaps with the other issue that is on your plate. |
timothysc
added
the
priority/important-soon
k8s-github-robot
pushed a commit
to kubernetes/kubernetes
that referenced
this issue
May 11, 2018
Automatic merge from submit-queue (batch tested with PRs 63673, 63712, 63691, 63684). If you want to cherry-pick this change to another branch, please follow the instructions <a href="https:/kubernetes/community/blob/master/contributors/devel/cherry-picks.md">here</a>. kubeadm - add preflight warning when using systemd-resolved **What this PR does / why we need it**: This PR adds a preflight warning when the host is running systemd-resolved. Newer Ubuntu releases (artful and bionic in particular) run systemd-resolved by default and in the dfeault configuration have an /etc/resolv.conf file that references 127.0.0.53 which is not accessible from containers running on the host. We will now provide a warning to the user to tell them that the kubelet args should include `--resolv-conf=/run/systemd/resolve/resolv.conf`. `/run/systemd/resolve/resolv.conf`. **Which issue(s) this PR fixes**: This does not resolve the following issues, but it does provide better output to the users affected by the issues: kubernetes/kubeadm#273 kubernetes/kubeadm#787 **Release note**: ```release-note NONE ```
|
Closing as a duplicate of #273. |
This was referenced Jun 24, 2018
kinghajj
pushed a commit
to AdvMicrogrid/amazon-eks-ami
that referenced
this issue
Oct 22, 2018
According to [an issue][0] with kubeadm, Ubuntu 18.04 changed its DNS configuration in a way that broke kube-dns. One user suggested removing `/etc/resolv.conf` and symlinking it to systemd's as a solution. This worked when prepended to the user-data, so it should also work as part of the bootstrap script. [0]: kubernetes/kubeadm#787
vannrt
added a commit
to platform9/nodeadm
that referenced
this issue
Dec 7, 2018
This problem occurs because systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. More discussion here: kubernetes/kubernetes#45828 Related issues: kubernetes/kubeadm#787 kubernetes/kubeadm#273 kubernetes/kubeadm#845 The upstream fix is now in v1.11.
vannrt
added a commit
to platform9/nodeadm
that referenced
this issue
Dec 7, 2018
This problem occurs because systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. More discussion here: kubernetes/kubernetes#45828 Related issues: kubernetes/kubeadm#787 kubernetes/kubeadm#273 kubernetes/kubeadm#845 The upstream fix is now in v1.11.
vannrt
added a commit
to platform9/nodeadm
that referenced
this issue
Dec 7, 2018
This problem occurs because kube-dns on systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. Since 127.0.0.53 is a loopback address, dns queries never get past kube-dns causing our conformance tests to fail on DNS related issues. More discussion here: kubernetes/kubernetes#45828 Related issues: kubernetes/kubeadm#787 kubernetes/kubeadm#273 kubernetes/kubeadm#845 The upstream fix is now in v1.11.
vannrt
added a commit
to platform9/nodeadm
that referenced
this issue
Dec 7, 2018
This problem occurs because kube-dns on systems using systemd-resolved copy 127.0.0.53 from the host's /etc/resolv.conf. Since 127.0.0.53 is a loopback address, dns queries never get past kube-dns causing our conformance tests to fail on DNS related issues. More discussion here: kubernetes/kubernetes#45828 Related issues: kubernetes/kubeadm#787 kubernetes/kubeadm#273 kubernetes/kubeadm#845 The upstream fix is now in v1.11. Without the fix, the kubedns and dnsmasq containers would copy the host's `/etc/resolv.conf`: ``` \# This file is managed by man:systemd-resolved(8). Do not edit. \# \# This is a dynamic resolv.conf file for connecting local clients to the \# internal DNS stub resolver of systemd-resolved. This file lists all \# configured search domains. \# \# Run "systemd-resolve --status" to see details about the uplink DNS servers \# currently in use. \# \# Third party programs must not access this file directly, but only through the \# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way, \# replace this symlink by a static file or a different symlink. \# \# See man:systemd-resolved.service(8) for details about the supported modes of \# operation for /etc/resolv.conf. nameserver 127.0.0.53 search platform9.sys ``` After the fix: ``` \# This file is managed by man:systemd-resolved(8). Do not edit. \# \# This is a dynamic resolv.conf file for connecting local clients directly to \# all known uplink DNS servers. This file lists all configured search domains. \# \# Third party programs must not access this file directly, but only through the \# symlink at /etc/resolv.conf. To manage man:resolv.conf(5) in a different way, \# replace this symlink by a static file or a different symlink. \# \# See man:systemd-resolved.service(8) for details about the supported modes of \# operation for /etc/resolv.conf. nameserver 10.105.16.2 nameserver 10.105.16.4 search platform9.sys ```
|
@detiber I am still facing this issue with minikube 1.9.2, k8s v1.15.12 and ubuntu 18.04.4. Tried the mentioned work around but it didn't work. Could you please suggest something? I am trying to access a service exposed as NodePort. Things used to work perfectly until my system crashed and I had to re-install Ubuntu and minikube. Facing this issue since 2 days. Please go through this issue for more details. I would really appreciate any help. Thanks |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
What keywords did you search in kubeadm issues before filing this one?
dns
Is this a BUG REPORT or FEATURE REQUEST?
BUG REPORT
Versions
kubeadm version 1.10.2
Environment:
What happened?
Following the kubeadm guide creates a cluster that doesn't have a functional DNS.
What you expected to happen?
Following the kubeadm guide creates a cluster that works.
How to reproduce it (as minimally and precisely as possible)?
Follow the kubeadm setup doc.
Anything else we need to know?
It seems that 18.04 had switched to systemd-resolved by default, which generates a managed
/etc/resolv.confon the host:it's not a problem for docker at it will override the DNS with google dns, but kubeadm spins up coredns pods (or kubedns) with dns policy set to "default", meaning that the dns pod will have itself as a resolver. For coredns that means it will crashloop on any external name resolution as it's OOMing on the query recursion.
I'd think the best way to move forward with this is to allow kubeadm to manage
dnsConfigof the resolver pod, but given it's an alpha feature, kubeadm should at least do a sanity check and not allow a cluster to be created if the host resolv.conf is pointing at localhost.The text was updated successfully, but these errors were encountered: