[<prev] [next>] [<thread-prev] [day] [month] [year] [list]
Message-ID: <20190913151109.GF9017@brightrain.aerifal.cx>
Date: Fri, 13 Sep 2019 11:11:09 -0400
From: Rich Felker <dalias@...c.org>
To: Andrey Arapov <andrey.arapov@...aid.com>
Cc: musl@...ts.openwall.com
Subject: Re: DNS FQDN query issue in musl starting 1.1.13
On Fri, Sep 13, 2019 at 02:19:48PM +0000, Andrey Arapov wrote:
> Hello Rich,
>
> thank you for your prompt reply.
>
> I agree that SERVFAIL must be reported as an error to the caller and have just realized
> that "ucp-controller.kube-system.svc.cluster.local" has only 4 ndots, hence it isn't
> tried unless a trailing (5th) dot was specified,
> e.g. "ucp-controller.kube-system.svc.cluster.local.".
>
> Probably one of the differences is that, I presume, glibc treats a domain name terminated
> by a length byte of zero (RFC1035 3.1 Name space definitions), hence resolving the FQDN
> with only 4 dots whilst 5 is set.
> Please correct me if I am wrong.
You're mistaken here. The input to the resolver interfaces is a plain
string; there is no "length byte". Length byte is part of the DNS
protocol on the wire and is only involved at a lower layer past the
search processing, which is just transformations on strings.
On both glibc and musl, the search domains are tried initially if the
query string does not end in a dot and does not contain at least ndots
dots. So in your above example with four dots and ndots==5, both will
do the search. Adding the final dot suppresses the search regardless
of whether it takes the dots count to 5 or more.
The only difference between glibc and musl here is that, as I
understand it at least, glibc will continue searching after hitting an
error like ServFail, thus producing results that depend on transient
failures. musl stops and reports the error.
In the opposite case, with something like a.b.c.d.example.com, where
there are 5 dots, the behaviors differ in the way you're wondering
about, but I don't think it's relevant to your usage/problem. glibc
will first try to resolve it as a FQDN, and if that fails, it will run
through the whole search. musl does not perform search at all for
queries with at least ndots dots in them. The motivation here is the
same: if you depend on that behavior, your application/configuration
is subject to breakage by third parties registering new things in the
global DNS namespace. This is actually what happened with all the new
TLDs -- lots of networks using those names as fake local TLDs via
search with ndots==1 broke when they appeared as real TLDs.
> Regarding usage constraints, it looks like that the whole point having the ndots > 1 is
> basically to make the internal cluster lookups faster (the more dots the faster) while
> cache the external DNS lookups so they are slow the first time but fast subsequently.
If all your internal cluster lookups use the .local fake TLD
explicitly, e.g. *.svc.cluster.local, etc., then search domain is not
needed whatsoever and just making things slow and fragile. ndots==1
would avoid it getting used, though.
If your internal cluster lookups are looking up names like "foo.bar"
and expecting it to resolve to foo.bar.kube-system.svc.cluster.local,
foo.bar.svc.cluster.local, or foo.bar.cluster.local depending on which
first defines it, then your setup actually does depend on search and
having ndots be greater than the number of dots in the longest
"foo.bar" part you use. The number of dots in the search part
("foo.bar.svc.cluster.local") is not relevant.
One proposal I recall hearing from Kubernetes folks way back was to
use names like "foo-bar" instead of "foo.bar" in the above, so that
ndots==1 suffices and the search behavior does not clash with lookups
of real domains.
> But having ndots = 1 to workaround the musl's unexpected behavior (when ndots > 1) is
> making all intra-cluster lookups slower, whilst upstream FQDN faster.
I don't understand why that would be. They should either fail entirely
or be just as fast. Decreasing ndots should not be able to slow down
any lookup that works both before and after the decreasing.
> After reading through the discussions it turns out that in the beginning people resorted
> to using the Kubernetes's dnsConfig for setting the ndots to 1 (default) as a workaround
> but later then they did not need that anymore as Kubernetes/CoreDNS dropped Alpine
> (not sure for what reasons though).
>
> I guess the whole point is that the projects using musl C library (>=1.1.13)
> should clearly make people aware of that difference in hostname lookups
> which cause unexpected behavior compared to the glibc.
We've tried to do this on the wiki about functional differences. Open
to improvements.
> Below is some brief story-line I gathered which might be handy to anyone reading this.
>
> ### September-December 2016/2017
>
> > - "We're smart enough to solve this for everyone" is not realistic. (c) BrianGallew
>
> The rationale for having ndots=5 is explained at length at #33554
> - https://github.com/kubernetes/kubernetes/issues/33554#issuecomment-266251056
>
> dnsConfig:
> - https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/#pod-s-dns-config
> - https://github.com/kubernetes/website/pull/5978
>
> ### June-August 2018 ... April 2019
>
> People are struggling with this issue as upstream/downstream projects
> updated (or switched to) their Alpine base distro to 3.4 (or higher with musl >= 1.1.13).
>
> ndots breaks DNS resolving
> - https://github.com/kubernetes/kubernetes/issues/64924
>
> Kubernetes pods /etc/resolv.conf ndots:5 option and why it may negatively affect your application performances
> - https://pracucci.com/kubernetes-dns-resolution-ndots-options-and-why-it-may-affect-application-performances.html
>
> Docker: drop alpine
> - https://github.com/coredns/coredns/pull/1843
>
> Rebase container images from alpine to debian-base.
> - https://github.com/kubernetes/dns/pull/294
I'm not up for reading through all that right now, but thanks for
collecting the relevant information. It's frustrating that, despite
having known way back that what they were doing was wrong, they seem
to have decided to "drop support for Alpine/musl" rather than "fix the
stuff that's obviously wrong and hurting performance even on glibc
based dists"...
Rich
Powered by blists - more mailing lists
Confused about mailing lists and their use?
Read about mailing lists on Wikipedia
and check out these
guidelines on proper formatting of your messages.
