[HELP] Failed Cloudflare DNS Validation and "Did Not Connect: Potential Security Issue" for Multiple Subdomains

[pr0927]

What happened?
Hi all, I run some self-hosted items using a number of servers. My domain's DNS is handled by Cloudflare, and I "gray cloud" most of the subdomains, only toggling on the proxy for some items (concerns about violating their ToS with too much handling of large files).

So I was enamored with what I'm seeing of Zoraxy, and decided to give it a whirl - my prior experience with reverse proxies was with SWAG on UnRAID, blindly following instructions from SpaceInvaderOne, then Nginx Proxy Manager, which worked but frequently broke, and later with Traefik on TrueNAS SCALE - though this was done by seeming magic by the TrueCharts crew, I have no idea how it works - and reading/watching tutorials for it gave me a headache.

I briefly tried to setup the "oh it's so easy" Caddy, and I must be dumb-as-rocks because this confused me, especially when it came to building a version to support DNS validation. Maybe it was too easy? I don't know - truthfully I didn't even try it live, I decided I wanted to first put my efforts into Zoraxy and see if it works for me.

I'm in the process of moving over to new Proxmox nodes from my existing TrueNAS SCALE setup with Traefik. I have my OPNsense handling port-forwarding to that reverse proxy.

I wanted to give it a live trial run, and ran into some issues. Namely, the DNS validation under "Open ACME Tool" on Zoraxy just gave me an infinite spinning loop after clicking "Get Certificate" and in the end shows a red "x" on the loaded certificate for my domain.

Also, for a handful of my subdomains pointed to Nextcloud, Jellyfin, Navidrome, and maybe 1-2 other items, I got this error:

Describe what have you tried
I have tried multiple times to get DNS validation working, to the point that Cloudflare rate limited me after too many attempts. I have also checked to make sure my domain is set with "Full (Strict)" SSL/TLS Encryption on Cloudflare's portal, and I've tried to mess around with options in the respective apps to make them work, thinking maybe I needed to list "https://drive.mydomain.tld" in the correct Nextcloud config file (I did) - but this still didn't fix it.

I also tried changing on-and-off requirements for HTTPS local addressing in Zoraxy for given problem apps, restarted Zoraxy, restarted problem apps, and I'm sure some other minor silly things I can't remember.

Describe the networking setup you are using
Here are some example, commonly asked questions from our maintainers:

• Are you using the docker build of Zoraxy? Yes, here is my Docker Compose for it:

services:
  zoraxy:
    image: zoraxydocker/zoraxy:latest
    container_name: zoraxy
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
      - 7000:8000
    volumes:
      - /data/zoraxy/config/:/opt/zoraxy/config/
      - /data/zoraxy/zerotier/:/var/lib/zerotier-one/
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime
    environment:
      FASTGEOIP: "true"
      TZ: "America/Los_Angeles"

• Your Zoraxy version? .3.1.0
• Are you using Cloudflare? Yes
• Are your system hosted under a NAT router? Yes, gateway is 192.168.0.1, port forwarding is set to send traffic from WAN on port 80 to Zoraxy's port 80, and traffic on WAN on port 443 to Zoraxy's port 443.
• DNS record (if any)? Unsure what this means to be honest.

Additional context
I setup Zoraxy in Docker on a Debian VM on Proxmox, then added a number of proxies (I used the "Pick from Docker Containers" option for apps running on that same VM - I have another Debian VM with Docker that has other apps, and I have two Home Assistant VMs - these are the items with the 192.168.0.x IPs that I had to add manually):

My main status page is setup like this:

My TLS/SSL Certificates page looks like this:

I went into "Open ACME Tool" and configured it as well (I initially tried "*.mydomain.tld" - thinking I needed it to be in wildcard format to cover all my subdomains, then it gave an error and failed to load, so I tried to list each subdomain, but with "Auto Detect Matching Rule" it defaulted to "mydomain.tld" in the end - so I assume only that root "mydomain.tld" is needed:

Looking at my logs in Portainer for Zoraxy, i see the following (somehow nothing prior to this is showing up, and I'm sure some of this is from when I got rate limited, and from when I switched my port-forward back to my Traefik reverse proxy on my TrueNAS SCALE server):

Would super-appreciate any assistance, as I'd really like to not start-from-scratch and switch to some other reverse proxy - this feels amazing otherwise!

[tobychui]

Hey @pr0927

You got a really detail log and description of your problem here, good job writing all the details!

All Zoraxy settings looks good to me (except, well, the blur out parts. But I assume you entered everything correctly), and from what I read from your description it seems to be a Cloudflare side setting issues.

When you are using Full (Strict) mode, you do not request TLS certificate using Zoraxy (i.e. your gateway node for handling all your homelab inbound requests). Instead, the certificate is handled by CloudFlare (Read more here). What you should install in Zoraxy is the Cloudflare certificates but not the standard TLS/SSL certificate offer by public CAs. This diagram explains how CF's full strict mode works.

Here is a quick walkthrough on the general idea on how you should setup your CF DNS record with Zoraxy

1. Enable Full (Strict) mode on CloudFlare
2. Download the 15 years certificate from CloudFlare dashboard
3. Go to Zoraxy > TLS / SSL > Fallback Certificate and upload the CloudFlare Signed Certifiacte
4. Enable TLS serving and set port to 443 (if this is not already been done on Zoraxy). You can skip the TLS 1.2 and HTTP to HTTPS redirection options on Zoraxy as those are handled by Cloudflare proxy.
5. Create a wildcard DNS A / CNAME record on Cloudflare DNS record (e.g. *.yourdomain.com)
6. Create a subdomain HTTP record in Zoraxy (e.g. nextcloud.yourdomain.com) and point it to your backend / local IP address

Then you should be able to visit your subdomain (e.g. nextcloud.yourdomain.com) or add more subdomains into Zoraxy for future projects.

[yeungalan]

checking in

[yeungalan]

https://community.cloudflare.com/t/lego-letsencrypt-cloudflare-unexpected-response-code-refused/582095/4

https://community.letsencrypt.org/t/error-renewing-certificate-from-le-ns-returned-refused-for-acme-challenge/174132/42

It sounds like the problem is due to the DNS server being unable to recognize your DNS record and go-acme giving out a bad nonce, going to investigate this as this maybe is a dependency problem

[pr0927]

Hey @pr0927

You got a really detail log and description of your problem here, good job writing all the details!

Thank you! I figured the least I could do was be thorough if I'm seeking help!

All Zoraxy settings looks good to me (except, well, the blur out parts. But I assume you entered everything correctly), and from what I read from your description it seems to be a Cloudflare side setting issues.

Haha yeah the blurred out stuff is just my "subdomain.mydomain.tld" or my e-mail address or Cloudflare API key, etc.

When you are using Full (Strict) mode, you do not request TLS certificate using Zoraxy (i.e. your gateway node for handling all your homelab inbound requests). Instead, the certificate is handled by CloudFlare (Read more here). What you should install in Zoraxy is the Cloudflare certificates but not the standard TLS/SSL certificate offer by public CAs. This diagram explains how CF's full strict mode works.

Here is a quick walkthrough on the general idea on how you should setup your CF DNS record with Zoraxy

1. Enable Full (Strict) mode on CloudFlare

2. Download the 15 years certificate from CloudFlare dashboard

3. Go to Zoraxy > TLS / SSL > Fallback Certificate and upload the CloudFlare Signed Certifiacte

4. Enable TLS serving and set port to 443 (if this is not already been done on Zoraxy). You can skip the TLS 1.2 and HTTP to HTTPS redirection options on Zoraxy as those are handled by Cloudflare proxy.

5. Create a wildcard DNS A / CNAME record on Cloudflare DNS record (e.g. *.yourdomain.com)

6. Create a subdomain HTTP record in Zoraxy (e.g. nextcloud.yourdomain.com) and point it to your backend / local IP address

Then you should be able to visit your subdomain (e.g. nextcloud.yourdomain.com) or add more subdomains into Zoraxy for future projects.

Oh interesting, never done that before, OK - presumably this "should" fix access for the other apps then too, like Jellyfin and Navidrome and whatnot? Do I toggle the TLS setting on per proxied subdomain too, or no?

https://community.cloudflare.com/t/lego-letsencrypt-cloudflare-unexpected-response-code-refused/582095/4

https://community.letsencrypt.org/t/error-renewing-certificate-from-le-ns-returned-refused-for-acme-challenge/174132/42

It sounds like the problem is due to the DNS server being unable to recognize your DNS record and go-acme giving out a bad nonce, going to investigate this as this maybe is a dependency problem

Ah, so this might be outside of my having done something incorrectly then?

If I do that other stuff mentioned by @tobychui, do I need to bother with DNS validation, or is that still a good thing to have?

[tobychui]

@pr0927 Alan is the maintainer of the ACME module, so it is kind of his "normal behavior" for wanting to dig more information from you and he can debug this issue later.

presumably this "should" fix access for the other apps then too, like Jellyfin and Navidrome and whatnot?

Yes and no. This should fix your inbound (downstream to Zoraxy) connection TLS issue, but for upstream (Zoraxy to Jellyfin), you can use the skip TLS validation option to let Zoraxy automatically bypass the validation check. There are a few tutorials on Youtube you can checkout regarding this option.

Ah, so this might be outside of my having done something incorrectly then?

More like your DNS setting / network environment. If you want to get it properly fixed, seek help from an IT professional.

If I do that other stuff mentioned by @tobychui, do I need to bother with DNS validation, or is that still a good thing to have?

The CF solution should be a work-around for Alan's proposed proper fix. If you use CF proxy solution, you do not need to deal with DNS validation yourself. But from Alan's point of view, this is an issue related to upstream dependencies and what he mean is, if you are still having the DNS problem, you should seek help from upstream dependencies maintainer instead.

[pr0927]

@pr0927 Alan is the maintainer of the ACME module, so it is kind of his "normal behavior" for wanting to dig more information from you and he can debug this issue later.

Ah, gotcha.

presumably this "should" fix access for the other apps then too, like Jellyfin and Navidrome and whatnot?

Yes and no. This should fix your inbound (downstream to Zoraxy) connection TLS issue, but for upstream (Zoraxy to Jellyfin), you can use the skip TLS validation option to let Zoraxy automatically bypass the validation check. There are a few tutorials on Youtube you can checkout regarding this option.

O.K., I'll do some digging/experimentation.

Ah, so this might be outside of my having done something incorrectly then?

More like your DNS setting / network environment. If you want to get it properly fixed, seek help from an IT professional.

Heh, I am an IT professional...this is for my homelab and learning more. xD

If I do that other stuff mentioned by @tobychui, do I need to bother with DNS validation, or is that still a good thing to have?

The CF solution should be a work-around for Alan's proposed proper fix. If you use CF proxy solution, you do not need to deal with DNS validation yourself. But from Alan's point of view, this is an issue related to upstream dependencies and what he mean is, if you are still having the DNS problem, you should seek help from upstream dependencies maintainer instead.

Ah I see. On that, note, I'm a bit lost. I went to this page, and I'm not sure what I'm uploading, or which button to click:

On the Cloudflare dashboard when creating the 15-year certificate, it gave me the "Origin Certificate" in "click-to-copy" text form, and the "Private Key" in the same form (with a drop-down to select .pem, .p7b, or .der formats).

And on the note you added about DNS record on Cloudflare, I already have my "mydomain.tld" as an A record pointing to my home IP (I have dynamic DNS setup on my OPNsense to update this) - do I need to add an additional wildcard CNAME with "*.mydomain.tld" as well? Or something else? I already have other CNAMEs for my subdomains that I'd been using in the past with existing reverse proxies.

[tobychui]

I am an IT professional...this is for my homelab and learning more. xD

Oh cool, seem you are like me whom never pay attention in uni's networking course. No worry, Zoraxy should be basic enough to get you started on learning some basic networking & proxy techniques.

I'm a bit lost. I went to this page, and I'm not sure what I'm uploading, or which button to click:

Upload default keypairs and select your private and public key respectively. (Copy those key generated from CF and save it into a file named .pem and .key if you havn't already do so). The upload dialog will pop up after you click the "Public Key" and "Private Key" buttons.

do I need to add an additional wildcard CNAME with "*.mydomain.tld" as well?

Yes

I already have other CNAMEs for my subdomains that I'd been using in the past with existing reverse proxies.

You can keep or remove it depending on your preferences. Generally speaking if you added * you don't need those anymore, subdomain handling then can be setup in Zoraxy instead.

[pr0927]

I am an IT professional...this is for my homelab and learning more. xD

Oh cool, seem you are like me whom never pay attention in uni's networking course. No worry, Zoraxy should be basic enough to get you started on learning some basic networking & proxy techniques.

Hahaha, I left being a lawyer when I was bored with it and have only been in IT for ~2 years - was afraid of networking till more recently, but still somehow always confused about anything related to websites and certificates. xD

On that note, I have my OPNsense set to DNS-over-TLS through Quad9 in the built-in Unbound DNS as such:

Do I need to be doing anything in my Firewall rules or NAT for Cloudflare's DNS validation to work by any chance? Or entirely unrelated?

I'm a bit lost. I went to this page, and I'm not sure what I'm uploading, or which button to click:

Upload default keypairs and select your private and public key respectively. (Copy those key generated from CF and save it into a file named .pem and .key if you havn't already do so). The upload dialog will pop up after you click the "Public Key" and "Private Key" buttons.

Oh, so that "Origin Certificate" goes into the public key? And then of course the "Private Key" into private key.

do I need to add an additional wildcard CNAME with "*.mydomain.tld" as well?

Yes

Sounds good!

I already have other CNAMEs for my subdomains that I'd been using in the past with existing reverse proxies.

You can keep or remove it depending on your preferences. Generally speaking if you added * you don't need those anymore, subdomain handling then can be setup in Zoraxy instead.

Oh wow I had no idea it could be done like this. Sounds good!

[tobychui]

@pr0927

Do I need to be doing anything in my Firewall rules or NAT for Cloudflare's DNS validation to work by any chance? Or entirely unrelated?

That is unrelated if you go with the CF full strict mode.

Oh, so that "Origin Certificate" goes into the public key? And then of course the "Private Key" into private key.

Yes

I guess till here you can figure out the rest yourself. Good luck with your new homelab proxy 👍🏻

[pr0927]

@pr0927

Do I need to be doing anything in my Firewall rules or NAT for Cloudflare's DNS validation to work by any chance? Or entirely unrelated?

That is unrelated if you go with the CF full strict mode.

Oh, O.K. But what I was doing initially, if I were to not do this "Full Strict Mode" and return to the DNS validation approach - then it's possibly this DNS setup is at fault?

Oh, so that "Origin Certificate" goes into the public key? And then of course the "Private Key" into private key.

Yes

I guess till here you can figure out the rest yourself. Good luck with your new homelab proxy 👍🏻

I hope so! Just to be clear - you mean the first set of "Public Key" and "Private Key" toward the top of the page, not the "Fallback Certificate" towards the bottom?

And apologies, but one last question - when you say that I can skip TLS validation and where I am using Cloudflare as a proxy - does this only regard this "Full Strict Mode" with the Origin Certificate? Or does this mean for subdomains for which I have the orange "cloud" toggled on the DNS page for proxying? I ask because for a few subdomains I do not have that toggled on the DNS page (deliberately).

Thanks again, this has been really instructive and I appreciate the time you're taking to help!

[tobychui]

you mean the first set of "Public Key" and "Private Key" toward the top of the page, not the "Fallback Certificate" towards the bottom?

I do mean "Fallback Certificate" (aka the Default Certificate when no cert is found). Cloudflare expect your web server to communicate with their proxy server with your "origin certificate" encoded data. So in that sense, uploading your origin server certificate to the "Fallback Certificate" is the correct setup here.

(Press those two buttons for a pop-up upload file selector)

when you say that I can skip TLS validation and where I am using Cloudflare as a proxy - does this only regard this "Full Strict Mode" with the Origin Certificate? Or does this mean for subdomains for which I have the orange "cloud" toggled on the DNS page for proxying?

Skip TLS validation is for upstream connections. It doesn't have direct relation with the Full strict Mode. Regarding if you should proxy via CF or not, unless you are having issue with Cloudflare POST size limits (e.g. nextcloud web ui file upload), you do not need to set it to off. For those subdomains where you cannot enable the CF proxy, you will need to request a subdomain certificate for them (i.e. requesting with a domain that contain no *, like nextcloud.yourdomain.com ).

In shorts, for subdomains that is not proxied by CF, you will need to handle the certificate in Zoraxy (using ACME tool, and due to your network environment issues, not using the DNS challenge option). For other subdomains, you can just leave the certificate handling to CF iff such subdomains is covered by your wildcard certificate.