Kubernetes - exposing cluster to external access with cloudflare proxy

Question

I currently have the following setup:

• Cloudflare https proxied dns which routes ALL traffic to my server (end-to-end [client->proxy and proxy->server] encryption with CloudFlare Origin CA)
• Nginx reverse-proxy that acts as an SSL termination for all traffic to my server (origin signed cert I believe)
• Nginx proxies anything that goes to kube.my-domain.com to my kubeapi server as follows: proxy_pass https://127.0.0.1:6443
• kubeapi running on my server

Currently I can connect to my cluster just fine by specifying in my kubeconfig:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: REDACTED
    server: https://127.0.0.1:6443  <-----------------------------
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    namespace: test-namespace
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED

But if I change the server in kubeconfig to point to my domain: server: https://kube.my-domain.com I get the following error:

user@master:~$ kubectl get deployments
Unable to connect to the server: x509: certificate signed by unknown authority

Now I assume this is because my traffic has to go through cloudflare's proxy and has different certs along the way. So I try appending --insecure-skip-tls-verify=true and get teh following response:

user@master:~$ kubectl get deployments --insecure-skip-tls-verify=true
error: the server doesn't have a resource type "deployments"

So I look this up and find out a lot of the time it's due to RBAC (even though I should be authenticating as kubernetes-admin, which worked when I connected locally...)

I0317 00:41:09.281404   31989 loader.go:375] Config loaded from file:  /home/user/.kube/config
I0317 00:41:09.281813   31989 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://kube.my-domain.com/api?timeout=32s'
I0317 00:41:09.493885   31989 round_trippers.go:443] GET https://kube.my-domain.com/api?timeout=32s 403 Forbidden in 212 milliseconds
I0317 00:41:09.493921   31989 round_trippers.go:449] Response Headers:
I0317 00:41:09.493970   31989 round_trippers.go:452]     Set-Cookie: REDACTED; expires=Thu, 16-Apr-20 04:41:09 GMT; path=/; domain=.my-domain.com; HttpOnly; SameSite=Lax; Secure
I0317 00:41:09.493991   31989 round_trippers.go:452]     Expect-Ct: max-age=604800, report-uri="REDACTED"
I0317 00:41:09.494002   31989 round_trippers.go:452]     Server: cloudflare
I0317 00:41:09.494027   31989 round_trippers.go:452]     Cf-Ray: REDACTED
I0317 00:41:09.494065   31989 round_trippers.go:452]     Date: Tue, 17 Mar 2020 04:41:09 GMT
I0317 00:41:09.494070   31989 round_trippers.go:452]     X-Content-Type-Options: nosniff
I0317 00:41:09.494091   31989 round_trippers.go:452]     Cf-Cache-Status: DYNAMIC
I0317 00:41:09.494112   31989 round_trippers.go:452]     Content-Type: application/json
I0317 00:41:09.498893   31989 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"forbidden: User \"system:anonymous\" cannot get path \"/api\"","reason":"Forbidden","details":{},"code":403}
I0317 00:41:09.503266   31989 round_trippers.go:423] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://kube.my-domain.com/apis?timeout=32s'
I0317 00:41:09.640948   31989 round_trippers.go:443] GET https://kube.my-domain.com/apis?timeout=32s 403 Forbidden in 137 milliseconds
I0317 00:41:09.640992   31989 round_trippers.go:449] Response Headers:
I0317 00:41:09.641000   31989 round_trippers.go:452]     Date: Tue, 17 Mar 2020 04:41:09 GMT
I0317 00:41:09.641006   31989 round_trippers.go:452]     Content-Type: application/json
I0317 00:41:09.641011   31989 round_trippers.go:452]     Cf-Cache-Status: DYNAMIC
I0317 00:41:09.641016   31989 round_trippers.go:452]     Expect-Ct: max-age=604800, report-uri="REDACTED"
I0317 00:41:09.641021   31989 round_trippers.go:452]     Set-Cookie: REDACTED; expires=Thu, 16-Apr-20 04:41:09 GMT; path=/; domain=.my-domain.com; HttpOnly; SameSite=Lax; Secure
I0317 00:41:09.641027   31989 round_trippers.go:452]     X-Content-Type-Options: nosniff
I0317 00:41:09.641031   31989 round_trippers.go:452]     Server: cloudflare
I0317 00:41:09.641056   31989 round_trippers.go:452]     Cf-Ray: REDACTED

What am I doing wrong? Also, what is the best way to expose my cluster and keep it secure?

I feel like it has to do with the SSL termination at NGINX removing what's necessary for the api-server to recognize me.

Thanks.

UPDATE: It seems like either cloudflare proxy pr NGINX is stripping away the authentication certificate and key and is therefore causing the user of the request to become anonymous. Still looking into this.

Answer 1

Невозможно подключиться к серверу: x509: сертификат, подписанный неизвестным органом

Вышеупомянутая ошибка указывает на то, что сертификат сервера, представленный конечной точкой https://kube.my-domain.com/, не подписан центром сертификации, который у вас есть в файле kubeconfig.certificate-authority-dataраздел. В kubeconfig вам необходимо настроить правильный центр сертификации (CA).