We have discussed ngix/gateway deployment on aks in the below posts.
Then we fixed an issue with cert renewal at deployment as explained in the below post.
The nginx gateway setup explained in above posts has two parts that talk to each other over a secure connection:
- The control part (Fabric) sends configuration.
- The traffic part (nginx pods) receives it and routes user traffic.
To trust each other, both use certificates. We use cert-manager to create and renew them. There are three certificates:
- A CA certificate. This is the "boss" certificate. It signs the other two.
- A server certificate for the control part.
- A client certificate for the nginx pods.
The server and client certificates must be signed by the same CA. If they are not, the two parts stop trusting each other. The problem explained in the Handle Nginx Gateway Certificate Refresh While In Place Upgrade in AKS is partly soved in that post but, if a deployment does not happen for a long time problem occurs again. We used the cert-manager defaults for all three certificates:
- Each certificate lives 90 days.
- Each one renews at day 60.
All three were created at the same moment as we explained in Handle Nginx Gateway Certificate Refresh While In Place Upgrade in AKS . So all three also renewed at the same moment, at day 60.
The problem
When the CA renews, it gets a new key. The two other certificates must be signed by this new CA. But all three renewed at the same second, so it was a race. Either of these could happen:
- One certificate was signed by the old CA and the other by the new CA. The two parts no longer trusted each other. New control plane and data plane connections failed from day 60.
- Both were signed by the old CA. This looks fine at first. But the old CA dies at day 90, while the new certificates live until day 150. New control plane and data plane connections failed from day 90.
Running pods kept serving users, because the certificate is only checked when a connection starts. The failure showed up only when a new pod started or a connection had to be created again.
Why did we not see it earlier? Our deploy pipeline deletes and recreates the certificates on every deploy. So most of the time everything was fresh. The problem only appeared when no deploy happened for 60 days or more.
We had below in the file
pipelines\aks_manifests\nginx_gateway\nginx_gateway_fabric_prerequisites.yaml as explained in the post
Deploy Nginx-Gateway-Fabric. So, we used the cert-manager defaults for all three certificates:
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned-issuer
namespace: nginx-gateway
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx-gateway-ca
namespace: nginx-gateway
spec:
isCA: true
commonName: nginx-gateway
secretName: nginx-gateway-ca
privateKey:
algorithm: RSA
size: 2048
issuerRef:
name: selfsigned-issuer
kind: Issuer
group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: nginx-gateway-issuer
namespace: nginx-gateway
spec:
ca:
secretName: nginx-gateway-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx-gateway
namespace: nginx-gateway
spec:
secretName: server-tls
usages:
- digital signature
- key encipherment
dnsNames:
- ngf-nginx-gateway-fabric.nginx-gateway.svc
issuerRef:
name: nginx-gateway-issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx
namespace: nginx-gateway
spec:
secretName: agent-tls
usages:
- "digital signature"
- "key encipherment"
dnsNames:
- "*.cluster.local"
issuerRef:
name: nginx-gateway-issuer
The fix
We stopped letting everything renew together. We set our own timings:
Three simple rules sit behind these numbers:
- A certificate must never live longer than the CA that signed it. The CA renews 90 days before it ends. That is more than the 60 days a certificate lives. So the CA always outlives the certificates it signed.
- The CA and the other certificates should not renew at the same time. The CA renews every 365 days. The others renew every 30 days. 365 is not a multiple of 30, so they do not line up. The first time they would meet again is after about 6 years.
- Both certificates always get the same CA. The CA changes rarely, so when the two certificates renew, they are signed by the same one.
We also kept the deploy pipeline step that recreates the certificates. It does not affect live traffic, and every deploy restarts all the timers. In practice, the 6-year meeting point is never reached.
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: selfsigned-issuer
namespace: nginx-gateway
spec:
selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx-gateway-ca
namespace: nginx-gateway
spec:
isCA: true
commonName: nginx-gateway
secretName: nginx-gateway-ca
# Renews every 365 days; renewBefore (90d) must exceed leaf duration (60d)
# and keep CA renewal off the 30-day leaf renewal days.
duration: 10920h # 455 days
renewBefore: 2160h # 90 days
privateKey:
algorithm: RSA
size: 2048
rotationPolicy: Always
issuerRef:
name: selfsigned-issuer
kind: Issuer
group: cert-manager.io
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: nginx-gateway-issuer
namespace: nginx-gateway
spec:
ca:
secretName: nginx-gateway-ca
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx-gateway
namespace: nginx-gateway
spec:
secretName: server-tls
duration: 1440h # 60 days
renewBefore: 720h # 30 days, renews every 30 days
usages:
- digital signature
- key encipherment
dnsNames:
- ngf-nginx-gateway-fabric.nginx-gateway.svc
issuerRef:
name: nginx-gateway-issuer
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
name: nginx
namespace: nginx-gateway
spec:
secretName: agent-tls
duration: 1440h # 60 days
renewBefore: 720h # 30 days, renews every 30 days
usages:
- "digital signature"
- "key encipherment"
dnsNames:
- "*.cluster.local"
issuerRef:
name: nginx-gateway-issuer
How we tested it
We tested on a test cluster with very short times, so we could see months of life in minutes: Below timeline is set.
- Certificates renewed every 5 minutes, and the CA every 8 minutes.
- We watched 3 CA renewals and 5 certificate renewals.
- Both certificates always had the same CA.
- A certificate never lived longer than its CA.
- No pod restarted.
Each certificate renewal caused about 2 minutes of connection resets between the control part and the nginx pods, then everything recovered. This connection only carries configuration, not user traffic, so routing was not interrupted.
The expectation was that, with the above settings, 40 minutes would proportionally represent six years of collision expectations under production conditions.
To monitor we used below script on wsl connected to the AKS where nginx is deployed.
CTX=ch-demo-dev-euw-002-aks-green-admin
NS=nginx-gateway
cert_info() { # $1 = secret, $2 = label
local s=$1 l=$2
kubectl --context $CTX -n $NS get secret $s -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/leaf.crt 2>/dev/null
kubectl --context $CTX -n $NS get secret $s -o jsonpath='{.data.ca\.crt}' | base64 -d > /tmp/ca.crt 2>/dev/null
printf "%-10s leaf serial=%s notAfter=%s | ca.crt serial=%s key=%s notAfter=%s\n" "$l" \
"$(openssl x509 -in /tmp/leaf.crt -noout -serial | cut -d= -f2 | cut -c1-8)" \
"$(openssl x509 -in /tmp/leaf.crt -noout -enddate | cut -d= -f2)" \
"$(openssl x509 -in /tmp/ca.crt -noout -serial | cut -d= -f2 | cut -c1-8)" \
"$(openssl x509 -in /tmp/ca.crt -noout -pubkey | sha256sum | cut -c1-8)" \
"$(openssl x509 -in /tmp/ca.crt -noout -enddate | cut -d= -f2)"
}
while true; do
echo "===== $(date -u +%H:%M:%S) UTC ====="
kubectl --context $CTX -n $NS get secret nginx-gateway-ca -o jsonpath='{.data.tls\.crt}' | base64 -d > /tmp/cacur.crt
printf "%-10s serial=%s key=%s notAfter=%s\n" "CA" \
"$(openssl x509 -in /tmp/cacur.crt -noout -serial | cut -d= -f2 | cut -c1-8)" \
"$(openssl x509 -in /tmp/cacur.crt -noout -pubkey | sha256sum | cut -c1-8)" \
"$(openssl x509 -in /tmp/cacur.crt -noout -enddate | cut -d= -f2)"
cert_info server-tls "server-tls"
cert_info agent-tls "agent-tls"
cert_info pocrmq-apps-gateway-nginx-agent-tls "agent-copy"
echo "fabric TLS errors (2m): $(kubectl --context $CTX -n $NS logs -l app.kubernetes.io/name=nginx-gateway-fabric --since=2m --tail=-1 --prefix 2>/dev/null | grep -ciE 'tls|certificate|handshake')"
echo "nginx agent resets (2m): $(kubectl --context $CTX -n $NS logs -l gateway.networking.k8s.io/gateway-name=pocrmq-apps-gateway -c nginx --since=2m --tail=-1 2>/dev/null | grep -ciE 'credential|reset')"
echo "restarts: $(kubectl --context $CTX -n $NS get pods --no-headers | awk '{s+=$4} END{print s}')"
sleep 10
done 2>&1 | tee ~/ca-leaf-rotation.log
Below is some of monitoring resutls.
What we learned
- Do not renew everything at the same time. Spread the timings out.
- A certificate must not outlive the CA that signed it.
- A problem that only shows up after 60 days is easy to miss, because deploys hide it.
- Test with short times. You can see the whole renewal process in minutes.
No comments:
Post a Comment