AKS and Ingress : Constraining Access

Published: Apr 3, 2020 by Isaac Johnson

In my last missive I detailed out AKS and Ingress using NGinx Ingress Controllers. For public facing services this can be sufficient. However, many businesses require tighter controls - either for canary style deployments, hybrid cloud mixed systems or geofencing using CIDR blocks.

While we have a public IP that can be reached globally, here are two methods we can explore to restrict traffic.

Network Security Groups

One way is to restrict traffic at the Network Security Group (NSG) level. This maintains the perimeter around our VNet that holds our cluster’s VMs. We can see rules have been added for us that allows all of the internet to access our NGinx.

Traffic checked at blocked at the Network Perimeter

However, I can reach that from anywhere.

Sending my traffic to Sweden, i can still hit it

Let’s head to the Inbound rules and first add a rule.

We are going to put it at a range above 500 but below the 65000 rules, then remove the old ones created by Nginx…

Which means, this:

becomes:

While i set source port to "*" i did limit target ports

One thing I discovered in testing is that my external facing IP address was not enough. I used traceroute to find the hops from Comcast and add CIDR rules for all of them.

This now means my network range can see the site, but others would get an unreachable page:

From my AKS cluster, i now see :

Restricting via NGinx

Perhaps you want to set controls on the NGinx ingress itself instead of the NSG. In other words, you want to allow traffic to hit the ingress, but reject anything but our allowable ranges. This can be controlled by annotations on the ingress deployment, this is something that can be parameterized and updated via pipelines allowing our CICD systems to control it. This could be particularly useful for slow roleouts (canary deployments) or allowing limited private access. The disadvantage is that your NGinx can get hammered from anywhere so you are more exposed from a security standpoint.

Traffic Checked by NGinx Ingress Controller and rejected with HTTP/S error

Let’s install our Helloworld example:

$ helm version
version.BuildInfo{Version:"v3.0.2", GitCommit:"19e47ee3283ae98139d98460de796c1be1e3975f", GitTreeState:"clean", GoVersion:"go1.13.5"}
$ helm repo add azure-samples https://azure-samples.github.io/helm-charts/
"azure-samples" has been added to your repositories
$ helm repo update
Hang tight while we grab the latest from your chart repositories...
...Successfully got an update from the "azure-samples" chart repository
...Successfully got an update from the "nginx-stable" chart repository
...Successfully got an update from the "banzaicloud-stable" chart repository
...Successfully got an update from the "bitnami" chart repository
Update Complete. ⎈ Happy Helming!⎈ 
$ helm install aks-helloworld azure-samples/aks-helloworld --namespace shareddev
NAME: aks-helloworld
LAST DEPLOYED: Thu Apr 2 10:49:46 2020
NAMESPACE: shareddev
STATUS: deployed
REVISION: 1
TEST SUITE: None

We can list our services there and add an ingress:

$ kubectl get svc -n shareddev
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
aks-helloworld ClusterIP 10.0.200.75 <none> 80/TCP 21s
ingress-nginx LoadBalancer 10.0.19.138 xxx.xxx.xxx.xxx 80:30466/TCP,443:30090/TCP 24h
$ vi helloworld-ingress.yaml
$ kubectl apply -f helloworld-ingress.yaml 
error: error validating "helloworld-ingress.yaml": error validating data: ValidationError(Ingress.spec.rules[0].http.paths[0]): unknown field "pathType" in io.k8s.api.networking.v1beta1.HTTPIngressPath; if you choose to ignore these errors, turn validation off with --validate=false
$ kubectl apply -f helloworld-ingress.yaml --validate=false
ingress.networking.k8s.io/aks-helloworld created

We can now restrict at the controller level.

Let’s look at the Nginx configmap:

$ kubectl get cm nginx-configuration -n shareddev
NAME DATA AGE
nginx-configuration 0 24h
$ kubectl get cm nginx-configuration -n shareddev -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"nginx-configuration","namespace":"shareddev"}}
  creationTimestamp: "2020-04-01T15:29:50Z"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-configuration
  namespace: shareddev
  resourceVersion: "177523"
  selfLink: /api/v1/namespaces/shareddev/configmaps/nginx-configuration
  uid: a0c5ef9f-742d-11ea-a4a0-bac2a8381a43

Here we have no settings.. Let’s add a data block to lock down access.

$ kubectl edit cm nginx-configuration -n shareddev
configmap/nginx-configuration edited
$ kubectl get cm nginx-configuration -n shareddev -o yaml
apiVersion: v1
data:
  whitelist-source-range: 10.2.2.2/24
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"nginx-configuration","namespace":"shareddev"}}
  creationTimestamp: "2020-04-01T15:29:50Z"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-configuration
  namespace: shareddev
  resourceVersion: "177827"
  selfLink: /api/v1/namespaces/shareddev/configmaps/nginx-configuration
  uid: a0c5ef9f-742d-11ea-a4a0-bac2a8381a43

Note that i added a data block with a whitelist-source-range. I used 10.2.2.2 knowing it won’t match any of my systems.

We can now see it’s immediately blocked.

Nginx validates our IP and rejects with a 403 error page

But what if we want to enable access to some by not all?

Let’s restore our configmap ( kubectl edit configmap nginx-configuration -n shareddev) and remove the datablock:

$ kubectl get cm nginx-configuration -n shareddev -o yaml
apiVersion: v1
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app.kubernetes.io/name":"ingress-nginx","app.kubernetes.io/part-of":"ingress-nginx"},"name":"nginx-configuration","namespace":"shareddev"}}
  creationTimestamp: "2020-04-01T15:29:50Z"
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  name: nginx-configuration
  namespace: shareddev
  resourceVersion: "178232"
  selfLink: /api/v1/namespaces/shareddev/configmaps/nginx-configuration
  uid: a0c5ef9f-742d-11ea-a4a0-bac2a8381a43

Let’s make two endpoints, one that is open, but the other has an annotation to block access:

$ cat helloworld-ingress.yaml 
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: aks-helloworld
  namespace: shareddev
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
      paths:
      - path: /mysimplehelloworld
        pathType: Prefix
        backend:
          serviceName: aks-helloworld
          servicePort: 80
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: aks-helloworld-blocked
  namespace: shareddev
  annotations:
    nginx.ingress.kubernetes.io/ssl-redirect: "false"
    nginx.ingress.kubernetes.io/rewrite-target: /
    nginx.ingress.kubernetes.io/whitelist-source-range: '10.2.2.0/24'
spec:
  rules:
  - http:
      paths:
      - path: /mysimplehelloworldb
        pathType: Prefix
        backend:
          serviceName: aks-helloworld
          servicePort: 80

$ kubectl apply -f helloworld-ingress.yaml --validate=false
ingress.networking.k8s.io/aks-helloworld configured
ingress.networking.k8s.io/aks-helloworld-blocked created

We can now see that the “b” (blocked) endpoint is blocked by Nginx but the other remains open:

You can see the top is rejected but the lower is allowed

Summary

When it comes to controlling access to the resources in our cluster, we have illustrated two ways we can constrain access. The first uses cloud native perimeter security techniques that would apply to any cloud compute. This is the stricter approach but does require access to modify Network Security Groups. The other technique uses kubernetes native annotations to selectively limit and enable access to specific ingress pathways.