<img height="1" width="1" style="display:none;" alt="" src="https://px.ads.linkedin.com/collect/?pid=4958233&amp;fmt=gif">
RSS Feed

Architecture | Armen Kojekians |
30 April 2019


This guide provides concepts and principles which are best considered during the design of a Kubernetes environment (abbreviated as K8s in this document). It aims to provide the reader good advice on how to maintain large workloads in a secure manner. Where possible the document will remain vendor agnostic, it may however refer to specific cloud providers where useful to allow reader to broaden their knowledge of how versatile K8s is and what approach the cloud vendors have used to offer a more reliable solution. It is assumed that the reader has good understanding of containerisation and some knowledge of orchestration. This document should not be considered a step-by-step guide on how to build a K8s cluster, for this there are plenty of resources online. Instead it aims harness the great features K8s offers after the cluster has been stood up.


Once the cluster is up, the following areas need careful planning.

  • Namespaces
  • Requests and Limits
  • Labels and Selectors
  • Secrets
  • Network Policies


Think of namespaces as a virtual cluster inside the Kubernetes cluster. Namespaces are all isolated from each other. They can help the project with organisation, security and even performance. You can deploy multiple namespaces which are all independent of each other. This allows the teams to have resource with same resource name inside different namespaces allowing scalability inside a single K8s cluster. A typical out of the box installation comes with 3 default namespaces; default, kube-system, kube-public. It is advised to leave these namespaces as they are with the exception of default which can be used in a POC and non-production workload environment.

Although earlier it was mentioned that names spaces were isolated, this does not mean pods within different namespace cannot communicate with each other, they can. Built in service discovery and DNS allows you to refer to other namespaces.

Organisations often start out with a single cluster that handles all workloads, both production and non-production, with services being developed by a single team. In this case it makes sense to define namespaces based on environment, for example, namespaces for development, test and production with network policies (see below) to provide a measure of network isolation between the environments. Over time though, organisations often spin up separate clusters for the environments which offers genuine network separation and the potential to run larger workloads. In this case a good practice is to organise the namespaces around teams and/or functional areas, particularly where application development follows Domain Driven Design principles and occurs within bounded contexts. In this case create namespaces per bounded context.

Bottom line:

Always organise your clusters into namespaces and don’t deploy pods into default. Use environments as the basis for namespaces where you only have a single cluster and base namespaces on team, product or functional / business area when environments have been allocated dedicated clusters.


Requests and limits are the features in K8s which control resources such as CPU and memory. Requests are what the container is guaranteed to get. If a container requests a resource, K8s will only schedule it on a node that can give it that resource. Limits, on the other hand, prevent a container to go above a certain value. The container is only allowed to go up to the limit, and then it is restricted.

K8s uses the concept of millicores as a measure of CPU . If your application needs a whole core, assign 1000 millicores to the container. If your container needs quarter of a core, assign 250 millicores to the container. One thing to note, if your manifest states that your container has say 8000 millicores, but largest node only has 4 cores, the scheduler will not be able to launch the container. Unless the application is designed to utilise multi cores, it is typically better to run multiple replicas of the containers with lower cores than larger containers but in less numbers. This introduces more flexibility and reliability into the cluster.

Memory is allocated in bytes where KB, MB, GB etc. are unit prefixes denoting Kilo, Mega or Giga bytes in decimal i.e multiples of 1000, or KiB, MiB, GiB are unit prefixes for binary, ie, multiples of 1024.

Kubernetes-Design Principles Part 1

Bottom Line:

Always use Requests and Limits to prevent cluster running out of resources. Requests and limits along with Resource Quotas work very well. Whereby Resource Quotas are set at the Namespace level and request and limits at the container level. Example; in a 3-node cluster where the aggregate memory is 48GB and CPU cores is 36, a Resource Quotas, the Containers can be allocated 40GB memory and 30 CPUs.


Labels are key – value pairs that are attached to K8s objects and are used to organise and select subsets of K8s objects. They allow users to map their own organisational structure onto their K8s clusters. Some example labels might be:

env: dev or env: prod
version: v1 or version: v2

In order to fully appreciate labels and selectors we need to understand the concept of Services. Once we understand this we can continue to look at labels. Every object in Kubernetes in particular pods should get labels. At high level:

Labels are metadata you can assign to any API object.
They are a grouping mechanism for pods
They are used by selectors

Let’s take an example where we have a back-end application in your K8s Cluster in the production environment and version 1.1, we label this as env: Prod, role:BE, version: 1.1. There are 2 replicas of this pod.

Docker container

We then place a Service above these pods to allow the front-end application to discover the back-end pods.

Kubernetes Design Principles 4

We then use the same three labels on the Services object which essentially creates a filter stating “Find all pods with the labels env: prod, role: BE and version 1.1 and route traffic to these pods”. Here very quickly and simplistically we have tied the Services Object to the back end pods.

Taking the same example as above, we go ahead a place a new pod in the cluster with the labels Dev and 1.0. Because these labels do not match, the Services object will not route traffic to this pod.

Bottom Line

Always use Labels and Selectors to tie your API objects together giving you both control and a way to quickly search for objects with common roles.


Kubernetes Secrets are another resource object which hold small amount of information such as passwords, tokens or keys. The benefit of this setup is that we avoid storing this sensitive information inside container images or configuration files. Once a pod is deployed it can reference secrets in these ways:

as files in a mounted volume
used by the node agent
environment variables to be used by a container in a pod

There is an issue with the default way that K8s stores since they are stored as base64 encoded plaintext, which means with the right privileges a hexdump of the secret can be obtained which is then very simple to decrypt.

Third Party Providers

Due to growth and popularity of K8s, cloud providers and third parties are exposing their services to be consumed by such products as K8s.


Microsoft Azure allows K8S to store its secrets within the Azure Vault. Vault fully encrypts all data and therefore even privileges users cannot view to decrypt the data. To use Azure keyvault, the yaml manifest will need to contain the endpoint: unix:///opt/azurekms.socket. and this will allow the pod to read/use/destroy keys.

Hashicorp Vault

Hashicorp provide a similar product which some subtle differences which can be seen here. The key difference between using this product over say Azure or Google Key Management Service is that you can run this anywhere which means you can run it on-prem if this is one of the requirements.
Always use Secrets and do not store the sensitive data on mounted volumes or files. Using a cloud providers offering or company like hashicorp is more robust as they have added more controls, features and security to ensure the sensitive data cannot be tampered with.

Bottom Line:

Store sensitive information, such as passwords, in K8s secrets.


In simple terms Network Policies control traffic from and to Pods. Network Policies use labels to select pods and define rules which specify what traffic is allowed to the selected pods. The default setting in K8s is to set pods as non-isolated which means ‘accept traffic from any pod in the cluster’. BBy default network policies are not applied and any pod can communicate with any other pod in the cluster. Applying network policies can give tight control over network communication. We can further define other constructs within Network Policies such as, which namespace, IP Range, and source destination ports. Thinking in traditional sense, one would never have a flat network with zero network ACLs, the notion should be no different in the containerised world.

One thing to note about Network Policies is that they require a networking plugin on the K8s nodes. Deploying the Network Policy without the plugin will have no effect.

An example of the plugins are:

Calico: https://kubernetes.io/docs/tasks/configure-pod-container/calico-network-policy/
Weave Net: https://kubernetes.io/docs/tasks/configure-pod-container/weave-network-policy/

Network policies are defined at namespace level and define ingress rules which define what traffic is allowed to pods in the namespace and egress rules that define what traffic is allowed from pods in the namespace. Traffic is allowed or disallowed based on matches on one of the following criteria

  1. Pods that match certain criteria such as having a set of labels
  2. Namespaces that match certain criteria such as having a set of labels
  3. Pods and namespaces that match certain criteria such as having a set of labels
  4. Blocks of IPs within the K8s network


Bottom Line

Always use Network Policies when deploying K8s clusters. This gives you robust control over pod communication. When pod communication leaves the K8s cluster, the firewalls and security appliance outside of the cluster will take care of the traffic filtering. A good practice we have used is to, by default, deny ingress from, or egress to, pods outside of a namespace.

Armen Kojekians

Senior Infrastructure Consultant

Armen enjoys working on all things Distributed and Cloud, helping clients embrace innovation while moving existing workloads or building new products. The only clouds Armen doesn't like are the kind that keep him and his two kids indoors. He also enjoys experimenting with cooking, but admits it doesn't always produce Master Chef style results.


From This Author

  • 09 April 2019

    Keeping Up With The Norm In An Era Of Software Defined Everything



  • 13 November 2023

    Delving Deeper Into Generative AI: Unlocking Benefits and Opportunities

  • 07 November 2023

    Retrieval Augmented Generation: Combining LLMs, Task-chaining and Vector Databases

  • 19 September 2023

    The Rise of Vector Databases

  • 27 July 2023

    Large Language Models Automating the Enterprise – Part 2

  • 20 July 2023

    Large Language Models Automating the Enterprise – Part 1

  • 11 July 2023

    Boost Your Game’s Success with Tools – Part 2

  • 04 July 2023

    Boost Your Game’s Success with Tools – Part 1

  • 01 June 2023

    Challenges for Adopting AI Systems in Software Development

  • 07 March 2023

    Will AI Transform Even The Most Creative Professions?

  • 14 February 2023

    Generative AI: Technology of Tomorrow, Today

  • 25 January 2023

    The Joy and Challenge of being a Video Game Tester

  • 14 November 2022

    Can Software Really Be Green

  • 26 July 2022

    Is Data Mesh Going to Replace Centralised Repositories?

  • 09 June 2022

    A Spatial Analysis of the Covid-19 Infection and Its Determinants

  • 17 May 2022

    An R&D Project on AI in 3D Asset Creation for Games

  • 07 February 2022

    Using Two Cloud Vendors Side by Side – a Survey of Cost and Effort

  • 25 January 2022

    Scalable Microservices Architecture with .NET Made Easy – a Tutorial

  • 04 January 2022

    Create Production-Ready, Automated Deliverables Using a Build Pipeline for Games – Part 2

  • 23 November 2021

    How User Experience Design is Increasing ROI

  • 16 November 2021

    Create Production-Ready, Automated Deliverables Using a Build Pipeline for Games – Part 1

  • 19 October 2021

    A Basic Setup for Mass-Testing a Multiplayer Online Board Game

  • 24 August 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 3

  • 20 July 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 2

  • 29 June 2021

    EHR to HL7 FHIR Integration: The Software Developer’s Guide – Part 1

  • 08 June 2021

    Elasticsearch and Apache Lucene: Fundamentals Behind the Relevance Score

  • 27 May 2021

    Endava at NASA’s 2020 Space Apps Challenge

  • 27 January 2021

    Following the Patterns – The Rise of Neo4j and Graph Databases

  • 12 January 2021

    Data is Everything

  • 05 January 2021

    Distributed Agile – Closing the Gap Between the Product Owner and the Team – Part 3

  • 02 December 2020

    8 Tips for Sharing Technical Knowledge – Part 2

  • 12 November 2020

    8 Tips for Sharing Technical Knowledge – Part 1

  • 30 October 2020

    API Management

  • 22 September 2020

    Distributed Agile – Closing the Gap Between the Product Owner and the Team – Part 2

  • 25 August 2020

    Cloud Maturity Level: IaaS vs PaaS and SaaS – Part 2

  • 18 August 2020

    Cloud Maturity Level: IaaS vs PaaS and SaaS – Part 1

  • 08 July 2020

    A Virtual Hackathon Together with Microsoft

  • 30 June 2020

    Distributed safe PI planning

  • 09 June 2020

    The Twisted Concept of Securing Kubernetes Clusters – Part 2

  • 15 May 2020

    Performance and security testing shifting left

  • 30 April 2020

    AR & ML deployment in the wild – a story about friendly animals

  • 16 April 2020

    Cucumber: Automation Framework or Collaboration Tool?

  • 25 February 2020

    Challenges in creating relevant test data without using personally identifiable information

  • 04 January 2020

    Service Meshes – from Kubernetes service management to universal compute fabric

  • 10 December 2019

    AWS Serverless with Terraform – Best Practices

  • 05 November 2019

    The Twisted Concept of Securing Kubernetes Clusters

  • 01 October 2019

    Cognitive Computing Using Cloud-Based Resources II

  • 17 September 2019

    Cognitive Computing Using Cloud-Based Resources

  • 03 September 2019

    Creating A Visual Culture

  • 20 August 2019

    Extracting Data from Images in Presentations

  • 06 August 2019

    Evaluating the current testing trends

  • 23 July 2019

    11 Things I wish I knew before working with Terraform – part 2

  • 12 July 2019

    The Rising Cost of Poor Software Security

  • 09 July 2019

    Developing your Product Owner mindset

  • 25 June 2019

    11 Things I wish I knew before working with Terraform – part 1

  • 30 May 2019

    Microservices and Serverless Computing

  • 14 May 2019

    Edge Services

  • 30 April 2019

    Kubernetes Design Principles Part 1

  • 09 April 2019

    Keeping Up With The Norm In An Era Of Software Defined Everything

  • 25 February 2019

    Infrastructure as Code with Terraform

  • 11 February 2019

    Distributed Agile – Closing the Gap Between the Product Owner and the Team

  • 28 January 2019

    Internet Scale Architecture