Autoscale your Kubernetes Workloads intelligently using Keda

Keda is event driven autoscaler built for Kubernetes

Why autoscaling?

One of the most crucial aspects in application hosting is how it can respond to variable load. During the lifetime of hosted applications, applications experience variability in load due to various reasons like seasonality, marketing events, etc. It is essential that our application can respond to such events successfully, without any impact on its SLA. But it is also not economical to hold huge amounts of hosting resources for seasonal and occasional variables in load. This is where autoscaler comes in. Autoscaler is a component which can trigger scale-up / scale down events in the kubernetes cluster to scale deployment, statefulsets and other such resources.

What are our options for autoscaling?

We can either scale horizontally (add more instances of the application) or vertically (make more resources available to the application). Typically, it’s more complicated to change the available resources for the pod in the kubernetes cluster. It’s quite easier to just scale one more / less replica of the application.

We have below options when it comes to scaling in kubernetes cluster

  • Horizontal Pod Autoscaler – HPA is a first class citizen of Kubernetes. You can configure the HPA manually for each application.
  • Keda – Keda is a Kubernetes operator for HPA – which adjusts HPA objects based on events. Events can happen within or completely outside the cluster.
  • VPA – Vertical Pod Autoscaler – VPA works as an operator which can adjust the requested resources based on actual usage of the application.
  • Goldilocks – Goldilocks helps to create VPA easily.
  • Cluster Autoscaler – Cluster Autoscaler worker rather at worker node level and helps you create or remove worker nodes depending upon the overall workload in the cluster.

Today, we would see HPA and Keda in depth.

Why Keda over Horizontal Pod Autoscaler?

KEDA Logo

KEDA is a Kubernetes-based Event Driven Autoscaler. With KEDA, Kubernetes workloads can be autoscaled based on the number of events needing to be processed.

Internally, Keda uses HPA to delegate actual scaling up and scaling down operations. Keda really shines to scale the “processing” jobs based on onslaught of incoming load. E.g. Kafka Queue depth, AWS Kinesis Stream’s shard count, number of files in Azure blob storage. See here for all available Keda scalers. It is easy to also extend and write your own scaler which can react to events of your choice.

Quick start using Keda (show me the money)!

Quick start using Keda

Keda defines a new Kubernetes Custom Resource (CR) – ScaledObject – which maps the Kubernetes Workload which needs to be scaled up/down with external triggers source which will trigger scale up/down.

To demonstrate simple event driven usage of keda – we will be creating a number of files in Azure blob storage. In the example below you will see The ScaledObject definition which will map nginx deployment as target with count of files in Azure Blob Storage as trigger source.

Pre-requisites:

  1. A kubernetes cluster with supported kubernetes version
  2. Helm installation
  3. Azure login with permissions to create storage account and containers. (or just create)

Deploy Keda using helm

Run below commands to install Keda operator in your kubernetes cluster

az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/1.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/2.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/3.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/4.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/5.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/6.txt”

Create a storage account in Azure

az storage account create -n k8sadventure -g k8s-adventure -l westus –sku Standard_LRS –allow-blob-public-access false
az storage account show-connection-string -g k8s-adventure -n k8sadventure
# Make note of this connection string as it will be needed in below command as well as later in definition of secret for Scaled Object below
az storage container create -n keda-test –account-name k8sadventure –resource-group k8s-adventure –connection-string

Deploy the target workload – which needs to be controlled by Keda

Lets deploy below yaml to install a simple nginx application in default namespace.

apiVersion: apps/v1
kind: Deployment
metadata:
name: nginx-deployment
labels:
app: nginx
spec:
replicas: 1
selector:
matchLabels:
app: nginx
template:
metadata:
labels:
app: nginx
spec:
containers:
– name: nginx
image: nginx:1.14.2
ports:
– containerPort: 80

Deploy ScaledObjects

Lets now deploy the actual Keda ScaledObject to start watching Azure blob container and adjust workload! We use secret and TriggerAuthentication to separate out the sensitive information out of ScaledObject definition.

The ScaledObject definition below says, it will monitor the keda-test storage container and whenever the count of files go beyond 5 (or multiple of 5), it will proportionately, increase / decrease the number replicas of target deployment which is defined via scaledTargetRef in the spec.

apiVersion: v1
kind: Secret
metadata:
name: azure-storage-connectionstr
namespace: default
data:
azure-storage-connectionstring: <BASE64 encoded Azure Storage connection string>

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
name: azure-blob-auth
spec:
secretTargetRef:
– parameter: connection
name: azure-storage-connectionstr
key: azure-storage-connectionstring

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
name: azure-blob-scaledobject
namespace: default
spec:
scaleTargetRef:
name: nginx-deployment
minReplicaCount: 1
triggers:
– type: azure-blob
metadata:
blobContainerName: keda-test
blobCount: “5”
blobDelimiter: “/”
authenticationRef:
name: azure-blob-auth

Observe the autoscaling behavior

Lets add 6 files to azure container

az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/1.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/2.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/3.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/4.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/5.txt”
az storage azcopy blob upload -c keda-test –account-name k8sadventure -s “./upload/6.txt”

Within 30 seconds or so, you will nginx deployment spawning another replica and you see below events related to hpa in kubernetes events

k get event | grep hpa
38m NormalSuccessfulRescale horizontalpodautoscaler/keda-hpa-azure-
blob-scaledobject New size: 2; reason: All metrics below target
37m NormalSuccessfulRescale horizontalpodautoscaler/keda-hpa-azure-
blob-scaledobject New size: 2; reason: external metric s0-azure-blob-keda-
test(&LabelSelector{MatchLabels:map[string]string{scaledobject.keda.sh/name: azure-blob-scaledobject,},MatchExpressions:[]LabelSelectorRequirement{},}) above target

If we remove one of the file from Azure storage container (to bring the total count of files to 5) now, you will see the HPA reducing the replicas back to one after the cool down period (default 5 minutes).

0s NormalSuccessfulRescale horizontalpodautoscaler/keda-hpa-azure blob-scaledobject New size: 1; reason: All metrics below target
0s NormalScalingReplicaSet deployment/nginx-deployment
Scaled down replica set nginx-deployment-9456bbbf9 to 1
0s NormalSuccessfulDelete replicaset/nginx-deployment-9456bbbf9
Deleted pod: nginx-deployment-9456bbbf9-2ssgr
0s NormalKilling pod/nginx-deployment-9456bbbf9-2ssgr
Stopping container nginx

Recent blogs