Unni Krishnan's Blog

Unni Krishnan's Blog

Get your OOM on Slack

Get your OOM on Slack

Image for post

Slack

Slack is like any other communication platform but simpler, more flexible & sometimes prettier. It is simple & awesome at the same time.

I love the integrations that Slack provides

  • Add slack provided apps
  • Add apps by other great developers
  • Create your own webhooks, with message formatting & embedding features
  • Create your own bots, who listen to your commands

Kubernetes

k8 is awesome too. I wonder how we would be deploying our apps if Docker & k8 were not developed

  • Container-based orchestration has been made easy
  • Designed for development & DevOps

Java

Programming in Java is my bread & butter

So what problem are we trying to solve by using the three of them?

  • Send alert when a pod restarts, automated deployment, manual patches etc
  • Send alert when pod restarts because of OOM Killed
  • Send alert when a java pod restarts due to OutOfMemory errors

We are using kubewatch, kubewatch is a Kubernetes watcher that currently publishes notification to Slack. Run it in your k8s cluster, and you will get event notifications in a slack channel.

We use the k8 lifecycle events to send notifications about the pod that is going to restart. The notification contains the pod name, any other environment specific details & the location of the log files.

**apiVersion: v1
kind: Pod
metadata:
  name: lifecycle-demo
spec:
  containers:
  - name: lifecycle-demo-container
    image: nginx
    lifecycle:
      preStop:
        exec:** **command: ["/bin/sh", "-c",'curl -X POST --data-urlencode "payload={ \"attachments\": [{ \"title\": \" Pod Stopped \", \"color\": \"#F35A00\", \"attachment_type\": \"default\", \"fields\": [{ \"title\": \"Environment\", \"value\": \"${APP_ENV}\", \"short\": false }, { \"title\": \"Pod name\", \"value\": \"${POD_NAME}\", \"short\": false }, { \"title\": \"Log name\", \"value\": \"${tar_name}\", \"short\": true } ] }] }" $SLACK_URL']**</span>

On prestop, the k8 pod executes the command provided by us, which is just a simple curl command with the payload with all information and slack message formatting. For pod must have curl installed or any other equivalent can be used

$SLACK_URL is the custom integration webhook configured and provided to the k8 pod using configmaps

Java OutOfMemoryError exception, this error is thrown when there is insufficient space to allocate an object in the Java heap.

The Xms & Xmx parameters are used to set the JVM heap memory but there can be out of memory issues because of the sheer volume of requests, because of memory leaks or incorrect implementations.

While running the java application along with the JVM & heap related arguments we also provide the below argument

-**XX:OnOutOfMemoryError**="$Slack_cmd"</span>

$Slack_cmd variable will hold the details about the slack URL & the heap dump location, on OutOfMemory exception this command would be executed & we would get a notification on Slack.

With the help of all these integrations, monitoring our applications is as simple as chatting on a Slack channel.

#kubernetes#java#slack
 
Share this