Skip to main content

Secure nginx with Let's Encrypt on Ubuntu 18.04

Monitoring Kubernetes with Prometheus - Linux Guru

How to deploy Prometheus on Kubernetes.

Image Source Prometheus (software)

Prometheus is an open-source systems monitoring and alerting toolkit, You can monitor various resources using Prometheus. Prometheus has multiple modes for visualisation data: a built-in expression browse. In this article we will deploy prometheus server in Kubernetes and Monitor resorces of kubernetes cluster. after deploying prometheus you can confgure prometheus server url in grafana to visulize metric graph.
Here are the steps to deploy prometheus on kubernetes.

Create namespace

We will create namespace called monitoring and deploy prometheus into that namespace. create a file name with namespace.yaml.
#Create namespace

apiVersion: v1
kind: Namespace
metadata:
  name: monitoring

kubectl create -f namespace.yaml

Create cluster role and binding.

Now we will create cluster role for correct permission to the cluster in order to deploy and access resources. create a file name with clusterrole.yaml.
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/proxy
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: default
  namespace: monitoring
kubectl create -f clusterrole.yaml

Create configmap.

Now we will create configmap for prometheus, You can define scrap config and targets into the configmap and we will manage prometheus.conf file from configmap. Create a file name with configmap.yaml.
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-server-conf
  labels:
    name: prometheus-server-conf
  namespace: monitoring
data:
  prometheus.yml: |-
    global:
      scrape_interval: 5s
      evaluation_interval: 5s

    scrape_configs:
      - job_name: 'kubernetes-apiservers'
        kubernetes_sd_configs:
        - role: endpoints
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        relabel_configs:
        - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
          action: keep
          regex: default;kubernetes;https

      - job_name: 'kubernetes-nodes'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics

      - job_name: 'kubernetes-pods'
        kubernetes_sd_configs:
        - role: pod
        relabel_configs:
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
          action: replace
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
          target_label: __address__
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_pod_name]
          action: replace
          target_label: kubernetes_pod_name

      - job_name: 'kubernetes-cadvisor'
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
        - role: node
        relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor

      - job_name: 'kubernetes-service-endpoints'
        kubernetes_sd_configs:
        - role: endpoints
        relabel_configs:
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape]
          action: keep
          regex: true
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme]
          action: replace
          target_label: __scheme__
          regex: (https?)
        - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path]
          action: replace
          target_label: __metrics_path__
          regex: (.+)
        - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port]
          action: replace
          target_label: __address__
          regex: ([^:]+)(?::\d+)?;(\d+)
          replacement: $1:$2
        - action: labelmap
          regex: __meta_kubernetes_service_label_(.+)
        - source_labels: [__meta_kubernetes_namespace]
          action: replace
          target_label: kubernetes_namespace
        - source_labels: [__meta_kubernetes_service_name]
          action: replace
          target_label: kubernetes_name


      - job_name: kubernetes-nodes-cadvisor
        scrape_interval: 10s
        scrape_timeout: 10s
        scheme: https  # remove if you want to scrape metrics on insecure port
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        kubernetes_sd_configs:
          - role: node
        relabel_configs:
          - action: labelmap
            regex: __meta_kubernetes_node_label_(.+)
          - target_label: __address__
            replacement: kubernetes.default.svc:443
          - source_labels: [__meta_kubernetes_node_name]
            regex: (.+)
            target_label: __metrics_path__
            replacement: /api/v1/nodes/${1}/proxy/metrics/cadvisor
        metric_relabel_configs:
          - action: replace
            source_labels: [id]
            regex: '^/machine\.slice/machine-rkt\\x2d([^\\]+)\\.+/([^/]+)\.service$'
            target_label: rkt_container_name
            replacement: '${2}-${1}'
          - action: replace
            source_labels: [id]
            regex: '^/system\.slice/(.+)\.service$'
            target_label: systemd_service_name
            replacement: '${1}'


here we have use the below jobs to monitoring metrics.
kubernetes-cadvisor : It will get caadvisor metrics from kubernetes.
kubernetes-service-endpoints : it will get the metrics of services.
kubernetes-pods : It will get metrics of kubernetes pods.
kubernetes-nodes : It will get metrics of kubernetes nodes.
kubernetes-apiservers : It will get metrics from kubernetes apis.

Apply prometheus Deployment and service.

Now final step is to deploy prometheus service in kubernetes using deployment.yaml and expose it to the load balancer port 9090 using service.yaml. Create a file name with deployment.yaml.
---
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
spec:
  replicas: 1
  template:
    metadata:
      labels:
        app: prometheus
    spec:
      containers:
        - name: prometheus
          image: prom/prometheus:v2.2.1
          args:
            - "--config.file=/etc/prometheus/prometheus.yml"
            - "--storage.tsdb.path=/prometheus/"
          ports:
            - containerPort: 9090
          volumeMounts:
            - name: prometheus-config-volume
              mountPath: /etc/prometheus/
            - name: prometheus-storage-volume
              mountPath: /prometheus/
      volumes:
        - name: prometheus-config-volume
          configMap:
            defaultMode: 420
            name: prometheus-server-conf

        - name: prometheus-storage-volume
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus-service
  namespace: monitoring
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/port:   '9090'

spec:
  selector:
    app: prometheus
  type: LoadBalancer
  ports:
    - port: 9090
      targetPort: 9090

Now wait for some time and then check the service which we expose to the namespace monitoring. it will provide load balancer ip to access prometheus endpoint.
❯ kubectl get services -n monitoring

NAME                 TYPE           CLUSTER-IP    EXTERNAL-IP   PORT(S)          AGE
prometheus-service   LoadBalancer   10.12.2.140   34.67.0.162   9090:31615/TCP   96s
So the load balancer url is created and now you can access prometheus using the URL.
http://34.67.0.162:9090


Comments

  1. Useful Article. Nicely written

    ReplyDelete
    Replies
    1. Glad you like it. More article on monitoring will come soon.

      Delete

Post a Comment

Popular posts from this blog

Another instance of Certbot is already running - Solved - Linux guru

Error : Another instance of Certbot is already runningcertbot --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns --installer nginx -d *.vishalvyas.com Another instance of Certbot is already running Reason : when you run certbot form your machine and unexpectedly stop the command, Then cert bot is not running but it left some .certbot.lock files behind.You need to kill the certbot instance form your machine. Solution : Run the below command to find killed certbot.find / -type f -name ".certbot.lock" You can see result of the command, If there are, you can remove them. Run below command to remove them.find / -type f -name ".certbot.lock" -exec rm {} \; And try again.

Install Apache Using Ansible Playbooks

Ansible is a automation tool which is widely used, you can install and install, configure and manage number of system and services remotely. you can install software and manage services and tasks without needing manually log in to each servers. you have to install ansible in one machine and use ssh to communicate host each other.



Ansible uses Playbooks which is written in YAML format. it's uses module base format. with playbook can run multiple tasks at time and provide more advance functionality, YAML file always start with "---" syntax. In this article we will see how to install and configure apahce2 using ansible YAML script. Master server : 10.80.253.11 [Ansible server, ] Slave 1 : 10.80.253.12 [need to install apache2 in slave 1] Slave 2 : 10.80.253.13 [need to install apache2 in slave 2] We need to configure slave server info in our ansible configuration file, click here to know how to add client machine to ansible. Let's update our apache2.yml file and install…

Create aws s3 bucket using cli

In this article i will explain you to how to manage s3 bucket and objects using AWS cli command line interface. for that you will have to configure AWS cli interface to your local linux machine. You can configure cli from. HERE.




After configured aws cli to your local machine, you can run all below commands. How to create bucket using AWS Cli.root@master:~# aws s3 mb s3://linuxgurubucket make_bucket: linuxgurubucket Listing Bucket : To list all bucket : root@master:~# aws s3 ls 2018-05-01 15:28:37 linuxgurubucket To list all objects from bucket :root@master:~# aws s3 ls s3://linuxgurubucket 2018-05-01 15:32:16 71515 vishal.jpg How to copy local file to s3 using aws cp command :root@master:~# aws s3 cp ansible.sh s3://linuxgurubucket/ upload: ./ansible.sh to s3://linuxgurubucket/ansible.sh To check it's upload or not :root@master:~# aws s3 ls s3://linuxgurubucket/ 2018-05-01 15:34:19 82 ansible.sh 2018-05-01 15:32:16 71515 vishal.jpg To remove file from s3 bucket using cli: