prometheus

修改config文件,在里面添加一行. spec: retention: 365d 完整的实例,更新完成之后,记得apply一下. [root@VM-64-25-centos manifests]# cat prometheus-prometheus.yaml apiVersion: monitoring.coreos.com/v1 kind: Prometheus metadata: labels: prometheus: k8s name: k8s namespace: monitoring spec: alerting: alertmanagers: - name: alertmanager-main namespace: monitoring port: web image: quay.io/prometheus/prometheus:v2.17.2 nodeSelector: kubernetes.io/os: linux podMonitorNamespaceSelector: {} podMonitorSelector: {} replicas: 2 resources: requests: memory: 500Mi ruleSelector: matchLabels: prometheus: k8s role: alert-rules securityContext: fsGroup: 2000 runAsNonRoot: true runAsUser: 1000 additionalScrapeConfigs: name: additional-configs key: prometheus-additional.yaml retention: 7d #添加一行,持久化7天 storage: volumeClaimTemplate: spec: storageClassName: prometheus-cfs resources: requests: storage: 100Gi serviceAccountName: prometheus-k8s serviceMonitorNamespaceSelector: {} serviceMonitorSelector: {} version: v2. [Read More]

下载源代码安装prometheus-operator的时候报错 git clone https://github.com/coreos/kube-prometheus.git 执行 kubectl apply -f manifests/ 报错 unable to recognize “manifests/0prometheus-operator-serviceMonitor.yaml”: no matches for kind “ServiceMonitor” in version “monitoring.coreos.com/v1” unable to recognize “manifests/alertmanager-alertmanager.yaml”: no matches for kind “Alertmanager” in version “monitoring.coreos.com/v1” unable to recognize "manifests/alertmanager-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" unable to recognize "manifests/kube-state-metrics-serviceMonitor.yaml": no matches for kind "ServiceMonitor" in version "monitoring.coreos.com/v1" unable to recognize “manifests/node-exporter-serviceMonitor.yaml”: no matches for kind “ServiceMonitor” in version “monitoring.coreos.com/v1” unable to recognize “manifests/prometheus-prometheus. [Read More]

参考资料 GitHub - coreos/prometheus-operator: Prometheus Operator creates/configures/manages Prometheus clusters atop Kubernetes git clone kube-prometheus;需要注意的是,代码已经从原来的地址迁移到这个地址 git clone https://github.com/coreos/prometheus-operator.git minikube安装部署,请参考另外一篇文章 安装 # create namespace and CRDs kubectl apply -f manifests/setup # wait for CRD creation to complete until kubectl get servicemonitors --all-namespaces ; do date; sleep 1; echo ""; done # create monitoring components kubectl apply -f manifests/ 查看 ➜ kube-prometheus git:(master) ✗ k get pods -n monitoring NAME READY STATUS RESTARTS AGE alertmanager-main-0 2/2 Running 4 104m alertmanager-main-1 2/2 Running 4 104m alertmanager-main-2 2/2 Running 4 104m grafana-58dc7468d7-tzsnh 1/1 Running 2 105m kube-state-metrics-769f4fd4d5-85f7p 3/3 Running 7 105m node-exporter-22l2n 2/2 Running 4 105m prometheus-adapter-5cd5798d96-ttfm2 1/1 Running 3 105m prometheus-k8s-0 3/3 Running 7 61m prometheus-k8s-1 3/3 Running 0 51m prometheus-operator-99dccdc56-klb8r 1/1 Running 4 105m ➜ kube-prometheus git:(master) ✗ 查看创建的服务,需要注意的是,我们需要修改prometheus-k8s和grafana这两个service的类型为NodePort;如下操作: ➜ kube-prometheus git:(master) ✗ k edit svc grafana -n monitoring ➜ kube-prometheus git:(master) ✗ k edit svc prometheus-k8s -n monitoring 查看修改完成的service; ➜ kube-prometheus git:(master) ✗ k get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-main ClusterIP 10. [Read More]

Prerequisites:

* macOS 10.12 (Sierra)
* A hypervisor such as Hyperkit, Parallels, VirtualBox, or VMware Fusion

Installation:

brew install minikube

Upgrading minikube:

brew update brew upgrade minikube

Hypervisor Setup for Hypervisor

Increasing memory allocation:

minikube config set memory 4096

常用命令:

Start a cluster by running: minikube start Access the Kubernetes Dashboard running within the minikube cluster: minikube dashboard minikube service list Stop your local cluster: minikube stop Delete your local cluster: minikube delete Delete all local clusters and profiles minikube delete —all

目标 在生产环境中,我们需要除了监控kubernetes中的一些资源对象、节点以及组件,有时候还需要根据实际的业务添加自定义监控.添加自定义步骤如下: 1. 先创建一个ServiceMonitor对象,用于为Prometheus添加监控项 2. 将ServiceMonitor对象关联metrics数据接口的一个Service对象; 3. Service对象可以正确获取metrics数据 查看nginx完整的带metrics的文件 ➜ manifests git:(master) ✗ cat nginx.yaml apiVersion: apps/v1 kind: Deployment metadata: name: nginx labels: app: nginx spec: replicas: 1 selector: matchLabels: app: nginx template: metadata: labels: app: nginx spec: containers: - name: nginx image: billy98/nginx-prometheus-metrics:latest ports: - name: http-metrics containerPort: 9527 - name: web containerPort: 80 - name: test containerPort: 1314 imagePullPolicy: IfNotPresent --- apiVersion: v1 kind: Service metadata: labels: app: nginx name: nginx namespace: default spec: ports: - name: http-metrics port: 9527 protocol: TCP targetPort: 9527 - name: web port: 80 protocol: TCP targetPort: 80 - name: test port: 1314 protocol: TCP targetPort: 1314 selector: app: nginx type: NodePort ➜ manifests git:(master) ✗ 配置prometheus监控 ➜ manifests git:(master) ✗ cat prometheus-serviceMonitornginx. [Read More]

首先把alertmanager-main这个service改为NodePort的server. ➜ manifests git:(master) ✗ k get svc -n monitoring NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE alertmanager-main NodePort 10.96.120.87 <none> 9093:31175/TCP 29h alertmanager-operated ClusterIP None <none> 9093/TCP,9094/TCP,9094/UDP 29h grafana NodePort 10.96.163.124 <none> 3000:31263/TCP 29h kube-state-metrics ClusterIP None <none> 8443/TCP,9443/TCP 29h node-exporter ClusterIP None <none> 9100/TCP 29h prometheus-adapter ClusterIP 10.96.121.50 <none> 443/TCP 29h prometheus-k8s NodePort 10.96.129.132 <none> 9090:30525/TCP 29h prometheus-operated ClusterIP None <none> 9090/TCP 29h prometheus-operator ClusterIP None <none> 8080/TCP 29h ➜ manifests git:(master) ✗ 可以使用svc“alertmanager-main”的nodeport查看Alertmanager的Status状态,获取访问prometheus地址和alertmanager地址. [Read More]

注意所在操作都在,manifests目录下面创建, ➜ manifests git:(master) ✗ cat prometheus-additional.yaml - job_name: ‘kubernetes-service-endpoints' kubernetes_sd_configs: - role: endpoints relabel_configs: - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scrape] action: keep regex: true - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_scheme] action: replace target_label: __scheme__ regex: (https?) - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_path] action: replace target_label: __metrics_path__ regex: (.+) - source_labels: [__address__, __meta_kubernetes_service_annotation_prometheus_io_port] action: replace target_label: __address__ regex: ([^:]+)(?::\d+)?;(\d+) replacement: $1:$2 - action: labelmap regex: __meta_kubernetes_service_label_(.+) - source_labels: [__meta_kubernetes_namespace] action: replace target_label: kubernetes_namespace - source_labels: [__meta_kubernetes_service_name] action: replace target_label: kubernetes_name ➜ manifests git:(master) ✗ 使用这个文件创建一个secret对象并查看 ➜ manifests git:(master) ✗ kubectl create secret generic additional-configs --from-file=prometheus-additional. [Read More]