springboot2.X整合prometheus監控的實例講解
springboot2.x暴露健康狀況通過prometheus監控
加入依賴
<!--prometheus監控 https://prometheus.io/docs/introduction/overview/--> <dependency> <groupId>org.springframework.boot</groupId> <artifactId>spring-boot-starter-actuator</artifactId> </dependency> <dependency> <groupId>io.micrometer</groupId> <artifactId>micrometer-registry-prometheus</artifactId> </dependency> <!--prometheus監控 https://prometheus.io/docs/introduction/overview/-->
application.yml加入相關配置
management: security: enabled: false #prometheus+grafana+springboot2監控集成配置 metrics: export: prometheus: enabled: true jmx: enabled: true endpoints: web: exposure: include: '*' base-path: /metrics #prometheus+grafana+springboot2監控集成配置
主啟動類加入配置
package com.drore.saas; import com.drore.saas.services.service.StorageService; import io.micrometer.core.instrument.MeterRegistry; import org.springframework.boot.CommandLineRunner; import org.springframework.boot.SpringApplication; import org.springframework.boot.actuate.autoconfigure.metrics.MeterRegistryCustomizer; import org.springframework.boot.autoconfigure.SpringBootApplication; import org.springframework.boot.web.servlet.ServletComponentScan; import org.springframework.context.annotation.Bean; @SpringBootApplication @ServletComponentScan public class TenantappApplication { public static void main(String[] args) { SpringApplication.run(TenantappApplication.class, args); } @Bean CommandLineRunner init(StorageService storageService) { return (args) -> { storageService.init(); }; } #prometheus+grafana+springboot2監控集成配置 @Bean MeterRegistryCustomizer meterRegistryCustomizer(MeterRegistry meterRegistry) { return meterRegistry1 -> { meterRegistry.config() .commonTags("application", "Tenantapp"); }; } #prometheus+grafana+springboot2監控集成配置 }
啟動之後通過路徑訪問查看健康狀況
http://xxxxxx/metrics/prometheus
查看到的數據如下:
[root@saas98 /]$ curl "http://10.98.94.220:80/ts/metrics/prometheus" # HELP process_start_time_seconds Start time of the process since unix epoch. # TYPE process_start_time_seconds gauge process_start_time_seconds{application="Tenantapp",} 1.556068841226E9 # HELP tomcat_threads_busy_threads # TYPE tomcat_threads_busy_threads gauge tomcat_threads_busy_threads{application="Tenantapp",name="http-nio-9081",} 1.0 # HELP tomcat_sessions_expired_sessions_total # TYPE tomcat_sessions_expired_sessions_total counter tomcat_sessions_expired_sessions_total{application="Tenantapp",} 0.0 # HELP tomcat_sessions_active_current_sessions # TYPE tomcat_sessions_active_current_sessions gauge tomcat_sessions_active_current_sessions{application="Tenantapp",} 0.0 # HELP jvm_gc_memory_promoted_bytes_total Count of positive increases in the size of the old generation memory pool before GC to after GC # TYPE jvm_gc_memory_promoted_bytes_total counter jvm_gc_memory_promoted_bytes_total{application="Tenantapp",} 1.18894656E8 # HELP tomcat_global_request_max_seconds # TYPE tomcat_global_request_max_seconds gauge tomcat_global_request_max_seconds{application="Tenantapp",name="http-nio-9081",} 3.366 # HELP jvm_memory_used_bytes The amount of used memory # TYPE jvm_memory_used_bytes gauge jvm_memory_used_bytes{application="Tenantapp",area="heap",id="Survivor Space",} 653880.0 jvm_memory_used_bytes{application="Tenantapp",area="nonheap",id="Metaspace",} 1.36445248E8 jvm_memory_used_bytes{application="Tenantapp",area="heap",id="Eden Space",} 7511504.0 jvm_memory_used_bytes{application="Tenantapp",area="nonheap",id="Code Cache",} 3.8031424E7 jvm_memory_used_bytes{application="Tenantapp",area="heap",id="Tenured Gen",} 1.3880212E8 jvm_memory_used_bytes{application="Tenantapp",area="nonheap",id="Compressed Class Space",} 1.7220968E7 # HELP tomcat_sessions_created_sessions_total # TYPE tomcat_sessions_created_sessions_total counter tomcat_sessions_created_sessions_total{application="Tenantapp",} 0.0 # HELP system_cpu_count The number of processors available to the Java virtual machine # TYPE system_cpu_count gauge system_cpu_count{application="Tenantapp",} 1.0 # HELP tomcat_global_sent_bytes_total # TYPE tomcat_global_sent_bytes_total counter tomcat_global_sent_bytes_total{application="Tenantapp",name="http-nio-9081",} 8168269.0 # HELP jvm_threads_daemon_threads The current number of live daemon threads # TYPE jvm_threads_daemon_threads gauge jvm_threads_daemon_threads{application="Tenantapp",} 34.0 # HELP logback_events_total Number of error level events that made it to the logs # TYPE logback_events_total counter logback_events_total{application="Tenantapp",level="debug",} 0.0 logback_events_total{application="Tenantapp",level="error",} 965.0 logback_events_total{application="Tenantapp",level="warn",} 4.0 logback_events_total{application="Tenantapp",level="info",} 1047.0 logback_events_total{application="Tenantapp",level="trace",} 0.0 # HELP tomcat_cache_access_total # TYPE tomcat_cache_access_total counter tomcat_cache_access_total{application="Tenantapp",} 0.0 # HELP tomcat_servlet_error_total # TYPE tomcat_servlet_error_total counter tomcat_servlet_error_total{application="Tenantapp",name="dispatcherServlet",} 0.0 tomcat_servlet_error_total{application="Tenantapp",name="statViewServlet",} 0.0 tomcat_servlet_error_total{application="Tenantapp",name="default",} 0.0 # HELP jvm_buffer_total_capacity_bytes An estimate of the total capacity of the buffers in this pool
prometheus-operator監控java應用整合
目前的prometheus-operator目錄下的相關文件
https://github.com/coreos/prometheus-operator.git
已經移植到
https://github.com/coreos/kube-prometheus.git
https://github.com/coreos/kube-prometheus/manifests
具體原因參照官網。
下面也有自己總結的文檔的地址,大傢可以參照下:
https://github.com/hkj123/kube-prometheus-manifests.git
具體的文件如下圖展示,這裡有很多文件,我們隻介紹關註的幾個文件,其他的可以參照官網瞭解:
構建springboot項目的時候需要註意的點:
--- kind: Service apiVersion: v1 metadata: labels: app: ms #prometheus配置監控需要通過app去監控,這個k8s-app和app都可以 namespace: drore-saas name: ms spec: ports: - name: http-ms #prometheus配置監控需要通過port name去識別,當然也可以直接配置端口,不過這樣看起來更專業 nodePort: 31945 protocol: TCP port: 80 targetPort: 80 selector: app: ms type: NodePort ---
部署prometheus需要註意的點
https://github.com/coreos/kube-prometheus/manifests
提供瞭一套整合的例子:
需要修改的文件: prometheus-clusterRole.yaml
默認的文件內容:
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus-k8s rules: - apiGroups: - "" resources: - nodes/metrics verbs: - get - nonResourceURLs: - /metrics verbs: - get
可以發現這裡的監控的resources:- nodes/metrics
我們需要監控自己構建的java應用需要擴大權限
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: prometheus-k8s rules: - apiGroups: - "" resources: - nodes - services - endpoints - pods - nodes/proxy verbs: - get - list - watch - apiGroups: - "" resources: - configmaps - nodes/metrics verbs: - get - nonResourceURLs: - /metrics verbs: - get
書寫monitor監控文件,提供瞭一下參考的文件:prometheus-serviceMonitor.yaml
自己編寫的 :prometheus-serviceMonitorSelfServiceMs.yaml
apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: name: drore-saas-ms namespace: monitoring spec: endpoints: - interval: 15s port: http-ms #prometheus配置註意這個port的名稱 path: /ms/metrics/prometheus #prometheus配置註意這個路徑 jobLabel: app #prometheus配置註意我們用的是app在springboot構建的時候 namespaceSelector: # 表示去匹配某一命名空間中的service,如果想從所有的namespace中匹配用any: true matchNames: - drore-saas selector: matchLabels: app: ms #prometheus配置註意我們用的是app在springboot構建的時候
配置完畢之後使用
kubectl apply -f prometheus-clusterRole.yaml -f prometheus-serviceMonitorSelfServiceMs.yaml
就可以登錄管理界面去看瞭
http://118.31.17.205:31144/targets
報警規則可以通過 prometheus-rules.yaml 去配置
參考
https://github.com/coreos/prometheus-operator
補充:spring boot2.x暴露監控endpoint並配置prometheus及grafana對多個targets進行監控
1. build.gradle中添加依賴
compile group: 'org.springframework.boot', name: 'spring-boot-starter-actuator' compile group: 'org.springframework.boot', name: 'spring-boot-starter-security' compile('io.micrometer:micrometer-registry-prometheus')
2. spring boot 2.x暴露信息
不同於之前的Actuator 1.x,Actuator 2.x 的大多數端點默認被禁掉。 Actuator 2.x 中的默認端點增加瞭/actuator前綴。
3. spring boot服務端改造
配置信息
server.port=8085 management.endpoints.web.exposure.include=prometheus management.endpoint.metrics.enabled=true spring.security.user.name=xxx spring.security.user.password=123456 spring.security.user.roles=ACTUATOR_ADMIN management.server.port=8090 management.endpoints.web.base-path=/admin
指定權限
@Configuration public class ActuatorSecurityConfig extends WebSecurityConfigurerAdapter { @Override protected void configure(HttpSecurity http) throws Exception { http.requestMatcher(EndpointRequest.toAnyEndpoint()).authorizeRequests().antMatchers("/actuator/*").hasRole( "ACTUATOR_ADMIN").and().httpBasic(); } }
這樣我們可以看到:
這個spring boot服務啟動瞭兩個端口:
8085端口,進行正常的業務處理
8090端口,admin,暴露出監控信息,並通過spring security進行權限控制
打開瀏覽器訪問http://192.168.211.2:8090/admin/prometheus
輸入用戶名 密碼
4. Prometheus監控多個spring boot服務
下載prometheus-2.11.1.linux-amd64.tar.gz
編寫配置文件prometheus.yml
global: scrape_interval: 10s scrape_timeout: 10s evaluation_interval: 10m scrape_configs: - job_name: app-gateway scrape_interval: 5s scrape_timeout: 5s metrics_path: /admin/prometheus scheme: http basic_auth: username: root password: 1230456 static_configs: - targets: - 192.168.211.2:8099 - job_name: search scrape_interval: 5s scrape_timeout: 5s metrics_path: /admin/prometheus scheme: http basic_auth: username: root password: 1230456 static_configs: - targets: - 192.168.211.2:8090
其中,我的spring boot服務部署在192.168.211.2上,而prometheus和grafana都部署在192.168.211.101上
配置瞭兩個服務
執行以下命令啟動 ./prometheus –config.file=prometheus.yml &
訪問http://192.168.211.101:9090/targets
可以看到
5. grafana可視化監控多個prometheus targets
下載grafana-6.2.5.linux-amd64.tar.gz
啟動grafana
./grafana-server &
http://192.168.211.101:3000
在dashboard中配置時,要采用這樣的方式
jvm_gc_pause_seconds_count{action=”end of minor GC”,cause=”Metadata GC Threshold”,instance=”192.168.211.2:8090″,job=”search”}
通過instance和job來進行區分prometheus的targets
可以看到dashboard中有兩個prometheus的targets
以上為個人經驗,希望能給大傢一個參考,也希望大傢多多支持WalkonNet。如有錯誤或未考慮完全的地方,望不吝賜教。
推薦閱讀:
- spring boot metrics監控指標使用教程
- SpringBoot Actuator埋點和監控及簡單使用
- SpringBoot使用prometheus監控的示例代碼
- SpringBoot可視化監控的具體應用
- SpringBoot+Prometheus+Grafana實現應用監控和報警的詳細步驟