searchusermenu
  • 发布文章
  • 消息中心
点赞
收藏
评论
分享
原创

k8s 组件观测指标

2024-08-16 09:37:00
24
0

1.运行状态、重启次数、实例个数、cpu/内存占用等基本信息

运行状态、重启次数、实例个数可以通过kube-state-metrics进行采集

这里是部分指标,详细的可以查看官方文档

DaemonSet Metrics

Metric name Metric type Description Labels/tags Status
kube_daemonset_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled  daemonset=<daemonset-name>
namespace=<daemonset-namespace>
annotation_DAEMONSET_ANNOTATION=<DAEMONSET_ANNOTATION>
EXPERIMENTAL
kube_daemonset_created Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_current_number_scheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_desired_number_scheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_available Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_misscheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_ready Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_unavailable Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_observed_generation Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_updated_number_scheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_metadata_generation Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_labels Gauge Kubernetes labels converted to Prometheus labels controlled daemonset=<daemonset-name>
namespace=<daemonset-namespace>
label_DAEMONSET_LABEL=<DAEMONSET_LABEL>
STABLE


Deployment Metrics

Metric name Metric type Description Labels/tags Status
kube_deployment_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled deployment=<deployment-name>
namespace=<deployment-namespace>
annotation_DEPLOYMENT_ANNOTATION=<DEPLOYMENT_ANNOTATION>
EXPERIMENTAL
kube_deployment_status_replicas Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_ready Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_available Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_unavailable Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_updated Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_observed_generation Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_condition Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
condition=<deployment-condition>
status=<true|false|unknown>
STABLE
kube_deployment_spec_replicas Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_spec_paused Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_spec_strategy_rollingupdate_max_unavailable Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_spec_strategy_rollingupdate_max_surge Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_metadata_generation Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_labels Gauge Kubernetes labels converted to Prometheus labels controlled deployment=<deployment-name>
namespace=<deployment-namespace>
label_DEPLOYMENT_LABEL=<DEPLOYMENT_LABEL>
STABLE
kube_deployment_created Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE

Stateful Set Metrics

Metric name Metric type Description Labels/tags Status
kube_statefulset_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled  statefulset=<statefulset-name>
namespace=<statefulset-namespace>
annotation_STATEFULSET_ANNOTATION=<STATEFULSET_ANNOTATION>
EXPERIMENTAL
kube_statefulset_status_replicas Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_replicas_current Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_replicas_ready Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_replicas_available Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
EXPERIMENTAL
kube_statefulset_status_replicas_updated Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_observed_generation Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_replicas Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_ordinals_start Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
ALPHA
kube_statefulset_metadata_generation Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_persistentvolumeclaim_retention_policy Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
when_deleted=<statefulset-when-deleted-pvc-policy>
when_scaled=<statefulset-when-scaled-pvc-policy>
EXPERIMENTAL
kube_statefulset_created Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_labels Gauge Kubernetes labels converted to Prometheus labels controlled  statefulset=<statefulset-name>
namespace=<statefulset-namespace>
label_STATEFULSET_LABEL=<STATEFULSET_LABEL>
STABLE
kube_statefulset_status_current_revision Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
revision=<statefulset-current-revision>
STABLE
kube_statefulset_status_update_revision Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
revision=<statefulset-update-revision>
STABLE

实例个数:通过DaemenSet, Deployment, Statefulset 指标获取

deployment可以参考:

kube_deployment_status_replicas  #表示实例总数量

kube_deployment_status_replicas_available# 可用的实例数量

kube_deployment_status_replicas_unavailable#  不可用的实例数量

DaemenSet可以参考:

kube_daemonset_status_desired_number_scheduled  #表示实例总数量

kube_daemonset_status_number_available# 可用的实例数量

kube_daemonset_status_number_unavailable#  不可用的实例数量

Statefulset可以参考:

kube_statefulset_status_replicas  #表示实例总数量

kube_statefulset_status_replicas_ready# 可用的实例数量

 

 

Pod Metrics

Metric name Metric type Description Unit (where applicable) Labels/tags Status Opt-in
kube_pod_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled   pod=<pod-name>
namespace=<pod-namespace>
annotation_POD_ANNOTATION=<POD_ANNOTATION>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_info Gauge Information about pod   pod=<pod-name>
namespace=<pod-namespace>
host_ip=<host-ip>
pod_ip=<pod-ip>
node=<node-name>
created_by_kind=<created_by_kind>
created_by_name=<created_by_name>
uid=<pod-uid>
priority_class=<priority_class>
host_network=<host_network>
STABLE -
kube_pod_ips Gauge Pod IP addresses   pod=<pod-name>
namespace=<pod-namespace>
ip=<pod-ip-address>
ip_family=<4 OR 6>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_start_time Gauge Start time in unix timestamp for a pod seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_completion_time Gauge Completion time in unix timestamp for a pod seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_owner Gauge Information about the Pod's owner   pod=<pod-name>
namespace=<pod-namespace>
owner_kind=<owner kind>
owner_name=<owner name>
owner_is_controller=<whether owner is controller>
uid=<pod-uid>
STABLE -
kube_pod_labels Gauge Kubernetes labels converted to Prometheus labels controlled    pod=<pod-name>
namespace=<pod-namespace>
label_POD_LABEL=<POD_LABEL>
uid=<pod-uid>
STABLE -
kube_pod_nodeselectors Gauge Describes the Pod nodeSelectors   pod=<pod-name>
namespace=<pod-namespace>
nodeselector_NODE_SELECTOR=<NODE_SELECTOR>
uid=<pod-uid>
EXPERIMENTAL Opt-in
kube_pod_status_phase Gauge The pods current phase   pod=<pod-name>
namespace=<pod-namespace>
phase=<Pending|Running|Succeeded|Failed|Unknown>
uid=<pod-uid>
STABLE -
kube_pod_status_qos_class Gauge The pods current qosClass   pod=<pod-name>
namespace=<pod-namespace>
qos_class=<BestEffort|Burstable|Guaranteed>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_ready Gauge Describes whether the pod is ready to serve requests   pod=<pod-name>
namespace=<pod-namespace>
condition=<true|false|unknown>
uid=<pod-uid>
STABLE -
kube_pod_status_scheduled Gauge Describes the status of the scheduling process for the pod   pod=<pod-name>
namespace=<pod-namespace>
condition=<true|false|unknown>
uid=<pod-uid>
STABLE -
kube_pod_container_info Gauge Information about a container in a pod   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
image=<image-name>
image_id=<image-id>
image_spec=<image-spec>
container_id=<containerid>
uid=<pod-uid>
STABLE -
kube_pod_container_status_waiting Gauge Describes whether the container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_status_waiting_reason Gauge Describes the reason the container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-waiting-reason>
uid=<pod-uid>
STABLE -
kube_pod_container_status_running Gauge Describes whether the container is currently in running state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_state_started Gauge Start time in unix timestamp for a pod container seconds container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_status_terminated Gauge Describes whether the container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_status_terminated_reason Gauge Describes the reason the container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_last_terminated_reason Gauge Describes the last reason the container was in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<last-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_last_terminated_exitcode Gauge Describes the exit code for the last container in terminated state.   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_last_terminated_timestamp Gauge Last terminated time for a pod container in unix timestamp.   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_ready Gauge Describes whether the containers readiness check succeeded   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_status_initialized_time Gauge Time when the pod is initialized. seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_ready_time Gauge Time when pod passed readiness probes. seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_container_ready_time Gauge Time when the container of the pod entered Ready state. seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_restarts_total Counter The number of container restarts per container   container=<container-name>
namespace=<pod-namespace>
pod=<pod-name>
uid=<pod-uid>
STABLE -
kube_pod_container_resource_requests Gauge The number of requested request resource by a container. It is recommended to use the kube_pod_resource_requests metric exposed by kube-scheduler instead, as it is more precise. cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_resource_limits Gauge The number of requested limit resource by a container. It is recommended to use the kube_pod_resource_limits metric exposed by kube-scheduler instead, as it is more precise. cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_overhead_cpu_cores Gauge The pod overhead in regards to cpu cores associated with running a pod core pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_overhead_memory_bytes Gauge The pod overhead in regards to memory associated with running a pod bytes pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_runtimeclass_name_info Gauge The runtimeclass associated with the pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_created Gauge Unix creation timestamp seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_deletion_timestamp Gauge Unix deletion timestamp seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_restart_policy Gauge Describes the restart policy in use by this pod   pod=<pod-name>
namespace=<pod-namespace>
type=<Always|Never|OnFailure>
uid=<pod-uid>
STABLE -
kube_pod_init_container_info Gauge Information about an init container in a pod   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
image=<image-name>
image_id=<image-id>
image_spec=<image-spec>
container_id=<containerid>
uid=<pod-uid>
restart_policy=<restart-policy>
STABLE -
kube_pod_init_container_status_waiting Gauge Describes whether the init container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_waiting_reason Gauge Describes the reason the init container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-waiting-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_status_running Gauge Describes whether the init container is currently in running state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_terminated Gauge Describes whether the init container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_terminated_reason Gauge Describes the reason the init container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_status_last_terminated_reason Gauge Describes the last reason the init container was in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<last-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_status_ready Gauge Describes whether the init containers readiness check succeeded   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_restarts_total Counter The number of restarts for the init container integer container=<container-name>
namespace=<pod-namespace>
pod=<pod-name>
uid=<pod-uid>
STABLE -
kube_pod_init_container_resource_limits Gauge The number of CPU cores requested limit by an init container cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_resource_requests Gauge The number of CPU cores requested by an init container cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_spec_volumes_persistentvolumeclaims_info Gauge Information about persistentvolumeclaim volumes in a pod   pod=<pod-name>
namespace=<pod-namespace>
volume=<volume-name>
persistentvolumeclaim=<persistentvolumeclaim-claimname>
uid=<pod-uid>
STABLE -
kube_pod_spec_volumes_persistentvolumeclaims_readonly Gauge Describes whether a persistentvolumeclaim is mounted read only bool pod=<pod-name>
namespace=<pod-namespace>
volume=<volume-name>
persistentvolumeclaim=<persistentvolumeclaim-claimname>
uid=<pod-uid>
STABLE -
kube_pod_status_reason Gauge The pod status reasons   pod=<pod-name>
namespace=<pod-namespace>
reason=<Evicted|NodeAffinity|NodeLost|Shutdown|UnexpectedAdmissionError>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_scheduled_time Gauge Unix timestamp when pod moved into scheduled status seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_status_unschedulable Gauge Describes the unschedulable status for the pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_tolerations Gauge Information about the pod tolerations   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
key=<toleration-key>
operator=<toleration-operator>
value=<toleration-value>
effect=<toleration-effect> toleration_seconds=<toleration-seconds>
EXPERIMENTAL -
kube_pod_service_account Gauge The service account for a pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
service_account=<service_account>
EXPERIMENTAL -
kube_pod_scheduler Gauge The scheduler for a pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
name=<scheduler-name>

运行状态、重启次数:通过pod指标获取

可以参考:

kube_pod_status_*   #pod的状态

kube_pod_container_status_restarts_total  #重启次数

 

 

cpu/内存占用通过kubelet进行采集

cadvisor插件,详细可以查看官方文档

这里贴了部分软件指标

Metric name Type Description Unit (where applicable) option parameter additional build flag
container_blkio_device_usage_total Counter Blkio device bytes usage bytes diskIO  
container_cpu_cfs_periods_total Counter Number of elapsed enforcement period intervals   cpu  
container_cpu_cfs_throttled_periods_total Counter Number of throttled period intervals   cpu  
container_cpu_cfs_throttled_seconds_total Counter Total time duration the container has been throttled seconds cpu  
container_cpu_load_average_10s Gauge Value of container cpu load average over the last 10 seconds   cpuLoad  
container_cpu_schedstat_run_periods_total Counter Number of times processes of the cgroup have run on the cpu   sched  
container_cpu_schedstat_runqueue_seconds_total Counter Time duration processes of the container have been waiting on a runqueue seconds sched  
container_cpu_schedstat_run_seconds_total Counter Time duration the processes of the container have run on the CPU seconds sched  
container_cpu_system_seconds_total Counter Cumulative system cpu time consumed seconds cpu  
container_cpu_usage_seconds_total Counter Cumulative cpu time consumed seconds cpu  
container_cpu_user_seconds_total Counter Cumulative user cpu time consumed seconds cpu  
container_file_descriptors Gauge Number of open file descriptors for the container   process  
container_fs_inodes_free Gauge Number of available Inodes   disk  
container_fs_inodes_total Gauge Total number of Inodes   disk  
container_fs_io_current Gauge Number of I/Os currently in progress   diskIO  
container_fs_io_time_seconds_total Counter Cumulative count of seconds spent doing I/Os seconds diskIO  
container_fs_io_time_weighted_seconds_total Counter Cumulative weighted I/O time seconds diskIO  
container_fs_limit_bytes Gauge Number of bytes that can be consumed by the container on this filesystem bytes disk  
container_fs_reads_bytes_total Counter Cumulative count of bytes read bytes diskIO  
container_fs_read_seconds_total Counter Cumulative count of seconds spent reading   diskIO  
container_fs_reads_merged_total Counter Cumulative count of reads merged   diskIO  
container_fs_reads_total Counter Cumulative count of reads completed   diskIO  
container_fs_sector_reads_total Counter Cumulative count of sector reads completed   diskIO  
container_fs_sector_writes_total Counter Cumulative count of sector writes completed   diskIO  
container_fs_usage_bytes Gauge Number of bytes that are consumed by the container on this filesystem bytes disk  
container_fs_writes_bytes_total Counter Cumulative count of bytes written bytes diskIO  
container_fs_write_seconds_total Counter Cumulative count of seconds spent writing seconds diskIO  
container_fs_writes_merged_total Counter Cumulative count of writes merged   diskIO  
container_fs_writes_total Counter Cumulative count of writes completed   diskIO  
container_hugetlb_failcnt Counter Number of hugepage usage hits limits   hugetlb  
container_hugetlb_max_usage_bytes Gauge Maximum hugepage usages recorded bytes hugetlb  
container_hugetlb_usage_bytes Gauge Current hugepage usage bytes hugetlb  
container_last_seen Gauge Last time a container was seen by the exporter timestamp -  
container_llc_occupancy_bytes Gauge Last level cache usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM). bytes resctrl  
container_memory_bandwidth_bytes Gauge Total memory bandwidth usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM). bytes resctrl  
container_memory_bandwidth_local_bytes Gauge Local memory bandwidth usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM). bytes resctrl  
container_memory_cache Gauge Total page cache memory bytes memory  
container_memory_failcnt Counter Number of memory usage hits limits   memory  
container_memory_failures_total Counter Cumulative count of memory allocation failures   memory  
container_memory_mapped_file Gauge Size of memory mapped files bytes memory  
container_memory_max_usage_bytes Gauge Maximum memory usage recorded bytes memory  
container_memory_migrate Gauge Memory migrate status   cpuset  
container_memory_numa_pages Gauge Number of used pages per NUMA node   memory_numa  
container_memory_rss Gauge Size of RSS bytes memory  
container_memory_swap Gauge Container swap usage bytes memory  
container_memory_usage_bytes Gauge Current memory usage, including all memory regardless of when it was accessed bytes memory  
container_memory_working_set_bytes Gauge Current working set bytes memory  
container_network_advance_tcp_stats_total Gauge advanced tcp connections statistic for container   advtcp  
container_network_receive_bytes_total Counter Cumulative count of bytes received bytes network  
container_network_receive_errors_total Counter Cumulative count of errors encountered while receiving   network  
container_network_receive_packets_dropped_total Counter Cumulative count of packets dropped while receiving   network  
container_network_receive_packets_total Counter Cumulative count of packets received   network  
container_network_tcp6_usage_total Gauge tcp6 connection usage statistic for container   tcp  
container_network_tcp_usage_total Gauge tcp connection usage statistic for container   tcp  
container_network_transmit_bytes_total Counter Cumulative count of bytes transmitted bytes network  
container_network_transmit_errors_total Counter Cumulative count of errors encountered while transmitting   network  
container_network_transmit_packets_dropped_total Counter Cumulative count of packets dropped while transmitting   network  
container_network_transmit_packets_total Counter Cumulative count of packets transmitted   network  
container_network_udp6_usage_total Gauge udp6 connection usage statistic for container   udp  
container_network_udp_usage_total Gauge udp connection usage statistic for container   udp  
container_oom_events_total Counter Count of out of memory events observed for the container   oom_event  
container_perf_events_scaling_ratio Gauge Scaling ratio for perf event counter (event can be identified by event label and cpu indicates the core for which event was measured).    perf_event libpfm
container_perf_events_total Counter Scaled counter of perf core event (event can be identified by event label and cpu indicates the core for which event was measured).   perf_event libpfm
container_perf_uncore_events_scaling_ratio Gauge Scaling ratio for perf uncore event counter (event can be identified by event label, pmu and socket lables indicate the PMU and the CPU socket for which event was measured).  Metric exists only for main cgroup (id="/").   perf_event libpfm
container_perf_uncore_events_total Counter Scaled counter of perf uncore event (event can be identified by event label, pmu and socket lables indicate the PMU and the CPU socket for which event was measured).  Metric exists only for main cgroup (id="/").   perf_event libpfm
container_processes Gauge Number of processes running inside the container   process  
container_referenced_bytes Gauge Container referenced bytes during last measurements cycle based on Referenced field in /proc/smaps file, with /proc/PIDs/clear_refs set to 1 after defined number of cycles configured through referenced_reset_interval cAdvisor parameter.
Warning: this is intrusive collection because can influence kernel page reclaim policy and add latency. 
bytes referenced_memory  
container_sockets Gauge Number of open sockets for the container   process  
container_spec_cpu_period Gauge CPU period of the container   -  
container_spec_cpu_quota Gauge CPU quota of the container   -  
container_spec_cpu_shares Gauge CPU share of the container   -  
container_spec_memory_limit_bytes Gauge Memory limit for the container bytes -  
container_spec_memory_reservation_limit_bytes Gauge Memory reservation limit for the container bytes    
container_spec_memory_swap_limit_bytes Gauge Memory swap limit for the container bytes    
container_start_time_seconds Gauge Start time of the container since unix epoch seconds    
container_tasks_state Gauge Number of tasks in given state (sleepingrunningstoppeduninterruptible, or ioawaiting)   cpuLoad  
container_threads Gauge Number of threads running inside the container   process  
container_threads_max Gauge Maximum number of threads allowed inside the container   process  
container_ulimits_soft Gauge Soft ulimit values for the container root process. Unlimited if -1, except priority and nice   process

例如cpu相关指标:

container_cpu_load_average_10s

container_cpu_system_seconds_total

container_cpu_usage_seconds_total

container_cpu_user_seconds_total

 

内存相关可以参考:

container_memory_max_usage_bytes

container_memory_usage_bytes

 

0条评论
作者已关闭评论
技术分享
4文章数
0粉丝数
技术分享
4 文章 | 0 粉丝
技术分享
4文章数
0粉丝数
技术分享
4 文章 | 0 粉丝
原创

k8s 组件观测指标

2024-08-16 09:37:00
24
0

1.运行状态、重启次数、实例个数、cpu/内存占用等基本信息

运行状态、重启次数、实例个数可以通过kube-state-metrics进行采集

这里是部分指标,详细的可以查看官方文档

DaemonSet Metrics

Metric name Metric type Description Labels/tags Status
kube_daemonset_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled  daemonset=<daemonset-name>
namespace=<daemonset-namespace>
annotation_DAEMONSET_ANNOTATION=<DAEMONSET_ANNOTATION>
EXPERIMENTAL
kube_daemonset_created Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_current_number_scheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_desired_number_scheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_available Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_misscheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_ready Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_number_unavailable Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_observed_generation Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_status_updated_number_scheduled Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_metadata_generation Gauge   daemonset=<daemonset-name>
namespace=<daemonset-namespace>
STABLE
kube_daemonset_labels Gauge Kubernetes labels converted to Prometheus labels controlled daemonset=<daemonset-name>
namespace=<daemonset-namespace>
label_DAEMONSET_LABEL=<DAEMONSET_LABEL>
STABLE


Deployment Metrics

Metric name Metric type Description Labels/tags Status
kube_deployment_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled deployment=<deployment-name>
namespace=<deployment-namespace>
annotation_DEPLOYMENT_ANNOTATION=<DEPLOYMENT_ANNOTATION>
EXPERIMENTAL
kube_deployment_status_replicas Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_ready Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_available Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_unavailable Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_replicas_updated Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_observed_generation Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_status_condition Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
condition=<deployment-condition>
status=<true|false|unknown>
STABLE
kube_deployment_spec_replicas Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_spec_paused Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_spec_strategy_rollingupdate_max_unavailable Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_spec_strategy_rollingupdate_max_surge Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_metadata_generation Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE
kube_deployment_labels Gauge Kubernetes labels converted to Prometheus labels controlled deployment=<deployment-name>
namespace=<deployment-namespace>
label_DEPLOYMENT_LABEL=<DEPLOYMENT_LABEL>
STABLE
kube_deployment_created Gauge   deployment=<deployment-name>
namespace=<deployment-namespace>
STABLE

Stateful Set Metrics

Metric name Metric type Description Labels/tags Status
kube_statefulset_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled  statefulset=<statefulset-name>
namespace=<statefulset-namespace>
annotation_STATEFULSET_ANNOTATION=<STATEFULSET_ANNOTATION>
EXPERIMENTAL
kube_statefulset_status_replicas Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_replicas_current Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_replicas_ready Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_replicas_available Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
EXPERIMENTAL
kube_statefulset_status_replicas_updated Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_status_observed_generation Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_replicas Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_ordinals_start Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
ALPHA
kube_statefulset_metadata_generation Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_persistentvolumeclaim_retention_policy Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
when_deleted=<statefulset-when-deleted-pvc-policy>
when_scaled=<statefulset-when-scaled-pvc-policy>
EXPERIMENTAL
kube_statefulset_created Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
STABLE
kube_statefulset_labels Gauge Kubernetes labels converted to Prometheus labels controlled  statefulset=<statefulset-name>
namespace=<statefulset-namespace>
label_STATEFULSET_LABEL=<STATEFULSET_LABEL>
STABLE
kube_statefulset_status_current_revision Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
revision=<statefulset-current-revision>
STABLE
kube_statefulset_status_update_revision Gauge   statefulset=<statefulset-name>
namespace=<statefulset-namespace>
revision=<statefulset-update-revision>
STABLE

实例个数:通过DaemenSet, Deployment, Statefulset 指标获取

deployment可以参考:

kube_deployment_status_replicas  #表示实例总数量

kube_deployment_status_replicas_available# 可用的实例数量

kube_deployment_status_replicas_unavailable#  不可用的实例数量

DaemenSet可以参考:

kube_daemonset_status_desired_number_scheduled  #表示实例总数量

kube_daemonset_status_number_available# 可用的实例数量

kube_daemonset_status_number_unavailable#  不可用的实例数量

Statefulset可以参考:

kube_statefulset_status_replicas  #表示实例总数量

kube_statefulset_status_replicas_ready# 可用的实例数量

 

 

Pod Metrics

Metric name Metric type Description Unit (where applicable) Labels/tags Status Opt-in
kube_pod_annotations Gauge Kubernetes annotations converted to Prometheus labels controlled   pod=<pod-name>
namespace=<pod-namespace>
annotation_POD_ANNOTATION=<POD_ANNOTATION>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_info Gauge Information about pod   pod=<pod-name>
namespace=<pod-namespace>
host_ip=<host-ip>
pod_ip=<pod-ip>
node=<node-name>
created_by_kind=<created_by_kind>
created_by_name=<created_by_name>
uid=<pod-uid>
priority_class=<priority_class>
host_network=<host_network>
STABLE -
kube_pod_ips Gauge Pod IP addresses   pod=<pod-name>
namespace=<pod-namespace>
ip=<pod-ip-address>
ip_family=<4 OR 6>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_start_time Gauge Start time in unix timestamp for a pod seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_completion_time Gauge Completion time in unix timestamp for a pod seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_owner Gauge Information about the Pod's owner   pod=<pod-name>
namespace=<pod-namespace>
owner_kind=<owner kind>
owner_name=<owner name>
owner_is_controller=<whether owner is controller>
uid=<pod-uid>
STABLE -
kube_pod_labels Gauge Kubernetes labels converted to Prometheus labels controlled    pod=<pod-name>
namespace=<pod-namespace>
label_POD_LABEL=<POD_LABEL>
uid=<pod-uid>
STABLE -
kube_pod_nodeselectors Gauge Describes the Pod nodeSelectors   pod=<pod-name>
namespace=<pod-namespace>
nodeselector_NODE_SELECTOR=<NODE_SELECTOR>
uid=<pod-uid>
EXPERIMENTAL Opt-in
kube_pod_status_phase Gauge The pods current phase   pod=<pod-name>
namespace=<pod-namespace>
phase=<Pending|Running|Succeeded|Failed|Unknown>
uid=<pod-uid>
STABLE -
kube_pod_status_qos_class Gauge The pods current qosClass   pod=<pod-name>
namespace=<pod-namespace>
qos_class=<BestEffort|Burstable|Guaranteed>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_ready Gauge Describes whether the pod is ready to serve requests   pod=<pod-name>
namespace=<pod-namespace>
condition=<true|false|unknown>
uid=<pod-uid>
STABLE -
kube_pod_status_scheduled Gauge Describes the status of the scheduling process for the pod   pod=<pod-name>
namespace=<pod-namespace>
condition=<true|false|unknown>
uid=<pod-uid>
STABLE -
kube_pod_container_info Gauge Information about a container in a pod   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
image=<image-name>
image_id=<image-id>
image_spec=<image-spec>
container_id=<containerid>
uid=<pod-uid>
STABLE -
kube_pod_container_status_waiting Gauge Describes whether the container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_status_waiting_reason Gauge Describes the reason the container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-waiting-reason>
uid=<pod-uid>
STABLE -
kube_pod_container_status_running Gauge Describes whether the container is currently in running state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_state_started Gauge Start time in unix timestamp for a pod container seconds container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_status_terminated Gauge Describes whether the container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_container_status_terminated_reason Gauge Describes the reason the container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_last_terminated_reason Gauge Describes the last reason the container was in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<last-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_last_terminated_exitcode Gauge Describes the exit code for the last container in terminated state.   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_last_terminated_timestamp Gauge Last terminated time for a pod container in unix timestamp.   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_ready Gauge Describes whether the containers readiness check succeeded   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_status_initialized_time Gauge Time when the pod is initialized. seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_ready_time Gauge Time when pod passed readiness probes. seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_container_ready_time Gauge Time when the container of the pod entered Ready state. seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_status_restarts_total Counter The number of container restarts per container   container=<container-name>
namespace=<pod-namespace>
pod=<pod-name>
uid=<pod-uid>
STABLE -
kube_pod_container_resource_requests Gauge The number of requested request resource by a container. It is recommended to use the kube_pod_resource_requests metric exposed by kube-scheduler instead, as it is more precise. cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_container_resource_limits Gauge The number of requested limit resource by a container. It is recommended to use the kube_pod_resource_limits metric exposed by kube-scheduler instead, as it is more precise. cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_overhead_cpu_cores Gauge The pod overhead in regards to cpu cores associated with running a pod core pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_overhead_memory_bytes Gauge The pod overhead in regards to memory associated with running a pod bytes pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_runtimeclass_name_info Gauge The runtimeclass associated with the pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_created Gauge Unix creation timestamp seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_deletion_timestamp Gauge Unix deletion timestamp seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_restart_policy Gauge Describes the restart policy in use by this pod   pod=<pod-name>
namespace=<pod-namespace>
type=<Always|Never|OnFailure>
uid=<pod-uid>
STABLE -
kube_pod_init_container_info Gauge Information about an init container in a pod   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
image=<image-name>
image_id=<image-id>
image_spec=<image-spec>
container_id=<containerid>
uid=<pod-uid>
restart_policy=<restart-policy>
STABLE -
kube_pod_init_container_status_waiting Gauge Describes whether the init container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_waiting_reason Gauge Describes the reason the init container is currently in waiting state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-waiting-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_status_running Gauge Describes whether the init container is currently in running state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_terminated Gauge Describes whether the init container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_terminated_reason Gauge Describes the reason the init container is currently in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<container-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_status_last_terminated_reason Gauge Describes the last reason the init container was in terminated state   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
reason=<last-terminated-reason>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_status_ready Gauge Describes whether the init containers readiness check succeeded   container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_init_container_status_restarts_total Counter The number of restarts for the init container integer container=<container-name>
namespace=<pod-namespace>
pod=<pod-name>
uid=<pod-uid>
STABLE -
kube_pod_init_container_resource_limits Gauge The number of CPU cores requested limit by an init container cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_init_container_resource_requests Gauge The number of CPU cores requested by an init container cpu=<core>
memory=<bytes>
resource=<resource-name>
unit=<resource-unit>
container=<container-name>
pod=<pod-name>
namespace=<pod-namespace>
node=< node-name>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_spec_volumes_persistentvolumeclaims_info Gauge Information about persistentvolumeclaim volumes in a pod   pod=<pod-name>
namespace=<pod-namespace>
volume=<volume-name>
persistentvolumeclaim=<persistentvolumeclaim-claimname>
uid=<pod-uid>
STABLE -
kube_pod_spec_volumes_persistentvolumeclaims_readonly Gauge Describes whether a persistentvolumeclaim is mounted read only bool pod=<pod-name>
namespace=<pod-namespace>
volume=<volume-name>
persistentvolumeclaim=<persistentvolumeclaim-claimname>
uid=<pod-uid>
STABLE -
kube_pod_status_reason Gauge The pod status reasons   pod=<pod-name>
namespace=<pod-namespace>
reason=<Evicted|NodeAffinity|NodeLost|Shutdown|UnexpectedAdmissionError>
uid=<pod-uid>
EXPERIMENTAL -
kube_pod_status_scheduled_time Gauge Unix timestamp when pod moved into scheduled status seconds pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_status_unschedulable Gauge Describes the unschedulable status for the pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
STABLE -
kube_pod_tolerations Gauge Information about the pod tolerations   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
key=<toleration-key>
operator=<toleration-operator>
value=<toleration-value>
effect=<toleration-effect> toleration_seconds=<toleration-seconds>
EXPERIMENTAL -
kube_pod_service_account Gauge The service account for a pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
service_account=<service_account>
EXPERIMENTAL -
kube_pod_scheduler Gauge The scheduler for a pod   pod=<pod-name>
namespace=<pod-namespace>
uid=<pod-uid>
name=<scheduler-name>

运行状态、重启次数:通过pod指标获取

可以参考:

kube_pod_status_*   #pod的状态

kube_pod_container_status_restarts_total  #重启次数

 

 

cpu/内存占用通过kubelet进行采集

cadvisor插件,详细可以查看官方文档

这里贴了部分软件指标

Metric name Type Description Unit (where applicable) option parameter additional build flag
container_blkio_device_usage_total Counter Blkio device bytes usage bytes diskIO  
container_cpu_cfs_periods_total Counter Number of elapsed enforcement period intervals   cpu  
container_cpu_cfs_throttled_periods_total Counter Number of throttled period intervals   cpu  
container_cpu_cfs_throttled_seconds_total Counter Total time duration the container has been throttled seconds cpu  
container_cpu_load_average_10s Gauge Value of container cpu load average over the last 10 seconds   cpuLoad  
container_cpu_schedstat_run_periods_total Counter Number of times processes of the cgroup have run on the cpu   sched  
container_cpu_schedstat_runqueue_seconds_total Counter Time duration processes of the container have been waiting on a runqueue seconds sched  
container_cpu_schedstat_run_seconds_total Counter Time duration the processes of the container have run on the CPU seconds sched  
container_cpu_system_seconds_total Counter Cumulative system cpu time consumed seconds cpu  
container_cpu_usage_seconds_total Counter Cumulative cpu time consumed seconds cpu  
container_cpu_user_seconds_total Counter Cumulative user cpu time consumed seconds cpu  
container_file_descriptors Gauge Number of open file descriptors for the container   process  
container_fs_inodes_free Gauge Number of available Inodes   disk  
container_fs_inodes_total Gauge Total number of Inodes   disk  
container_fs_io_current Gauge Number of I/Os currently in progress   diskIO  
container_fs_io_time_seconds_total Counter Cumulative count of seconds spent doing I/Os seconds diskIO  
container_fs_io_time_weighted_seconds_total Counter Cumulative weighted I/O time seconds diskIO  
container_fs_limit_bytes Gauge Number of bytes that can be consumed by the container on this filesystem bytes disk  
container_fs_reads_bytes_total Counter Cumulative count of bytes read bytes diskIO  
container_fs_read_seconds_total Counter Cumulative count of seconds spent reading   diskIO  
container_fs_reads_merged_total Counter Cumulative count of reads merged   diskIO  
container_fs_reads_total Counter Cumulative count of reads completed   diskIO  
container_fs_sector_reads_total Counter Cumulative count of sector reads completed   diskIO  
container_fs_sector_writes_total Counter Cumulative count of sector writes completed   diskIO  
container_fs_usage_bytes Gauge Number of bytes that are consumed by the container on this filesystem bytes disk  
container_fs_writes_bytes_total Counter Cumulative count of bytes written bytes diskIO  
container_fs_write_seconds_total Counter Cumulative count of seconds spent writing seconds diskIO  
container_fs_writes_merged_total Counter Cumulative count of writes merged   diskIO  
container_fs_writes_total Counter Cumulative count of writes completed   diskIO  
container_hugetlb_failcnt Counter Number of hugepage usage hits limits   hugetlb  
container_hugetlb_max_usage_bytes Gauge Maximum hugepage usages recorded bytes hugetlb  
container_hugetlb_usage_bytes Gauge Current hugepage usage bytes hugetlb  
container_last_seen Gauge Last time a container was seen by the exporter timestamp -  
container_llc_occupancy_bytes Gauge Last level cache usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM). bytes resctrl  
container_memory_bandwidth_bytes Gauge Total memory bandwidth usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM). bytes resctrl  
container_memory_bandwidth_local_bytes Gauge Local memory bandwidth usage statistics for container counted with RDT Memory Bandwidth Monitoring (MBM). bytes resctrl  
container_memory_cache Gauge Total page cache memory bytes memory  
container_memory_failcnt Counter Number of memory usage hits limits   memory  
container_memory_failures_total Counter Cumulative count of memory allocation failures   memory  
container_memory_mapped_file Gauge Size of memory mapped files bytes memory  
container_memory_max_usage_bytes Gauge Maximum memory usage recorded bytes memory  
container_memory_migrate Gauge Memory migrate status   cpuset  
container_memory_numa_pages Gauge Number of used pages per NUMA node   memory_numa  
container_memory_rss Gauge Size of RSS bytes memory  
container_memory_swap Gauge Container swap usage bytes memory  
container_memory_usage_bytes Gauge Current memory usage, including all memory regardless of when it was accessed bytes memory  
container_memory_working_set_bytes Gauge Current working set bytes memory  
container_network_advance_tcp_stats_total Gauge advanced tcp connections statistic for container   advtcp  
container_network_receive_bytes_total Counter Cumulative count of bytes received bytes network  
container_network_receive_errors_total Counter Cumulative count of errors encountered while receiving   network  
container_network_receive_packets_dropped_total Counter Cumulative count of packets dropped while receiving   network  
container_network_receive_packets_total Counter Cumulative count of packets received   network  
container_network_tcp6_usage_total Gauge tcp6 connection usage statistic for container   tcp  
container_network_tcp_usage_total Gauge tcp connection usage statistic for container   tcp  
container_network_transmit_bytes_total Counter Cumulative count of bytes transmitted bytes network  
container_network_transmit_errors_total Counter Cumulative count of errors encountered while transmitting   network  
container_network_transmit_packets_dropped_total Counter Cumulative count of packets dropped while transmitting   network  
container_network_transmit_packets_total Counter Cumulative count of packets transmitted   network  
container_network_udp6_usage_total Gauge udp6 connection usage statistic for container   udp  
container_network_udp_usage_total Gauge udp connection usage statistic for container   udp  
container_oom_events_total Counter Count of out of memory events observed for the container   oom_event  
container_perf_events_scaling_ratio Gauge Scaling ratio for perf event counter (event can be identified by event label and cpu indicates the core for which event was measured).    perf_event libpfm
container_perf_events_total Counter Scaled counter of perf core event (event can be identified by event label and cpu indicates the core for which event was measured).   perf_event libpfm
container_perf_uncore_events_scaling_ratio Gauge Scaling ratio for perf uncore event counter (event can be identified by event label, pmu and socket lables indicate the PMU and the CPU socket for which event was measured).  Metric exists only for main cgroup (id="/").   perf_event libpfm
container_perf_uncore_events_total Counter Scaled counter of perf uncore event (event can be identified by event label, pmu and socket lables indicate the PMU and the CPU socket for which event was measured).  Metric exists only for main cgroup (id="/").   perf_event libpfm
container_processes Gauge Number of processes running inside the container   process  
container_referenced_bytes Gauge Container referenced bytes during last measurements cycle based on Referenced field in /proc/smaps file, with /proc/PIDs/clear_refs set to 1 after defined number of cycles configured through referenced_reset_interval cAdvisor parameter.
Warning: this is intrusive collection because can influence kernel page reclaim policy and add latency. 
bytes referenced_memory  
container_sockets Gauge Number of open sockets for the container   process  
container_spec_cpu_period Gauge CPU period of the container   -  
container_spec_cpu_quota Gauge CPU quota of the container   -  
container_spec_cpu_shares Gauge CPU share of the container   -  
container_spec_memory_limit_bytes Gauge Memory limit for the container bytes -  
container_spec_memory_reservation_limit_bytes Gauge Memory reservation limit for the container bytes    
container_spec_memory_swap_limit_bytes Gauge Memory swap limit for the container bytes    
container_start_time_seconds Gauge Start time of the container since unix epoch seconds    
container_tasks_state Gauge Number of tasks in given state (sleepingrunningstoppeduninterruptible, or ioawaiting)   cpuLoad  
container_threads Gauge Number of threads running inside the container   process  
container_threads_max Gauge Maximum number of threads allowed inside the container   process  
container_ulimits_soft Gauge Soft ulimit values for the container root process. Unlimited if -1, except priority and nice   process

例如cpu相关指标:

container_cpu_load_average_10s

container_cpu_system_seconds_total

container_cpu_usage_seconds_total

container_cpu_user_seconds_total

 

内存相关可以参考:

container_memory_max_usage_bytes

container_memory_usage_bytes

 

文章来自个人专栏
监控指标
1 文章 | 1 订阅
0条评论
作者已关闭评论
作者已关闭评论
0
0