Skip to content
share

Kubernetes Metrics

Kubernetes Metrics

Sematext Agent seamlessly integrates with Kubernetes and cooperates with its leader election mechanism to elect one instance of the agent that is responsible for querying the API server and gathering the metrics. The list of supported metrics is summarized in the next section.

Cluster Metrics

Name Type Unit Description
kubernetes.cluster.pod.count gauge number of pods in the cluster
kubernetes.cluster.deployment.count gauge number of deployments in the cluster
kubernetes.cluster.node.count gauge number of node comprising the cluster

Pod Metrics

Name Type Unit Description
kubernetes.pod.restarts counter long number of pod restarts
kubernetes.pod.container.count gauge long number of containers inside pod
kubernetes.pod.count gauge long pod count which is always equal to one
kubernetes.pod.count.succeeded gauge long equal to one if all containers inside pod have terminated in success
kubernetes.pod.count.failed gauge long equal to one if all containers inside pod have terminated and at least one container has terminated in failure
kubernetes.pod.count.unknown gauge long equal to one if pod state can't be obtained
kubernetes.pod.count.pending gauge long equal to one if the pod has been accepted by the scheduler and his containers are waiting to be created
kubernetes.pod.count.running gauge long equal to one if the pod has been scheduled on a node and at least one of his containers is running

Deployment Metrics

Name Type Unit Label Description
kubernetes.deployment.count gauge deployment count number of active deployments
kubernetes.deployment.replicas gauge replica count number of active replicas
kubernetes.deployment.replicas.avail gauge number of available replicas. Replicas are marked as available if they are passing the health check
kubernetes.deployment.replicas.desired gauge number of desired replicas as defined in the deployment

DaemonSet Metrics

Name Type Unit Description
kubernetes.daemonset.number.available gauge long number of nodes that should be running the daemon pod
kubernetes.daemonset.number.misscheduled gauge long number of nodes that are not supposed to run the daemon pod
kubernetes.daemonset.number.ready gauge long number of nodes that have one or more of the daemon pods running and ready
kubernetes.daemonset.number.unavailable gauge long number of nodes that have none of the daemon pods running and available
kubernetes.daemonset.scheduled.updated gauge long number of nodes that are running updated daemon pod
kubernetes.daemonset.scheduled.current gauge long number of nodes that are running at least 1 daemon pod
kubernetes.daemonset.scheduled.desired gauge long number of nodes that should be running the daemon pod

StatefulSet Metrics

Name Type Unit Description
kubernetes.statefulset.desired.replicas gauge long number of desired replicas
kubernetes.statefulset.replicas gauge long number of active replicas
kubernetes.statefulset.replicas.current gauge long number of current replicas
kubernetes.statefulset.replicas.updated gauge long number of updated replicas
kubernetes.statefulset.replicas.ready gauge long number of Pods that have a Ready Condition
kubernetes.statefulset.revision.update gauge long update revision
kubernetes.statefulset.revision.current gauge long the current version of the StatefulSet used to generate Pods

CronJob Metrics

Metric Name Type Unit Description
kubernetes.cronjob.suspended boolean has suspended state or not
kubernetes.cronjob.successful boolean has successful state or not
kubernetes.cronjob.failed boolean has failed state or not
kubernetes.cronjob.active number '1' if is active and '0' if not
kubernetes.cronjob.timestamp timestamp last time scheduled timestamp
kubernetes.cronjob.creation.timestamp timestamp time creation timestamp
kubernetes.cronjob.total gauge total number of cron jobs
kubernetes.cronjob.suspended.total gauge total number of suspended cron jobs
kubernetes.cronjob.successful.total gauge total number of successful cron jobs
kubernetes,cronjob.failed.total gauge total number of failed cron jobs

Job Metrics

Metric Name Type Unit Description
kubernetes.job.condition number is the job finish condition, completed: 2, failed: 0 or suspended: 1
kubernetes.job.executions number job number of executions
kubernetes.job.failures number job number of failures
kubernetes.job.creation.timestamp number time creation timestamp
kubernetes.job.total gauge total number of jobs
kubernetes.job.completed.total gauge total number of completed jobs
kubernetes.job.failed.total gauge total number of failed jobs

Storage Metrics

Metric Name Type Unit Description
kubernetes.pvc.used gauge bytes number of used inodes in the volume
kubernetes.pvc.available gauge bytes number of available bytes in the volume
kubernetes.pvc.capacity gauge bytes capacity in bytes of the volume

Kubelet Metrics

Runtime Operations

Metric Name Type Unit Description
kubelet.runtime_operation.count gauge cumulative number of runtime operations by operation type
kubelet.runtime_operation.errors gauge cumulative number of runtime operation errors by operation type
kubelet.runtime_operation.total_num gauge total number of runtime operations
kubelet.runtime_operation.duration gauge seconds duration of runtime operations
kubelet.runtime_operation.p50latency gauge seconds p50 latency in seconds of runtime operations
kubelet.runtime_operation.p75latency gauge seconds p75 latency in seconds of runtime operations
kubelet.runtime_operation.p90latency gauge seconds p90 latency in seconds of runtime operations
kubelet.runtime_operation.p95latency gauge seconds p95 latency in seconds of runtime operations
kubelet.runtime_operation.p99latency gauge seconds p99 latency in seconds of runtime operations

Kubelet

Metric Name Type Unit Description
kubelet.pods.instances gauge pods Instances
kubelet.pods.running gauge running Pods
kubelet.pods.started gauge started Pods
kubelet.pods.started_error gauge pods Started Errors
kubelet.containers.created gauge containers with status Created
kubelet.containers.exited gauge containers with status Exited
kubelet.containers.running gauge containers with status Running
kubelet.containers.unknown gauge containers with status Unknown

Pod Start Duration

Metric Name Type Unit Description
kubelet.pod_start.total_num gauge number of pod starts
kubelet.pod_start.duration gauge seconds duration of a pod start in seconds
kubelet.pod_start.p50latency gauge seconds p50 latency in seconds for a single pod to go from pending to running
kubelet.pod_start.p75latency gauge seconds p75 latency in seconds for a single pod to go from pending to running
kubelet.pod_start.p90latency gauge seconds p90 latency in seconds for a single pod to go from pending to running
kubelet.pod_start.p95latency gauge seconds p95 latency in seconds for a single pod to go from pending to running
kubelet.pod_start.p99latency gauge seconds p99 latency in seconds for a single pod to go from pending to running

Pod Worker Duration

Metric Name Type Unit Description
kubelet.pod_worker.total_num gauge pod worker counter
kubelet.pod_worker.duration gauge seconds pod worker duration
kubelet.pod_worker.p50latency gauge seconds p50 latency in seconds to sync a single pod
kubelet.pod_worker.p75latency gauge seconds p75 latency in seconds to sync a single pod
kubelet.pod_worker.p90latency gauge seconds p90 latency in seconds to sync a single pod
kubelet.pod_worker.p95latency gauge seconds p95 latency in seconds to sync a single pod
kubelet.pod_worker.p99latency gauge seconds p99 latency in seconds to sync a single pod

Pod Worker Start Duration

Metric Name Type Unit Description
kubelet.pod_worker_start.total_num gauge worker start counter
kubelet.pod_worker_start.duration gauge seconds duration for starting a worker
kubelet.pod_worker_start.p50latency gauge seconds p50 latency in seconds from seeing a pod to starting a worker
kubelet.pod_worker_start.p75latency gauge seconds p75 latency in seconds from seeing a pod to starting a worker
kubelet.pod_worker_start.p90latency gauge seconds p90 latency in seconds from seeing a pod to starting a worker
kubelet.pod_worker_start.p95latency gauge seconds p95 latency in seconds from seeing a pod to starting a worker
kubelet.pod_worker_start.p99latency gauge seconds p99 latency in seconds from seeing a pod to starting a worker

Volume Manager

Metric Name Type Unit Description
kubelet.volume_manager.count gauge total volumes managed by the volume manager
kubelet.volume_manager.desired.count gauge total volumes desired by the volume manager

Storage Operation

Metric Name Type Unit Description
kubelet.storage.total_num gauge total number of storage operations
kubelet.storage.duration gauge seconds duration of storage operations in seconds
kubelet.storage.p50latency gauge seconds p50 latency to perform storage operations
kubelet.storage.p75latency gauge seconds p75 latency to perform storage operations
kubelet.storage.p90latency gauge seconds p90 latency to perform storage operations
kubelet.storage.p95latency gauge seconds p95 latency to perform storage operations
kubelet.storage.p99latency gauge seconds p99 latency to perform storage operations

Cgroup Manager

Metric Name Type Unit Description
kubelet.cgroup.total_num gauge total number of cgroup management operations by the kubelet
kubelet.cgroup.duration gauge seconds duration of cgroup management by the kubelet in seconds
kubelet.cgroup.p50latency gauge seconds p50 latency for cgroup manager operations
kubelet.cgroup.p75latency gauge seconds p75 latency for cgroup manager operations
kubelet.cgroup.p90latency gauge seconds p90 latency for cgroup manager operations
kubelet.cgroup.p95latency gauge seconds p95 latency for cgroup manager operations
kubelet.cgroup.p99latency gauge seconds p99 latency for cgroup manager operations

PLEG Relist Interval

Metric Name Type Unit Description
kubelet.pleg_relist_interval.total_num gauge total number of intervals between pod relisting operations
kubelet.pleg_relist_interval.duration gauge seconds duration of intervals between pod relisting operations
kubelet.pleg_relist_interval.p50latency gauge seconds p50 latency of intervals between pod relisting operations
kubelet.pleg_relist_interval.p75latency gauge seconds p75 latency of intervals between pod relisting operations
kubelet.pleg_relist_interval.p90latency gauge seconds p90 latency of intervals between pod relisting operations
kubelet.pleg_relist_interval.p95latency gauge seconds p95 latency of intervals between pod relisting operations
kubelet.pleg_relist_interval.p99latency gauge seconds p99 latency of intervals between pod relisting operations

Docker Operations

Metric Name Type Unit Description
kubelet.docker_operation.errors gauge number of Docker operation errors
kubelet.docker_operation.operations gauge total number of Docker operations
kubelet.docker_operation.total_num gauge number of Docker operations
kubelet.docker_operation.duration gauge seconds duration of docker operations in seconds
kubelet.docker_operation.p50latency gauge seconds p50 latency in seconds of Docker operations
kubelet.docker_operation.p75latency gauge seconds p75 latency in seconds of Docker operations
kubelet.docker_operation.p90latency gauge seconds p90 latency in seconds of Docker operations
kubelet.docker_operation.p95latency gauge seconds p95 latency in seconds of Docker operations
kubelet.docker_operation.p99latency gauge seconds p99 latency in seconds of Docker operations

HTTP Inflight Request

Metric Name Type Unit Description
kubelet.http_inflight_request.total_num gauge number of inflight requests

HTTP Request Duration

Metric Name Type Unit Description
kubelet.http_request.operations gauge total number of HTTP request operations
kubelet.http_request.total_num gauge total number of HTTP requests
kubelet.http_request.duration gauge seconds HTTP request duration
kubelet.http_request.p50latency gauge seconds p50 HTTP request latency in seconds
kubelet.http_request.p75latency gauge seconds p75 HTTP request latency in seconds
kubelet.http_request.p90latency gauge seconds p90 HTTP request latency in seconds
kubelet.http_request.p95latency gauge seconds p95 HTTP request latency in seconds
kubelet.http_request.p99latency gauge seconds p99 HTTP request latency in seconds

Network Plugin Operations

Metric Name Type Unit Description
kubelet.network_plugin_operation.errors gauge network plugin operation errors
kubelet.network_plugin_operation.operations gauge
kubelet.network_plugin_operation.total_num gauge total number of network plugin operations
kubelet.network_plugin_operation.duration gauge seconds network plugin operations duration
kubelet.network_plugin_operation.p50latency gauge seconds p50 latency in seconds of network plugin operations
kubelet.network_plugin_operation.p75latency gauge seconds p75 latency in seconds of network plugin operations
kubelet.network_plugin_operation.p90latency gauge seconds p90 latency in seconds of network plugin operations
kubelet.network_plugin_operation.p95latency gauge seconds p95 latency in seconds of network plugin operations
kubelet.network_plugin_operation.p99latency gauge seconds p99 latency in seconds of network plugin operations

PLEG Relist Duration

Metric Name Type Unit Description
kubelet.pleg_relist.total_num gauge number of pod relisting performed by kubelet
kubelet.pleg_relist.duration gauge seconds duration of pod relisting performed by kubelet in seconds
kubelet.pleg_relist.p50duration gauge seconds p50 latency of pod relisting
kubelet.pleg_relist.p75duration gauge seconds p75 latency of pod relisting
kubelet.pleg_relist.p90duration gauge seconds p90 latency of pod relisting
kubelet.pleg_relist.p95duration gauge seconds p95 latency of pod relisting
kubelet.pleg_relist.p99duration gauge seconds p99 latency of pod relisting

Started Containers

Metric Name Type Unit Description
kubelet.started_containers.errors gauge number of started containers with errors
kubelet.started_containers.total_num gauge total number of started containers

Volume Stats Inode

Metric Name Type Unit Description
kubelet.volume_stats_inode.free gauge number of free inodes in the volume
kubelet.volume_stats_inode.used gauge number of used inodes in the volume
kubelet.volume_stats_inode.maximum gauge maximum number of inodes in the volume

PLEG Events

Metric Name Type Unit Description
kubelet.pleg_events.discard gauge number of discarded events by kubelet's PLEG
kubelet.pleg_events.last_seen timestamp last timestamp in which PLEG was observed

API Server Metrics

Inflight Requests

Metric Name Type Unit Description
apiserver.inflight_requests.total_num gauge number of in-flight requests being processed by API server

Audit Events

Metric Name Type Unit Description
apiserver.audit_events.total_num gauge number of audit events generated and sent to the audit backend

Request Duration

Metric Name Type Unit Description
apiserver.etcd_request_duration.p50latency gauge seconds p50 latency of etcd requests
apiserver.etcd_request_duration.p75latency gauge seconds p75 latency of etcd requests
apiserver.etcd_request_duration.p90latency gauge seconds p90 latency of etcd requests
apiserver.etcd_request_duration.p95latency gauge seconds p95 latency of etcd requests
apiserver.etcd_request_duration.p99latency gauge seconds p99 latency of etcd requests
apiserver.etcd_request_duration.duration gauge seconds duration of etcd requests measured in seconds
apiserver.etcd_request_duration.total_num gauge number of etcd requests

Storage Objects

Metric Name Type Unit Description
apiserver.storage_objects.total_num gauge number of stored objects at the time of last check split by kind

Client Requests

Metric Name Type Unit Description
apiserver.client_requests.total_num gauge number of HTTP requests

Rejected Audit Requests

Metric Name Type Unit Description
apiserver.rejected_audit_req.total_num gauge number of API server audit requests rejected

Current Executing Requests

Metric Name Type Unit Description
apiserver.current_executing_reqs.total_num gauge number of requests in regular execution phase in the API Priority and Fairness system

Current Inqueue Requests

Metric Name Type Unit Description
apiserver.current_inqueue_reqs.total_num gauge number of requests currently pending in queues of the API Priority and Fairness system

Admission Duration

Metric Name Unit Description
apiserver.admission_duration.p50latency gauge seconds p50 latency of admission controller processing
apiserver.admission_duration.p75latency gauge seconds p75 latency of admission controller processing
apiserver.admission_duration.p90latency gauge seconds p90 latency of admission controller processing
apiserver.admission_duration.p95latency gauge seconds p95 latency of admission controller processing
apiserver.admission_duration.p99latency gauge seconds p99 latency of admission controller processing
apiserver.admission_duration.duration gauge seconds duration of admission controller processing in the API server
apiserver.admission_duration.total_num gauge number of admission controller processing in the API server

TLS Error

Metric Name Type Unit Description
apiserver.tls_handshake_error.total_num gauge number of requests dropped with 'TLS handshake error from' error

Authentication Duration

Metric Name Type Unit Description
apiserver.auth_duration.p50latency gauge seconds p50 latency of authentication process
apiserver.auth_duration.p75latency gauge seconds p75 latency of authentication process
apiserver.auth_duration.p90latency gauge seconds p90 latency of authentication process
apiserver.auth_duration.p95latency gauge seconds p95 latency of authentication process
apiserver.auth_duration.p99latency gauge seconds p99 latency of authentication process
apiserver.auth_duration.duration gauge seconds duration of authentication processes in seconds
apiserver.auth_duration.total_num gauge number of authentication processes

Authenticated User Requests

Metric Name Type Unit Description
apiserver.auth_user_req.total_num gauge number of authenticated requests

Client Certificate Expiry

Metric Name Type Unit Description
apiserver.client_cert_expiry.p50latency gauge seconds p50 latency of expiration time
apiserver.client_cert_expiry.p75latency gauge seconds p75 latency of expiration time
apiserver.client_cert_expiry.p90latency gauge seconds p90 latency of expiration time
apiserver.client_cert_expiry.p95latency gauge seconds p95 latency of expiration time
apiserver.client_cert_expiry.p99latency gauge seconds p99 latency of expiration time
apiserver.client_cert_expiry.duration gauge seconds expiration time in seconds of the client certificates used to authenticate with the API server
apiserver.client_cert_expiry.total_num gauge number of clients certificates used to authenticate with the API server

Key Geneneration Fails

Metric Name Type Unit Description
apiserver.key_gen_fails.total_num gauge number of failed data encryption key(DEK) generation operations

Key Generation Duration

Metric Name Type Unit Description
apiserver.key_gen_duration.p50latency gauge seconds p50 latency of key generation for storing data
apiserver.key_gen_duration.p75latency gauge seconds p75 latency of key generation for storing data
apiserver.key_gen_duration.p90latency gauge seconds p90 latency of key generation for storing data
apiserver.key_gen_duration.p95latency gauge seconds p95 latency of key generation for storing data
apiserver.key_gen_duration.p99latency gauge seconds p99 latency of key generation for storing data
apiserver.key_gen_duration.duration gauge seconds duration of key generation for storing data in the API server's storage in seconds
apiserver.key_gen_duration.total_num gauge number of key generation for storing data in the API server's storage

Requests Total

Metric Name Type Unit Description
apiserver.request_total.total_num gauge number of API server requests

Requests Duration

Metric Name Type Unit Description
apiserver.request_duration.p50latency gauge seconds p50 latency of API server requests
apiserver.request_duration.p75latency gauge seconds p75 latency of API server requests
apiserver.request_duration.p90latency gauge seconds p90 latency of API server requests
apiserver.request_duration.p95latency gauge seconds p95 latency of API server requests
apiserver.request_duration.p99latency gauge seconds p99 latency of API server requests
apiserver.request_duration.duration gauge seconds duration of API server requests in seconds
apiserver.request_duration.total_num gauge number of API server requests

Etcd Metrics

Generic

Metric Name Type Unit Description
etcd.has_leader gauge whether or not a leader exists
etcd.leader.changes_seen gauge number of leader changes seen
etcd.proposals.applied gauge number of consensus proposals applied
etcd.proposals.committed gauge number of consensus proposals committed
etcd.proposals.failed gauge number of failed proposals seen
etcd.proposals.pending gauge number of pending proposals to commit
etcd.version version version of etcd running

Commit

Metric Name Type Unit Description
etcd.disk_backend_commit.p50latency gauge seconds p50 latency of commit operations
etcd.disk_backend_commit.p75latency gauge seconds p75 latency of commit operations
etcd.disk_backend_commit.p90latency gauge seconds p90 latency of commit operations
etcd.disk_backend_commit.p95latency gauge seconds p95 latency of commit operations
etcd.disk_backend_commit.p99latency gauge seconds p99 latency of commit operations
etcd.disk_backend_commit.duration gauge seconds duration of commit operations for etcd's disk backend in seconds
etcd.disk_backend_commit.total_num gauge number of commit operations for etcd's disk backend commits

WAL

Metric Name Type Unit Description
etcd.disk_wal_fsync.p50latency gauge seconds p50 latency of fsync operations
etcd.disk_wal_fsync.p75latency gauge seconds p75 latency of fsync operations
etcd.disk_wal_fsync.p90latency gauge seconds p90 latency of fsync operations
etcd.disk_wal_fsync.p95latency gauge seconds p95 latency of fsync operations
etcd.disk_wal_fsync.p99latency gauge seconds p99 latency of fsync operations
etcd.disk_wal_fsync.total_num gauge number of fsync (flush to disk) operations for etcd's write-ahead log (WAL) on the disk backend
etcd.disk_wal_fsync.duration gauge seconds duration of fsync (flush to disk) operations for etcd's write-ahead log (WAL) on the disk backend in seconds

Backend Snapshots

Metric Name Type Unit Description
etcd.disk_backend_snapshot.p50latency gauge seconds p50 latency of snapshot creation
etcd.disk_backend_snapshot.p75latency gauge seconds p75 latency of snapshot creation
etcd.disk_backend_snapshot.p90latency gauge seconds p90 latency of snapshot creation
etcd.disk_backend_snapshot.p95latency gauge seconds p95 latency of snapshot creation
etcd.disk_backend_snapshot.p99latency gauge seconds p99 latency of snapshot creation
etcd.disk_backend_snapshot.total_num gauge number of snapshot creation for etcd's disk backend
etcd.disk_backend_snapshot.duration gauge seconds duration of snapshot creation for etcd's disk backend in seconds

GRPC Total

Metric Name Type Unit Description
etcd.grpc.handled gauge number of RPCs completed on the server, regardless of success or failure

MVCC

Metric Name Type Unit Description
etcd.mvcc.keys gauge number of keys
etcd.mvcc.events gauge number of events sent by this member
etcd.mvcc.keys_bytes gauge number of MVCC keys involved in etcd debugging operations
etcd.mvcc.watch_stream gauge number of watch streams
etcd.mvcc.watcher gauge number of watchers
etcd.mvcc.db_read gauge number of currently open read transactions
etcd.mvcc.db_size gauge bytes size of the underlying database physically allocated in bytes
etcd.mvcc.db_used gauge bytes size of the underlying database logically in use in bytes
etcd.mvcc.puts gauge number of puts seen by this member
etcd.mvcc.deletes gauge number of deletes seen by this member
etcd.mvcc.slow_watches gauge number of unsynced slow watchers
etcd.watch.requests gauge number of incoming watch requests (new or reestablished)
etcd.store.watchers gauge number of currently active watchers

Commit Rebalance

Metric Name Type Unit Description
etcd.commit_rebalance_duration.p50latency gauge seconds p50 latency of commit and rebalance operations
etcd.commit_rebalance_duration.p75latency gauge seconds p75 latency of commit and rebalance operations
etcd.commit_rebalance_duration.p90latency gauge seconds p90 latency of commit and rebalance operations
etcd.commit_rebalance_duration.p95latency gauge seconds p95 latency of commit and rebalance operations
etcd.commit_rebalance_duration.p99latency gauge seconds p99 latency of commit and rebalance operations
etcd.commit_rebalance_duration.total_num gauge number of commit and rebalance operations for etcd disk backend
etcd.commit_rebalance_duration.duration gauge seconds duration of commit and rebalance operations for etcd disk backend in seconds

Commit Write

Metric Name Type Unit Description
etcd.commit_write.p50latency gauge seconds p50 latency of write operations during commit
etcd.commit_write.p75latency gauge seconds p75 latency of write operations during commit
etcd.commit_write.p90latency gauge seconds p90 latency of write operations during commit
etcd.commit_write.p95latency gauge seconds p95 latency of write operations during commit
etcd.commit_write.p99latency gauge seconds p99 latency of write operations during commit
etcd.commit_write.total_num gauge number of write operations during commit in etcd disk backend
etcd.commit_write.duration gauge seconds duration of write operations during commit in etcd disk backend

DB Compaction Duration

Metric Name Type Unit Description
etcd.mvcc_db_compaction.p50latency gauge seconds p50 latency of MVCC database compaction operations
etcd.mvcc_db_compaction.p75latency gauge seconds p75 latency of MVCC database compaction operations
etcd.mvcc_db_compaction.p90latency gauge seconds p90 latency of MVCC database compaction operations
etcd.mvcc_db_compaction.p95latency gauge seconds p95 latency of MVCC database compaction operations
etcd.mvcc_db_compaction.p99latency gauge seconds p99 latency of MVCC database compaction operations
etcd.mvcc_db_compaction.total_num gauge number of MVCC database compaction operations
etcd.mvcc_db_compaction.duration gauge seconds duration of MVCC database compaction operations in miliseconds

Kube-proxy Metrics

Generic

Metric Name Type Unit Description
kubeproxy.endpoint.changes gauge number of proxy rules endpoint changes
kubeproxy.endpoint.changes_pending gauge number of proxy rules endpoint changes pending
kubeproxy.restore.failures gauge number of proxy iptables restore failures
kubeproxy.service.changes gauge number of proxy rules service changes
kubeproxy.service.changes_pending gauge number of proxy rules service changes pending

Client Requests

Metric Name Type Unit Description
kubeproxy.rest_client.total_num gauge number of HTTP requests, partitioned by status code, method, and host

Client Duration

Metric Name Type Unit Description
kubeproxy.rest_client_duration.p50latency gauge seconds p50 latency of REST client requests
kubeproxy.rest_client_duration.p75latency gauge seconds p75 latency of REST client requests
kubeproxy.rest_client_duration.p90latency gauge seconds p90 latency of REST client requests
kubeproxy.rest_client_duration.p95latency gauge seconds p95 latency of REST client requests
kubeproxy.rest_client_duration.p99latency gauge seconds p99 latency of REST client requests
kubeproxy.rest_client_duration.duration gauge seconds duration of REST client requests in seconds
kubeproxy.rest_client_duration.total_num gauge number of REST client requests

Sync Proxy Rules Duration

Metric Name Type Unit Description
kubeproxy.sync_proxy_rules.p50latency gauge seconds p50 latency of proxy rules synchronization
kubeproxy.sync_proxy_rules.p75latency gauge seconds p75 latency of proxy rules synchronization
kubeproxy.sync_proxy_rules.p90latency gauge seconds p90 latency of proxy rules synchronization
kubeproxy.sync_proxy_rules.p95latency gauge seconds p95 latency of proxy rules synchronization
kubeproxy.sync_proxy_rules.p99latency gauge seconds p99 latency of proxy rules synchronization
kubeproxy.sync_proxy_rules.duration gauge seconds duration of proxy rules synchronization in seconds
kubeproxy.sync_proxy_rules.total_num gauge number of proxy rules synchronization

CoreDNS Metrics

Generic

Metric Name Type Unit Description
coredns.cache.entries gauge number of elements in the cache
coredns.cache.hits gauge number of cache hits
coredns.cache.misses gauge number of cache misses
coredns.panics.total_num gauge number of panics
coredns.failed_reloads.total_num gauge number of failed reload attempts
coredns.healthcheck_broken.total_num gauge number of complete failures of the healthchecks
coredns.forward_max_concurrent_rejects.total_num gauge number of queries rejected because the concurrent queries were at maximum

Request Duration

Metric Name Type Unit Description
coredns.request_duration.p50latency gauge seconds p50 latency of DNS requests handled
coredns.request_duration.p75latency gauge seconds p75 latency of DNS requests handled
coredns.request_duration.p90latency gauge seconds p90 latency of DNS requests handled
coredns.request_duration.p95latency gauge seconds p95 latency of DNS requests handled
coredns.request_duration.p99latency gauge seconds p99 latency of DNS requests handled
coredns.request_duration.duration gauge seconds duration of DNS requests handled by CoreDNS in seconds
coredns.request_duration.total_num gauge number of DNS requests handled by CoreDNS

Requests

Metric Name Unit Description
coredns.request_counter.total_num gauge number of requests handled by CoreDNS

Response Status Codes

Metric Name Unit Description
coredns.response_code.total_num gauge number of response status codes

Forward Cache Hits

Metric Name Type Unit Description
coredns.forward_cache.hits gauge number of connection cache hits per upstream and protocol

Forward Cache Misses

Metric Name Type Unit Description
coredns.forward_cache.misses gauge number of connection cache misses per upstream and protocol

Forward Request Duration

Metric Name Type Unit Description
coredns.forward_request.p50latency gauge seconds p50 latency of forwarded DNS requests handled
coredns.forward_request.p75latency gauge seconds p75 latency of forwarded DNS requests handled
coredns.forward_request.p90latency gauge seconds p90 latency of forwarded DNS requests handled
coredns.forward_request.p95latency gauge seconds p95 latency of forwarded DNS requests handled
coredns.forward_request.p99latency gauge seconds p99 latency of forwarded DNS requests hanndled
coredns.forward_request.duration gauge seconds duration of forwarding DNS requests handled by CoreDNS in seconds
coredns.forward_request.total_num gauge number of forwarding DNS requests handled by CoreDNS

Scheduler Metrics

Preemption Attempts

Metric Name Type Unit Description
scheduler.preemption_attempts.total_num gauge number of preemption attempts in the cluster

Pending Pods

Metric Name Type Unit Description
scheduler.pending_pods.total_num gauge number of pending pods, by the queue type

Queue Incoming Pods

Metric Name Type Unit Description
scheduler.incoming_pods.total_num gauge number of pods added to scheduling queues by event and queue type

Scheduler Attempts

Metric Name Type Unit Description
scheduler.schedule_attempts.total_num gauge number of attempts to schedule pods, by the result

Go Routines

Metric Name Type Unit Description
scheduler.go_routines.total_num gauge number of running goroutines split by the work they do such as binding

Cache

Metric Name Type Unit Description
scheduler.cache.total_num gauge number of nodes, pods, and assumed (bound) pods in the scheduler cache

E2E Scheduling Duration

Metric Name Type Unit Description
scheduler.e2e_duration.p50latency gauge seconds p50 latency of end-to-end scheduling operations
scheduler.e2e_duration.p75latency gauge seconds p75 latency of end-to-end scheduling operations
scheduler.e2e_duration.p90latency gauge seconds p90 latency of end-to-end scheduling operations
scheduler.e2e_duration.p95latency gauge seconds p95 latency of end-to-end scheduling operations
scheduler.e2e_duration.p99latency gauge seconds p99 latency of end-to-end scheduling operations
scheduler.e2e_duration.total_num gauge number of end-to-end scheduling operations
scheduler.e2e_duration.duration gauge seconds duration of end-to-end scheduling operations measured in seconds (scheduling algorithm + binding)

Scheduling Algorithm Duration

Metric Name Type Unit Description
scheduler.scheduling_algorithm.p50latency gauge seconds p50 latency of scheduling algorithm executions
scheduler.scheduling_algorithm.p75latency gauge seconds p75 latency of scheduling algorithm executions
scheduler.scheduling_algorithm.p90latency gauge seconds p90 latency of scheduling algorithm executions
scheduler.scheduling_algorithm.p95latency gauge seconds p95 latency of scheduling algorithm executions
scheduler.scheduling_algorithm.p99latency gauge seconds p99 latency of scheduling algorithm executions
scheduler.scheduling_algorithm.total_num gauge number of scheduling algorithm execution
scheduler.scheduling_algorithm.duration gauge seconds duration of scheduling algorithm execution in seconds

Preemption Victims

Metric Name Type Unit Description
scheduler.preemption_victims.p50latency gauge seconds p50 latency of preemption victims
scheduler.preemption_victims.p75latency gauge seconds p75 latency of preemption victims
scheduler.preemption_victims.p90latency gauge seconds p90 latency of preemption victims
scheduler.preemption_victims.p95latency gauge seconds p95 latency of preemption victims
scheduler.preemption_victims.p99latency gauge seconds p99 latency of preemption victims
scheduler.preemption_victims.total_num gauge number of preemption victims identified by the scheduler for resource reclamation
scheduler.preemption_victims.duration gauge seconds duration of preemption victims identified by the scheduler for resource reclamation in seconds

Scheduling Duration

Metric Name Type Unit Description
scheduler.scheduling_duration.p50latency gauge seconds p50 latency of pod scheduling operations
scheduler.scheduling_duration.p75latency gauge seconds p75 latency of pod scheduling operations
scheduler.scheduling_duration.p90latency gauge seconds p90 latency of pod scheduling operations
scheduler.scheduling_duration.p95latency gauge seconds p95 latency of pod scheduling operations
scheduler.scheduling_duration.p99latency gauge seconds p99 latency of pod scheduling operations
scheduler.scheduling_duration.total_num gauge number of pod scheduling operations in the scheduler
scheduler.scheduling_duration.duration gauge seconds duration of pod scheduling operations in the scheduler measured in seconds

Framework Extension Duration

Metric Name Type Unit Description
scheduler.framework_extension.p50latency gauge seconds p50 latency of framework extension point executions
scheduler.framework_extension.p75latency gauge seconds p75 latency of framework extension point executions
scheduler.framework_extension.p90latency gauge seconds p90 latency of framework extension point executions
scheduler.framework_extension.p95latency gauge seconds p95 latency of framework extension point executions
scheduler.framework_extension.p99latency gauge seconds p99 latency of framework extension point executions
scheduler.framework_extension.total_num gauge number of framework extension point execution measured
scheduler.framework_extension.duration gauge seconds duration of framework extension point execution measured in seconds

Scheduler Attempts

Metric Name Type Unit Description
scheduler.attempts.total_num gauge number of attempts to schedule pods

Kube-controller Metrics

Workqueue Wait

Metric Name Type Unit Description
kubecontroller.workqueue_wait.p50latency gauge seconds p50 wait time latency before item is processed
kubecontroller.workqueue_wait.p75latency gauge seconds p75 wait time latency before item is processed
kubecontroller.workqueue_wait.p90latency gauge seconds p90 wait time latency before item is processed
kubecontroller.workqueue_wait.p95latency gauge seconds p95 wait time latency before item is processed
kubecontroller.workqueue_wait.p99latency gauge seconds p99 wait time latency before item is processed
kubecontroller.workqueue_wait.duration gauge seconds duration of waiting time before item is processed
kubecontroller.workqueue_wait.total_num gauge number of items being processed

Workqueue Process

Metric Name Type Unit Description
kubecontroller.workqueue_process.p50latency gauge seconds p50 latency of item being processed by Kube-controller
kubecontroller.workqueue_process.p75latency gauge seconds p75 latency of item being processed by Kube-controller
kubecontroller.workqueue_process.p90latency gauge seconds p90 latency of item being processed by Kube-controller
kubecontroller.workqueue_process.p95latency gauge seconds p95 latency of item being processed by Kube-controller
kubecontroller.workqueue_process.p99latency gauge seconds p99 latency of item being processed by Kube-controller
kubecontroller.workqueue_process.duration gauge seconds duration of processing an item by Kube-controller
kubecontroller.workqueue_process.total_num gauge number of items being processed by Kube-controller

Workqueue Depth

Metric Name Type Unit Description
kubecontroller.workqueue_depth.total_num gauge number of items queued for processing

Workqueue Unfinished Work

Metric Name Type Unit Description
kubecontroller.workqueue_unfinished_work.total_num gauge number of items in the workqueue

Workqueue Longest Running

Metric Name Type Unit Description
kubecontroller.workqueue_longest_running.total_num gauge number of longest-running task processors in workqueue

Registered Nodes

Metric Name Type Unit Description
kubecontroller.registered_nodes.total_num gauge number of registered nodes in the cluster

Healthy Nodes

Metric Name Type Unit Description
kubecontroller.healthy_nodes.percentage gauge number of healthy nodes in a specific zone

Unhealthy Nodes

Metric Name Type Unit Description
kubecontroller.unhealthy_nodes.percentage gauge number of unhealthy nodes in a specific zone

Workqueue Retries

Metric Name Type Unit Description
kubecontroller.workqueue_retries.total_num gauge number of retries attempted by workqueue items

Runtime Metrics

Metric Name Type Unit Description
kubernetes.runtime.os_threads gauge number of operating system threads used
kubernetes.runtime.goroutines gauge number of goroutines active
kubernetes.runtime.resident_memory gauge bytes amount of memory for process code and data occupied
kubernetes.runtime.virtual_memory gauge bytes amount of virtual memory used
kubernetes.runtime.number_frees gauge number of allocated memory blocks that have been freed
kubernetes.runtime.number_mallocs gauge number of memory allocations performed
kubernetes.runtime.heap_obtained gauge bytes memory obtained from the operating system for the heap
kubernetes.runtime.heap_used gauge bytes amount of heap memory in use
kubernetes.runtime.heap_waiting gauge bytes amount of heap memory currently unused and can be potentially allocated
kubernetes.runtime.number_heap_objects gauge number of allocated objects in the heap memory
kubernetes.runtime.stack_obtained gauge bytes memory obtained from the operating system for the stack space
kubernetes.runtime.stack_used gauge bytes amount of stack memory in use
kubernetes.runtime.gc_duration gauge seconds duration of a garbage collection event
kubernetes.runtime.gc_count gauge number of garbage collection events