\n| rancher1.cloudwerkstatt.com<\/td>\n | prom.rancher1.cloudwerkstatt.com\/federate<\/td>\n | 30s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Here’s an example YAML for the ingress object that exposes Prometheus with federation:<\/p>\n apiVersion: networking.k8s.io\/v1\nkind: Ingress\nmetadata:\n annotations:\n cert-manager.io\/cluster-issuer: cluster-acme\n name: expose-prom\n namespace: cattle-monitoring-system\nspec:\n rules:\n - host: prom.rancher1.cloudwerkstatt.com\n http:\n paths:\n - backend:\n service:\n name: rancher-monitoring-prometheus\n port:\n number: 9090\n path: \/federate\n pathType: ImplementationSpecific\n tls:\n - hosts:\n - prom.rancher1.cloudwerkstatt.com\n secretName: prom-secret<\/code><\/pre>\nFeel free to adapt this example to match your specific cluster configurations and requirements. This YAML snippet demonstrates the creation of an ingress object, including the necessary annotations for certificate management. The host<\/code> field specifies the URL for accessing the federation endpoint, while the backend<\/code> section defines the service and port to route the traffic to. Additionally, the TLS section enables secure communication with the appropriate secret name for the SSL certificate.<\/p>\nWith the federation configured, we can proceed to explore the comparison between federating all job metrics and federating specific metrics in the subsequent sections of this blog post.<\/p>\n \n\n<\/p>\n \n\n<\/p>\n \n\n<\/p>\n Federate All metrics<\/h2>\nFirst, we will begin by federating all metrics to showcase the sheer amount of data that can be collected. However, it is essential to understand and consider the purpose for which you want to collect all metrics. It is typically not considered a best practice to federate everything.<\/p>\n To federate all metrics, we need to configure the central Prometheus instance accordingly. By specifying the appropriate configuration, the central Prometheus instance will collect and aggregate metrics from all federated sources. It is important to note that federating all metrics can have implications on resource consumption, such as memory, CPU, and disk usage.<\/p>\n Within the prometheus.yml<\/code> file of the central Prometheus configuration, you’ll find the section that handles the collection of all metrics from the federated endpoint.<\/p>\nscrape_configs:\n - job_name: rancher-federate\n scrape_timeout: 30s\n honor_labels: true\n metrics_path: '\/federate'\n scheme: https\n params:\n 'match[]':\n - '{__name__=~".+"}' # Collect all\n tls_config:\n # For simplicity, let's skip SSL verification to focus on the main point of this blog\n insecure_skip_verify: true\n static_configs:\n - targets:\n - prom.rancher1.cloudwerkstatt.com\n labels:\n cluster_id: 'rancher1'\n env: "Dev"<\/code><\/pre>\nLet’s examine some real data. Please note that this Rancher cluster is currently empty, meaning it does not run any applications and thus contains less data compared to a cluster with multiple running apps (pods). The data provided will illustrate the amount of data scraped in 24 hours. Now, envision having 10 or 100 clusters, each being scraped in the same manner. Such an approach would result in significant resource wastage due to suboptimal setup.<\/p>\n \n\n\n| Central Prometheus Instance<\/th>\n | TSDB Size after 24h<\/th>\n<\/tr>\n<\/thead>\n | \n\n| centralprom.cloudwerkstatt.com<\/td>\n | 630MB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Now, let’s perform some calculations based on the expanded setup:<\/p>\n \n- There are 40 clusters in total.<\/li>\n
- The approximate size of each federated cluster’s TSDB (Time-Series Database) is 630MB.<\/li>\n
- The retention period for the central Prometheus would be set to 30 days.<\/li>\n<\/ul>\n
If we intend to retain all this data for 30 days<\/code>, we would require around (40 * 630 * 30) \/ 1024 \u2248 738GB<\/code> of storage. However, this would significantly increase the memory footprint for Prometheus too. Consequently, you would need to allocate more memory and CPU resources to handle the increased workload.<\/p>\nIn the next section, we will explore an alternative approach: federating specific metrics. This approach allows for a more targeted and efficient monitoring strategy, focusing only on the metrics that are most relevant to your specific use case and objectives.<\/p>\n \n\n<\/p>\n \n\n<\/p>\n \n\n<\/p>\n Federate Specific Metrics<\/h2>\nIn this example, we will explore the approach of federating only minimal set of metrics that are necessary for alerting purposes. By focusing on specific metrics, we can optimize resource usage and streamline monitoring operations.<\/p>\n To begin, we need to configure the central Prometheus instance to federate only the desired metrics. By specifying the appropriate configuration, we ensure that only the selected metrics are collected and aggregated. This targeted approach allows for more efficient resource utilization, including memory, CPU, and disk usage.<\/p>\n Please keep in mind that the snippet below of the prometheus.yml<\/code> is just an example to demonstrate federation with a specific set of metrics.<\/p>\nscrape_configs:\n - job_name: rancher-federate\n scrape_timeout: 30s\n honor_labels: true\n metrics_path: '\/federate'\n scheme: https\n params:\n 'match[]':\n - '{job="node-exporter"}'\n - '{job="rancher-monitoring-prometheus"}'\n - '{job="apiserver"}'\n - '{job="kube-state-metrics"}'\n - '{job="kube-scheduler"}'\n - '{job="kube-etcd"}'\n - '{job="kube-controller-manager"}'\n - '{job="kubelet"}'\n - '{job="coredns"}'\n - '{job="cert-manager"}'\n tls_config:\n insecure_skip_verify: true\n metric_relabel_configs:\n - source_labels: [__name__]\n action: keep\n regex: "(kube_pod_status_phase|\\\n etcd_network_peer_round_trip_time_seconds_bucket|\\\n kube_node_status_condition|\\\n kube_pod_container_status_restarts_total|\\\n kube_persistentvolume_status_phase|\\\n node_memory_MemAvailable_bytes|\\\n node_memory_MemTotal_bytes|\\\n kubelet_volume_stats_available_bytes|\\\n kubelet_volume_stats_capacity_bytes|\\\n node_filesystem_avail_bytes|\\\n node_filesystem_size_bytes|\\\n node_filesystem_readonly|\\\n apiserver_client_certificate_expiration_seconds_count|\\\n apiserver_client_certificate_expiration_seconds_bucket|\\\n aggregator_unavailable_apiservice|\\\n kube_pod_status_phase|\\\n kube_pod_info|\\\n kube_node_status_capacity|\\\n node_network_up|\\\n node_nf_conntrack_entries|\\\n node_nf_conntrack_entries_limit|\\\n prometheus_tsdb_head_samples_appended_total|\\\n coredns_panics_total|\\\n coredns_dns_responses_total|\\\n etcd_network_peer_round_trip_time_seconds_bucket|\\\n etcd_server_has_leader|\\\n etcd_network_peer_sent_failures_total|\\\n etcd_mvcc_db_total_size_in_bytes|\\\n etcd_server_quota_backend_bytes|\\\n etcd_server_is_leader|\\\n container_memory_working_set_bytes|\\\n kube_pod_container_resource_limits_memory_bytes|\\\n kube_node_role|\\\n kube_pod_container_resource_requests|\\\n kube_pod_container_resource_limits|\\\n certmanager_certificate_ready_status|\\\n certmanager_http_acme_client_request_count|\\\n certmanager_certificate_expiration_timestamp_seconds|\\\n kube_node_status_allocatable|\\\n kubelet_node_name|\\\n up)"\n static_configs:\n - targets:\n - prom.rancher1.cloudwerkstatt.com\n labels:\n cluster_id: 'rancher1'\n env: "Dev"<\/code><\/pre>\nLet’s take a look at some real data where only a specific set of jobs is scraped, and then only a specific set of metrics is stored in the central Prometheus. This ensures that the central Prometheus only stores metrics from the listed jobs and those defined in the metric_relabel_configs<\/code>. The inclusion of action: keep<\/code> is vital, as it ensures that any other metrics not meeting the criteria will be discarded, resulting in a substantial reduction in TSDB size.<\/p>\n\n\n\n| Central Prometheus Instance<\/th>\n | TSDB Size after 24h<\/th>\n<\/tr>\n<\/thead>\n | \n\n| centralprom.cloudwerkstatt.com<\/td>\n | 11MB<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n Once again let’s perform some calculations based on the expanded setup:<\/p>\n \n- There are 40 clusters in total<\/li>\n
- The approximate size of each federated cluster’s TSDB (Time-Series Database) is 11MB.<\/li>\n
- The retention period for the central Prometheus would be set to 30 days.<\/li>\n<\/ul>\n
So the data would persist for 30 days in the central Prometheus, and we will require around ( 40 * 11 * 30) \/ 1024 \u2248 12.9GB<\/code> of storage.
\nNow it is evident that the central Prometheus instance will contain much less data compared to the previous case where all metrics were collected. This also decreases the memory and CPU footprint on the central Prometheus.<\/p>\nBy federating specific metrics, we can create a monitoring setup that aligns closely with our alerting requirements. This approach enables us to focus on the metrics that are most crucial for detecting and responding to critical events. By eliminating unnecessary noise and reducing the volume of collected data, we can enhance the effectiveness and responsiveness of our alerting system.<\/p>\n In the next section, we will further explore the benefits and considerations of federating specific metrics, as well as provide guidance on best practices for implementing this approach effectively.<\/p>\n \n\n<\/p>\n \n\n<\/p>\n \n\n<\/p>\n Benefits and Considerations of Federating Specific Metrics<\/h2>\nFederating specific metrics offers several advantages, allowing for a more targeted and efficient monitoring strategy. Let’s explore some of the key benefits and considerations associated with this approach:<\/p>\n \n- \n
Reduced Resource Consumption:<\/strong> Federating specific metrics optimizes resource usage on the central Prometheus instance.<\/p>\n<\/li>\n- \n
Reduce federation failure:<\/strong> Federation is not designed to handle the federation of all jobs, and attempting to federate a large number of metrics from all jobs can lead to issues. You may encounter errors in the Prometheus logs with a federation failed<\/code> message.<\/p>\n<\/li>\n<\/ul>\n |
|
|
![\"Prometheus](\"https:\/\/www.cloudwerkstatt.com\/wp-content\/uploads\/2023\/08\/prom_federation_all_vs_specific.jpg\")
<\/p>\n