Prometheus redis exporter autodiscovery config

Problem

Some time ago at my job I had to implement prometheus monitoring for all redis instances. In this case redis was used as cache, one instance for each project and environment, running on kubernetes.

After some time I had ExternalName setup that connected prometheus namespace with all redis instances, and static list of all instances inside prometheus config. It was fine, just lot of manual work to add another one, and prometheus restart/reload to get new config going. Also not everybody in my team was able to add another instance.

I have set up kubernetes prometheus autodiscovery before, at that time it was quite easy- a lot of examples on the internet. But this time, for I couldn’t find anything for exporter that I was using.

After some trial and error (mostly errors) I came up with something at least kinda ok.

Solution

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

- job_name: 'redis autodiscovery'
  scrape_interval: 1m
  scrape_timeout: 55s
  metrics_path: /scrape
  kubernetes_sd_configs:
  - role: endpoints
  relabel_configs:
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_redis_scrape]
    action: keep
    regex: true
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_redis_scheme]
    action: keep
    regex: (rediss?)
  - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_redis_scheme, __address__]
    action: replace
    regex: (rediss?);(.*)
    replacement: "${1}://${2}"
    target_label: __param_target
  - source_labels: [__param_target]
    action: replace
    target_label: instance
  - replacement: redis-exporter.monitoring.svc.cluster.local:9121
    action: replace
    target_label: __address__
  - source_labels: [__meta_kubernetes_namespace]
    action: replace
    target_label: kubernetes_namespace
  - source_labels: [__meta_kubernetes_service_name]
    action: replace
    target_label: kubernetes_service
  - source_labels: [__meta_kubernetes_pod_name]
    action: replace
    target_label: kubernetes_pod
 

Of course to get this to work you have to also have RBAC setup, the same as for prometheus kubernetes discovery.

Now you just need to add annotations to redis service:

1
2
3

annotations:
  prometheus.io.redis/scrape: "true"
  prometheus.io.redis/scheme: "redis"
 

I think first one is self-explanatory, second one is redis protocol- redis for non ssl, rediss for tls.

For more inspiration check out my deploy repo - prometheus config in monitoring folder.