Use Prometheus monitoring

Use Prometheus with W&B Server. Prometheus installs are exposed as a kubernetes ClusterIP service.

Follow the procedure below to access your Prometheus metrics endpoint (/metrics):

1. Connect to the cluster with Kubernetes CLI toolkit, kubectl. See kubernetes’ Accessing Clusters documentation for more information.

2. Find the internal address of the cluster with:

   kubectl describe svc prometheus
   

3. Start a shell session inside your container running in your Kubernetes cluster with kubectl exec. Hit the endpoint at <internal address>/metrics.

   Copy the command below and execute it in your terminal and replace <internal address> with your internal address:

   kubectl exec <internal address>/metrics
   

A test pod starts, which you can exec into just to access anything in the network:

kubectl run -it testpod --image=alpine bin/ash --restart=Never --rm

From there you can choose to keep access internal to the network or expose it yourself with a kubernetes nodeport service.