FEAT(observability): enable HA with replica 2 and soft anti-affinity

- Add replicaCount: 2 to goldilocks, vpa, alertmanager
- Add replicas: 2 to loki singleBinary
- Add soft pod anti-affinity for node distribution
- Keep kube-state-metrics at replica 1 to prevent duplicate metrics

FIX(loki): revert to replica 1 for Single Binary mode

- Single Binary mode cannot run more than 1 replica without object storage
- Remove affinity configuration for single replica
- Keep filesystem storage backend

goldilocks/helm-values.yaml

@@ -4,7 +4,7 @@
 # Dashboard configuration
 dashboard:
   enabled: true
-  replicaCount: 1
+  replicaCount: 2
 
   resources:
     requests:
@@ -52,3 +52,15 @@ controller:
 vpa:
   # Set to false since we're installing VPA separately
   enabled: false
+
+# Affinity - Soft Anti-Affinity to spread pods across nodes
+dashboard:
+  affinity:
+    podAntiAffinity:
+      preferredDuringSchedulingIgnoredDuringExecution:
+        - weight: 100
+          podAffinityTerm:
+            labelSelector:
+              matchLabels:
+                app.kubernetes.io/name: goldilocks
+            topologyKey: kubernetes.io/hostname