Bug - upstream (loki) behind rproxy

[Ronan-WeScale]

I tried to deploy observatorium on dev cluster and call loki on other cluster, loki is exposed by nginx ingress.

It seems the api don't set the right Host header on the call, observatorium log :

level=debug name=observatorium ts=2024-08-09T09:26:42.711366336Z caller=proxy.go:61 request=observatorium-api-bbf9b9dd6-5dsqd/uam7XGqFNW-000018 msg="request to upstream" url="https://loki.xxx.com/loki/api/v1/query?direction=backward&query=vector%281%29%2Bvector%281%29&time=4000000000"
level=debug name=observatorium ts=2024-08-09T09:26:42.714032285Z caller=instrumentation.go:36 request=observatorium-api-bbf9b9dd6-5dsqd/uam7XGqFNW-000018 proto=HTTP/1.1 method=GET status=404 content= path=/api/logs/v1/xxx/loki/api/v1/query duration=3.310348ms bytes=146

With tcpdump I can catch the header and see the Host isn't my loki address "loki.xxx.com" but my observatorium url (obs.xxx.com), so it's normal the nginx ingress answer 404.

Host: obs.xxx.com
User-Agent: Grafana/10.4.3
Accept-Encoding: gzip
Authorization: Bearer yyyy
Forwarded: for=10.128.4.27;host=obs.xxx.com;proto=https
X-Datasource-Uid: e9b1dd3f-49c9-4bff-bdec-f53bd2ee7ea9
X-Forwarded-For: 10.128.4.27
X-Forwarded-Host: obs.xxx.com
X-Forwarded-Port: 443
X-Forwarded-Prefix: /api/logs/v1/infra
X-Forwarded-Proto: https
X-Grafana-Org-Id: 1
X-Id-Token: yyyy

As workaround I had bypass my ingress with metallb to forward traffic directly on loki-gateway svc.