Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hydra doesn't like Grafana Cloud OTLP endpoint #778

Open
3 of 5 tasks
Moosemorals opened this issue Jan 12, 2024 · 0 comments
Open
3 of 5 tasks

Hydra doesn't like Grafana Cloud OTLP endpoint #778

Moosemorals opened this issue Jan 12, 2024 · 0 comments
Labels
bug Something is not working. effort/small It will probably need less than a day to solve this problem. good first issue A good issue to tackle when being a novice to the project. help wanted We are looking for help on this one.

Comments

@Moosemorals
Copy link

Preflight checklist

Ory Network Project

No response

Describe the bug

When I set tracing.providers.otlp.server_url to the URL given by Grafana cloud (https://otlp-gateway-prod-gb-south-0.grafana.net/otlp), Hydra logs show "The configuration contains values or keys which are invalid" and three regexes that look like they should match an ipv4 address and host, ipv6 address and host, or domain name and host.

Digging into the code SetupOTLP in otelx/otlp.go, you're calling otlptracehttp.WithEndpoint, which does only accept a host/address and port number, but that should also be called with WithURLpath since not all providers (e.g., Grafana Cloud) use the standard path.

What I'd really like you to do is not call any of the With* methods at all, and let otlptracehttp pick up it's config from the environment, in line with other otlp collectors.

Reproducing the bug

  1. Configure hydra with a url under tracing.providers.otlp.server_url
tracing:
  providers:
    otlp:
      server_url: otlp-gateway-prod-gb-south-0.grafana.net/otlp

Relevant log output

Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing: map[provider:otel providers:map[otlp:map[server_url:https://otlp-gateway-prod-gb-south-0.grafana.net/otlp]]]
Jan 12 23:29:15 example.com hydra-server[56720]:          ^-- doesn't validate with "ory://tracing-config#"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- anyOf failed
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- does not match pattern "^\\[(([0-9a-fA-F]{1,4}:){7}[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,7}:|([0-9a-fA-F]{1,4}:){1,6}:[0-9a-fA-F]{1,4}|([0-9a-fA-F]{1,4}:){1,5}(:[0-9a-fA-F]{1,4}){1,2}|([0-9a-fA-F]{1,4}:){1,4}(:[0-9a-fA-F]{1,4}){1,3}|([0-9a-fA-F]{1,4}:){1,3}(:[0-9a-fA-F]{1,4}){1,4}|([0-9a-fA-F]{1,4}:){1,2}(:[0-9a-fA-F]{1,4}){1,5}|[0-9a-fA-F]{1,4}:((:[0-9a-fA-F]{1,4}){1,6})|:((:[0-9a-fA-F]{1,4}){1,7}|:)|fe80:(:[0-9a-fA-F]{0,4}){0,4}%[0-9a-zA-Z]{1,}|::(ffff(:0{1,4}){0,1}:){0,1}((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])|([0-9a-fA-F]{1,4}:){1,4}:((25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9])\\.){3,3}(25[0-5]|(2[0-4]|1{0,1}[0-9]){0,1}[0-9]))]:([0-9]*)$"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- does not match pattern "^([0-9]{1,3}\\.){3}[0-9]{1,3}:([0-9]*)$"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: The configuration contains values or keys which are invalid:
Jan 12 23:29:15 example.com hydra-server[56720]: tracing.providers.otlp.server_url: https://otlp-gateway-prod-gb-south-0.grafana.net/otlp
Jan 12 23:29:15 example.com hydra-server[56720]:                                    ^-- does not match pattern "^(([a-zA-Z0-9]|[a-zA-Z0-9][a-zA-Z0-9\\-]*[a-zA-Z0-9])\\.)*([A-Za-z0-9]|[A-Za-z0-9][A-Za-z0-9\\-]*[A-Za-z0-9]):([0-9]*)$"
Jan 12 23:29:15 example.com hydra-server[56720]:
Jan 12 23:29:15 example.com hydra-server[56720]: {"audience":"application","event":"&jsonschema.ValidationError{Message:\"doesn't validate with \\\"ory://tracing-config#\\\"\", InstancePtr:\"#/tracing\", SchemaURL:\"https://github.com/ory/hydra/spec/config.json\", SchemaPtr:\"#/properties/tracing/$ref\", Context:jsonschema.ValidationErrorContext(nil), Causes:[]*jsonschema.ValidationError{(*jsonschema.ValidationError)(0xc00095a000)}}","level":"error","msg":"The changed configuration is invalid and could not be loaded. Rolling back to the last working configuration revision. Please address the validation errors before restarting the process.","service_name":"Ory Hydra","service_version":"v2.2.0-rc.3","time":"2024-01-12T23:29:15.154613904Z"}

Relevant configuration

tracing:
  providers:
    otlp:
      server_url: otlp-gateway-prod-gb-south-0.grafana.net/otlp

Version

2.2.0-rc.3

On which operating system are you observing this issue?

Linux

In which environment are you deploying?

Other

Additional Context

I'm running the container under podman on a linux host. Otherwise my install is fine (it's great!).

@Moosemorals Moosemorals added the bug Something is not working. label Jan 12, 2024
@alnr alnr transferred this issue from ory/hydra Mar 28, 2024
@alnr alnr added help wanted We are looking for help on this one. good first issue A good issue to tackle when being a novice to the project. effort/small It will probably need less than a day to solve this problem. labels Apr 5, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something is not working. effort/small It will probably need less than a day to solve this problem. good first issue A good issue to tackle when being a novice to the project. help wanted We are looking for help on this one.
Projects
None yet
Development

No branches or pull requests

2 participants