Incorrect generation of harbor-core URL in part of jobservice

[Tonkari]

Summary

Jobservice logs show that some calls from hook_agent go to an incorrect harbor-core URL

Environment

We are using this chart to deploy harbor 2.10.2 (chart version 1.14.2) with internal TLS enabled, certSource is secret and the secrets are managed by cert-manager. This means our services all listen exclusively on port 443

Problem

When troubleshooting an unrelated issue with the jobservice we noticed some errors in the pod logs that contain failed requests to "http://harbor-core:80", when all other requests from the pod go to the correct URL of "https://harbor-core:443"

Log messages

# failed call
2024-07-11T07:57:13Z [ERROR] [/jobservice/hook/hook_agent.go:159]: Retry: still failed after all retries: Post "http://harbor-core:80/service/notifications/tasks/xxx": context deadline exceeded (Client.Timeout exceeded while awaiting headers), evt=status change: job=xxxxxxxxxxxxxxxxxxxxxxxx@1720656000, status=Success, revision=1720655940->http://harbor-core:80/service/notifications/tasks/xxx

# working call from same pod
2024-07-11T08:00:00Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://harbor-core:443/api/v2.0/internalconfig

I don't know if it's a configuration error in this chart of if it's hardcoded in the docker image itself - I would appreciate any help or direction. I also don't know what functions of harbor are impaired here or what earlier versions are affected.

[zyyw]

@Tonkari thanks for reporting this issue. Could you please share with us what triggers the # failed call.

Tried configuring a webhook for image scanning on an internalTLS enabled Harbor (harbor-helm main branch / harbor oss main branch), do not see the http://harbor-core:80 error log.

2024-07-25T08:28:11Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"IMAGE_SCAN","id":"fe263f2eddb3492c72786b57","t":1721896089,"args":null}
2024-07-25T08:28:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:16Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f57d80b348d3711a7be26fc8","t":1721896091,"args":null}
2024-07-25T08:28:16Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:16Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:17Z [INFO] [/jobservice/runner/redis.go:152]: Job 'WEBHOOK:f57d80b348d3711a7be26fc8' exit with success
2024-07-25T08:28:31Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"8792a36d444c0a213f42259f","t":1721896110,"args":null}
2024-07-25T08:28:31Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:31Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:32Z [INFO] [/jobservice/runner/redis.go:152]: Job 'WEBHOOK:8792a36d444c0a213f42259f' exit with success
2024-07-25T08:28:33Z [INFO] [/pkg/scan/postprocessors/report_converters.go:196][report="bdf79a35-1494-42ed-9e18-34b394573625" scanner="e18e636d-4a5e-11ef-b796-7a71a6a03a13" vulnerabilityRecords="20"]: Converted vulnerability records to the new schema
2024-07-25T08:28:33Z [INFO] [/jobservice/runner/redis.go:152]: Job 'IMAGE_SCAN:fe263f2eddb3492c72786b57' exit with success
2024-07-25T08:28:33Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f7c4c07113cd5bd13ac3e05a","t":1721896113,"args":null}
2024-07-25T08:28:33Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:33Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T08:28:34Z [INFO] [/jobservice/runner/redis.go:152]: Job 'WEBHOOK:f7c4c07113cd5bd13ac3e05a' exit with success

[zyyw]

Also tried replication with webhook on an internalTLS enabled Harbor (v2.10.2, harbor-helm 1.14.2). Not seeing http://harbor-core:80 error log either.

2024-07-25T09:23:58Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"REPLICATION","id":"6e366f8623a6ca639f078b45","t":1721899436,"args":null}
2024-07-25T09:23:58Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:23:58Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"d45b9170f1c79f961efad041","t":1721899438,"args":null}
2024-07-25T09:24:00Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:00Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:00Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:d45b9170f1c79f961efad041' exit with success
2024-07-25T09:24:02Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"e09e1552894eb181a6a3a408","t":1721899442,"args":null}
2024-07-25T09:24:02Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:02Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:02Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:e09e1552894eb181a6a3a408' exit with success
2024-07-25T09:24:03Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"473e584319cd5a939f05eba5","t":1721899443,"args":null}
2024-07-25T09:24:03Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:03Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:04Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:473e584319cd5a939f05eba5' exit with success
2024-07-25T09:24:04Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f870f187b6595097bc5851ed","t":1721899444,"args":null}
2024-07-25T09:24:04Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:04Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:05Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:f870f187b6595097bc5851ed' exit with success
2024-07-25T09:24:06Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"1252722613e2105462c6389d","t":1721899446,"args":null}
2024-07-25T09:24:06Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:06Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:07Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:1252722613e2105462c6389d' exit with success
2024-07-25T09:24:07Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"204a55fdd1104289c763a615","t":1721899447,"args":null}
2024-07-25T09:24:07Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:07Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:09Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:204a55fdd1104289c763a615' exit with success
2024-07-25T09:24:09Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"a0b7acddbf04580fcd61abec","t":1721899448,"args":null}
2024-07-25T09:24:09Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:09Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:10Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"5b2286920779f2e22770f43a","t":1721899450,"args":null}
2024-07-25T09:24:10Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:10Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:10Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:a0b7acddbf04580fcd61abec' exit with success
2024-07-25T09:24:11Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"58c58549b55cd43089d1a0c9","t":1721899451,"args":null}
2024-07-25T09:24:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:11Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:12Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:5b2286920779f2e22770f43a' exit with success
2024-07-25T09:24:13Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:58c58549b55cd43089d1a0c9' exit with success
2024-07-25T09:24:13Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"7d7f06d63a55d227d81c9a1c","t":1721899452,"args":null}
2024-07-25T09:24:13Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:13Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:7d7f06d63a55d227d81c9a1c' exit with success
2024-07-25T09:24:14Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"f711fcdc772a57aa69406af5","t":1721899453,"args":null}
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/jobservice/runner/redis.go:151]: Job 'REPLICATION:6e366f8623a6ca639f078b45' exit with success
2024-07-25T09:24:14Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"145b9765575f365b7bcc789d","t":1721899454,"args":null}
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:f711fcdc772a57aa69406af5' exit with success
2024-07-25T09:24:14Z [INFO] [/jobservice/worker/cworker/c_worker.go:77]: Job incoming: {"name":"WEBHOOK","id":"7b9c0397e795f064f85b2718","t":1721899454,"args":null}
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:14Z [INFO] [/pkg/config/rest/rest.go:47]: get configuration from url: https://myharbor-core:443/api/v2.0/internalconfig
2024-07-25T09:24:15Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:7b9c0397e795f064f85b2718' exit with success
2024-07-25T09:24:15Z [INFO] [/jobservice/runner/redis.go:151]: Job 'WEBHOOK:145b9765575f365b7bcc789d' exit with success

[chlins]

@Tonkari If you initially used port 80 and later switched to 443, there might be some old tasks that send requests to the old address. Please try restarting the harbor core and jobservice containers, and check their environment variables for CORE_URL related configurations.

[Tonkari]

Hi, thanks for the investigation,

We were indeed using port 80 initially, but that was many years and versions ago. In the meantime all pods have been updated and restarted multiple times (we are running everything with at least 3 replicas), and a manual restart of jobservice did not get rid of the error.
How can I identify the job that triggers the notification? We are not running any replication between other registries but everything else should be configured (GC, scans with built-in trivy, retention mostly). It always calls the same URL as well http://harbor-core:80/service/notifications/tasks/186, not multiple different ones.

[github-actions]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.