Nginx keeps restarting after upgrade

[ntex777]

Hi

Our registry was a bit outdated back in version 2.4.1, so we decided to upgrade using these versions -> 2.5.6 -> 2.7.4 -> 2.9.5 -> 2.11.1 - keeping the rule no more than 2 previous minor builds.

We have HTTPS enabled, but as soon we jumped to 2.5.6 nginx keeps restarting:

Sep  5 17:51:39 172.31.0.1 proxy[102991]: nginx: [emerg] cannot load certificate "/etc/cert/server.crt": PEM_read_bio_X509_AUX() failed (SSL: error:0480006C:PEM routines::no start line:Expecting: TRUSTED CERTIFICATE)
Sep  5 17:52:39 172.31.0.1 proxy[102991]: 2024/09/05 16:52:39 [emerg] 1#0: cannot load certificate "/etc/cert/server.crt": PEM_read_bio_X509_AUX() failed (SSL: error:0480006C:PEM routines::no start line:Expecting: TRUSTED CERTIFICATE)

Being the same cert from our internal PKI, with previous files on volumes mounted on compose, also being system docker folder and also at ca-trust OS folder, this shouldn't be breaking or ?

Does the install needs something at folder where script is despite we've mapped /:hostfs and also the previous /registry/data has old secret ?

...
loaded secret from file: /data/secret/keys/secretkey
...

Did something change or we missed some step during the prepare with harbor.yml ?

In the end, we upgraded to 2.11.1 easily, just nginx not working.

Thanks in advance

[ntex777]

I found the underlying cause, after the first install some years ago, someone changed the filenames while renewing the cert last year.
So the file named as PEM was in fact binary and Binary filename was PEM, so when Upgrade (migrate step) picks these certs, was overwriting the good one PEM with a binary format instead.

Now, I'm still curious on this step, do you have to do prepare step when you need to renew certs ?

version=2.11.1
cd /harbor/$version
docker run --rm -v /harbor/$version:/input \
    -v /harbor/$version:/compose_location \
    -v /harbor/$version/common/config:/config \
    -v /registry/harbor/data:/data 
    -v /:/hostfs 
    --privileged goharbor/prepare:v$version prepare --with-trivy $@ 

Or we can just replace the ones in data/ and docker/certs.d and restart the stack ?

[wy65701436]

yes, you need to execute the prepare step once the certs is renewed.

[ntex777]

Thanks for confirming it.