[Bug] Old RayServices not deleted after operator update to 1.2.1

[aravind-anantha]

Search before asking

• I searched the issues and found no similar issues.

KubeRay Component

ray-operator

What happened + What you expected to happen

After kuberay operator update to 1.2.1 new ray services were created and are up and running as expected. But, the old ray services never got deleted. The expected behavior is for the old services to be torn down once the new ones are up but it did not happen.

Reproduction script

Following instructions from https://docs.ray.io/en/latest/cluster/kubernetes/user-guides/upgrade-guide.html to upgrade the operator to 1.2.1

Reconciling the ingress and service resources on the active Ray cluster. No pending Ray cluster found.

^ is a message in the logs but the old cluster exists and is still dangling.

Anything else

No response

Are you willing to submit a PR?

• Yes I am willing to submit a PR!

[andrewsykim]

The expected behavior is for the old services to be torn down once the new ones are up but it did not happen.

To clarify, do you mean old RayService or Service?

[aravind-anantha]

The expected behavior is for the old services to be torn down once the new ones are up but it did not happen.

To clarify, do you mean old RayService or Service?

Yes, I meant the old RayService

[andrewsykim]

I don't think upgrading kuberay to v1.2.1 is suppose to delete exist RayService objects. Can you share a link to code or documentation that implies that we do this?

[andrewsykim]

Or are you referring to the "old" RayCluster used for the RayService?

[aravind-anantha]

Or are you referring to the "old" RayCluster used for the RayService?

Yes, the old cluster being used for the RayService is not being deleted.

[andrewsykim]

I believe that's expected, if you upgrade kuberay without changing any fields of the RayService, the RayCluster should be not replaced

[aravind-anantha]

I believe that's expected, if you upgrade kuberay without changing any fields of the RayService, the RayCluster should be not replaced

Oh, we also made another change to the resources, specifically the amount of memory the headGroup requests. Could it be because both happened around the same time something went wrong?

[andrewsykim]

Yes, if you change some property of the RayService, it will trigger a replacement of the RayCluster. However, this doesn't apply for all fields like minReplicas and maxReplicas