Cascading binding deletion #2711

piotrmiskiewicz · 2019-09-24T09:28:35Z

Implements proposal: #2734

Adds a cascading deletion feature which deletes existing service bindings for a service instance which is being deleted.

Which issue(s) this PR fixes
Fixes: #481 kyma-project/kyma#5180

k8s-ci-robot · 2019-09-24T09:28:58Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: piotrmiskiewicz

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [piotrmiskiewicz]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

piotrmiskiewicz · 2019-09-24T13:57:05Z

How to test:
Install the service-catalog with the CascadingDeletion feature flag.

Install the test-broker:

helm install charts/test-broker --name test-broker --namespace test-broker

and register it in the service-catalog:

cat <<EOF | kubectl apply -f -
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ClusterServiceBroker
metadata:
  name: test-broker
spec:
  url: http://test-broker-test-broker.test-broker.svc.cluster.local
EOF

Create a namespace:
kubectl create ns test-ns

Create an instance

cat <<EOF | kubectl apply -f -
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceInstance
metadata:
  name: test-instance
  namespace: test-ns
spec:
  clusterServiceClassExternalName: test-service
  clusterServicePlanExternalName: default
EOF

Create 3 bindings:

cat <<EOF | kubectl apply -f -
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  name: test-binding1
  namespace: test-ns
spec:
  instanceRef:
    name: test-instance
---
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  name: test-binding2
  namespace: test-ns
spec:
  instanceRef:
    name: test-instance
---
apiVersion: servicecatalog.k8s.io/v1beta1
kind: ServiceBinding
metadata:
  name: test-binding3
  namespace: test-ns
spec:
  instanceRef:
    name: test-instance     
EOF

Then, delete the instance:

kubectl delete -n test-ns serviceinstance test-instance

check events:

kubectl get events -n test-ns --sort-by=.metadata.creationTimestamp

piotrmiskiewicz · 2019-09-25T18:30:45Z

The PR implements service bindings deletion in the controller due to a business requirement - deprovision must be done after all unbind operations.
The idea of having ownerReference in the ServiceBinding instances and use K8s garbage collector does not work. The k8s garbage collector will try to delete bindings, when service instance is deleted, but it can be deleted, when all unbind operation is done.

MHBauer · 2019-09-27T18:41:47Z

OMG, this is part of the matrix of deleting things! #2229

jberkhahn · 2019-10-01T21:09:17Z

/retest

mszostok · 2019-10-01T21:33:42Z

I still want to discuss that in details

why kubectl delete .. --cascade=true cannot be used? Maybe we can reuse "propagationPolicy":"Foreground" concept but implement it on our own?
or why the spec was not changed? Maybe we should have a new field under ServiceInstance spec, sth like deletionPolicy. By default, it will have old behavior. If a user will change that then the cascading opt will be used.

The current solution is not backward compatible and user cannot select different strategies for different ServiceInstance. With current impl you enable/disable it globally. Let's discuss that if we want to implement this feature in that way.

/hold

piotrmiskiewicz · 2019-10-03T08:48:33Z

There is a feature flag which enables cascading deletion, so the change is backward compatible. The ownerReference cannot be used because of a problem when the delete is done with propagationPolicy=background. The binding must be deleted before the instance. The risk is someone could do kubectl delete serviceinstance ... without the --cascade=true flag

mszostok · 2019-10-03T15:17:09Z

Feature gates are a set of key=value pairs that describe alpha or experimental features and can be turned on or off

so it's not backward compatible because you cannot remove that feature flag and make that behavior as the core one because u will change the deletion policy for all clients when they will update to new service catalog version.

I'm talking about the "propagationPolicy":"Foreground" not the ownerReference. We can impl that GC on our own as we did here but based on propagationPolicy but I'm not fully convinced to do that in that way. It's just an option and maybe better will be to extend the ServiceInstance Spec. IMO it will be also worth to show those two approaches in sig-api-machinery and ask them about the recommendation.

let's discuss that offline and post here clear strategy for having that feature in ServiceCatalog:)

mszostok · 2019-10-16T16:02:59Z

docs/feature-gates.md

@@ -32,6 +32,7 @@ different Service Catalog.
 | `ResponseSchema` | `false` | Alpha | v0.1.12 | |
 | `ServicePlanDefaults` | `false` | Alpha | v0.1.32 | |
 | `UpdateDashboardURL` | `false` | Alpha | v0.1.13 | |
+| `CascadingDeletion` | ` false` | Alpha | v0.2.3 | |


mszostok · 2019-10-16T16:03:43Z

pkg/controller/controller_instance.go

+	bindingLister := c.bindingLister.ServiceBindings(instance.Namespace)
+
+	selector := labels.NewSelector()
+	bindingList, err := bindingLister.List(selector)


just inline all of those statements:

c.bindingLister.ServiceBindings(instance.Namespace).List(labels.NewSelector())

mszostok · 2019-10-16T16:05:26Z

pkg/controller/controller_instance.go

+	klog.V(4).Infof("Delete existing bindings for the instance %s", instance.Name)
+	bindings, err := c.listExistingBindings(instance)
+	if err != nil {
+		return err


u're missing context for each error
e.g. here u should have

errors.Wrapf(err, "while fetching ServiceBindings for instance %s", instance.Name)

I've added wrapping errors

mszostok · 2019-10-17T10:41:47Z

also, adjust the chart with the new param that can be set by our clients

piotrmiskiewicz · 2019-10-21T11:20:25Z

/test pull-build-all-images-for-s390x

mszostok · 2019-10-21T15:33:11Z

@piotrmiskiewicz you probably missed this comment #2711 (comment). The Helm Chart is still not adjusted, please add opt to enable it, sth similar what we have already: https://github.com/kubernetes-sigs/service-catalog/blob/master/charts/catalog/values.yaml#L105-L106

piotrmiskiewicz · 2019-10-24T13:46:32Z

/test pull-service-catalog-test-migration

mszostok · 2019-10-28T12:08:07Z

just add disable-cascading-deletion flag to Helm Chart beside that
/lgtm

more info can be found here: #2734

piotrmiskiewicz · 2019-11-04T12:32:08Z

/test pull-service-catalog-test-integration

jberkhahn · 2019-11-04T23:36:00Z

charts/catalog/values.yaml

@@ -82,6 +82,8 @@ controllerManager:
  serviceAccount: service-catalog-controller-manager
  # Whether the controller will expose metrics on /metrics
  enablePrometheusScrape: false
+  # Whether the cascading binding deletion is disabled, it overrides the CascadingDeletion feature gate


why the double feature gate? i dont really understand how this is supposed to work if I set one and not the other.

jberkhahn · 2019-11-04T23:56:59Z

so, i can't seem to get this to work. I set the "cascadingDeletionEnabled" flag to true, and leave the "cascadingDeletionDisabled" to false, as it's a negative, which should make this work? these flags are confusing. why do we not just have a single top level flag like the other feature gates?

I go to deprovision the instance, and it gets stuck in the "DeprovisionBlockedByExistingCredentials" state. I'm using 'svcat' for this, btw.

piotrmiskiewicz · 2019-11-05T09:23:31Z

As I understand the process, the feature gate will be removed in the future and the feature will be enabled by default. The second (disabling) flag is designed for all users which does not want this feature and it should be set explicitly. Long term the feature gate will be removed and the cascading deletion will be enabled by default. There will be only the cascadingDeletionDisabled flag.

Currently, the cascading deletion feature is an alpha feature. This is the reason to use a feature gate.
I understand the names in the values.yaml could be confusing: cascadingDeletionEnabled and cascadingDeletionDisabled. If you have better ideas I can change it.

piotrmiskiewicz · 2019-11-05T09:57:49Z

I've made a mistake in the chart, which I've just fixed.

jberkhahn · 2019-11-05T22:02:48Z

why do we not leave the to-be-used-in-the-future-flag out of the chart now, and add it it later, then? I also think the double negative is weird, wouldn't it make more sense to make it --enable-foobar and set it to true by default?

piotrmiskiewicz · 2019-11-06T07:34:38Z

I understand that two options may be confusing, at the beginning I had only the feature gate.
On the other hand, the whole feature "cascading binding deletion" is an alpha feature and needs to be explicitly enabled (by default disabled). After some time - it can be enabled by default.
Are you sure we can change the way how the controller works just with the next release? No "alpha" stage for few releases?

jberkhahn · 2019-11-06T18:15:02Z

you can leave the option on the actual controller itself, although I would prefer it was a simple "enable-feature-foobar" flag rather than a weird double-negative one. I'm just saying take the ability to set it out of the helm charts so people don't get confused about why there's two settings that look like they do the same thing.

piotrmiskiewicz · 2019-11-06T21:11:10Z

We want to have the cascading deletion feature enabled by default in the future.
Do you mean something like: --enable-cascading-deletion which is enabled by default.
I someone doesn't want to use this feature, he could pass: --enable-cascading-deletion=false

Sounds good, but the problem is we are adding such feature without an alpha stage and this feature breaks the backward compatibility.

piotrmiskiewicz · 2019-11-12T11:25:01Z

/retest

piotrmiskiewicz · 2019-11-12T13:04:12Z

I removed the "disabling" flag. Now, there is only enabling flag

jberkhahn · 2019-11-14T23:18:02Z

/lgtm

mszostok · 2019-11-15T09:28:31Z

/hold cancel

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Sep 24, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 24, 2019

k8s-ci-robot requested review from jberkhahn and mszostok September 24, 2019 09:29

piotrmiskiewicz force-pushed the cascading-binding-deletion branch from 04c89ba to c607665 Compare September 25, 2019 18:23

piotrmiskiewicz force-pushed the cascading-binding-deletion branch 2 times, most recently from 9f31d40 to 5f5e7ce Compare September 26, 2019 09:51

piotrmiskiewicz marked this pull request as ready for review September 26, 2019 10:51

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Sep 26, 2019

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 1, 2019

mszostok reviewed Oct 16, 2019

View reviewed changes

piotrmiskiewicz force-pushed the cascading-binding-deletion branch 2 times, most recently from 41086a1 to 64dbea6 Compare October 21, 2019 11:01

mszostok mentioned this pull request Oct 21, 2019

Service Instances cascading delete proposal #2734

Closed

piotrmiskiewicz force-pushed the cascading-binding-deletion branch 3 times, most recently from 6924d6a to db932a7 Compare October 24, 2019 13:23

k8s-ci-robot assigned mszostok Oct 28, 2019

k8s-ci-robot added lgtm Indicates that a PR is ready to be merged. and removed lgtm Indicates that a PR is ready to be merged. labels Oct 28, 2019

jberkhahn reviewed Nov 4, 2019

View reviewed changes

piotrmiskiewicz force-pushed the cascading-binding-deletion branch from 425023a to ee78615 Compare November 12, 2019 10:59

Cascading deletion

89cfea1

piotrmiskiewicz force-pushed the cascading-binding-deletion branch from de9f93f to e786c49 Compare November 12, 2019 11:09

Chart update

52d7f45

piotrmiskiewicz force-pushed the cascading-binding-deletion branch from e786c49 to 52d7f45 Compare November 12, 2019 11:33

k8s-ci-robot assigned jberkhahn Nov 14, 2019

k8s-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Nov 14, 2019

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Nov 15, 2019

k8s-ci-robot merged commit b6afbc9 into kubernetes-retired:master Nov 15, 2019

This was referenced Nov 17, 2019

Enable Cascading delete for ServiceInstances in Kyma kyma-project/kyma#6140

Closed

Cascading delete for ServiceInstances kyma-project/kyma#5180

Closed

mszostok mentioned this pull request May 25, 2020

Update version from 0.3.0-beta.2 to 0.3.0 #2811

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cascading binding deletion #2711

Cascading binding deletion #2711

piotrmiskiewicz commented Sep 24, 2019 •

edited by mszostok

Loading

k8s-ci-robot commented Sep 24, 2019

piotrmiskiewicz commented Sep 24, 2019 •

edited

Loading

piotrmiskiewicz commented Sep 25, 2019

MHBauer commented Sep 27, 2019

jberkhahn commented Oct 1, 2019

mszostok commented Oct 1, 2019 •

edited

Loading

piotrmiskiewicz commented Oct 3, 2019

mszostok commented Oct 3, 2019

mszostok Oct 16, 2019

mszostok Oct 16, 2019

mszostok Oct 16, 2019

piotrmiskiewicz Oct 21, 2019

mszostok commented Oct 17, 2019 •

edited

Loading

piotrmiskiewicz commented Oct 21, 2019

mszostok commented Oct 21, 2019 •

edited

Loading

piotrmiskiewicz commented Oct 24, 2019

mszostok commented Oct 28, 2019 •

edited

Loading

piotrmiskiewicz commented Nov 4, 2019

jberkhahn Nov 4, 2019 •

edited

Loading

jberkhahn commented Nov 4, 2019 •

edited

Loading

piotrmiskiewicz commented Nov 5, 2019

piotrmiskiewicz commented Nov 5, 2019

jberkhahn commented Nov 5, 2019

piotrmiskiewicz commented Nov 6, 2019

jberkhahn commented Nov 6, 2019

piotrmiskiewicz commented Nov 6, 2019

piotrmiskiewicz commented Nov 12, 2019

piotrmiskiewicz commented Nov 12, 2019

jberkhahn commented Nov 14, 2019

mszostok commented Nov 15, 2019

	\| `CascadingDeletion` \| ` false` \| Alpha \| v0.2.3 \| \|
	\| `CascadingDeletion` \| ` false` \| Alpha \| v0.3.0 \| \|

Cascading binding deletion #2711

Cascading binding deletion #2711

Conversation

piotrmiskiewicz commented Sep 24, 2019 • edited by mszostok Loading

k8s-ci-robot commented Sep 24, 2019

piotrmiskiewicz commented Sep 24, 2019 • edited Loading

piotrmiskiewicz commented Sep 25, 2019

MHBauer commented Sep 27, 2019

jberkhahn commented Oct 1, 2019

mszostok commented Oct 1, 2019 • edited Loading

piotrmiskiewicz commented Oct 3, 2019

mszostok commented Oct 3, 2019

mszostok Oct 16, 2019

Choose a reason for hiding this comment

mszostok Oct 16, 2019

Choose a reason for hiding this comment

mszostok Oct 16, 2019

Choose a reason for hiding this comment

piotrmiskiewicz Oct 21, 2019

Choose a reason for hiding this comment

mszostok commented Oct 17, 2019 • edited Loading

piotrmiskiewicz commented Oct 21, 2019

mszostok commented Oct 21, 2019 • edited Loading

piotrmiskiewicz commented Oct 24, 2019

mszostok commented Oct 28, 2019 • edited Loading

piotrmiskiewicz commented Nov 4, 2019

jberkhahn Nov 4, 2019 • edited Loading

Choose a reason for hiding this comment

jberkhahn commented Nov 4, 2019 • edited Loading

piotrmiskiewicz commented Nov 5, 2019

piotrmiskiewicz commented Nov 5, 2019

jberkhahn commented Nov 5, 2019

piotrmiskiewicz commented Nov 6, 2019

jberkhahn commented Nov 6, 2019

piotrmiskiewicz commented Nov 6, 2019

piotrmiskiewicz commented Nov 12, 2019

piotrmiskiewicz commented Nov 12, 2019

jberkhahn commented Nov 14, 2019

mszostok commented Nov 15, 2019

piotrmiskiewicz commented Sep 24, 2019 •

edited by mszostok

Loading

piotrmiskiewicz commented Sep 24, 2019 •

edited

Loading

mszostok commented Oct 1, 2019 •

edited

Loading

mszostok commented Oct 17, 2019 •

edited

Loading

mszostok commented Oct 21, 2019 •

edited

Loading

mszostok commented Oct 28, 2019 •

edited

Loading

jberkhahn Nov 4, 2019 •

edited

Loading

jberkhahn commented Nov 4, 2019 •

edited

Loading