Getting 404 Not Found with terraform

[mkazla]

Hi,

My tofu controller is using a terraform repository which basically creates confluent_kafka_topic, As a part of DR testing I am working on a scenario where I am trying to create the confluent topic on my primary cluster using the configuration.

For example I have cluster A (primary), configuration of this is passed to tofu and tofu creates a topic for me.

Now I try to switch the configuration from primary (cluster A) to secondary (cluster B) by editing my terraform object, it creates the topic for me on secondary cluster and updates the backend properly. But it also gives error on terraform describe as follows

Events:
  Type     Reason       Age                   From           Message
  ----     ------       ----                  ----           -------
  Warning  Progressing  7m7s (x120 over 90m)  tf-controller  Apply error: rpc error: code = Internal desc = exit status 1

Error: 404 Not Found: HTTP 404 Not Found

  with module.test_kafka_topic.confluent_kafka_topic.kafka_topic,
  on .terraform/modules/kafka_topic/modules/kafka-topic/main.tf line 1, in resource "confluent_kafka_topic" "kafka_topic":
   1: resource "confluent_kafka_topic" "kafka_topic" {
   

I tried replicating the same error with terraform by switching the configuration but terraform works fine, it is able to apply both configurations properly.

[ilithanos]

Just to make sure I understand this one correctly.

Does the plan + apply go through and apply the changes, but the describe is showing a 404, or does the 404 make the apply fail as well?

It's most likely some kind of networking issue either way, but since I haven't encountered it anywhere I'm trying to wrap my head around a way for us to replicate it to help fix it.

[mkazla]

Hey @ilithanos - Thanks for replying on the issue.

For me plan works fine, without any issues but on apply (which fails partially), it fails with 404 error above. which means, I see 404 when I describe terraform object. But on cluster tofu is able to create the resource.

Networking: May not be the issue since I have both the clusters in same region as part of testing.

error output:

Plan generated: set approvePlan: "plan1" to approve this plan.
  Normal  Progressing              75s (x2 over 35m)  tf-controller  Applied successfully
  Normal  TerraformAppliedSucceed  75s                tf-controller  Outputs written.
2 output(s): kafka_topic_name, kafka_topic_id
  Warning  Progressing  6s (x2 over 48s)  tf-controller  Apply error: rpc error: code = Internal desc = exit status 1

Error: 404 Not Found: HTTP 404 Not Found

  with module.test_kafka_topic.confluent_kafka_topic.kafka_topic,
  on .terraform/modules/test_kafka_topic/modules/kafka-topic/main.tf line 1, in resource "confluent_kafka_topic" "kafka_topic":
   1: resource "confluent_kafka_topic" "kafka_topic" {
  Warning  TFExecApplyFailed  6s (x2 over 48s)  tf-controller  error running Apply: rpc error: code = Internal desc = exit status 1

[mkazla]

I also put debug to the logLevel and gets following output

{"level":"error","ts":"2024-06-07T14:08:57.779Z","msg":"error applying","controller":"terraform","controllerGroup":"infra.contrib.fluxcd.io","controllerKind":"Terraform","Terraform":
{"name":"test-clusterlink1-kafka-topic-test-clusterlink1","namespace":"testl-sbx"},"namespace":"testl-sbx","name":"test-clusterlink1-kafka-topic-test-clusterlink1","reconcileID":"c5b3a4bd-5f64-4a1f-af00-2880ddadb193","reconciliation-loop-id":"91430eba-9bb4-48b4-acaa-25f92f39c502","start-time":"2024-06-07T14:08:36.085Z","error":"error running Apply: rpc error: code = Internal desc = exit status 1\n\nError: 404 Not Found: HTTP 404 Not Found\n\n  with module.test_kafka_topic.confluent_kafka_topic.kafka_topic,\n  on 

.terraform/modules/test_kafka_topic/modules/kafka-topic/main.tf line 1, in resource \"confluent_kafka_topic\" \"kafka_topic\":\n   1: resource \"confluent_kafka_topic\" \"kafka_topic\" {\n\n","stacktrace":"github.com/flux-iac/tofu-controller/controllers.(*TerraformReconciler).reconcile\n\t/workspace/controllers/tf_controller_reconcile.go:205\ngithub.com/flux-iac/tofu-

controller/controllers.(*TerraformReconciler).Reconcile\n\t/workspace/controllers/tf_controller.go:464\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.

(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/[email protected]/pkg/internal/controller/controller.go:227"}

[mkazla]

Hey @ilithanos - if you could please check the issue?

[ilithanos]

I havn't forgotten about you @mkazla but i havn't currently been able to replicate the issue you are having. It might be some environment specific issue.

I don't currently have a good suggestion as to what to look at though. It smells like a network issue with something that specific module is doing, but what it is i can't say for sure.