[Issue] [aspire] azd deploy removes custom domain bindings, can't work around after 1.9

[mip1983]

Output from azd version
Run azd version and copy and paste the output here:

azd version 1.9.0 (commit 651394c)

Describe the bug

After running 'azd deploy' (locally or via CI/CD), custom domain bindings are removed from my container apps.

I was working around this using 'azd infra synth' as I've outlined here, with custom domains added to the output of the bicep files and referenced in the yaml files.

As of 1.9, the *.tmpl.yaml files have moved to the AppHost of the aspire project, and despite adding back in the yaml for the custom domain, it doesn't seem to be picking it up n working anymore.

To Reproduce
Deploy the aspire starter template. Set up a custom domain binding with a bring your own certificate on the deployed web app. Redeploy via azd having run azd infra synth. See domain binding disappear.

Expected behavior
Domain bindings should be retained, and ideally #1765 addressed so there's a clear way of doing this.

Environment
Information on your environment:
* .NET 8 Aspire preview 6
* IDE and version : Visual Studio 2022 10.0 Preview 7

Additional context
Add any other context about the problem here.

[ellismg]

As of 1.9, the *.tmpl.yaml files have moved to the AppHost of the aspire project, and despite adding back in the yaml for the custom domain, it doesn't seem to be picking it up n working anymore.

@mip1983 I'm a little surprised about this, the intention was to just move the files to the infra folder under the app host (named by the resource name) but allow customization there - I'll take a look as to why that isn't working.

Regarding #1765, the long term direction is to be able to do these customizations in C# code in the app host, instead of having to fall back to infra synth to make customizations, but the story is still being flushed out - it's a key priority for us post the initial GA of Aspire - we want to get to a place where you have full access to the IaC within the aspire app host so you can make customizations instead of needing to run azd infra synth and edit.

[ellismg]

@mip1983 I just did a quick experiment locally where I took the aspire starter app, ran azd infra synth followed by azd up and then modified the webfrontend.tmpl.yaml to add an extra environment variable (a simplification for me instead of setting up a custom domain and everything, but should still let us know if the changes were getting picked up or not) and after running azd deploy again I could see my changes were deployed.

You might consider running with --debug and re-directing stderr to a file like 2>debug.log. We print a log message about what file we are loading and reading:

2024/05/08 13:01:30 service_target_dotnet_containerapp.go:186: using container app manifest from /Users/matell/dd/ellismg/AspireStarterChanges/AspireStarterChanges.AppHost/infra/webfrontend.tmpl.yaml

If we can't find the file, we log a message saying we are generating the infrastructure again from the manifest - I wonder if that's what is happening here?

[mip1983]

Thanks @ellismg, I'll do some digging today with debug and see what I can see. I do have another instance of me and someone else finding it's not picking things up from yaml files in #3597, so maybe it is being generated again.

[mip1983]

I've tried it running azd deploy locally with debug and my domain binding remains. I can see in the debug output:

2024/05/10 11:41:31 service_target_dotnet_containerapp.go:186: using container app manifest from 
D:\source\ecoDriverAnalytics\ecoDriverAnalytics.Aspire\ecoDriverAnalytics.Aspire.AppHost\infra\ecodriver-analytics-api.tmpl.yaml

Which looks correct.

However, when I try it from Azure DevOps, the domain binding is removed again, the output on the Azure DevOps console is:

2024/05/10 10:52:02 service_target_dotnet_containerapp.go:195: generating container app manifest from /home/vsts/work/1/s/.\ecoDriverAnalytics.Aspire\ecoDriverAnalytics.Aspire.AppHost\ecoDriverAnalytics.Aspire.AppHost.csproj for project ecodriver-analytics-api

yaml files are checked in, I've tried copy to output to see if that makes a difference, but no luck.

[mip1983]

@ellismg

Looking at the manifest path where it's not picked up the file running the DevOps pipeline, it looks a bit odd with the different slash directions, so I wondered if it was working locally on windows 11, but having trouble on the build agent which is ubuntu.

I've just tried updating my Azure DevOps build pipeline to use 'windows_latest' rather than 'ubuntu_latest', and that seems to work, my app has deployed without losing the custom domain binding set in my yaml file.

Hopefully that helps you track this down, seems like a linux path issue.

[vhvb1989]

@mip1983 , you can now use azd config set alpha.aca.persistDomains on to make azd to persist your domains w/o requiring to infra synth. See: #3955

[mip1983]

Thanks @vhvb1989, I'll give that a try. Do you know if the problem with it not finding the manifest on the linux build agent has been resolved as well? Still using a windows agent at the moment.

[mip1983]

/unresolve

Hi @vhvb1989, I've been running with 'azd config set alpha.aca.persistDomains on' configured in my build pipeline for a little while and it's generally worked until now.

Just did a build/release just now, one environment was fine but on production it took out the domain name binding for one of the container apps (certificate was still in place on the container apps environment and the binding was still there on the other container app).

Both are using a wildcard certificate uploaded to the container apps environment, both are using 'A' records and the domain names validate in the UI.

[vhvb1989]

Thanks @vhvb1989, I'll give that a try. Do you know if the problem with it not finding the manifest on the linux build agent has been resolved as well? Still using a windows agent at the moment.

Still an issue :(

[rajeshkamal5050]

Marking this a complete for persisting the custom domains.

We can use #3891 for tracking the CI/CD pipeline path issue.