Ingress Stuck in Redeployment Loop After Being Removed

[kaykhan]

Describe the bug

On a fresh kubernetes cluster - I am deploying an ingress it successfully deploys, i wait for all resources to be up and im able to access the service.

I then delete the ingress and aws load balancer begins to delete and then shortly after initiates a redeployment on its own. It then becomes stuck in a loop of redeploying and attempting to delete a sg that no longer exists.

Steps to reproduce

1. Create

kubectl apply -f echoserver.ym

2. Wait until all resources are created and up.

3. Delete

kubectl delete -f echoserver.yml

Expected outcome

I expect it to not be redeployed.

Environment

• AWS Load Balancer controller version: 2.8.3
• Kubernetes version: 1.31
• Using EKS (yes/no), if so version?: YES - 1.31

Additional Context:

echoserver.yaml

apiVersion: v1
kind: Namespace
metadata:
  name: echoserver
---
apiVersion: v1
kind: Service
metadata:
  name: echoserver
  namespace: echoserver
spec:
  ports:
    - port: 80
      targetPort: 8080
      protocol: TCP
  type: NodePort
  selector:
    app: echoserver
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: echoserver
  namespace: echoserver
spec:
  selector:
    matchLabels:
      app: echoserver
  replicas: 3
  template:
    metadata:
      labels:
        app: echoserver
    spec:
      containers:
      - image: k8s.gcr.io/e2e-test-images/echoserver:2.5
        imagePullPolicy: Always
        name: echoserver
        ports:
        - containerPort: 8080
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: echoserver
  namespace: echoserver
  labels:
    app: echoserver
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/tags: Environment=dev,Team=test
    alb.ingress.kubernetes.io/certificate-arn: <redacted>
    alb.ingress.kubernetes.io/listen-ports: '[{"HTTP":80,"HTTPS": 443}]'
    alb.ingress.kubernetes.io/healthcheck-path: /health
    alb.ingress.kubernetes.io/ssl-redirect: '443'
    alb.ingress.kubernetes.io/ssl-policy: ELBSecurityPolicy-TLS-1-2-Ext-2018-06
spec:
  ingressClassName: alb
  rules:
    - host: echoserver.<redacted>.com
      http:
        paths:
          - path: /
            pathType: Exact
            backend:
              service:
                name: echoserver
                port:
                  number: 80

kubectl logs -f --selector=app.kubernetes.io/instance=aws-load-balancer-controller -n kube-system

{"level":"info","ts":"2024-09-30T13:58:40Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"echoserver/echoserver\",\"resources\":{\"AWS::EC2::SecurityGroup\":{\"ManagedLBSecurityGroup\":{\"spec\":{\"groupName\":\"k8s-echoserv-echoserv-27ee8b7486\",\"description\":\"[k8s] Managed SecurityGroup for LoadBalancer\",\"tags\":{\"Environment\":\"dev\",\"Team\":\"test\"},\"ingress\":[{\"ipProtocol\":\"tcp\",\"fromPort\":80,\"toPort\":80,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]},{\"ipProtocol\":\"tcp\",\"fromPort\":443,\"toPort\":443,\"ipRanges\":[{\"cidrIP\":\"0.0.0.0/0\"}]}]}}},\"AWS::ElasticLoadBalancingV2::Listener\":{\"443\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":443,\"protocol\":\"HTTPS\",\"defaultActions\":[{\"type\":\"fixed-response\",\"fixedResponseConfig\":{\"contentType\":\"text/plain\",\"statusCode\":\"404\"}}],\"certificates\":[{\"certificateARN\":\"arn:aws:acm:eu-west-2:<redacted_aws_id>:certificate/7f27a4da-b536-4186-b5b7-60a816d3347b\"}],\"sslPolicy\":\"ELBSecurityPolicy-TLS-1-2-Ext-2018-06\",\"tags\":{\"Environment\":\"dev\",\"Team\":\"test\"}}},\"80\":{\"spec\":{\"loadBalancerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::LoadBalancer/LoadBalancer/status/loadBalancerARN\"},\"port\":80,\"protocol\":\"HTTP\",\"defaultActions\":[{\"type\":\"redirect\",\"redirectConfig\":{\"port\":\"443\",\"protocol\":\"HTTPS\",\"statusCode\":\"HTTP_301\"}}],\"tags\":{\"Environment\":\"dev\",\"Team\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::ListenerRule\":{\"443:1\":{\"spec\":{\"listenerARN\":{\"$ref\":\"#/resources/AWS::ElasticLoadBalancingV2::Listener/443/status/listenerARN\"},\"priority\":1,\"actions\":[{\"type\":\"fixed-response\",\"fixedResponseConfig\":{\"contentType\":\"text/plain\",\"messageBody\":\"Backend service does not exist\",\"statusCode\":\"503\"}}],\"conditions\":[{\"field\":\"host-header\",\"hostHeaderConfig\":{\"values\":[\"echoserver.<redacted>-test.com\"]}},{\"field\":\"path-pattern\",\"pathPatternConfig\":{\"values\":[\"/\"]}}],\"tags\":{\"Environment\":\"dev\",\"Team\":\"test\"}}}},\"AWS::ElasticLoadBalancingV2::LoadBalancer\":{\"LoadBalancer\":{\"spec\":{\"name\":\"k8s-echoserv-echoserv-ef6dc8e92c\",\"type\":\"application\",\"scheme\":\"internet-facing\",\"ipAddressType\":\"ipv4\",\"subnetMapping\":[{\"subnetID\":\"subnet-078b1848a18d5a884\"},{\"subnetID\":\"subnet-09b0339ba7a3d8f00\"},{\"subnetID\":\"subnet-0b09d712e06effc8d\"}],\"securityGroups\":[{\"$ref\":\"#/resources/AWS::EC2::SecurityGroup/ManagedLBSecurityGroup/status/groupID\"},\"sg-08f68ec78c0022204\"],\"tags\":{\"Environment\":\"dev\",\"Team\":\"test\"}}}}}}"}
{"level":"info","ts":"2024-09-30T13:58:40Z","msg":"deRegistered targets","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:targetgroup/k8s-echoserv-echoserv-d57316ac4e/0fa63780c6cafe5d"}
{"level":"info","ts":"2024-09-30T13:58:40Z","msg":"revoking securityGroup ingress","securityGroupID":"sg-08a05473ebef099f4","permission":[{"FromPort":32519,"IpProtocol":"tcp","IpRanges":null,"Ipv6Ranges":null,"PrefixListIds":null,"ToPort":32519,"UserIdGroupPairs":[{"Description":"elbv2.k8s.aws/targetGroupBinding=shared","GroupId":"sg-08f68ec78c0022204","GroupName":null,"PeeringStatus":null,"UserId":"<redacted_aws_id>","VpcId":null,"VpcPeeringConnectionId":null}]}]}
{"level":"info","ts":"2024-09-30T13:58:40Z","msg":"revoked securityGroup ingress","securityGroupID":"sg-08a05473ebef099f4"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"modifying listener rule","stackID":"echoserver/echoserver","resourceID":"443:1","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:listener-rule/app/k8s-echoserv-echoserv-ef6dc8e92c/6fab3f126efc5ac1/7948e0ee7f7a6e5d/e9f7b4cfced877cb"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"modified listener rule","stackID":"echoserver/echoserver","resourceID":"443:1","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:listener-rule/app/k8s-echoserv-echoserv-ef6dc8e92c/6fab3f126efc5ac1/7948e0ee7f7a6e5d/e9f7b4cfced877cb"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"deleting targetGroupBinding","targetGroupBinding":{"name":"k8s-echoserv-echoserv-d57316ac4e","namespace":"echoserver"}}
{"level":"info","ts":"2024-09-30T13:58:41Z","msg":"deRegistering targets","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:targetgroup/k8s-echoserv-echoserv-d57316ac4e/0fa63780c6cafe5d","targets":[{"AvailabilityZone":null,"Id":"i-0e5924671a80ed6ca","Port":32519},{"AvailabilityZone":null,"Id":"i-0d290adc887d0a024","Port":32519},{"AvailabilityZone":null,"Id":"i-01a0d703fe4474dad","Port":32519},{"AvailabilityZone":null,"Id":"i-0b70e00e75787fd59","Port":32519},{"AvailabilityZone":null,"Id":"i-0f4a32103adf758be","Port":32519},{"AvailabilityZone":null,"Id":"i-0360cf2c212456096","Port":32519}]}
{"level":"info","ts":"2024-09-30T13:58:41Z","msg":"deRegistered targets","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:targetgroup/k8s-echoserv-echoserv-d57316ac4e/0fa63780c6cafe5d"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"deleted targetGroupBinding","targetGroupBinding":{"name":"k8s-echoserv-echoserv-d57316ac4e","namespace":"echoserver"}}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"deleting targetGroup","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:targetgroup/k8s-echoserv-echoserv-d57316ac4e/0fa63780c6cafe5d"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"deleted targetGroup","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:targetgroup/k8s-echoserv-echoserv-d57316ac4e/0fa63780c6cafe5d"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"echoserver/echoserver"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"echoserver/echoserver\",\"resources\":{}}"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"deleting loadBalancer","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:loadbalancer/app/k8s-echoserv-echoserv-ef6dc8e92c/6fab3f126efc5ac1"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"deleted loadBalancer","arn":"arn:aws:elasticloadbalancing:eu-west-2:<redacted_aws_id>:loadbalancer/app/k8s-echoserv-echoserv-ef6dc8e92c/6fab3f126efc5ac1"}
{"level":"info","ts":"2024-09-30T13:58:41Z","logger":"controllers.ingress","msg":"deleting securityGroup","securityGroupID":"sg-0b9dedd643442d039"}
{"level":"info","ts":"2024-09-30T13:59:14Z","logger":"controllers.ingress","msg":"deleted securityGroup","securityGroupID":"sg-0b9dedd643442d039"}
{"level":"info","ts":"2024-09-30T13:59:14Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"echoserver/echoserver"}
{"level":"error","ts":"2024-09-30T14:01:14Z","msg":"Reconciler error","controller":"ingress","object":{"name":"echoserver","namespace":"echoserver"},"namespace":"echoserver","name":"echoserver","reconcileID":"866b8109-f88b-4ca9-b9b0-609a3b371b6f","error":"failed to delete securityGroup: timed out waiting for the condition"}
{"level":"info","ts":"2024-09-30T14:01:14Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"echoserver/echoserver\",\"resources\":{}}"}
{"level":"info","ts":"2024-09-30T14:01:15Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"echoserver/echoserver"}
{"level":"error","ts":"2024-09-30T14:03:15Z","msg":"Reconciler error","controller":"ingress","object":{"name":"echoserver","namespace":"echoserver"},"namespace":"echoserver","name":"echoserver","reconcileID":"19ab9cb4-761a-4efc-afc8-2902196f77e3","error":"failed to delete securityGroup: timed out waiting for the condition"}
{"level":"info","ts":"2024-09-30T14:03:15Z","logger":"controllers.ingress","msg":"successfully built model","model":"{\"id\":\"echoserver/echoserver\",\"resources\":{}}"}
{"level":"info","ts":"2024-09-30T14:03:16Z","logger":"controllers.ingress","msg":"successfully deployed model","ingressGroup":"echoserver/echoserver"}

log file

logs-from-aws-load-balancer-controller-in-aws-load-balancer-controller-6b6457c65b-vs2tj.log

aws-load-balancer-controller configuration settings - installed via helm chart 1.8.4 - 2.8.3 app version

    {
      name  = "clusterName"
      value = local.k8s_cluster_name
    },
    {
      name  = "serviceAccount.create"
      value = false
    },
    {
      name  = "serviceAccount.name"
      value = local.k8s_service_account_name
    },
    {
      name  = "nodeSelector.acme/node-type"
      value = "worker"
    }

[huangm777]

Thanks for your report. We will look at this as soon as possible.

[huangm777]

/kind bug

[kaykhan]

Thanks for your report. We will look at this as soon as possible.

If it's of any value,

I've also tried with helm 1.8.0 and app version 2.8.0 and experience the same issue. Version 2.8.0 works on eks/kubernetea 1.30.

So I suspect this is an issue with the use of k8s 1.31.

I'm in the middle of preparing to out of place upgrade our cluster to 1.31 when I came across this issue.

Please let me know if there is anything else I can do/provide to help as this bug is blocking us from our upgrade.

Thank you

[peterbosalliandercom]

Same here. It got stuck deleting on this error: ,"error":"ingress: default/pax-custom-domain-certificate-k8ntw-wjpwd: no certificate found for host: xxx.dummy.nl"}

[peterbosalliandercom]

Reproductionscenario:

1. Create ingress for non existing certificate.
2. Create another ingress for non existing certificate.

Delete one ingress, it remains in deleting and will not finalize.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale