feat: k8s-worker3/k8s-worker5をiBGPフルメッシュに参加させる

[kigawa01]

Summary

kinfraのR2変更検出フィルタ実地検証を通じて、Kubernetes Pod（CI Runner含む）がDNS VIP（10.0.0.53、iBGP経由で広告）に到達できないことが判明した。原因は、iBGPフルメッシュがk8s1/k8s2/k8s4のコントロールプレーンのみで構成されており、ワーカーノード（k8s-worker3/k8s-worker5）がメッシュに参加していないため。

docs/network.mdには既にk8s-worker3(10.0.0.30)/k8s-worker5(10.0.0.40)向けの静的IPが予約済みだったが、実際にはまだ割り当てられておらず、BGPモジュールも呼び出されていなかった。

⚠️⚠️ 重要: このPRは実運用中の物理サーバーのネットワークインターフェース設定を変更します ⚠️⚠️

• null_resource.bgp_static_ipが、各ワーカーの既存netplanファイルに直接セカンダリIPを追記する（新規drop-inファイルではなく、既存ファイルへの追記+バックアップ）。既存のプライマリIP（192.168.1.x/24）を上書きしないよう設計・ローカルでdry-run検証済みだが、実機での適用は必ずplanで差分を確認してから、可能であれば1台ずつapplyすることを強く推奨
• k8s-worker3は現在DiskPressure状態（別件、本PRとは無関係）。ディスク容量に余裕がない状態での追加変更はリスクが上がる可能性があるため、先に確認することを推奨
• 万一SSH接続を失った場合に備え、変更前に物理コンソール等の代替アクセス手段を確保してから適用することを推奨

Changes

• hardware/k8s-worker3, hardware/k8s-worker5:
  • null_resource.bgp_static_ip: 既存プライマリIPを維持したまま、セカンダリIP（var.server_ip）をnetplan経由で追加（バックアップ付き、追記のみ、netplan generateで検証後にapply）
  • module "bgp"（bgp-bird）呼び出しを追加
  • k8s-worker3の既存だが未配線だったbgp_local_as/bgp_peers変数を修正（AS番号を他ノードと同じ65000に、型をlist(string)に統一）。k8s-worker5には同等の変数を新規追加
• hardware/{k8s1,k8s2,k8s4}/variables.tf: bgp_peersのデフォルト値に10.0.0.30/10.0.0.40を追加し、5ノードのフルメッシュに拡張

Test plan

• terraform init -backend=false && terraform validate を影響する5モジュール全てで成功確認
• netplanへのsed追記ロジックを、実際にk8s-worker3から読み取った既存netplan内容のコピーに対してローカルでdry-run実行し、既存アドレスを保持したまま正しく追記されYAML構文も有効であることを確認
• マージ後、まずk8s-worker5（DiskPressureが無い方）でterraform plan→慎重にapplyし、SSH接続とBGPピアリングを確認してからk8s-worker3に進める
• 5ノード全体でBGPピアリングが確立し、DNS VIP(10.0.0.53)への到達性がPodから確認できることを確認
• terraform-kinfra.ymlを再実行し、k8s-worker3のplanが完走することを確認

🤖 Generated with Claude Code

[github-actions]

Terraform Plan hardware/k8s-worker3

data.external.sudo_password: Reading...
data.external.join_info: Reading...
data.external.ssh_key: Reading...
data.external.sudo_password: Read complete after 0s [id=-]
data.external.ssh_key: Read complete after 0s [id=-]
data.external.join_info: Read complete after 1s [id=-]
null_resource.worker_node: Refreshing state... [id=1921248082511887334]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement

Terraform planned the following actions, but then encountered a problem:

  # null_resource.bgp_static_ip will be created
  + resource "null_resource" "bgp_static_ip" {
      + id       = (known after apply)
      + triggers = {
          + "server_ip" = "10.0.0.30"
        }
    }

  # null_resource.worker_node must be replaced
-/+ resource "null_resource" "worker_node" {
      ~ id       = "1921248082511887334" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "ca_cert_hash"  = "sha256:6115df9d7965c06d99704d9b20a4d2783c2293c729415528429795f2aa8566e8" -> ""
          ~ "token"         = "nb388k.zbhl56o3564uzl7f" -> ""
            # (2 unchanged elements hidden)
        }
    }

  # module.bgp.null_resource.bird will be created
  + resource "null_resource" "bird" {
      + id       = (known after apply)
      + triggers = {
          + "advertised_vips" = ""
          + "bgp_loopback_ip" = "127.0.0.2"
          + "bird_conf"       = <<-EOT
                log syslog all;
                
                router id 10.0.0.30;
                
                protocol device {}
                
                protocol direct {
                  ipv4;
                }
                
                protocol kernel {
                  ipv4 {
                    export all;
                    import all;
                  };
                  learn;
                  persist;
                }
                
                protocol bgp peer0 {
                  local 10.0.0.30 as 65000;
                  neighbor 10.0.0.103 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer1 {
                  local 10.0.0.30 as 65000;
                  neighbor 10.0.0.120 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer2 {
                  local 10.0.0.30 as 65000;
                  neighbor 10.0.0.140 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer3 {
                  local 10.0.0.30 as 65000;
                  neighbor 10.0.0.40 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                
                
                protocol bgp kube_vip {
                  local 127.0.0.2 as 65000;
                  neighbor 127.0.0.1 as 65001;
                  multihop;
                  passive on;
                  ipv4 {
                    import filter {
                      bgp_next_hop = 10.0.0.30;
                      accept;
                    };
                    export none;
                  };
                }
            EOT
          + "external_peers"  = jsonencode([])
          + "host"            = "192.168.1.30"
          + "pod_yaml"        = <<-EOT
                apiVersion: v1
                kind: Pod
                metadata:
                  name: bird
                  namespace: kube-system
                  annotations:
                    config-hash: "19417ca4a0cf57fae23871c4d463e93fae63d9fe51ff28e1748fb78194da53f8"
                spec:
                  hostNetwork: true
                  priorityClassName: system-node-critical
                  containers:
                  - name: bird
                    image: kathara/bird2:latest
                    command: ["bird", "-f", "-c", "/etc/bird/bird.conf"]
                    securityContext:
                      capabilities:
                        add:
                        - NET_ADMIN
                        - NET_RAW
                    volumeMounts:
                    - name: bird-config
                      mountPath: /etc/bird
                      readOnly: true
                    - name: bird-run
                      mountPath: /var/run/bird
                  volumes:
                  - name: bird-config
                    hostPath:
                      path: /etc/bird
                      type: DirectoryOrCreate
                  - name: bird-run
                    hostPath:
                      path: /var/run/bird
                      type: DirectoryOrCreate
            EOT
        }
    }

Plan: 3 to add, 0 to change, 1 to destroy.

Error: Resource postcondition failed

  on main.tf line 149, in resource "null_resource" "worker_node":
 149:       condition     = length(self.triggers.token) > 0
    ├────────────────
    │ self.triggers.token is ""

join_info did not return a token

Error: Resource postcondition failed

  on main.tf line 153, in resource "null_resource" "worker_node":
 153:       condition     = length(self.triggers.ca_cert_hash) > 0
    ├────────────────
    │ self.triggers.ca_cert_hash is ""

join_info did not return a ca_cert_hash

[github-actions]

Terraform Plan hardware/k8s-worker5

data.external.sudo_password: Reading...
data.external.join_info: Reading...
data.external.ssh_key: Reading...
data.external.ssh_key: Read complete after 1s [id=-]
data.external.sudo_password: Read complete after 1s [id=-]
data.external.join_info: Read complete after 2s [id=-]
null_resource.worker_node: Refreshing state... [id=2183205705926198087]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # null_resource.bgp_static_ip will be created
  + resource "null_resource" "bgp_static_ip" {
      + id       = (known after apply)
      + triggers = {
          + "server_ip" = "10.0.0.40"
        }
    }

  # module.bgp.null_resource.bird will be created
  + resource "null_resource" "bird" {
      + id       = (known after apply)
      + triggers = {
          + "advertised_vips" = ""
          + "bgp_loopback_ip" = "127.0.0.2"
          + "bird_conf"       = <<-EOT
                log syslog all;
                
                router id 10.0.0.40;
                
                protocol device {}
                
                protocol direct {
                  ipv4;
                }
                
                protocol kernel {
                  ipv4 {
                    export all;
                    import all;
                  };
                  learn;
                  persist;
                }
                
                protocol bgp peer0 {
                  local 10.0.0.40 as 65000;
                  neighbor 10.0.0.103 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer1 {
                  local 10.0.0.40 as 65000;
                  neighbor 10.0.0.120 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer2 {
                  local 10.0.0.40 as 65000;
                  neighbor 10.0.0.140 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer3 {
                  local 10.0.0.40 as 65000;
                  neighbor 10.0.0.30 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                
                
                protocol bgp kube_vip {
                  local 127.0.0.2 as 65000;
                  neighbor 127.0.0.1 as 65001;
                  multihop;
                  passive on;
                  ipv4 {
                    import filter {
                      bgp_next_hop = 10.0.0.40;
                      accept;
                    };
                    export none;
                  };
                }
            EOT
          + "external_peers"  = jsonencode([])
          + "host"            = "192.168.1.150"
          + "pod_yaml"        = <<-EOT
                apiVersion: v1
                kind: Pod
                metadata:
                  name: bird
                  namespace: kube-system
                  annotations:
                    config-hash: "82e47dd030fedd4c55417cb10c4e5a73049f798f17e2e227ae6f766f9663f8b3"
                spec:
                  hostNetwork: true
                  priorityClassName: system-node-critical
                  containers:
                  - name: bird
                    image: kathara/bird2:latest
                    command: ["bird", "-f", "-c", "/etc/bird/bird.conf"]
                    securityContext:
                      capabilities:
                        add:
                        - NET_ADMIN
                        - NET_RAW
                    volumeMounts:
                    - name: bird-config
                      mountPath: /etc/bird
                      readOnly: true
                    - name: bird-run
                      mountPath: /var/run/bird
                  volumes:
                  - name: bird-config
                    hostPath:
                      path: /etc/bird
                      type: DirectoryOrCreate
                  - name: bird-run
                    hostPath:
                      path: /var/run/bird
                      type: DirectoryOrCreate
            EOT
        }
    }

  # module.node_exporter.null_resource.node_exporter will be created
  + resource "null_resource" "node_exporter" {
      + id       = (known after apply)
      + triggers = {
          + "host"           = "192.168.1.150"
          + "listen_address" = ":9100"
          + "version"        = "1.7.0"
        }
    }

Plan: 3 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

[github-actions]

Terraform Plan hardware/k8s1

data.external.join_info: Reading...
data.external.sudo_password: Reading...
data.external.ssh_key: Reading...
data.external.sudo_password: Read complete after 1s [id=-]
data.external.ssh_key: Read complete after 1s [id=-]
module.control_plane.null_resource.disable_swap: Refreshing state... [id=2884415110834909690]
module.wireguard[0].null_resource.wireguard: Refreshing state... [id=6656762921329990223]
module.keepalived.null_resource.keepalived: Refreshing state... [id=7637171177745551209]
data.external.join_info: Read complete after 2s [id=-]
module.control_plane.null_resource.control_plane: Refreshing state... [id=8957225143565195658]
module.kube_vip.null_resource.kube_vip: Refreshing state... [id=3972940073172515771]
module.bgp.null_resource.bird: Refreshing state... [id=7710895868079908516]
module.knot.null_resource.knot: Refreshing state... [id=1087168737206409085]
module.knot_resolver.null_resource.knot_resolver: Refreshing state... [id=6307309063132079256]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.bgp.null_resource.bird must be replaced
-/+ resource "null_resource" "bird" {
      ~ id       = "7710895868079908516" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "bird_conf"       = <<-EOT
                log syslog all;
                
                router id 10.0.0.103;
                
                protocol device {}
                
                protocol direct {
                  ipv4;
                }
                
                protocol kernel {
                  ipv4 {
                    export all;
                    import all;
                  };
                  learn;
                  persist;
                }
                
                protocol bgp peer0 {
                  local 10.0.0.103 as 65000;
                  neighbor 10.0.0.120 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer1 {
                  local 10.0.0.103 as 65000;
                  neighbor 10.0.0.140 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
              + protocol bgp peer2 {
              +   local 10.0.0.103 as 65000;
              +   neighbor 10.0.0.30 as 65000;
              +   ipv4 {
              +     import all;
              +     export all;
              +   };
              + }
              + 
              + protocol bgp peer3 {
              +   local 10.0.0.103 as 65000;
              +   neighbor 10.0.0.40 as 65000;
              +   ipv4 {
              +     import all;
              +     export all;
              +   };
              + }
              + 
                protocol static local_vips {
                  ipv4;
                  route 10.0.0.53/32 blackhole;
                }
                
                
                
                protocol bgp kube_vip {
                  local 127.0.0.2 as 65000;
                  neighbor 127.0.0.1 as 65001;
                  multihop;
                  passive on;
                  ipv4 {
                    import filter {
                      bgp_next_hop = 10.0.0.103;
                      accept;
                    };
                    export none;
                  };
                }
            EOT
          ~ "pod_yaml"        = <<-EOT
                apiVersion: v1
                kind: Pod
                metadata:
                  name: bird
                  namespace: kube-system
                  annotations:
              -     config-hash: "373b8025e9a79422295cd497128cbacaaecfe0a1c228f173a2ec738b90e10eeb"
              +     config-hash: "9bd353e67d79fe2b7db120f76e8ffff8c90a683db7649317265fa78793318934"
                spec:
                  hostNetwork: true
                  priorityClassName: system-node-critical
                  containers:
                  - name: bird
                    image: kathara/bird2:latest
                    command: ["bird", "-f", "-c", "/etc/bird/bird.conf"]
                    securityContext:
                      capabilities:
                        add:
                        - NET_ADMIN
                        - NET_RAW
                    volumeMounts:
                    - name: bird-config
                      mountPath: /etc/bird
                      readOnly: true
                    - name: bird-run
                      mountPath: /var/run/bird
                  volumes:
                  - name: bird-config
                    hostPath:
                      path: /etc/bird
                      type: DirectoryOrCreate
                  - name: bird-run
                    hostPath:
                      path: /var/run/bird
                      type: DirectoryOrCreate
            EOT
            # (4 unchanged elements hidden)
        }
    }

  # module.node_exporter.null_resource.node_exporter will be created
  + resource "null_resource" "node_exporter" {
      + id       = (known after apply)
      + triggers = {
          + "host"           = "192.168.1.103"
          + "listen_address" = ":9100"
          + "version"        = "1.7.0"
        }
    }

  # module.wireguard[0].null_resource.wireguard will be destroyed
  # (because module.wireguard[0] is not in configuration)
  - resource "null_resource" "wireguard" {
      - id       = "6656762921329990223" -> null
      - triggers = {
          - "allowed_ips"       = "172.31.255.0/24"
          - "host"              = "192.168.1.103"
          - "server_endpoint"   = "161.248.62.66:51820"
          - "server_public_key" = "70061113b24f5ac462235f3907f5c5ceb69b318024f6465c4d1ad4ce77a348b3"
          - "setup_version"     = "3"
          - "wireguard_address" = "172.31.255.11/24"
        } -> null
    }

Plan: 2 to add, 0 to change, 2 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

[github-actions]

Terraform Plan hardware/k8s2

data.external.ssh_key: Reading...
data.external.join_info: Reading...
data.external.sudo_password: Reading...
data.external.sudo_password: Read complete after 1s [id=-]
data.external.ssh_key: Read complete after 1s [id=-]
module.control_plane.null_resource.disable_swap: Refreshing state... [id=2078522867301445248]
module.keepalived.null_resource.keepalived: Refreshing state... [id=77951099548562352]
module.wireguard[0].null_resource.wireguard: Refreshing state... [id=2269961954052301461]
data.external.join_info: Read complete after 2s [id=-]
module.control_plane.null_resource.control_plane: Refreshing state... [id=3058245187027734364]
module.knot.null_resource.knot: Refreshing state... [id=298362026524776807]
module.kube_vip.null_resource.kube_vip: Refreshing state... [id=4617693817191532127]
module.bgp.null_resource.bird: Refreshing state... [id=6967168182303976108]
module.knot_resolver.null_resource.knot_resolver: Refreshing state... [id=4610183504413275223]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # module.bgp.null_resource.bird must be replaced
-/+ resource "null_resource" "bird" {
      ~ id       = "6967168182303976108" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "bird_conf"       = <<-EOT
                log syslog all;
                
                router id 10.0.0.120;
                
                protocol device {}
                
                protocol direct {
                  ipv4;
                }
                
                protocol kernel {
                  ipv4 {
                    export all;
                    import all;
                  };
                  learn;
                  persist;
                }
                
                protocol bgp peer0 {
                  local 10.0.0.120 as 65000;
                  neighbor 10.0.0.103 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer1 {
                  local 10.0.0.120 as 65000;
                  neighbor 10.0.0.140 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
              + protocol bgp peer2 {
              +   local 10.0.0.120 as 65000;
              +   neighbor 10.0.0.30 as 65000;
              +   ipv4 {
              +     import all;
              +     export all;
              +   };
              + }
              + 
              + protocol bgp peer3 {
              +   local 10.0.0.120 as 65000;
              +   neighbor 10.0.0.40 as 65000;
              +   ipv4 {
              +     import all;
              +     export all;
              +   };
              + }
              + 
                protocol static local_vips {
                  ipv4;
                  route 10.0.0.53/32 blackhole;
                }
                
                
                
                protocol bgp kube_vip {
                  local 127.0.0.2 as 65000;
                  neighbor 127.0.0.1 as 65001;
                  multihop;
                  passive on;
                  ipv4 {
                    import filter {
                      bgp_next_hop = 10.0.0.120;
                      accept;
                    };
                    export none;
                  };
                }
            EOT
          ~ "pod_yaml"        = <<-EOT
                apiVersion: v1
                kind: Pod
                metadata:
                  name: bird
                  namespace: kube-system
                  annotations:
              -     config-hash: "74a4f0966ffb6e4f1e4147f02e8c8cd34b2a812752bda1e53366099e7a08343d"
              +     config-hash: "609f91de726294455a43caf0f1916517250d1c65049b61e0105b9756251bc27f"
                spec:
                  hostNetwork: true
                  priorityClassName: system-node-critical
                  containers:
                  - name: bird
                    image: kathara/bird2:latest
                    command: ["bird", "-f", "-c", "/etc/bird/bird.conf"]
                    securityContext:
                      capabilities:
                        add:
                        - NET_ADMIN
                        - NET_RAW
                    volumeMounts:
                    - name: bird-config
                      mountPath: /etc/bird
                      readOnly: true
                    - name: bird-run
                      mountPath: /var/run/bird
                  volumes:
                  - name: bird-config
                    hostPath:
                      path: /etc/bird
                      type: DirectoryOrCreate
                  - name: bird-run
                    hostPath:
                      path: /var/run/bird
                      type: DirectoryOrCreate
            EOT
            # (4 unchanged elements hidden)
        }
    }

  # module.node_exporter.null_resource.node_exporter will be created
  + resource "null_resource" "node_exporter" {
      + id       = (known after apply)
      + triggers = {
          + "host"           = "192.168.1.20"
          + "listen_address" = ":9100"
          + "version"        = "1.7.0"
        }
    }

  # module.wireguard[0].null_resource.wireguard will be destroyed
  # (because module.wireguard[0] is not in configuration)
  - resource "null_resource" "wireguard" {
      - id       = "2269961954052301461" -> null
      - triggers = {
          - "allowed_ips"       = "172.31.255.0/24"
          - "host"              = "192.168.1.20"
          - "server_endpoint"   = "161.248.62.66:51820"
          - "server_public_key" = "70061113b24f5ac462235f3907f5c5ceb69b318024f6465c4d1ad4ce77a348b3"
          - "setup_version"     = "3"
          - "wireguard_address" = "172.31.255.12/24"
        } -> null
    }

Plan: 2 to add, 0 to change, 2 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

[github-actions]

Terraform Plan hardware/k8s4

data.external.inuyama_wireguard_public_key: Reading...
data.external.join_info: Reading...
data.external.sudo_password: Reading...
data.external.ssh_key: Reading...
data.external.inuyama_wireguard_private_key: Reading...
data.external.inuyama_wireguard_private_key: Read complete after 0s [id=-]
data.external.ssh_key: Read complete after 0s [id=-]
data.external.sudo_password: Read complete after 0s [id=-]
module.control_plane.null_resource.disable_swap: Refreshing state... [id=2329390693052091540]
data.external.inuyama_wireguard_public_key: Read complete after 1s [id=-]
data.external.join_info: Read complete after 2s [id=-]
module.control_plane.null_resource.control_plane: Refreshing state... [id=5585367606577211439]
null_resource.alice_gateway_services: Refreshing state... [id=1072532008218401708]
module.kube_vip.null_resource.kube_vip: Refreshing state... [id=6160400128829816737]
null_resource.inuyama_wireguard: Refreshing state... [id=3885716026194835594]
module.knot.null_resource.knot: Refreshing state... [id=6183639464161112917]
module.knot_resolver.null_resource.knot_resolver: Refreshing state... [id=420020468497207407]
module.bgp.null_resource.bird: Refreshing state... [id=158986863230925580]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
-/+ destroy and then create replacement

Terraform will perform the following actions:

  # null_resource.alice_gateway_services must be replaced
-/+ resource "null_resource" "alice_gateway_services" {
      ~ id       = "1072532008218401708" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "ingress_vip"            = "192.168.1.240" -> "10.0.0.240"
          ~ "metallb_base_range"     = "192.168.1.50-192.168.1.99" -> "10.0.0.50-10.0.0.99"
          ~ "metallb_reserved_range" = "192.168.1.240-192.168.1.249" -> "10.0.0.240-10.0.0.249"
          ~ "minecraft_vip"          = "192.168.1.241" -> "10.0.0.241"
            # (3 unchanged elements hidden)
        }
    }

  # module.bgp.null_resource.bird must be replaced
-/+ resource "null_resource" "bird" {
      ~ id       = "158986863230925580" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "advertised_vips" = "192.168.1.53" -> "10.0.0.53"
          ~ "bird_conf"       = <<-EOT
                log syslog all;
                
              - router id 192.168.1.120;
              + router id 10.0.0.140;
                
                protocol device {}
                
                protocol direct {
                  ipv4;
                }
                
                protocol kernel {
                  ipv4 {
                    export all;
                    import all;
                  };
                  learn;
                  persist;
                }
                
                protocol bgp peer0 {
              -   local 192.168.1.120 as 65000;
              -   neighbor 192.168.1.103 as 65000;
              +   local 10.0.0.140 as 65000;
              +   neighbor 10.0.0.103 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
                protocol bgp peer1 {
              -   local 192.168.1.120 as 65000;
              -   neighbor 192.168.1.20 as 65000;
              +   local 10.0.0.140 as 65000;
              +   neighbor 10.0.0.120 as 65000;
                  ipv4 {
                    import all;
                    export all;
                  };
                }
                
              + protocol bgp peer2 {
              +   local 10.0.0.140 as 65000;
              +   neighbor 10.0.0.30 as 65000;
              +   ipv4 {
              +     import all;
              +     export all;
              +   };
              + }
              + 
              + protocol bgp peer3 {
              +   local 10.0.0.140 as 65000;
              +   neighbor 10.0.0.40 as 65000;
              +   ipv4 {
              +     import all;
              +     export all;
              +   };
              + }
              + 
                protocol static local_vips {
                  ipv4;
              -   route 192.168.1.53/32 blackhole;
              +   route 10.0.0.53/32 blackhole;
                }
                
                protocol bgp external0 {
                  local 172.31.255.1 as 65010;
                  neighbor 172.31.255.2 as 65020;
                  ipv4 {
                    import filter {
                
                      reject;
                    };
                    export filter {
              -       if net = 192.168.1.0/24 then accept;
              +       if net = 10.0.0.0/16 then accept;
                      reject;
                    };
                  };
                }
                
                protocol bgp kube_vip {
                  local 127.0.0.2 as 65000;
                  neighbor 127.0.0.1 as 65001;
                  multihop;
                  passive on;
                  ipv4 {
                    import filter {
              -       bgp_next_hop = 192.168.1.120;
              +       bgp_next_hop = 10.0.0.140;
                      accept;
                    };
                    export none;
                  };
                }
            EOT
          ~ "external_peers"  = jsonencode(
              ~ [
                  ~ {
                      ~ export_prefixes = [
                          - "192.168.1.0/24",
                          + "10.0.0.0/16",
                        ]
                        # (5 unchanged attributes hidden)
                    },
                ]
            )
          ~ "pod_yaml"        = <<-EOT
                apiVersion: v1
                kind: Pod
                metadata:
                  name: bird
                  namespace: kube-system
                  annotations:
              -     config-hash: "db9aaa663fb169a42c76932ec34f14d43fcf2898df26a1082eb594c11b175fea"
              +     config-hash: "9c120d794c0369d48261662ab0415e99c0ee14679c0430209e02a03bd1349512"
                spec:
                  hostNetwork: true
                  priorityClassName: system-node-critical
                  containers:
                  - name: bird
                    image: kathara/bird2:latest
                    command: ["bird", "-f", "-c", "/etc/bird/bird.conf"]
                    securityContext:
                      capabilities:
                        add:
                        - NET_ADMIN
                        - NET_RAW
                    volumeMounts:
                    - name: bird-config
                      mountPath: /etc/bird
                      readOnly: true
                    - name: bird-run
                      mountPath: /var/run/bird
                  volumes:
                  - name: bird-config
                    hostPath:
                      path: /etc/bird
                      type: DirectoryOrCreate
                  - name: bird-run
                    hostPath:
                      path: /var/run/bird
                      type: DirectoryOrCreate
            EOT
            # (2 unchanged elements hidden)
        }
    }

  # module.keepalived.null_resource.keepalived will be created
  + resource "null_resource" "keepalived" {
      + id       = (known after apply)
      + triggers = {
          + "conf_hash" = "438a7827e984a7b9c95481d31fb7d7ccf146dd3927bdad5e3b64337d26914b47"
          + "host"      = "192.168.1.120"
        }
    }

  # module.knot_resolver.null_resource.knot_resolver must be replaced
-/+ resource "null_resource" "knot_resolver" {
      ~ id       = "420020468497207407" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "kresd_conf" = <<-EOT
                logging:
                  level: notice
                
                network:
                  listen:
                    - interface: "0.0.0.0@53"
                    - interface: "0.0.0.0@853"
                      kind: dot
              +     - interface: "10.0.0.53@53"
                
                cache:
                  size-max: 512M
                  storage: /var/cache/knot-resolver
                
                forward:
                  - subtree: "cluster.local."
                    servers:
                      - "10.96.0.10"
                  - subtree: "haproxy.kigawa.net."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "k8s.kigawa.net."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "mod.kigawa.net."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "oyu.kigawa.net."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "kizuna.kigawa.net."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "atm10.kigawa.net."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "kigawa.net."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "onemc.world."
                    servers:
                      - "127.0.0.1@5353"
                  - subtree: "."
                    servers:
              -       - 8.8.8.8
              -       - 8.8.4.4
                      - 1.1.1.1
                      - 1.0.0.1
              +       - 8.8.8.8
              +       - 8.8.4.4
            EOT
            # (1 unchanged element hidden)
        }
    }

  # module.kube_vip.null_resource.kube_vip must be replaced
-/+ resource "null_resource" "kube_vip" {
      ~ id       = "6160400128829816737" -> (known after apply)
      ~ triggers = { # forces replacement
          ~ "pod_yaml"  = <<-EOT
                apiVersion: v1
                kind: Pod
                metadata:
                  name: kube-vip
                  namespace: kube-system
                spec:
                  containers:
                  - name: kube-vip
                    image: ghcr.io/kube-vip/kube-vip:v0.8.9
                    imagePullPolicy: IfNotPresent
                    args:
                    - manager
                    env:
                    - name: vip_arp
              -       value: "true"
              +       value: "false"
                    - name: bgp_enable
                      value: "true"
                    - name: bgp_routerid
              -       value: 192.168.1.100
              +       value: 10.0.0.100
                    - name: bgp_as
                      value: "65001"
                    - name: bgp_peeraddress
                      value: "127.0.0.2"
                    - name: bgp_peeras
                      value: "65000"
                    - name: PORT
                      value: "6443"
                    - name: vip_interface
                      value: ens18
                    - name: address
              -       value: 192.168.1.100
              +       value: 10.0.0.100
                    - name: cp_enable
                      value: "true"
                    - name: cp_namespace
                      value: kube-system
                    - name: vip_leaderelection
                      value: "true"
                    securityContext:
                      capabilities:
                        add:
                        - NET_ADMIN
                        - NET_RAW
                        - SYS_TIME
                    volumeMounts:
                    - mountPath: /etc/kubernetes/admin.conf
                      name: kubeconfig
                      readOnly: true
                  hostNetwork: true
                  hostAliases:
              -   - ip: 192.168.1.104
              +   - ip: 10.0.0.100
                    hostnames:
                    - kubernetes
                  volumes:
                  - hostPath:
                      path: /etc/kubernetes/admin.conf
                    name: kubeconfig
            EOT
          ~ "vip"       = "192.168.1.100" -> "10.0.0.100"
            # (2 unchanged elements hidden)
        }
    }

  # module.node_exporter.null_resource.node_exporter will be created
  + resource "null_resource" "node_exporter" {
      + id       = (known after apply)
      + triggers = {
          + "host"           = "192.168.1.120"
          + "listen_address" = ":9100"
          + "version"        = "1.7.0"
        }
    }

Plan: 6 to add, 0 to change, 4 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"