Support IPA IPA Trust with additional IPA server

[justin-stephenson]

Add new server master2.ipa2.test which deploys an IPA domain ipa2.test to be used in IPA IPA trust.

with this PR checked out
sudo make down
sudo make build
`sudo REGISTRY="localhost/sssd" make up

Linked PRs:
SSSD/sssd-test-framework#119
SSSD/sssd#7517

[justin-stephenson]

Hi @pbrezina Can you help me understand why docker.io/ubuntu:rolling build fails (https://github.com/SSSD/sssd-ci-containers/actions/runs/10370744252/job/28711693110?pr=106) when sshd does not restart properly on master2.ipa2.test. I see that the host keys are not found.

Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.ecdsa_key                                                    
Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.ed25519_key                                          
Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.rsa_key  

I ran src/tools/gen-ssh-keys.sh and added the ssh keys for master2.ipa2.test into this PR. I also added master2.ipa2.test into the for loop in src/tools/gen-ssh-keys.sh

However, these are not being copied into the master2.ipa2.test system.

[root@master2 /]# ll data/ssh-keys/hosts/
total 192       
-rw-------. 1 root root  525 Aug 13 02:05 client.test.ecdsa_key                                                        
-rw-------. 1 root root  189 Aug 13 02:05 client.test.ecdsa_key.pub                                                    
-rw-------. 1 root root  419 Aug 13 02:05 client.test.ed25519_key                                                      
-rw-------. 1 root root  109 Aug 13 02:05 client.test.ed25519_key.pub                                                  
-rw-------. 1 root root 2610 Aug 13 02:05 client.test.rsa_key                                                          
-rw-------. 1 root root  581 Aug 13 02:05 client.test.rsa_key.pub                                                      
-rw-------. 1 root root  525 Aug 13 02:05 dc.samba.test.ecdsa_key                                                      
-rw-------. 1 root root  189 Aug 13 02:05 dc.samba.test.ecdsa_key.pub                                                  
-rw-------. 1 root root  419 Aug 13 02:05 dc.samba.test.ed25519_key                                                    
-rw-------. 1 root root  109 Aug 13 02:05 dc.samba.test.ed25519_key.pub                                                
-rw-------. 1 root root 2622 Aug 13 02:05 dc.samba.test.rsa_key                                                        
-rw-------. 1 root root  581 Aug 13 02:05 dc.samba.test.rsa_key.pub                                                    
-rw-------. 1 root root  525 Aug 13 02:05 dns.test.ecdsa_key                                                           
-rw-------. 1 root root  189 Aug 13 02:05 dns.test.ecdsa_key.pub                                                       
-rw-------. 1 root root  419 Aug 13 02:05 dns.test.ed25519_key                                                         
-rw-------. 1 root root  109 Aug 13 02:05 dns.test.ed25519_key.pub                                                     
-rw-------. 1 root root 2610 Aug 13 02:05 dns.test.rsa_key
-rw-------. 1 root root  581 Aug 13 02:05 dns.test.rsa_key.pub                                                         
-rw-------. 1 root root  525 Aug 13 02:05 kdc.test.ecdsa_key                                                           
-rw-------. 1 root root  189 Aug 13 02:05 kdc.test.ecdsa_key.pub                                                       
-rw-------. 1 root root  419 Aug 13 02:05 kdc.test.ed25519_key                                                         
-rw-------. 1 root root  109 Aug 13 02:05 kdc.test.ed25519_key.pub                                                     
-rw-------. 1 root root 2622 Aug 13 02:05 kdc.test.rsa_key
-rw-------. 1 root root  581 Aug 13 02:05 kdc.test.rsa_key.pub                                                         
-rw-------. 1 root root  525 Aug 13 02:05 master.ipa.test.ecdsa_key                                                    
-rw-------. 1 root root  189 Aug 13 02:05 master.ipa.test.ecdsa_key.pub                                                
-rw-------. 1 root root  419 Aug 13 02:05 master.ipa.test.ed25519_key                                                  
-rw-------. 1 root root  109 Aug 13 02:05 master.ipa.test.ed25519_key.pub                                              
-rw-------. 1 root root 2622 Aug 13 02:05 master.ipa.test.rsa_key                                                      
-rw-------. 1 root root  581 Aug 13 02:05 master.ipa.test.rsa_key.pub                                                  
-rw-------. 1 root root  525 Aug 13 02:05 master.keycloak.test.ecdsa_key                                               
-rw-------. 1 root root  189 Aug 13 02:05 master.keycloak.test.ecdsa_key.pub                                           
-rw-------. 1 root root  419 Aug 13 02:05 master.keycloak.test.ed25519_key                                             
-rw-------. 1 root root  109 Aug 13 02:05 master.keycloak.test.ed25519_key.pub                                         
-rw-------. 1 root root 2622 Aug 13 02:05 master.keycloak.test.rsa_key                                                 
-rw-------. 1 root root  581 Aug 13 02:05 master.keycloak.test.rsa_key.pub                                             
-rw-------. 1 root root  525 Aug 13 02:05 master.ldap.test.ecdsa_key                                                   
-rw-------. 1 root root  189 Aug 13 02:05 master.ldap.test.ecdsa_key.pub                                               
-rw-------. 1 root root  419 Aug 13 02:05 master.ldap.test.ed25519_key                                                 
-rw-------. 1 root root  109 Aug 13 02:05 master.ldap.test.ed25519_key.pub                                             
-rw-------. 1 root root 2622 Aug 13 02:05 master.ldap.test.rsa_key                                                     
-rw-------. 1 root root  581 Aug 13 02:05 master.ldap.test.rsa_key.pub                                                 
-rw-------. 1 root root  525 Aug 13 02:05 nfs.test.ecdsa_key                                                           
-rw-------. 1 root root  189 Aug 13 02:05 nfs.test.ecdsa_key.pub                                                       
-rw-------. 1 root root  419 Aug 13 02:05 nfs.test.ed25519_key                                                         
-rw-------. 1 root root  109 Aug 13 02:05 nfs.test.ed25519_key.pub                                                     
-rw-------. 1 root root 2610 Aug 13 02:05 nfs.test.rsa_key
-rw-------. 1 root root  581 Aug 13 02:05 nfs.test.rsa_key.pub   

[pbrezina]

Hi @pbrezina Can you help me understand why docker.io/ubuntu:rolling build fails (https://github.com/SSSD/sssd-ci-containers/actions/runs/10370744252/job/28711693110?pr=106) when sshd does not restart properly on master2.ipa2.test. I see that the host keys are not found.

Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.ecdsa_key                                                    
Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.ed25519_key                                          
Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.rsa_key  

I ran src/tools/gen-ssh-keys.sh and added the ssh keys for master2.ipa2.test into this PR. I also added master2.ipa2.test into the for loop in src/tools/gen-ssh-keys.sh

However, these are not being copied into the master2.ipa2.test system.

Ubuntu does not provide ipa package so base-ipa container is actually pulled from quay.io/sssd/ci-base-$svc:lates which does not contain your changes. Maybe, it would be possible to use the base image we just created? But build.sh would have to change. Maybe introduce UNAVAILABLE_BASE_IMAGE variable or something like that.

[pbrezina]

Hi @pbrezina Can you help me understand why docker.io/ubuntu:rolling build fails (https://github.com/SSSD/sssd-ci-containers/actions/runs/10370744252/job/28711693110?pr=106) when sshd does not restart properly on master2.ipa2.test. I see that the host keys are not found.

Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.ecdsa_key                                                    
Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.ed25519_key                                          
Aug 13 13:19:21 master2.ipa2.test sshd[6905]: Unable to load host key: /data/ssh-keys/hosts/master2.ipa2.test.rsa_key  

I ran src/tools/gen-ssh-keys.sh and added the ssh keys for master2.ipa2.test into this PR. I also added master2.ipa2.test into the for loop in src/tools/gen-ssh-keys.sh
However, these are not being copied into the master2.ipa2.test system.

Ubuntu does not provide ipa package so base-ipa container is actually pulled from quay.io/sssd/ci-base-$svc:lates which does not contain your changes. Maybe, it would be possible to use the base image we just created? But build.sh would have to change. Maybe introduce UNAVAILABLE_BASE_IMAGE variable or something like that.

Actually it wouldn't work because we run each distro on different host. We would need to store it as artifact and then download it and install it.

[justin-stephenson]

Ubuntu does not provide ipa package so base-ipa container is actually pulled from quay.io/sssd/ci-base-$svc:lates which does not contain your changes. Maybe, it would be possible to use the base image we just created? But build.sh would have to change. Maybe introduce UNAVAILABLE_BASE_IMAGE variable or something like that.

Actually it wouldn't work because we run each distro on different host. We would need to store it as artifact and then download it and install it.

Can ssh keys from both IPA servers master.ipa.test and master2.ipa2.test be added to quay.io/sssd/ci-base-ipa:latest ?

[jakub-vavra-cz]

The base_ipa2 is still present.

[justin-stephenson]

The base_ipa2 is still present.

Removed fully.

[pbrezina]

Ubuntu does not provide ipa package so base-ipa container is actually pulled from quay.io/sssd/ci-base-$svc:lates which does not contain your changes. Maybe, it would be possible to use the base image we just created? But build.sh would have to change. Maybe introduce UNAVAILABLE_BASE_IMAGE variable or something like that.

Actually it wouldn't work because we run each distro on different host. We would need to store it as artifact and then download it and install it.

Can ssh keys from both IPA servers master.ipa.test and master2.ipa2.test be added to quay.io/sssd/ci-base-ipa:latest ?

No until this PR is merged. But you could do it manually, however it's probably not worth the effort.

[pbrezina]

Justin, you can try removing the ssh host keys completely when you will rebase on top of Jakub's changes. I'm pretty sure I added them as a workaround for something, but I don't remember anymore. Maybe, it is not needed anymore.

[justin-stephenson]

Justin, you can try removing the ssh host keys completely when you will rebase on top of Jakub's changes. I'm pretty sure I added them as a workaround for something, but I don't remember anymore. Maybe, it is not needed anymore.

I rebased and removed the host keys.

[danlavu]

Just be aware that we can't copy this that easily in idmci (without hadcoding specific openstack networks). We also need to make sure that the same network will still work.

Ack, I did run it in IDMCI and the provision passes, I'll go ahead and test it.

2024-12-10T05:40:24 PLAY RECAP *********************************************************************
2024-12-10T05:40:24 client.test                : ok=82   changed=46   unreachable=0    failed=0    skipped=45   rescued=0    ignored=0   
2024-12-10T05:40:24 dc-nhhq.ad-nhhq.test       : ok=17   changed=8    unreachable=0    failed=0    skipped=8    rescued=0    ignored=0   
2024-12-10T05:40:24 dc.samba.test              : ok=67   changed=44   unreachable=0    failed=0    skipped=44   rescued=0    ignored=0   
2024-12-10T05:40:24 dns.test                   : ok=57   changed=26   unreachable=0    failed=0    skipped=42   rescued=0    ignored=0   
2024-12-10T05:40:24 kdc.test                   : ok=57   changed=27   unreachable=0    failed=0    skipped=46   rescued=0    ignored=0   
2024-12-10T05:40:24 master.ipa.test            : ok=61   changed=35   unreachable=0    failed=0    skipped=47   rescued=0    ignored=0   
2024-12-10T05:40:24 master.ldap.test           : ok=61   changed=31   unreachable=0    failed=0    skipped=44   rescued=0    ignored=0   
2024-12-10T05:40:24 master2.ipa2.test          : ok=60   changed=34   unreachable=0    failed=0    skipped=48   rescued=0    ignored=0   
2024-12-10T05:40:24 nfs.test                   : ok=54   changed=24   unreachable=0    failed=0    skipped=46   rescued=0    ignored=0  

[justin-stephenson]

I rebased this PR, and also added capabilities to the 'ipa2' section in docker-compose.yml similar to

211908d

[danlavu]

@justin-stephenson, I have the configuration provisioning but the networks are not routable to one another so I'm trying to figure that out.

[sumit-bose]

Hi,

jfyi, I was still able to setup a test environment with this PR.

bye,
Sumit

[danlavu]

We have to visit the multiple network stuff later. To @jakub-vavra-cz point, the priority is that it works on the same net.

[danlavu]

We have to visit the multiple network stuff later. To @jakub-vavra-cz point, the priority is that it works on the same net.

[justin-stephenson]

Hi @pbrezina Do you think this is ready to be merged?

Related PRs SSSD/sssd-test-framework#119 and SSSD/sssd#7517 will still be blocked until FreeIPA IDM IDM Trust code is merged into FreeIPA.

Based on your earlier comment:

Justin, you can try removing the ssh host keys completely when you will rebase on top of Jakub's changes. I'm pretty sure I added them as a workaround for something, but I don't remember anymore. Maybe, it is not needed anymore.

I created #122 for this but please check if I missed anything.

[pbrezina]

Hi, see a nitpick inline. Otherwise it looks good.

[justin-stephenson]

Hi, see a nitpick inline. Otherwise it looks good.

@pbrezina friendly ping reminder

[pbrezina]

Ack, but I am rerunning c8s before pushing as there is a failure.

[justin-stephenson]

Ack, but I am rerunning c8s before pushing as there is a failure.

No host keys were added to data/ssh-keys/hosts/ for master.ipa2.test in this PR, therefore the below step fails because c8s openssh version does not appear to support this drop directory /etc/ssh/sshd_config.d/. How do you prefer to fix it?

- name: Configure SSH daemon without pre-generated hostkey
  ansible.builtin.copy:
    dest: /etc/ssh/sshd_config.d/sshd.conf
    owner: root
    group: root
    mode: 0600
    content: |
      PermitRootLogin yes
  when: not stat_ecdsa_key.stat.exists

[pbrezina]

The problem is not that sshd_config.d is not supported on C8S, but copy does not create the folder, while template which is used to set the generated keys does.

[danlavu]

Last week, I tried to get the networks working. Everything is provisioned, but the hosts are not routable to one another. Do you think you can test it? Perhaps it's an issue with my workstation?

justin-stephenson#1

I suspect that it may be a firewall-related issue. I've tried the following with no such luck.

firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -i podman1 -o podman2  -j ACCEPT
firewall-cmd --direct --add-rule ipv4 filter FORWARD 0 -i podman2 -o podman1  -j ACCEPT

Then, explicitly defined the networks

firewall-cmd --zone=FedoraWorkstation --add-rich-rule='rule family="ipv4" source address="172.16.100.0/24" destination address="172.16.110.0/24" accept'
firewall-cmd --zone=FedoraWorkstation --add-rich-rule='rule family="ipv4" source address="172.16.110.0/24" destination address="172.16.100.0/24" accept'

[jakub-vavra-cz]

Hi @danlavu @justin-stephenson ,
You might need to setup policy for routing and make sure that interfaces are in proper zones
See: https://firewalld.org/2020/09/policy-objects-introduction

I found about it when my home server based on fedora suddenly stopped routing traffic between networks :-D.