Use persistent XPC connection for IP allocation management.#1395
Use persistent XPC connection for IP allocation management.#1395jglogan wants to merge 3 commits intoapple:mainfrom
Conversation
- Closes apple#1378. - At present, IP allocation involves four parties: the ContainersService and NetworksService in the API server, the SandboxService running in `container-runtime-linux`, and the NetworkService running in `container-network-vmnet`. - Each NetworkService `allocate()` requires a corresponding `deallocate()` or an IP address gets leaked. - This change gets rid of `deallocate()` and instead changes `allocate()` so that the caller received an XPC connection. The caller holds this connection for as long as the network attachment is alive (essentially as long as the container is running). When the connection disconnects, the NetworkService automatically unregisters the hostname:IP association. - Removed IP allocation from the API server, restoring the old process by which sandboxes directly allocate from networks. To deal with the eventual possibility of more network plugins than `container-network-vmnet`, the bootstrap call requires an additional parameter `networkPluginInfos` that is the same size as the attachments array in the container configuration, and identifies the plugin for each attachment.
| request.set(key: NetworkKeys.macAddress.rawValue, value: macAddress.description) | ||
| } | ||
|
|
||
| let client = createClient() |
There was a problem hiding this comment.
If we changed this client to the model that some of our others have taken (maintain a long lived xpc connection instead of creating a new one for every request), does this screw things up with this persistent routes idea?
There was a problem hiding this comment.
It'd probably be easier to just discuss it. I could see the API server ContainerNetworksService holding long lived XPC connections with each network. The API server uses different calls than the runtime plugin, so I imagine there's a way to make it work.
| /// Persistent XPC connections to each network plugin, one per network attachment. | ||
| /// Held for the container lifetime; deallocation happens automatically when these are released. | ||
| private var networkConnections: [XPCClient] = [] |
There was a problem hiding this comment.
Maybe this is a naive question, but could this ever cause an issue if we have a lot of running containers with a lot of networks in use?
There was a problem hiding this comment.
Not naive, I can do a bit of research on that.
A napkin calculation is that we can have a max of ~256 VMs, and I think maybe 4 network interfaces per VM (I think I just saw that recently but could be wrong there) so if you worst case the network interfaces per VM number you're looking at 4K XPC connections.
|
@katiewasnothere @dcantah Thanks for the initial comments! I pushed a PR to fix a bug in the disconnect handler, it was canceling the server XPC connection even if the container wasn't in the running state, which caused startup errors to get handled improperly. I still need to think through the sandbox changes more closely myself, so putting this into draft until I do that. |
4 participants
IP allocator cannot be disabled with active containers#1318.container-runtime-linux, and the NetworkService running incontainer-network-vmnet.allocate()requires a correspondingdeallocate()or an IP address gets leaked.deallocate()and instead changesallocate()so that the caller received an XPC connection. The caller holds this connection for as long as the network attachment is alive (essentially as long as the container is running). When the connection disconnects, the NetworkService automatically unregisters the hostname:IP association.container-network-vmnet, the bootstrap call requires an additional parameternetworkPluginInfosthat is the same size as the attachments array in the container configuration, and identifies the plugin for each attachment.Type of Change
Motivation and Context
deallocate()call was always a leak hazard.Testing