Excessive NBT Data in a network with a Storage Drawers mod multiblock can cause instability & deadlocks

[Puxta-lab]

Issue type:

• 🐛 Bug

---

Short description:

A continuation of Integrated Terminals #185. In that report, I had little idea of what was going on or how to replicate it. After some testing, I have managed to replicate it; though it's a little bit more involved. And my conclusion is: time to change my storage solution again.

My speculation is that this is more of a problem on Storage Drawer's side of things, and not Integrated Dynamics; but I'm reporting here first to provide closure on the previous report, and to see if there's anything that can be done on this side of things at least to prevent crashes.

But, to actually get on to the problem description. In short; there is a certain threshold of NBT data in a network, which I wasn't quite able to pinpoint, which starts to cause instability if you have a storage drawers multiblock connected to the network. The threshhold occurs earlier if Serialization multithreading is enabled in Integrated Terminals, but in my live world, it still can occur with multithreading OFF.

The instability is characterized primarily by two things.

• 1. Integrated Dynamics will cease to be able to read/extract from (but can still insert into) Storage Drawers multiblocks. This occurs even with a 1x2 multiblock with only 1 item inside each drawer. This appears to be triggered by opening the Storage Terminal on the network. This starts out as a temporary hiccup in the network below a certain threshhold of high NBT items in the network, but eventually becomes consistent as you add more items. I observe a less pronounced hiccup in network item counts when I place logic cables, but I think this might be expected InDy behavior as the network is expanded.
• 2. Once the network can fail to read from the Storage Drawer, this eventually can cause a deadlock, causing the server to stall indefinitely. This is far more likely when multithreading is ON.

Steps to reproduce the problem:

This was replicated in a new world in ATM10; I didn't try with fewer mods mostly because I would need to think about how to get lots of NBT data fast.

1. Place a storage drawer controller, storage drawer controller I/O, and two 1x1 Oak Drawers. Place any item in the drawers, and attach an item interface to the controller I/O. Ensure all blocks are connected. It's best, for step 5, if the items you place in here aren't anywhere else in your network by the end of things.
2. Place a storage terminal on the network, and pre-type the name of one of the items in these drawers into the terminal.
3. Place an inventory that can hold a large number of unique items with a large amount of NBT data. For this purpose, I used a 14x Colossal Chest
4. Find a way to insert a large amount of unique NBT items into the inventory you made in step 3. For this purpose, I made a simple machine to squish Productive Bees into their genes and piped them to inventory 2.
5. It helps if you just have a ton of items in the network too. Make another inventory (I again used 14x colossal chests) which you fill with random junk. I did this by using the /loot command to fill a vanilla chest, piping the contents to Storage Drawers with vending upgrades, and then once I have enough unique items to my liking I pipe them all into the chest.
6. Helpful, but not required, is to set up a network reader on this network you are creating. And use the "item count in network" aspect to get item counts of the item you are keeping in the storage drawers. apply the aspect to those items and put their counts in some display panels. It's best if you can see these panels in your periphery when you open the terminal.
7. Once you have enough items in the network, you should be able to open the terminal and observe the item you pre-typed in Step 2 not appearing while everything in the other inventories do. You might even get a deadlock. I was able to get this working with ~6 million items and ~1-2k bee genes.

Expected behaviour:

Integrated Dynamics should always be able to read from inventories, or at least should not freeze the game when attempting to do so.

---

Versions:

Cyclops Core: 1.27.2
Common Capabilities: 2.11.2-296 beta
Colossal Chests: 1.8.14
Storage Drawers: 13.11.4
Integrated Dynamics: 1.29.3
Integrated Tunnels: 1.9.2
Integrated Terminals: 1.6.20
Minecraft: 1.21.1
Mod loader version: Neoforge 21.1.211

Log file:

Respectfully, I am making this report as I am going to sleep, and didn't collect any crash logs during my previous tests. There are some crash logs related to this issue in my previous report, but if a more recent one in a clean world might elucidate something I can provide it next time I am online.

[rubensworks]

Thanks for reporting!

[rubensworks]

Once you can provide some logs, I can look into it.

[Puxta-lab]

This is a server profile which should contain several instances of the InDy network, upon opening the Storage Terminal, initially displaying the correct item counts for 1-2s, then failing to access the Storage Drawers for a few seconds, and then returning to normal. I have some display panels set up to read specific item counts from the storage drawer; they drop to 0 upon opening the terminal too, but I observe a delay on this happening, and it will occur even if I close the terminal before the display panels show 0. The Storage Drawer multiblock is larger here than detailed in replication because i find it easier to observe the failure this way.

Here is the crash log of me killing the server, for posterity.

[rubensworks]

Note to self: the crashlog above indeed shows a clear deadlock when packetSerializationEnableMultithreading is enabled.

Note to self 2: The root cause seems to be the call to getMaxQuantity. Let's make sure we only call this (and any other indirect calls to positioned storage) in the main thread.