The lag is a very real concern.
It's gonna get old, telling people to repeat themselves because they quick keyed and didn't leave a long enough space.
Even when people use internet linking a lot and know better, sometimes they slip up and don't wait long enough.
It may not sound like a big deal, and it's really not a big deal, but it does require an adjustment of how people operate. I have used linking a lot and when I use it, I wait for the carrier to drop completely then key up, wait 3 seconds, then start talking. That's going to take some training and some adjustment for people who are just accustomed to keying up and being heard.
When you're rag chewing and making long transmissions anyway, if a word or two get clipped out, it's not a big deal but in a special event scenario when transmissions are brief to begin with, you might miss entire transmissions if people don't leave pauses.
Depending on what you use for linking, one nice thing is that if there is a "double" involving more than one repeater there's almost always a clear winner because only audio input stream will be allowed at a time.
I am familar with irlp or echolink, as its more a linking one location to another, this is one to 5 and I am learning what to expect.
IRLP and Echolink aren't just for linking one station to another. They can link many stations together.
Have you thought of using something like a mobile crossband repeater setup so that stations in the field can remain on the same frequency and only one person will need to change the channel on the crossband repeater to get into the appropriate repeater? What this might also enable you to do, aside from eliminating the confusion of when everyone is supposed to change channels, is maintain reliable comms in the repeater dead spots, as on the local side field stations will hear themselves on simplex. You won't have portable stations trying to get into repeaters but rather a mobile crossband repeater controlled by one person who is in constant contact with net control and can decide on the fly when to change repeaters without having to get everyone else in gear. That's always a PITA when you have to get everyone to switch repeaters at an event. You may not even need to use any outside repeaters if net control isn't far enough away that they can't hear the crossband repeater's output.