Author Topic:   Networking
LDA Seumas
posted September 08, 1999 12:09 PM           
I just sent this to the Software-Engineering-Game-Development mailing list, and though I'd report it here in case anyone is interested.

For Tread Marks, I've built a totally UDP based networking layer to handle all client to server and server to client communications in the game. It uses a single BSD Socket on each end, set to non-blocking IO mode, which I suck dry of incoming UDP packets during each network "pulse".

The heart of my layer is the mini-packet, which is a light weight packet of data which only requires a 2-byte header when transmitted (identification as to type (Unreliable, Reliable, Ordered) and length packed into 16-bits). The application deals in mini-packets, and thus can send a brand new "packet" for even very small amounts of data (the state for a single entity, say) without incurring a huge penalty. The network layer has three outgoing queues for each remote connection, one each for Unreliable, Reliable, and Ordered mini-packets. The application adds packets to these queues as it does its processing (not directly, of course; the internals are well hidden), and then once per frame (or more, I think I use 3 or 4 times per frame) the app calls the network layer's ProcessTraffic member, which builds a single UDP packet for each connected client containing as many queued mini-packets and mini-packet reception ACKs as possible, based on packet priorities assigned at queue-up time. ACKs are bundled first, then OrderedReliable packets, then Reliable packets (they get through, but no order guarantee), then Unreliable packets, up to the byte-rate limit for this client on this "pulse".

I'm really glad I structured things this way, as it provides a nice separation between the networking guts and the application, while giving the application much more control than it would have using UDP and/or TCP directly. I simply tell the networking layer that this client only gets this many bytes per second, and it does. The packets that need to get through most badly still do, based on priorities.

At least with a fast action game like Tread Marks, networking is a bit like lossy data compression. You have to figure out which data you can completely throw out or ignore for a while without unduly damaging the final "image", which is the sequence of events the player sees. Actually setting packet priorities, and deciding when or when not to send mini-packets is left up to the app, which does so based on where the entity sending the packet originated in space compared to the location of the client computer's physical manifestation.

Byte rate limits and priorities also go hand in hand, I've found. I implemented byte rates before priorities, and to my chagrin found the game would become unplayable at modem-quality byte rates, due to the random (or worse, predictable _and_ totally "wrong") order with which mini-packets would be queued and then thrown out due to the rate. At this point, I think I was actually not sending as many packets for distant entities, but even then, the flood of packets that did need sending would often keep required packets dumped due to rate. Once I added priorities, which would order the Unreliable mini-packets while in the queue so that the "most important" ones would get through under the byte rate limit, everything suddenly started working great.

Also, when delivering packets to the app, the network layer only deals with data and lengths, no IDs or priorities. Even though the network layer has its own packet headers internally, it leaves it up to the app to tag and decipher the mini-packet data it sends and receives, since the app usually uses tight Bit Packing, and can often flag the contents of a mini-packet with just a few bits.

Note that I've only had a couple of "real" modem over internet tests of this whole system so far, but they were lousy modem clients connecting to cable-modem dedicated servers, and things appeared to hold up well. The big test of all this should be in a couple of days, to see how it really stacks up.

-- Seumas McNally, Lead Programmer, Longbow Digital Arts