Seeking advice on design of application protocol

UPDATE 1 as requested by Brendan.

We are developing a Unix batch application for storing millions of customer records into a relational database. In order to allow multiple batch jobs to run in parallel, and to achieve a certain amount of concurrency while processing an input record, we’ve distributed the work across nine server daemons, all within the same LAN as the client, each of which is responsible for an isolated task (e.g. store the name in the name table, store the address in the address table, etc.).

Each daemon will be able to accept connections and requests (concurrently) from multiple clients.

Finally, whatever communication protocol we design (or adopt) must be extensible to accommodate different kinds of connections – for instance, we’d like for a monitor tool to be able to connect to a daemon, over the same port as the clients, and request statistics or send commands. There will also eventually be different kinds of clients that require different functions to be performed by the daemons. In other words, a client connecting to a daemon will have to be able to declare “I want to post customer records,” or “I want to apply a batch of changes-of-address to the database” or “I want to send you a few commands.”

I think it’s a foregone conclusion that the underlying transport of these connections be TCP.

Finally, apart from the “command session”, these will be multi-million record batch jobs, and performance is critical. The processing is iterative, and every record has the same layout (so wrapping each one in XML, say, would be unnecessary and highly wasteful).

I’ve no idea if this is enough information for someone to suggest some formats and protocols to use, but I’ll gladly try to clarify or supply additional details if asked.

END OF UPDATE 1

(Below is the original post, which may or may not be of any value.)

I’m in charge of designing an application protocol for a set of in-house, batch-oriented, client-server applications. I’m familiar with IBM LU 6.2 (a.k.a. “Advanced Program-to-Program Communication”), and for the past 15 years have worked in Unix environments.

The client is a batch job that connects to a server process — which, in turn, connects to nine other “sub-server” processes. The client passes a customer transaction record to the server, which distributes different portions of the record to the sub-servers. Each sub-server processes its portion independently and concurrently, and passes some results back to the server which then passes them back to the client.

These are TCP connections, and we decided that, first of all, we’d use newline-terminated ASCII strings (we call these lines) as our Protocol Data Unit. A record is a line that is subdivided into tab-separated fields, and since each process will have its own data requirements, field#1 of every record will contain a record type. This is used as a key into a metadata store to find the field names, in the order they’re expected to appear in the record, in order to know how to interpret the record. For example:

Metadata: NAME=(prefix, first_name, middle_name, last_name, personal_suffix, professional_suffix)
Record:   NAME||CHAP||HARRISON|JR|CTO#

(Here I use ‘|’ to represent the tab character, and ‘#’ to represent newline character)

Very straightforward so far, I think.

But then, new twists arose. The first was that, instead of sending all the data about one transaction in a single [large] record, we’d like to possibly use several records: a NAME record, an ADDRESS record, an EMAIL record, etc. So we invented the block – a series of records, preceded by a “special” BB record and followed by an EB record. The BB record’s first field is, of course, ‘BB’, and the second field holds the block type. Block type, like record type, is a key into metadata listing the record types (required, or optional) in this block type.

Understandably, each of these ten or so processes, having its own unique data requirements, has led to its own set of record types, enclosed in their own set of block types. That’s a lot of metadata, but it isn’t really a showstopper.

The next thing requirement that emerged was a special “context-setting” message, to transmit certain configuration variables that would remain constant from transaction to transaction. This gave rise to a new record and block type, as well as a new column in the block metadata that indicated this kind of block was somehow “special”.

Then, envisioning circumstances where the “context” could change during the runtime of the client, we lifted the rule that stipulated there could be only one “context” message, at the beginning of a connection. At the same time, we realized there was still a need for a true “initialization” message that could only come once, at the beginning, conveying “session-global” parameters. Thus the INIT block was born, and now there were three flavors of block: session-initialization, context-switching, and normal application data.

Then, how about a connection that isn’t for transaction-posting at all, but rather a control session for querying the health of the server, retrieving statistics, changing operational settings? The INIT block’s role became more general, declaring the “mode” of the session, which could now be either “posting”, or “control.”

We’ve been trying to wedge all of this into a single-layered application protocol, and it’s getting out of hand. Just defining a “special” BB record should have raised a red flag – it’s constructed from application-level objects and metadata, but it is not application data – rather, it frames the application data.

I think what has evolved is a 2-layer architecture atop TCP: one layer that simply provides typed containers (blocks and records), and a higher layer that uses those containers for a specific purpose, be it posting, or control, or something else – the true application protocols. However, I’m not well-versed enough in the design of protocols to recognize “patterns” that might be obvious to the pros. I’d appreciate any feedback on how to approach this.

It also seems possible that someone may have written a guide to designing protocols, that addresses these very concerns. I just haven’t found it.

And my apologies for the length of this post. I hope none of it was immaterial. My deep thanks for your patience!

Chap

First, break the problem into “requests” and “replies”. A request (from server to sub-server) should begin with some sort of request type identifier, and all of the other data in the request should depend on the request type identifier (e.g. if you have a “set name” request, then you don’t include irrelevant data like “number of cats” in that request). The type of request also determines the type/s of reply the sub-server might send back (e.g. if you send a “set name” request then you might get a “name set” reply or an “error reason” reply; but won’t get a “number of cats” reply).

Second; avoid having modes/contexts if at all possible. For example, rather than having a “set mode to foo” request and a “get name” request, just have a “get name using mode foo” request. This reduces the amount of state in the sub-servers and makes code cleaner; and it can also be important for things like fault tolerance and scalability later on.

Third; allow groups of requests (and groups of replies) to be sent across the network in the same packet. The only reason for this is that small packets waste bandwidth which can become a bottleneck under load. Basically you’d have a “send_request()” function that tries to add the request to the current packet (and sends the packet and starts a new one if the request won’t fit). You will also need timing to avoid latency problems under light load. For example, when you add the first request to a new packet, set a timer for about 2 ms, and when that timer expires send the packet regardless of whether it’s been filled or not.

Fourth; I personally dislike sending data that isn’t text as text. Some people think it makes it easier for debugging (and it can a little bit if you’re unable to work with hex dumps or write a simple decoder); but this doesn’t justify the unnecessary increase in bloat. For a simple example, sending a signed 32-bit integer would mean converting to ASCII, sending up to 11 bytes of data, then converting from ASCII back into an integer; rather than sending 4 bytes of data with no conversions (other than serialisation for endianness that would be almost always optimised to nothing). Sadly, there’s a lot of people using scripting languages for more than just scripting now, so “doing it right(tm)” might be impractical and avoiding bloat may be impossible (e.g. a lot of scripting languages store integers as strings internally).

Fifth; document the protocol well. You should write a “formal-ish” specification that describes each individual request (what its intended for, its “request ID”, which fields in which format, which replies are possible, etc) and describes each individual reply. It should also mention if there’s any special requirements (e.g. a “lock entry” request needs to be followed by one or more “modify entry” requests and then an “unlock entry request”).

Sixth; examine how the data is used. Typically (for databases) finding the right row/s takes most of the time and it doesn’t matter how many columns are in that row; and typically you need multiple pieces of data from a row at the same time. It’s likely that having one sub-server for “name”, one for “address”, one for “age”, etc is going to give bad performance, because if you want the name, address and age at the same time then you end up with 3 sub-servers trying to find the right rows (with triple the overhead). Normally there’s some sort of unique key involved (e.g. a user ID) and it’s far better for the server to use this to determine which sub-server is responsible (e.g. for 9 sub-servers, the server would do “sub_server_number = user_ID % 9” to determine which sub-server to talk to for all transactions involving that user ID). Note that this may also be important for fault tolerance – e.g. if the sub-server for “name” explodes then the entire system will probably fail; but if the sub-server for every nth unique key explodes then all transactions involving other unique keys can continue uninterrupted.

Filed under: softwareengineering - @ 09:21

Thẻ: applications, protocol

Thiết kế website giá rẻ

Danh mục

Seeking advice on design of application protocol