Infrastructure I/O
Low level I/O carries all interprocess communications
and as such, is the backbone of the infrastructure.
Design Philosophy
In a daemon there should be only one place to actually "wait" the process.
This may seem like a religious or political statement, but it has been my experience that this greatly simplifies overall process coding and structure and eliminates one of the most common structural/performance errors, that is, not being able to process any event at any time.
I know, sometime you'd rather not process some kinds of events, but, if you can, your process' throughput (and stability) will be greatly improved.
There is nothing more frustrating than to have a process hang, waiting on some event when an unexpected error occurred and the process isn't looking for it. If you adopt the single wait mindset in your design, you will always be able to process anything that happens.
In order to accomplish this objective you must have a good set of underlying infrastructure tools to register, deregister, and execute waits.
Since most call-backs, alarms, and input/output servicing is called from ioWait,you should always call ioWait when nothing else needs doing or even occasionally (with a timeout of zero) just to be sure nothing else needs attention.
The infrastructure's I/O uses non-blocking sockets to eliminate deadly embrace
and improve performance under load.
The I/O system always transmitts only the required number of bytes, so if you receive a message and want to place a return message into the same space, the response must be smaller than the in-bound message was.
The only way to insure that you have suffucient space is to create a response message (a much better solution since the msg layer controls all internal type, response, and routing fields).
Protocol Addition (Low Level Straight Skinny)
The infrastructure's I/O and specifically ioWait, is very flexible and allows the coder to build a daemon that can wait on and service any file or socket descriptor event.
To do this the coder must register callbacks to the socket for processing Input, Output, and Exception events.
The call-backs return one of 3 codes, NOTFND (or zero) when a partial read or write done, OK when a read or write is finished, or ERR when something distasteful has happended.
If a call-back returns NOTFND, the ioWait does not return, either error or OK is returned immediately to the caller.
Call-Back registration is accomplished using the function:
ioRegWat( int sd, pfi_t rdFunc, pfi_t wtFunc, pfi_t excFunc, void *ctxt )
The rdFunc and wtFunc must process reads and writes from/to the socket/file, and excFunc must process exceptions.
The process launcher is a good example of how to use this, since it registers sockets to read the output of it's launched sub-process.
Right about here I need to say that the coder is responsible to understand the protocol in enough detail so that the support code properly handles reads, writes, and exceptions.
The ctxt (context) should be a structure pointer that contains the sd and any other info required to perform input, output, or exception handling.
The ctxt pointer is the only argument passed to the drivers.
By default a new sd will have the read and exception waits enabled but not the write.
When output interrupts are expected, you must enable them using ioSetWatOut( int sd ), and when the output is complete clear the wait flag with ioClrWatOut( int sd ).
When the process is finished with a socket or file descriptor, it must call ioDregWat( int sd ) to unregister the socket from the wait mechanism and free the allocated wait structure.
How Infrastructure IO TCP Works:
The basic I/O of the infrastructure is currently based on TCP and is viewed from the aspect of opening a server to accept messages and respond to them or connecting, as a client, to a server then sending and receiving messages to that server.
To send a message to another process you must call three functions:
ioGetPath( char *dest )
where dest is a logical queue (or process name) already defined or a fully qualified logical address "locicalName@host:port encTyp encKey"
This function finds or creates the logical queue address, connects to the server, sending and receiving ping packets, then returns a path handle which may be used to build messages to be sent to the server queue.
This function creates a virtual circuit between the two processes across which all subsequent messages will flow bidirectionally.
ioMakMsg( int pathHndl, int maxLength )
A message structure is allocated with the requested "maxLength" message space then the "pathHndl" is used to find the connection control block and attach it to the message structure.
A pointer the the message space is returned.
ioSnd( char *msg, int length )
The "msg" pointer is used to find the message structure, and through that the connection control block.
The actual data length int the message structure is set to "length" then the message structur is passed to the lower lever transmission layer.
After transmission the message structure and it's data area are freed.
To receive a message (after the queue is opened) you must make three function calls:
ioWait( int waitMs )
Waits until either a message arrives, an error occurs, or the timeout expires.
If ioWait returns "OK" a message has arrived, if it returns "NOTFND" the timeout expired, else "ERR" an error occured.
If "OK" was returned, you must get the message off the receive queue:
ioRcv()
Checks the receive queue and if a message is present, returns the pointer to it's data space, else if no message is available, it returns "NULL".
After processing the data in the message you must call:
ioFree( char *msg )
to de-allocate the message structure and it's data space.
Each process in the infrastructure, unless configured not to, automatically opens a communication queue which is identified by the process' logical name from it's configuration file.
This comm queue is usually on the loopback device so that only processes on the same host can communicate with this process.
If a process is configured to do so, it may open a number of additional queues on different ports, on differenct interface devices, and with different encryption.
Even though the process may have many open queues, all in-bound messages are placed on the same internal message receive queue, hence, if the source of the message is important, it must be stated in the message contents.
In almost all infrastructure based systems, the only process which listens on an external interface is the ORB and all inter-process communication goes througn the ORB.
The most notable exception would be a special process to handle some kind of communication protocol not directly compatable with the infrastructure.
The io1 module and it's minions do connection tracking and a lot of exception handling but it is currently TCP/STRM centric.
Io1 actually has an ioAddProtocol( int protoNbr, char *pName, pfi_t pInit ) which sets up a protocol_t block and calls the pInit function to initialize the underlying protocol control structures with handlers for snd, rcv, exc, conn, dConn, opnSrv, opnClt, getSd, makSockAddr.
In the case of TCP, ioAddProtocol is passed the address of tcpAddVects() which calls ioDrvInitP() to set up the ioRx and ioTx vectors for TCP read and write.
When ioConn is called it creates a new connection control block and calls ioRegWat(), above.
TCP uses a small 16 byte header which contains a version, message type, dataLength, status, and flags.
Since all I/O is event driven, the type field determines what event will fire when the message arrives.
This provides for in-band and out-of-band data and control of all process and communication functions.
Calls to ioAddCfg( int proto, char *name, char *llc, char *phy, char *cmd, char *key ) populate the IO layer's configuration database.
Once this is done a call to ioOpen( char *name ) will open a server listening for connections, and calls to ioGetPath( char *name ) will attempt to connect to other servers.
An in-bound connection creates a new connection control block with it's sd and links it to the associated port control block.
If you are using io1 for connection tracking, one additional piece of code must support possible disconnects.
The coder should register a disconnect processor using ioRegEvent( EVTDCONN, pfi_t dConnFun, void *context ).
The call-back prototype is:
int callBack( void *registeredContext, void *doEvtOption )
The argument "registeredContext" is the variable supplied with the registration.
The second argument "doEvtOption" is the argument supplied by the event trigger call, in the case of conn or dconn it would be the socket descriptor.