So I've been learning more than I ever wanted to about the syslog protocol. There's the non-spec that is RFC-3164 that is optimistic in terms of the protocol. Then there's the cleaned-up spec that no one is using that is RFC-5424 (which is quite nice, if a bit over-engineered).
RFC-3164 documents the use of UDP as the transport protocol for the syslog protocol, reading that RFC one gets the impression that one should never actually use UDP as the transport mechanism, least some cracker intercept or change the messages, or worse yet—you lose some packets and get nailed in an Sarbanes—Oxley audit (or even worse still—an ISO-9000 audit—the horror! The horror!).
Well, you could try running the syslog protocol over TCP, but even that isn't good enough for some people, claiming that you can still lose logging information under certain circumstances. No, for reliability you need to add a layer of framing over TCP and wrap the syslog protocol in XML and call it a day.
I personally haven't seen an issue with using UDP for the syslog protocol. Not only do I relay syslog messages to a centralized server (my desktop box at Chez Boca, so I can watch the stuff in real time) but copies are kept locally (just in case). Also, there have been times when a TCP version (yes, even if I was using RFC 3195 or the lighter RELP) would have failed (at one point, our upstream provider upgraded a firewall that filtered out TCP traffic routed asymetrically and guess what? Our traffic was routed asymetrically; UDP traffic was unaffected and thus in that case, we were able to isolate the issue faster). Even the design of SNMP centered around UDP simply because it was “fire and forget” and thus on a congested network, there was a greater chance of UDP traffic of making it out and accepted than TCP traffic (which requires an acknowledgment that might never make it back).
But in looking over these, I'm struck that a reliable syslog protocol doesn't use SCTP, which has the reliability, ordering and (most importantly, congestion control) of TCP with the message-based semantics of UDP. Heck, for “reliability” SCTP has one feature that neither TCP nor UDP have: either peer can change the IP address used for the session.
For now, I'll just stick with UDP.