Red Hat's Linux changes: Fixes or ISV positioning?

Syslog, filesystem plans raise new questions about direction of Red Hat

Recent proposals from the Fedora Project that could change the fundamental infrastructure of Linux have been met with caution and critical analysis. They also may raise questions on why Fedora and its commercial parent Red Hat seem to be pushing away from existing standards.

After initial negative reaction to a proposed new daemon to improve system logging functions within Linux, experts in system logging are now adding more comprehensive analysis on the proposal--and aren't liking what they see, either.

Red Hat developers Lennart Poettering and Kay Sievers recently proposed that the current 30-year-old syslog system is inefficient and too easy to misread and hack to properly perform even its most basic function: store a log of system events on a given Linux box. To address these and other perceived problems, Poettering and Sievers proposed the new Journal daemon.

Initially, reaction to the Journal was decidedly mixed, though definitely leaning towards more of a negative response. Much of the first reaction focused on the Journal's use of a binary key-value form of data to log system events, as opposed to the text-based data used by the current syslog system. With this binary implementation, The Journal daemon would enable the addition of metadata to each system event, such as the process ID and name of the sender, user and group IDs, and other key system data.

Many critics of the Journal lamented that this would be the technical equivalent of bringing Windows Event Log functionality into the core of Linux administration. Given Linux users' typical reaction to all things Microsoft, you can guess how well that one went over.

But Rainer Gerhards, the lead developer for the rsyslog tool, has now had a chance to analyze Poettering's and Sievers' paper in detail and says that the similarities to the Windows Event Log is actually a good thing, since there are aspects of the Windows Event Log tool that would actually be useful in.

But, Gerhards argues, such a drastic change in the way Linux handles system event logging may not be necessary, given that Gerhards' rsyslog tool, as well as functionality in the competing syslog-ng tool, already can address many of the problems Sievers and Poettering have addressed.

One example that Gerhards focused on in his initial assessment of the Journal proposal was the issue of timestamps in syslog, about which Sievers and Poettering wrote: "The timestamps generally do not carry timezone information, even though some newer specifications define support for it."

Gerhard's initial take on timestamps was that syslog most definitely can handle detailed timestamps… but in practice this is not done:

"Lennart [Poetering] claims that syslog does not contain a timezone and mentions that journald will provide much finer resolution. Actually, the timestamp is a source of deep frustration to me. Ages ago (2006?) I implemented high-precision timestamps (including TZ info) in rsyslog, and RFC5424 has brought them to the on-the-wire protocol. As far as I know, syslog-ng supports them for quite a while as well (but I am not a syslog-ng expert ;)). However, all distributions turn high precision timestamps off and set the dumb old format as this is a requirement to keep old tools working."

In a later, more detailed analysis, Gerhards repeats the assessment of the timestamp problem, and uses it to enforce his arguments. While he agrees with some of the assertions made by Poettering and Sievers in the Journal proposal, Gerhards stresses his belief that while the two Red Hat developers have done a fair job of accurately portraying the current state of system logging problems on Linux, many of the problems already have technological solutions available.

Solutions, Gerhards emphasizes, that aren't being used. The problem, he asserts, is a lack of proper developer implementation. Here's how Gerhards addresses the problem of multiple system logs on a Linux machine, as first introduced by Sievers and Poettering:

"4. Syslog is only one of many log systems on local machines. Separate logs are kept for utmp/wtmp, lastlog, audit, kernel logs, firmware logs, and a multitude of application-specific log formats. This is not only unnecessarily complex, but also hides the relation between the log entries in the various subsystems.

"Rhetorically True - but what why is that the failure of syslog? In fact, this problem would not exist if developers had consistently used syslog. So the problem is not rooted in syslog but rather in the fact that syslog is not being used. Lesson learned: even if standards exist, many developers simply ignore them…"

Without a standardized way of applications sending information to and pulling information from syslog, the problems with event logging continue to propagate. A new form of tool, therefore, does not instantly wave a magic wand over the problems, because there's little guarantee the problems wouldn't simply continue.

Remember Gerhards' honest assessment of the Windows Event Log, which he says the Journal seems to emulate? It hasn't had a lot of success, either:

"In Windows Event Log, there are also 'structured fields' available, but in the form of an array (this is [arguably] a bit different from name-value pairs but far from totally different). This system [has been] in place since the earliest versions of Windows NT, more than 15 years ago. So it would be a decent assumption that the problem described as a syslog problem does not exist in the Windows world, right (especially given the fact that Windows purposefully does not support syslog)? Well, have a look at the problems related to Windows log analysis: these are exactly the same!"

The problem, therefore, is one of discipline: if developers of applications and distributions would enforce a consistent methodology within Linux system logging practices, then a lot of the problems with syslog would ultimately go away.

As I mentioned the last time I discussed the proposed Journal daemon, Journal is planned to be implemented by the release of Fedora 17. I also made the observation that the Fedora team and Red Hat seem to be on the cutting edge of many internal infrastructure changes to the Linux operating system. (Sievers, for instance, is one of the Fedora Project team members who proposed to move all executable files into the /usr/bin directory and their libraries into /usr/lib or /usr/lib64, as needed.)

Upon learning that there are compelling arguments for using existing technology to address these syslog problems--indeed, that similar technological approaches to the Journal have not successfully eliminated many of the same problems in other operating systems--I now believe questions about Red Hat's plans need to be raised.

Simplifying the filesystem hierarchy and event logging in Fedora (and by extension, Red Hat Enterprise Linux) seem like worthy goals, but it could come at the expense of circumventing existing standards, like the Linux Standard Base or the Common Event Expression, respectively.

You can argue the effectiveness of these standards (and many have), but if Red Hat is planning on just going their own way here with Linux infrastructure, as they seem to be doing, this represents a game-changer for the Linux ecosystem. Such differentiators will most certainly affect how independent software vendors work with Red Hat's Linux versus the other distributions.

In a world where app development can make all the difference for commercial Linux vendors, that's a telling path to take.

Read more of Brian Proffitt's Open for Discussion blog and follow the latest IT news at ITworld. Drop Brian a line or follow Brian on Twitter at @TheTechScribe. For the latest IT news, analysis and how-tos, follow ITworld on Twitter and Facebook.

ITWorld DealPost: The best in tech deals and discounts.
Shop Tech Products at Amazon