Wrappers and Compound Targets described

[Jarek Kowalski]

Hi NLoggers!

Yesterday I committed a large piece of code that addded many new features.
I'd like to describe two of them, namely: Wrapper targets and Compound
Targets.

WRAPPER TARGETS
==============================

Wrapper Targets are special targets which send their output to other
targets. Each wrapper target has exactly one such target.

The following wrapper targets have been implemented:

1. AsyncWrapper - processes writes in a separate thread which frees the main
thread more quickly therefore improving performance.

2. AutoFlushWrapper - forces a Flush() after each write.

3. BufferingWrapper - buffers log events and when the buffer is full, sends
the buffer as a whole. Some targets may take advantage of it. For example
Mail target is able to send multiple events in a single mail message. File
target is also optimized for batch writes - it can minimize the number of
open/close operations by sorting by filename.

4. RepeatingWrapper - repeats each log message written (this may be useful
to increase the probabilility of message reaching the destination when using
unreliable media - like UDP network protocol)

5. RetryingWrapper - retries the write up to a specified number of times
until the operation succeeds.

The following targets are not complete yet, but the code is in SVN:

6. ASPNetRequestBufferingWrapper - gathers ALL log messages from each
ASP.NET request and when the request is complete, it sends them down to the
wrapped target with possible filtering. Each request has a separate buffer
that's stored in HttpContext.

7. ThreadRequestBufferingWrapper - same as ASPNetRequestBufferingWrapper but
for threaded applications (each thread has a separate buffer that's stored
in Thread local storage)

8. GlobalRequestBufferingWrapper - same as ASPNetRequestBufferingWrapper but
for non-threaded applications (there's one global buffer).

The typical use case for 6,7,8 will be: whenever any Error message occurs
during the request, send down the most detailed trace possible (level TRACE
and above), in the "normal" cases - just send log events whose level is INFO
and above. The filtering is not decided yet, but will definitely be
expandable and make use of the existing Filters infrastructure.

There will be a new API to mark the beginning and the end of the buffer that
the targets will expose:

target.BeginRequest(); // marks the start of the request
target.EndRequest(); // marks the end of the request. Buffer Flush occurs
here.

A convenience shortcut using the IDisposable pattern:

using (target.BeginRequest())
{
}

Configuration file usage is very simple. Just put the wrapped <target>
inside the wrapping <target>. The wrapped target doesn't need a "name"
attribute, but if it has one, you can use either the wrapped or unwrapped
version:

<nlog>
<targets>
    <target name="buffered-mail" type="BufferingWrapper" buffersize=100">
        <target name="mail" type="Mail" ... />
    </target>
</targets>

<rules>
    <logger minlevel="Error" writeto="mail" />    <!-- send error messages
individually -->
    <logger minlevel="Debug" writeTo="buffered-mail" /> <!-- send debug
messages groupped by 100s -->
</rules>
</nlog>

Programmatic usage is very simple, too: a wrapper target is a subclass of
NLog.Targets.Wrappers.WrapperTargetBase and the wrapped target is pointed to
by the "WrappedTarget" property.

You can of course stack up the wrappers. This is the ultimate retrying,
repeating, buffering, async file target:

<target name="xxx" type="AsyncWrapper" batchSize="500">
    <target type="BufferingWrapper" bufferSize="100">
        <target type="RepeatingWrapper" repeatCount="3">
            <target type="RetryingWrapper" retryCount="5"
retryDelayMilliseconds="100">
                <target type="Network" address="udp://somehost:4444"
layout="${message}" />
            </target>
        </target>
    </target>
</target>

COMPOUND TARGETS
==============================

Compound Targets are quite similar to Wrapper Targets but they act on a
group of targets. This can be used to selectively choose which target to
write to, provide some degree of reliability, and so on.

The following compound targets are available:

1. FallbackGroup - writes to the first target, if it fails - writes to the
second one, if this one fails - writes to the third one, and so on.
If any of the writes succeeds, the target remembers the successful target
and uses it in subsequent cases. Alternatively you can tell it to always
return to the first target on success.

2. RandomizeGroup - randomly chooses one of the defined targets and sends
the message to it. (Theoretically) it can be used to provide some load
balancing. Assuming you have 3 network receivers or 3 databases which get
logs, you can randomly choose the one that will get the message which helps
reduce load on each server.

3. RoundRobinGroup - chooses the targets in a round-robin fashion (Nth
message goes to (N % M)th target where M is the number of targets).

4. SplitGroup - writes the message to ALL targets

Configuration usage is simple. Just include multiple <target /> elements
under a compound target:

<target name="random-file" type="RandomizeGroup">
    <target type="File" filename="file1.txt" />
    <target type="File" filename="file2.txt" />
</target>

This will make the random-file target write to either "file1.txt" or
"file2.txt".

This demonstrates the fallback-on-error feature of FallbackGroup:

<target name="failover-target" type="FallbackGroup"
returnToFirstOnSuccess="false">
    <target type="Database" ... />
    <target type="Network" ... />
    <target type="File" ... />
</target>

This will try to send logs to the database first, if it fails, logs will be
sent over the network and if it fails too, they will be written to a file. A
database will be tried first on each write. If you change the
returnToFirstOnSuccess to "true", it will remember last-known-good target
and try to write to it first.

You can of course mix and match compound targets and wrappers by stacking
and nesting them:

<target name="fancy-target" type="RoundRobinGroup">
    <target type="BufferingWrapper">
        <target type="File" ... />
    </target>
    <target type="RandomizeGroup">
        <target type="Database" ... />
        <target type="Database" ... />
        <target type="AsyncWrapper">
        </target>
    </target>
    <target type="Mail" ... />
</target>

That should hopefully cover all of this subject. If you have any further
questions, just ask them on the mailing list so that everyone can benefit
from the answers.

I'd like to ask you for some serious field testing so that we can be sure
things are stable for the 0.95 release.
The bits are here: http://nlog.sourceforge.net/snapshots/20050929/

--
Jaroslaw Kowalski
http://blog.jkowalski.net/ 



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Nlog-list mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nlog-list

[Ron Grabowski]

--- Jaroslaw Kowalski <[hidden email]> wrote:

> 6. ASPNetRequestBufferingWrapper - gathers ALL log messages from each
>
> ASP.NET request and when the request is complete, it sends them down
> to the
> wrapped target with possible filtering. Each request has a separate
> buffer
> that's stored in HttpContext.

This is interesting. Similiar to how the O/R Mappers allow a user to
make changes to groups of object then a single commit starts the
process of writing all the changes to the database.

> This will try to send logs to the database first, if it fails, logs
> will be
> sent over the network and if it fails too, they will be written to a
> file. A
> database will be tried first on each write. If you change the
> returnToFirstOnSuccess to "true", it will remember last-known-good
> target
> and try to write to it first.

One of the features that I remember users asking for every now and then
is something that will buffer records if the target/appender goes
offline for a brief period of time. For example if my database is
recycling and its offline for 5 or 10 seconds it would be nice to be
able to buffer messages sent during this period then commit them to the
database once its available again. Of course there would need to be an
expiration on the buffer such that if the buffer expires the buffered
messages would be sent to the next target/appender in the chain. Every
once in a while there's a post on the log4j list about people worried
about loosing a handful of log messages when their RollingFileAppender
is being rolled.


-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Nlog-list mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nlog-list

[Jarek Kowalski]

... SourceForge sucks. The message has been delayed for at least 48 hours...

From: "Ron Grabowski" <[hidden email]>

> --- Jaroslaw Kowalski <[hidden email]> wrote:
>
>> 6. ASPNetRequestBufferingWrapper - gathers ALL log messages from each
>>
>> ASP.NET request and when the request is complete, it sends them down
>> to the
>> wrapped target with possible filtering. Each request has a separate
>> buffer
>> that's stored in HttpContext.
>
> This is interesting. Similiar to how the O/R Mappers allow a user to
> make changes to groups of object then a single commit starts the
> process of writing all the changes to the database.

Kind of. The key point here is "filtering". For example, you'll be able to
provide detailed trace for all requests where a user queries for a specific
record in a database. Of course you don't know it at the very beginning (you
have to parse the request first), and with ASPNetRequestBufferingWrapper
you'll be able to do the filtering post-factum, when all log messages have
been gathered.

>> This will try to send logs to the database first, if it fails, logs
>> will be
>> sent over the network and if it fails too, they will be written to a
>> file. A
>> database will be tried first on each write. If you change the
>> returnToFirstOnSuccess to "true", it will remember last-known-good
>> target
>> and try to write to it first.
>
> One of the features that I remember users asking for every now and then
> is something that will buffer records if the target/appender goes
> offline for a brief period of time.

This is something that RetryingWrapper does. Just set the retry timeout to
be 5 seconds or so.

> For example if my database is
> recycling and its offline for 5 or 10 seconds it would be nice to be
> able to buffer messages sent during this period then commit them to the
> database once its available again. Of course there would need to be an
> expiration on the buffer such that if the buffer expires the buffered
> messages would be sent to the next target/appender in the chain. Every
> once in a while there's a post on the log4j list about people worried
> about loosing a handful of log messages when their RollingFileAppender
> is being rolled.

Or perhaps this requires a smarter Wrapper. Would you care to write one,
Ron?

Jarek



-------------------------------------------------------
This SF.Net email is sponsored by:
Power Architecture Resource Center: Free content, downloads, discussions,
and more. http://solutions.newsforge.com/ibmarch.tmpl
_______________________________________________
Nlog-list mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/nlog-list