[AWFFULL] no results some months

[Steve McInerney]

On Mon, 2010-05-17 at 14:43 -0400, Zack Galbreath wrote:

> The following log causes all of my April 2010 data to disappear.  It's
> only three lines long.
> 
> http://public.kitware.com/stats/awffull/test/bad.log
> 
> 
> I can't reproduce this bug if I apply these three lines one at a time,
> so I think you're right that it has something to do with the order
> that awffull is processing the log entries.

Oh excellent work! And yes, my guess was right. the problem is that logs
aren't written in the order they arrive; but the order they finish; so
have a few slower pages; and time sorting goes out the window.

> Any suggestions on what I should do to overcome this issue?

In *this* case you have 2 options in the easy case; plus a third.

1. as the bad is an ipv6, we can simply apply an external prefilter (eg
egrep) to strip all ipv6 addresses.

2. do a time sort on the logs in the first place. This is easy to do;
but *can* be painfully slow.
http://www.mrunix.net/webalizer/faq.html has a great example:

$ sort -t ' ' -k 4.9,4.12n -k 4.5,4.7M -k 4.2,4.3n -k 4.14,4.15n -k
4.17,4.18n -k 4.20,4.21n

as a full sort string


3. upgrade the version of awffull you use. I'm quite confidant that bug
was eliminated in later versions. Classic nasty edge case unf.


Cheers!
- Steve