[AWFFULL] month end problem

[Steve McInerney]

On Tue, 6 Jan 2009 04:59:36 am Anthony J. Biacco wrote:
> I'm having a problem when awffull is running for the last day of the
> month. I'm currently using 3.9.1-beta3.
> So, some background..
> My analyzing script fires every 4 hours. In cron, "0 0,4,8,12,16,20".
> Now, in my script, I run the awffull twice, each on the same log file,
> each with a different config file. Once for the monthly cumulative
> update, and once to make a running daily update (to a separate output
> directory, separate cumulative file). This is because I wanted a
> separate details page for every day of the month to compare day-to-day
> totals, like referrer hits, etc..
> For the running daily update, at midnight on the following day, it
> removes the cumulative file for the previous day. E.g. when processing
> data for 12/30/08, I might output to OutputBaseDir/20081230 and then
> when midnight rolls around on 12/31/08, I'll remove
> OutputBaseDir/20081230/webalizer.current and start with
> OutputBaseDir/20081231/webalizer.current.
>
> So I've found that when I get to the end of the previous month, the
> reporting for the month gets kind of truncated. So if I just ran at
> midnight on 01/01/09 for the last of the 12/xx/08 month data, the 12/08
> data in the main "Summary by Month" has all the "Daily Avg" columns set
> to 0, but the "Monthly Totals" columns are all correct. Furthermore, if
> I click on the 12/08 month to get the detailed data, the Daily Usage
> graph all has bars for every day, yet the table underneath it for
> Hits/Files/Pages/etc.. per day only has data for the first day of the
> month, i.e. 12/01/08.
>
> I don't expertly know how the end of the month processing works for
> awffull, but I'm assuming maybe the double processing of the same log
> file in conjunction with the end of the month is causing this missing
> data problem..so I'm deferring to anybody that can exaplin this to me,
> and possibly suggest an alternative or workaround, if possible.

Hmmm. Interesting.
Aren't sure atm as to what the problem could be, but if I explain how things 
work, that might help somewhat.


Basically there are two data/state files that tell awffull what's what from 
one run to the next.
awffull.current and awffull.hist.
Both are plain text and easily editable. Tho I'd advise generally avoiding 
awffull.current. :-)
Both are stored in the output directory - so so long as the output directories 
don't overlap - shouldn't have a problem. Key word: shouldn't :-)


awffull.hist - is *really* simple. This is the one that generates the history 
shown on the very front page.
The one to watch for in there is the startday and end day fields - actually 
had a bug in 3.10.1 on that - fixed in 10.2. These are the 7th and 8th fields 
on each data line.
And they are what determine the average # days in a month. I'm... unconvinced 
this is the best way - actually I'll rephrase - it's downright sucky, but it's 
what we have atm.
ie endday - startday == number of days, which in turn then is the divisor for 
getting the average.

awffull.current is what holds the most state data. By the end of a month, if 
doing dailies - this can get huge. It's essentially an ascii dump of all the 
memory structures created during a run. Such that awffull can reload and 
pickup from where it left off. More or less.
At end of month, this one gets zeroed. vs awffull.hist which is updated - but 
updated FROM awffull.current.


So in your case where you're generating dailies and want them separate from 
the actual monthly. You really are double processing - as we discussed 
earlier. Somewhat sped up, but the daily analysis not having to reload 
awffull.current like the normal monthly is.


Now as to why the monthly is getting truncated? It shouldn't. The only reason 
it'd be doing so is if the .hist file is wrong - so I'd start checking on that 
first. Make sure that the start day is the first (assuming you had traffic on 
the first).


Does that make sense, and hopefully give a few clues to get started?


Cheers!
- Steve