Reporting

Three types of usage reports are available in surgemail, each provides slightly different information:

  • Monthly usage reports
  • Current status reports
  • High use log file analysis

Monthly usage reports (fast to generate)

The Monthly usage reports are an efficient way of providing usage statistics on a per account, per domain or per server basis. This will typically be used for accounting purposes or for routine monitoring of the behaviour of users of the system.

A variety of statistics are recorded on a per account basis as each message is processed. These statistics can be broadly classified as relating to mail sent and received, and to mail retrieved via POP and IMAP. For local accounts both these sets of information are recorded. If surgemail is running in a surgewall configuration or has gateway rules defined, statistics are also gathered on the "gateway" accounts. These are obviously not real accounts on the system and this usage information is based on mail delivered to unique gateway email addresses in this case.

Sample report

Account usage (all domains) - matched 5 records today

account                      msg Rcv    mb Rcv  msg Sent   mb sent  --  p/i sess   mb_read  --     ~~~~~~~
                                                                                                   wrapped
joebloggs@mydomain2.com            3       0.0         7       0.0  --         1       0.0  --      below
marijn@mydomain.com                8       0.0         4       1.6  --        10       0.0  --     ~~~~~~~
user1@mydomain.com                 4       1.6         0       0.0  --         1       0.0  --
user1@mydomain2.com                4       0.4         0       0.0  --         0       0.0  --
user2@mydomain2.com               14       1.4         0       0.0  --         0       0.0  --

Total                             33       3.4        11       1.6  --        12       0.0  --

~~~~~~~
       last connected             lastip   cpu_sec  elap_sec  mb quota  -- virus blk  --  msgsleft   mb_left

 12-Oct-2004 12:25:30          127.0.0.1         0        54       0.0  --         0  --         3       0.0
 12-Oct-2004 12:20:09          127.0.0.1         0        14       0.0  --         0  --         0       0.0
 12-Oct-2004 12:24:34          127.0.0.1         0        22       1.6  --         0  --         4       1.6
                never                            0         0       0.4  --         0  --         0       0.0
                never                            0         0       1.4  --         0  --         0       0.0

                never                            0        90       3.4  --         0  --         7       1.6
~~~~~~~

The above report contains the following information:

  • Number of messages received + size in Mb
  • Number of messages sent + size in Mb
  • POP / IMAP sessions + size of data read in Mb
  • Time and IP address of last POP/IMAP connection
  • Actual CPU time used (only accurate on windows) and elapsed CPU time
  • Current quota used (in Mb)
  • Viruses blocked on this account
  • Messages and size of messages left online during last pop session

Domain usage reports provide a summary per domain with a total of all domains at the bottom. Account usage reports list each account individually with a total of all entries at the bottom. In addition the reports can be sorted on any column and limited to the top N accounts. In advanced mode wildcard exclude and limit filters can be applied similar to the main log file searching.

Available from

These reports are available to domain administrators and to server administrators using the Reports link in the web admin interface. In addition these reports are available on the command line using the following command: tellmail report report_type [output file] [days=n sort_type=user ...]

eg. Top 20 senders of mail over the last 5 days saved to output.txt:

tellmail report account_usage c:\output.txt days=5 top=20 sort_type=bytesent

eg. Top 10 domains mail over the last 20 days counted by messages sent output to the console:

tellmail report domain_usage days=20 top=10 sort_type=msgsent

Valid values to pass each of the options of this tellmail command can be found by checking the na_reports.htm template file. You can pass the tellmail report command the following optional parameters: domain, days, sort_type, top, exclude, limit.
In addtion you can pass a date range over which to generate the report. To do so use "range=DDMMYYYY-DDMMYYYY" instead of "days=n".

Miscellaneous

The data for these reports is stored in daily usage snapshots (recYYMM/statYYMMDD.dat) with a monthly summary (recYYMM/summary.dat). When generating these reports it is most efficient to generate a report for "previous calendar month" as the monthly summary just gets loaded and appropriately filtered and displayed. In any of the n days to-date reports, each of the daily summary files gets totaled before the report gets generated. These statistics snapshot files are stored in binary format but can be dumped as text format using the command: tellmail pstat_dump filename

There are several other aspects one should be aware of as to the detailed operation of the mail sent and received logging:

- Under default installed operation "mail sent" is attributed to the account logged in using SMTP authentication. If the message is sent without SMTP authentication then the mail from address is used to determine the account sending mail. This can obviously be faked. The setting g_acctlog_authonly allows you to only record messages sent from a particular account if smtp authentication is used.

- For surgewalled accounts / gatewayed accounts the delivery of a message to a unique gatewayed email address will start the recording of mail sent and received on a particular account.

- When a message is redirected either from a particular account (eg user.cgi level forwarding or using aliases or redirection rules) this is attributed to a particular accounts as having "sent this message". This allows for the easy identification of accounts that have high volume redirections to non local addresses. This logging can be disabled using g_acctlog_noaliases.

- If you have a large number of inactive accounts in your userdatabase the recording of deliveries to these accounts can make the record files grow excessively large. If g_acctlog_sum_inactive is set, any mail delivered to an account that has never logged in yet is attributed to the not_logged_in_yet@yourdomain.com.

In addition to normal accounts usage information is also logged for the following special case accounts. Aliases as noted above, mailing lists and robots including the isspam address. eg:

Account usage (all domains) - matched 5 records today

account                            MSG Rcv    Mb Rcv  MSG Sent   Mb Sent  -- ...
alias@mydomain.com                       0       0.0        11       0.0  -- ...
isspam@mydomain.com 0 0.0 12 0.0 -- ...
testlist_marijn-bounce@mydomain.com 0 0.0 4 0.0 -- ...
testlist_marijn@mydomain.com 1 0.0 0 0.0 -- ...
|c:\surgemail\some_robot.exe 4 0.0 0 0.0 -- ...
|isspam 70 0.5 0 0.0 -- ...
Total 75 0.5 27 0.1 -- ...

 

Current status reports (may be slow to generate)

The current status reports are generated based on actual users currently in the authentication database. If you have a slow authentication database and a large number of users the generation of this report can be rather slow.

Currently only a disk quota status report is available.

As for the monthly usage reports this report may be sorted on any column and limited to the top n entries or limited via the filters in advanced mode. Also this report is available to domain administrators.

Also as for the monthly reports this report can be generated on the command line using : tellmail report quota c:\output.txt top=20 sort=quota_pct

Account name                         % used   bytes used   byte limit


user2@mydomain.com                      34%        6.9mb       20.0mb
user1@mydomain.com                       7%        1.5mb       20.0mb
marijn@mydomain.com                      0%        8.0kb       20.0mb
user1@mydomain2.com                      0%        392kb    unlimited
joebloggs@mydomain2.com                  0%        2.6kb    unlimited
user2@mydomain2.com                      0%        1.3mb    unlimited


Total                                    NA       10.2mb    unlimited

High use log file analysis (may be very slow to generate)

This report screen lets you analyse a section of the delivery logs on the fly to trouble shoot a particular "high use" problem.

  • From IP address
  • From Email address
  • To Email address

The report can be sorted by the number of messages:

  • Sent
  • Stored
  • Rejected
  • Failed
  • Spam
  • Received

You can also specify the time period, and number of lines of the report to display. For Example:

     Size   Stored Rejected  Spam  Sent Received Failed   InQue To
      23k        0        1     0     0        2      0       1 chrisp@netwin.co.nz
       4k        3        0     0     0        3      0       0 test3@lap
       3k        0        1     0     0        0      0       0 spam2@netwin.co.nz

The complete delivery logs are processed each time for the generation of this report. If the log files are large then this is likely to be a rather disk io intensive process. For a typical system of 50K active users the delivery logs could easily be 250mb per day.