Backend Operation Manual

Table of Contents:
About this Document
Components of MailDepot
Configuration Files
Startup Procedure
Command Line Options of Backend Process
Logging of Backend Process
Process Monitoring Daemon
Alert Monitoring Daemon
Job Management Daemon
Mail Archiving Daemon
Daily Processing Script
Monthly Processing Script
Backup Processing Script
Apache, PostgreSQL Log

About this Document

This document describes the backend processes and operations in MailDepot.

For command line tools of MailDepot, please refer to "Command Line Tools Manual".

Components of MailDepot

The components of the backend processes in MailDepot are shown below.

  md_watch --+-- md_alert
             |
             +-- md_job
             |
             +-- md_import
             |
             +-- md_daily.php
                    |
                    +-- md_monthly.php

md_watch: Process Monitoring Daemon

This process monitors the status of all processes, and starts them up if not running. Besides, it starts up daily processing once a day.

md_alert: Alert Monitoring Daemon

This process monitors the system status and issues an alert if abnormal status is found.

md_job: Job Management Daemon

This process manages the Mail Importing Process.

md_import: Mail Importing Daemon

This process retrieve archive mail messages from the POP server, and stores them in the database.

md_daily.php: Daily Processing Script

This script runs once a day when the date changes, and does processing such as updating the previous day's statistics, etc.. This script also move old mail partitions (monthly archived data) to offline status if your archive volume is near your license limit.

md_monthly.php: Monthly Processing Script

This script runs once a month when the month changes, and does processing such as rotating log files, etc..

Configuration Files

maildepot.conf

After install, maildepot.conf is created on /opt/maildepot3/etc, and this file is used as a template for each domain.
On creating new domain, maildepot.conf is created on /opt/maildepot3/domain/NEW_DOMAIN/etc according to template. This configuration file is used as system settings for each domain.

md_license.conf

This configuration file keeps the license infomation. Don't edit this file directly because you can manage the licenses with md_licnese command. See License Management for more.

md_authenticity.conf

The md_authenticity.conf file holds the important information which is used for non-falsification checking feature. So, if you want to restore backup file into another MailDepot server, you must restore this configuration file before mail archiving is started. If you forget to restore this configuration file, non-falsification checking for restored mail messages will be allways failed.

md_import.conf

In archiving mails, mail archiving daemon (md_import) refers following configuration file for each POP account. 

/ope/maildepot3/domain/DOMAIN_NAME/etc/md_import_POP_USER@POP_HOST:POP_PORT.conf

In this configuration file, you can specifiy following option.

md_import_disk.conf

This configuration file keeps a disk name (table space name) that will be used when the next monthly partition is made.

next_table_space = "pg_default"

Startup Procedure

If the mail archiving is started on Administration Console, the md_watch daemon is started first, then md_alert. md_job and md_import is started by md_watch daemon.

When started, each program saves it's process ID in a pid file (ex. /opt/maildepot3/domain/DOMAIN_NAME/var/run/md_watch.pid). When stopped, this pid file is deleted automatically.

Each program can be started up only by a Maildepot super-user.

Command Line Options of Backend Process

md_watch, md_alert, md_job and md_import take the following options as arguments.

Logging of Backend Process

The backend process of MailDepot saves all logs in syslog. Further more, logs that have a log level of NOTICE and above are saved (if possible) in the database operation log as well. The operation log can be viewed in Administration Console.

MailDepot uses LOCAL1 as syslog facility to output logging messages. By changing the settings file for syslog (/etc/syslog.conf), MailDepot can create an independent log file. If you wish to produce the MailDepot log under /var/log/maildepot.log, add the following setting to syslog.conf. 

  local1.*			/var/log/maildepot.log

The information which is produced is as follows (Log level is shown in brackets).

Normal Information (INFO)

Important Information (NOTICE)

Warning (WARNING)

Error (ERROR)

Alert (ALERT)

Process Monitoring Daemon

The process monitoring daemon (md_watch) checks whether the alert monitoring daemon (md_alert) and mail archiving daemon (md_import) are running in every 10 seconds. If they are not running, they will be started automatically. An error log will be produced if failed to start in 3 attempts.

If "stop" argument is passed when executing monitoring daemon, or if an terminate signal (SIGHUP, SIGINT, SIGTERM, SIGQUIT) is sent, monitoring daemon will be stopped after sending an terminate signal to md_alert, md_job and md_import.

Alert Monitoring Daemon

The alert monitoring daemon (md_alert) monitors the system status at every 1 minute interval, and produces an alert message in the syslog in case of any problem in the system. An alert messages is also sent to the mail address set in Administration Console (except when the database cannot be accessed).

The following status are monitored.

System Abnormality

Abnormality in Statistical Values

If the same alert continues, in case of system abnormality, new alert is issued only after more than 1 hour has passed since the last alert was issued. In case of abnormal statistical values, an alert is issued only after 1 day or more has passed since the last alert was issued.

The threshold values for the alerts can be changed in Administration Console.

Job Management Daemon

The job management daemon (md_job) checks the POP account information every 20 seconds, and registers the mail archiving job in the database.

Mail Archiving Daemon

The mail archiving daemon (md_import) checks the mail archive job regularly, then imports the copied mail messages from the POP server at specified intervals, and stores them in the database.

When archiving mails at the beginning of the month, a new mail partition for that month is created. If disks have been added, the disk with the least usage volume is automatically selected for the mail partition.

If multiple POP servers are specified, although the archiving time is overlaped, mails are sequentially archived from the 1st server in order. In that case, the server with the last archive time will be given priority. If the start of importing is delayed beyond the prescribed time, then that job will be skipped.

In order to enable automatic identification of japanese chanaratr encoding, the locale setting in system startup must be set to a Japanese locate such as LANG=ja_JP.UTF-8.

In the case that your operating system has /etc/mime.types file which describes a list of MIME types and extensions, the file type of the attachment can also be ascertained from the extension (if Content-Type: is set to application/octet-stream). The mime.types files are provided in the mailcap package.

Mail Archiving Daemon (md_import) will store prcess status information into following file while imporing mail messages. 

/opt/maildepot3/var/tmp/md_session-XXXX.log
This file will be removed after imporing process is finished successfully.

Process Monitoring Daemon (md_watch) will restart Mail Archiving Daemon (md_import) when abnormal erros are occured in imporing process. In this case, Mail Archiving Daemon will refer saved status file, then recover aborted imporing process.

In recovery process, attachments in aborted mail messages may be stored as unknown data format and cannot be searched with full-text search.

If unexpected erros are occured in imporing process, aborted mail messages may be saved in following directory with EML format.

/opt/maildepot3/var/mail

Daily Processing Script

Daily processing script (md_daily.php) is invoked everyday by md_watch at 12:00 A.M. (when the date changes), and does the following tasks.

Even if the daily processing script could not be invoked because of reasons such as the MailDepot system being temporarily stopped, you do not need to do anything for recovery because the necessary processing will be done together with that of the next day. In this case, the statistical processing etc. will be delayed by one day. But if you want to reflect the information immediately, you can manually run the md_daily.php script. In this case, statistical information will be reflected after the md_daily.php is executed. You must run md_daily.php as maildepot super-user. 

  % sudo -u maildepot -i /opt/maildepot3/bin/md_daily.php DOMAIN

Further, the md_daily.php takes the following options as arguments. It is useful for times such as during manual execution, in the case of you running only specific task.

Monthly Processing Script

The monthly processing script (md_monthly.php) is invoked every month on the first day (when the month changes) through md_daily.php script, and does the following tasks.

If the monthly processing script could not be invoked because of reasons such as the MailDepot system being temporarily stopped, the necessary processing is done together with that of the following month. If you wish to do processing without waiting until the following month, you can manually run the md_monthly.php script. You must run md_monthly.php as maildepot super-user. 

  % sudo -u maildepot /opt/maildepot3/bin/md_monthly.php DOMAIN

Backup Processing Script

When the backup operation is requested in Administration Console, md_backup is invoked and create the backup files for specified mail partition. While running, md_backup upates the backup status for specified mail partition.

Apache, PostgreSQL Log

As a norm, the Apache or PostgreSQL Log is produced into the file below. The log file that contains [YYYY-MM-DD (Y-M-D)] in the file name is rotated day-wise. Further, the log file given below is not automatically deleted. Please ensure that you regularly delete past log files to prevent using too much disk space.

Apache Access Log
/opt/maildepot3/var/log/httpd/access-YYYY-MM-DD.log
Apache Error Log
/opt/maildepot3/var/log/httpd/error-YYYY-MM-DD.log

PHP Error messages, etc. are also produced into this file. In cases such as nothing is displayed in Administration Console, check this file.

PostgreSQL Log
$PGDATA/pg_log/postgresql-YYYY-MM-DD.log

$PGDATA is the directoty for database cluster. As a norm, the database cluster directory is set to "/opt/maildepot3/data". and configured as db_datadir parameters in /opt/maildepot3/etc/maildepot.conf.

PostgreSQL Startup Log
/opt/maildepot3/var/log/postgresql-startup.log

Only if the startup of PostgreSQL has failed, an error message will be produced into this file.