HOWTO: Protecting access to MSGDIR file in automated systems

ZTN1888

ID: ZTN1888

This Zetafax technical note applies to:

• Zetafax 2012 and earlier versions
• Automated applications using the Zetafax API or ZSUBMIT program

Summary

This document gives best practice guidelines for reducing access contention for the shared message information file, MSGDIR.CTL, when using Zetafax with automated processes and applications.  This helps improve fax throughput, and minimizes the chance of system problems which may affect correct operation of the Zetafax Server and other applications.

It includes advice for software designers integrating their applications with Zetafax using the Zetafax API (Application Programmers Interface), and also for system administrators and support staff wishing to optimize system performance with existing software.

Some of the items rely on features which were introduced with Zetafax 2012 for this purpose, but most will apply to older system versions as well.

More information

Zetafax can be used as part of an automated system, where faxes are sent and/or received faxes are processed automatically.  The Zetafax Automation Toolkit includes a variety of methods to submit and process faxes, including a rich API (Application Programmers Interface) which can be used to build fax functionality into applications.

Zetafax uses a series of shared files to store information about submitted messages, and to pass information between the Zetafax server and client applications.  This Zetafax API also uses these shared files, and many of the functions in the API result in reading or updating shared files.

For this mechanism to work correctly, it is necessary to ensure that these files are generally accessible.  Zetafax programs needing to access a file will wait for a period if the file is locked (being updated) by another application.  However, if the file is locked for an extended period then the access attempt will eventually timeout, and the operation will fail.

This need to maintain access is rarely an issue when Zetafax is used as a network fax server by multiple users, since the architecture of the shared file system stores messages for each user independently so there is no contention between users. 

However, in automated use it is common to use a single Zetafax user account for submitting faxes automatically.  In such systems it is advisable to design the application so that the Zetafax Server still has sufficient opportunity to access the shared files, ensuring that the application does not block other programs from accessing the files for a disproportionate amount of time.  This is particularly important for the message information files, MSGDIR.CTL, which are used to store information about all messages for a user, and can become a bottleneck if an application is not designed to use the Zetafax API following best practice guidelines.

This document gives advice on ways of reducing contention for shared message files, and so improving the robustness and reliability of the resulting system.  In addition to advice about how to design the application it also includes configuration suggestions which may be followed by a system administrator or other staff responsible for deploying and maintaining an application which has already been developed.

Zetafax MSGDIR.CTL file

Zetafax stores each fax message in a series of files with the same file body name and different file extensions (eg ~SEND123.CTL, ~SEND123.G3F)

Zetafax also stores a summary of all messages in a folder in a file called MSGDIR.CTL in that folder, to make it quicker to display a list of all messages.  Each message (eg “~SEND123”) has one entry in the MSGDIR file.

When a new message is added to the Outbox, Zetafax first finds a filename which is not already used, then checks the MSGDIR file to ensure it does not already contain an entry for that filename before adding a new entry.  When a message is deleted by Zetafax (Zetafax Client, Zetafax Server or Zetafax API), the software removes the entry from MSGDIR.CTL before deleting the message files.

As described above, in cases where a large number of faxes are sent using a single user account it is necessary to protect access to the MSGDIR file to prevent system performance degrading. There are three main techniques to achieve this:

• Reduce the size of the file, by reducing the number of messages in the user’s Outbox
• Reduce the frequency of accessing the file
• Improve the disk access speed

These are described in more detail below.

Application design – reducing the number of messages in a user folder

The following areas will reduce the number of messages in the Outbox for an individual user, reducing the size of the MSGDIR file which will in turn reduce the time taken by applications to access and update the file.

As a general guideline, we recommend that each user folder (Outbox or Inbox) should contain a maximum of 250 messages to maintain system performance, though the exact number will depend on the specific system configuration.

• Automatically delete successful faxes after sending.  The Zetafax API and SUBMIT file format includes a “Delete” message option.  This includes a setting which causes the Zetafax Server to delete the message if it is sent successfully to all recipients.  The successful fax will still be stored in the user’s Sent Items folder and the central server archive (depending on configuration settings), so the user can still check what faxes have been sent if required, but will be deleted from the Outbox.
  
  For many systems the user is only required to take action for unsuccessful faxes, so deleting successful faxes automatically can be better for the user as well as reducing the size of the MSGDIR file.
• Delete completed messages programmatically.  The Zetafax API enables an application to retrieve information about completed messages.  If your application needs to check when the message has been sent (eg to log transmission details), consider deleting the message after these details have been retrieved and stored so that the message list only includes messages which have not yet completed.
  
  If your application needs to display a historic list of completed messages, consider storing that list outside Zetafax and deleting the message from the Zetafax Outbox once it has completed transmission, rather than using Zetafax to store the list of completed faxes.
• Use multiple user accounts.  For systems submitting large numbers of faxes, consider using multiple Zetafax user accounts to spread the load and reduce the number of fax messages stored for any one user.
• Use a single message with multiple recipients.  When sending the same fax to multiple recipients, submit it as a single message.  A message can have multiple recipients, and the Zetafax coversheet designer can be used to customise the content for each recipient if required.

The following areas will reduce the frequency of access to the MSGDIR file, which will in turn reduce the likelihood of delays in access by the Zetafax Server impacting system performance.

• When retrieving the status of all messages for a user, use the GetMsgList() API function where possible, rather than creating an object for each individual message or using the more detailed GetMsgInfo() function
• Reduce the frequency of checking for new messages and status updates.  A two page fax typically takes around 1 minute to send.  For most applications in production use, a check once every 1 minute (or longer) is quite sufficient, and will give better overall performance with a large number of messages than a check every 5 or 10 seconds.
• Pause after operations which access MSGDIR.  When a new message is submitted, the Zetafax API locks the MSGDIR file to add a new entry for the message.  If a batch of messages are submitted in quick succession the repeated locking of the file can block access by other programs for an extended period.  You can protect against this by adding a short pause (eg 5 to 10 seconds) after submitting a new fax or deleting a message - this is unlikely to make a significant impact on the application or user experience, and can help increase throughput of the server.

System configuration – reducing the number of messages in a user folder

The following areas are changes which can be made by system administrators or support staff, without needing any change to the application.

• Auto-delete successful messages.  With Zetafax 2012, the Zetafax Client can be configured so that successful messages are deleted from the Outbox automatically on completion.  Applications using the Zetafax API use the client setting as a default, so if the application does not explicity override this setting, changing the client setting to delete successful messages will apply to messages submitted by the API application as well.
  
  

  Note that this setting should not be changed if the application checks the status of messages and performs any processing for successful messages – eg updating a separate log of sent messages stored in the application or elsewhere.  Please check with the application developer if in doubt about whether this applies.

• Configure Auto Cleanup quota rules for the specified Zetafax user account.  Zetafax 2012 introduced a new Auto Cleanup feature which can be used to restrict the number or age of messages in a user’s Outbox.  These rules can be specified independently for each user account.  On systems where a single account is being used for high volumes of faxes then we recommend that Auto Cleanup rules are enabled for that account to limit the number of stored messages by deleting older completed faxes.
• Regularly delete completed faxes using the Zetafax Client.  The Zetafax Client includes a Tidy-Up feature which enables you to delete messages from the Outbox based on selected criteria (age and whether they were successful).  If it is not possible to delete messages automatically using Auto Cleanup (as described above), then we recommend regularly deleting older messages as part of the standard daily operating procedures.

System configuration – reducing access to the MSGDIR file

The following areas are changes which can be made by system administrators or support staff, without needing any change to the application.

• Install the FaxBroadcast add-on licence.  With a standard Zetafax system, the Zetafax Server records progress for each message by updating the status in the MSGDIR file.  This enables users to see the progress of the message from submitted, preparing, waiting for device, dialling, sending, etc. 
  
  The FaxBroadcast licence changes several aspects of the Zetafax Server functionality to optimise it for higher volume transmission, including reducing the number of interim states which are reported.  In most automated systems it is not necessary to track progress of the message, and it is sufficient just to be notified when the message completes.  Installing the FaxBroadcast licence add-on can significantly reduce the amount of access to the MSGDIR, both by the server in updating the status and by the Zetafax Client in rereading the file when the status of a message changes.
• Disable the Automatic Submission Program “delete message” option in the configuration program.  The ZSUBMIT program can be configured to delete messages for a user.  Although this is sometimes enabled to alleviate MSGDIR access problems, on higher volume systems it can have the opposite effect by adding an extra program which is reading and updating the MSGDIR file.  This option has now been superceded by the server “delete” message option and Auto Cleanup, which are more efficient options, and that this option is disabled unless required for historic reasons.
• Close the Zetafax Client program after use.  The Zetafax Client program monitors the Outbox for the logged in user, and will reread the MSGDIR.CTL file if a change is detected.  On high volume systems it is recommended that users are advised to close the Zetafax Client program when it is not being used, to reduce the number of programs which are reading the shared message files.
  
  For hybrid systems which are also used for manual faxing, this is only necessary for user accounts which are being used for automatic faxing – typically a single account.
• Disable antivirus software.  The Zetafax user folders (zfax\USERS) are not used for storing applications, but only data files for use by the Zetafax applications.  We recommend excluding these folders from antivirus checks on file access, to reduce the access time by Zetafax applications and reduce the file being locked by the antivirus program after being updated.

System configuration – improving file access speed

The following areas are changes which can be made by system administrators or support staff, without needing any change to the application.

• Improve the fileserver access speed.  For higher volume systems, system throughput is affected by the access speed for the Zetafax Server file system.  Moving the file system onto a fileserver with faster access speed may improve throughput, and will reduce the likelihood of file contention problems on automated systems.
  
  The default installation options for Zetafax store the files on the server computer local hard drive.  This can be changed during installation, or the file system can be moved after installation if required.  Please contact Equisys technical support staff for further information if needed.

References

Further information about the options described above can be found in the Zetafax API reference guide, the Zetafax Configuration program online help and the Zetafax installation guide.

Last updated: 24^th April 2012 (GW/DH)