SEP sesam extension for Lotus Domino Server

From SEPsesam

SEP sesam Extension for Lotus Domino Server

(C)SEP AG

Copyright 1999-2009 by SEP AG. All Rights reserved.

Any form of reproduction of the contents or parts of this manual is allowed only with the express written permission from SEP AG. When compiling and designing user documentation SEP AG uses great diligence and attempts to deliver accurate and correct information. However, SEP AG cannot issue a guarantee for the contents of this manual.

Introduction

The SEP sesam Client SBCLN is a program library for backups of Lotus Notes/Domino databases. To backup a Lotus Notes Server transaction oriented Transaction Logging has to be activated in Archived Mode on the Lotus Notes Server. Transaction Logging is part of the Lotus Notes System. Notes databases can also be backed up without Transaction Logging although the databases are always saved as full backups this way.

Another important point in planning a backup strategy is considering a valid DBIID (Database Instance ID). Transactions only referr to information with an identical DBIID of a database. If tools like fixup or compact are used on databases the database usually gets a new DBIID and all transaction information that was previously created for it is then turned invalid. We therefore strongly recommend to run a full backup of the database right after using these Notes tools. Neither planning nor operating these Lotus Notes tools is part of SEP sesam.

Additional information on the topics Transaction Logging and DBIID can be found in the Lotus Notes documentation of IBM (a link is provided at the end of this document).

The Backup-API of Lotus Notes offers no direct backup or restore options for single eMails or documents. The API is designed more like a relational database and set up for disaster recovery. A single mail recovery is possible but needs profound knowledge of Notes administration. The chapter Restore/Single Mail Restore briefly describes the practical procedure of a Single Mail Restore.

Prerequisites

Right now the Lotus Notes Online Backup is available on the following platforms:

• Windows (Windows Server 2003/2008)
• Linux
• Solaris (Version 9 and 10)

To use the Lotus Notes Extension the following SEP sesam components are necessary:

• Sesam Server Version 3.4 or higher
• Sesam Klient Version 3.4.1.88 or higher

Configuration

Linux

• A fully installed SEP sesam Linux client that can be backed up from the Sesam Server via path-backup.
• Copy the content of the SEP sesam Extension for Lotus Notes into the directory <SESAM_ROOT>/bin/sesam/.
• To work with Notes and to load the library LOTUS, NOTES_DATA_DIR, Notes_ExecDirectory and PATH have to be set according to the Notes environment and the LD_LIBRARY_PATH has to direct to the directories of the libraries libsbcln.so (located in <SESAM_BIN>/bin/sesam) and libnotes.so (in the binary directory of the Notes installation).

The [ENVIRONMENT] section has to be added in <SESAM ROOT>/var/ini/sm.ini at the client. The values listed below depend on the Notes installation. Please use values that apply to your Notes environment.

 [ENVIRONMENT] ... LOTUS=/opt/ibm/lotus NOTES_DATA_DIR=/srv/notesdata Notes_ExecDirectory=$LOTUS/notes/latest/linux Notes_Resources=$Notes_ExecDirectory/res SESAM_LIB_PATH=/opt/sesam/bin/sesam NOTES_LIB_PATH=$NOTES_DATA_DIR:$Notes_ExecDirectory:$Notes_Resources PATH=$PATH:$NOTES_DATA_DIR:$Notes_ExecDirectory:$Notes_Resources LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$SESAM_LIB_PATH:$NOTES_LIB_PATH ...

If no [ENVIRONMENT] section exists in sm.ini it has to be created.

Afterwards, the SEP sesam Daemon has to be restarted.

 /etc/init.d/sesam restart

Windows

• A fully installed SEP sesam Windows Client that can be backed up from the Sesam Server via path backup.
• The Lotus Notes Client Module is already included in the regular Sesam Client for Windows. However, to use it a seperate license for the online extension of Lotus Notes is necessary.
• The system variable PATH must include the path of the library NNOTES.DLL. If this is not the case the variable has to be set accordingly under system – environment variables, e.g.:

 PATH=...;C:\Program Files\Lotus\Domino

And the SEP sesam service has to be stopped and restarted:

net stop sm_mainnet start sm_main

Backup

All further steps of the setup are performed using the SEP sesam GUI. When performing a Lotus Notes Online Backup files with Notes-relevant filename extensions (e.g. ".nsf", ".ntf", ".dic", ".box", ".dsk", ".id" etc.) are backed up over the Notes API at the operating time of the Notes Server.

How does a Notes Online Backup work?

If a backup task with backup type Lotus Notes, backup type Full and backup source all is started the Sesam client connects with the Notes API and requests a backup of all files relevant to Lotus Notes there. Notes takes the first database, copy it to the temporary Sesam Path (gv_rw_tmp in PATHES-section in the sm.ini) to the filesystem and then finds out if any transactions are still in queue. If this is the case these are executed and right after that the database will be backed up by the Sesam client. This way all the Lotus Notes databases are transfered from Notes to the file system and backed up by the Sesam client one after another. Therefore storage space of the temporary space has to be at least as big as the largest Notes database.

When restoring over the Notes API the database that’s restored travels the same way back. The database file is initially put into the temporary Sesam space in the file system by Sesam and then transfered to the Notes API.

Backup-Type

FULL

Here the Notes databases selected in the backup source are backed up completely. The file source all means that all files in the Notes Data Root that are identified as Notes files over the Notes-API are backed up. In addition to the database files all backup-relevant transaction-logs (TXN-files) are backed up as well.

DIFF

Currently not supported !!!

INC

When executing an incremental backup only backup-relevant transaction-logs are saved. Only all is possible as backup source for incremental backups.

COPY

Same as a Full-Backup without consideration of the transaction-logs.

Entering the backup source

All backup tasks for Notes are set as Lotus Notes in the GUI. The following patterns can be entered as backup source:

Full backup

Of all Notes relevant files of the Notes Server (supported backup types: FULL, INC, COPY). The placeholder all has to be entered in the field Source.

Selective Backup

Using the File Wizard the backup source can be designated here. As an example let’s take the Notes file directory E:\notesdata for Windows and /srv/notesdata for Linux.

• Backup with path statement with one or two directories. Here all Notes-relevant files are backed up with their subdirectories. Contrary to normal procedure with path backups of Linux clients Lotus Notes backups in Linux also follow the directories beyond file system boundaries and symbolic links.
  • Examples Windows:

 a) E:/notesdata/mail b) E:/notesdata/mail1,E:/notedata/help

• • Examples Linux:

 a) /srv/notesdata/mail b) /srv/notesdata/mail1,/srv/notesdata/help

• Backup with file pattern:
  e.g. all databases from directory mail in Windows:

 E:/notesdata/mail/*.nsf

or all templates from the directory help in Linux:

 /srv/notesdata/help/*.ntf 

• Backup of single files (1st line in Linux / 2nd line in Windows):

 1. /srv/notesdata/mail/user1.nsf 2. E:/notesdata/mail/user1.nsf

Options for Lotus Notes in Linux

By default the values for the user that runs the Notes service in Linux and the path of notes.ini server can be stored inin the section [ENVIRONMENT] of sm.ini of the Sesam client. If the Notes user is not notes or differs from the standard value it can be entered in the task options in the register 'Options 1' in backup- and restore options with

 -a USER={ihr-notes-user}

Likewise, if notes.ini is not located in /local/notesdata this has to be entered in the backup- and restore options with

 -a NOTESINI={complete path the notes.ini file}

Example for backup and restore options:

 -a USER=nadmin -a NOTESINI=/srv/notesdata/notes.ini

This information has to be entered for every backup task that wants to backup files on the respective Notes server with the backup type Lotus Notes.

Restore

A restore of Lotus Notes databases begins with starting the restore wizard in the Sesam GUI. There, in the first rider choose the backup task from which you want to restore the database and confirm with <Next>. Then browse through the backup tree to select the file and click <Next> again.

The restore of Lotus Notes databases can be done in two different ways.

To the original location

Here the database(s) is/are restored over Notes API to its original location.

To a different location

Here the database(s) is/are restored outside the visibility of the Notes server into the file system to be processed further there.

To the original location

The restore over the Notes API can be executed selectively (single or several databases) or fully (might only be necessary for disaster recovery).

Here, the following execution options (see red box in the image above) can be chosen:

Upper field)

Don’t overwrite existing files

The database chosen for the restore must not exist in the Notes file directory because otherwise the restore will fail.

Overwrite existing files

If the original database still exists in the Notes system overwrite existing files has to be selected in the options of the restore wizard in order to substitute the existing database. Deleting the original and substituting it with the database from the saveset is done completely from the Note API. The restored file can be accessed immediately after.

Lower field)

No recover after restore

Possibly existing transaction-logs are not performed on the database after the restore, i.e. the databases are just stored at the target system.

Auto Recover and Online after Restore

Applies transaction-logs to every restored database after the restore if there are any.

With the additional restore option -a logs only the transaction loggings from the saveset are stored on the client! This option can be amended under "Expert Options" in the restore wizard.

To a different location

When restoring a database (e.g. user1.nsf) Lotus Notes automatically detects if it is to be restored into a path inside the Notes data path. If a different path (e.g. e:/temp/restore) is used (the restore path can be chosen via the file wizard – Button with black arrow pointing downward on the right hand side of the field) the restore is performed as a path restore without transfer to the Notes-API. I.e. the database is stored in e:/temp/restore/<notespfad>/user1.nsf.

From this point on the admin has to copy the database file to the necessary location in the Notes file path manually according to the requirements of the file restoration.

Point-In-Time-Recovery

The Lotus Notes-API offers Point-In-Time-Recovery. This means it’s possible to bring a database back to a state it was in at some point in time between the last full backup and now. For this the program dbrecover is included in the SEP sesam client. dbrecover originally from the Notes Backup SDK was slightly modified by SEP. The only option thats used for a PITR (Point-In-Time-Recovery) is recover.

To perform a PITR the following prerequisites are necessary:

1. A successful full backup of the database
2. Transaction logging must be turned on and active in Archived mode.
3. All transaction-logs (TXN-files) since the last full backup up until the time of recovery must be available.

To perform a Point-In-Time-Recovery for a database proceed as follows:

1. Restore the database at a different location (e.g. C:\temp). Choose the values Don’t overwrite and No Recover after Restore in the execution options.
2. Make sure that all necessary transaction logs exist in the LogDir of the Notes Server (if necessary through seperate restores) because dbrecover gets the required recover information over the Notes Backup-API.
3. Start the program dbrecover via the command line. For example, in order to restore the database rbogner.nsf from 08/11/2009 04:00 am the following command has to be executed from the directory <SESAM_BIN>\bin\sesam:

 dbrecover recover c:\temp\rbogner.nsf "11/08/09 04:00 am"

After that the following output appears in the console:

 $Id: dbrecover.c,v 1.9 2008/07/25 14:06:41 miraculix Exp $   Recovering backup file ...  Media Recovery Replay (60 MB): 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 11.08.2009 10:08:52   Recovery Manager: Media Recovery complete for c:\temp\rbogner.nsf, last update applied 11.08.2009 03:58:20.   Backup file c:\temp\rbogner.nsf recovered.  Program completed successfully.

Finally the database has to be copied into the file space of Lotus Notes.

Single-Mail-Restore

In this example a possibility is shown with which the Notes administrator can restore single documents/mails.

The user of the mail database user1.nsf accidently deleted an Email today. Therefore it must have existed in last night’s backup. Lotus Notes can’t restore an existing database into a different directory or with a different name on the same server directly over the Notes Backup-API because Notes will notice that a database with the same Replica-ID already exists.

The only solution is to restore the database into the file system, create a copy of this database and then copy the required eMail documents from it to the restore destination.

To do this the following procedure is necessary:

1. Restore the database user1.nsf on the Notes Server into the file system (not in Notes Data Root)
2. Open user1.nsf locally with the Notes Client or the Notes Administrator and create a copy with the name user1a.nsf on the Notes Server. Creating this new copy assignes a different Replica-ID to the database user1a.nsf than the original user1.nsf has. Therefore Lotus Notes sees them as two completely different databases.
3. Given the necessary access rights both databases can now be opened and the required eMail documents can be copied from user1a.nsf to user1.nsf.
4. After restoring the eMail documents just delete the database user1a.nsf.

Desaster Restore

Prerequisites

To prepare a Lotus Notes Server for a possible disaster recovery the following backups are necessary:

1. Operating System
   1. Windows
      1. Systempartiton with VSS
      2. Backup of system data (system_state)
      3. Backup of the Notes application (directory with the Notes binaries)
   2. Linux
      1. Complete Linux system without Notes data (all-Backup)
      2. Backup of partition data (disk_info)
2. Lotus Notes online backup of all Notes data with the source all.

Execution

1. In case of a disk crash the first things that are restored are the OS and the Notes program files. Please do not start the Notes service!
2. Next, the Lotus Notes data is restored as a path backup to the original Notes datapath – with overwrite. Hereby all databases, transaction loggings, ID-files and the notes.ini are copied to the server.
3. Before starting the file(s) nlogctrl.* has to be deleted in the log directory.
4. After the successful restore of all Notes data the Notes Server service can be started. While starting the Notes server automatically checks the consistency of the databases and recovers their states. The Notes recover procedure can last some time. It depends on the number of the databases of this server and their volume.

Troubleshooting

Backup reports Recovery may fail

Problem

When doing the backup the following warning appears for several or all Notes databases:

sbc-2076: Warning: Item [D:\notus03data\mailboxes\mail1\cruoff.nsf] is not logged. Recovering may fail.

There may be two reasons for this:

1. The Transactional logging was not turned on or is not in Archived mode.
2. Transactional logging is running in Archived' mode but is explicitely turned off for this database in the options of the database.

Solution

This message is a warning of the Notes Backup-API and is only forwarded by SEP Sesam. Either turn on the Transactional logging in general or just for this database. Other than that this message can be ingnored. The warning is given during the backup for safety reasons because only the admin can determine if administrative intervention is neccessary.

Temporary space is not enough

Problem

Since every database has to be copied on the client first temporary space may not suffice.

Solution

Change the path gv_rw_tmp in <SESAM_VAR>/ini/sm.ini to a directory with enough space.

Transactional logging not active

Problem

The INCR backups fail.

Solution

The "Transactional logging in Archived mode for Notes has to be activated.

Transaction logs already in progress

Problem

In the Sesam log appears the message:

DB Module: [Archiving of transaction logs already in progress.]

Solution

The Lotus Notes API sends this message to show that the transactional logs are already in state Backup. The state cannot set twice. If a backup could not be completed successfully this state can appear. To finish the Backup status all logasio processes should be removed on the Notes system:

 UNIX:     killall -9 logasio WINDOWS:  sm_kill logasio

Alternatively, you can reboot the Notes server.

Backup reports incorrect Transactional loggings

Problem

After a restore of Transactional loggings Notes reports incorrect Transactional loggings.

Solution

Delete the files nlogctrl.* in the log directory. After this the Notes server has to be restarted.

Restart of the server after Notes server crash fails

Problem

The server can not be started without reboot anymore after a crash of the Lotus Notes server.

Solution

If a server crash occurs during a log backup logasio processes can stay active. The processes must be stopped.

 UNIX:     killall -9 logasio WINDOWS:  sm_kill logasio

After this the server can be restarted.

Backup with missing options

Problem

Notes restore on Linux client ends with

 "RESTORE STATUS: Restore failed. 2007-05-02 08:52:32:  sbc-1146: Error:    DB Module: [Notes API NotesInitExtended() failure"

Solution

The restore options must be set identically to the backup options. Please set the restore options accordingly in the task. Additionally, the options can be set in the Restore Wizard under <Expert Options> in the tab Options.
Example:

 -v 3 -a USER=nadmin -a NOTESINI=/srv/notedata/notes.ini

Lotus Domino Console repeats: C:\..\SEPSesam\bin\sesam\sbc.exe Process has terminated abnormally

In case of a Backup failure which caused the Sesam Backup Client to terminate abnormally, the Lotus Domino ServerAPI will notice this and repeat this error each minute in its logging.

If this consists over multiple reboots, please see this article from IBM how to remove theMessage from the Domino logging:

http://www-01.ibm.com/support/docview.wss?uid=swg21252974

Solution is to stop the Domino Server Forcibly by using "nsd -kill"

Further Links/Literature

• SEP sesam Extension for Lotus Domino Server
• IBM Notes Backup & Recovery API
• Transaction Logging on Domino Servers
• Tranactional Logging and DBIID