Home » GFI User Forums » Kerio Connect » Differential Backups Seem to Miss Emails
| Differential Backups Seem to Miss Emails [message #117725] |
Mon, 24 November 2014 13:46  |
 |
jcooper
Messages: 113
Registered: May 2009
Location: Syracuse, NY
|
|
|
|
Hello All,
I'd like a reality check before I submit this as a rather serious bug. This weekend I moved our KMS to a new server. I did the migration using "Method 1" from here:
http://kb.kerio.com/product/kerio-connect/server-configurati on/export-and-migration/transferring-an-installation-of-keri o-connect-to-another-server-or-operating-system-360.html
A full backup takes about 15 hours and it's hardly practical to shut down my mail server for that long. So, I did a full backup earlier in the week (Tuesday). My plan was to do a Differential backup the morning of the move, then use kmsrecover to pull the data into the new server.
The live server was on during the full backup earlier in the week. Prior to the differential backup, I turned off all the services to effectively "stop" emails but keep the admin UI working so I could schedule the backup (our gateway held the incoming messages while KMS was off).
To recover the data, I used > ./kmsrecover /volumes/mailbackup/backup
I did a couple test runs prior to the move and it all seemed to go well; it took about 90 minutes to recover the data after which I turned the server on and all seemed fine. A test email I sent to myself just prior to the differential backup appeared on the new server. I should have looked closer.
No sooner had I completed the actual migration that I started getting calls from users who were missing emails. A comparison of the two servers indicated that most (but not all) emails for the two or three days prior to the migration did not migrate.
We only have 50 or so users, and we're on Mac, so I was able to mount the old server as a share and use Carbon Copy Cloner to manually copy the missing contents of each user's store, reindex the mailbox, and they came back (iOS users had to turn their mail accounts off then on again for the missing emails to show up). "Only" took a few hours.
In the end the migration worked, albeit with a few panic attacks. But I think this problem means there's something seriously wrong with Kerio's backup system.
Like most places, we rely on these backups in case of a server meltdown, fire, etc. I have the full backup off-site and take the differentials off-site regularly. My assumption was if the server room exploded we'd have all our emails up to the last backup. But now it seems we have our emails except for the couple days leading up to the last backup.
I did find a couple threads alluding to others having similar issues, but with no clear resolution. Did I forget a needed option for kmsrecover or to check a box somewhere?
Thanks,
Jeff
|
|
|
|
| Re: Differential Backups Seem to Miss Emails [message #121830 is a reply to message #117725] |
Fri, 05 June 2015 10:34  |
greg_m
Messages: 19
Registered: July 2008
Location: Cheltenham, UK
|
|
|
|
Hi, did you get to the bottom of this? I'm seeing the same thing. We're trying to migrate from an old Xserve to a new HP Prolient server and after a test run I found that some, but strangely not all emails, were missing from the few days before the backup was run.
I've spoken to Kerio support and they admit that the recovery process is not 100% reliable, and say we will just have to accept the loss of the emails.
This has seriously dented my confidence in Kerio's backup and recovery tools. We are now considering migrating to hosted Exchange or Google as we can't trust KC anymore, but I just wanted to see if I'd missed anything.
Thanks
|
|
|
|
|
|
| Re: Differential Backups Seem to Miss Emails [message #121835 is a reply to message #121830] |
Fri, 05 June 2015 13:09 |
|
jcooper
Messages: 113
Registered: May 2009
Location: Syracuse, NY
|
|
|
|
Didn't get a resolution. Moved on once I was able to manually migrate the missing messages. It's been a while so I was re-reading my post a bit more objectively (as if someone else had written it) and it now occurs to me perhaps the restore command doesn't work as I thought it did (or how I read it in the docs). With differential backups, I'm wondering if they need to be restored independently of base full backup and/or explicitly selected. Perhaps my backup command only grabbed the last full backup and if there are differential they need to be restored by running the command again, but explicitly specifying the D...zip file.
Just a theory and I just thought of it five minutes ago; other projects took over space in my brain following the migration. So I hadn't really thought of it since.
If I get some time in the coming weeks I may run a test to see what happens. If and when I do, I'll post the results.
Thanks,
Jeff
|
|
|
|
|
|
| Re: Differential Backups Seem to Miss Emails [message #121840 is a reply to message #121835] |
Fri, 05 June 2015 16:17 |
greg_m
Messages: 19
Registered: July 2008
Location: Cheltenham, UK
|
|
|
|
Hi Jeff, thanks for getting back to me, I appreciate it was sometime ago. As I understand it pointing kmsrecover to the backup folder will cause it to use the latest full plus any diffs associated with it. You can specify a particular backup set by using the filename in the command.
The command I used was;
kmsrecover -v -s D:\store E:\[backup directory]
This reported a successful recovery. However, there were messages missing. The important thing that I noticed, and you noticed too, was that NOT ALL emails were missing. If the full backup was performed on day 1, the incremental on day 2 and the recovery on day 3, then if the incremental had not been included in the restore, than you would expect to see only emails from day 1 on the restored server. We saw some emails from day 2, but not all. For instance, one user had emails from 9am in the morning, but some from 11am were missing. This is after I had reindexed all mailboxes, so the messages reported by the server should be correct.
I took screen shots of the user list showing the total item counts before and after the reindex (attached) and there were differences of up to several hundred files.
Mark/Pavel
Your confidence in the backup/recovery process is admirable and comforting, so perhaps you can help me, as my experience (and perhaps Jeff's) seems to contradict your claims? We can't reliably migrate from an old server to a new one right now. This is understandably causing us a headache. I've tried first line support and the advice given is that there isn't much I can do about these missing files, so if the wider Kerio userbase or second line support have any good ideas as to how I can fix this, I'm all ears!
Having spent £6,000 on a nice shiny new HP server, I'm not looking forward to telling my MD we can't move away from the 7 year old server that is failing.
|
|
|
|
| Re: Differential Backups Seem to Miss Emails [message #121867 is a reply to message #117725] |
Mon, 08 June 2015 12:20 |
greg_m
Messages: 19
Registered: July 2008
Location: Cheltenham, UK
|
|
|
|
It has been pointed out by Scott Riegelhaupt-Herzig (Kerio's Technical Director in the US) that I have misrepresented Kerio's UK support team on this issue and I have to agree with him, to a point. Kudos to him for writing to me directly to clarify the matter.
I made the assertion that UK support had told me that the recovery process was not 100% reliable, a statement that I must retract. What the support representative actually said was "Sadly there is no real way to determine how an index has become corrupted after a migration like this. Sometimes it can be to do with the change in OS and how the OS is reading the files, if it locks the files, if there are too many emails in the folder the index belongs too etc... But there is no 100% answer as to what causes this issue.". So not quite the same thing, but still enough for me to be concerned about.
And this still doesn't alter my position that I am unable to complete a reliable recovery from a backup whilst migrating from a Mac to a Windows server. Perhaps there are factors relating to my environment (and possibly Jeff Cooper's) that have caused this, but the essence of the support response was that there was little they could do to help. Hence the reason for coming to these forums for further help and insight.
I wish Kerio would invest some money into bringing the recovery tool into the management console and expanding its capabilities. Using the command line tool has required a lot of work, searching and support tickets to explain various errors. Real time replication to a second backup server would be great. None of this old zip file nonsense. I could then put my KC server in the cloud and keep a second instance running to cover me in case of outage or corruption. Kerio have consistently told me I should never use any other backup tool on the live mail store, otherwise I'd use Veeam or something similar.
|
|
|
|
|
|
| Re: Differential Backups Seem to Miss Emails [message #146540 is a reply to message #146501] |
Wed, 04 September 2019 19:35 |
tonci
Messages: 1
Registered: July 2012
|
|
|
|
Hello to all,
unfortunately I'm facing the same problems and can confirm that restore is not reliable any more. I'm hosting kerio on debian and at one of my client premises diff-backup procedure always generates two D* files in spite of that they are pretty little i.e. 128MB and 350MB together far away from 2G. I'm even not sure how to restore this two D* files ?
What would be correct syntax :
1. kmsrecover /bckpath/D*date*.*
2. kmsercover /bckpath/D*date*.zip
3. kmsrecover /bckpath/D*date*.zip /bckpath/D*date*.zip.1
None of the above will produce error , each will execute , but non of them will do correct restore and I can confirm that
I think this is very serious issue and at this moment I'm not able to migrate to another host because it takes cca 16h for full backup.
Some time ago I used to follow this steps:
1. make restore on new host
2. shut all services down on source host
3. make diff backup on source host
4. restore diff backup on another host
5. start engine on new host and everything was correct
Now I'm not able to achieve this and message-store difference is pretty obvious
BTW, Diff-files restore is not mentioned in the guide/documentation any more ?! ... Have I missed something?
Thank you very much for help and support
Best regards
Tonci
|
|
|
|
| Re: Differential Backups Seem to Miss Emails [message #146544 is a reply to message #146540] |
Thu, 05 September 2019 18:49 |
Bud Durland
Messages: 586
Registered: December 2013
Location: Plattsburgh, NY
|
|
|
|
Here's the general steps on how I did it last time, moving from one Debian machine to another
- Mount new host file system on old server
- make sure kerio service is NOT running on the new server
- rsync mail store old to new (no need to stop Kerio yet). This initial copy will take the longest amount of time.
- Stop kerio service on old server, but do not shut down the box
- Rsync again. This will run much quicker, as it is just the updates
- shutdown old server box
- config new server with proper IP address & etc
- Check Kerio config files on new server using Nano
- Start Kerio on new server
|
|
|
|
| Re: Differential Backups Seem to Miss Emails [message #146545 is a reply to message #146501] |
Thu, 05 September 2019 19:21 |
j.a.duke
Messages: 239
Registered: October 2006
|
|
|
|
jcooper wrote on Wed, 28 August 2019 13:51OK, it's been five years and I'm upgrading again, time to a newer MM with an SSD RAID 1+0. It appears this problem is still here. I have specific examples of emails which, when they should have been included in a Differential Backup, are simply skipped. Fortunately I was just running a test.
Has anyone come up with any tricks to absolutely guarantee the viability of KC Backups in Mac OS? Permissions fix or something? I can use rsync to copy the store again, but I'd sleep better at night if I knew my email backups actually BACKED UP my emails in case of catastrophe.
Thanks,
Jeff
Jeff,
I hope you saw my post in your other thread.
In case you missed it: https:// forums.gfi.com/index.php?t=msg&th=37721&goto=146500& amp; amp;#msg_146500
Works 100% of the time in my experience.
Also can be adapted for disaster recovery (DR) purposes.
Cheers,
Jon
[Updated on: Thu, 05 September 2019 19:21] Report message to a moderator |
|
|
|
|
|
Goto Forum:
Current Time: Mon May 29 17:09:36 CEST 2023
Total time taken to generate the page: 0.05519 seconds
|
Members
Search
Help
Register
Login
Home
