| Suddenly Failing Backups [message #143895] |
Mon, 17 September 2018 02:45  |
El-Man
Messages: 11
Registered: October 2011
Location: New Zealand
|
|
|
|
Hello, all.
A client's Kerio Connect server is suddenly failing to run its backup consistently. The Differentials work now and again, but there hasn't been a successful Full backup for a long time now. Here are the relevant bits from the logs:
Error Log:
[16/Sep/2018 16:10:04] backup.cpp: Cannot write file C:\Program Files\Kerio\MailServer\store/mail/<redacted>/#msgs/0000009e.eml (file size 13.40 MB) to backup ZIP file \\BoundaryNAS\Public\Keriobackup/F20180915T181003Z
[16/Sep/2018 16:10:06] backup.cpp: BACKUP: Backup failed in folder <redacted>, 3632865 MB (3632865 total) of 8501032 MB free on destination disk
Debug Log:
[16/Sep/2018 16:10:05][3448] Compression of message store backup finished. 231478 files were added to the archive \\BoundaryNAS\Public\Keriobackup/F20180915T181003Z in 46 volumes.
[16/Sep/2018 16:10:05][3448] {backup} Connection to folder \\BoundaryNAS\Public\Keriobackup is OK
[16/Sep/2018 16:10:06][3448] {backup} Estimate amount of folders were 3691 and amount of really visited folders is 2524.
As you can see, the backups happen to a NAS, which is connected on a gigabit network. You might think that this connection is unreliable, but Connect says it's okay, and there have been no other connection or disk errors. Nor is it running out of space, there's over 3.5Tb free. It just gets up to that stage saying compression is finished, then that the backup has failed, and deletes all the backup files it just wrote out.
Also, the folder it fails on is always different.
Any ideas?
|
|
|
|
| Re: Suddenly Failing Backups [message #143905 is a reply to message #143895] |
Mon, 17 September 2018 16:20  |
Maerad
Messages: 275
Registered: August 2013
|
|
|
|
Do you run an antivirus or intrusion detection program on the NAS or Kerio host? If yes, the kerio folder should be whitelisted, because if you open (copy) a file, the realtime scanner might block the mail OR needs more time to scan it and the copy command gets a time out.
Also what does the Windows Event Log and NAS Log say?
Try to copy a part of the store folder with the error mail to the NAS manually, also try to copy a big file and see, if there is a connection issue.
I would suspect something blocking the network transfer (locks the file) or the connection itself is bad (that why copy a big file over).
Another ID would be, that the NAS has some kind of timeout added for shares. Look tru the options.
[Updated on: Mon, 17 September 2018 16:21] Report message to a moderator |
|
|
|
|
|
| Re: Suddenly Failing Backups [message #143924 is a reply to message #143910] |
Wed, 19 September 2018 04:58 |
El-Man
Messages: 11
Registered: October 2011
Location: New Zealand
|
|
|
|
Okay. In order:
Yes, the mail server runs Eset for Kerio Connect, and there is a built-in antivirus on the NAS. But the store is definitely excluded on the server, and the destination is excluded on the NAS.
Nothing in the Windows Event Logs regarding connection or privilege at the time of the backups.
I copied 5x 2Gb files to the NAS, and it worked perfectly. However, when I copied one of the mail store folders (70k files, 45Gb) to the NAS, there was an alert box saying 'Error 0x8007003B: An unexpected network error occurred'. On an .eml file that was 120Kb. Pressed Try Again and it carried on fine.
Oh, lord, I hate these Windows code errors - there's hardly ever a definitive fix.
I need to narrow this down further, so I'll change the destination to somewhere else. That will at least tell us if the problem is with the NAS or the network. If it still doesn't work, I'll test a USB disk in the mail server.
|
|
|
|
| Re: Suddenly Failing Backups [message #143930 is a reply to message #143924] |
Wed, 19 September 2018 11:22 |
Maerad
Messages: 275
Registered: August 2013
|
|
|
|
Before you try too much, disable the antivir on your NAS and on the server (if you have eset for kerio, there shouldn't even be an option to disable a special folder...) and try coping the folder again with the load of small files.
Also I would check, if you can disable additional services on the NAS, do a firmware update and reboot it. It could be, that there are too many small files and the NAS gets overloaded.
But first - try with disabled antivirus. This is in most cases the main culprit. Also you could copy the files to a usual windows share - that could give you an idea, if the NAS has problems.
|
|
|
|
|
|
| Re: Suddenly Failing Backups [message #145923 is a reply to message #143962] |
Fri, 07 June 2019 09:29 |
dzjowk
Messages: 6
Registered: August 2008
|
|
|
|
Hi El-Man,
We have been experiencing the same issue for several weeks now. Rebooting, updating, ... on both ends did not solve the issue.
Are you still experiencing failing backups, or what did you do to fix ?
|
|
|
|
| Re: Suddenly Failing Backups [message #145943 is a reply to message #145923] |
Thu, 13 June 2019 10:44 |
zebby
Messages: 154
Registered: March 2009
|
|
|
|
We also saw this where it was struggling to save a 2.5GB file on a drive with 137GB free. In our case it was only when doing full backups.
We reported but no solution was found so in the end I just created VB script to delete the contents of the backup directory and scheduled it to run before the KC full backup ran.
Since we implemted this we've had no issues.
|
|
|
|
| Re: Suddenly Failing Backups [message #145950 is a reply to message #143962] |
Thu, 13 June 2019 16:02 |
Maerad
Messages: 275
Registered: August 2013
|
|
|
|
El-Man wrote on Sun, 23 September 2018 17:03Turned off the Antivirus on the NAS, but the full backup still failed. Will try updating its firmware next.
Another thing that came to mind - copy the folder again, this time disable windows defender and any kind of antivirus on the client system or server.
Also disable indexing for the network share, if it's enabled.
How is the NAS itself integrated into the domain? COnnected to the AD?
I run the backup on a windows server 2012 r2, backing up to a server 2012. I only gave rights to the machine itself to open the shares, no user is allowed (in case someone has a encrypt trojan). If you allow access, maybe public reading, there might be some tool or whatever also accessing the folder, thus creating a problem.
Also - what NAS do you have? How's the RAM/CPU load on it, if you copy the folder itself (many small files etc.)? Did you disable every service on the NAS aside from what you really need?
The windowserror you had tells us there's a problem with the network, share or nas. Not with the server or kerio. I guess it could be, that the NAS can't do all the files so fast, so the share goes on timeout for a bit. Also search the NAS options for anything like timeout, energy save or whatever, that could explain this behavior.
|
|
|
|
|
|
|
|
|
|
| Re: Suddenly Failing Backups [message #147957 is a reply to message #147952] |
Wed, 08 April 2020 10:45 |
AndreasL
Messages: 124
Registered: July 2008
Location: Germany
|
|
|
|
|
maybe, but the zip files will be created in local temp storage first and then moved to the destination. Your system partition who belongs to the temp storage seems to full or have not enough space left to create the zip files.
|
|
|
|
| Re: Suddenly Failing Backups [message #147963 is a reply to message #147957] |
Fri, 10 April 2020 09:13 |
Wilco
Messages: 107
Registered: July 2005
Location: The Netherlands
|
|
|
|
298Gb free on system disk. The last time there was no problem the backed up file was 2Gb big.
Could it also be that there is a corrupt line in the backup file that made it fail to backup?
Kerio Connect 9.4.2 on Windows Server 2022 (Dutch)
[Updated on: Fri, 10 April 2020 09:16] Report message to a moderator |
|
|
|
Members
Search
Help
Register
Login
Home
