Indexing taking days, huge index files, etc
|
Logged in as: Guest
|
|
Users viewing this topic:
none
|
|
Login  | |
|
Indexing taking days, huge index files, etc - 22.Oct.2008 11:26:48 AM
|
|
|
charlesw
Posts: 9
Joined: 31.Jan.2008
Status: offline
|
Currently we have our archive stores split by calendar year, going back to 2000. We use a SQL backend, and the databases for the most recent years exceed 50GB each. This is huge, and indexing the contents takes forever.
We recently had to reindex because searches were returning incorrect results. The indexer has been running for over 24 hours now, and the status for our two largest stores is as follows:
- MailArchive-2007
- Store contains: 410447 emails
- DB size: 51372.25 MB
- Index size: 8681.5 MB
- Indexed emails: 172000
- MailArchive-2008
- Store contains: 435362 emails
- DB size: 47189.38 MB
- Index size: 6374.27 MB
- Indexed emails: 257999
Also, the "indexed emails" statistic hasn't changed since I got in today, about 3 hours ago.
I'm not exactly sure of what is happening, or why this is taking so long. The only potential solution I have arrived at is splitting the archives down into 3-month chunks - but I'm not sure than an archive can be dynamically split in such a way.
My monitoring system doesn't indicate any bottlenecks on the network, SQL or MailArchiver servers.
Are there any suggestions for speeding up indexing that can be accomplished without re-importing all my data?
|
|
|
|
RE: Indexing taking days, huge index files, etc - 22.Oct.2008 11:33:51 AM
|
|
|
junglism
Posts: 8
Joined: 24.Aug.2007
Status: offline
|
I've got one set of 6-month data at about 50GB
I had to rebuild the indexes on it once and it took about 60hours!
Interested to see if they can be split up. I looked into this about a year ago and couldnt really find any info
|
|
|
|
|
RE: Indexing taking days, huge index files, etc - 23.Oct.2008 3:32:47 AM
|
|
|
roderickb
Posts: 26
Joined: 30.May2008
Status: offline
|
Hi charlesw,
Which version of MailArchiver are you using ?
_____________________________
Thanks and kind regards,
Roderick Buhagiar
GFI Software Ltd
|
|
|
|
|
RE: Indexing taking days, huge index files, etc - 23.Oct.2008 5:30:29 AM
|
|
|
clcgroup
Posts: 4
Joined: 10.Oct.2008
Status: offline
|
We use Version 6 and our database contained about 640,000 e-mails. I found that archiving this number of e-mails took about 3-4 days.
I also discovered a major issue as well whilst doing it. The pause in indexing was, in our case caused by the maintenance schedule which runs at night on the indexes to merge them in.
Effectively what would happen is after about 2 days the indexing would simply stop.
We got round the problem by performing the following action (as advised by GFI support)
1.- Make a backup copy of the file ..\GFI\MailArchiver\Search\Data\Management.xml
2.- Edit the file and modify the field <NextMerge> to be done in a few days instead instead today at 23:00 (or whenever it is scheduled). Basically allow enough time for the indexing to finish (mine was 3.5days).
3.- Restart the MailArchiver services
What this did for me is allow the re-indexing to complete successfully. When the maintenance next runs it merges in all of the indexing subfolders and makes the index smaller.
Our archiving server is fairly powerful ( Dual Quad Core XEON, 24GB RAM, 15K RPM SAS HDDs) so you may find indexing takes a lot longer.
Hope this helps
|
|
|
|
|
RE: Indexing taking days, huge index files, etc - 31.Oct.2008 8:26:25 AM
|
|
|
charlesw
Posts: 9
Joined: 31.Jan.2008
Status: offline
|
Not sure why it didn't tell me people had been replying!
GFI has been helping via e-mail, but hasn't (yet) suggested stopping the scheduled maintenance. I'm going to give that a shot and see what happens.
As of right now, it seems like the indexer runs at about 100 messages/minute - is this more or less in line with average performance?
|
|
|
|
|
RE: Indexing taking days, huge index files, etc - 3.Nov.2008 9:32:27 PM
|
|
|
John Letourneau
Posts: 1264
Joined: 28.Apr.2008
From: Clayton, NC
Status: online
|
charlesw,
This does seem a bit slow but I could not guess if it was normal without knowing everything running on this server along with hardware specs. If you are looking to speed up the overall indexing process one thing that could be disabled would be the attachment indexing. This would index the current emails but not index anything that was attached. You can read more about this at http://kbase.gfi.com/showarticle.asp?id=KBID003164.
_____________________________
Regards,
John Letourneau - Senior Technical Support Representative
GFI Software - www.gfi.com
|
|
|
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
 Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts |
|
|
Printable Version
