How to speed up re-indexing
|
Logged in as: Guest
|
|
Users viewing this topic:
none
|
|
Login  | |
|
How to speed up re-indexing - 11.Jun.2007 6:32:42 PM
|
|
|
srakkhit
Posts: 77
Joined: 7.Feb.2006
Status: offline
|
Hi All
I have upgraded MARC 4 to the latest version. Everything is working fine, however the re-indexing taking a long time. We have approximately 4 million emails in 4 differenting databases ( two are MARC v3 databases). So far it has re-indexed 1.4 million email in 3.5 days. We had a long week-end, so there were no activity on that server, plus I have disabled the backup too.
Server is dedicated for GFI including SQL 2005 on the same server with 6 GB memory and seperate disk for databases and indexes.
I know that MARC 4 indexes 1000 email at a time and at a pre-determined interval and cycle through one database at a time.
IS THERE SOME SORT OF CONFIGURATION THAT I CAN TWEAK TO MAKE INDEXING FASTER.
Regards
Sukumar
|
|
|
|
|
RE: How to speed up re-indexing - 12.Jun.2007 12:34:15 PM
|
|
|
wood36
Posts: 3
Joined: 8.Mar.2007
Status: offline
|
We have run into a similar problem. It seems that the re-build periodically stalls. I have a call open now, but the troubleshooter logs I have sent, don't seem to point to any particular issue. This started happening after we installed build 20070424.
We have about 800,000 e-mails in the DB we are rebuilding, and it takes about a week to index. We also ran into a problem when the rebuild finishes, or gets close to finishing the process starts all over.
|
|
|
|
|
RE: How to speed up re-indexing - 13.Jun.2007 5:50:56 PM
|
|
|
keatscon
Posts: 22
Joined: 13.Dec.2006
Status: offline
|
We updated to build 20070424 today and have also noticed that the rebuilding of the indexes is taking an excruciatingly long time. Our db's are small and so should not take a great deal of time to rebuild.
|
|
|
|
|
RE: How to speed up re-indexing - 14.Jun.2007 3:11:27 AM
|
|
|
Rodion
Posts: 834
Joined: 15.Nov.2004
Status: offline
|
Hi
Can you please enable debug logging, reproduce the issue and then send me the files from the MailArchiver\Search\DebugLogs folder to rodion@gfi.com
_____________________________
Best regards
Rodion Denisyuk
GFI Software Ltd
|
|
|
|
|
RE: How to speed up re-indexing - 14.Jun.2007 6:10:01 AM
|
|
|
srakkhit
Posts: 77
Joined: 7.Feb.2006
Status: offline
|
Hi Rodion
Thanks for the offer. Two of my databases completed the re-indexing in 6 day. At current rate other two will complete this week-end. Therefore I do not want to disturb anything at this stage.
What I am really interested is some sort of config file which will dedicate 100% of the resources to re-indexing. One of the reason I chose to upgrade in the long weekend so the indexing will complete by the time users come back on tuesday. But at current rate it will take 12 days to reindex 3.6 million emails. Can't imagine what will happen at next upgrade when our archived email might be 5 million.
I hopr you understand my situation. I don't thing there is a big problem with my server. Just the whole re-indexing itself is slow or the it was given low piority.
Regards
Sukumar
|
|
|
|
|
RE: How to speed up re-indexing - 3.Jul.2007 10:44:54 PM
|
|
|
engelm
Posts: 18
Joined: 29.Dec.2005
Status: offline
|
Indexing in version 4 is worthless. I just upgraded from version 3 where everything was working fine. My indexes take a long time if they build at all. It doesn't matter whether it is a new v4 db or an exisitng v3 db. My server is dedicated to MailArchiver, SQL running locally, dual ZEON processors and 3GB of RAM. My db's are broken down quarterly and are around 20GB each. I have been in contact with support to no avail. Just look at the forum posts in regards to indexing it's a big problem. What I can't understand is why they would create there own indexing system in place of the SQL offerings like they used in v3. Searches may have been slow in v3 but at least they worked. If you can't build an index in v4 you can't search. GFI needs to step up and provide answers to customers.
|
|
|
|
|
RE: How to speed up re-indexing - 4.Jul.2007 8:12:27 AM
|
|
|
Mark Busuttil
Posts: 4836
Joined: 16.Oct.2005
Status: offline
|
This issue is currently being handled by e-mail support. The reference number(s) used are:
CAS-19979 - engelm
NOTE: We have sent you an email on the address that you have registered over the forums with.
Should you require any updates or further information, kindly contact us using the support form at the following link:
http://support.gfi.com/supportrequestform.asp
_____________________________
Regards,
Mark Busuttil
GFI Software Ltd - www.gfi.com
Messaging, Content Security & Network Security Software
GFI: MailEssentials - MailSecurity - MailArchiver - FAXmaker - LANguard – WebMonitor
|
|
|
|
|
RE: How to speed up re-indexing - 21.Jul.2007 11:07:25 AM
|
|
|
robbied31
Posts: 45
Joined: 27.Apr.2007
Status: offline
|
Why can't you guys show results or resolution to a problem in the thread in the forum? I am having the same problem, and if you guys move to email, what good is this thread? This forum is littered with unresolved threads!..... "This issue is currently being handled by e-mail support"
Why so many problems with rebuilding index in new version?
Rob
|
|
|
|
|
RE: How to speed up re-indexing - 21.Jul.2007 11:47:44 AM
|
|
|
engelm
Posts: 18
Joined: 29.Dec.2005
Status: offline
|
Here's what I have learned on my own (GFI support has been of no help on this issue). Index files must be placed on a seperate drive (not partition) from your MailArchiver installation and SQL transaction logs and databases files. This eliminates all disk contention. Next if you're running SQL server and have maintenance jobs that have optimizations enabled that reorganize data pages and indexes turn them off. I found that as soon as the maintenance jobs ran the MailArchiver indexng stopped and could not be restarted without blowing away the GFI indexes and starting them over again. I'm not a SQL expert so I can't explain why SQL optimizations corrupt the GFI indexes but it does.
My indexes continue to build but very slowly. I have been watching the performance of the server very closely. The marc.search.exe page faults like crazy. I have reported this to GFI support several times over the last few weeks and sent them many troubleshooting files. The only anwser I get is upgrade to the latest build which I am running and finally "am i happy with the support" and can the case be closed.
The bottom line is that they really meesed up the indexng with version 4 and probably don't have a clue how to fix it. I've aksed why they created their own indexing and I've heard excuses from GFI like "SQL indexing doesn't allow searching of attachments so we created an index system that would." That's a poor excuse. SharePoint running on SQL server can do it. If you follow the steps above you should be able to get the initial indexes built (albeit it may take several weeks depending on the size of your stores) once built updating should go quickly. Just pray you never have to reindex, starting them all over from scratch.
|
|
|
|
|
RE: How to speed up re-indexing - 21.Jul.2007 12:09:49 PM
|
|
|
robbied31
Posts: 45
Joined: 27.Apr.2007
Status: offline
|
Thanks for some input!
My searches all last week have been spotty. I have just now pointed all indexes to a new location and started re-indexing on the smallest database we have [1.2 GB]. I see nothing happening though whatsoever and am now thinking I may be waiting for quite sometime. A simple progress indicator of some kind to let me know something is happening would be great, but no, I have nothing.
I can not stand to search thru a forum littered with incomplete threads. You actually think you are getting somewhere and then you run into "This issue is currently being handled by e-mail support". Come on man, support the community!
Rob
|
|
|
|
|
RE: How to speed up re-indexing - 21.Jul.2007 12:15:41 PM
|
|
|
engelm
Posts: 18
Joined: 29.Dec.2005
Status: offline
|
If your indexing schedule is set to index new emails every 30 minutes (the default) it will start indexing then.
|
|
|
|
|
RE: How to speed up re-indexing - 23.Jul.2007 3:55:50 AM
|
|
|
Rodion
Posts: 834
Joined: 15.Nov.2004
Status: offline
|
Hello guys
We do understand your concern about the indexing and are researching every issue submitted to our support. 90% of the cases were closed by installing the latest build and rebuilding the index or changing the location of the index. As you have pointed out already the performance of the indexing and search heavily depends on the performance of the IO subsystem on the MailArchiver machine. Also the immediate indexing is a requirement for heavily loaded environments.
_____________________________
Best regards
Rodion Denisyuk
GFI Software Ltd
|
|
|
|
|
RE: How to speed up re-indexing - 25.Jul.2007 7:24:56 PM
|
|
|
robbied31
Posts: 45
Joined: 27.Apr.2007
Status: offline
|
Rodion,
I hadn't run into this problem until upgrading to latest build. I did however upgrade after moving new server. I haven't had any luck getting rebuild to complete, or show any results whatsoever. I have changed location of index, unattached and reattached stores, and have now just waited for 4 or 5 days and still no results. The search service seems to become unresponsive after a couple hours and I have to reboot the server just to get into ma.
What was the problems with the other 10% of the cases?
Rob
|
|
|
|
|
RE: How to speed up re-indexing - 26.Jul.2007 2:54:29 AM
|
|
|
Rodion
Posts: 834
Joined: 15.Nov.2004
Status: offline
|
Hi Rob
The other 10% are corrupted configuration files and certain attachments that just fail to index.
Please enable debug logging, wait for at least 2 hours, then run the troubleshooter and upload the resulting files to our ftp as described here: http://forums.gfi.com/General_Information/m_900727063/tm.htm
Then send me the name of the file to rodion@gfi.com
_____________________________
Best regards
Rodion Denisyuk
GFI Software Ltd
|
|
|
|
|
RE: How to speed up re-indexing - 26.Jul.2007 4:36:32 PM
|
|
|
robbied31
Posts: 45
Joined: 27.Apr.2007
Status: offline
|
Hi Rodion,
Thanks for the reply. I came back here too late. The reason I had moved gfi to a new server is because we upgraded our main sql server. We moved sql and databases for accounting, construction and service to new hardware. I had temporarily moved gfi to another box to upgrade old sql server. I completed that yesterday, reinstalled server 2003, sql server 2000 sp4 and and moving all gfi databases [backup and restore]. Just got done with a clean install of GFI, reattaching databases, attaching gfi stores and everything else. As of now, I am not actively archiving email, and one of the four databases has already completed rebuilding index. I am hoping to see success on all the others.
Sorry for the long story, but I don't have the other box archiving right now so I can't accomplish your request.....
Thank you though!!!
Rob
|
|
|
|
 New Messages |
 No New Messages |
 Hot Topic w/ New Messages |
 Hot Topic w/o New Messages |
 Locked w/ New Messages |
 Locked w/o New Messages |
|
 Post New Thread
Reply to Message
Post New Poll
Submit Vote
Delete My Own Post
Delete My Own Thread
Rate Posts |
|
|
Printable Version
