How to speed up re-indexing

How to speed up re-indexing (Full Version)

All Forums >> [Archiving & Fax] >> GFI MailArchiver

Message

srakkhit -> How to speed up re-indexing (11.Jun.2007 6:32:42 PM)
Hi All I have upgraded MARC 4 to the latest version. Everything is working fine, however the re-indexing taking a long time. We have approximately 4 million emails in 4 differenting databases ( two are MARC v3 databases). So far it has re-indexed 1.4 million email in 3.5 days. We had a long week-end, so there were no activity on that server, plus I have disabled the backup too. Server is dedicated for GFI including SQL 2005 on the same server with 6 GB memory and seperate disk for databases and indexes. I know that MARC 4 indexes 1000 email at a time and at a pre-determined interval and cycle through one database at a time. IS THERE SOME SORT OF CONFIGURATION THAT I CAN TWEAK TO MAKE INDEXING FASTER. Regards Sukumar

wood36 -> RE: How to speed up re-indexing (12.Jun.2007 12:34:15 PM)
We have run into a similar problem. It seems that the re-build periodically stalls. I have a call open now, but the troubleshooter logs I have sent, don't seem to point to any particular issue. This started happening after we installed build 20070424. We have about 800,000 e-mails in the DB we are rebuilding, and it takes about a week to index. We also ran into a problem when the rebuild finishes, or gets close to finishing the process starts all over.

keatscon -> RE: How to speed up re-indexing (13.Jun.2007 5:50:56 PM)
We updated to build 20070424 today and have also noticed that the rebuilding of the indexes is taking an excruciatingly long time. Our db's are small and so should not take a great deal of time to rebuild.

Rodion -> RE: How to speed up re-indexing (14.Jun.2007 3:11:27 AM)
Hi Can you please enable debug logging, reproduce the issue and then send me the files from the MailArchiver\Search\DebugLogs folder to rodion@gfi.com

srakkhit -> RE: How to speed up re-indexing (14.Jun.2007 6:10:01 AM)
Hi Rodion Thanks for the offer. Two of my databases completed the re-indexing in 6 day. At current rate other two will complete this week-end. Therefore I do not want to disturb anything at this stage. What I am really interested is some sort of config file which will dedicate 100% of the resources to re-indexing. One of the reason I chose to upgrade in the long weekend so the indexing will complete by the time users come back on tuesday. But at current rate it will take 12 days to reindex 3.6 million emails. Can't imagine what will happen at next upgrade when our archived email might be 5 million. I hopr you understand my situation. I don't thing there is a big problem with my server. Just the whole re-indexing itself is slow or the it was given low piority. Regards Sukumar

engelm -> RE: How to speed up re-indexing (3.Jul.2007 10:44:54 PM)
Indexing in version 4 is worthless. I just upgraded from version 3 where everything was working fine. My indexes take a long time if they build at all. It doesn't matter whether it is a new v4 db or an exisitng v3 db. My server is dedicated to MailArchiver, SQL running locally, dual ZEON processors and 3GB of RAM. My db's are broken down quarterly and are around 20GB each. I have been in contact with support to no avail. Just look at the forum posts in regards to indexing it's a big problem. What I can't understand is why they would create there own indexing system in place of the SQL offerings like they used in v3. Searches may have been slow in v3 but at least they worked. If you can't build an index in v4 you can't search. GFI needs to step up and provide answers to customers.

Mark Busuttil -> RE: How to speed up re-indexing (4.Jul.2007 8:12:27 AM)
This issue is currently being handled by e-mail support. The reference number(s) used are: CAS-19979 - engelm NOTE: We have sent you an email on the address that you have registered over the forums with. Should you require any updates or further information, kindly contact us using the support form at the following link: http://support.gfi.com/supportrequestform.asp

robbied31 -> RE: How to speed up re-indexing (21.Jul.2007 11:07:25 AM)
Why can't you guys show results or resolution to a problem in the thread in the forum? I am having the same problem, and if you guys move to email, what good is this thread? This forum is littered with unresolved threads!..... "This issue is currently being handled by e-mail support" Why so many problems with rebuilding index in new version? Rob

engelm -> RE: How to speed up re-indexing (21.Jul.2007 11:47:44 AM)
Here's what I have learned on my own (GFI support has been of no help on this issue). Index files must be placed on a seperate drive (not partition) from your MailArchiver installation and SQL transaction logs and databases files. This eliminates all disk contention. Next if you're running SQL server and have maintenance jobs that have optimizations enabled that reorganize data pages and indexes turn them off. I found that as soon as the maintenance jobs ran the MailArchiver indexng stopped and could not be restarted without blowing away the GFI indexes and starting them over again. I'm not a SQL expert so I can't explain why SQL optimizations corrupt the GFI indexes but it does. My indexes continue to build but very slowly. I have been watching the performance of the server very closely. The marc.search.exe page faults like crazy. I have reported this to GFI support several times over the last few weeks and sent them many troubleshooting files. The only anwser I get is upgrade to the latest build which I am running and finally "am i happy with the support" and can the case be closed. The bottom line is that they really meesed up the indexng with version 4 and probably don't have a clue how to fix it. I've aksed why they created their own indexing and I've heard excuses from GFI like "SQL indexing doesn't allow searching of attachments so we created an index system that would." That's a poor excuse. SharePoint running on SQL server can do it. If you follow the steps above you should be able to get the initial indexes built (albeit it may take several weeks depending on the size of your stores) once built updating should go quickly. Just pray you never have to reindex, starting them all over from scratch.

robbied31 -> RE: How to speed up re-indexing (21.Jul.2007 12:09:49 PM)
Thanks for some input! My searches all last week have been spotty. I have just now pointed all indexes to a new location and started re-indexing on the smallest database we have [1.2 GB]. I see nothing happening though whatsoever and am now thinking I may be waiting for quite sometime. A simple progress indicator of some kind to let me know something is happening would be great, but no, I have nothing. I can not stand to search thru a forum littered with incomplete threads. You actually think you are getting somewhere and then you run into "This issue is currently being handled by e-mail support". Come on man, support the community! Rob

engelm -> RE: How to speed up re-indexing (21.Jul.2007 12:15:41 PM)
If your indexing schedule is set to index new emails every 30 minutes (the default) it will start indexing then.

Rodion -> RE: How to speed up re-indexing (23.Jul.2007 3:55:50 AM)
Hello guys We do understand your concern about the indexing and are researching every issue submitted to our support. 90% of the cases were closed by installing the latest build and rebuilding the index or changing the location of the index. As you have pointed out already the performance of the indexing and search heavily depends on the performance of the IO subsystem on the MailArchiver machine. Also the immediate indexing is a requirement for heavily loaded environments.

robbied31 -> RE: How to speed up re-indexing (25.Jul.2007 7:24:56 PM)
Rodion, I hadn't run into this problem until upgrading to latest build. I did however upgrade after moving new server. I haven't had any luck getting rebuild to complete, or show any results whatsoever. I have changed location of index, unattached and reattached stores, and have now just waited for 4 or 5 days and still no results. The search service seems to become unresponsive after a couple hours and I have to reboot the server just to get into ma. What was the problems with the other 10% of the cases? Rob

Rodion -> RE: How to speed up re-indexing (26.Jul.2007 2:54:29 AM)
Hi Rob The other 10% are corrupted configuration files and certain attachments that just fail to index. Please enable debug logging, wait for at least 2 hours, then run the troubleshooter and upload the resulting files to our ftp as described here: http://forums.gfi.com/General_Information/m_900727063/tm.htm Then send me the name of the file to rodion@gfi.com

robbied31 -> RE: How to speed up re-indexing (26.Jul.2007 4:36:32 PM)
Hi Rodion, Thanks for the reply. I came back here too late. The reason I had moved gfi to a new server is because we upgraded our main sql server. We moved sql and databases for accounting, construction and service to new hardware. I had temporarily moved gfi to another box to upgrade old sql server. I completed that yesterday, reinstalled server 2003, sql server 2000 sp4 and and moving all gfi databases [backup and restore]. Just got done with a clean install of GFI, reattaching databases, attaching gfi stores and everything else. As of now, I am not actively archiving email, and one of the four databases has already completed rebuilding index. I am hoping to see success on all the others. Sorry for the long story, but I don't have the other box archiving right now so I can't accomplish your request..... Thank you though!!! Rob

Page: [1] 2 next > >>