John Eikenberry wrote:
Long answer: If you check out the source and/or hit it with the profiler you'll see that the way the partial search works is to first do a more general search then to limit the hits as much as possible via regex's. Both these steps have to happen no matter the batch size, and this is where you take the performance hit.
Can you tell us a bit about how many indexes (and what types) you were maintaining about each object? Another poster reported no problems with 60,000 objects.
So you're saying that you are doing all searching using SQL statements, and not just object retreival and storage, correct? How are you handling full text searches?
Yes. I'll use MySQL's built in pattern matching facilities. It can do full text searches with partial matching, and it can do this fast. I'm working on a system that will return the DataSkin's in responce to the query. Allowing me to deal with just the objects yet use all of MySQL's facilities.
Sounds interesting.
I've just started to work on this as part of a larger project, but I'm doing it full time and should have something fairly soon. My company is very free software friendly, so I'll be able to share it once its ready. If you happen to be interested.
Sure, keep me posted. Thanks, Michael Bernstein.