Help with Search in Zope
Hi I am a newbie to Zope and have been trying to get ZCatalogs to work with indexing my postgres data to be avaiable for search within my site. I have no clues on how this is possible (if possible at all?). I am able to follow the examples on the site to have the search working on the DTML documents etc. But I want to be able to search my database content. Can any one guide me on this please ? Thanks bobby ==================== Bobby Mathew Dialognet Technologies Y220, 2nd Avenue Annanagar Chennai - 600 040 Tel : 91-44-6216851/52 Fax : 91-44-6216873 Email : bobby@dialognet.com WWW : www.dialognet.com
Hi Bobby, The ZCatalog doesn't generally index relational data. Its usually used to index things in the ZODB. It's possible that in the future it will be easier to index relational data with the ZCatalog, but for now it's not supported. - C ----- Original Message ----- From: "Bobby Mathew" <bobby@dialognet.com> To: <zope@zope.org> Sent: Tuesday, April 10, 2001 2:09 AM Subject: [Zope] Help with Search in Zope
Hi I am a newbie to Zope and have been trying to get ZCatalogs to work with indexing my postgres data to be avaiable for search within my site. I have no clues on how this is possible (if possible at all?). I am able to follow the examples on the site to have the search working on the DTML documents etc. But I want to be able to search my database content. Can any one guide me on this please ? Thanks
bobby
==================== Bobby Mathew Dialognet Technologies Y220, 2nd Avenue Annanagar Chennai - 600 040 Tel : 91-44-6216851/52 Fax : 91-44-6216873 Email : bobby@dialognet.com WWW : www.dialognet.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
Hello!
The ZCatalog doesn't generally index relational data. Its usually used to index things in the ZODB.
There are so many questions and problems and deficiencies about ZCatalog. Think for example I want to index a site on bobobase_modification_time and search for date ranges. Impossible! ZCatalog does not have ranges. Recently added textindex_operator allows to use "and", "near", "andnot", "or" operators, but it is insufficient. I can only use the operator for one index. What if I want to search for "title LIKE '%ZOPE% OR body LIKE '%python%'"? Alas! Developing special SQL-like language for ZCatalog? It is hard. What than? A crazy idea sprang into my mind. What if I, the site programmer, could provide a callback function, and ZCatalog will call the function for every index? Then I can code such simple or complex search as I need. Examples: <dtml-in "Catalog(indicies=['id', 'titile', 'time'], callback=mySearch)"> def mySearch(id, titile, time): if re.search("ZOPE", id): return 1 if re.search("python", title, re.I): return 1 if "2000/01/01" <= time <= "2000/12/31": return 1 return 0 Let the function return 1 for objects that correspond the serach criteria, 0 for failed objects, and raise StopSearch exception if it need to stop search immediatly. Oleg. ---- Oleg Broytmann http://www.zope.org/Members/phd/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
There are so many questions and problems and deficiencies about ZCatalog. Think for example I want to index a site on bobobase_modification_time and search for date ranges. Impossible! ZCatalog does not have ranges.
Yes it does. See the section titled "Searching Field Indexes" in the Zope Book.
Recently added textindex_operator allows to use "and", "near", "andnot", "or" operators, but it is insufficient. I can only use the operator for one index. What if I want to search for "title LIKE '%ZOPE% OR body LIKE '%python%'"? Alas! Developing special SQL-like language for ZCatalog? It is hard. What than?
You currently need to union and/or intersect the result sets manually. It *is* hard, yes. ;-)
A crazy idea sprang into my mind. What if I, the site programmer, could provide a callback function, and ZCatalog will call the function for every index? Then I can code such simple or complex search as I need. Examples:
<dtml-in "Catalog(indicies=['id', 'titile', 'time'], callback=mySearch)">
def mySearch(id, titile, time): if re.search("ZOPE", id): return 1 if re.search("python", title, re.I): return 1 if "2000/01/01" <= time <= "2000/12/31": return 1 return 0
Let the function return 1 for objects that correspond the serach criteria, 0 for failed objects, and raise StopSearch exception if it need to stop search immediatly.
THis is a pretty cool idea... except I think that it might be better to just use the callback to aggregate search results in interesting ways on a per-index basis. For instance, if you specify three indexes in a query and specify a callback, the callback can do unions or intersections on the result sets returned from each index manually...
On Tue, 10 Apr 2001, Chris McDonough wrote:
There are so many questions and problems and deficiencies about ZCatalog. Think for example I want to index a site on bobobase_modification_time and search for date ranges. Impossible! ZCatalog does not have ranges.
Yes it does. See the section titled "Searching Field Indexes" in the Zope Book.
Aha. Nice!
index. What if I want to search for "title LIKE '%ZOPE% OR body LIKE '%python%'"? Alas!
You currently need to union and/or intersect the result sets manually. It *is* hard, yes. ;-)
And I'll have duplicates :(
A crazy idea sprang into my mind. What if I, the site programmer, could provide a callback function, and ZCatalog will call the function for every index? Then I can code such simple or complex search as I need.
Let the function return 1 for objects that correspond the serach criteria, 0 for failed objects, and raise StopSearch exception if it need to stop search immediatly.
THis is a pretty cool idea... except I think that it might be better to just use the callback to aggregate search results in interesting ways on a per-index basis. For instance, if you specify three indexes in a query and specify a callback, the callback can do unions or intersections on the result sets returned from each index manually...
May be. These ideas cretainly deserve some thoughts. Oleg. ---- Oleg Broytmann http://www.zope.org/Members/phd/ phd@phd.pp.ru Programmers don't die, they just GOSUB without RETURN.
Hi Chris How can I add my data directly into the ZODB and thus make the search possible ? Can I add relational data into objects in ZODB and then make Zcatlogs or any other component search upon them ? If so, can anyone guide me on how to do this ? Are there any drawbacks or limits of doing this ? bobby On Tue, 10 Apr 2001, Chris McDonough wrote:
Hi Bobby,
The ZCatalog doesn't generally index relational data. Its usually used to index things in the ZODB. It's possible that in the future it will be easier to index relational data with the ZCatalog, but for now it's not supported.
- C
----- Original Message ----- From: "Bobby Mathew" <bobby@dialognet.com> To: <zope@zope.org> Sent: Tuesday, April 10, 2001 2:09 AM Subject: [Zope] Help with Search in Zope
Hi I am a newbie to Zope and have been trying to get ZCatalogs to work with indexing my postgres data to be avaiable for search within my site. I have no clues on how this is possible (if possible at all?). I am able to follow the examples on the site to have the search working on the DTML documents etc. But I want to be able to search my database content. Can any one guide me on this please ? Thanks
bobby
==================== Bobby Mathew Dialognet Technologies Y220, 2nd Avenue Annanagar Chennai - 600 040 Tel : 91-44-6216851/52 Fax : 91-44-6216873 Email : bobby@dialognet.com WWW : www.dialognet.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
You can turn your relational data into objects, if you want. This is a more advanced topic than I can cover in this email. Or you could take a hybrid approach like Ron Bickers described in his last email to this thread. ----- Original Message ----- From: "Bobby Mathew" <bobby@dialognet.com> To: "Chris McDonough" <chrism@digicool.com> Cc: <zope@zope.org> Sent: Tuesday, April 10, 2001 2:16 PM Subject: Re: [Zope] Help with Search in Zope
Hi Chris How can I add my data directly into the ZODB and thus make the search possible ? Can I add relational data into objects in ZODB and then make Zcatlogs or any other component search upon them ? If so, can anyone guide me on how to do this ? Are there any drawbacks or limits of doing this ?
bobby
On Tue, 10 Apr 2001, Chris McDonough wrote:
Hi Bobby,
The ZCatalog doesn't generally index relational data. Its usually used
to
index things in the ZODB. It's possible that in the future it will be easier to index relational data with the ZCatalog, but for now it's not supported.
- C
----- Original Message ----- From: "Bobby Mathew" <bobby@dialognet.com> To: <zope@zope.org> Sent: Tuesday, April 10, 2001 2:09 AM Subject: [Zope] Help with Search in Zope
Hi I am a newbie to Zope and have been trying to get ZCatalogs to work with indexing my postgres data to be avaiable for search within my site. I have no clues on how this is possible (if possible at all?). I am able to follow the examples on the site to have the search working on the DTML documents etc. But I want to be able to search my database content. Can any one guide me on this please ? Thanks
bobby
==================== Bobby Mathew Dialognet Technologies Y220, 2nd Avenue Annanagar Chennai - 600 040 Tel : 91-44-6216851/52 Fax : 91-44-6216873 Email : bobby@dialognet.com WWW : www.dialognet.com
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
I recently did this with MySQL to integrate the books for sale into the entire site search at socap.org. Here's what I did: 1) Create a ZSQL Method to pull out the fields you want to index. The field names must be the same as the indexes/meta data you want to store. For example, getBooksToCatalog contains the following: SELECT Number, Title, concat(Description, Title, Author) as PrincipiaSearchSource, 'Book' as meta_type, Number as id, Description as summary FROM Books (I use the meta_type 'Book' to show in the results that it's a Book, vs. just another page on the site) 2) Create a Python Script (a DTML Method would do, but this code, so it really belongs in a script) that calls catalog_object() for each record returned by the ZSQL Method above. The parameters are a) the record object, and b) the uid -- the path you want to be used as a URL for viewing the record. For example for book in container.getBooksToCatalog(): catalog.catalog_object(book, '/Publications/getBook/'+book.Number+'/bookdetails.html') "Test" the script and that will get the data into the catalog. When a search matches, getURL() will return something like http://www.socap.org/Publications/getBook/00123/bookdetails.html. This is the URL that can be used to view the book. getBook is a ZSQL Method that has the advanced "Allow simple direct traversal" option set, and a single argument of Number (which is a string). The method looks like this: SELECT * FROM Books WHERE <dtml-sqltest number column="Number" type=nb> bookdetails.html is a DTML Method that displays the record returned by getBook. It's really quite simple once you get the gist. Using the above principles, you can get almost anything into the catalog. I also have some articles that are listed and summarized in MySQL and have associated PDF files in the file system. I use a similar method, but I also have a class defined in Extensions that is used by the ZSQL Method. The class defines PrincipiaSearchSource as the text of its PDF file (actually, of the converted text file). This gets me the source of the article in the catalog, and result URLs that point to the article summary page where it can be ordered or viewed as a PDF online. One important down side to note is that if you do an 'Update Catalog' in the ZCatalog Advanced tab, you'll mess up the catalog because it will try to suck data based on the uid, not from the database as you intend. Because of this, it's best to maintain a script (or several) that can be run to update the data. I hope this helps. Anyone interested in a more complete How-To? _______________________ Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
-----Original Message----- From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Bobby Mathew Sent: Tuesday, April 10, 2001 2:09 AM To: zope@zope.org Subject: [Zope] Help with Search in Zope
Hi I am a newbie to Zope and have been trying to get ZCatalogs to work with indexing my postgres data to be avaiable for search within my site. I have no clues on how this is possible (if possible at all?). I am able to follow the examples on the site to have the search working on the DTML documents etc. But I want to be able to search my database content. Can any one guide me on this please ? Thanks
Here's the first version of the How-To, but I will add more examples and details soon. Let me know if you have any questions or comments. http://www.zope.org/Members/rbickers/cataloganything _______________________ Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
-----Original Message----- From: Steve Drees [mailto:drees@the-bridge.net] Sent: Tuesday, April 10, 2001 10:38 AM To: Ron Bickers; Bobby Mathew; zope@zope.org Subject: RE: [Zope] Help with Search in Zope (SQL data)
I hope this helps. Anyone interested in a more complete How-To?
Definitely. This is the kind of Zen that needs to be shared.
This is an excellent howto! ----- Original Message ----- From: "Ron Bickers" <rbickers@logicetc.com> To: "Norman Khine" <khine@btinternet.com>; "Steve Drees" <drees@the-bridge.net>; "Bobby Mathew" <bobby@dialognet.com>; <zope@zope.org> Sent: Tuesday, April 10, 2001 5:27 PM Subject: [Zope] How-To: Cataloging SQL Data and Almost Anything Else
Here's the first version of the How-To, but I will add more examples and details soon. Let me know if you have any questions or comments.
http://www.zope.org/Members/rbickers/cataloganything
_______________________
Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
-----Original Message----- From: Steve Drees [mailto:drees@the-bridge.net] Sent: Tuesday, April 10, 2001 10:38 AM To: Ron Bickers; Bobby Mathew; zope@zope.org Subject: RE: [Zope] Help with Search in Zope (SQL data)
I hope this helps. Anyone interested in a more complete How-To?
Definitely. This is the kind of Zen that needs to be shared.
_______________________________________________ Zope maillist - Zope@zope.org http://lists.zope.org/mailman/listinfo/zope ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope-dev )
I have updated (and completed) what I hope is a sufficiently detailed How-To on cataloging SQL data and just about anything else. Your comments and questions are welcome. http://www.zope.org/Members/rbickers/cataloganything Enjoy! _______________________ Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
-----Original Message----- From: Steve Drees [mailto:drees@the-bridge.net] Sent: Tuesday, April 10, 2001 10:38 AM To: Ron Bickers; Bobby Mathew; zope@zope.org Subject: RE: [Zope] Help with Search in Zope (SQL data)
I hope this helps. Anyone interested in a more complete How-To?
Definitely. This is the kind of Zen that needs to be shared.
Thanks Ron, Thats just fantastic to have whipped up a howto in such a quick time. I have neverhad such fast support from any kind of support group till now. You certainly boost my confidence in using Zope. Fo a newbie like me this is a great enouragement. Hats off to you. bobby ==================== Bobby Mathew Dialognet Technologies Y220, 2nd Avenue Annanagar Chennai - 600 040 Tel : 91-44-6216851/52 Fax : 91-44-6216873 Email : bobby@dialognet.com WWW : www.dialognet.com On Wed, 11 Apr 2001, Ron Bickers wrote:
I have updated (and completed) what I hope is a sufficiently detailed How-To on cataloging SQL data and just about anything else. Your comments and questions are welcome.
http://www.zope.org/Members/rbickers/cataloganything
Enjoy! _______________________
Ron Bickers Logic Etc, Inc. rbickers@logicetc.com
-----Original Message----- From: Steve Drees [mailto:drees@the-bridge.net] Sent: Tuesday, April 10, 2001 10:38 AM To: Ron Bickers; Bobby Mathew; zope@zope.org Subject: RE: [Zope] Help with Search in Zope (SQL data)
I hope this helps. Anyone interested in a more complete How-To?
Definitely. This is the kind of Zen that needs to be shared.
participants (5)
-
Bobby Mathew -
Chris McDonough -
Oleg Broytmann -
Ron Bickers -
Steve Drees