AW: AW: [Zope] Source from form result.
Loren Stafford
lstafford@morphics.com
Thu, 3 May 2001 17:02:40 -0700
Did you get enough new material to write a HowTo? "How To Screen Scrape"
-- Loren
> -----Original Message-----
> From: zope-admin@zope.org [mailto:zope-admin@zope.org]On Behalf Of Marc
> Fischer
> Sent: Thursday, May 03, 2001 16:31
> To: Tino Wildenhain; Marc Fischer; Danny William Adair; Paula Mangas
> Cc: zope@zope.org
> Subject: AW: AW: AW: [Zope] Source from form result.
>
>
> Hey, thanks to all of you, that tried to help me.
>
> Finally I had success with the httplib funktion. ... :-)))
>
> Marc
>
>
>
> > -----Ursprungliche Nachricht-----
> > Von: zope-admin@zope.org [mailto:zope-admin@zope.org]Im Auftrag von Tino
> > Wildenhain
> > Gesendet: Donnerstag, 3. Mai 2001 22:51
> > An: Marc Fischer; Danny William Adair; Paula Mangas
> > Cc: zope@zope.org
> > Betreff: Re: AW: AW: [Zope] Source from form result.
> >
> >
> > Hi Marc,
> >
> > its simply:
> >
> > <dtml-with "web_client('http://server/',x='a',y='b')">
> > <dtml-var body>
> > </dtml-with>
> >
> > in your case :)
> > x and y go into this **kw argument. If this is set,
> > webclient uses POST.
> >
> > HTH
> > Tino Wildenhain.
> >
> > --On Mittwoch, 2. Mai 2001 20:45 +0200 Marc Fischer
> > <marcbpc@gmx.de> wrote:
> >
> > > Hi Tino,
> > >
> > > I think, thats it, what I am looking for. Your Example works
> > fine so far.
> > > But my understanding in python I so low, that I am not able
> change it in
> > > my needed way. Perhabs you could tell me how to realize this:
> > >
> > > <form action="http://server" method=POST>
> > > <input type=hidden name="x" value="a">
> > > <input type=hidden name="y" value="b">
> > > <input type=submit>
> > > </form>
> > >
> > > and show me how to get the source of the result page?! :-)
> > >
> > > I would be very happy, ... thanx a lot ...
> > >
> > > Marc
> > >
> > >
> > >
> > >
> > >> -----Ursprungliche Nachricht-----
> > >> Von: zope-admin@zope.org [mailto:zope-admin@zope.org]Im
> > Auftrag von Tino
> > >> Wildenhain
> > >> Gesendet: Mittwoch, 2. Mai 2001 15:37
> > >> An: Marc Fischer; Danny William Adair; Paula Mangas
> > >> Cc: zope@zope.org
> > >> Betreff: Re: AW: [Zope] Source from form result.
> > >>
> > >>
> > >> Hi Marc,
> > >>
> > >> zope has a "client" deep in its core :)
> > >> See lib/python/ZPublisher/Client.py
> > >> for more info.
> > >>
> > >> I currently use a external method like this:
> > >>
> > >> > from ZPublisher import Client
> > >>
> > >> def web_client(url = '', username = None, password = None, **kw):
> > >> '''access http servers'''
> > >> class gen_res:
> > >> __allow_access_to_unprotected_subobjects__=1
> > >> f=gen_res()
> > >> if kw:
> > >> f.headers,f.body=Client.call(url,username,password,kw)
> > >> else:
> > >> f.headers,f.body=Client.call(url,username,password)
> > >> return(f)
> > >>
> > >>
> > >> And use it like this:
> > >>
> > >> <dtml-with "web_client('http://server/uri')">
> > >> <dtml-var body>
> > >> </dtml-with>
> > >>
> > >> In Client.py are also methods and functions to map arguments
> > >> to a POST request as you wish.
> > >>
> > >> I think, the above could be a starting point for you.
> > >>
> > >> HTH
> > >> Tino Wildenhain
> > >>
> > >> --On Mittwoch, 2. Mai 2001 12:12 +0200 Marc Fischer
> > >> <marcbpc@gmx.de> wrote:
> > >>
> > >> > Hi Danny,
> > >> >
> > >> > first of all thank you for your detailed help!!! That is
> > >> nearly, what I am
> > >> > looking for, but one problem exists:
> > >> > The page I want to extract (you are right ... its on an external
> > >> > server) does not work with passing the variables in the url.
> > I mean ...
> > >> >
> > >> > http://server?x=a&y=b does not work.
> > >> >
> > >> > I need to do it with a form like:
> > >> >
> > >> > <form action="http://server" method=POST>
> > >> > <input type=hidden name="x" value="a">
> > >> > <input type=hidden name="y" value="b">
> > >> > <input type=submit>
> > >> > </form>
> > >> >
> > >> > That's my big big problem. Btw, ... its legal :-)))
> > >> >
> > >> > Marc
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >
> > >> >> -----Ursprungliche Nachricht-----
> > >> >> Von: Danny William Adair [mailto:Danny@Adair.net]
> > >> >> Gesendet: Mittwoch, 2. Mai 2001 01:26
> > >> >> An: Marc Fischer; Paula Mangas
> > >> >> Cc: zope@zope.org
> > >> >> Betreff: Re: [Zope] Source from form result.
> > >> >>
> > >> >>
> > >> >> Hi Marc,
> > >> >>
> > >> >> When you submit a form to one of your DTML methods or
> > >> documents, all the
> > >> >> form variables will be available without any further work
> to be done
> > >> >> (REQUEST object, see docs). But if I get you right, you are not
> > >> >> submitting your form to a method of your own. So things are a bit
> > >> >> different here.
> > >> >>
> > >> >> As I understand it, you are submitting your form to someone
> > >> else's server
> > >> >> (since otherwise you would produce the output yourself).
> > Maybe you're
> > >> >> providing an input form that is directed to a search engine,
> > >> or something
> > >> >> similar.
> > >> >>
> > >> >> --------------------------------------------
> > >> >> <form action="http://server" method=POST>
> > >> >> <input type=hidden name="x" value="a">
> > >> >> <input type=hidden name="y" value="b">
> > >> >>
> > >> >> <input type="text" name="myInput">
> > >> >>
> > >> >> <input type=submit>
> > >> >> </form>
> > >> >> --------------------------------------------
> > >> >>
> > >> >> If so, then you should take a look at the "KebasData" Product:
> > >> >> http://www.zope.org/Members/kedai/KebasData
> > >> >>
> > >> >> Just instantiate a KebasData object, let's say "results". Take a
> > >> >> look at the
> > >> >> product's documentation to see how you can extract the
> > >> relevant code from
> > >> >> the external server's result page. I'll try to get you
> started right
> > >> >> away:
> > >> >>
> > >> >> Set the regular expression patterns properly, start with
> > >> >>
> > >> >> .*
> > >> >>
> > >> >> as the search pattern and leave the rest blank ("url" will be set
> > >> >> by a DTML
> > >> >> method you create, see below), then you'll get the entire
> > >> >> external page, you
> > >> >> can change that later to fit your needs.
> > >> >>
> > >> >> Then create a DTML method next to it, lets say
> > "getResults". It could
> > >> >> look like this:
> > >> >>
> > >> >> <dtml-var standard_html_header>
> > >> >> <h2>RESULTS</h3>
> > >> >> <dtml-with results>
> > >> >> <dtml-call "REQUEST.set('new_url', 'http://server?search=' +
> > >> myInput)">
> > >> >> <dtml-call "get_matched(new_url)">
> > >> >> <dtml-in match>
> > >> >> <dtml-var result><br/>
> > >> >> </dtml-in>
> > >> >> </dtml-with>
> > >> >> <dtml-var standard_html_footer>
> > >> >>
> > >> >> This will fetch the page you want to display by calling
> > >> >> "get_matched(new_url)", you construct that new_url
> > dynamically, using
> > >> >> your form variable "myInput". Then it iterates through the results
> > >> >> list and spits
> > >> >> out the parts that matched your query, appending a <br/> after
> > >> each match
> > >> >> (if you started with .* as the pattern, the list will only
> > >> have one item,
> > >> >> holding the complete page).
> > >> >>
> > >> >> Oh, right: your DTML method should have a proxy role that has the
> > >> >> right to "change KebasData" (or maybe even "View
> > >> >> management_screens"?) on the KebasData object "results", since it
> > >> >> calls its function
> > >> >> "get_matched", which
> > >> >> is usually only called by clicking "Initialize!" in the
> > >> >> management screen of
> > >> >> the KebasData object.
> > >> >>
> > >> >> Now you just send your form to "getResults" (instead of the
> > >> >> external server
> > >> >> directly), and let your DTML method do the rest:
> > >> >>
> > >> >> --------------------------------------------
> > >> >> <form action="getResults" method=POST>
> > >> >> <input type=hidden name="x" value="a">
> > >> >> <input type=hidden name="y" value="b">
> > >> >>
> > >> >> <input type="text" name="myInput">
> > >> >>
> > >> >> <input type=submit>
> > >> >> </form>
> > >> >> --------------------------------------------
> > >> >>
> > >> >> Later, you could put the "getResults" code in a python script,
> > >> then it's
> > >> >> easier to do further processing with your fetched data.
> > >> >>
> > >> >> hth,
> > >> >> Danny
> > >> >>
> > >> >> P.S.: Check the legal stuff before extracting other
> > people's pages ;-)
> > >> >>
> > >> >> >>> -----Ursprungliche Nachricht-----
> > >> >> >>> Von: zope-admin@zope.org [mailto:zope-admin@zope.org]Im
> > >> >> Auftrag von Marc
> > >> >> >>> Fischer
> > >> >> >>> Gesendet: Dienstag, 1. Mai 2001 11.43p
> > >> >> >>> An: Paula Mangas; Marc Fischer
> > >> >> >>> Cc: zope@zope.org
> > >> >> >>> Betreff: AW: [Zope] Source from form result.
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> Hi,
> > >> >> >>>
> > >> >> >>> I think there was an understanding problem. I want to get the
> > >> >> >>> source code of
> > >> >> >>> the result that is displayed in the browser, after klicking on
> > >> >> >>> the submit
> > >> >> >>> button. I want to be able to handle this source perhabs
> > to create
> > >> >> >>> a dtmlmethod wiht it, or do some string extractions with it.
> > >> >> >>>
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> > -----Ursprungliche Nachricht-----
> > >> >> >>> > Von: Paula Mangas [mailto:pamm@students.si.fct.unl.pt]
> > >> >> >>> > Gesendet: Dienstag, 1. Mai 2001 13:25
> > >> >> >>> > An: Marc Fischer
> > >> >> >>> > Cc: zope@zope.org
> > >> >> >>> > Betreff: Re: [Zope] Source from form result.
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > On Tue, 1 May 2001, Marc Fischer wrote:
> > >> >> >>> >
> > >> >> >>> > > Hi,
> > >> >> >>> > >
> > >> >> >>> > > I want to get the HTML result from a form into a
> > dtml method.
> > >> >> >>> > So I need a
> > >> >> >>> > > way to handle this result, but HOW.
> > >> >> >>> > >
> > >> >> >>> > > The form ist like this:
> > >> >> >>> > >
> > >> >> >>> > > <form action="http://server" method=POST>
> > >> >> >>> > > <input type=hidden name="x" value="a">
> > >> >> >>> > > <input type=hidden name="y" value="b">
> > >> >> >>> > > <input type=submit>
> > >> >> >>> > > </form>
> > >> >> >>> > >
> > >> >> >>> > > If I klick on submit, I get the result displayed in the
> > >> >> >>> Browser, but I
> > >> >> >>> > > cannot work with it.
> > >> >> >>> > > Perhaps someone could help me???
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > Hi,
> > >> >> >>> >
> > >> >> >>> >
> > >> >> >>> > I think that, if you try
> > >> >> >>> > <form action="http://server/the_method_you_want_to_call"
> > >> >> method=POST>
> > >> >> >>> >
> > >> >> >>> > it will work.
> > >> >> >>> >
> > >> >> >>> > Paula
> > >> >> >>> >
> > >> >> >>>
> > >> >> >>>
> > >> >> >>> _______________________________________________
> > >> >> >>> Zope maillist - Zope@zope.org
> > >> >> >>> http://lists.zope.org/mailman/listinfo/zope
> > >> >> >>> ** No cross posts or HTML encoding! **
> > >> >> >>> (Related lists -
> > >> >> >>> http://lists.zope.org/mailman/listinfo/zope-announce
> > >> >> >>> http://lists.zope.org/mailman/listinfo/zope-dev )
> > >> >> >>>
> > >> >>
> > >> >
> > >> >
> > >> > _______________________________________________
> > >> > Zope maillist - Zope@zope.org
> > >> > http://lists.zope.org/mailman/listinfo/zope
> > >> > ** No cross posts or HTML encoding! **
> > >> > (Related lists -
> > >> > http://lists.zope.org/mailman/listinfo/zope-announce
> > >> > http://lists.zope.org/mailman/listinfo/zope-dev )
> > >>
> > >>
> > >>
> > >>
> > >>
> > >> _______________________________________________
> > >> Zope maillist - Zope@zope.org
> > >> http://lists.zope.org/mailman/listinfo/zope
> > >> ** No cross posts or HTML encoding! **
> > >> (Related lists -
> > >> http://lists.zope.org/mailman/listinfo/zope-announce
> > >> http://lists.zope.org/mailman/listinfo/zope-dev )
> > >>
> > >
> >
> >
> >
> >
> >
> > _______________________________________________
> > Zope maillist - Zope@zope.org
> > http://lists.zope.org/mailman/listinfo/zope
> > ** No cross posts or HTML encoding! **
> > (Related lists -
> > http://lists.zope.org/mailman/listinfo/zope-announce
> > http://lists.zope.org/mailman/listinfo/zope-dev )
> >
>
>
> _______________________________________________
> Zope maillist - Zope@zope.org
> http://lists.zope.org/mailman/listinfo/zope
> ** No cross posts or HTML encoding! **
> (Related lists -
> http://lists.zope.org/mailman/listinfo/zope-announce
> http://lists.zope.org/mailman/listinfo/zope-dev )
>