[ZWeb] FYI: Readying the "Zope Download Center"

[Paul Everitt]

Guido van Rossum wrote:
>>>>>Have you read my general mirroring proposal?  The help we need
>>>>>right now would concern the architecture, design and
>>>>>implementation of the system I propose there.  It shouldn't be
>>>>>much code, since the essence of mirroring can be done using rsync
>>>>>or ftpmirror.py, but we need tools to find downloadable bits, and
>>>>>to collate download stats.
>>>>
>>>>Sure, though given the history of such efforts, I don't think we
>>>>should wait for new software before fixing this problem.  We've
>>>>talked for over two years about writing software to collect download
>>>>stats, even before spreading download across multiple sites!
>>>
>>>Hm.  For python.org, we use something called Webalizer, which scans
>>>Apache-style log file.  Since Zope wrotes those too, I'm not sure what
>>>the problem is for a single site.  Integrating the results back into
>>>Zope?
>>
>>I did this back in July.  Here's the problems I discovered:
>>
>>1) URLs make it hard to accurately say "what is zope".  There's 
>>different versions of Zope, there's patch files, there are the 
>>different binaries.
> 
> 
> I don't understand this.  Where do you need to know "what is Zope"
> when scanning the log files?  Probably you skipped a step in your
> reasoning.

Ok, tell me the regular expression that you would match on in the log 
file.  I'm not as good as you are at regular expressions.

I was trying to match on the processed results in webalizer, since I 
didn't have access.

>>2) The log files are currently being munched by webalizer and then 
>>thrown out.  Thus, if the URL you want isn't in the list of top 20 
>>hits, you're screwed.  All the information is lost, since the original 
>>data are deleted.  (This was the problem that blocked me, as I never 
>>had access to the boxes where the munching was done).
> 
> 
> That's an operational problem that can be fixed.

I was told that as well.  It's been 7 months.  Is the data still being 
tossed?

Of course these problems can be fixed.  And probably fixed pretty 
easily.  But it proved easier to put things in SF than raise this high 
enough on the priority list for zope.org to get it done.  Hopefully 
you'll have more luck, cause we need download statistics!

>>I was able to extract some useful information and compile it into an 
>>Excel spreadsheet last summer.  But it wasn't authoritative.
> 
> 
> Sorry, you fail to have proved that using Webalizer for zope.org won't
> work.  All you have proved is that it didn't work within the
> constraints you tried it last July.

Ok.  Show me I'm wrong.  I didn't have login access to the box where it 
runs.  I know that you can configure webalizer to keep more data, but 
all that I had access to was the info in the web interface from the 
reports that were produced.

Do you think you can tell us how many downloads of Zope were there in 
2002?  That would really be helpful marketing information.

I sent a full writeup of what I did, and the challenges faced, to geeks 
in June.

[snip]

 > Of course it has to wait for NZO (see Sidnei's post).  I think we can

Why wait?  Even if NZO were up today, and even if this new, extra work 
were also done, this would still be good for everybody.  The only 
downside is that, a few times a year, someone needs to update the 
download center.

 > wait that much longer.  In the mean time, all I'm asking (and have
 > still not heard) is where on zope.org the pointers to the SF downloads
 > are.  If there aren't any, how are people going to find these?

My email at the top of this thread explained this:

"""
Based on the agreed decision to pursue this, Sidnei and I said we'd do 
some work and come back with the results.  Well, it looks good and we're 
ready to take the next step:
[...snip...]
The next step is to get this woven into the /Products page, which could 
also use some sprucing up.  We hope to get this done soon, in advance of 
the 2.6.1 release.
"""

Sidnei and I promised on this list to do some investigation and report 
back.  We did the investigation, we're now reporting back.

Changing your question to where the links will be, I'll put them on the 
/Products page this week, along with an improved organization and 
presentation of the download options.

--Paul