[Grok-dev] grokproject/buildout problems

[Maurits van Rees]

Hi Martijn,

Martijn Faassen, on 2008-09-03:
> Hi there,
>
> There are a bunch of problems with grokproject/grok buildout that I'd 
> like to see investigated and addressed. The big, big problem we need to 
> fix sooner rather than later is that the grokproject generated buildouts 
> are actually fundamentally broken - I can't run bootstrap.py for a 
> grokproject generated project in a fresh environment and have things 
> work! We can't live with that...
>
> The problems:
>
> enter a fresh virtual environment
> remove .buildout and everything in it, etc
> $ easy_install grokproject (0.8)
> $ grokproject something
>
> * "Download error: unknown url type: svn -- Some packages may not be 
> found!". non-urgent but very long standing and confuses people

setuptools shows this message.  One of the packages has an svn url
in setup.py, which shows up in the egg index as possible download
location.  I don't think there is much we can do about that, except
not make that mistake again ourselves.

Ah, in this case lovely.recipe is the culprit.  The 0.3.1b1 and
0.3.1b5 home pages point to svn. See:
http://pypi.python.org/simple/lovely.recipe

> * "Couldn't find index page for 'grok' (maybe misspelled?)" - what index 
> page? What is triggering this?

Somehow grok 0.13 or one of its dependencies can not be found in the
tarball.  This Should Not Happen (tm) but apparently it does.

So buildout looks in the index page.  The index page is in fact an
empty directory made by eggbasket.  Naturally the missing dependency
cannot be found there either, so buildout prints this error.

The empty directory/index is vital when creating the tarball.  But it
makes no sense when installing from that tarball.  It *should* not
matter, but apparently it does.  I have now changed that in the
eggbasket recipe.

> * "warning: no files found matching '0.1.0-changes.txt'"

This is from mechanize 0.1.7b.  This has a MANIFEST.in file that says
among other:

  include 0.1.0-changes.txt

But that file does not exist; there is only a 0.1-changes.txt file.

> * "no previously-included directories found matching 'docs-in-progress'"

This is also from mechanize.  The MANIFEST.in file has:

  prune docs-in-progress

But there is no such file or directory.

> * "eggbasket: Failed to install required eggs with the tar ball. 
> Continuing with buildout instead." - repeating twice

See below.

> The generated project does work in the sense that bin/zopectl is there 
> starts.
>
> Now create a new fresh virtual environment, activate it, and go into 
> 'something', removing this:
>
> $ rm -rf develop-eggs/
> $ rm -rf .installed.cfg
> $ rm -rf parts/
> $ rm -rf bin/
>
> and also putting aside .buildout again:
>
> after this, do:
>
> $ python bootstrap.py
>
> * One thing I notice is a lot of duplication of messages:
>
> Getting distribution for 'z3c.recipe.eggbasket==0.3.1'.
> Getting distribution for 'z3c.recipe.eggbasket==0.3.1'.
> Got z3c.recipe.eggbasket 0.3.1.
> Got z3c.recipe.eggbasket 0.3.1.
>
> basically all messages are duplicated (and these are a lot of 
> messages!). Why? Needs to be fixed

Philipp reported this bug about that:
https://bugs.launchpad.net/zopeproject/+bug/125729

> * "eggbasket: Failed to install required eggs with the tar ball. 
> Continuing with buildout instead." (duplicated). After this "continuing 
> with buildout instead" doesn't do anything - it just stops here. We 
> don't end up with a working buildout!

No idea.  Do you mean it ends and you are back on the command line?
Or is it still continuing without printing anything, probably heading
for a timeout while trying to download something?

> * Because I happen to know what's up with buildout, I run 
> 'bin/buildout'. It seems to be getting the docutils 0.4 distribution 
> online for some reason - why isn't this in the eggbasket tarball? Or 
> perhaps it is and it isn't picked up for some reason? The whole point of 
> it after all is to avoid such dangerous side-trips to sourceforge..

It is not in the tarball.  Okay, this is strange.  When I create the
grok 0.13 tarball (bin/bundlemaker in the grok trunk checkout) I get
these differences with the original tarball:

  < grok-eggs-0.13/ClientForm-0.2.7.zip
  ---
  > grok-eggs-0.13/ClientForm-0.2.7.tar.gz
  > grok-eggs-0.13/docutils-0.4.tar.gz

The zip instead of tar.gz is fine with me.  But for some unknown
reason docutils is not included in our official tarball.  Running the
bin/bundlemaker normally or from a virtualenv with --no-site-packages
made no difference.

> * Things do appear to work then.
>
> Another issue I'm noticing now:
>
> * I installed grokproject already in one virtual environment. Now in 
> another virtual environment I install grokproject again and run it. It 
> now downloads something big (it doesn't seem to specify what, but it 
> must be eggbasket). Then I get:

Yes, eggbasket should be more informative.  But there is plenty logged
at log level 'info', so I'm not sure why that is not showing up.
'paster create -t grok' should be more talkative iirc.

Okay, we had fixed the bootstrap.py in the grokproject template so it
ran the install of the eggbasket part.  But the buildout bootstrapping
done by grokproject does nothing with our bootstrap.py.  I am pretty
sure this worked a few months ago...

Right, I have now changed grokproject so it really runs the install of
eggbasket itself.  And that makes it print more info.

BTW, when it prints lines like "Getting distribution for..." it is
actually getting the distribution from within the tarball.

> * "eggbasket: Failed to install required eggs with the tar ball. 
> Continuing with buildout instead." Does it not pick up the stuff in 
> .buildout/eggs? Do we need a separate one per virtual env? If we use 
> virtualenv anyway (it looks more and more inevitable and we already 
> recommend it in various places), can we do away with our egg cache there 
> altogether and simply use the virtual env's site-packages?

eggbasket looks in the specified eggs_directory to see if the required
grok version and its dependencies are installed there.  If not, it
gets the big tarball.

> * Even after this it takes forever, still not outputting what it is 
> doing. I happen to be on a slow network right now so I probably notice 
> these problems more quickly. When I run grokproject twice while in the 
> same virtualenv, starting without .buildout directory, grokproject 
> *does* run quickly the same time. I wonder what causes this...

The second time it *does* have a filled .buildout/eggs directory,
right?  Then it is logical.


So I fixed a few small things in eggbasket and grokproject while
answering this message.  That should help.  What can still be done is:

- Recreate the 0.13 grok eggs tarball, make sure docutils is in it,
  and upload it.  Why it is not in the current tarball, I don't know.

- Change the eggbasket creator so it accepts more than one egg.  Then
  in the bundlemaker section of grok trunk: add the recipes used by a
  fresh grokproject to the egg(s) parameter and add good versions of
  them (and their dependencies) in the versions.cfg - probably for the
  new 0.14.

-- 
Maurits van Rees | http://maurits.vanrees.org/
            Work | http://zestsoftware.nl/
"This is your day, don't let them take it away." [Barlow Girl]