[Zope] Big improvement for load_site.py, patch included

[Jerome Alet]

  This message is in MIME format.  The first part should be readable text,
  while the remaining parts are likely unreadable without MIME-aware tools.
  Send mail to mime@docserver.cac.washington.edu for more info.

--889292347-288126141-995979212=:31882
Content-Type: TEXT/PLAIN; charset=US-ASCII

Hi,

I'm pleased to send the attached patch for load_site.py (tested against
load_site.py from Zope 2.3.2), just tell me if it's useful.

The original load_site.py determined the content-type of the file to
upload to the ZODB from the file's extension, and a wrong content type
(text/plain, the default one) was set for unrecognized extensions. For
example PDF files were uploaded as text/plain.

The modified version uses urllib to get the real content-type. urllib is
able to load local content, so why not use it ?

This allowed me to delete the methods to handle images, which are now
handled automagically, as well as upload a bunch of documents to my ZODB
while preserving their original content-type. 

Another thing at which urllib would be useful is at allowing load_site to
load sites from the web instead of from the local filesystem, however more
work is needed for this to work with directories.

Some sanitization is done to document's ids before uploading them to the
ZODB, because I encountered problems trying to upload files which names
contained spaces or accented characters: each invalid character is
replaced with an underscore. However this may produce invalid ids too, so
perhaps a better solution is needed.

<dtml-rant mode="LOUD">

load_site is wonderful, however I really think it could be even better
if my idea of putting the ZODB object's meta-type in a single comment line
at the top of DTML Documents and DTML Methods was finally implemented (see
zope@zope.org archives). 

This comment, why not a # comment instead of a <!-- --> or <dtml-comment
...>, would be set on EXPORT/ViewSource only, and stripped by Zope on
IMPORT/CREATION This would allow people to easily work with both DTML
Documents and DTML Methods in external tools and have them uploaded with
the correct metatype.  Given that PythonScripts already contain such a
stupid comment which means nothing to external tools but only to Zope, I
really can't see why DTML Documents and Methods wouldn't have the same
thing !  Of course on import/creation if the said comment is absent, then
the actual behavior would be preserved, i.e. the 50% chances to be wrong
would still be there for those who don't want to understand how much this
would simplify some people's work. 

I'd be glad with something like:
	
	# DTML Method
	<dtml-something ...>
	...

or:

	<!-- DTML Document -->
	<dtml-something ...>
	...

I'd be just as glad to counter any objection, but please send them
privately and I'll summarize. 

</dtml-rant>

hope-this-helps-and-won't-be-lost-into-the-Collector'sly yours.

Jerome Alet - alet@unice.fr

--889292347-288126141-995979212=:31882
Content-Type: TEXT/PLAIN; charset=US-ASCII; name="load_site.py.patch"
Content-Transfer-Encoding: BASE64
Content-ID: <Pine.LNX.3.96.1010724145332.31882C@cortex.unice.fr>
Content-Description: 

LS0tIGxvYWRfc2l0ZS5weS5vcmlnCVR1ZSBKdWwgMjQgMTQ6MjE6MDUgMjAw
MQ0KKysrIGxvYWRfc2l0ZS5weQlUdWUgSnVsIDI0IDE0OjIyOjM0IDIwMDEN
CkBAIC0xMTksNyArMTE5LDcgQEANCiAgICAgICAgICBVc2UgKm9sZCogem9w
ZSBtZXRob2QgbmFtZXMuDQogIiIiDQogDQotaW1wb3J0IHN5cywgZ2V0b3B0
LCBvcywgc3RyaW5nDQoraW1wb3J0IHN5cywgZ2V0b3B0LCBvcywgc3RyaW5n
LCB1cmxsaWINCiBTZXJ2ZXJFcnJvcj0nJw0KIHZlcmJvc2U9MA0KIG9sZD0w
DQpAQCAtMTczLDYgKzE3MywxNyBAQA0KIA0KICAgICBmb3IgZiBpbiBmaWxl
czogdXBsb2FkX2ZpbGUob2JqZWN0LCBmKQ0KIA0KK2RlZiBzYW5pdGl6ZShp
ZCkgOg0KKyAgICAjIHNhbml0aXplIHRoZSBpZCBpbiBjYXNlIGl0IGNvbnRh
aW5zIHNwZWNpYWwgY2hhcmFjdGVycw0KKyAgICAjIG1vcmUgY2xlYW4gc2Fu
aXRpemF0aW9uIHNob3VsZCBiZSBkb25lLCBvZiBjb3Vyc2UuLi4NCisgICAg
dmFsaWQgPSAiMDEyMzQ1Njc4OUFCQ0RFRkdISUpLTE1OT1BRUlNUVVZXWFla
YWJjZGVmZ2hpamtsbW5vcHFyc3R1dnd4eXpfLiINCisgICAgZm9yIGkgaW4g
cmFuZ2UobGVuKGlkKSkgOg0KKwkjIGlmIGNoYXJhY3RlciBpcyBpbnZhbGlk
DQorCWlmIGlkW2ldIG5vdCBpbiB2YWxpZCA6DQorCSAgICAjIHJlcGxhY2Ug
d2l0aCBhbiB1bmRlcnNjb3JlDQorCSAgICBpZD0gaWRbOmldICsgJ18nICsg
aWRbaSsxOl0NCisgICAgcmV0dXJuIGlkDQorDQogZGVmIGNhbGwoZiwgKmFy
Z3MsICoqa3cpOg0KICAgICAjIENhbGwgYSBmdW5jdGlvbiBpZ25vcmluZyBy
ZWRpcmVjdCBiY2kgZXJyb3JzLg0KICAgICB0cnk6IGFwcGx5KGYsYXJncywg
a3cpDQpAQCAtMTgxLDYgKzE5Miw3IEBADQogICAgICAgICAgICAgcmFpc2Ug
c3lzLmV4Y19pbmZvKClbMF0sIHN5cy5leGNfaW5mbygpWzFdLCBzeXMuZXhj
X2luZm8oKVsyXQ0KIA0KIGRlZiB1cGxvYWRfZmlsZShvYmplY3QsIGYpOg0K
KyAgICBmID0gb3MucGF0aC5leHBhbmR1c2VyKGYpCSMgdGFrZXMgY2FyZSBv
ZiB+dXNlcm5hbWUvLi4uIGFuZCB+Ly4uLg0KICAgICBpZiBvcy5wYXRoLmlz
ZGlyKGYpOiByZXR1cm4gdXBsb2FkX2RpcihvYmplY3QsIGYpDQogICAgIGRp
ciwgbmFtZSA9IG9zLnBhdGguc3BsaXQoZikNCiAgICAgcm9vdCwgZXh0ID0g
b3MucGF0aC5zcGxpdGV4dChuYW1lKQ0KQEAgLTE5MywxMSArMjA1LDM3IEBA
DQogICAgICAgICByZXR1cm4gZ2xvYmFscygpWyd1cGxvYWRfJytleHRdKG9i
amVjdCwgZikNCiANCiAgICAgaWYgdmVyYm9zZTogcHJpbnQgJ3VwbG9hZF9m
aWxlJywgZiwgZXh0DQotICAgIGNhbGwob2JqZWN0Lm1hbmFnZV9hZGRGaWxl
LCBpZD1uYW1lLCBmaWxlPW9wZW4oZiwncmInKSkNCisNCisgICAgIyB3ZSBu
b3cgdXNlIHVybGxpYiB0byBnZXQgdGhlIHJlYWwgY29udGVudC10eXBlLg0K
KyAgICBuYW1lID0gc2FuaXRpemUobmFtZSkNCisgICAgdHJ5IDoNCisJIyBX
QVJOSU5HOiBib3RoIHVybG9wZW4gYW5kIG9iamVjdC5yZWFkIG1heQ0KKwkj
IHJhaXNlIGFuIElPRXJyb3IsIGluIHRoZSBsYXR0ZXIgY2FzZSB0aGF0J3Mg
d2hlbg0KKwkjIG9iamVjdCBpcyBhIGZpbGVzeXN0ZW0gZGlyZWN0b3J5LCBi
dXQgdGhpcyBvbmUgc2hvdWxkbid0IG9jY3VyDQorCSMgYmVjYXVzZSBkaXJl
Y3RvcmllcyBhcmUgdXBsb2FkZWQgYnkgdXBsb2FkX2RpcigpDQorCWZzZmls
ZSA9IHVybGxpYi51cmxvcGVuKGYpDQorCWluZm8gPSBmc2ZpbGUuaW5mbygp
DQorCWN0eXBlID0gaW5mby5nZXR0eXBlKCkNCisJbXR5cGUgPSBpbmZvLmdl
dG1haW50eXBlKCkNCisJcmVhbHVybCA9IGZzZmlsZS5nZXR1cmwoKQ0KKwlp
ZiBtdHlwZSA9PSAiaW1hZ2UiIDoNCisJICAgICMgSW1hZ2UNCisJICAgIGNh
bGwob2JqZWN0Lm1hbmFnZV9hZGRJbWFnZSwgaWQgPSBuYW1lLCBmaWxlID0g
ZnNmaWxlLCB0aXRsZSA9IHJlYWx1cmwsIHByZWNvbmRpdGlvbiA9ICcnLCBj
b250ZW50X3R5cGUgPSBjdHlwZSkNCisJZWxpZiBjdHlwZSA9PSAndGV4dC9o
dG1sJyA6DQorCSAgICAjIERUTUwgRG9jdW1lbnQsIGp1c3QgaW4gY2FzZSBp
dCB3YXMgbm90IHNlZW4gYnkgdXBsb2FkX2h0bWwvdXBsb2FkX2h0bQ0KKwkg
ICAgY2FsbChvYmplY3QubWFuYWdlX2FkZERUTUxEb2N1bWVudCwgaWQgPSBu
YW1lLCB0aXRsZSA9IHJlYWx1cmwsIGZpbGUgPSBmc2ZpbGUpDQorCWVsc2Ug
Og0KKwkgICAgIyBub3JtYWwgRmlsZQ0KKwkgICAgY2FsbChvYmplY3QubWFu
YWdlX2FkZEZpbGUsIGlkID0gbmFtZSwgZmlsZSA9IGZzZmlsZSwgdGl0bGUg
PSByZWFsdXJsLCBwcmVjb25kaXRpb24gPSAnJywgY29udGVudF90eXBlID0g
Y3R5cGUpDQorCWZzZmlsZS5jbG9zZSgpDQorCWRlbCBmc2ZpbGUNCisgICAg
ZXhjZXB0IElPRXJyb3IsbXNnIDoNCisJc3lzLnN0ZGVyci53cml0ZSgnRXJy
b3IgJXMsIG9jY3VyZWQgd2hpbGUgcmV0cmlldmluZyAlc1xuJyAlIChtc2cs
IGYpKQ0KIA0KIGRlZiB1cGxvYWRfZGlyKG9iamVjdCwgZik6DQogICAgIGlm
IHZlcmJvc2U6IHByaW50ICd1cGxvYWRfZGlyJywgZg0KICAgICBkaXIsIG5h
bWUgPSBvcy5wYXRoLnNwbGl0KGYpDQorICAgIG5hbWUgPSBzYW5pdGl6ZShu
YW1lKQ0KICAgICBjYWxsKG9iamVjdC5tYW5hZ2VfYWRkRm9sZGVyLCBpZD1u
YW1lKQ0KICAgICBvYmplY3Q9b2JqZWN0Ll9fY2xhc3NfXyhvYmplY3QudXJs
KycvJytuYW1lLA0KICAgICAgICAgICAgICAgICAgICAgICAgICAgICB1c2Vy
bmFtZT1vYmplY3QudXNlcm5hbWUsDQpAQCAtMzA5LDYgKzM0Nyw3IEBADQog
DQogZGVmIHVwbG9hZF9odG1sKG9iamVjdCwgZik6DQogICAgIGRpciwgbmFt
ZSA9IG9zLnBhdGguc3BsaXQoZikNCisgICAgbmFtZSA9IHNhbml0aXplKG5h
bWUpDQogICAgIGY9b3BlbihmKQ0KIA0KICAgICBpZiBkb2N0b3I6DQpAQCAt
MzU1LDYgKzM5NCw3IEBADQogDQogZGVmIHVwbG9hZF9kdG1sKG9iamVjdCwg
Zik6DQogICAgIGRpciwgbmFtZSA9IG9zLnBhdGguc3BsaXQoZikNCisgICAg
bmFtZSA9IHNhbml0aXplKG5hbWUpDQogICAgIGY9b3BlbihmKQ0KIA0KICAg
ICBpZiBvbGQ6DQpAQCAtMzYzLDEzICs0MDMsNiBAQA0KICAgICBlbHNlOg0K
ICAgICAgICAgY2FsbChvYmplY3QubWFuYWdlX2FkZERUTUxNZXRob2QsIGlk
PW5hbWUsIGZpbGU9ZikNCiAgICAgICAgIA0KLQ0KLWRlZiB1cGxvYWRfZ2lm
KG9iamVjdCwgZik6DQotICAgIGRpciwgbmFtZSA9IG9zLnBhdGguc3BsaXQo
ZikNCi0gICAgY2FsbChvYmplY3QubWFuYWdlX2FkZEltYWdlLCBpZD1uYW1l
LCBmaWxlPW9wZW4oZiwncmInKSkNCi0NCi11cGxvYWRfanBnPXVwbG9hZF9n
aWYNCi11cGxvYWRfcG5nPXVwbG9hZF9naWYNCiANCiBpZiBfX25hbWVfXz09
J19fbWFpbl9fJzogbWFpbigpDQogDQo=
--889292347-288126141-995979212=:31882--