[Zope3-checkins] CVS: Zope3/src/zope/fssync - README.txt:1.9.2.1 fsmerger.py:1.2.2.1 fssync.py:1.24.2.1 main.py:1.15.2.1 merger.py:1.8.2.1 metadata.py:1.2.2.1 snarf.py:1.1.2.1 compare.py:NONE
Grégoire Weber
zope@i-con.ch
Sun, 22 Jun 2003 10:24:13 -0400
Update of /cvs-repository/Zope3/src/zope/fssync
In directory cvs.zope.org:/tmp/cvs-serv24874/src/zope/fssync
Modified Files:
Tag: cw-mail-branch
README.txt fsmerger.py fssync.py main.py merger.py metadata.py
snarf.py
Removed Files:
Tag: cw-mail-branch
compare.py
Log Message:
Synced up with HEAD
=== Zope3/src/zope/fssync/README.txt 1.9 => 1.9.2.1 ===
--- Zope3/src/zope/fssync/README.txt:1.9 Fri May 16 08:52:28 2003
+++ Zope3/src/zope/fssync/README.txt Sun Jun 22 10:23:42 2003
@@ -8,6 +8,13 @@
This version is based loosely on a prototype written by Jim Fulton and
Deb Hazarika. It is now maintained by Guido van Rossum.
+Possibly also relevant background:
+
+ http://dev.zope.org/Zope3/ThroughTheWebSiteDevelopment
+
+The "bundles" mentioned there are likely candidates for filesystem
+synchronization. (See section "Working with bundles" below.)
+
User stories
------------
@@ -50,49 +57,68 @@
status, and the simplest form of diff) must be performed entirely
offline.
+* An interesting possibility: you could couple your filesystem copy to
+ a revision control system like CVS or Subversion, to have an
+ auditable revision history of a site. Typically, you'd do a cvs
+ commit after each sync update and after each sync commit, after
+ verifying that the state committed to Zope actually works. It would
+ be handy if files added to or removed from Zope are automatically
+ added or removed from CVS. The "binary" flag for CVS might be set
+ automatically based on the Zope object type.
+
+* Another possibility: export and import (a la Zope 2 export/import)
+ should be easily implemented on top of this. Export would be done
+ with checkout; import could be a new "checkin" command. (This is
+ now implemented.)
+
+* And last but not least, this will form the basis of bundles; see the
+ ThroughTheWebSiteDevelopment reference above.
+
BUGS
----
-* Sometimes when committing additions or removals, the Entries.xml
- file doesn't get updated properly.
+* When committing an added file, you must commit the directory
+ containing it; you can't commit the file itself, since the command
+ tries to send the request to a view of the corresponding object,
+ which doesn't exist yet.
+
+* When doing an update, somehow the absolute pathnames of all files
+ are reported rather than the nice relative names.
TO DO
-----
-* Rewrite fromFS to integrate uptodate checking; because the db is
- transactional it's ok to have made some changes and later raise an
- exception. Then it could also update the disk copy in-place to
- reflect changes, ready to be zipped and sent back.
-
-* Don't rely on external zip/unzip tools. Maybe switch to tar as the
- archival format, because it is easier to stream and compress at the
- same time. The file can be probably streamed right to the socket,
- without going to a temp file first, assuming the receiver can handle
- not having a Content-length header.
-
-* more unit tests for fsmerge, to check that entries are handled
- correctly in all cases (including addition/deletion/change of
- type).
-
-* unit tests for the fssync core functionality
-
-* more refactoring and cleanup of the fssync core functionality
-
-* more diff options:
- -2 diffs between local and remote
- -3 diffs between original and remote
- -N shows diffs for added/removed files as diffs with /dev/null
- more GNU diff options? e.g. --ignore-space-change etc.
+- Implement bundle commands.
+
+- On the server side:
+
+ * Nothing ATM.
+
+- In the sync application:
+
+ * Implement diff using difflib.
-* something akin to cvs -n update, which shows what update would do
- without actually doing it
+ * More diff options:
+ -2 diffs between local and remote
+ -3 diffs between original and remote
+ -N shows diffs for added/removed files as diffs with /dev/null
-* commit shouldn't commit new versions of unchanged objects to ZODB
+ * More GNU diff options? e.g. --ignore-space-change etc.
-* refine the adapter protocol or implementation to leverage the
- file-system representation protocol
+ * Something akin to cvs -n update, which shows what update would do
+ without actually doing it.
+
+- Code maintenance:
+
+ * Rewrite toFS() to use the Metadata class, and add unit tests.
+
+ * Unit tests for the fssync utility.
+
+ * More refactoring and cleanup of the fssync utility.
+
+ * Use camelCase for public method names.
TO DO LATER
@@ -100,12 +126,23 @@
* Work out security details.
+* A commit unpickles user-provided data. Unpickling is not a safe
+ operation. Possible solution: have an unpickler that finds globals
+ in a secure way. Use an import on a security proxy for sys.modules.
+
+* The adapters returned by the fs registry should optionally have
+ a permission associated with them. If you have an adapter that
+ calls removeAllProxies, the adapter should require a permission.
+
+* Refine the fssync adapter protocol or implementation to leverage the
+ file-system representation (== FTP, WebDAV) protocol.
+
* In common case where extra data are simple values, store extra data
in the entries file to simplify representation and updates. Maybe
do something similar w annotations.
* Maybe do some more xmlpickle refinement with an eye toward
- impproving the usability of simple dictionary pickles.
+ improving the usability of simple dictionary pickles.
* Maybe leverage adaptable storage ideas to assure losslessness.
@@ -116,3 +153,147 @@
* Commit to multiple Zope instances?
* Diff/merge multiple working sets (a la bitkeeper)?
+
+
+Working with bundles
+--------------------
+
+- Bundles aren't quite as easy to use as they are supposed to be as
+ described in the ThroughTheWebSiteDevelopment Wiki page referenced
+ above, but you can do some basic bundle-ish things. All these need
+ is a little better packaging.
+
+- The fssync command. Below, examples use a command named fssync.
+ This doesn't yet exist. Best is to have a shell alias that points
+ to the file <Zope3>/src/fssync/main.py, where <Zope3> is the root of
+ the Zope3 tree.
+
+- Permissions. Everything described here requires the
+ zope.ManageServices permission, which usually requires being logged
+ in with the manager role.
+
+- Bundle status. There is not yet an explicit notion of "bundle-ness"
+ for site management folders. Any site management folder can be
+ treated as a bundle. Exception: the Bundle view works for the
+ default folder but the form included in the view refuses to change
+ it. This is a safety measure: the bundle form can do a lot of
+ damage, e.g. it can disable all services at once. By convention, I
+ propose that bundles have a folder name of the form
+ <name>-<version>, where <version> is two or more decimal numbers
+ separated by dots and <name> is unconstrained.
+
+- Creating a bundle. There is no specific command to create a
+ bundle. Instead, you create a new site management folder by going
+ to the Contents view of the site (e.g. /++etc++site/@@contents.html)
+ and clicking on "Add" in the actions menu. A box will appear in
+ which you should type the name + version of your bundle. Then in
+ that bundle you should create the things that you want to go into
+ the bundle, e.g. modules, templates, services, utilities, etc.
+
+- Creating a bundle from an existing folder. If you have some
+ existing work done in the default folder or another non-bundle
+ folder, you can save your work to the filesystem using the fssync
+ checkout command, and then check it in under a different name using
+ the fssync checkin command. Example; replace u:p with your manager
+ username and password:
+
+ $ fssync checkout http://u:p@localhost:8080/++etc++site/default
+ <lots of output>
+ All done.
+ $ fssync checkin http://u:p@localhost:8080/++etc++site/bundle-1.0 default
+ $
+
+ Now go back to your web browser and check out the contents of
+ /++etc++site/; a new folder bundle-1.0 should exist, containing a
+ copy of the default folder. You should delete unnecessary things;
+ especially the standard service definitions are not needed.
+
+- Exporting a bundle. First deactivate the bundle bu using the
+ "Deactivate bundle" button on the Bundle tab (see below). Then save
+ the bundle to the filesystem using fssync checkout. Finally tar or
+ zip it up. Make sure to include the @@Zope directory at the same
+ level as the bundle directory in the archive. Example:
+
+ $ fssync checkout http://u:p@localhost:8080/++etc++site/bundle-1.0
+ <lots of output>
+ All done.
+ $ tar tf - bundle-1.0 @@Zope | gzip >bundle-1.0.tgz
+ $
+
+ Now distribute the gzipped tar file file via the web.
+
+- Importing a bundle. First extract the zip or tar file to the
+ filesystem. Then use fssync checkin command to add it to your Zope
+ server. Warning: the checkin command will happily overwrite an
+ existing site management folder!
+
+- Activating a bundle. An imported bundle is completely inactive.
+ Its configuration records (the objects in the bundle's
+ RegistrationManager subfolder, and in RegistrationManager subfolders
+ of subfolders of the bundle) are not registered with their
+ respective services. To activate the bundle, navigate your web
+ browser to its contents and select the Bundle tab; it is probably
+ the second tab from the left. The Bundle tab displays two sections
+ and a few buttons.
+
+ - Section one of the Bundle tab shows the services needed by the
+ bundle. This list is created by inspection of the bundle's
+ configuration records: for example, if there is a configuration
+ record for a utility, the service needs the utility service.
+ For each needed service, there are three possibilities:
+
+ 1) The service is already active in the site. This is probably
+ because it exists in the default folder or in a previous
+ bundle.
+
+ 2) The service is not yet active in the site but the bundle
+ provides a configuration for the service.
+
+ 3) No usable definition of the service can be found. Note that a
+ service active in a parent site cannot be used. This is called
+ an unfulfilled dependency. This means that the bundle cannot
+ be activated. A helpful link to the "Add service" view of the
+ default folder is provided, where you can create (and
+ activate!) the service and then navigate back to the bundle;
+ but you may also import the service as part of another bundle.
+
+ - Section two of the Bundle tab shows, for each of the service types
+ shown in section one, all configuration records in the bundle for
+ that service type. Initially, all configurations are in the
+ "Unregistered" state. At the bottom of the list you will find a
+ button which will register all configurations, and activate the
+ ones that aren't in conflict with pre-existing registrations.
+ Conflicts are indicated in red and provide a link to the
+ conflicting active configuration record, probably in another
+ bundle. The automatic resolution of conflicts in favor of a newer
+ version of the same bundle, mentioned in the Wiki, is not yet
+ implemented; by default, whenever there is a conflict the
+ conflicting configuration record in the bundle is not activated.
+ You can resolve conflicts yourself in favor of the new bundle by
+ clicking the radio button labeled "Register and activate". You
+ can also leave a configuration record inactive by clicking the
+ radio button "Register only". When you are satisfied with the
+ selections, click the "Activate bundle" button below the list to
+ register and activate the bundle's configuration records; this
+ performs the actions selected by the radio buttons. If you later
+ change your mind, you can always go back to the Bundle tab and
+ change your selections.
+
+ - At the very bottom of the page is a button labeled "Deactivate
+ bundle". This is used for uninstalling a bundle; it makes all
+ configuration records contained in the bundle inactive and
+ unregistered. It is also used for exporting a bundle; before you
+ export a bundle, you should deactivate it (see above). In
+ contrast to the description in the Wiki, deactivating a bundle
+ does not reactivate any configuration records that were active
+ before the bundle was activated, because the configuration
+ registries don't record this information; it can't distinguish
+ between previously active and previously registered. A redesign
+ of the registries would be necessary to accommodate this feature.
+
+- To delete a deactivated bundle, go to the site manager's contents
+ display (/++etc++site/@@contents.html), select the checkbox in front
+ of the bundle name, and click the Delete button below the list.
+ Deleting an active bundle usually doesn't work because of the
+ dependencies between the configuration records and the configured
+ objects in the bundle.
=== Zope3/src/zope/fssync/fsmerger.py 1.2 => 1.2.2.1 ===
--- Zope3/src/zope/fssync/fsmerger.py:1.2 Thu May 15 11:32:23 2003
+++ Zope3/src/zope/fssync/fsmerger.py Sun Jun 22 10:23:42 2003
@@ -17,10 +17,13 @@
"""
import os
+import shutil
from os.path import exists, isfile, isdir, split, join
from os.path import realpath, normcase, normpath
+from zope.xmlpickle import dumps
+
from zope.fssync.merger import Merger
from zope.fssync import fsutil
@@ -53,19 +56,49 @@
self.reporter("XXX %s" % local)
self.merge_extra(local, remote)
self.merge_annotations(local, remote)
+ if not exists(local) and not self.metadata.getentry(local):
+ self.remove_special(local, "Extra")
+ self.remove_special(local, "Annotations")
+ self.remove_special(local, "Original")
def merge_extra(self, local, remote):
+ """Helper to merge the Extra trees."""
lextra = fsutil.getextra(local)
rextra = fsutil.getextra(remote)
self.merge_dirs(lextra, rextra)
def merge_annotations(self, local, remote):
+ """Helper to merge the Anotations trees."""
lannotations = fsutil.getannotations(local)
rannotations = fsutil.getannotations(remote)
self.merge_dirs(lannotations, rannotations)
+ def remove_special(self, local, what):
+ """Helper to remove an Original, Extra or Annotations file/tree."""
+ head, tail = fsutil.split(local)
+ dir = join(head, "@@Zope", what)
+ target = join(dir, tail)
+ if exists(target):
+ if isdir(target):
+ shutil.rmtree(target)
+ else:
+ os.remove(target)
+ if isdir(dir):
+ try:
+ os.rmdir(dir)
+ except os.error:
+ pass
+
def merge_files(self, local, remote):
"""Merge remote file into local file."""
+
+ # Reset sticky conflict if file was edited or removed
+ entry = self.metadata.getentry(local)
+ conflict = entry.get("conflict")
+ if conflict and (not os.path.exists(local) or
+ conflict != os.path.getmtime(local)):
+ del entry["conflict"]
+
original = fsutil.getoriginal(local)
action, state = self.merger.classify_files(local, original, remote)
state = self.merger.merge_files(local, original, remote,
@@ -74,12 +107,12 @@
def merge_dirs(self, localdir, remotedir):
"""Merge remote directory into local directory."""
- lnames = self.metadata.getnames(localdir)
- rnames = self.metadata.getnames(remotedir)
+ lentrynames = self.metadata.getnames(localdir)
+ rentrynames = self.metadata.getnames(remotedir)
lentry = self.metadata.getentry(localdir)
rentry = self.metadata.getentry(remotedir)
- if not lnames and not rnames:
+ if not lentrynames and not rentrynames:
if not lentry:
if not rentry:
@@ -87,9 +120,11 @@
self.reportdir("?", localdir)
else:
if not exists(localdir):
- fsutil.ensuredir(localdir)
+ self.make_dir(localdir)
+ lentry.update(rentry)
self.reportdir("N", localdir)
else:
+ self.make_dir(localdir)
self.reportdir("*", localdir)
return
@@ -110,31 +145,51 @@
return
if not rentry:
- try:
- os.rmdir(localdir)
- except os.error:
- pass
- self.reportdir("D", localdir)
- lentry.clear()
+ self.clear_dir(localdir)
return
if exists(localdir):
- self.reportdir("/", localdir)
+ if lentry.get("flag") == "added":
+ if exists(remotedir):
+ self.reportdir("U", localdir)
+ del lentry["flag"]
+ else:
+ self.reportdir("A", localdir)
+ else:
+ if rentry or exists(remotedir):
+ self.reportdir("/", localdir)
+ else:
+ # Tree removed remotely, must recurse down locally
+ for name in lentrynames:
+ self.merge(join(localdir, name), join(remotedir, name))
+ self.clear_dir(localdir)
+ return
+
lnames = dict([(normcase(name), name)
for name in os.listdir(localdir)])
else:
- if lentry.get("flag") != "removed" and (rentry or rnames):
- fsutil.ensuredir(localdir)
+ flag = lentry.get("flag")
+ if flag == "removed":
+ self.reportdir("R", localdir)
+ return # There's no point in recursing down!
+ if rentry or rentrynames:
+ self.make_dir(localdir)
lentry.update(rentry)
self.reportdir("N", localdir)
lnames = {}
+ for name in lentrynames:
+ lnames[normcase(name)] = name
+
if exists(remotedir):
rnames = dict([(normcase(name), name)
for name in os.listdir(remotedir)])
else:
rnames = {}
+ for name in rentrynames:
+ rnames[normcase(name)] = name
+
names = {}
names.update(lnames)
names.update(rnames)
@@ -146,6 +201,45 @@
for ncname in ncnames:
name = names[ncname]
self.merge(join(localdir, name), join(remotedir, name))
+
+ def make_dir(self, localdir):
+ """Helper to create a local directory.
+
+ This also creates the @@Zope subdirectory and places an empty
+ Entries.xml file in it.
+ """
+ fsutil.ensuredir(localdir)
+ localzopedir = join(localdir, "@@Zope")
+ fsutil.ensuredir(localzopedir)
+ efile = join(localzopedir, "Entries.xml")
+ if not os.path.exists(efile):
+ data = dumps({})
+ f = open(efile, "w")
+ try:
+ f.write(data)
+ finally:
+ f.close()
+
+ def clear_dir(self, localdir):
+ """Helper to get rid of a local directory.
+
+ This zaps the directory's @@Zope subdirectory, but not other
+ files/directories that might still exist.
+
+ It doesn't deal with extras and annotations for the directory
+ itself, though.
+ """
+ lentry = self.metadata.getentry(localdir)
+ lentry.clear()
+ localzopedir = join(localdir, "@@Zope")
+ if os.path.isdir(localzopedir):
+ shutil.rmtree(localzopedir)
+ try:
+ os.rmdir(localdir)
+ except os.error:
+ self.reportdir("?", localdir)
+ else:
+ self.reportdir("D", localdir)
def reportdir(self, letter, localdir):
"""Helper to report something for a directory.
=== Zope3/src/zope/fssync/fssync.py 1.24 => 1.24.2.1 ===
--- Zope3/src/zope/fssync/fssync.py:1.24 Tue May 20 15:09:15 2003
+++ Zope3/src/zope/fssync/fssync.py Sun Jun 22 10:23:42 2003
@@ -27,7 +27,6 @@
import filecmp
import htmllib
import httplib
-import commands
import tempfile
import urlparse
import formatter
@@ -38,6 +37,8 @@
from os.path import dirname, basename, split, join
from os.path import realpath, normcase, normpath
+from zope.xmlpickle import dumps
+
from zope.fssync.metadata import Metadata
from zope.fssync.fsmerger import FSMerger
from zope.fssync.fsutil import Error
@@ -158,20 +159,22 @@
finally:
f.close()
- def httpreq(self, path, view, datafp=None,
- content_type="application/x-snarf"):
+ def httpreq(self, path, view, datasource=None,
+ content_type="application/x-snarf",
+ expected_type="application/x-snarf"):
"""Issue an HTTP or HTTPS request.
The request parameters are taken from the root url, except
that the requested path is constructed by concatenating the
path and view arguments.
- If the optional 'datafp' argument is not None, it should be a
- seekable stream from which the input document for the request
- is taken. In this case, a POST request is issued, and the
- content-type header is set to the 'content_type' argument,
- defaulting to 'application/x-snarf'. Otherwise (if datafp is
- None), a GET request is issued and no input document is sent.
+ If the optional 'datasource' argument is not None, it should
+ be a callable with a stream argument which, when called,
+ writes data to the stream. In this case, a POST request is
+ issued, and the content-type header is set to the
+ 'content_type' argument, defaulting to 'application/x-snarf'.
+ Otherwise (if datasource is None), a GET request is issued and
+ no input document is sent.
If the request succeeds and returns a document whose
content-type is 'application/x-snarf', the return value is a tuple
@@ -196,42 +199,33 @@
path += "/"
path += view
if self.roottype == "https":
- h = httplib.HTTPS(self.host_port)
+ conn = httplib.HTTPSConnection(self.host_port)
else:
- h = httplib.HTTP(self.host_port)
- if datafp is None:
- h.putrequest("GET", path)
- filesize = 0 # for PyChecker
+ conn = httplib.HTTPConnection(self.host_port)
+ if datasource is None:
+ conn.putrequest("GET", path)
else:
- datafp.seek(0, 2)
- filesize = datafp.tell()
- datafp.seek(0)
- h.putrequest("POST", path)
- h.putheader("Content-type", content_type)
- h.putheader("Content-length", str(filesize))
+ conn.putrequest("POST", path)
+ conn.putheader("Content-type", content_type)
+ conn.putheader("Transfer-encoding", "chunked")
if self.user_passwd:
auth = base64.encodestring(self.user_passwd).strip()
- h.putheader('Authorization', 'Basic %s' % auth)
- h.putheader("Host", self.host_port)
- h.endheaders()
- if datafp is not None:
- nbytes = 0
- while True:
- buf = datafp.read(8192)
- if not buf:
- break
- nbytes += len(buf)
- h.send(buf)
- assert nbytes == filesize
- errcode, errmsg, headers = h.getreply()
- fp = h.getfile()
- if errcode != 200:
+ conn.putheader('Authorization', 'Basic %s' % auth)
+ conn.putheader("Host", self.host_port)
+ conn.putheader("Connection", "close")
+ conn.endheaders()
+ if datasource is not None:
+ datasource(PretendStream(conn))
+ conn.send("0\r\n\r\n")
+ response = conn.getresponse()
+ if response.status != 200:
raise Error("HTTP error %s (%s); error document:\n%s",
- errcode, errmsg,
- self.slurptext(fp, headers))
- if headers["Content-type"] != "application/x-snarf":
- raise Error(self.slurptext(fp, headers))
- return fp, headers
+ response.status, response.reason,
+ self.slurptext(response.fp, response.msg))
+ elif expected_type and response.msg["Content-type"] != expected_type:
+ raise Error(self.slurptext(response.fp, response.msg))
+ else:
+ return response.fp, response.msg
def slurptext(self, fp, headers):
"""Helper to read the result document.
@@ -253,6 +247,30 @@
return data.strip()
return "Content-type: %s" % ctype
+class PretendStream(object):
+
+ """Helper class to turn writes into chunked sends."""
+
+ def __init__(self, conn):
+ self.conn = conn
+
+ def write(self, s):
+ self.conn.send("%x\r\n" % len(s))
+ self.conn.send(s)
+
+class DataSource(object):
+
+ """Helper class to provide a data source for httpreq."""
+
+ def __init__(self, head, tail):
+ self.head = head
+ self.tail = tail
+
+ def __call__(self, f):
+ snf = Snarfer(f)
+ snf.add(join(self.head, self.tail), self.tail)
+ snf.addtree(join(self.head, "@@Zope"), "@@Zope/")
+
class FSSync(object):
def __init__(self, metadata=None, network=None, rooturl=None):
@@ -298,36 +316,49 @@
for name in names:
method(join(target, name), *more)
- def commit(self, target, note="fssync"):
+ def commit(self, target, note="fssync_commit", raise_on_conflicts=False):
entry = self.metadata.getentry(target)
if not entry:
raise Error("nothing known about", target)
self.network.loadrooturl(target)
path = entry["path"]
- snarffile = tempfile.mktemp(".snf")
+ view = "@@fromFS.snarf?note=%s" % urllib.quote(note)
+ if raise_on_conflicts:
+ view += "&raise=1"
head, tail = split(realpath(target))
+ data = DataSource(head, tail)
+ fp, headers = self.network.httpreq(path, view, data)
try:
- f = open(snarffile, "wb")
- try:
- snf = Snarfer(f)
- snf.add(join(head, tail), tail)
- snf.addtree(join(head, "@@Zope"), "@@Zope/")
- finally:
- f.close()
- infp = open(snarffile, "rb")
- view = "@@fromFS.snarf?note=%s" % urllib.quote(note)
- try:
- outfp, headers = self.network.httpreq(path, view, infp)
- finally:
- infp.close()
- finally:
- pass
- if isfile(snarffile):
- os.remove(snarffile)
- try:
- self.merge_snarffile(outfp, head, tail)
+ self.merge_snarffile(fp, head, tail)
finally:
- outfp.close()
+ fp.close()
+
+ def checkin(self, target, note="fssync_checkin"):
+ rootpath = self.network.rootpath
+ if not rootpath:
+ raise Error("root url not set")
+ if rootpath == "/":
+ raise Error("root url should name an inferior object")
+ i = rootpath.rfind("/")
+ path, name = rootpath[:i], rootpath[i+1:]
+ if not path:
+ path = "/"
+ if not name:
+ raise Error("root url should not end in '/'")
+ entry = self.metadata.getentry(target)
+ if not entry:
+ raise Error("nothing known about", target)
+ qnote = urllib.quote(note)
+ qname = urllib.quote(name)
+ head, tail = split(realpath(target))
+ qsrc = urllib.quote(tail)
+ view = "@@checkin.snarf?note=%s&name=%s&src=%s" % (qnote, qname, qsrc)
+ data = DataSource(head, tail)
+ fp, headers = self.network.httpreq(path, view, data,
+ expected_type=None)
+ message = self.network.slurptext(fp, headers)
+ if message:
+ print message
def update(self, target):
entry = self.metadata.getentry(target)
@@ -378,8 +409,8 @@
return
print "Index:", target
sys.stdout.flush()
- os.system("diff %s %s %s" %
- (diffopts, commands.mkarg(orig), commands.mkarg(target)))
+ cmd = ("diff %s %s %s" % (diffopts, quote(orig), quote(target)))
+ os.system(cmd)
def dirdiff(self, target, mode=1, diffopts=""):
assert isdir(target)
@@ -390,7 +421,7 @@
if e and "flag" not in e:
self.diff(t, mode, diffopts)
- def add(self, path):
+ def add(self, path, type=None, factory=None):
if not exists(path):
raise Error("nothing known about '%s'", path)
entry = self.metadata.getentry(path)
@@ -408,9 +439,22 @@
zpath += tail
entry["path"] = zpath
entry["flag"] = "added"
- if isdir(path):
- entry["type"] = entry["factory"] = "zope.app.content.folder.Folder"
+ if type:
+ entry["type"] = type
+ if factory:
+ entry["factory"] = factory
self.metadata.flush()
+ if isdir(path):
+ # Force Entries.xml to exist, even if it wouldn't normally
+ zopedir = join(path, "@@Zope")
+ efile = join(zopedir, "Entries.xml")
+ if not exists(efile):
+ if not exists(zopedir):
+ os.makedirs(zopedir)
+ self.network.writefile(dumps({}), efile)
+ print "A", join(path, "")
+ else:
+ print "A", path
def remove(self, path):
if exists(path):
@@ -426,6 +470,7 @@
else:
entry["flag"] = "removed"
self.metadata.flush()
+ print "R", path
def status(self, target, descend_only=False):
entry = self.metadata.getentry(target)
@@ -495,3 +540,23 @@
extra = fsutil.getextra(target)
if isdir(extra):
self.status(extra, True)
+
+def quote(s):
+ """Helper to put quotes around arguments passed to shell if necessary."""
+ if os.name == "posix":
+ meta = "\\\"'*?[&|()<>`#$; \t\n"
+ else:
+ meta = " "
+ needquotes = False
+ for c in meta:
+ if c in s:
+ needquotes = True
+ break
+ if needquotes:
+ if os.name == "posix":
+ # use ' to quote, replace ' by '"'"'
+ s = "'" + s.replace("'", "'\"'\"'") + "'"
+ else:
+ # (Windows) use " to quote, replace " by ""
+ s = '"' + s.replace('"', '""') + '"'
+ return s
=== Zope3/src/zope/fssync/main.py 1.15 => 1.15.2.1 ===
--- Zope3/src/zope/fssync/main.py:1.15 Thu May 15 18:22:58 2003
+++ Zope3/src/zope/fssync/main.py Sun Jun 22 10:23:42 2003
@@ -23,6 +23,7 @@
fssync [global_options] status [local_options] [TARGET ...]
fssync [global_options] add [local_options] TARGET ...
fssync [global_options] remove [local_options] TARGET ...
+fssync [global_options] checkin [local_options] URL [TARGETDIR]
``fssync -h'' prints the global help (this message)
``fssync command -h'' prints the local help for the command
@@ -38,7 +39,7 @@
from os.path import dirname, join, realpath
# Find the zope root directory.
-# XXX This assumes this script is <root>/src/zope/fssync/sync.py
+# XXX This assumes this script is <root>/src/zope/fssync/main.py
scriptfile = sys.argv[0]
scriptdir = realpath(dirname(scriptfile))
rootdir = dirname(dirname(dirname(scriptdir)))
@@ -158,7 +159,7 @@
fs.checkout(target)
def commit(opts, args):
- """fssync commit [-m message] [TARGET ...]
+ """fssync commit [-m message] [-r] [TARGET ...]
Commit the TARGET files or directories to the Zope 3 server
identified by the checkout command. TARGET defaults to the
@@ -171,12 +172,15 @@
The -m option specifies a message to label the transaction.
The default message is 'fssync'.
"""
- message = "fssync"
+ message = "fssync_commit"
+ raise_on_conflicts = False
for o, a in opts:
if o in ("-m", "--message"):
message = a
+ if o in ("-r", "--raise-on-conflicts"):
+ raise_on_conflicts = True
fs = FSSync()
- fs.multiple(args, fs.commit, message)
+ fs.multiple(args, fs.commit, message, raise_on_conflicts)
def update(opts, args):
"""fssync update [TARGET ...]
@@ -193,15 +197,35 @@
fs.multiple(args, fs.update)
def add(opts, args):
- """fssync add TARGET ...
+ """fssync add [-t TYPE] [-f FACTORY] TARGET ...
Add the TARGET files or directories to the set of registered
objects. Each TARGET must exist. The next commit will add them
to the Zope 3 server.
+
+ The options -t and -f can be used to set the type and factory of
+ the newly created object; these should be dotted names of Python
+ objects. Usually only the factory needs to be specified.
+
+ If no factory is specified, the type will be guessed when the
+ object is inserted into the Zope 3 server based on the filename
+ extension and the contents of the data. For example, some common
+ image types are recognized by their contents, and the extensions
+ .pt and .dtml are used to create page templates and DTML
+ templates, respectively.
"""
+ type = None
+ factory = None
+ for o, a in opts:
+ if o in ("-t", "--type"):
+ type = a
+ elif o in ("-f", "--factory"):
+ factory = a
+ if not args:
+ raise Usage("add requires at least one TARGET argument")
fs = FSSync()
for a in args:
- fs.add(a)
+ fs.add(a, type, factory)
def remove(opts, args):
"""fssync remove TARGET ...
@@ -210,6 +234,8 @@
objects. No TARGET must exist. The next commit will remove them
from the Zope 3 server.
"""
+ if not args:
+ raise Usage("remove requires at least one TARGET argument")
fs = FSSync()
for a in args:
fs.remove(a)
@@ -254,17 +280,50 @@
fs = FSSync()
fs.multiple(args, fs.status)
+def checkin(opts, args):
+ """checkin [-m message] URL [TARGETDIR]
+
+ URL should be of the form ``http://user:password@host:port/path''.
+ Only http and https are supported (and https only where Python has
+ been built to support SSL). This should identify a Zope 3 server;
+ user:password should have management privileges; /path should be
+ the traversal path to a non-existing object, not including views
+ or skins.
+
+ TARGETDIR should be a directory; it defaults to the current
+ directory. The object tree rooted at TARGETDIR is copied to
+ /path. subdirectory of TARGETDIR whose name is the last component
+ of /path.
+ """
+ message = "fssync_checkin"
+ for o, a in opts:
+ if o in ("-m", "--message"):
+ message = a
+ if not args:
+ raise Usage("checkin requires a URL argument")
+ rooturl = args[0]
+ if len(args) > 1:
+ target = args[1]
+ if len(args) > 2:
+ raise Usage("checkin requires at most one TARGETDIR argument")
+ else:
+ target = os.curdir
+ fs = FSSync(rooturl=rooturl)
+ fs.checkin(target, message)
+
command_table = {
"checkout": ("", [], checkout),
"co": ("", [], checkout),
"update": ("", [], update),
- "commit": ("m:", ["message="], commit),
- "add": ("", [], add),
+ "commit": ("m:r", ["message=", "raise-on-conflicts"], commit),
+ "add": ("f:t:", ["factory=", "type="], add),
"remove": ("", [], remove),
"rm": ("", [], remove),
"r": ("", [], remove),
"diff": ("bBcC:iuU:", ["brief", "context=", "unified="], diff),
"status": ("", [], status),
+ "checkin": ("m:", ["message="], checkin),
+ "ci": ("m:", ["message="], checkin),
}
if __name__ == "__main__":
=== Zope3/src/zope/fssync/merger.py 1.8 => 1.8.2.1 ===
--- Zope3/src/zope/fssync/merger.py:1.8 Wed May 14 18:24:42 2003
+++ Zope3/src/zope/fssync/merger.py Sun Jun 22 10:23:42 2003
@@ -204,6 +204,10 @@
lmeta = self.getentry(local)
rmeta = self.getentry(remote)
+ # Special-case sticky conflict
+ if "conflict" in lmeta:
+ return ("Nothing", "Conflict")
+
# Sort out cases involving additions or removals
if not lmeta and not rmeta:
@@ -279,7 +283,8 @@
def cmpfile(self, file1, file2):
"""Helper to compare two files.
- Return True iff the files are equal.
+ Return True iff the files both exist and are equal.
"""
- # XXX What should this do when either file doesn't exist?
+ if not (isfile(file1) and isfile(file2)):
+ return False
return filecmp.cmp(file1, file2, shallow=False)
=== Zope3/src/zope/fssync/metadata.py 1.2 => 1.2.2.1 ===
--- Zope3/src/zope/fssync/metadata.py:1.2 Tue May 13 12:15:21 2003
+++ Zope3/src/zope/fssync/metadata.py Sun Jun 22 10:23:42 2003
@@ -106,22 +106,16 @@
def flushkey(self, key):
entries = self.cache[key]
- todelete = [name for name, entry in entries.iteritems() if not entry]
- for name in todelete:
- del entries[name]
- if entries != self.originals[key]:
+ # Make a copy containing only the "live" (non-empty) entries
+ live = {}
+ for name, entry in entries.iteritems():
+ if entry:
+ live[name] = entry
+ if live != self.originals[key]:
zdir = join(key, "@@Zope")
efile = join(zdir, "Entries.xml")
- if not entries:
- if isfile(efile):
- os.remove(efile)
- if exists(zdir):
- try:
- os.rmdir(zdir)
- except os.error:
- pass
- else:
- data = dumps(entries)
+ if exists(efile) or live:
+ data = dumps(live)
if not exists(zdir):
os.makedirs(zdir)
f = open(efile, "w")
@@ -129,4 +123,4 @@
f.write(data)
finally:
f.close()
- self.originals[key] = copy.deepcopy(entries)
+ self.originals[key] = copy.deepcopy(live)
=== Zope3/src/zope/fssync/snarf.py 1.1 => 1.1.2.1 ===
--- Zope3/src/zope/fssync/snarf.py:1.1 Tue May 20 15:09:15 2003
+++ Zope3/src/zope/fssync/snarf.py Sun Jun 22 10:23:42 2003
@@ -16,13 +16,12 @@
This is for transferring collections of files over HTTP where the key
need is for simple software.
-The format is as follows:
+The format is dead simple: each file is represented by the string
-- for directory entries:
- '/ <pathname>\n'
+ '<size> <pathname>\n'
-- for file entries:
- '<size> <pathname>\n' followed by exactly <size> bytes
+followed by exactly <size> bytes. Directories are not represented
+explicitly.
Pathnames are always relative and always use '/' for delimiters, and
should not use '.' or '..' or '' as components. All files are read
@@ -58,6 +57,7 @@
def filter(fspath):
return True
names = os.listdir(root)
+ names.sort()
for name in names:
fspath = os.path.join(root, name)
if not filter(fspath):
@@ -112,16 +112,13 @@
if not infoline.endswith("\n"):
raise IOError("incomplete info line %r" % infoline)
infoline = infoline[:-1]
- what, path = infoline.split(" ", 1)
- if what == "/":
- self.makedir(path)
- else:
- size = int(what)
- f = self.createfile(path)
- try:
- copybytes(size, self.istr, f)
- finally:
- f.close()
+ sizestr, path = infoline.split(" ", 1)
+ size = int(sizestr)
+ f = self.createfile(path)
+ try:
+ copybytes(size, self.istr, f)
+ finally:
+ f.close()
def makedir(self, path):
fspath = self.translatepath(path)
=== Removed File Zope3/src/zope/fssync/compare.py ===