Re: [Zope] Q: Zope2 version of "Contributions/DTML Example Folder"?

25 Jul 1999

      Jens,

doh!  Here's the FS which may need other stuff.  The install for
Zope2Beta1 was painless, so that's probably (at this point) the easier
way
to upgrade (and get the latest :-).  The beta is available at:

    http://www.zope.org/Download/Releases/Zope-2.0.0b1

Cheers,

= Joe =

Jens Vagelpohl wrote:
...
hi joe,
thanks a lot for providing that bbb.py. the scripts bbb.py and FS.py are
interrelated, however, and the new bbb.py you sent doesn't like the old
FS.py. now i'm getting this:
[root@diesel utilities]# python bbb.py -x -f export.zexp export.bbe
Traceback (innermost last):
  File "bbb.py", line 388, in ?
    if __name__=='__main__': main(sys.argv[1:])
  File "bbb.py", line 381, in main
    rpt=FS.ZEXP(v, file).rpt
AttributeError: ZEXP
hrmph :(
Jens Vagelpohl
...
-----Original Message-----
From: Joe Grace [mailto:occam@serv.net]
Sent: Sunday, July 25, 1999 13:38
To: Jens Vagelpohl; zope@zope.org
Subject: Re: [Zope] Q: Zope2 version of "Contributions/DTML Example
Folder"?
Jens,
if you don't see an -x switch, then there must be a difference.
I've attached
the bbb.py from Zope2b1 for your convenience.
= Joe =
Jens Vagelpohl wrote:
...
hm... is there some difference between the bbb.py on alpha3 and
beta1? i'm
trying to convert that dtmlExamples export.bbe as well and
first off i don't
see an -x switch and secondly i am getting an error message:
[root@diesel utilities]# python bbb.py -f export.zexp export.bbe
Corrupted data record at 9
Traceback (innermost last):
  File "bbb.py", line 409, in ?
    _read_and_report(file, rpt, fromEnd, both, n, show)
  File "bbb.py", line 218, in _read_and_report
    rpt(None, None, None, None, None, None, None, None, None)
  File "/usr/src/Zope-2.0.0a3-src/utilities/FS.py", line 110, in rpt
    self.tpc_finish()
  File "/usr/src/Zope-2.0.0a3-src/utilities/FS.py", line 173,
in tpc_finish
    id=self._serial
AttributeError: _serial
not sure what's happening here :(
Jens Vagelpohl
##############################################################################
# 
# Zope Public License (ZPL) Version 1.0
# -------------------------------------
# 
# Copyright (c) Digital Creations.  All rights reserved.
# 
# This license has been certified as Open Source(tm).
# 
# Redistribution and use in source and binary forms, with or without
# modification, are permitted provided that the following conditions are
# met:
# 
# 1. Redistributions in source code must retain the above copyright
#    notice, this list of conditions, and the following disclaimer.
# 
# 2. Redistributions in binary form must reproduce the above copyright
#    notice, this list of conditions, and the following disclaimer in
#    the documentation and/or other materials provided with the
#    distribution.
# 
# 3. Digital Creations requests that attribution be given to Zope
#    in any manner possible. Zope includes a "Powered by Zope"
#    button that is installed by default. While it is not a license
#    violation to remove this button, it is requested that the
#    attribution remain. A significant investment has been put
#    into Zope, and this effort will continue if the Zope community
#    continues to grow. This is one way to assure that growth.
# 
# 4. All advertising materials and documentation mentioning
#    features derived from or use of this software must display
#    the following acknowledgement:
# 
#      "This product includes software developed by Digital Creations
#      for use in the Z Object Publishing Environment
#      (http://www.zope.org/)."
# 
#    In the event that the product being advertised includes an
#    intact Zope distribution (with copyright and license included)
#    then this clause is waived.
# 
# 5. Names associated with Zope or Digital Creations must not be used to
#    endorse or promote products derived from this software without
#    prior written permission from Digital Creations.
# 
# 6. Modified redistributions of any form whatsoever must retain
#    the following acknowledgment:
# 
#      "This product includes software developed by Digital Creations
#      for use in the Z Object Publishing Environment
#      (http://www.zope.org/)."
# 
#    Intact (re-)distributions of any official Zope release do not
#    require an external acknowledgement.
# 
# 7. Modifications are encouraged but must be packaged separately as
#    patches to official Zope releases.  Distributions that do not
#    clearly separate the patches from the original work must be clearly
#    labeled as unofficial distributions.  Modifications which do not
#    carry the name Zope may be packaged in any form, as long as they
#    conform to all of the clauses above.
# 
# 
# Disclaimer
# 
#   THIS SOFTWARE IS PROVIDED BY DIGITAL CREATIONS ``AS IS'' AND ANY
#   EXPRESSED OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
#   IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
#   PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL DIGITAL CREATIONS OR ITS
#   CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
#   SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
#   LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF
#   USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
#   ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
#   OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT
#   OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF
#   SUCH DAMAGE.
# 
# 
# This software consists of contributions made by Digital Creations and
# many individuals on behalf of Digital Creations.  Specific
# attributions are listed in the accompanying credits file.
# 
##############################################################################

import struct, tempfile, string, time, pickle, os, sys
from struct import pack, unpack
from cStringIO import StringIO

class FS:
    """FileStorage 'report' writer for converting BoboPOS to FileStorage
    """

    def __init__(self, fname, file):
        file.seek(0,2)
        self._input_size=file.tell()
        file.seek(0)
        self.__name__=fname
        self._file=open(fname,'w+b')
        self._file.write('FS21')

        self._index={}
        self._indexpos=self._index.get
        self._tindex=[]
        self._tappend=self._tindex.append
        self._tfile=tempfile.TemporaryFile()
        self._pos=4
        self._progress=None

    def rpt(self, pos, oid, start, tname, user, t, p, first, newtrans):
        if pos is None:
            self.tpc_finish()
            sys.stderr.write(' %% 100     \n')
            return
        else:
            progress=' %% %.1f \r' % (pos*100.0/self._input_size)
            if progress != self._progress:
                sys.stderr.write(progress)
                self._progress=progress

        if newtrans:
            try: string.atof(tname)
            except:
                # Ugh, we have a weird tname.  We'll just ignore the transaction
                # boundary and merge transactions
                if first:
                    # But we can't ignore the first one, so we'll hack in a
                    # bogus start date
                    self.tpc_begin('100', user, t)
            else:
                if not first: self.tpc_finish()
                self.tpc_begin(tname, user, t)
        self.store(oid, p)

    def store(self, oid, data):
        old=self._indexpos(oid, 0)
        pnv=None

        tfile=self._tfile
        write=tfile.write
        pos=self._pos
        here=tfile.tell()+pos+self._thl
        self._tappend(oid, here)
        serial=self._serial
        data=fixpickle(data, oid)
        write(pack(">8s8s8s8sH8s",
                   p64(oid+1),serial,p64(old),p64(pos),
                   0,p64(len(data))
                   )
              )

        write(data)

    def tpc_begin(self, tname, user, desc):
        del self._tindex[:]   # Just to be sure!
        self._tfile.seek(0)

        t=string.atof(tname)
        y,m,d,h,mi=time.gmtime(t)[:5]
        s=t%60
        self._serial=struct.pack(
            ">II",
            (((((y-1900)*12)+m-1)*31+d-1)*24+h)*60+mi,
            long(s * (1L << 32) / 60)
            )

        # Ugh, we have to record the transaction header length
        # so that we can get version pointers right.
        self._thl=23+len(user)+len(desc)

        # And we have to save the data used to compute the
        # header length. It's unlikely that this stuff would
        # change, but if it did, it would be a disaster.
        self._ud=user, desc

    def tpc_finish(self):
        file=self._file
        write=file.write
        tfile=self._tfile
        dlen=tfile.tell()
        tfile.seek(0)
        id=self._serial
        user, desc = self._ud
        self._ud=None

        tlen=self._thl
        pos=self._pos
        tl=tlen+dlen
        stl=p64(tl)
        write(pack(
            ">8s" "8s" "c"  "H"        "H"        "H"
            ,id, stl, ' ', len(user), len(desc), 0,
            ))
        if user: write(user)
        if desc: write(desc)

        cp(tfile, file, dlen)

        write(stl)
        self._pos=pos+tl+8

        tindex=self._tindex
        index=self._index
        for oid, pos in tindex: index[oid]=pos
        del tindex[:]

class ZEXP:
    """Zope Export format 'report' writer
    """
    def __init__(self, fname, file):
        file.seek(0,2)
        self._input_size=file.tell()
        file.seek(0)
        self.__name__=fname
        self._file=open(fname,'w+b')
        self._file.write('ZEXP')

        self._pos=4
        self._progress=None

    def rpt(self, pos, oid, start, tname, user, t, p, first, newtrans):
        write=self._file.write
        if pos is None:
            write('\377'*16) # end marker
            sys.stderr.write(' %% 100     \n')
            return
        else:
            progress=' %% %.1f \r' % (pos*100.0/self._input_size)
            if progress != self._progress:
                sys.stderr.write(progress)
                self._progress=progress

        data=fixpickle(p, oid)
        l=len(data)
        write(p64(oid+1)+p64(l))
        write(data)
        self._pos=self._pos+l+16

t32 = 1L << 32

def p64(v, pack=struct.pack):
    if v < t32: h=0
    else:
        h=v/t32
        v=v%t32
    return pack(">II", h, v)

def cp(f1, f2, l):
    read=f1.read
    write=f2.write
    n=8192

    while l > 0:
        if n > l: n=l
        d=read(n)
        write(d)
        l = l - len(d)

class Ghost: pass

class Global:
    __safe_for_unpickling__=1

    def __init__(self, m, n):
        self._module, self._name = m, n

    def __call__(self, *args):
        return Inst(self, args)

    def __basicnew__(self):
        return Inst(self, None)

def _global(m, n):
    if m[:8]=='BoboPOS.':
        if m=='BoboPOS.PickleDictionary' and n=='Root':
            m='ZODB.conversionhack'
            n='hack'
        elif m=='BoboPOS.PersistentMapping': m='Persistence'
        elif m=='BoboPOS.cPickleJar' and n=='ec':
            m=n='ExtensionClass'
        else:
            raise 'Unexpected BoboPOS class', (m, n)
    elif m=='PersistentMapping': m='Persistence'

    return Global(m,n)

class Inst:

    _state=None

    def __init__(self, c, args):
        self._cls=c
        self._args=args

    def __setstate__(self, state): self._state=state

from pickle import INST, GLOBAL, MARK, BUILD, OBJ, REDUCE

InstanceType=type(Ghost())

class Unpickler(pickle.Unpickler):

    dispatch={}
    dispatch.update(pickle.Unpickler.dispatch)

    def load_inst(self):
        k = self.marker()
        args = tuple(self.stack[k+1:])
        del self.stack[k:]
        module = self.readline()[:-1]
        name = self.readline()[:-1]
        klass = _global(module, name)
        value=Inst(klass, args)
        self.append(value)

    dispatch[INST] = load_inst

    def load_global(self):
        module = self.readline()[:-1]
        name = self.readline()[:-1]
        klass = _global(module, name)
        self.append(klass)
    dispatch[GLOBAL] = load_global

    def persistent_load(self, oid,
                        TupleType=type(()), Ghost=Ghost, p64=p64):
        "Remap object ids from ZODB 2 stype to ZODB 3 style"

        if type(oid) is TupleType:
            oid, klass = oid
            oid = p64(oid+1), klass
        else:
            oid = p64(oid+1)

        Ghost=Ghost()
        Ghost.oid=oid
        return Ghost

class Pickler(pickle.Pickler):

    dispatch={}
    dispatch.update(pickle.Pickler.dispatch)

    def persistent_id(self, object, Ghost=Ghost):
        if hasattr(object, '__class__') and object.__class__ is Ghost:
            return object.oid

    def save_inst(self, object):
        d = id(object)
        cls = object.__class__

        memo  = self.memo
        write = self.write
        save  = self.save

        if cls is Global:
            memo_len = len(memo)
            write(GLOBAL + object._module + '\n' + object._name + '\n' +
                  self.put(memo_len))
            memo[d] = (memo_len, object)
            return

        self.save_inst(object._cls)
        save(object._args)

        memo_len = len(memo)
        write(REDUCE + self.put(memo_len))

        memo[d] = (memo_len, object)

        stuff=object._state
        save(stuff)
        write(BUILD)

    dispatch[InstanceType] = save_inst

    def save_global(self, object, name = None):
        write = self.write
        memo = self.memo

        if (name is None):
            name = object._name

        module=object._module

        memo_len = len(memo)
        write(GLOBAL + module + '\n' + name + '\n' +
            self.put(memo_len))
        memo[id(object)] = (memo_len, object)

    dispatch[type(Ghost)] = save_global

def fixpickle(p, oid):

    pfile=StringIO(p)
    unpickler=Unpickler(pfile)

    newp=StringIO()
    pickler=Pickler(newp,1)

    pickler.dump(unpickler.load())
    state=unpickler.load()
    if oid==-1: state={'_container': state}
    pickler.dump(state)
    p=newp.getvalue()

    return p