[ZODB-Dev] Scalable Container for Log-Entries

[Casey Duncan]

On Wed, 28 Jan 2004 10:44:01 +0100
Thomas Guettler <hv at tbz-pariv.de> wrote:

> Hi!
> 
> How would you code a scalable container which holds
> log-entries ordered by the time the data was inserted into
> the container.
> 
> There are several process which add log-entries (ZEO + Filestorage).
> The API is very simple: 
> 
> 1. logContainer.add(data)
> 2. data_list=logContainer.get(from_date, to_date)
> 
> I think of a BTree with the time as key. The time in seconds
> since 1970 is not enough because there will be several
> log entries per second. If you have an other solution, please
> let me know.
> 
> 
> I think a IOBTree would be good. But can a IOBTree
> handle long integers? 

No, an IOBTree allows short ints as keys only.
 
> I think these keys would be possible:
> 
> t=time.time()
> t2=time.time()
> print long(t *10000000)
> print long(t2*10000000)
> --> 10752829557540122
> --> 10752829557540250

The float value returned by time should be OK, but the resolution of it
may not be good enough in your application. The resolution varies by
platform AFAIK, but I'm not sure if this is a problem in your case.

You might consider using an OOTreeSet as well, which is like an OOBTree
except it only has keys. The keys could be a tuple of (time,
log_msg_object). You could do a range search to find entries for
specific times. You wouldn't have to worry about key collisions in that
case, but you would need to make sure the log message objects compared
stabily to one another.

To help with database bloat and contention, you might consider
collecting batches of log entries in memory and commiting them to the
database only every so often. The downside there is that you could loose
entries if the app dies (unless you also write them to a separate file).

You could also consider putting the log in a separate storage so that it
doesn't content with access to the main storage server. This could be
done later, however.

-Casey