I am most interested and will check out your efforts.
You are welcome to join as a member on the Plone4Artists site. There is also a newsletter and mailing list to make it easier to stay in touch with the developments. http://plone4artists.jazkarta.com
my wiki address (when it is up) is 192.117.110.160:8080/nigunim/thewiki
I checked out the wiki, and it looks like you have a very ambitious project indeed!
My problem is that the audio is only one aspect of the picture and I need to keep the metadata in sync with, notation and information.
I am currently leaning towards keeping the metadata outside the actual ogg files and having it added when the playlists are created.
Why not keep the metadata in the OGG files themselves. It's in the OGG spec, so you might as well take advantage of it!
First people will search the metadata and then reaquest to hear the songs.
You can store the metadata in the Zope database as well, so that it is searchable.
I unfortuately got all my reponses off list.
And I can see that this message you sent to me was also sent privately instead of to the list. You should really use "Reply to All" in your email client, so that BOTH the list and the recipient get a copy of your msg.
Some suggested a way to use the Zope db for large files.
Yes, Sergey Volobuev mentioned that Zope 2.7.1 will have some improvements in this area.
I am still so new to zope that I am still unsure what will provide me with the best solution.
You have some pretty special requirements in that you need to keep metadata in sync with audio files and notation files. I would strongly recommend that you have a look at Archetypes. It gives you a lot of flexibility in this area. You could for example extend the ATAudio product to be referenceable to a ATNotation type. In this way, you can upload your Audio files with all metadata intact, and then "link" this audio file to a corresponding Notation file that you have also uploaded. Using WebDAV, you can get these files into your site very quickly by drag-n-drop, and you won't have to enter all the metadata into your site because it will already be embedded in the OGG files. As for the notation, I know that Andrew Kuchling from the docutils project wrote a directive for the ABC music language. http://sourceforge.net/mailarchive/message.php?msg_id=7625994 Michael Hauser is working on getting this to work in a Zope/Plone site, which means you could upload a notation file, and it would be rendered in the site as a PNG image. It looks like the only dependencies besides docutils are abcm2ps, Ghostscript and PIL. Now to reply to your comments...
Have you seen the audio products? In what way will they help me?
CMFAudio, written by Marc Bowery is fine for small MP3 files,
I have small ogg files but lots of them
If they are small and Zope 2.7.1 is able to stream them sufficiently, you may not even want to bother with storing them on the file system. There are some advantages to having all your data in the ZODB.
want to workflow, edit and search for them.
but not for the general public only a small group of developers
You can define these permissions however you want.
The cool thing is that we've implemented an ID3Storage layer which acts as an interface for reading/writing the ID3 tags from/to the MP3 files This was done using pyid3lib (http://pyid3lib.sourceforge.net).
This saves you from doing a lot of data-entry, especially if your audio files already have metadata embedded in them.
In either case the metadata is yet to be added. The question is whether it should reside withing the ogg files or the notation or ?.
Well, that depends on your use cases. You could also make an abstract "Song" type which contains ONLY metadata but has references to the actual audio and notation files. Or if you don't want to bother with Archetypes, you could make this "Song" type folderish, so it could be a container for the audio and notation files. In my opinion, this would be the simplest way to do it.
I also have the language issue where I need to serve metadata in multiple languages (hebrew and english)
Ugh.. Then that complicates things a bit. Currently, the only solution that I know of is I18NLayer, but it gets very messy. If you can wait a month, there will be a new product called LinguaPlone which makes handling multilingual content much easier. However, I think it requires that your content types are Archetypes-based since it relies on the new Archetypes references engine.
Unless you don't even use Zope to serve the file, but simply provide a pointer to the URL where another 3rd party tool (such as Apache, Icecast, Shoutcast, Edna, etc.) actually serves up the file.
sounds interesting
But again, if your OGG files are small, you may not want to complicate matters with 3rd party streaming tools. On the other hand, your solution will be more scalable if you store your files outside of the ZODB because then you can use any streaming server and easily change to a better one if the need arises.
Well I think the largest ogg file will be 9 megs. Certainly not 30- 50 meg.
9 Meg is not what I would consider "small". If you are going to have many of these large files, I would definitely consider storing them on the file system instead of the ZODB. Or else you will quickly find yourself with ZODB bloat.
This is how we do it with ATAudio. When you click on the 'Play' icon, it auto-generates an .m3u file which points to the MP3 file being served from your Apache server.
Also ogg?
I answered this in the last mail. No ogg support yet, but it's coming...
Jens Klein (jensens) is working on providing support for OGG files, using the pyOGG libraries. http://www.andrewchatham.com/pyogg/
I am not planning to run apache in addition to zope. I haven't looked at ZEO and I will do that soon I guess.
Why aren't you planning to run Apache? It's pretty foolish to run Zope 'in the wild' without Apache in front of it. Also, Apache (esp. with mod_mp3 or Apache::MP3) makes an excellent server for serving up those large MP3/OGG files.
Well I have seen those who use it in the wild as you say and just use a lot of cache.
Well, the Zope caching is not very robust and eats up a lot of RAM. You can reduce the hit on your Zope server a lot by using Apache or Squid caching in front of Zope. Also, using Apache is not only for performance reasons, but also for security reasons.
I may use apache etc. but for development I am working locally. I am trying presently to avoid learning another thing, at least until I understand Zope enough to get off the ground.
It's fine while you're developing the site, but once you launch it, you should really use Apache. It's not very difficult to set up. Just a few lines in your Apache's httpd.conf file (VirtualHost directive).
If anyone is willing/interested in giving some feedback about how to best use zope for my project, My Wiki explains the project details and design issues.
What is the URL to your wiki? Mine is http://plone4artists.jazkarta.com/development/Audio
The ATAudio product is still a little rough around the edges, but you are welcome to join in the development efforts. I will take a look thanks.
The bugs/issues collector for ATAudio describes the known issues and features which are planned. http://tinyurl.com/37u7k
You can also read the TODO.txt if you grab the product from the collective CVS. http://sf.net/projects/collective
I'm looking forward to this talk in a few weeks at the EuroPython conference: "The Railroad Project: Managing Large files Around Zope" http://www.europython.org/conferences/epc2004/info/talks/zope/kitblake01
Nate
Thanks Aaron
I checked out the wiki, and it looks like you have a very ambitious project indeed! And I as I see it I keep getting further and further away from completion.
My problem is that the audio is only one aspect of the picture and I need to keep the metadata in sync with, notation and information.
I am currently leaning towards keeping the metadata outside the actual ogg files and having it added when the playlists are created.
Why not keep the metadata in the OGG files themselves. It's in the OGG spec, so you might as well take advantage of it!
My ogg files are conversions from casette/reel-to-reel tapes that I or another collaborator has created. The metadata for the most part either doesn't exist, or is in print and must be added later. I see no problem with storing the metadata in in ogg file and having it added to fields later, but what about notation files that have metadata but no recording exists?
First people will search the metadata and then reaquest to hear the songs.
You can store the metadata in the Zope database as well, so that it is searchable.
I unfortuately got all my reponses off list.
Yes because you answered me off-list. This is a mistake as if these mails are lost I won't be able to retreive them from the archives and others won't benefit from them. We should somehow get this discussion on the list again. I will cc. this to the list.
And I can see that this message you sent to me was also sent privately instead of to the list. You should really use "Reply to All" in your email client, so that BOTH the list and the recipient get a copy of your msg.
Some suggested a way to use the Zope db for large files.
Yes, Sergey Volobuev mentioned that Zope 2.7.1 will have some improvements in this area.
I am still so new to zope that I am still unsure what will provide me with the best solution.
You have some pretty special requirements in that you need to keep metadata in sync with audio files and notation files. I would strongly recommend that you have a look at Archetypes. It gives you a lot of flexibility in this area.
Truthfully, I see CMF products and Archetype products and can't figure out what they are/do? I don't see them in the Zope Book....
You could for example extend the ATAudio product to be referenceable to a ATNotation type. In this way, you can upload your Audio files with all metadata intact, and then "link" this audio file to a corresponding Notation file that you have also uploaded.
Using WebDAV, you can get these files into your site very quickly by drag-n-drop, and you won't have to enter all the metadata into your site because it will already be embedded in the OGG files.
Again I have no metadata in the ogg files yet.
As for the notation, I know that Andrew Kuchling from the docutils project wrote a directive for the ABC music language. http://sourceforge.net/mailarchive/message.php?msg_id=7625994
Michael Hauser is working on getting this to work in a Zope/Plone site, which means you could upload a notation file, and it would be rendered in the site as a PNG image. It looks like the only dependencies besides docutils are abcm2ps, Ghostscript and PIL.
hmm this sounds interesting, unfortunetly lilypond relies on Latex although in version 3. they plan also to output postscript only.
Now to reply to your comments...
Have you seen the audio products? In what way will they help me?
CMFAudio, written by Marc Bowery is fine for small MP3 files,
I have small ogg files but lots of them
If they are small and Zope 2.7.1 is able to stream them sufficiently, you may not even want to bother with storing them on the file system. There are some advantages to having all your data in the ZODB.
want to workflow, edit and search for them.
but not for the general public only a small group of developers
You can define these permissions however you want.
The cool thing is that we've implemented an ID3Storage layer which acts as an interface for reading/writing the ID3 tags from/to the MP3 files This was done using pyid3lib (http://pyid3lib.sourceforge.net).
Yes I has the id3lib but it is only mp3...
This saves you from doing a lot of data-entry, especially if your audio files already have metadata embedded in them.
In either case the metadata is yet to be added. The question is whether it should reside withing the ogg files or the notation or ?.
Well, that depends on your use cases. You could also make an abstract "Song" type which contains ONLY metadata but has references to the actual audio and notation files. Thats what I have so far although I can't figure out how to make it fly.
I have folders nigunim/notation/audio/ my idea was to have the metadata for each nigun(song) reside in the highest level nigun folder and then have only the ogg specific metadata reside in the ogg folder, and only the notation specific metadata reside in the notation folder, a so forth. My thought was that since no metadata exists at least I should make it as easy as possible to add the metadata. Once the metadata is added for the nigun, only the specific metadata for the recording will need to be added, and in my case there could be many ogg files for each notation file. (the same song perfomed by different singers)
Or if you don't want to bother with Archetypes, you could make this "Song" type folderish, so it could be a container for the audio and notation files. In my opinion, this would be the simplest way to do it.
So it seems.
I also have the language issue where I need to serve metadata in multiple languages (hebrew and english)
Ugh.. Then that complicates things a bit. Currently, the only solution that I know of is I18NLayer, but it gets very messy. If you can wait a month, there will be a new product called LinguaPlone which makes handling multilingual content much easier. However, I think it requires that your content types are Archetypes-based since it relies on the new Archetypes references engine.
Ok, but for the beginning I will content myself with english.
Unless you don't even use Zope to serve the file, but simply provide a pointer to the URL where another 3rd party tool (such as Apache, Icecast, Shoutcast, Edna, etc.) actually serves up the file.
sounds interesting
But again, if your OGG files are small, you may not want to complicate matters with 3rd party streaming tools. On the other hand, your solution will be more scalable if you store your files outside of the ZODB because then you can use any streaming server and easily change to a better one if the need arises.
Well I think the largest ogg file will be 9 megs. Certainly not 30- 50 meg.
9 Meg is not what I would consider "small". If you are going to have many of these large files, I would definitely consider storing them on the file system instead of the ZODB. Or else you will quickly find yourself with ZODB bloat.
This is how we do it with ATAudio. When you click on the 'Play' icon, it auto-generates an .m3u file which points to the MP3 file being served from your Apache server.
Also ogg?
I answered this in the last mail. No ogg support yet, but it's coming...
Jens Klein (jensens) is working on providing support for OGG files, using the pyOGG libraries. http://www.andrewchatham.com/pyogg/
I am not planning to run apache in addition to zope. I haven't looked at ZEO and I will do that soon I guess.
Why aren't you planning to run Apache? It's pretty foolish to run Zope 'in the wild' without Apache in front of it. Also, Apache (esp. with mod_mp3 or Apache::MP3) makes an excellent server for serving up those large MP3/OGG files.
Well I have seen those who use it in the wild as you say and just use a lot of cache.
Well, the Zope caching is not very robust and eats up a lot of RAM. You can reduce the hit on your Zope server a lot by using Apache or Squid caching in front of Zope. Also, using Apache is not only for performance reasons, but also for security reasons.
I may use apache etc. but for development I am working locally. I am trying presently to avoid learning another thing, at least until I understand Zope enough to get off the ground.
It's fine while you're developing the site, but once you launch it, you should really use Apache. It's not very difficult to set up. Just a few lines in your Apache's httpd.conf file (VirtualHost directive).
If anyone is willing/interested in giving some feedback about how to best use zope for my project, My Wiki explains the project details and design issues.
What is the URL to your wiki? Mine is http://plone4artists.jazkarta.com/development/Audio
The ATAudio product is still a little rough around the edges, but you are welcome to join in the development efforts. I will take a look thanks.
The bugs/issues collector for ATAudio describes the known issues and features which are planned. http://tinyurl.com/37u7k
You can also read the TODO.txt if you grab the product from the collective CVS. http://sf.net/projects/collective
I grabbed the product, and thought to give it a whirl but I got a show stopping error so I removed it. I will read the TODO.txt though.
I'm looking forward to this talk in a few weeks at the EuroPython conference: "The Railroad Project: Managing Large files Around Zope" http://www.europython.org/conferences/epc2004/info/talks/zope/kitblake01
Whew so much to learn and so little time, I was hoping that Zope would speed up the process, but I am starting to be a bit overwhelmed.
Any additional insights you can offer, would be most appreciated Aaron
Nate
Thanks Aaron
On Thu, May 27, 2004 at 06:37:17AM +0300, Aaron wrote:
Truthfully, I see CMF products and Archetype products and can't figure out what they are/do? I don't see them in the Zope Book....
They're not in there. There are a lot of layers at work here: The Zope Book only documents zope itself. CMF is a framework for building content management apps that is implemented on top of Zope. There aren't really any good docs about CMF :-( (Well, there are docs about bits of it.. there's a good DCWorkflow pdf doc around.) Plone is a content management app implemented for CMF. Good Plone docs, but a bit out of date: http://plone.org/documentation/book/1 Also check out the plone-users mailing list and the #plone IRC channel, both will help you immensely. Archetypes is a content-type development framework for Plone. http://www.zopemag.com/Issue006/Section_Articles/article_IntroToArchteypes.h...
Whew so much to learn and so little time, I was hoping that Zope would speed up the process, but I am starting to be a bit overwhelmed.
Yes, unfortunately there is a lot to learn. But for something as sophisticated as you describe, that's hard to avoid. -- Paul Winkler http://www.slinkp.com
-- Nate Aune - natea@jazkarta.com Plone4Artists - http://plone4artists.jazkarta.com "Build your own artist community website!"
I see no problem with storing the metadata in in ogg file and having it added to fields later, but what about notation files that have metadata but no recording exists?
Then maybe it would be best to have an abstract song folderish content type which *only* contains the metadata, and contained within this folder you could have the actual audio and notation files.
Yes because you answered me off-list. This is a mistake as if these mails are lost I won't be able to retreive them from the archives and others won't benefit from them.
We should somehow get this discussion on the list again. I will cc. this to the list.
That's strange - I thought for sure that I replied to the list as well. I'm pretty sure I did because otherwise you couldn't have gotten my mail. When I sent it to your address, I got this msg back: I'm sorry but your email with subject [SPAM] [Zope] Re: audio order addressed to "Aaron" <aamehl@bezeqint.net> may not have been delivered because forbidden keyword check out As part of our ongoing battle against SPAM, Insane Technologies (www.insane.net.au) uses a combination of registered ban lists (RBL) and keywords. If you feel your communication has been blocked inadvertantly by this process, please contact us via phone on +61 7 5539 6116 or fax on +61 7 5539 6130
Truthfully, I see CMF products and Archetype products and can't figure out what they are/do? I don't see them in the Zope Book....
Nothing about CMF? Archetypes is a relatively new thing, so it wouldn't be in the Zope book. Here are a few links where you can read more: http://plone.org/documentation/archetypes http://www.zopemag.com/Issue006/Section_Articles/article_IntroToArchteypes.h tml http://plone.org/documentation/archetypes/wiki/UpdatedContentTypesHelp http://www.neuroinf.de/LabTools/MySite/
hmm this sounds interesting, unfortunetly lilypond relies on Latex although in version 3. they plan also to output postscript only.
Well, if you have it in postscript, you can convert it to PNG or PDF.
Well, that depends on your use cases. You could also make an abstract "Song" type which contains ONLY metadata but has references to the actual audio and notation files. Thats what I have so far although I can't figure out how to make it fly.
I have folders nigunim/notation/audio/ my idea was to have the metadata for each nigun(song) reside in the highest level nigun folder and then have only the ogg specific metadata reside in the ogg folder, and only the notation specific metadata reside in the notation folder, a so forth.
My thought was that since no metadata exists at least I should make it as easy as possible to add the metadata. Once the metadata is added for the nigun, only the specific metadata for the recording will need to be added, and in my case there could be many ogg files for each notation file. (the same song perfomed by different singers)
This sounds like a perfect use case for using Archetypes, because you can define some BaseSchema (all the metadata about the song which is common to the audio and notation files - song title, composer, etc.) Then the audio and notation files could inherit these properties, so you wouldn't have to add them again. You would only have to add the metadata that is specific to the audio file (when it was recorded, who was it performed by, etc.) and notation file.
Whew so much to learn and so little time, I was hoping that Zope would speed up the process, but I am starting to be a bit overwhelmed.
Yes, I get overwhelmed sometimes too. There is a lot to learn with Zope/CMF/Plone/Python, but once you get a handle on it, it starts to come together. Nate
Hi I see also that the list rejects the name of a compressed audio file as spam m p 3... So reply to all was only going to us off list. At least their spam filter is working. I feel that after months of figuring out what I need to do to get my project off the ground I am once again going back to the drawing board. My next question is must I lean zpt or dhtml. Thanks so far Aaron
On Thu, May 27, 2004 at 01:45:25PM +0300, Aaron wrote:
My next question is must I lean zpt or dhtml.
++votes for zpt. (btw, zope uses dtml, not dhtml.) -- Paul Winkler http://www.slinkp.com
Hi I see also that the list rejects the name of a compressed audio file as spam m p 3...
Hmm.. Are you sure? It was my understanding that the rejection was happening from your mail host, not the list server.
I feel that after months of figuring out what I need to do to get my project off the ground I am once again going back to the drawing board.
I know the feeling. The best way to cure that is to just dig in and start making a prototype. You'll learn a lot this way, and can usually eliminate the uncertainty (or at least minimize it). Probably the fastest way to get up and running is to make a folderish Song content type, and then upload your audio and notation files into that folder. Using Archetypes, it's super easy to make these types in no time.
My next question is must I lean zpt or dhtml.
ZPT. Don't even mess with dtml since it's hardly ever used anymore. Here is a good tutorial for learning ZPT: http://zope.org/Documentation/Books/ZopeBook/2_6Edition/ZPT.stx Nate
thanks, I will do just that, and you have been more than helpful thanks Aaron On ה', 2004-05-27 at 23:35, Nate Aune wrote:
Hi I see also that the list rejects the name of a compressed audio file as spam m p 3...
Hmm.. Are you sure? It was my understanding that the rejection was happening from your mail host, not the list server. could be, I didn't pay that much attention. But I will avoid that dirty word from now on.
I feel that after months of figuring out what I need to do to get my project off the ground I am once again going back to the drawing board.
I know the feeling. The best way to cure that is to just dig in and start making a prototype. You'll learn a lot this way, and can usually eliminate the uncertainty (or at least minimize it).
Probably the fastest way to get up and running is to make a folderish Song content type, and then upload your audio and notation files into that folder. Using Archetypes, it's super easy to make these types in no time. I looked at the Archetypes article in Zope mag and saw a bunch of python. I gather that these are premade chunks to add. I will need to as you said dig in to get a feel for it.
My next question is must I lean zpt or dhtml.
ZPT. Don't even mess with dtml since it's hardly ever used anymore. Here is a good tutorial for learning ZPT: http://zope.org/Documentation/Books/ZopeBook/2_6Edition/ZPT.stx
I will take a look thanks Aaron
Nate
On Thu, 27 May 2004, Nate Aune wrote:
My next question is must I lean zpt or dhtml.
ZPT. Don't even mess with dtml since it's hardly ever used anymore. Here is a good tutorial for learning ZPT: http://zope.org/Documentation/Books/ZopeBook/2_6Edition/ZPT.stx
Nate
DTML is not dead! Nate, where'd you get the information that supports your pronouncement? We actually use both, but mostly we use DTML because it's a better match to what we do.
My next question is must I lean zpt or dhtml.
ZPT. Don't even mess with dtml since it's hardly ever used anymore. Here is a good tutorial for learning ZPT: http://zope.org/Documentation/Books/ZopeBook/2_6Edition/ZPT.stx
DTML is not dead! Nate, where'd you get the information that supports your pronouncement? We actually use both, but mostly we use DTML because it's a better match to what we do.
Oops.. I thought I might end up eating my words by posting such a statement on the Zope list. I didn't mean to suggest that it was dead, but that for new development in Plone, it is recommended to use ZPT in favor of DTML. I know that DTML is good for some things, but for someone just starting out trying to learn Zope, it's preferable to steer them towards ZPT combined with Python scripts. Nate
What is the URL to your wiki? Mine is http://plone4artists.jazkarta.com/development/Audio
The ATAudio product is still a little rough around the edges, but you are welcome to join in the development efforts. I will take a look thanks.
The bugs/issues collector for ATAudio describes the known issues and features which are planned. http://tinyurl.com/37u7k
You can also read the TODO.txt if you grab the product from the collective CVS. http://sf.net/projects/collective
I grabbed the product, and thought to give it a whirl but I got a show stopping error so I removed it. I will read the TODO.txt though.
Try grabbing the most recent one out of CVS. Please note that this version has a dependency on the pyid3lib python library for handling the ID3 tags, so you'll need to install that. Instructions are in the README-ID3.txt file. Let me know how it goes with the install! Nate -- Nate Aune - natea@jazkarta.com Plone4Artists - http://plone4artists.jazkarta.com "Build your own artist community website!"
Thanks, But it will be a while, I am stuck at the beginning of a lengthy learning curve. I am starting with the zope tutorial and somehow I will need to graduate to Archtypes. Part of the problem is the huge amount of choices and the newness of it all. Aaron On א', 2004-06-06 at 18:52, Nate Aune wrote:
What is the URL to your wiki? Mine is http://plone4artists.jazkarta.com/development/Audio
The ATAudio product is still a little rough around the edges, but you are welcome to join in the development efforts. I will take a look thanks.
The bugs/issues collector for ATAudio describes the known issues and features which are planned. http://tinyurl.com/37u7k
You can also read the TODO.txt if you grab the product from the collective CVS. http://sf.net/projects/collective
I grabbed the product, and thought to give it a whirl but I got a show stopping error so I removed it. I will read the TODO.txt though.
Try grabbing the most recent one out of CVS. Please note that this version has a dependency on the pyid3lib python library for handling the ID3 tags, so you'll need to install that. Instructions are in the README-ID3.txt file. Let me know how it goes with the install!
Nate
participants (4)
-
Aaron -
Dennis Allison -
Nate Aune -
Paul Winkler