collective.portlet.itemview

The idea is simple. Let Plone manage the content and the portlet be simply a small browser view.

Plone is content centric, but a web page need to display more than the current content. This is why I have created this add-on. The add form let you choose one content item (like in the collection portlet) and a view mode. The default view is covering all plone content type.

There are many examples to cover, to show you how it is simple to use and to add support for.
I have already add conditional support with collective.gallery. So now you can choose a content item to act as gallery in a portlet (link, folder or topic + ‘Gallery’ view in Itemview portlet). The same has been done on nmd.plonelinkasvideoembed, so you can publish a video from a link content item.

Comments and feed back are welcome !
This package is on pypi

Nantes-developpement.com a new Plone website by Makina-Corpus

I’m proud to announce it, the new web site of “The greater Nantes economic agency” is now available here: http://www.nantes-developpement.com .

This web site use Plone with some community add-ons (available on pypi). Some of these are new, mean they have been created to fit the customer needs, others has been contributed , mean they has been modified upstream, and last has just been used:

• collective.z3cform.norobots [new]
• collective.groupdashboard [used]
• collective.gallery [new]
• collective.dewplayer [new]
• collective.portlet.actions [used]
• collective.portlet.contact [new]
• collective.portlet.feedmixer [used]
• qi.portlet.TagClouds [contributed]
• collective.plonebookmarklets [contributed]
• collective.singing & collective.dancing [contributed]
• collective.contentlicensing [contributed]
• nmd.plonelinkasvideoembed [new]
• archetypes.uploadreferencewidget [contributed]
• babel [used]
• collective.autopermission [used]
• collective.contentrules.mail [used]
• collective.harlequin [new]
• collective.monkeypatcher [used]
• collective.z3cform.grok [new]
• collective.z3cform.norobots [new]
• Products.Maps [contributed]
• Products.PloneFormGen [contributed]
• Products.TinyMCE [contributed]
• Products.TextIndexNG3 [used]
• Products.CompoundField [contributed]
• Products.ZNagios [used]
• five.grok [used]
• experimental.contentcreation [used]
• experimental.aggressiveopaquespeedup [new]
• collective.remove.kss [new]
• collective.remove.kupu [new]
• collective.remove.rules [new]
• Products.CacheSetup [used]
• Products.MemcachedManager [used]
• plone.app.ldap [used]
• iw.fss [used]
• Relstorage [used]

Custom code contains:

• the theme done as a usual plonetheme (not with xdv).
• policy (integration of all community add-ons and plone configuration)
• portlets
• content-types (collection, folderish document, video/audio interview, …)

System architecture

We have apache2 in front end. Next we have varnish as proxy cache (but not load balancer). Configuration has been taken from Products.PloneOrg subversion. It is already compatible with varnish 2.1.2

Next we have haproxy as load balancer that dispatch request over 4 zope instances. Each zope instances are configured with 2 threads (Zope default configuration is 4 threads).

For the database we use relstorage with postgresql, iw.fss to store files on the server’s file system and OpenLDAP to store users.

Monitoring & maintenace

We use nagios & cacti to monitor all services. We have split services in two categories:

• system services: apache, postgresql, memcached, ldap
• app services: varnish, haproxy, zope

This has some advantages: You can deploy the application without system services. This has been possible with buildout because it is very flexible. Supervisord and haproxy has just been taken from the über buildout.

Relstorage has a script to export the database to any database format: mysql, filestorage (data.fs) or whatever you want. So we use a script every night just after the backup to build a file storage “data.fs”, so any developper can work without having to install system services and has the last database available. It is easier to work on bugs when you have the content produce by the customer.

What I would like to improve / add / in to  Plone this year

For sure after a project like this one, I would like to contribute to Plone, share my experience. Here it is.

portlets dynamic template: I would like to be able to register new templates to provide multiple choice of portlet display. As you can choose how a content is displayed with dynamic view fti feature, you should be able to choose how a portlet is displayed.  I overrided collection and static portlet just to do that.

portlets order: I want to have first portlet for users&groups, next portlet for content types and next contextual portlets. To achieve this I overrided portlet mecanism because it was hardcoded.

performances

Plone3 is slow. Right, but it is great. So how can I do a site with traffic and a newsletter of 8300 subscribers. Here are my keys:

• install experimental add-ons listed below
• install cachesetup with varnish as proxy cache and configure both (debug is easy with wireshark & firebug)
• configure of zope: 2 threads, object cache and jarn.checkinterval
• externalise binary content: fss is the best for plone3. Use blob for plone4
• collective.solr: I have tryed out solr but I was not satisfied: some plone features are broken
• add one instance only to deliver the newsletter. This one can not be reached by request, only the clock server

Because we followed the way plone community does website, we succeeded in doing a great web site with only free (GNU & BSD) softwares. We are working on linux desktop and we produce only free software because we love this.

The buildout used to deploy this web site has been push on the collective subversion under the name of nmd. Feel free to use it, to contribute, to share ideas. It has been designed to be useable in offline mode, and built upon reusable pieces depends on what you want.

I would like to thank the community for help, efforts, bug fixes and all great add-ons that have been done. I would like to thank nantes-developpement.com for this experience and their warm welcome of Plone.

Extract all images of your skin with a script to serve them by apache

Following the unloading Plone conference at budapest last year, you may want to serve images from skin directly by apache.

To do this you will need first to extract images. So here is a small python script to help you:

skintool = context.portal_skins
skin = skintool.getDefaultSkin()
PATHS = skintool.getSkinPath(skin).split(',')

images = skintool.ZopeFind(skintool, obj_metatypes=['Filesystem Image'],search_sub=1)
images_skin = [path for path, obj in images if path.split('/')[0] in PATHS and skintool.isFirstInSkin(path)]
images_str = "\n".join(['/'.join(path.split('/')[1:]) for path,obj in images])
print "You have extracted %s images. Here is the result:"%(len(images_skin))
print images_str
return printed

To use it, you just have to go in the Zope Management Interface (ZMI), Add a Script (Python) with the drop down menu and paste this script. Next you can excecute it with the “test” tab or access it by it’s url.

Write the content of the result page in a text file and then follow this blog post: http://www.makina-corpus.org/blog/use-rewritemap-prevent-proxying-some-static-contents

Hope this will help you to unloading Plone.

How to add a random time between two emails sent from zope.sendmail

I’m using collective.dancing for a mass mailing: +8000 emails to send.

One of the question is how to not be blacklisted ? A first thing I have done today is to add a random time between two sent.

To achieve this I have read the code of zope.sendmail. It is a well documented egg so this is not hard to understand how it works:

• A maildir (a directory) store mails under files ready to be sent
• A mailer (a utility) is ready to be called to send email one by one
• A queue mail delivrery (utility) is ready to put messages inside a maildir
• A mailqueue processor (not a component, a thread implementation) parse every 3 seconds if there is new emails to send in the mailbox

So to achieve this you just have to unconfigure the plone.smtp utility and declare a new utility called plone.smtp with this code:

import random
import time

from zope.sendmail import mailer

class SMTP(mailer.SMTPMailer):
    """Override SMTPMailer to let a random time"""

    def send(self, fromaddr, toaddrs, message):
        time.sleep(random.uniform(0,2))
        super(SMTP, self).send(fromaddr, toaddrs, message)

It’s working !

A more simple editorial process in Plone: Classic CMS + InPlace Topic view

For a project I have tried to create a more simple editorial process for Plone.To understand this you have first to know the normal editorial process of Plone:

* First you have to go to the folder where you want to add your article.
* Next you click on add an article
* You fill the form (content + categorisation)
* Finaly you publish your article.

Pages are created with topic to fake publication of one article in multiple sections.

Contents are stored like on a computer : inside folders.

In many project this is a pain because the writer must first thinking about “where do I will add this article”.

So the idea is here. Remove the first step and simply use a link “add article” available anywhere in the site.

The way to achieve this is to create one “database” (large plone folder) per content type and add one action category with all “add xxx” actions + a portlet. At the moment I initialise this in policy.

Now we are able to create article without that first step. We need to know How to keep my tree that make so nice URL with Plone ? Let me introduce you the magic “acquisition” used with topic: collective.inplacetopicview.

A topic is like a persistent search done in Plone. The webmaster add criteria, and the display is like folders but links point to the real destination where article has been added. This add a real issue when you add contextual theming inside Plone. The user can be very surprised if just by clicking on an article link he moved outside the current context (so theme can change).

With inplacetopicview the link has a different behaviour. When you click you just stay in the current context so theme just stay the same.

There are different way to achieve this. inplacetopicview doesn’t add any new content type or what so ever, it just add some new views that let your visitor stay in the context of the topic but in different ways:
* With acquisition (domain.com/mytopic/articles/myarticle with myarticle real path = /myplone/articles/myarticles)
* With beatifulsoup by rendering both article and topic, and replace the content of the topic with the content of the article.

There are some other idea to implements in inplacetopicview like
* Ajax
* Popup

Now let’s try it by installing collective.inplacetopicview and add an initial structure with transmogrifier in your profile.

This process is not perfect but it is one first step done. For example image content handle threw wysiwyg editor need customization. The breadcrum show the real path.

Let me know what you think of this process and if you have idea to improve it.

Functional doctest snippet of the day

I was looking for an easy way to get the current browser.contents in my real browser to easy debug what is happening in my doctest.

Here is the snippet:

Updated the 01/13/2009

>>> import tempfile as tmp
>>> def contents():
...    fd, fn = tmp.mkstemp(suffix=".html", prefix="testbrowser-")
...    file = open(fn, 'w')
...    file.write(browser.contents)
...    file.close()
...    print fn
>>> import pdb; pdb.set_trace()

So now you can just call contents() and copy paste the file path in your real browser and see what’s happening in your tests. May it can be added to testbrowser.Browser ?

collective.remove

Plone has lots of features. collective.remove is a project to remove some features from Plone and be usefull to integrators.

Why ? My customer don’t need them and for performance purpose, it’s good to remove things not used.

What ? This project has three eggs at the moment:

• collective.remove.kss
• collective.remove.kupu
• collective.remove.rules

To know what to remove, it’s per project need. For example I’m using TinyMCE on my last project, so I have removed kupu. Vicent Fretin tells me rules tab permission check is slow and I don’t need it. Limi has tweet kss make Plone slower so I have removed it.

The goal is to remove one thing at a time when a feature is not needed. Most of the time I was doing this in the policy but I have written the same thing multiple times, so lets released it in a reusable package.

It is more than uninstall profiles. It is uninstall packages.

Import contents & keywords with generic setup

I really like generic setup to build plone instance.

But, I was not able to do some things like setup contents (folders for structure, documents to make some example pages, …), keywords and portlets.

Note: The default import structure step of Plone doesn’t do enought things for me, and I find it hard to use. So I have contributed to some third part modules:

CSVReplicata

csvreplicata first give you a way to import/export content within the csv file format. So you can import/export atcontent types (and more) with it. I have added a csvimportStep to be able to use it with genericsetup:

def importcsvStep(context):
"""This step give you a way to import content in your website with
csvreplicata.

How it works:
read replicata.cfg file to load configuration

read replicata.csv and call csvreplicata with the config.

Config:
[replicable_types]
#List schemata for each types you want to import
#Folder : default, categorization, dates, ownership, settings
#News Item : default

Document: default

[settings]
#list here every csv settings that will be given to replicator.importcsv
encoding:utf-8
delimiter:,
stringdelimiter:"
datetimeformat:%d/%m/%Y
wf_transition = publish
conflict_winner:SERVER

"""

So you just have to drop replicata.csv and replicata.cfg files in your profile to be able to import contents in your Plone site. An other way of doing, is to create the content with Plone, then export it, and save the file in your profile.

PloneKeywordManager

I have done the same with PloneKeywordManager. There is a new import step named ‘keywords’. It reads a ‘keywords.txt’ file in the current profile, it creates a document called ‘keywords’. The file must have one keyword per line. It is simple and it works.

Note: To use this you need to use the svn collective trunk version at the moment but they will be released soon.

Portlet ??

Now I m trying to make the same thing with portlets but I don’t have any idea how to do this. If any one has ideas about this, please mail me, I would be glade to implement it !

My Sprint Report : Funkload and Buildbot @ ploneconf2009

Plone need to reach a new QA level: Adding nightly performance tests. It is important when you make a modification on the code to know if there is a impact on the performance of the application. As Elisabeth said in the “unloading plone” conference, measure !

To understand and use technologies, you first need to know what are the names of all these packages:

• Funkload Functional and load web tester
• Buildbot System to automate the compile/test cycle
• collective.coreloadtests Three funkload tests for Plone + some setup profiles
• collective.loadtesting One buildout to setup a buildbot with funkload + some examples.
• collective.funkload Complex functional load testing and benchmarking
• collective.buildbot A set of zc.buildout recipes and support for declarative configuration for Buildbot
• collective.recipe.funkload Easy running of funkload tests and generation of reports.

So what has been done during the sprint:

Day One

I have reviewed pasteFunBot. I have kept some scripts and the template script but I have removed all the templates.

I have read the code of most of the differents eggs and tryed most of them.

What I had in mind before starting:

• Buildbot + Plone4 + Funkload + coreloadtests
• Use collective.loadtesting as an example
• Add a paster script to start continuous performance tests faster

By the end of the day 1 I have done collective.funkbot a paster that create a new performance test bot in three questions:

• vcs ['svn']
• vcs_url ['']
• email ['']

In the Plone community we are using buildout, so the vcs_url is supposed to be the buildout url to checkout / update every night. email is the email adress where will be sent the reports. Also if the buildbot failed to build the buildout, a mail with all stdio will be sent.

At the end of the day I had some troubles with funkload. I was not able to launch performance tests.

Day Two

In the morning tomlazar and vincinc has helped me and I have disovered their work on collective.recipe.funkload done during the last performance sprint in Bristol. In fact funkload script need to know in which eggs it can looking for tests. I have find a bug and fixed it. So now there are a new released of this egg (done with mkrelease)

After lunch I had decided to work on Plone4 buildout to fix coreloadtests.cfg. Because I m new to the community I had done this work under the collective.funkbot repository (still collective but not svn plone).

Next I have used my paster to initialize the buildout of the futur nighly performance bot (still under collective.funkbot repository)

Before the sprint report of the day I have succeed in fixing most of bugs. The only remaining one was on the creation of the plone4 site. That has been a bit funny to report this: “I had succeed except I can’t create a Plone site …”

After this report I have finished and add the ability to collective.recipe.plonesite to create a Plone4 site. I have next discovered a bug with the basic theme of Plone, so I have report it to limi and added the profile of plonetheme.sunbirst to launch funkload.

I have also refresh a bit coreloadtests by moving genericsetup registrations and steps in zcml.

I have also made this report :)

You can check this picture:

TODO

• Push my patch on plone4 buildout
• Refresh the scripts that send the performance reports by email (atm their is a simple python script) and the one who serve html reports throw paste server #static (this one is also used by links inside the email)
• Make a better documentation
• Make contentgenerator faster (it seems not working at all)

Conclusion

I want to thanks my companie Makina-Corpus to have give me the ability to comes to conference and sprint.

I want to thanks the Plone community to be so full of great and impressive people !

I have learn a lot about many different topics (hosting, testing, performances, futur of Plone, …)

I want to be at the next sprints and at the next Plone conferences !

I will continue to work on that topic, so to stay connected, you can follow me on twitter with the nickname toutpt that you can’t pronounce in english, so if you try , do it like “toopatai” :)

Links

• collective.funkbot
• plone4 buildout patched to be triggered by buildout with coreloadtests
• The nighly buildout buildbot (pasted from funkbot and adapted).

Collection + related items: A great feature of Plone

Today I have worked on Collection (ATTopic). I have discovered a great feature.

You can use related items to choose first results. It can be very useful to manage a home page. First results are the most important items because they are the most read.

So choosing first items is important and at the same time making the page content changing with the web site activity is just great !

At the time of writing this post, this feature has a bug: If criteria match related items they just appear twice. I have reported that bug with a test and a patch here: https://dev.plone.org/plone/ticket/9591. I hope it will be accepted.

It is not finished yet. You can use archetypes.referencebrowserwidget to add order handling to related items. It add up and down arrows on each related items. So you can move up and down each one of them like in folder listing with items.

Enjoyed it !