Archive for the zope Category

Nantes-developpement.com a new Plone website by Makina-Corpus

Posted in plone, zope on July 7, 2010 by toutpt

I’m proud to announce it, the new web site of “The greater Nantes economic agency” is now available here: http://www.nantes-developpement.com .

This web site use Plone with some community add-ons (available on pypi). Some of these are new, mean they have been created to fit the customer needs, others has been contributed , mean they has been modified upstream, and last has just been used:

Custom code contains:

  • the theme done as a usual plonetheme (not with xdv).
  • policy (integration of all community add-ons and plone configuration)
  • portlets
  • content-types (collection, folderish document, video/audio interview, …)

System architecture

We have apache2 in front end. Next we have varnish as proxy cache (but not load balancer). Configuration has been taken from Products.PloneOrg subversion. It is already compatible with varnish 2.1.2

Next we have haproxy as load balancer that dispatch request over 4 zope instances. Each zope instances are configured with 2 threads (Zope default configuration is 4 threads).

For the database we use relstorage with postgresql, iw.fss to store files on the server’s file system and OpenLDAP to store users.

Monitoring & maintenace

We use nagios & cacti to monitor all services. We have split services in two categories:

  • system services: apache, postgresql, memcached, ldap
  • app services: varnish, haproxy, zope

This has some advantages: You can deploy the application without system services. This has been possible with buildout because it is very flexible. Supervisord and haproxy has just been taken from the über buildout.

Relstorage has a script to export the database to any database format: mysql, filestorage (data.fs) or whatever you want. So we use a script every night just after the backup to build a file storage “data.fs”, so any developper can work without having to install system services and has the last database available. It is easier to work on bugs when you have the content produce by the customer.

What I would like to improve / add / in to  Plone this year

For sure after a project like this one, I would like to contribute to Plone, share my experience. Here it is.

portlets dynamic template: I would like to be able to register new templates to provide multiple choice of portlet display. As you can choose how a content is displayed with dynamic view fti feature, you should be able to choose how a portlet is displayed.  I overrided collection and static portlet just to do that.

portlets order: I want to have first portlet for users&groups, next portlet for content types and next contextual portlets. To achieve this I overrided portlet mecanism because it was hardcoded.

performances

Plone3 is slow. Right, but it is great. So how can I do a site with traffic and a newsletter of 8300 subscribers. Here are my keys:

  • install experimental add-ons listed below
  • install cachesetup with varnish as proxy cache and configure both (debug is easy with wireshark & firebug)
  • configure of zope: 2 threads, object cache and jarn.checkinterval
  • externalise binary content: fss is the best for plone3. Use blob for plone4
  • collective.solr: I have tryed out solr but I was not satisfied: some plone features are broken
  • add one instance only to deliver the newsletter. This one can not be reached by request, only the clock server

Because we followed the way plone community does website, we succeeded in doing a great web site with only free (GNU & BSD) softwares. We are working on linux desktop and we produce only free software because we love this.

The buildout used to deploy this web site has been push on the collective subversion under the name of nmd. Feel free to use it, to contribute, to share ideas. It has been designed to be useable in offline mode, and built upon reusable pieces depends on what you want.

I would like to thank the community for help, efforts, bug fixes and all great add-ons that have been done. I would like to thank nantes-developpement.com for this experience and their warm welcome of Plone.

Advertisements

How to add a random time between two emails sent from zope.sendmail

Posted in plone, zope on June 15, 2010 by toutpt

I’m using collective.dancing for a mass mailing: +8000 emails to send.

One of the question is how to not be blacklisted ? A first thing I have done today is to add a random time between two sent.

To achieve this I have read the code of zope.sendmail. It is a well documented egg so this is not hard to understand how it works:

  • A maildir (a directory) store mails under files ready to be sent
  • A mailer (a utility) is ready to be called to send email one by one
  • A queue mail delivrery (utility) is ready to put messages inside a maildir
  • A mailqueue processor (not a component, a thread implementation) parse every 3 seconds if there is new emails to send in the mailbox

So to achieve this you just have to unconfigure the plone.smtp utility and declare a new utility called plone.smtp with this code:

import random
import time

from zope.sendmail import mailer

class SMTP(mailer.SMTPMailer):
    """Override SMTPMailer to let a random time"""

    def send(self, fromaddr, toaddrs, message):
        time.sleep(random.uniform(0,2))
        super(SMTP, self).send(fromaddr, toaddrs, message)

It’s working !

Monkey ! When patching can’t be merged: experimental.aggressiveopaquespeedup

Posted in plone, zope on August 12, 2009 by toutpt

Next to my first plone patch that improve performance I have profiling the code on a simple query: get a javascript from portal_javascript already computed.

The result is clear: 28% of the time is spent in opaqueitems. So I have discuss about it on CMF mailing list.

A monkey patch was already existing: experimental.opaquespeedup. I have read the code. It replace the time consuming calls on all attributes by a catalog request. But I know that catalog can be slow if the web site has lots of contents. I have decided to make something more aggressive, by just removing opaqueitems that seems to be not used.

So use it at your own risk: experimental.aggressiveopaquespeedup

build a zope 3.4 with Postgresql and YUI framework & get ready to work with

Posted in python, zope on March 3, 2009 by toutpt

I just want to try to make a simple application with the best of current opensource technologies. Got a try with those one:

  • Python 2.5
  • Zope 3.4
  • Postgresql 8.3
  • SqlAlchemy 0.5
  • YUI 2.7
  • ZODB in Postgresql with RelStorage

I have take an hour to install this on my laptop, with the excellent virtualenv to build a copy of an already installed python in a directory.

/path/2/python/bin/virtualenv --no-site-packages /path/2/project

Next i have use zopeproject to install the last zope3 technologies. Now zopeproject use the KGS to set the version of eggs to use.

cd  /path/2/project
source bin/activate
easy_install zopeproject
zopeproject myproject

You get it, you have the last release of zope3.4 release. Now add the eggs you want to use

eggs = ...
       psycopg2
       RelStorage
       ore.alchemist
       hurry.zopeyui
       z3c.form
#       chameleon.zpt require zope.i18n > 3.4

modify your zope.conf to configure RelStorage:

%import relstorage
<zodb>
  <relstorage>
    <postgresql>
      # The dsn is optional, as are each of the parameters in the dsn.
      dsn dbname='zope' user='zope' host='localhost'
    </postgresql>
  </relstorage>
</zodb>

Now, you get your zope3.4 running on python2.5 and store the zodb data in postgresql. I have lost most of the time on RelStorage and postgresql configuration. I’m to work with all those technologies.

I will work on making a simple blog in pure zope3, and next i will try to make the same application with SqlAlchemy.