Plone need to reach a new QA level: Adding nightly performance tests. It is important when you make a modification on the code to know if there is a impact on the performance of the application. As Elisabeth said in the “unloading plone” conference, measure !

To understand and use technologies, you first need to know what are the names of all these packages:

• Funkload Functional and load web tester
• Buildbot System to automate the compile/test cycle
• collective.coreloadtests Three funkload tests for Plone + some setup profiles
• collective.loadtesting One buildout to setup a buildbot with funkload + some examples.
• collective.funkload Complex functional load testing and benchmarking
• collective.buildbot A set of zc.buildout recipes and support for declarative configuration for Buildbot
• collective.recipe.funkload Easy running of funkload tests and generation of reports.

So what has been done during the sprint:

Day One

I have reviewed pasteFunBot. I have kept some scripts and the template script but I have removed all the templates.

I have read the code of most of the differents eggs and tryed most of them.

What I had in mind before starting:

• Buildbot + Plone4 + Funkload + coreloadtests
• Use collective.loadtesting as an example
• Add a paster script to start continuous performance tests faster

By the end of the day 1 I have done collective.funkbot a paster that create a new performance test bot in three questions:

• vcs ['svn']
• vcs_url ['']
• email ['']

In the Plone community we are using buildout, so the vcs_url is supposed to be the buildout url to checkout / update every night. email is the email adress where will be sent the reports. Also if the buildbot failed to build the buildout, a mail with all stdio will be sent.

At the end of the day I had some troubles with funkload. I was not able to launch performance tests.

Day Two

In the morning tomlazar and vincinc has helped me and I have disovered their work on collective.recipe.funkload done during the last performance sprint in Bristol. In fact funkload script need to know in which eggs it can looking for tests. I have find a bug and fixed it. So now there are a new released of this egg (done with mkrelease)

After lunch I had decided to work on Plone4 buildout to fix coreloadtests.cfg. Because I m new to the community I had done this work under the collective.funkbot repository (still collective but not svn plone).

Next I have used my paster to initialize the buildout of the futur nighly performance bot (still under collective.funkbot repository)

Before the sprint report of the day I have succeed in fixing most of bugs. The only remaining one was on the creation of the plone4 site. That has been a bit funny to report this: “I had succeed except I can’t create a Plone site …”

After this report I have finished and add the ability to collective.recipe.plonesite to create a Plone4 site. I have next discovered a bug with the basic theme of Plone, so I have report it to limi and added the profile of plonetheme.sunbirst to launch funkload.

I have also refresh a bit coreloadtests by moving genericsetup registrations and steps in zcml.

I have also made this report :)

You can check this picture:

TODO

• Push my patch on plone4 buildout
• Refresh the scripts that send the performance reports by email (atm their is a simple python script) and the one who serve html reports throw paste server #static (this one is also used by links inside the email)
• Make a better documentation
• Make contentgenerator faster (it seems not working at all)

Conclusion

I want to thanks my companie Makina-Corpus to have give me the ability to comes to conference and sprint.

I want to thanks the Plone community to be so full of great and impressive people !

I have learn a lot about many different topics (hosting, testing, performances, futur of Plone, …)

I want to be at the next sprints and at the next Plone conferences !

I will continue to work on that topic, so to stay connected, you can follow me on twitter with the nickname toutpt that you can’t pronounce in english, so if you try , do it like “toopatai” :)

Links

• collective.funkbot
• plone4 buildout patched to be triggered by buildout with coreloadtests
• The nighly buildout buildbot (pasted from funkbot and adapted).

Today I have worked on Collection (ATTopic). I have discovered a great feature.

You can use related items to choose first results. It can be very useful to manage a home page. First results are the most important items because they are the most read.

So choosing first items is important and at the same time making the page content changing with the web site activity is just great !

At the time of writing this post, this feature has a bug: If criteria match related items they just appear twice. I have reported that bug with a test and a patch here: https://dev.plone.org/plone/ticket/9591. I hope it will be accepted.

It is not finished yet. You can use archetypes.referencebrowserwidget to add order handling to related items. It add up and down arrows on each related items. So you can move up and down each one of them like in folder listing with items.

Enjoyed it !

I have succeed in seting up a full private pypi. So I want to share this.

First you need to install a Plone on a server with plone software center. Our system administrators has setup DNS to this plone with apache + http auth + SSL (https) and has given me one login / password.

Next you need to add a new software center to the content of your site. You can also change base workflow of a project to be approved by default.

The final step for the server is to add a user at the root of Zope with the same user/password has http auth

Now you are ready to the next step: setup your environnement on your computer. You need to add http auth ability to buildout and collective.dist. For this:

• configure the pypirc file like describe on plone.org
• Add lovely.buildouthttp extension to your buildout
• Add ‘.buildout’ folder to your home directory if not already exists
• Add ‘.buildout/.httpauth’ with realm, https://yourdomain, user, password

I have looking for the ‘realm’ of my server during 2 hours. So what is a realm ? Thanks to Tarek who have take times to answer me on irc:

  toupt the realm is the domain the server sends back when you do a challenge
  for instance zope sends "Zope"
  trac sends "trac"
  etc

I have apache as server, so is it “Apache” ? The documentation of lovely.buildouthttp says” My domain” but it still doesn’t work. Finaly I have found the realm. It was ‘Members Only Area’. this is the default apache realm. This is a good things to know.

A contribution: add documentation to lovely.buildouthttp, collective.dist and software center. The realm and the user in Zope trick were not easy to find.

The benefits of a private pypi:

• Release private eggs for customers
• Test some eggs before release on pypi and plone.org (fix rest for example)
• Release your last commit on collective to use it now in production
• Learn release process of an egg

I often see lots of information about version number on release. I would like to explain it. For sure their is lots of different use case but here, I speak about a code you are writing and want to package.

For me there are 4 parts:

• Major version number
• Feature version indice
• Current level of release
• Status of the release

Major version number is about layout or about backward compatibility. When you change that number it means you have totaly change most of things or you have broke the layout because you have done a big mistake and you don’t have other choice. For example Zope 2.X.X and Zope 3.X.X.

Feature version indice is all about features. For me this number is increased when you have added feature(s). For example adding undo/redo to the software.

Current level of release is all about bug fixes. For example an i18n broken reported by someone.

The status is a character or a work like beta, rc, … and is all about release status. The order is:

• alpha (a) often means only for developers, first calls for testing (try to install)
• beta (b) often it is used to call for testing (install and use it)
• candidate (c or rc) means no more bugs reported just about packaging and last call for testing

I m sure there are some formals paper around Internet that explains this a lot better, but I was just wanting to explain my point of view.

Next to my first plone patch that improve performance I have profiling the code on a simple query: get a javascript from portal_javascript already computed.

The result is clear: 28% of the time is spent in opaqueitems. So I have discuss about it on CMF mailing list.

A monkey patch was already existing: experimental.opaquespeedup. I have read the code. It replace the time consuming calls on all attributes by a catalog request. But I know that catalog can be slow if the web site has lots of contents. I have decided to make something more aggressive, by just removing opaqueitems that seems to be not used.

So use it at your own risk: experimental.aggressiveopaquespeedup

I m proud of it, so I want to blog about it. Let me tell you the story of my first patch on Plone.

Two students from ‘Ecole polytechnique de l université de Nantes‘ has worked for me during 6 month. They have discover so much technologies:

• Python
• Zope
• Plone
• JMeter
• …

Their subject was : “Just improve authentication performance of Plone”. They do not have succeed in doing this, but they has showed me some graphics where i have found a performance issue: the user properties. So i have kept Nasreddine BERCHIDA for the summer to continue on this. Thuesday 7 July, he has found that enumerateUsers from PlonePas.plugins.property was the source code that matters.

We have done a report with funkload with lots of users and got it: lots of green means performance improved.

The patch is about to be merged (I hope).

References:
mailing list discussion
bug tracker ticket (need a plone.org account)

Remember an old post i have done about the way i manage eclipse third part plugins dependencies, i have discover a web site where you can select the plugins you want and it build a bundle for you ready to be downloaded:

Yoxos eclipse service

I have tried it and it rocks ! For 131 MO to download i got:

• Pydev
• WTP (XML + JS + …)
• Git plugin
• Mylyn + Mylyn connectors for Track and Mantis
• AnyEdit (manage tabs and spaces)

And you don’t have all Java’s stuff in your eclipse so it’s stay clean and start quickly.

I have work hard on testing/debugging an already developed specific app that handle local roles in a Plone site. I was sure that local roles was just manageable throw a permission but i have finally read the code that handle local roles and understand things:

* you can only add a role to somebody if you have the permission “Change local roles”
* you can only add a role to somebody if you this role in that context
* role are quiet on error (if you don’t have the role you are trying to add, you will not have any log)
* if you have manager role you can do what ever you want (don’t do your tests in manager !!! )
* else candidate local roles are only yours without ‘Anonymous’, ‘Authenticated’ and ‘Shared’
* local roles are stored in a dict called __ac_local_roles__ on the context

After one day of work with local roles, i think the most painful part is to don’t have log, even in debug mode. When a process is called to do a job and it do half of it, i think it has to said something.

But local roles are an essential part of Plone that i really like.

PS: i have discover to late a good module to display local roles in Plone: zopyx.plone.cassandra.

yet an other post to say svn sucks. yes it is. But many people who use svn are happy with it and i was happy to.

It’s hard to maintain a trunk when 5 people are commiting on it during the project life. So i have started to look at DVCS (git hg, bzr). I have tryied GIT because of it’s popularity.

At work, someone is using mercurial (hg) and we are two people intersting in GIT. So i have decided to test it but over SVN. it means the central repository is still an svn one.

Here is the way i m using it:
1. clone the repo (git svn clone blabla)
2. branche
3. work on your code (commit, …)
4. merge on master
5. git svn dcommit

And you get it, only one commit on the trunk (master) and the power of git to merge, commit, …

Some cool features and properties of git:

• space on hdd of a git clone < svn checkout (and you get the all history
• fast, git is damn faster than svn. When you type enter key, you get it (except from git svn clone, because here it’s svn)
• qgit2 (qt4) is a very good tool (you can browse your svn history, look at diffs, ….)
• the help on command line. git status give you help for next command you will make

But there are some point i have not found handy:

• git st mygitrepo/ doesn’t work (“is not a git repo” but it is)
• no default alias (st, co, …) but can be easily configured
• svn externals can be seen has submodules, but no managed in that way. git svn do not handle svn:externals and you can’t use git submodules over svn
• eclipse do not manage git (and i know it s a pain for eclipse guys because it break the way plateform is thought)

I hope all that things will be improved in the futur of git, but now i m using git at work. Next step will be code review TTW.

I just want to try to make a simple application with the best of current opensource technologies. Got a try with those one:

• Python 2.5
• Zope 3.4
• Postgresql 8.3
• SqlAlchemy 0.5
• YUI 2.7
• ZODB in Postgresql with RelStorage

I have take an hour to install this on my laptop, with the excellent virtualenv to build a copy of an already installed python in a directory.

/path/2/python/bin/virtualenv --no-site-packages /path/2/project

Next i have use zopeproject to install the last zope3 technologies. Now zopeproject use the KGS to set the version of eggs to use.

cd  /path/2/project
source bin/activate
easy_install zopeproject
zopeproject myproject

You get it, you have the last release of zope3.4 release. Now add the eggs you want to use

eggs = ...
       psycopg2
       RelStorage
       ore.alchemist
       hurry.zopeyui
       z3c.form
#       chameleon.zpt require zope.i18n > 3.4

modify your zope.conf to configure RelStorage:

%import relstorage
<zodb>
  <relstorage>
    <postgresql>
      # The dsn is optional, as are each of the parameters in the dsn.
      dsn dbname='zope' user='zope' host='localhost'
    </postgresql>
  </relstorage>
</zodb>

Now, you get your zope3.4 running on python2.5 and store the zodb data in postgresql. I have lost most of the time on RelStorage and postgresql configuration. I’m to work with all those technologies.

I will work on making a simple blog in pure zope3, and next i will try to make the same application with SqlAlchemy.