Import contents & keywords with generic setup

Posted in plone on November 17, 2009 by toutpt

I really like generic setup to build plone instance.

But, I was not able to do some things like setup contents (folders for structure, documents to make some example pages, …), keywords and portlets.

Note: The default import structure step of Plone doesn’t do enought things for me, and I find it hard to use. So I have contributed to some third part modules:

CSVReplicata

csvreplicata first give you a way to import/export content within the csv file format. So you can import/export atcontent types (and more) with it. I have added a csvimportStep to be able to use it with genericsetup:

def importcsvStep(context):
"""This step give you a way to import content in your website with
csvreplicata.

How it works:
read replicata.cfg file to load configuration

read replicata.csv and call csvreplicata with the config.

Config:
[replicable_types]
#List schemata for each types you want to import
#Folder : default, categorization, dates, ownership, settings
#News Item : default

Document: default

[settings]
#list here every csv settings that will be given to replicator.importcsv
encoding:utf-8
delimiter:,
stringdelimiter:"
datetimeformat:%d/%m/%Y
wf_transition = publish
conflict_winner:SERVER

"""

So you just have to drop replicata.csv and replicata.cfg files in your profile to be able to import contents in your Plone site. An other way of doing, is to create the content with Plone, then export it, and save the file in your profile.

PloneKeywordManager

I have done the same with PloneKeywordManager. There is a new import step named ‘keywords’. It reads a ‘keywords.txt’ file in the current profile, it creates a document called ‘keywords’. The file must have one keyword per line. It is simple and it works.

Note: To use this you need to use the svn collective trunk version at the moment but they will be released soon.

Portlet ??

Now I m trying to make the same thing with portlets but I don’t have any idea how to do this. If any one has ideas about this, please mail me, I would be glade to implement it !

My Sprint Report : Funkload and Buildbot @ ploneconf2009

Posted in plone on November 2, 2009 by toutpt

Plone need to reach a new QA level: Adding nightly performance tests. It is important when you make a modification on the code to know if there is a impact on the performance of the application. As Elisabeth said in the “unloading plone” conference, measure !

To understand and use technologies, you first need to know what are the names of all these packages:

  • Funkload Functional and load web tester
  • Buildbot System to automate the compile/test cycle
  • collective.coreloadtests Three funkload tests for Plone + some setup profiles
  • collective.loadtesting One buildout to setup a buildbot with funkload + some examples.
  • collective.funkload Complex functional load testing and benchmarking
  • collective.buildbot A set of zc.buildout recipes and support for declarative configuration for Buildbot
  • collective.recipe.funkload Easy running of funkload tests and generation of reports.

So what has been done during the sprint:

Day One

I have reviewed pasteFunBot. I have kept some scripts and the template script but I have removed all the templates.

I have read the code of most of the differents eggs and tryed most of them.

What I had in mind before starting:

  • Buildbot + Plone4 + Funkload + coreloadtests
  • Use collective.loadtesting as an example
  • Add a paster script to start continuous performance tests faster

By the end of the day 1 I have done collective.funkbot a paster that create a new performance test bot in three questions:

  • vcs [‘svn’]
  • vcs_url [”]
  • email [”]

In the Plone community we are using buildout, so the vcs_url is supposed to be the buildout url to checkout / update every night. email is the email adress where will be sent the reports. Also if the buildbot failed to build the buildout, a mail with all stdio will be sent.

At the end of the day I had some troubles with funkload. I was not able to launch performance tests.

Day Two

In the morning tomlazar and vincinc has helped me and I have disovered their work on collective.recipe.funkload done during the last performance sprint in Bristol. In fact funkload script need to know in which eggs it can looking for tests. I have find a bug and fixed it. So now there are a new released of this egg (done with mkrelease)

After lunch I had decided to work on Plone4 buildout to fix coreloadtests.cfg. Because I m new to the community I had done this work under the collective.funkbot repository (still collective but not svn plone).

Next I have used my paster to initialize the buildout of the futur nighly performance bot (still under collective.funkbot repository)

Before the sprint report of the day I have succeed in fixing most of bugs. The only remaining one was on the creation of the plone4 site. That has been a bit funny to report this: “I had succeed except I can’t create a Plone site …”

After this report I have finished and add the ability to collective.recipe.plonesite to create a Plone4 site. I have next discovered a bug with the basic theme of Plone, so I have report it to limi and added the profile of plonetheme.sunbirst to launch funkload.

I have also refresh a bit coreloadtests by moving genericsetup registrations and steps in zcml.

I have also made this report :)

You can check this picture:

TODO

  • Push my patch on plone4 buildout
  • Refresh the scripts that send the performance reports by email (atm their is a simple python script) and the one who serve html reports throw paste server #static (this one is also used by links inside the email)
  • Make a better documentation
  • Make contentgenerator faster (it seems not working at all)

Conclusion

I want to thanks my companie Makina-Corpus to have give me the ability to comes to conference and sprint.

I want to thanks the Plone community to be so full of great and impressive people !

I have learn a lot about many different topics (hosting, testing, performances, futur of Plone, …)

I want to be at the next sprints and at the next Plone conferences !

I will continue to work on that topic, so to stay connected, you can follow me on twitter with the nickname toutpt that you can’t pronounce in english, so if you try , do it like “toopatai” :)

Links

Collection + related items: A great feature of Plone

Posted in plone on October 7, 2009 by toutpt

Today I have worked on Collection (ATTopic). I have discovered a great feature.

You can use related items to choose first results. It can be very useful to manage a home page. First results are the most important items because they are the most read.

So choosing first items is important and at the same time making the page content changing with the web site activity is just great !

At the time of writing this post, this feature has a bug: If criteria match related items they just appear twice. I have reported that bug with a test and a patch here: https://dev.plone.org/plone/ticket/9591. I hope it will be accepted.

It is not finished yet. You can use archetypes.referencebrowserwidget to add order handling to related items. It add up and down arrows on each related items. So you can move up and down each one of them like in folder listing with items.

Enjoyed it !

Private Pypi setup

Posted in plone on September 11, 2009 by toutpt

I have succeed in seting up a full private pypi. So I want to share this.

First you need to install a Plone on a server with plone software center. Our system administrators has setup DNS to this plone with apache + http auth + SSL (https) and has given me one login / password.

Next you need to add a new software center to the content of your site. You can also change base workflow of a project to be approved by default.

The final step for the server is to add a user at the root of Zope with the same user/password has http auth

Now you are ready to the next step: setup your environnement on your computer. You need to add http auth ability to buildout and collective.dist. For this:

I have looking for the ‘realm’ of my server during 2 hours. So what is a realm ? Thanks to Tarek who have take times to answer me on irc:

  toupt the realm is the domain the server sends back when you do a challenge
  for instance zope sends "Zope"
  trac sends "trac"
  etc

I have apache as server, so is it “Apache” ? The documentation of lovely.buildouthttp says” My domain” but it still doesn’t work. Finaly I have found the realm. It was ‘Members Only Area’. this is the default apache realm. This is a good things to know.

A contribution: add documentation to lovely.buildouthttp, collective.dist and software center. The realm and the user in Zope trick were not easy to find.

The benefits of a private pypi:

  • Release private eggs for customers
  • Test some eggs before release on pypi and plone.org (fix rest for example)
  • Release your last commit on collective to use it now in production
  • Learn release process of an egg

A word about version number in packaging / release with pypi

Posted in plone, python on August 17, 2009 by toutpt

I often see lots of information about version number on release. I would like to explain it. For sure their is lots of different use case but here, I speak about a code you are writing and want to package.

For me there are 4 parts:

  • Major version number
  • Feature version indice
  • Current level of release
  • Status of the release

Major version number is about layout or about backward compatibility. When you change that number it means you have totaly change most of things or you have broke the layout because you have done a big mistake and you don’t have other choice. For example Zope 2.X.X and Zope 3.X.X.

Feature version indice is all about features. For me this number is increased when you have added feature(s). For example adding undo/redo to the software.

Current level of release is all about bug fixes. For example an i18n broken reported by someone.

The status is a character or a work like beta, rc, … and is all about release status. The order is:

  • alpha (a) often means only for developers, first calls for testing (try to install)
  • beta (b) often it is used to call for testing (install and use it)
  • candidate (c or rc) means no more bugs reported just about packaging and last call for testing

I m sure there are some formals paper around Internet that explains this a lot better, but I was just wanting to explain my point of view.

Monkey ! When patching can’t be merged: experimental.aggressiveopaquespeedup

Posted in plone, zope on August 12, 2009 by toutpt

Next to my first plone patch that improve performance I have profiling the code on a simple query: get a javascript from portal_javascript already computed.

The result is clear: 28% of the time is spent in opaqueitems. So I have discuss about it on CMF mailing list.

A monkey patch was already existing: experimental.opaquespeedup. I have read the code. It replace the time consuming calls on all attributes by a catalog request. But I know that catalog can be slow if the web site has lots of contents. I have decided to make something more aggressive, by just removing opaqueitems that seems to be not used.

So use it at your own risk: experimental.aggressiveopaquespeedup

My first patch ! Improve performance of Plone.

Posted in plone on August 12, 2009 by toutpt

I m proud of it, so I want to blog about it. Let me tell you the story of my first patch on Plone.

Two students from ‘Ecole polytechnique de l université de Nantes‘ has worked for me during 6 month. They have discover so much technologies:

  • Python
  • Zope
  • Plone
  • JMeter

Their subject was : “Just improve authentication performance of Plone”. They do not have succeed in doing this, but they has showed me some graphics where i have found a performance issue: the user properties. So i have kept Nasreddine BERCHIDA for the summer to continue on this. Thuesday 7 July, he has found that enumerateUsers from PlonePas.plugins.property was the source code that matters.

We have done a report with funkload with lots of users and got it: lots of green means performance improved.

The patch is about to be merged (I hope).

References:
mailing list discussion
bug tracker ticket (need a plone.org account)