Site News

Post

Fixing Broken ZODB Object references

I'm not an expert on this by any means, but here are some notes on my latest episode.

Introduction

If you start seeing POSKeyErrors on certain object, it most likely means your database is in some form of inconsistency. The problem is very well described by Elizabeth Leddy on her blog here. Her blog didn't quite handle the case that I encountered, missing objects--no oid in ZODB.

Getting Started

Run fsrefs.py to test your database and have it tell you which objects are bad.

python /path/to/eggs/ZODB/scripts/fsrefs.py /path/to/zodb/Data.fs

Will yield results like:

 oid 0x959755L BTrees.OOBTree.OOBucket
last updated: 2011-04-15 13:31:28.380634, tid=0x38DA88B79173877L
refers to invalid object:
	oid 0x0135ca66 missing: ''

oid 0x135CA59L Products.ATContentTypes.content.document.ATDocument
last updated: 2011-04-11 22:21:16.544874, tid=0x38D941D46976A11L
refers to invalid objects:
	oid 0x0135ca65 missing: ''
	oid 0x0135ca5c missing: ''

oid 0x135CA6AL BTrees.OOBTree.OOBTree
last updated: 2011-04-11 22:16:14.294142, tid=0x38D94183CFD03CCL
refers to invalid object:
	oid 0x0135ca6b missing: ''

Testing Out The Bad Object

from ZODB.utils import p64
from persistent import Persistent
obj = app._p_jar[p64(0x959755L)] obj

Should give the error:

2011-05-24 09:23:31 ERROR ZODB.Connection Couldn't load state for 0x0135ca59
Traceback (most recent call last):
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZODB/Connection.py", line 811, in setstate
    self._setstate(obj)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZODB/Connection.py", line 870, in _setstate
    self._reader.setGhostState(obj, p)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZODB/serialize.py", line 604, in setGhostState
    state = self.getState(pickle)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZODB/serialize.py", line 597, in getState
    return unpickler.load()
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZODB/serialize.py", line 471, in _persistent_load
    return self.load_oid(reference)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZODB/serialize.py", line 537, in load_oid
    return self._conn.get(oid)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZODB/Connection.py", line 244, in get
    p, serial = self._storage.load(oid, self._version)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py", line 712, in load
    return self.loadEx(oid, version)[:2]
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZEO/ClientStorage.py", line 735, in loadEx
    data, tid, ver = self._server.loadEx(oid, version)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZEO/ServerStub.py", line 196, in loadEx
    return self.rpc.call("loadEx", oid, version)
  File "/opt/Zope/buildout-cache/eggs/ZODB3-3.8.4wc1-py2.4-linux-x86_64.egg/ZEO/zrpc/connection.py", line 699, in call
    raise inst # error raised by server
POSKeyError: 0x0135ca65

Post

Notes on a More Secure Plone Deployment

Some things to think about if you're planning on providing a more secure Plone site. While Plone is a very secure CMS with an incredible track record, there are still plenty of things you can do to protect sites that might be larger targets.

Read-only Public Site

Making your public site read-only will prevent even a compromised site from taking any damage--even if a malicious user does somehow gain access, they can't save any different data to the database.

There are a few ways to do this:

  • Zope Replication Services(ZRS) allow you replicate a read-write backend private server to a read-only public facing site
  • You can also use RelStorage for you zeoserver. Then use the replication facilities provided by some RDMSs to replicate to a read-only zeoserver on the public site.
  • It is also possible to have read-only zeo clients connected to a read-write zeo server.
  • zeoraid might even be an option(never tried it)

One thing to note is that there are some cases where Plone will try to write on read unfortunately. To get around this, I create a before commit event handler in a policy product to abort every transaction when the server is read-only. It's kind of hackish but a necessary evil to prevent a user from getting a nasty ReadOnly database error thrown at them. It would look something like:

from zope.component import adapter
from ZPublisher.interfaces import IPubBeforeCommit
import App.config
import transaction
configuration = App.config.getConfiguration()
readonly = configuration.read_only_database
@adapter(IPubBeforeCommit)
def abortTransactionOnReadOnly(event):
if readonly:
transaction.abort()

Rewrite Login URLs

You can also rewrite login urls on the public site to restrict anyone from seeing a login form. Just do normal rewrites at your proxy server.

Urls you'll want to rewrite are:

  • /manage
  • /login
  • /logged_out
  • /require_login
  • /acl_users

This will prevent anyone from seeing a login form and an unauthorized page.

Post

Using Plone as a Document Repository

Sharing my experience in using plone to OCR PDF documents and displaying the documents in the browser with Flex Paper.

Update

It is recommended that you do not use this method anymore. Please use collective.documentviewer now which should cover all the use cases.

We just released a new site that houses thousands of scanned PDF documents that are now viewable in the browser via Flex Paper. We started with PDFs that were just scanned images. Plone, with the help of a few packages, then OCR'd and replaced the PDF with a searchable PDF counterpart.

Features

  • Convert Image PDFs to searchable versions
  • Split large PDFs into multiple documents
  • Overwrite metadata of PDF
  • OCR text is then searchable via Plone search
  • Online viewable version
  • All document processing is done via asynchronous processes so adding documents is not slow
  • Can monitor conversion asynchronous processes

Requirements

  • wc.pageturner : For online viewable PDFs
  • wildcard.pdfpal : heavy lifting in PDF processing
  • plone.app.async : asynchronously process PDF documents
  • Tesseract > 3.0.1 system package
  • swftools system package
  • ghostscript system package
  • hocr2pdf system package
  • pdftk system package
  • tiff2pdf system package

Caveats

  • Probably only works in Linux
  • wildcard.pdfpal is pretty specific and isn't smart at if it should process the PDF. For instance, if the PDF is already searchable, it'll still try to convert it regardless.
  • We're not really interested in wildly supporting pdfpal beyond our use case(that's why it's not listed on plone.org, but in the collective and on pypi). So if you're interested in implementing this, you might end up contributing to the project and cleaning up some of the cruft in the package.
Post

Running Plone 4b4 with Zope 2.13.0a1

Just some guidelines on getting Plone 4b4 to work with the new Zope 2.13 release to save you some time.

Update

This method is most likely no longer suitable for a plone WSGI setup. With stable releases of Zope 2.13 coming out and Plone 4.1 almost ready for alpha, it'd be best to start there and ignore this post.

Introductions

I'm not going to go into detail of the wheres and hows everything is done. This post expects you to know a bit about plone, zope and buildout. Maybe I'll be more detailed later. Use this if you want to save yourself a lot of time in getting a working setup with Plone 4 and Zope 2.13. Zope 2.13 adds native WSGI support in Zope. I tested it a bit and seems to work well but results may vary and I'm sure there will be a more supported way to do this soon.

Extends

Make sure your buildout extends the http://download.zope.org/Zope2/index/2.13.0a1/versions.cfg versions file.

Checkouts

You'll need to checkout Products.CMFCore, Products.PluggableAuthService, Products.TinyMCE and plone.locking from svn. You can use these locations right now until there is a new release,

http://svn.zope.org/repos/main/Products.CMFCore/branches/2.2/ Products.CMFCore

http://svn.zope.org/repos/main/Products.PluggableAuthService/trunk Products.PluggableAuthService

http://svn.plone.org/svn/collective/Products.TinyMCE/trunk/ Products.TinyMCE

http://svn.plone.org/svn/plone/plone.locking/trunk/ plone.locking

You'll also need to add these packages to your develop buildout section.

Extra Versions Pins

You'll need to pin these versions since Zope 2.13 doesn't pin version that plone's setup used to assume were pinned,

Add these extra version pins

Products.CMFCore = Products.PluggableAuthService = Products.TinyMCE = plone.locking = five.formlib = 1.0.2 zope.formlib = 3.7.0 zope.app.apidoc = 3.6.2 zope.app.applicationcontrol = 3.5.0 zope.app.appsetup = 3.11 zope.app.authentication = 3.6.0 zope.app.basicskin = 3.4.1 zope.app.broken = 3.5.0 zope.app.cache = 3.6.0 zope.app.catalog = 3.8.0 zope.app.component = 3.8.3 zope.app.container = 3.8.0 zope.app.content = 3.4.0 zope.app.dav = 3.5.1 zope.app.debug = 3.4.1 zope.app.dependable = 3.4.0 zope.app.dtmlpage = 3.5.0 zope.app.error = 3.5.2 zope.app.exception = 3.5.0 zope.app.file = 3.5.0 zope.app.folder = 3.5.1 zope.app.form = 3.8.1 zope.app.generations = 3.5.0 zope.app.http = 3.6.0 zope.app.i18n = 3.6.1 zope.app.interface = 3.5.0 zope.app.intid = 3.7.0 zope.app.locales = 3.6.1 zope.app.localpermission = 3.7.2 zope.app.pagetemplate = 3.7.1 zope.app.principalannotation = 3.7.0 zope.app.publication = 3.8.1 zope.app.publisher = 3.8.4 zope.app.renderer = 3.5.1 zope.app.rotterdam = 3.5.0 zope.app.schema = 3.5.0 zope.app.security = 3.7.3 zope.app.securitypolicy = 3.5.1 zope.app.server = 3.4.2 zope.app.session = 3.6.1 zope.app.testing = 3.7.3 zope.app.traversing = 3.4.0 zope.app.undo = 3.5.0 zope.app.wsgi = 3.6.0 zope.app.zapi = 3.4.1 zope.app.zcmlfiles = 3.5.5 zope.app.zopeappgenerations = 3.5.0 zope.app.zptpage = 3.5.0 plone.app.form = 2.0b6 plone.app.contentrules = 2.0b4 plone.app.portlets = 2.0b11 plone.app.users = 1.0b9 plone.app.contentmenu = 2.0b3

Extra Eggs

You'll also need to add extra add dependencies to your buildout that the Plone 4b4 egg doesn't require and should.

    zope.formlib

    five.formlib

    zope.app.schema


WSGI

You can take a look at my previous post for doing WSGI Zope2 for guidelines on how to setup the paste config and such.

Post

Running Plone 4 with a Zope2 WSGI

Guide to running Plone 4 with the Zope2 WSGI branch

Update

Tres has managed to merge his WSGI branch into trunk and Hanno tells me the unofficial plan is to include this in a release for Zope 2.13, in time for Plone 4.1. This is not decided upon yet though.

Overview

I was planning on implementing WSGI for Zope2 during the Penn State Symposium Sprints, and I did a few things to help out; however, Tres Seaver did most of the work on his own before and during part of the Sprint :) The rest of the time I spent just testing it out and helping with the Theme Editor sprint.

Now, it's really quite trivial to get it working now and makes all of the repoze.zope2 nonsense unneeded now. This article is just here for a reference if anyone else is interested in getting it going on their setup.

Guide

This guide assumes you have an existing Plone 4 installation to work from. I don't provide any buildouts here--just modifying an existing buildout to make it work with a branch of Zope2 and creating an ini that Paste can consume to serve WSGI.

Supplying the WSGI'd Zope2

First off, go to the src directory of the installation. If you've installed using the unified installer, that will bin in instance-home/zinstance/src or if you just used straight buildout, it'll be in instance/src. Then checkout the Zope2 branch:

svn co http://svn.zope.org/repos/main/Zope/branches/tseaver-fix_wsgi/ Zope2

Stringing up buildout

Next thing you'll need to do is modify your buildout.cfg file to add the checked out Zope 2 to the develop section:

develop = 
...
Zope2
...

Still modifying your buildout.cfg, add Paste, PasteScript, repoze.tm2 and repoze.retry to your eggs section:

eggs =
Plone
Paste
PasteScript
repoze.tm2
repoze.retry 

Again, editing your buildout.cfg, add a paster part for the paster script:

parts =
...
paster
...

[paster]
recipe = repoze.recipe.egg
scripts = paster
eggs = ${instance:eggs}

Then, you'll need to add the updated Zope2 versions for the WSGI branch. To do this, basically, just add the versions.cfg file provided in the branch after every other version file listed in the extends directive. It'll look like this:

extends =
...
src/Zope2/versions.cfg

Then run your buildout like normal:

./bin/buildout

Creating a WSGI Configuration File

You'll now need to create a WSGI configuration file. Right now, we'll just server it using the Paste server and wsgi ini configuration way. You can also do this to string up Apache's WSGI implementation but that is beyond the scope of this article.

Create a file in the instance directory called, zope2.ini with the contents of:

[app:zope]
use = egg:Zope2#main
zope_conf = %(here)s/parts/instance/etc/zope.conf

[pipeline:main]
pipeline =
    egg:paste#evalerror
    egg:repoze.retry#retry
    egg:repoze.tm2#tm
    zope

[server:main]
use = egg:paste#http
host = localhost
port = 8080

The zope_conf value in the app:zope section can be the path to any zope.conf file. I'm just exampling the standard location of it and not going through the configuration of that file itself.

Fire it up!

If all went well, you should now be able to start up your Plone 4 instance on WSGI like this:

./bin/paster serve zope2.ini

Your server should now be able to visit your site on http://localhost:8080

Caveats

I did run into a snag with the Mac OS X unified installer and the version of python it has configured. Basically, it wouldn't compile the Zope2 dependencies so I had to use my own version of python that I had compiled with the python buildout found in the plone collective svn. The bug is sort of referenced in the zope bug tracker.

Future Considerations

I'm hoping to maybe get a release with this branch implementation out--maybe as an alpha or beta release since I don't think they are planning on merging this to core any time soon; although, I really have no understanding of what that whole process is.

I'd like to see the Zope2 package implement mkzope2instance and other convenience methods so it'd be possible to install Plone 4/Zope2 without buildout at all maybe using pip. I'm looking into how this might be able to happen with a pip versions file and other things. Maybe more on this later.

 

Post any comments if you run into any issues.

Post

Plone 4 Upgrade

Thoughts on moving the server to Plone 4.

Last night I decided to take some time to move this site over to Plone 4. The repoze.zope2 build I had previously was buggy and I am starting to dislike Deliverance a bit.

Snags

  • the search didn't work because of this migration issue http://dev.plone.org/plone/ticket/10360
  • old file types aren't migrated to blob http://dev.plone.org/plone/ticket/10365
  • archetypes don't want to use the new add views--should probably report this
  • If you have the jquery UI package installed, it'll mess with some of Plone's JS for some reason--I just disabled it.
  • some other control panel entries and menu items were not fully migrated, still pointing to old templates

 

Benefits

  • FAST--running this on a linode with 512MB of ram. 2 zeo clients and with CacheSetup, this thing is very fast. Much faster than Plone 3.
  • Just more polished than Plone 3--with usability enhancements and a new theme, it's great.
  • I haven't found a product that wasn't compatible with Plone 4 yet--although, sometimes I needed to do some digging or use an svn checkout to get the compatible version

 

Post

Using the zope debug console

A short howto on using the zope debug console.

First off, go into your instance directory and issue a command like this,

./bin/instance debug

Things to Note

  • "app" is the zope root
  • You can access your Plone instances and other objects just by issuing the normal ways for traversing objects.  For instance,
    app.Plone
    app['Plone']

Login as user

from AccessControl.SecurityManagement import newSecurityManager
user = app.acl_users.getUser(user_name_or_id)
newSecurityManager(None, user.__of__(app.acl_users))

Editing Objects

So what do you do if you want to edit objects and make the changes persist?  Pretty much just treat objects like you'd normal do in a python script.  Just when you're done, make sure you issue the following commands.

import transaction
transaction.commit()
app._p_jar.sync()
Post

Automatically Pack The ZODB

This just goes over how you can automatically have the ZODB packed once a day. Does not use a cron job either.

  • Add Products.ClockServer to your egg section in buildout.cfg
  • Add something like this in your instance section
    zope-conf-additional = 
        <clock-server>
          method /pack_it_all
          period 86400
          user admin
          password password
          host localhost
       </clock-server>  
    
  • Re-run buildout
  • Start server
  • In the root of zope, create a "Script(Python)" with the id of "pack_it_all"
  • To the contents, add something like this,
    dbs=context.Control_Panel.Database
    names = dbs.getDatabaseNames()
    
    for name in names:
      if name != 'temporary':
        dbs[name].manage_pack(days=3)
        print "packed %s" % name
    
    return printed
  • This will pack all the databases you have in Zope once a day
Post

Backing Up Plone

Simple reference for backing up plone...

Installation

  • apt-get install rsync

Create SSH Keys

backup file

#!/bin/sh

#
# ZODB
#

PYTHON_DIR=/home/plone/Plone-3.1/Python-2.4/bin/python
REPOZO_FILE=/home/plone/Plone-3.1/zinstance/bin/repozo
ZODB_FILE=/home/plone/Plone-3.1/zinstance/var/filestorage/Data.fs

REMOTE_USER=nathan
REMOTE_HOST=76.222.67.149
REMOTE_PATH=/home/nathan/plonebackups/linode/datafs
LOCAL_PATH=/home/plone/Plone-3.1/zinstance/backups/

echo "backing up plone zodb"
$PYTHON_DIR $REPOZO_FILE -Bvz -r $LOCAL_PATH -f $ZODB_FILE
echo "using rsync to backup to home server"
rsync --delete -azvv -e ssh $LOCAL_PATH $REMOTE_USER@$REMOTE_HOST:$REMOTE_PATH
echo "completed successfully"

#
# BUILDOUT
#
LOCAL_BUILDOUT=/home/plone/Plone-3.1/zinstance/buildout.cfg
REMOTE_BUILDOUT=/home/nathan/plonebackups/linode/settings/buildout.cfg

echo "backing up buildout"
rsync --delete -azvv -e ssh $LOCAL_BUILDOUT $REMOTE_USER@$REMOTE_HOST:$REMOTE_BUILDOUT

#
# PRODUCTS
#
LOCAL_PRODUCTS=/home/plone/Plone-3.1/zinstance/products
REMOTE_PRODUCTS=/home/nathan/plonebackups/linode/products

echo "backing up products"
rsync --delete -azvv -e ssh $LOCAL_PRODUCTS $REMOTE_USER@$REMOTE_HOST:$REMOTE_PRODUCTS

#
# eggs
#
LOCAL_EGGS=/home/plone/Plone-3.1/buildout-cache/eggs
REMOTE_EGGS=/home/nathan/plonebackups/linode/eggs

echo "backing up eggs"
rsync --delete -azvv -e ssh $LOCAL_EGGS $REMOTE_USER@$REMOTE_HOST:$REMOTE_EGGS

 

Cron Jobs

  • crontab -e
  • 0 3 * * * sh /path/to/script
Post

Use Custom Fields in Smart Folders Plone 2.5

This shows how you can add new columns/fields to sort by and retrieve in your smart folders for new content types. This is useful if you have custom content types that you want to still use smart folders with.

Content Type

You'll need to make changes to the fields in your content type.

StringField('firstName',
    widget=StringWidget(
        label="First Name",
    ),
    index='FieldIndex:schema',
);

The part in bold is what you'll need to do for your field in the content type.

Site Setup

Next you'll need to specify in site setup of your plone instance that you want smart folders to use it.

  1. Go to site setup
  2. Click "Smart Folder Settings"
  3. Select "smart folder metadata" tab
  4. Click "all fields"
  5. Find the field you just added and give it a new name(by default it uses its getter name)
  6. Save

AddFieldSmartFolder

 

comments powered by Disqus

Navigation