Tuesday, June 2, 2009

Using Sphinx and git as a simple CMS

Suppose you want to be able to easily publish content to a website without using a CMS like Plone or Drupal. Let’s say in order to minimize the attack surface of your website you don’t want any dynamic HTML generation to take place. In this post I’ll demonstrate one way this can be done using existing tools. The main building blocks are Sphinx and git. First, I’ll set up a self-contained installation using virtualenv:
$ mkdir sphinx_playground
$ virtualenv --no-site-packages sphinx_playground
New python executable in sphinx_playground/bin/python
Installing setuptools............done.
$ cd sphinx_playground
$ source bin/activate
(sphinx_playground)$ easy_install -U Sphinx
Searching for Sphinx
Reading http://pypi.python.org/simple/Sphinx/
Reading http://sphinx.pocoo.org/
Best match: Sphinx 0.6.1
Downloading http://pypi.python.org/packages/2.6/S/Sphinx/Sphinx-0.6.1-py2.6.egg#md5=0c5baac650e48792124f71eabb0eb029
[...]
Finished processing dependencies for Sphinx
(sphinx_playground)$
  
Installing Sphinx with easy_install installs all dependencies, including docutils and Pygments. The next step is to create directories for the git repository containing files in restructuredText and for the HTML that Sphinx will produce from these files. Sphinx includes a script named sphinx-quickstart, that will do this, as well as create some initial files to get started.
(sphinx_playground)$ sphinx-quickstart
[...]
(sphinx_playground)$ cd source
(sphinx_playground)$ git init
Initialized empty Git repository in /some/path/sphinx_playground/source/.git/
(sphinx_playground)$ git add conf.py index.rst
(sphinx_playground)$ git commit -m "initial commit"
Created initial commit ce317ec: initial commit
 2 files changed, 214 insertions(+), 0 deletions(-)
 create mode 100644 conf.py
 create mode 100644 index.rst
(sphinx_playground)$
  
sphinx-quickstart asks a few questions regarding the configuration. I chose seperate directories for source and build and accepted the default for most of the other choices. The script created a makefile, which I can use to generate the HTML by typing
(sphinx_playground)$ make html
  
The HTML files will end up in build/html. By default, the directory containing the stylesheets used in these files is build/html/_static. You probably want to change how the resulting HTML looks. Luckily, Sphinx has the concept of themes. I won’t go into detail on how to use this feature, since I don’t have the slightest clue. It’s probably not very hard, though. Check out the Sphinx documentation on how to use HTML themes. What’s left is adding a post commit hook in the git repository. Creating hooks in git is just a matter of putting a file named post-commit (or pre-commit, etc.) in /path/to/repo/.git/hooks. Here’s an example for a simple post-commit script that calls make to generate the HTML:
(sphinx_playground)$ cat source/.git/hooks/post-commit
#!/bin/sh

PROJECT_DIR=/some/path/sphinx_playground

cd $PROJECT_DIR
make html
(sphinx_playground)$ chmod 744 source/.git/hooks/post-commit
  
Make sure the script is executable and then modify the file source/index.rst and commit the changes.
(sphinx_playground)$ cat index.rst
.. sphinx-playground documentation master file, created by
sphinx-quickstart on Sun May 31 23:01:49 2009.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.

Welcome to sphinx-playground's documentation!
=============================================

I'm a headline! Look at me!
---------------------------

Contents:

.. toctree::
:maxdepth: 2

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

(sphinx_playground)$ git add index.rst
(sphinx_playground)$ git commit -m "added silly headline"
sphinx-build -b html -d build/doctrees   source build/html
Making output directory...
Running Sphinx v0.6.1
loading pickled environment... not found
building [html]: targets for 1 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
writing additional files... genindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in build/html.
Created commit 43a506c: added silly headline
1 files changed, 3 insertions(+), 0 deletions(-)
(sphinx_playground)$
  
If you point your browser to /some/path/build/html/index.html you should see the new headline on the page. Updates work as well, just change the file and commit again:
(sphinx_playground)$ cat index.rst
.. sphinx-playground documentation master file, created by
sphinx-quickstart on Sun May 31 23:01:49 2009.
You can adapt this file completely to your liking, but it should at least
contain the root `toctree` directive.

Welcome to sphinx-playground's documentation!
=============================================

I'm a different headline! Look at me!
-------------------------------------

Contents:

.. toctree::
:maxdepth: 2

Indices and tables
==================

* :ref:`genindex`
* :ref:`modindex`
* :ref:`search`

(sphinx_playground)$ git add index.rst
(sphinx_playground)$ git commit -m "changed silly headline"
sphinx-build -b html -d build/doctrees   source build/html
Running Sphinx v0.6.1
loading pickled environment... done
building [html]: targets for 1 source files that are out of date
updating environment: 0 added, 1 changed, 0 removed
reading sources... [100%] index
looking for now-outdated files... none found
pickling environment... done
checking consistency... done
preparing documents... done
writing output... [100%] index
writing additional files... genindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded.

Build finished. The HTML pages are in build/html.
Created commit 0d6ef5a: changed silly headline
1 files changed, 2 insertions(+), 2 deletions(-)
(sphinx_playground)$
  
You might wonder whether Sphinx regenerates the HTML for all source files, regardless of them being out of date or not. Luckily, Sphinx is smarter than that, as you can see below:
(sphinx_playground)$ cat newfile.rst
I'm just a new file
===================

Move along, there is nothing to see here.

(sphinx_playground)$ git add newfile.rst
(sphinx_playground)$ git commit -m "added new file"
sphinx-build -b html -d build/doctrees   source build/html
Running Sphinx v0.6.1
loading pickled environment... done
building [html]: targets for 0 source files that are out of date
updating environment: 1 added, 0 changed, 0 removed
reading sources... [100%] newfile
looking for now-outdated files... none found
pickling environment... done
checking consistency... /some/path/sphinx_playground/source/newfile.rst:: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] newfile
writing additional files... genindex search
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded, 1 warning.

Build finished. The HTML pages are in build/html.
Created commit 5a287cd: added new file
 1 files changed, 5 insertions(+), 0 deletions(-)
 create mode 100644 newfile.rst
(sphinx_playground)$
  
As you can see in the output, Sphinx correctly determined, that existing source files have not changed. Also note, that Sphinx emits a warning, saying that the new file is not mentioned in a table of contents. Well, that’s pretty much it. Obviously you can enhance the post commit hook in any way you see fit. Like uploading the generated HTML to a web server via FTP or ssh, for example.

Sunday, March 29, 2009

Bembel With Care

Check out http://www.bembel-with-care.de/

went out last night to do some bemblin'
my head is huge my hands are tremblin'
the room is turning all around
despite me lying on the ground
this ain't no harmless apple juice
i got the careless bemblin' blues

i got bad news the other day
what can i do, what can i say
thursday a ramblin bemblin' man
friday no more a bembel fan
i have to say, it is the truth
i have the careless bemblin' blues

i have no memory of last night
maybe i got into a fight
maybe i was raped by a bear
why did i not bembel with care
i might have puked on my new shoes
i have the careless bemblin' blues

Saturday, March 28, 2009

Building Okular on Kubuntu Jaunty

The idea behind Okular is to have a universal document viewer, instead of one application for PDF, one for CHM, etc. Much like Preview.app on Mac OS X, only better of course. ;)
Unfortunately, on the current release of Kubuntu, Jaunty Jackalope, Okular doesn’t display CHM files. There’s a bug report about this problem with status “fix released”. This isn’t actually as good as it sounds, since the fix is to remove the claim that the kdegraphics package (which contains Okular) supports the CHM format from the package description. Reminds me of what they say about how mathematicians answer questions; the answer is both correct and useless. ;)
So, what to do if you would like your Okular to actually display CHM files? How about building it from source? I know, this is tedious and basically defeats the whole point of having a distribution with “smart” package management. It would be slightly better to build a .deb from this custom binary and install it properly. I never bother doing this, though.
Anyway, here’s what I did. First make sure that you have your system set up to compile KDE stuff. It’s a good idea to pick a directory where you put all the code that you check out from KDE repositories (I chose ~/code/kde4) and a directory where you put the binaries (i.e. ~/apps/kde4). It’s recommended to set up some shell scripts to make the process more convenient. Check out this procedure on how to build KDevelop and/or the techbase article.
Install the development packages of the libraries that enable Okular to read all those file formats.
$ sudo apt-get install libchm-dev libepub-dev libdjvulibre-dev libpoppler-qt4-dev libspectre-dev libtiff-dev
You might assume that you can go ahead and just build Okular now, as described at http://okular.kde.org/download.php. That didn’t work for me. CMake failed with the following message:
CMake Error at ui/data/icons/CMakeLists.txt:1 (kde4_install_icons):
  Unknown CMake command "kde4_install_icons".


CMake Warning (dev) in CMakeLists.txt:
  No cmake_minimum_required command is present.  A line of code such as

    cmake_minimum_required(VERSION 2.6)

  should be added at the top of the file.  The version specified may be lower
  if you wish to support older CMake versions for this project.  For more
  information run "cmake --help-policy CMP0000".
This warning is for project developers.  Use -Wno-dev to suppress it.

-- Configuring incomplete, errors occurred!
The macro CMake talks about can be found in the file KDE4Macros.cmake, which is part of the package kdelibs5-dev. I had this package installed, but apparently CMake doesn’t look for modules where kdelibs5-dev puts it. The file resides in the directory /usr/share/kde4/apps/cmake/modules/. Sounds like a directory that CMake really should be looking for modules in, doesn’t it? To fix this, I added the line
prepend CMAKE_MODULE_PATH /usr/share/kde4/apps/cmake/modules
to ~/.kde/env/kde4_setup_build_environment.sh. If you ran CMake before doing this (like I did when I got the error message shown above), make sure to delete the file kdegraphics/CMakeCache.txt. I didn’t plan to build the entire kdegraphics package, but when I ran cmake in kdegraphics/okular I got the error about the missing kde4_install_icons macro again. Therefore I followed the steps given on the Okular download page. CMake then informed me, that I’m missing some optional dependencies, which I installed with:
sudo apt-get install libqimageblitz-dev libgphoto2-dev libsane-dev libxxf86vm-dev libexiv2-dev libqca2-dev
That only set me back about 32 MB. Bring on the bloat! I’m well prepared. My root partition is 50 GB large. :) After that running
cmake -DCMAKE_INSTALL_PREFIX=$KDEDIR ..
in kdegraphics/build finally finished without complaining and make did what it’s supposed to. Unfortunately, the binary that fell out of make refused to open chm files, showing the same error message as before. I got fed up with it and stopped trying to get Okular working. I got back to it tonight, trying to figure out what went wrong. But after svn up and make install it worked. So, I don’t know what problem was, but at least it works now. :)


Friday, March 20, 2009

First Post!

Update: Since writing this post, I've moved the blog to Blogger. Turns out, I'm not fond of system administration. Besides, Blogger has shiny gadgets, widgets and whatnots.

So I decided to try this newfangled blogging thing that everyone is talking about. Maybe it's not just a fad after all. At this adoption rate expect me to start using twitter in about five years. ;)

In good internet tradition the first post is completely self-referential, describing what software I used to set this blog up.


Tools

One of the coolest features of the Django framework is its support for reusable applications. The concept of a generic foreign key is crucial for this to work, as well as some conventions like naming url patterns and adding a parameter for template names to view functions. By using some existing reusable apps, I was able to create this blog in a short amount of time while still having more flexibility for modification and extension than using a shrink-wrapped blog app like wordpress would provide. It seems to be a rite of passage for django programmers to write their own blog. I find the idea of doing that to be boring and pointless. Luckily, there's basic-apps. Since there are so many blogs built with Django out there I stole everything that seemed useful from them, like syntax highlighting and the help text for comment formatting.

Development and deployment

Using virtualenv I created a bootstrap script with an after_install function that installs pip and fetches some code from svn repositories.
import os, subprocess
def after_install(options, home_dir):
     subprocess.call([join(home_dir, 'bin', 'easy_install'), 'pip'])
     src = join(home_dir, 'src')
     if not os.path.exists(src):
         os.makedirs(src)
     curdir = os.getcwd()
     os.chdir('src')
     subprocess.call(['svn', 'co', 
      'http://django-basic-apps.googlecode.com/svn/trunk/', 'basic'])
     subprocess.call(['svn', 'co',
      'http://django-trackback.googlecode.com/svn/trunk/', 'django-trackback'])
     os.chdir(curdir)
     target = join(curdir, 'src', 'basic')
     link = join(curdir, 'lib', 'python2.5', 'site-packages', 'basic')
     os.symlink(target, link)
     target = join(curdir, 'src', 'django-trackback', 'trackback')
     link = join(curdir, 'lib', 'python2.5', 'site-packages', 'trackback')
     os.symlink(target, link)

Calling svn in a subprocess and symlinking to site-packages is pretty kludgy. Normally those repository URLs belong in the requirements.txt file. But basic-apps and django-trackback were missing setup.py files when I checked out the source. Maybe pip has a way to deal with that. Maybe I'll get around to look for this in the docs some day.

Speaking of pip, here's how my requirements.txt file looks like. I use rope for code completion in vim. Along the same line, ipython, django-extensions and Werkzeug are useful for development and debugging, but not really neccessary for running the blog.
flup
ipython
rope
markdown
docutils
BeautifulSoup
Werkzeug
pygments
-e svn+http://django-tagging.googlecode.com/svn/trunk/#egg=django-tagging
-e svn+http://code.djangoproject.com/svn/django/trunk/#egg=django-trunk
-e git+git://github.com/django-extensions/django-extensions.git#egg=django-extensions

After creating those files I created a virtualenv and ran pip to install the requirements.
$ mkdir myblog && cd myblog
$ python myblog-boot.py --no-site-packages .
[...]
$ source bin/activate
$ pip install -r requirements.txt


Django project

To glue all this goodness together I performed the following steps.

  • Create django project
  • $ django-admin startproject myblog
  • Edit settings.py, add to INSTALLED_APPS
'django.contrib.admin',<br />'django.contrib.markup',<br />'django.contrib.comments',<br />'django_extensions',<br />'tagging',<br />'basic.inlines',<br />'basic.blog',<br />'trackback',<br />

Here's my urls.py

# -*- coding: UTF-8 -*-
# vim: set fileencoding: utf-8

from django.conf import settings
from django.conf.urls.defaults import *
from django.contrib import admin
from basic.blog import views as blog_views
from basic.blog.feeds import BlogPostsFeed
from basic.blog.feeds import BlogPostsByCategory
from feeds import AllCommentsFeed
from feeds import AtomAllCommentsFeed
from feeds import AtomBlogPostsFeed
from feeds import AtomBlogPostsByCategory
from feeds import AtomCommentsForEntryFeed
from feeds import CommentsForEntryFeed
import views

admin.autodiscover()

rss_feeds = {
    'entries': BlogPostsFeed,
    'full-entries': BlogPostsFeed,
    'categories': BlogPostsByCategory,
    'entry-comments': CommentsForEntryFeed,
    'comments': AllCommentsFeed,
}

atom_feeds = {
    'entries': AtomBlogPostsFeed,
    'full-entries': AtomBlogPostsFeed,
    'categories': AtomBlogPostsByCategory,
    'entry-comments': AtomCommentsForEntryFeed,
    'comments': AtomAllCommentsFeed,
}

urlpatterns = patterns('',
    url(r'^(?P\d{4})/(?P\w{3})/(?P\d{1,2})/(?P[-\w]+)/$',
        view=blog_views.post_detail,
        name='blog_detail'),

    url(r'^(?P\d{4})/(?P\w{3})/(?P\d{1,2})/$',
        view=blog_views.post_archive_day,
        name='blog_archive_day'),

    url(r'^(?P\d{4})/(?P\w{3})/$',
        view=blog_views.post_archive_month,
        name='blog_archive_month'),

    url(r'^(?P\d{4})/$',
        view=blog_views.post_archive_year,
        name='blog_archive_year'),

    url('^$',
        view=blog_views.post_list,
        name='blog_index'),

    url('^archive/$',
        view=views.archive_list,
        name='archive_list'),

    url(r'^categories/(?P[-\w]+)/$',
        view=blog_views.category_detail,
        name='blog_category_detail'),

    url (r'^categories/$',
        view=blog_views.category_list,
        name='blog_category_list'),

    url (r'^search/$',
        view=blog_views.search,
        name='blog_search'),

    url(r'^page/(?P\w)/$',
        view=blog_views.post_list,
        name='blog_index_paginated'),

    url(r'^ping/', include('trackback.urls')),

    (r'^rss/(?P.*)/$', 'django.contrib.syndication.views.feed',
        {'feed_dict': rss_feeds}),

    (r'^atom/(?P.*)/$', 'django.contrib.syndication.views.feed',
        {'feed_dict': atom_feeds}),

    (r'^comments/', include('django.contrib.comments.urls')),

    (r'^admin/doc/', include('django.contrib.admindocs.urls')),
    (r'^admin/(.*)', admin.site.root),

    (r'^dev/random/$', views.randomize),

    (r'^pygments_lexers/$', views.pygments_lexers),
)

I wrote three little view functions of my own, but didn't bother creating an app for them. One is a primitive archive page, the second is a list of all lexers that pygments has installed and the last one is /dev/random. It will return either 4 or NINE NINE NINE NINE NINE NINE. I've linked to stackoverflow so you can see that I even stole that idea somewhere. But at least I improved it by adding the second value. It's actually using the python random module making it a true random device. ;)

For now I can only receive trackbacks/pingbacks, not send them. And I haven't really tried that, so it probably doesn't work either. I'll worry about that later, when there is some evidence that someone is actually reading this blog. The same strategy will be used for dealing with comment spam. ;)

Webserver Configuration

  • lighttpd
  • FastCGI
  • Hint: You might want to add FORCE_SCRIPT_NAME = "" to your settings.py.
TODO:

Yeah, I kinda got lazy in the end. ;) Anyway, that's it. Let's see whether I can come up with something more interesting for the next post.