Turbogears, Lighttpd and WSGI for real dummies

Just so to set the records straight, I’m the real dummy here, as you would obviously see.

I absolutely have no experience with python, lighttpd, turbogears or wsgi, fcgi but I was given a task to set it up on a server. All I used was some online posts and common sense. Normally I wouldn’t have to post a blog entry about it, most of the stuff is already out there. On this occasion however, I felt there’s some missing documentation, or else – simply because I don’t know anything about any of these it was more tricky than it usually is. Perhaps it’s one indicator how this all seems from the sysadmin perspective, as opposed to a turbogears developer.

I’m not going to cover every step, because I feel the basics should be easy enough to follow if you’ve done some linux server setup. Documentation is pretty clear on the basic install stuff. Nevertheless, some things aren’t too clear, and I hope to cover at least to some extent. For example, I wasn’t quite sure whether I should rely on the debian packages or the python easy_setup ones, that can be retrieved automagically. I guess easy_setup is a good bet for the latest versions, as I’m using Debian 4 stable (etch), and try to avoid mixing stable and unstable, test – mostly because I’m not confident enough about not screwing things up

Ok, down to business then
I’m running:

Debian stable (Etch)
python 2.4
python setup tools (debian package)
Turbogears 1.0.1 (debian package)
Lighttpd (debian package)
flup and scgi installed via easy_setup

First thing I wasn’t sure about, is where to place the turbogears apps. I then decided to create /var/turbogears and put them under there. I don’t think it makes a huge difference, but seemed sensible enough.

I then created a turbogears app using
/var/turbogears# tg-admin quickstart
Enter project name: newproject
Enter package name [newproject]:
Do you need Identity (usernames/passwords) in this project? [no]
…

so now there’s a new project called newproject under /var/turbogears/newproject.

I then tried to follow this link (or this one – which is essentially a mirror), to have it all served by lighty and wsgi, but then started having problems… Well – it simply wouldn’t work

I wasn’t very surprised considering all I did was a copy&paste job, without really understanding what’s going on. Getting to the root cause was somehow more of a challenge.

I wish I noted all the tiny different things I tried, but I’ll try to describe a couple of ones that helped along the way:

enabling more detailed debugging in lighttpd.conf:

debug.log-request-handling = "enable"
scgi.debug = 1

Another thing that took me ages to figure out from reading on the links above, what is actually /full/path/to/your/application supposed to point at?? I tried /var/turbogears/newproject/newproject-start-scgi.py (which is the new script I created by following the instructions) and /var/turbogears/newproject/ etc.

the correct answer is /var/turbogears/newproject/newproject (i.e. the folder where the application is UNDER the project, they are named the same when using the tg-admin quickstart)

so my lighttpd configuration looks like (after resolving it, I obviously took the debug statements out):

server.document-root = “/var/turbogears/newproject/newproject”
#debug.log-request-handling = “enable”
#scgi.debug = 1

$HTTP[“url”] !~ “^/static/” {
scgi.server = (
“/” =>
( “127.0.0.1” =>
(
“host” => “127.0.0.1”,
“port” => 4000,
“check-local” => “disable”
)
)
)
}

That didn’t solve the problem unfortunately. I was getting lots of these errors (in /var/log/lighttpd/error.log), even when running the new modified turbogears script (newproject-start-scgi.py):

2007-09-10 16:38:57: (response.c.258) URI-path     :  /
2007-09-10 16:38:57: (mod_scgi.c.2760) handling it in mod_scgi
2007-09-10 16:38:57: (response.c.372) -- before doc_root
2007-09-10 16:38:57: (response.c.373) Doc-Root     : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.374) Rel-Path     : /
2007-09-10 16:38:57: (response.c.375) Path         :
2007-09-10 16:38:57: (response.c.423) -- after doc_root
2007-09-10 16:38:57: (response.c.424) Doc-Root     : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.425) Rel-Path     : /
2007-09-10 16:38:57: (response.c.426) Path         : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.443) -- logical -> physical
2007-09-10 16:38:57: (response.c.444) Doc-Root     : /var/turbogears/newproject/
2007-09-10 16:38:57: (response.c.445) Rel-Path     : /
2007-09-10 16:38:57: (response.c.446) Path         : /var/turbogears/newproject/
2007-09-10 16:38:57: (mod_scgi.c.1325) connect delayed, will continue later: 7
2007-09-10 16:38:57: (mod_scgi.c.2239) got proc: 7 0  4000 1
2007-09-10 16:38:57: (mod_scgi.c.1179) release proc: 7 0
2007-09-10 16:38:57: (mod_scgi.c.2029) proc: 127.0.0.1 4000  1 0 0 0
2007-09-10 16:38:58: (mod_scgi.c.2029) proc: 127.0.0.1 4000  1 0 0 0
2007-09-10 16:38:59: (mod_scgi.c.2029) proc: 127.0.0.1 4000  1 0 0 0

And at some other point (apologies, don’t remember exactly when was what)

2007-09-10 16:49:48: (response.c.206) URI-path     :  /
2007-09-10 16:49:48: (response.c.207) URI-query    :
2007-09-10 16:49:48: (response.c.257) -- sanatising URI
2007-09-10 16:49:48: (response.c.258) URI-path     :  /
<strong>2007-09-10 16:49:48: (mod_scgi.c.2825) no fcgi-handler found for: /</strong>
2007-09-10 16:49:48: (mod_scgi.c.2029) proc: 127.0.0.1 4000  5 0 0 0
2007-09-10 16:49:49: (mod_scgi.c.2029) proc: 127.0.0.1 4000  5 0 0 0

After very long googling experience, I came across this link! and then modified my turbogears startup script as it recommended, but with the following modifications where it said modulename=”gtextranet.config.app” I changed it to newproject.config (without the .app), and obviously the line that now says from newproject.controllers import Root

#!/usr/bin/env python
import pkg_resources
pkg_resources.require("TurboGears")

import turbogears
import cherrypy
cherrypy.lowercase_api = True

from os.path import *
import sys

# first look on the command line for a desired config file,
# if it's not on the command line, then
# look for setup.py in this directory. If it's not there, this script is
# probably installed
if len(sys.argv) > 1:
    turbogears.update_config(configfile=sys.argv[1],
        modulename="newproject.config")
elif exists(join(dirname(__file__), "setup.py")):
    turbogears.update_config(configfile="dev.cfg",
        modulename="newproject.config")
else:
    turbogears.update_config(configfile="prod.cfg",
        modulename="newproject.config")

from newproject.controllers import Root
from cherrypy._cpwsgi import wsgiApp
#from paste.util.scgiserver import serve_application
from flup.server.scgi  import WSGIServer
#from flup.server.scgi_fork  import WSGIServer

port = 4000
if len(sys.argv) > 2:
    port = int(sys.argv[2])

cherrypy.config.update({
'global': {
    'autoreload.on': False,
}})
cherrypy.root = Root()
cherrypy.server.start(initOnly=True, serverClass=None)
#serve_application(application=wsgiApp, prefix='/', port=port)
WSGIServer(application=wsgiApp, bindAddress=('localhost', port)).run()

Don’t ask me what made the difference with this script, as I said before I know nothing about python or turbogears, but it seemed to work.
I had to launch the new script

/var/turbogears/newproject #./newproject-start-scgi.py

last thing I did was to set up a bash script to launch this newproject-start-scgi.py script when the system boots. I used a rather basic bash script placed under /etc/init.d:

#!/bin/sh
SCRIPT="/var/turbogears/newproject/newproject-start-scgi.py"
case "$1" in
start)  echo "STARTING $SCRIPT"
        $SCRIPT &
        ;;
stop)   echo "STOP $SCRIPT"
        pid=`/bin/ps -ef | /usr/bin/awk '($9~"newproject" && $3==1){printf " "$2}'`
##
## please check if this computation of pid is correct on your system
##
        if [ -n "$pid" ]; then
            /bin/kill $pid
        fi
        ;;
*)      echo "Usage: $0 {start|stop}"
        exit 2
        ;;
esac
exit 0

Even this script is probably crap and shows my lack of refined (or even basic) bash scripting experience, but it seemed to do the trick too