Pylons Execution Analysis

See QuickWiki in action

Install and run QuickWiki according to section 2 of the QuickWiki tutorial. For convenience, here's how I installed it on Linux with SQLite.

1. Create a "~/pylons" directory and cd into it.

2. Create "bin" and "lib" subdirectories.

3. Create a file "setup.cfg" containing:

   [easy_install]
   install_dir = ~/pylons/lib
   script_dir = ~/pylons/bin
   zip_ok = 0
   

   ("zip_ok = 0" forces packages to be installed as directories rather than zip files, so we can inspect their source code easily.)

4. Create a file "environ.sh" containing:

   export PATH=~/pylons/bin:$PATH
   export LIB=~/pylons/lib
   export PYTHONPATH=$LIB
   

5. Run "source environ.sh". Later when you want to run QuickWiki again, you'll have to "cd ~/pylons" and run "source environ.sh" to initialize the shell session.

6. If you haven't set up easy_install yet, download ez_setup.py and run it as root. See the Easy Install manual for details. (We won't use the "Custom Install Locations" described in the manual; our "setup.cfg" does the equivalent.)

7. Run "easy_install QuickWiki". This will install several Python packages according to the following dependency graph (the dependency relationships may not be 100% accurate):

   QuickWiki (0.1.2)
       Pylons  (0.9.3)
           Beaker (0.6.1)
           FormEncode (0.6)
           Myghty (1.1)
               MyghtyUtils (0.5.2)
           Paste(1.0)
           PasteDeploy (1.0)
               Cheetah 1.0
           PasteScript (1.0)
           Routes (1.5.2)
           simplejson (1.4)
           WebHelpers (0.2.2)
       SQLAlchemy (0.3.1)
   

   Important

   A module pylons.middleware in the "Pylons" egg is located at "~/pylons/lib/Pylons-VERSION-pyVERSION.egg/pylons/middleware.py".

8. Install SQLite if you don't already have it.

9. Run "easy_install pysqlite" (2.3.2).

10. Run "paster make-config QuickWiki quick-wiki.ini". This creates quick_wiki.ini in the current directory.

11. Edit 'quick_wiki.ini'. Change the "sqlalchemy.dburi" line to:

    sqlalchemy.dburi = sqlite:////PATH/TO/CURRENT/DIRECTORY/quick_wiki.sqlite
    

    Don't change the "sqlobject.dburi" line by mistake; it won't be used. If desired, uncomment the "sqlalchemy.echo" line. Comment the "set debug = false" line so we get error tracebacks in the browser. (This is not appropriate for a production system but we're just testing right now.)

12. Run "paster setup-app quick-wiki.ini". This sets up the database tables required by the application.

13. Run paster serve quick-wiki.ini. Point your web browser to "http://localhost:5000/" and add some wiki pages. Note the drag-n-drop AJAX feature for deleting pages. Press ctrl-C on the console when you get bored.

    Note

    The tutorial recommends "paster serve --reload quick-wiki.ini". This automatically reloads modules when they are modified so you don't have to restart the server, which is useful in development. However, I found it to leave threads running after I stopped the server, so I got an "address is already in use" error on port 5000 when I started it again. This may depend on the OS. I had to manually find these threads using "lsof -i @:5000" and kill the first one. On these systems it's easier to skip "--reload" and just restart the server manually after making changes.

# @@MO: Deleted due to DeprecationWarning
#app = pylons.wsgiapp.PylonsApp(config, helpers=quickwiki.lib.helpers)
#g = app.globals

# @@MO: Added per DeprecationWarning advice
import quickwiki.lib.app_globals as app_globals
import quickwiki.lib.helpers
app = pylons.wsgiapp.PylonsApp(config, helpers=quickwiki.lib.helpers,
    g=app_globals.Globals)
# @@MO: End of additions.

Application startup (PasteScript)

Note

Substitute your actual Python library directory for $LIB, and your program directory for $BIN. If you followed the checklist above, $LIB is "~/pylons/lib" and $BIN is "~/pylons/bin".

When you run paster serve --reload quick_wiki.ini, it runs the "paster" executable. This program consists of three short lines that will look unfamiliar to those who haven't started using Python eggs:

__requires__ = 'PasteScript==1.0'
import pkg_resources
pkg_resources.run_script('PasteScript==1.0', 'paster')

This says to run a script "paster" located in version 1.0 of an egg "PasteScript". The actual script location is "$LIB/PasteScript-1.0-pyVERSION.egg/EGG-INFO/scripts/paster", but we'll skip the details of how pkg_resources finds it.

The "paster" script is short too. It inserts the paste package into sys.path and calls paste.script.command.run(). I added print statements to the paste.script.command module to to decipher it. Here's a simplified description

1. paste.script.command.run() parses the command-line options into a command name "serve" with options ["--reload", "quick-wiki.ini"].

2. It calls get_commands(), which calls paste.script.pluginlib.load_global_commands()`. This uses a pkg_resources feature:

   pkg_resources.iter_entry_points("paste.global_paster_command")
   

   This reads a file "$LIB/PasteScript-1.0-pyVERSION/EGG-INFO/entry_points.txt". The file is in .ini format and contains among other things:

   [paste.global_paster_command]
   serve=paste.script.serve:ServeCommand [Config]
   ... other commands like "make-config", "setup-app", etc ...
   

   This defines a paster command "serve" in class "ServeCommand" in module "paste.script.serve". [@@MO: What's the "Config" part for?] The iter function yields a parse object whose .load() method returns the ServeCommand class.

3. command.run() function then calls invoke(), which effectively does: ServeCommand(["--reload", "quick-wiki.ini"]).run(). This in turn calls ServeCommand.command(), which handles daemonizing and other top-level stuff. Since our command line is short there's no top-level stuff to do. It creates 'server' and 'app' objects based on the config file, and calls server(app).

Loading the server and application (PasteDeploy)

This happens during step 3 of the application startup. We need to find and instantiate the WSGI application and server based on the config file. The application is our QuickWiki application. The server is Paste's built-in multithreaded HTTP server. A simplified version of the code is:

# Inside paste.script.serve module, ServeCommand.command() method.
from paste.deploy.loadwsgi import loadapp, loadserver
server = loadserver(uri="config:quick-wiki.ini", name=None,
    relative_to="/DIR/CONTAINING/CONFIG/FILE")
app = loadapp(uri="config:quick-wiki.ini", name=None,
    relative_to="/DIR/CONTAINING/CONFIG/FILE")

loadserver() and loadapp() are defined in module paste.deploy.loadwsgi. The code here is complex and delves into the details of Python eggs and entry points, so we'll just look at its general behavior. Both functions see the "config:" URI and read our config file. Since there is no server name or app name they both default to "main". Therefore loadserver() looks for a "[server:main]" section in the config file, and loadapp()` looks for "[app:main]". Here's what they find in "quick-wiki.ini":

[server:main]
use = egg:Paste#http
host = 0.0.0.0
port = 5000

[app:main]
use = egg:QuickWiki
sqlalchemy.dburi = sqlite:////..../quick_wiki.sqlite
...

The "use =" line in each section tells which object to load. The other lines are configuration paramaters for that object, or for plugins that object is expected to load.

Instantiating the application (QuickWiki)

loadapp() calls quickwiki.make_app(), imported from $LIB/QuickWiki*.egg/quickwiki/config/middleware.py:

def make_app(global_conf, **app_conf):
    config = load_environment(global_conf, app_conf)
    config.init_app(global_conf, app_conf, package="quickwiki")
    app = pylons.wsgiapp.PylonsApp(config, helpers=..., g=...)
    app = ConfigMiddleware(app, ...)

quickwiki.config.environment.load_environment() returns a pylons.config.Config object containing all our application's configuration information, including Pylons standard path locations. See the Config docstring in $LIB/Pylons*.egg/config.py. The attributes of interest here are:

.global_conf
    Dict representing the "[DEFAULT]" section of the config file.
.app_conf
    Dict representing the "[app:main]" section of the config file.

config.init_app() sets up the application's logging and error reporting, and sets some variables for Myghty. It sets config.app_conf["package"] to the 'package' argument, presumably so the application can remind itself what its own top-level package name is.

The first app assignment creates a Pylons WSGI application, whose .__call__() method will be called by the WSGI server for each request. This in turn calls .resolve() to determine the controller (presumably using Routes to find a class in QuickWiki), and .dispatch() to create the response. Dispatch calls the controller and returns the response. PylonsApp also sets up the global object 'g' used by the controller, and manages the session and cache.

The other app assignments wrap the application in successive layers of middleware. We'll skip looking at these until we analyze an actual request.

Anatomy of a request

Let's say you run the demo and choose the "Title List" link. It shows an index of all pages, with a Javascript feature that allows you to drag undesired links to a "delete box". We'll look at how this page was created in Pylons.

1. server(app) is running, called in the ServeCommand.command() method in module paster.serve.

2. server is actually paste.httpserver.serve(), which trivially calls server_runner() in the same module. The use_threadpool arg defaults to true so the actual server is a WSGIThreadPoolServer, which has the following inheritance:

   SocketServer.BaseServer     # In SocketServer.py in Python stdlib.
   SocketServer.TCPServer
   BaseHTTPServer.HTTPServer  # In BaseHTTPServer.py in Python stdlib.
   paste.httpserver.SecureHTTPServer  # Adds SSL (HTTPS).
   paste.httpserver.WSGIServerBase    # Adds WSGI.
   paste.httpserver.WSGIServer        # Adds multithreading.
   paste.httpserver.WSGIThreadPoolServer  # Adds thread pool.
   

   (SSL is not enabled in our example.)

4. Right now the server is waiting for a request, following this call stack:

   # In paste.httpserver.serve(), calling 'server.serve_forever()'
   ThreadPoolMixIn.serve_forever()  # Defined in paste.httpserver.
   -> TCPServer.handle_request()    # Called for every request.
   -> WSGIServerBase.get_request()
   -> SecureHTTPServer.get_request()
   -> self.socket.accept()          # Defined in stdlib socket module.
   

5. The request comes in and self.socket.accept() returns a new socket for the connection. TCPServer.handle_request() continues. It calls ThreadPoolMixIn.process_request() which puts the request in a thread queue:

   self.thread_pool.put(
       lambda: self.process_request_in_thread(request, client_address))
       # 'request' is the connection socket.
   

   The thread pool is defined in the ThreadPool class. It spawns a number of threads which each wait on the queue for a callable to run. In this case the callable will be a complete Web transaction including sending the HTML page to the client. Each thread will repeatedly process transactions from the queue until they receive a special value ordering them to exit.

   The main thread goes back to listening for other requests, so we're no longer interested in it.

6. Thread #2 pulls the lambda out of the queue and calls it:

   lambda
   -> ThreadPoolMixIn.process_request_in_thread()
   -> BaseServer.finish_request()
   -> self.RequestHandlerClass(request, client_address, self)  # Instantiates.
      The actual class is paste.httpserver.WSGIHandler; i.e.,
      the 'handler' variable in serve().
   

7. The request handler takes over:

   SocketServer.BaseRequestHandler.__init__(request, client_address, server)
   -> WSGIHandler.handle()
   -> BaseHTTPRequestHandler.handle()  # In stdlib BaseHTTPServer.py
   -> BaseHTTPRequestHandler.handle_one_request()
      Reads the command from the socket.  The command is
      "GET /page/list HTTP/1.1" plus several HTTP headers.
      BaseHTTPRequestHandler.parse_request() parses this into attributes
      .command, .path, .request_version, and .headers.
   -> BaseHTTPRequestHandler.do_GET()
      This method is overridden in WSGIHandler, and is actually
      WSGIHandlerMixin.wsgi_execute().  POST would be the same.
   -> WSGIHandlerMixin.wsgi_setup()
      Creates the .wsgi_environ dict.
   

   The WSGI environment dict is described in PEP 333, the WSGI specification. It contains various keys specifying the URL to fetch, query parameters, server info, etc. All keys required by the CGI specification are present, as are other keys specific to WSGI or to paricular middleware. The application will calculate a response based on the dict. The application is wrapped in layers of middleware -- nested function calls -- which modify the dict on the way in and modify the response on the way out.

8. The request handler calls the application thus:

   # In WSGIHandlerMixin.wsgi_execute(), simplified.
   result = app(wsgi_environ_dict, wsgi_start_response)
   

   wsgi_start_response is a callable mandated by the WSGI spec. The application will call it to specify the HTTP headers. The return value is an iteration of strings, which when concatenated form the HTML document to send to the browser. Other MIME types are handled analagously.

9. The application, as we remember, was returned by quickwiki.config.middleware.make_app(). It's wrapped in several layers of middleware, so calling it will execute the middleware in reverse order of how they're listed in $LIB/QuickWiki*.egg/config/middleware.py:

   • The RegistryManager middleware makes certain module globals both thread-local and middleware-local. Pylons and Myghty depend on certain module globals containing the context of the current request. (Defined in the paste.registry module.)

   • ErrorDocuments intercepts any HTTP error status returned by the

     application (e.g., "Page Not Found", "Internal Server Error") and sends another request to the application to get the appropriate error page to display instead. (Defined in pylons.middleware.)

   • Cascade lists a series of applications which will be tried on order. (Defined in paste.cascade.):

     1. The first is StaticURLParser (defined in paste.urlparser. It tries to return a file under our static directory ($LIB/QuickWiki*.egg/quickwiki/public/). For QuickWiki this is used only for the stylesheet.
     2. If the first application returns "Not Found", the cascader tries the second application, StaticJavascripts (defined in pylons.middleware). This tries to return a Javascript script in the WebHelpers package (defined in webhelpers). These include a variety of Javascript tools including a full port of the Ruby on Rails utilties. See the WebHelpers site for details.
     3. If that returns "Not Found" too, the cascader falls back to the third application, your QuickWiki app. But there's still some other middleware wrapped around the app....

   • ErrorHandler sends a nice helpful traceback to the browser if the app raises an exception. It's active only if "debug" is true in the config file. (Defined in pylons.middleware.)

   • httpexceptions (defined in paste.httpexceptions) converts HTTP exceptions raised into proper WSGI responses.

   • User-defined middleware. If the user added any middleware in make_app() it would be executed here.

   • ConfigMiddleware (defined in paste.deploy.config) makes a paste.config key in the WSGI environ dict that contains the effective values in the config file. This is a dict containing the merger of the "[app:main]" and "[DEFAULT]" sections of the config file.

   • The innermost middleware calls the PylonsApp instance it was initialized with.

   • Surprise! PylonsApp is itself middleware. It calls the pylons.wsgiapp.PylonsBaseWSGIApp instance in its self.app attribute.

10. PylonsBaseWSGIApp is a middleware too. Its .__call__() method does:

    self.setup_app_env(environ, start_response)
    controller = self.resolve(environ, start_response)
    response = self.dispatch(controller, environ, start_response)
    return response
    

    .setup_app_env() modifies the environ dict.

    .resolve() calculates the controller class using Routes, our package name ("quickwiki"), and a routing map defined in quickwiki.config.routing. Here's QuickWiki's routing map (simplified):

    map.connect('error/:action/:id', controller='error')
    map.connect(':controller/:action/:title', controller='page', action='index', title='FrontPage')
    map.connect(':title', controller='page', action='index', title='FrontPage')
    map.connect('*url', controller='template', action='view')
    

    The first arg is a URL pattern to match. "Controller" is a class in a same-name module under quickwiki.controllers. (Pylons converts the URL to TitleCase and "/" to ".".) The action is a method in that class. The keyword arguments are variables which will be available to the controller. ":var" means take the variable's value from the URL. If the URL is too short to contain that part, the keyword arguments provide defaults. Routing maps can contain other features not covered here.

    .resolve() invokes Routes and gets back a match dict for the requested URL. The match dict for our URL "/page/list" is:

    {'action': 'list', 'controller': 'page', 'title': 'FrontPage'}
    

    Note how it matched the second routing rule, and the title defaulted to the keyword arg. .resolve() puts the match dict under WSGI environ key "pylons.routes_dict". Then it looks for a module quickwiki.controller.page, converting any "/" in the 'controller' key to ".". It imports this module and looks for a class PageController inside it, using pylons.util.class_name_from_module_name() to TitleCase the class name to TitleCase and replace any "-" with "_" (defined in ).

    Hint

    Put "print map" in pylons.wsgiapp line 176 (in PylonsBaseWSGIApp.resolve() after "match = config.mapper_dict" to see the routing variables for every request.

    .dispatch instantiates the controller class and calls in the WSGI manner. If the controller does not exist (.resolve() returned None), raise HTTPNotFound.

11. quickwiki.controllers.page.PageController does not have a ``.__call__() method so control falls to its grandparent, pylons.controllers.WSGIController. It looks up the action method .index() defined in PageController and calls it. The action method may have any number of positional arguments as long as they correspond to variables in the routing rule. In addition, the c global will contain all variables as attributes. If an action method name starts with "_", it's private and HTTPNotFound is raised.

12. If the controller has .__before__() and/or .__after__() methods, they are called before and after the action, respectively. These can perform authorization, lock OS resources, etc.

13. The action method returns a pylons.Response object (imported from paste.wsgiwrappers.WSGIResponse). We'll look at action methods more closely in Pylons actions tips.

14. WSGIController.__call__() continues, converting the Response object to an appropriate WSGI return value. (First it calls the start_response callback to specify the HTTP headers, then it returns an iteration of strings. The Response object converts unicode to utf-8 encoded strings, or whatever encoding you've specified in the config file.)

15. The stack of middleware calls unwinds, each modifying the return value and headers if desired.

16. The server receives the final return value. (We're way back in paste.httpserver.WSGIHandlerMixin.wsgi_execute() now.) The outermost middleware has called back to server.start_response(), which has saved the status and HTTP headers in .wsgi_curr_headers. .wsgi_execute() then iterates the application's return value, calling .wsgi_write_chunk(chunk) for each encoded string yielded. .wsgi_write_chunk() formats the status and HTTP headers and sends them on the socket if they haven't been sent yet, then sends the chunk. The convoluted header behavior here is mandated by the WSGI spec.

17. Control returns to BaseHTTPRequestHandler.handle(). .close_connection is true so this method returns. The call stack continues unwinding all the way to paste.httpserver.ThreadPoolMixIn.process_request_in_thread(). This calls SocketServer.close_request(), which does nothing.

18. The request lambda finishes and control returns to ThreadPool.worker_thread_callback(). It waits for another request in the thread queue. If it receives a special shutdown value, thread #2 dies.

What's in the QuickWiki egg?

The QuickWiki-0.1.2-py2.4.egg contains two subdirectories: quickwiki and EGG-INFO.

What's in the setup.py?

Here's the setup.py file in QuickWiki's source. To download QuickWiki's source from its Subversion repository, run "svn co http://pylonshq.com/svn/sandbox/examples/QuickWiki".

from setuptools import setup, find_packages

setup(
    name='QuickWiki',
    version="0.1.1",
    description="Result of following the Pylons 0.9 Tutorial",
    url="http://www.pylonshq.com/docs/0.9/quick_wiki.html",
    #author="",
    #author_email="",
    install_requires=["Pylons>=0.9", "docutils==0.4", "SQLAlchemy>=0.2.6"],
    packages=find_packages(),
    include_package_data=True,
    test_suite = 'nose.collector',
    package_data={'quickwiki': ['i18n/*/LC_MESSAGES/*.mo']},
    entry_points="""
    [paste.app_factory]
    main=quickwiki:make_app
    [paste.app_install]
    main=paste.script.appinstall:Installer
    """,
)

It begins by importing setuptools rather than distutils. Distutils is included in current versions of Python but setuptools is not, so you must install setuptools via "ez_setup.py" before installing QuickWiki. We installed setuptools back at the beginning of this article.

The first several arguments give basic information about the package. "install_requires" lists other package/versions QuickWiki depends on; "easy_install" will install these if they're not already present. "package_data" lists some files to include in the installed package which would not otherwise be included.

"entry_points" is the most interesting argument. This is how the installer created the QuickWiki*.egg/EGG-INFO/entry_points.txt file we saw in the "Application loading" section above. The string here matches the file's contents exactly.

The "setup.cfg" file in the same directory is also worth perusing. Most of it tells Pudge how to generate documentation from the QuickWiki source.