opensource.activestate.com
About opensource.activestate.com
/ Projects / Papers /
ASPN ActiveState
PyPerl
PyXPCOM
Smoke
Getting Started with Smoke

Getting Started with Smoke

Introduction

Smoke is system for managing build, test and performance data for software projects. It is similar to Tinderbox, developed and used by Mozilla. (Could also look at the Snake Farm). Smoke consists of server and client parts. A Smoke server provides (1) an interface with which clients can submit data to a backend database and (2) a web front end for reporting submitted build, test and performance data.

System Requirements

The Smoke server software is implemented as a Python CGI script using a MySQL database backend. It is currently being developed on Windows but there is no technical reason it should not run on Linux (or, indeed, on any platform with a web server that can execute Python CGI scripts and connect to a MySQL database).

The Smoke client is currently distributed as a Python package and script (providing a Python module interface and a command-line interface), so requires a Python (version 2.2.2 or greater -- email.MIMEMultiPart was added in Python 2.2.2) installation. It is currently tested on Windows and Linux. In the future a free-standing command-line interface will be provided so that clients will not require a Python installation. It is possible that a Perl module interface will be provided as well.

Installing the Smoke Client

  1. Get the latest Smoke source tree. Current this is only available via Perforce:

    p4 sync //depot/main/support/smoke/...
    
  2. Change to the smoke client source dir:

    cd src/client
    
  3. Build and install (note you must have MSVC6 setup and Python 2.2. on your PATH):

    python build.py launcher
    python build.py install
    

TBD: Give a command here to have the user run to verify that smoke is up and running.

TBD: Start distributing a Smoke Client installer.

Installing the Smoke Server

The Smoke server is a Python CGI script so requires a Web server that can execute Python CGI scripts. Presumably any such web server will do, but we develop using Apache and ActivePython 2.2 so any specific instructions will be for this configuration.

Nominally, installing the smoke server involves the following simple steps (run from the base smoke source directory):

python configure.py
cd src/server
python build.py install

But before you can do that you must setup a somewhat daunting number of applications, servers and Python packages as described below. Good luck! :)

Apache

Download Apache and install it if you do not already have it. We currently use Apache version 1.3.x, but I know of no reason why Smoke should not work with Apache 2.0.x. Configuring Apache for general use is beyond the scope of this document, please refer to the Apache Documentation for help with that.

Python

The Smoke server requires Python 2.2 or greater. We use ActivePython, but the reference Python from python.org should work too. Here after, any reference to python is the Python interpreter you just installed.

Chaco/Kiva

For displaying changes in performance tests, Smoke uses the Chaco plotting program which is part of SciPy. As of this writing (Scipy version 0.2.0), Smoke requires the following packages from the CVS version of scipy layered on top of the full Scipy install:

  • traits
  • kiva
  • chaco
  • freetype

The next release of Scipy will include the needed changes.

In the meanwhile, you need to:

  1. Install the SciPy package. The SciPy downloads page has a binary installer for Python 2.2 installations on Windows. I have not tried this on Linux yet.

  2. Install Numeric 22.0. You can download the appropriate package from Numeric's downloads page. Note that Numeric 22.0 should be installed and not Numeric 23.0. [David, is Numeric 23.0 really a breaker? --TM]

  3. In a temporary directory, run the following commands:

    cvs -d :pserver:anonymous@scipy.org:/home/cvsroot login 
    cvs -d :pserver:anonymous@scipy.org:/home/cvsroot co chaco_all
    cd chaco
    python setup.py install
    cd ..
    

If you want to generate bitmaps, you need to also install PIL. (ActivePython users can just type pyppm install PIL).

If you want to generate SVG output (highest quality), you need a browser with an SVG viewer. Currently the only working SVG plugin is the Adobe SVG Plugin for Internet Explorer. Mozilla has plans for SVG support, but the last build tested failed miserably. Lobby Mozilla to get them to support SVG!.

MySQL

The Smoke server uses a MySQL backend. You must:

Download MySQL and install it (note that installing to a path with spaces may cause you unnecessary grief). The following are the steps I use to setup MySQL on Windows (refer to http://www.mysql.com/doc/en/Windows.html for details):

  1. Create %windir%\my.ini with the following content, presuming that you installed MySQL to c:\mysql:

    [mysqld]
    basedir=c:/mysql
    datadir=c:/mysql/data
    

    Note the use of Unix-style directory slashes in the configuration file.

  2. Install MySQL as a service by running the following:

    cd c:\mysql\bin
    mysqld-max-nt.exe --install
    
  3. Start the service by running:

    net start MySQL
    

    This service will be automatically shutdown and restarted whenever the computer reboots.

To install the Python/MySQL interface download mysql-python from here.

TBD: Setting up a MySQL database for Smoke.

Python CGI and Quixote

Smoke uses the Quixote Web Application Framework. Don't worry though, Quixote is a cinch to install. You must install at least Quixote version 0.6 (in beta at the time of this writing). Smoke will probably not work with Quixote version 0.5.x. To install Quixote, unpack the source tarball and run:

python setup.py install

Configure your browser for running Python CGI scripts. For starters, Apache users can just add these lines to their httpd.conf file:

ScriptAlias /cgi-bin/ "C:/path/to/your/cgi-bin/"
AddHandler cgi-script cgi
PassEnv PYTHONPATH 
SetEnv PYTHONUNBUFFERED 1

TBD: investigate setting up and using FastCGI.

phpMyAdmin

phpMyAdmin is a tool for administering a MySQL database via a web browser. It is a fantastic tool, and while it is not at all necessary for using Smoke you will find that it is helpful for browsing the guts of your Smoke database if you are ever so inclined. Here are the steps I took to get phpMyAdmin up and running on Windows (see PHP's install.txt for more details):

  1. Download PHP. (NOTE: Don't use the Windows "installer", use the "zip package". The "installer" package does not include some stuff that you may want.) Unzip the package to C:\ and rename the base directory to C:\PHP.

  2. Copy one of the starter C:\PHP\php.ini-* files to %windir%\php.ini.

  3. Copy C:\PHP\sapi\php4apache.dll up one directory to C:\PHP. (It must be in the same directory as php4ts.dll.)

  4. Add the following lines to your httpd.conf:

    LoadModule php4_module c:/PHP/php4apache.dll
    AddType application/x-httpd-php .php
    
  5. Restart Apache.

  6. Test your configuration by creating a test PHP file, say info.php in your Apache document root with the following content:

    <?php phpinfo();?>
    

    And then browse to: http://localhost/info.php

    You should get a page with a number of tables giving a dump of your PHP setup status. Search for "mysql" to ensure that MySQL support is included.

  7. Download phpMyAdmin. Unzip the package to some directory you choose. Edit config.inc.php as appropriate for your setup. Setup an alias in your httpd.conf:

    Alias /mysqladmin "D:/trentm/www/phpMyAdmin"
    

    And be sure that you have "index.php" as a DirectoryIndex:

    <IfModule mod_dir.c>
        DirectoryIndex index.html index.php
    </IfModule>
    

    Restart Apache and browse to: http://localhost/mysqladmin/

Smoke Server

Now that you have your web server, database and Python CGI setup, the Smoke server is made up of two pieces: (1) the smokeserver Python package and (2) the smokeserver.cgi driver script.

To install the smokeserver Python package:

cd src/server   # TBD: implement this, it doesn't exist yet
python setup.py install

To install and configure the smokeserver driver CGI script (TBD: write these files):

  1. Copy src/server/smokeserver.cgi and src/server/smokeserver.conf into your server's cgi-bin directory.
  2. Configure smokeserver.conf as appropriate for your setup.

You should now be able to run the smokeserver CGI script directly by opening: http://localhost/cgi-bin/smokeserver.cgi/

This URL is not very user friendly though, so it is preferable to setup your web server to rewrite URL references to "/cgi-bin/smokeserver.cgi/..." to something more user-friendly, say "/smoke/...". On Apache 1.3 this can be done by adding the following httpd.conf lines:

RewriteEngine On
RewriteRule ^/smoke$ /smoke/ [redirect=permanent]
# You may need 'passthrough' for Apache 1.3 to allow your
# ScriptAlias directive for /cgi-bin/ to work.
RewriteRule ^/smoke(/.*) /full/path/to/cgi-bin/smokeserver.cgi$1 [last]

If you are using VirtualHost's sections in your Apache configuration then you must report the RewriteRule's for each such section. If Apache complains about the RewriteEngine On line with:

Invalid command 'RewriteEngine', perhaps mis-spelled or defined...

you may have to configure mod_rewrite. For Apache 1.3.27 on Windows this meant adding the following two lines each in the appropriate place to httpd.conf (see httpd.conf-dist-win that came with your Apache installation):

LoadModule rewrite_module modules/mod_rewrite.so
AddModule mod_rewrite.c

You should now be able to run the smokeserver CGI script directly by opening:

http://localhost/smoke/

Windows+Apache Note:

If you are setting this up with Apache on Windows the instructions in this section to rewrite the ugly .../cgi-bin/smokeserver.cgi/... into the user-friendly .../smoke/... will not work as described. This is because the:

RewriteRule ^/smoke(/.*) /full/path/to/cgi-bin/smokeserver.cgi$1 [last]

Apache directive will not properly translate this for a Windows-style absolute path. It will try to prefix C:/full/path/to/cgi-bin/smokeserver.cgi with your configured Apache DocumentRoot which, obviously, is incorrect. I am inclined to assume that this is a bug in mod_rewrite.c.

The current work-around that I am using is as follows:

  1. Copy smokeserver.cgi and smokeserver.conf to your document root instead of to your cgi-bin directory.

  2. Change the above RewriteRule line to:

    RewriteRule ^/smoke(/.*) /smokeserver.cgi$1 [last]
    
  3. Ensure that the ExecCGI option is set for the document root directory.

This is not ideal because (1) we have openned up the document root for CGI scripts and (2) smokeserver.cgi is under the document root which, apparently, is a possible security hole (is it?).

Troubleshooting:

If you are unable to get the smokeserver CGI script running on your Apache setup, please look at the excellent setup/troubleshooting docs for Quixote: Running the Quixote Demo and Web Server Configuration for Quixote. These docs describe how to setup and run Quixote's demo.cgi. If you can get that running, smokeserver.cgi should run fine.

TBD: Start distributing a Smoke Server installer which bundles Quixote and MySQL-python. And possibly bundle Apache, MySQL and FastCGI (or at least a wizard to help set them up)??? Might want to look at how Subversion handles the Apache bundling, if that is relevant.

mod_python

There are generally a number of options for speeding up Python CGI scripts with Apache:

  • mod_python
  • FastCGI
  • SCGI

However, we have found that mod_python is the only one that will work on Windows. This section describes how to get a Smoke server running via mod_python up and running.

  1. Download the mod_python 2.7.8 Unix source package from. Note that we use Apache 2.3 (not Apache 2.0), so you must use mod_python 2.7.x and not mod_python 3.0.x.

  2. Manually copy the libpythonmod_python directory to your Python site-packages directory.

  3. You must now also place the compiled mod_python.dll into your Apache "modules" directory. You can either figure out how to compile that from source or you can use the slightly outdated pre-built mod_python.dll for version 2.7.6. Extract mod_python.dll from the appropriate zip file there and copy it to your Apache "modules" directory.

  4. Add the following lines in the "appropriate" places in your Apache httpd.conf file:

    LoadModule python_module modules/mod_python.dll
    AddModule mod_python.c
    
  5. Now we can configure your server to serve Smoke via Smoke's mod_python_handler.py. Note that this setup is appropriate for Smoke development and not really for deployment. This should get you started though. Add something like the following to your httpd.conf file:

    <LocationMatch "^/smoke2(/|$)">
        SetHandler python-program
        PythonHandler mod_python_handler
        PythonOption quixote-root-namespace smokeserver
        PythonInterpreter quixote.smokeserver
        PythonDebug On
        PythonOption smokeserver-ini-path "C:\trentm\www\htdocs\smokeserver.ini"
        PythonPath "sys.path+[r'C:\trentm\as\support\smoke\src\server']"
    </LocationMatch>
    

Introduction to Developing Smoke

Smoke is currently developed in ActiveState's Perforce repository at:

//depot/main/support/smoke/...

Smoke is implemented primarily in Python (it requires Python >= 2.2). Smoke's specification documents (docs/specs/...) are integrated into ActiveState's specs server:

http://specs.activestate.com/Smoke

A Smoke setup is basically a Smoke server (src/server/...) and one or more Smoke clients (src/client/...). Smoke clients add data to the smoke server via the Smoke Client API. Smoke clients would typically be automated build and test scripts. Users can browse the Smoke server pages for pretty representations of the build, test and performance data. The URLs for these pages are specified in the Smoke Server API.

Read the top-level ReadMe.txt for other details.

© ActiveState 2003 All rights reserved