PVS update

29 07 2007

It’s been quiet here lately, but only because I’m busy with PVS – my SoC project. ;-) PVS stands for Patch Verification System and I’m building it to support patch review process for CPython. There’s not much time left before the end of SoC 2007, so I won’t tell you the whole story now. Instead, I will share with you few insights about development process and tools I’ve used. OK, here we go.

Trac logoSometimes Trac is an overkill. Text files in reST format are often more than enough. You also get a bonus by keeping all your project-related stuff (tests, documentation, todos) in one place (i.e. not scattered among repository, wiki and bugs manager).

Django logoI’m using two ORMs inside PVS: Elixir+SQLAlchemy and Django ORM. Both work great and saved me a lot of tedious work. I like the way SQLAlchemy handles references to other tables, although its select syntax is slightly weird (although surely powerful). Django ORM may not be as functional as SQLAlchemy, but its main advantage is its good integration with the rest of the framework. Tweaking the admin views is both pleasant and addictive (something like customizing desktop settings of your new Ubuntu install ;-). You’ve been warned.

Did I mention that both ORMs work on the same database without any problems at all? A big thanks to both libraries’ authors for not using magic tables/fields or strange implicit conventions.

Ohloh Metric for PythonDuplicating the whole source checkout tree of a project of CPython size (about 100Mb in almost 12.000 files) is quite expensive, as I soon realized (better sooner than later). Hopefully I don’t have to do this – patch has –dry-run option and subversion client supports revert operation. This combination has truly saved me. :-)

Twill rocks. I written a patch reporter for Roundup in less than half an hour. Thanks Titus!

@Python decorators are not used often, but applied carefully can be very powerful. In PVS I defined daemonized decorator which makes decorated function execute in the background as a separate process. It also creates a PID file in a configured directory (like /var/run/pvs/), so you can easily manage the daemon (e.g. send signals to it).

Bazaar logoAfter initially using Subversion I recently switched to Bazaar, mostly for its ability to work offline. It’s been working fine so far. Because I’m working alone I didn’t have occasion to use its merging capabilities, so can’t really comment on those. When it comes to speed, it’s visibly faster than Subversion, mostly because it doesn’t have to process anything over the wire. Making a backup and sharing is easier – just archive the branch directory and you’re done. svn2bzr worked without problems, so if you’re a Subversion user – give Bazaar a try.

There are functions which I use over and over again in different Python applications that I write. Simple ones like read_file_contents/write_file_contents and more complicated beasts like daemonize or run_command. Those get copy&pasted between code repositories, which isn’t really DRY. Yeah, I could clean them up and put mk-utils package on PyPI, but I’m too lazy and this is a stupid idea anyway. A central repository of those functions and a simple way to cherry-pick – now that would make my day. Hey, why do we need whole libraries anyway? ;-)

And finally: grab the code. :-)





Minor annoyance

19 05 2007

Map operation on sets isn’t closed, both in Python:

>>> type( map(lambda x: x**2, set([1,2,3,4])) )
<type 'list'>

and Ruby:
>> Set.new([1,2,3,4]).map { |x| x**2 }.class
=> Array

At least Haskell does the right thing:
Main> typeOf( Set.map (\x -> x^2) (Set.fromList([1,2,3,4::Integer])) )
Set Integer

An obvious workaround is to operate on lists and convert to sets once you need them. You have to ask yourself whether it will cause performance problems in your application.





Accepted to Summer of Code 2007!

19 04 2007

It actually happened a week ago, but I was really busy and had no time to share this happy news with the world. But here I am again, a lucky student ready to hack some Python code during summer. This time I will not only write in Python, but also for Python – I’m going to implement a verification system for patches contributed to Python project via its patch tracker. See the application for detailed description. I have already set up a Trac instance for this project, so go ahead and post your comments there.

Last summer I learnt quite a bit about Python and its toolset, made some great friends and got involved into even more projects. I will be happy if this year will be no different.





Free your blog!

27 02 2007

During my search for free blogs today I got surprised by the fact that a really small percent of Python blogs (either on Planet Python or Unofficial Planet Python) is licensed under a free license (full list below). I believe it’s caused mostly by omission or unawareness, as in my example — I added a CC button pretty recently, although my intention was to share my knowledge with others from the very beginning.

Most of you already publish your code in open source projects, so you know the benefits of openness. Next step is freeing your ideas and hard work you’ve put into your blog. Choose a license that work for you and open your blog content for collaboration.

For a quick explanation of CC licensing, watch this video:

What follows is a list of Planet Python and Unofficial Planet Python blogs that are licensed under Creative Commons. If I omitted your blog or you’ve added a CC licensing recently, please let me know in the comments, so I can update this list.





Copying methods in Python

17 02 2007

Doctests should be self-explanatory:

class Copycat(object):
    def extract(self, klass, method):
        """Extract a `method` from given `klass` so it can be used on this object.

        >>> class A(object):
        ...    def inc(self, n):
        ...        return n + 1
        >>> p = Copycat()
        >>> p.extract(A, 'inc')(5)
        6
        """
        return lambda *args, **kwds: klass.__dict__[method](self, *args, **kwds)

    def copy_method(self, klass, method):
        """Copy a method from `klass` into this object.

        >>> class A(object):
        ...     def inc(self, n):
        ...         return n + 1
        >>> p = Copycat()
        >>> p.inc(11)
        Traceback (most recent call last):
          ...
        AttributeError: 'Copycat' object has no attribute 'inc'
        >>> p.copy_method(A, 'inc')
        >>> p.inc(11)
        12
        """
        self.__dict__[method] = self.extract(klass, method)

    def copy_methods(self, klass, *methods):
        """Copy methods from `klass` into this object.
        """
        for method in methods:
            self.copy_method(klass, method)

Normally you can’t call foreign method on a given object, so this won’t work:

>>> class A(object):
...     def inc(self, n):
...             return n + 1
>>> class B(object): pass
>>> b = B()
>>> A.inc(b, 7)
Traceback (most recent call last):
  File "", line 1, in ?
TypeError: unbound method inc() must be called with A instance as first argument (got B instance instead)

Using the trick with __dict__ you can overcome the type check and use the power of duck typing:

>>> A.__dict__['inc'](b, 7)
8

I used this technique with Mock class (from python-mock module) to easily test particular methods of a class, while mocking others. Let me give an example.

Imagine you’re testing a class which has few methods that touch the filesystem/database (so they are slow and rely on external unpredictable state) and few that are pure and do some logic, while delegating all the dirty work to the first group of methods. For example:

class ClassWeWantToTest(object):
    def __init__(self):
        self.that = self._init_that()

    def _init_that(self):
        # Touching the filesystem/database/...
        pass

    def dirty_work(self, argument):
        # Touching the filesystem/database/...
        pass

    def referentially_transparent(self, argument):
        # Working on argument and calling self.dirty_work()
        #  from time to time.
        pass

_init_that and dirty_work are “dirty”, while referentially_transparent is “pure”. Now, we want to test the referentially_transparent. We can’t make a mock and call referentially_transparent on it:

def test_it():
    m = Mock()
    m.that = [1,3,5,42]

    ClassWeWantToTest.referentially_transparent(m, 13)

This won’t work for the same reason the example with A and B classes didn’t work:

Traceback (most recent call last):
  File "", line 7, in ?
    test_it()
  File "", line 5, in test_it
    ClassWeWantToTest.referentially_transparent(m, 13)
TypeError: unbound method referentially_transparent() must be called with ClassWeWantToTest instance as first argument (got Mock instance instead)

This is where Copycat class comes in. Using the class we can rewrite test code into this:

class CopycatMock(Mock, Copycat): pass

def test_it():
    m = CopycatMock()
    m.that = [1,3,5,42]
    m.copy_method(ClassWeWantToTest, 'referentially_transparent')

    m.referentially_transparent(13)

Now everything seems to work. We can mock return values and set expectations as usual. Happy mocking!

Note: To make it work with current (0.1.0) version of python-mock, use this small patch.





Make your tests self-documenting

14 02 2007

Inspired by RSpec (which in turn was inspired by TestDox).

A piece of code that transforms standard unittest output:

test_doesnt_raise_and_exception_when_None_was_passed (test_webui.TestSomeArbitraryMethod) ... ok
test_returns_this_and_that_when_string_was_passed (test_webui.TestSomeArbitraryMethod) ... ok

into:

Some arbitrary method doesn't raise and exception when None was passed ... ok
Some arbitrary method returns this and that when string was passed ... ok

Test case that generated output above looks like this:

class TestSomeArbitraryMethod(TestCaseWithSpec):
    def test_returns_this_and_that_when_string_was_passed(self):
        assert "this and that" == some_arbitrary_method("string")

    def test_doesnt_raise_and_exception_when_None_was_passed(self):
        try:
            some_arbitrary_method(None)
        except:
            assert False

If you like it, grab the source code of TestCaseWithSpec.





Test isolation in nose

7 12 2006

NosePython has a built-in testing framework called unittest. As time went by Python programmers looked for better-suited solutions. Some of them created a new quality (like doctest) and others build on top of unittest. One of frameworks which intends to replace unittest is nose. I use it constantly during development of Cheesecake and I must admit it’s been very helpful, being easy to setup, integrate and extend to my needs. But it has to be said that among many of its features it also have slightly different philosophy than unittest. In this short article I’m going to describe one of the main issues that nose users may accidentally step into – weak test isolation.

Unittest, which nose is a great successor, was based on JUnit. One of the core priciples of JUnit was that (quoting Martin Fowler) no test should ever do anything that would cause other tests to fail. Nose doesn’t give you such certainty, because (for performance reasons) it uses one interpreter for all tests. We’ll look at the examples of possible bugs that this approach can cause.

This post is not meant to be a rant of any sort. Actually, by learning from this cases I’ve improved my understanding of testing in general. I’d be happy if at least one of the readers will learn something new from my experience. To those of you seeking a quick fix, I inform you that this issue has been bureported and there is a solution under way. Please remember that tests isolation won’t be a default though.

Just a note: last time I checked py.test had the same kind of flaw.

External state

Problem

State of tested modules preserve from one test to another.

Example

Imagine you have a “Hello world” module with following contents:

message = "Hello world!"

def show():
    return message

And two test files which exercise two different possible uses. Test A uses a custom message:

import hello

class TestHelloMessage:
    def setup(self):
        hello.message = "Bye world!"

    def test_hello_message(self):
        assert hello.show() == 'Bye world!'

while test B uses the default:

import hello

class TestHelloDefault:
    def test_hello_default(self):
        assert hello.show() == 'Hello world!'

All files are in the same directory, so we’re ready to execute a test runner:

$ nosetests
.F
======================================================================
FAIL: test_hello_b.TestHelloDefault.test_hello_default
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/nose-0.9.1.dev_r101-py2.4.egg/nose/case.py", line 129, in runTest
    self.testCase(*self.arg)
  File "/home/ruby/nose/ext/test_hello_b.py", line 5, in test_hello_default
    assert hello.show() == 'Hello world!'
AssertionError

----------------------------------------------------------------------
Ran 2 tests in 0.007s

FAILED (failures=1)

Ouch, you didn’t expect that, right? Where’s the bug? Before you spend your evening staring at the code, try naming your test files differently:

$ mv test_hello_a.py test_hello_c.py

and both tests will pass:

$ nosetests
..
----------------------------------------------------------------------
Ran 2 tests in 0.005s

OK

Why is that? Nose reads tests from the directory in lexical order, so test_hello_a.py gets read and executed before test_hello_b.py. Nose runs both tests in the same environment, so hello module is read and initiated only once. This means any changes made by test A to the module hello will be present during run of test B.

Resolution

This particular example seems like an error on the test runner side, because we have two different scripts that interfere each other in a way that wouldn’t happen if you run them in separate runs. On the other hand, it may pinpoint a bug in your setup/teardown logic. If you’re changing global state during setup to exercise behaviour of objects in certain environment you should revert to the original state during teardown. Having this in mind, test A should look more like this:

import hello

class TestHelloMessage:
    def setup(self):
        self.original_message = hello.message
        hello.message = "Bye world!"

    def teardown(self):
        hello.message = self.original_message

    def test_hello_message(self):
        assert hello.show() == 'Bye world!'

Reverting changes may seem impossible in some cases, but you should try hard to do it. Remember what they say: Your code sucks if it isn’t testable. And it isn’t really testable if you can’t isolate a state you’re interested in testing.

Mocking errors

Problem

State of builtin modules preserve from one test to another.

Example

This is similar to the first example, but approaches the problem from a different direction. Now we’re writing a simple locking mechanism that uses files as locks. Currently code looks like this:

import os

class Locker:
    def __init__(self, lock_file):
        self.lock_file = lock_file

        if self.is_locked():
            self.unlock()

    def is_locked(self):
        return os.path.exists(self.lock_file)

    def lock(self):
        file(self.lock_file, 'w').close()

    def unlock(self):
        os.remove(self.lock_file)

We also have a test for an unlocked locker:

import locker

class TestUnlockedLocker:
    def setup(self):
        self.locker = locker.Locker('/path/to/lock/file')

    def test_that_is_locked_is_false(self):
        assert self.locker.is_locked() is False

It worked well until we added a new test case:

from mock import Mock

# class TestUnlockedLocker:
#    ...

class TestLockedLocker:
    def setup(self):
        # Mock file(), os.path.exists() and os.remove() for locker.
        locker.file = lambda p, *a: Mock({ 'close': None })
        locker.os.path.exists = lambda p: True
        locker.os.remove = lambda p: None

        self.locker = locker.Locker('/path/to/lock/file')
        self.locker.lock()

    def test_that_is_locked_is_true(self):
        assert self.locker.is_locked() is True

Now we’ll get a failing test for unlocked locker:

$ nosetests
.F
======================================================================
FAIL: locker_test.TestUnlockedLocker.test_that_is_locked_is_false
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/nose-0.9.1.dev_r101-py2.4.egg/nose/case.py", line 129, in runTest
    self.testCase(*self.arg)
  File "/home/ruby/nose/mock/locker_test.py", line 9, in test_that_is_locked_is_false
    assert self.locker.is_locked() is False
AssertionError

----------------------------------------------------------------------
Ran 2 tests in 0.005s

FAILED (failures=1)

Does the TestLockedLocker leave a lock file? No. But it leaves mocked version of os.path.exists, which always return True. Builtin modules doesn’t get reloaded for each test, so any changes you do to them will remain during execution of other tests. The same goes for builtins.

Resolution

State of the interpreter should be considered external, just like the state of a filesystem or environmental variables: If you’re making any changes to it, remember to revert this change at the end. This especially affect __builtins__ and standard library, where change of one function/variable can affect majority of other tests.

Import clashes

Problem

Subdirectories you keep your tests in cannot contain modules with the same names.

Example

As your set of test cases grow, you will probably start arranging them in a hierarchical structure. First obvious level of partitioning is placing your unit tests and functional tests in separate directories. That is what Grig done for Cheesecake project and this is convention I’ve followed. As I was writing more unit and functional tests a bit of redundancy in code started to show up. When the time to refactor came, I took common functionality and placed in a handy module helper.py. That’s when nose came into my way. My directory structure looked like that:

tests/
    unit/
        helper.py
        #test cases#
    functional/
        helper.py
        #test cases#

It was the common name for helper module that caused problems. When run separately, tests had no problem running:

$ nosetests -i unit
.
----------------------------------------------------------------------
Ran 1 test in 0.003s

OK
$ nosetests -i functional
.
----------------------------------------------------------------------
Ran 1 test in 0.003s

OK

On the other hand, when I tried to run them at once, at least one of test will always fail due to difference in unit/functional helper contents.

$ nosetests -i unit -i functional
.E
======================================================================
ERROR: test_this.TestThis.test_this
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/lib/python2.4/site-packages/nose-0.9.1.dev_r101-py2.4.egg/nose/case.py", line 129, in runTest
    self.testCase(*self.arg)
  File "/home/ruby/nose/import/unit/test_this.py", line 5, in test_this
    helper.unit_help()
AttributeError: 'module' object has no attribute 'unit_help'

----------------------------------------------------------------------
Ran 2 tests in 0.005s

FAILED (errors=1)

In the example above, unit test test_this tried to access unit_help function, which exists in unit/helper.py, but doesn’t in functional/helper.py. Apparently functional helper got imported first, and it was saved as ‘helper‘ in sys.modules, thus inhibiting “import helper” in unit test (remember that all tests share the same interpreter).

Resolution

Name your helper modules differently or use reload().

Conclusion

Next release of nose will contain an isolation plugin, which responsibility will be to purge sys.modules list between running different test files. This will fix some of isolation problems, but not all of them. You will still be able to kill the runner or modify builtins in a way that breaks other tests. Complete solution would involve spawning a new Python process for each test, which is unacceptable from a performance point of view. Current state is just good enough, so with a bit of nose-specific knowledge and common sense you can get the best of both worlds: testing speed and stability.