Cheesecake enhancements and its integration with PyPI.
Benefits to the Community
Cheesecake is an application designed to evaluate and estimate the overall quality (or so called ‘kwalitee’) of a given software package written in Python. It emphasizes a need for well-written documentation and unit tests, encouraging good programming practices and penalize sloppy design and careless distribution. Using Cheesecake to check your code gives you confidence that your software doesn’t merely run, but is usable and easy to test and modify as well.
Because Python is very easy to learn and use there exists a vast variety of software written in it, most of which was scattered until PyPI was created. Now, when new packages are being indexed on Cheese Shop every day, effort can be made to spread the spirit of good software design and code reuse among the Python community. This can be achieved by combining the power of Cheesecake and Cheese Shop. Everytime a new version of a package would be uploaded to Cheese Shop, its cheesecake index will be calculated and published on web. Having a way to measure a quality of a package with accordance to other existing packages will be of invaluable help for all developers. It will promote well built packages and in the long run raise the overall quality of Python software.
Adding Cheesecake functionality to PyPI has been already mentioned by Phillip J. Eby on catalog-sig mailing list. Together with Cheesecake maintainer Grig Gheorghiu we’ve discussed modifications needed to be done to Cheesecake code to be reliable enough so it could be incorporated into PyPI service. A working copy of our ideas is accessible on the project wiki. It includes enhancing Cheesecake code scoring techniques to take into account unit tests of a package, running tests in secure environment, extending supported archive formats and fixing all known bugs. Development of Cheesecake will adhere to best practices such as unit testing, continuous integration (via buildbot), pylint verification, etc. This step will take about 5-6 weeks.
The next part of this project will include collaboration with Richard Jones, PyPI maintainer, and merging Cheesecake into PyPI service. Upon completion all PyPI uploads will be automatically scored by Cheesecake. It will be possible to browse packages archive by cheesecake index, sorting results by installability, documentation and code kwalitee index. Statistics in numeric and graphical form will also be made available. This part of a project, involving writing server-side code, with emphasis on security and robustness, will take as well 5-6 weeks.
The remaining time will be spent on resolving all problems that would occur during usage of Cheesecake and PyPI. Along with fixing bugs, I will develop a simple Hello world package that can be taken as an example of good development practices for all Python developers. It should also score 100% in Cheesecake test of course. ;-) It will be what hello is for GNU Project. Implementation shouldn’t take more than two weeks, so I’ll hopefully complete the whole project before the deadline.
UPDATE: Testing of packages
Testing of package code will be performed by running its own tests, either by setup.py test target or by piping each module through doctest. Making unittest-based tests work could be a bit tricky, as we don’t know if running a module will execute an application or its tests, but some common cases could be probably handled as well. Points will be granted for test coverage (proportion of number of code/functions and tests) and their positive results. Package will earn bonus points for adhering to standard setup.py test target, as it’s a preferred method for all setuptools packages.
Packages should be tested often, not only when a new version of given package is published, but also when there are new releases of modules this package require. On the other hand, there’s no need to test changes in not-related packages.
To implement such a system a secure environment must be created, so that malicious tests won’t harm the whole system. Since python doesn’t have restricted execution capabilities since 2.3, some other means of protection would need to be used. I just haven’t decided yet, but will try to keep it as simple as possible. If needed, testing can be moved to another box. Cheese Shop will use a web interface to request testing of certain packages from this test-server and testing software on that box will run all tests and publish results in a fixed location. Although creating such a buildbot may seem difficult, I think it’s possible and can be done in the timeline I’ve presented. I have suitable unix experience as I’ve done some administrative work during my years spent on learning unix systems. Building a web interface is also not a problem, especially in Python. Problems will certainly arise, but that’s what make programming so exciting. I think hard work will do the job and project will be completed before the deadline.
I’ve already contacted Grig Gheorghiu and he expressed his willingness to mentor my project. Richard Jones will also be able to co-mentor my efforts to integrate Cheeseshop into PyPI.
Name: Michał Kwiatkowski
Jabber address: firstname.lastname@example.org