team: PySoy

I write this blog post from PyCon 2010, sitting on the floor in a tiny crowded room the hotel, Hyatt Regency Atlanta, has relocated us to.

For the PyCon sprints we had reserved the large room used for lunches over the weekend plus seven break-out rooms; just after the sprints began we found another convention (Christian Colleges) setting up in the large dining room and setting up signage throughout the area. As it turns out the hotel double-booked the space we were using starting a chain of reducing our space and relocating us. Today, the final day of the sprints, we were relocated to a small room in the basement of the International Tower; a separate building entirely. The PyCon organizers have opened up the Chicago Room for work space as well (previously used for the media and organizers), but many people have simply left because there's no space to work.

This is only a long chain of disrespect and negligence the hotel has demonstrated toward us; turning heat off in our work space at night (last night everyone had to relocate to the only room with heat) and leaving exterior doors open near the space, refusing to remove garbage (or even provide garbage cans) in the work space, continually rearranging tables in the open space, the list goes on.

Above this, I found the catering over the weekend extremely sub-par and some of the waiters rude. I actually had to argue with the waitress Sunday to give me a vegan meal because so many other people had claimed to be vegan for an alternative to the apparently less palatable meat lunch they were offering. I don't think most of the attendees would have chosen to pay what they must have been charging for these meals and many of us found them so skimpy that we went out for lunch anyway.

I truly hope the PyCon organizers get either a full refund for this convention, including some reimbursement for us attendees, or cancel the second year of the contract and find a better hotel for next year. We all took time off work/school/life to be here, and frankly, I would have left Monday if this was known ahead of time. If PyCon is at the same hotel next year I know I'll be seeking alternative housing arrangements.

At least the PSF is buying us pizza for lunch today.

We've been moving forward on the PySoy rewrite using Genie in faith that a Python 3 binding for GObject would come about.

At PyCon this weekend I've brainstormed with Kirk McDonald, author of PyD which serves a similar role to use D types/objects in Python, and several other authors of FFIs for Python. A rough strategy is coming together to solve this in a dynamic manner.

The first part of the solution is libvala; using a small part of its functionality to parse .vapi and .gir files for existing GObject libraries such as GTK and Cairo as well as any library written in Vala or Genie. This gets us the C API for the library in all it's object oriented goodness, and with that we can dynamically generate bindings using Python's ctypes.

The second part is a GObject type in Python which supports subclassing in Python. Kirk detailed how this is done in PyD; each generated GObject type gets a subtype with all of its virtual methods all running Python methods which call the superclass's C method by default. When a Python subclass of a GObject class is made this GObject subclass is used allowing Python to override any of its methods and pass objects of the subclass back to GObject where they'll perform as expected. Not simple or easy, but its a strategy.

The end result is dynamic Python 3 bindings for GTK as well as PySoy and all the other tools we work on.

I'm organizing a sprint for this at PyCon. If anyone has ideas on this please comment, or if you'd like to help (whether at PyCon 2010 or not) get in touch with me.

Recently Arc tuned me on to a wonderful concept deemed a Hackerspace. For those of you who haven't heard about them they are places where people can get together and do things related to art, technology, science, and much more! After a quick google search I found a local hackerspace.

The hackerspace has a plethora of neat items such as a makerbot (a type of reprap machine) and a multitude of arcade cabinets a group of people are currently modifying. Everyone at the space is extremely welcoming and really, really freaking helpful with a multitude of subjects. The type of people that show up really did surprise me from engineers, to music artists, to just plain artists. Take a look around your city! I bet you have one near.

After porting Canvas over to Genie in the new libsoy library, I decided to try it out on my new Windows 7 machine. Take a look at the result, it shows a 1 Dimensional texture applied to a Canvas widget. This is somewhat of a "Black Triangle" moment, where it's a simple test, but a step in the right direction. The Canvas widget just displays a texture on the window. Libsoy works on Linux, Windows, and somewhat on Mac.

We purchased a new printer for the house last night to replace the windows-only Dell printer that ran out of ink (and would cost almost as much as a new printer for replacements).

First, some advice: plan ahead, order online, don't buy from Staples. We paid $170 for this when Epson is offering them direct for $130 with free shipping. If we were not in need of immediate replacement we could have saved $40 and waited a week. Further, the rep at the store lied when he told us the printer was going to the back to get a unit "fresh off the truck", it was a unit another customer had returned with ink spilled and dried on the inside and some of the packaging torn open. Buyer beware. Otherwise, they seemed to clean it up and repackaged it well. I have half a mind to order the printer online and return this one within 14 days for refund to save the $40.

The printer itself is fairly impressive. Two paper trays (letter/legal plus photo/cardstock) and a built-in CD printing tray that slides out when you select CD printing, scanner with auto-document feeder, and built in 10/100 and wifi so we don't need to depend on any computer in the house to properly "share" the printer. All I had to do is select our access point on the front controls and give it a static IP on our router.

Ubuntu 9.10 detects the printer over the network in seconds and I was printing the Ubuntu test page in under a minute, so more than half the systems in the house were easy. Windows setup was not hard, but time consuming as loads of bundled software installed from the CD. The only complaint I have is the color seems a little off; printing the same image in both Linux and Windows shows a fairly radical difference in quality. The prints from Linux are still much better than printing from Windows to our older printer.

Next I tried to use the scanner function. Using GIMP to launch SANE (Scanner Access Now Easy), the printer was automatically detected and let me select either the flatbed or auto-document feeder. Both options work fine until I start the scan, at which point the Epson Artisan 810 locks up and needs to be rebooted.

There is a scan function from the front panel, not supporting anywhere near the scanner's full resolution and storing in JPEG format with obvious artifacts around text/etc, but at least we don't need to use a Windows system for basic scans until SANE is fixed. Since I have a stand-alone Epson scanner that works beautifully this isn't a huge issue for me, though it would be great to be able to use the auto-document feeder from Linux and be able to put this older scanner into storage.

I'm seeing "red" now with our Windows, in more ways than one. The rendering code for soy.windows.Windows is to the point that, by changing the glClear color, we're seeing red in the window rather than black. This is a big step forward given that porting to use GDK is a big part of the libsoy migration.

On the other side of "red" is anger over GLEW. I really wish I knew of a decent, mature alternative to GLEW. The project is so immature that, though one of the maintainers is a Debian developer, they do not provide the pkgconfig file for it. Without this file, which provides Waf with the location of the library and it's headers, we would have to write a search script to verify that it's installed and find it's location. None of the major distros I looked at provide the pkgconfig on their own.

It's also at least partially redundant with the functionality of gtkglext which I believe should be able to replace GLEW entirely.

For the first time in 3.5 years, we had soy.windows.Window running on OSX. Just the basics; opening, (re)setting the title, resizing - but it worked. This is thanks to GDK, the windowing and event backend to GTK+, running on Mac OSX via native Quartz backend (not X11).

Shortly after we moved the window code to a background thread and it broke. The code runs fine on GNU/Linux and Windows, but on OSX all we get are white rectangles where the windows would normally be rendered. The rectangles even resize properly, and window events are getting processed by our code, just nothing rendered inside (not even the window title or buttons). I filed this bug and hope the problem to be solved quickly.

The problem could be as simple as a missing NSGraphicsContext.flushGraphics call (which is only needed when rendering is done in a non-main thread). If the bug hasn't been fixed by our next release I'll look into patching this myself.

After some work to update our ODE bindings for Vala, we discussed Bullet again last night and returned to the conclusion to drop further work on ODE and focus on Bullet for the next release.

It quickly became clear that the C bindings for Bullet are utter horse crap; praise for the effort for whoever wrote them, and I by no means intend to disparage your ability or intentions, but they're horribly incomplete and usable for little more than the two demo apps included in the source. Moreso, the C bindings are "higher level" than the C++ library, performing several steps automatically (such as calculating body inertia from mass and shape) which our 3d game engine needs more direct control over. I first wrote an alternative (ABI compatible) C header for it, and then started to rewrite it entirely.

I decided that if we're going to write our own Bullet compatibility layer, we may as well go all the way in wrapping it with GObject. The code is simple enough, but defining GObject in C involves a lot of boilerplate - in some cases more than the code itself. To get through this and retain my sanity, I started work this morning on a C++ code generator that takes the relevant functions and wraps them in the standard boilerplate.

The format I'm using is Genie-like for class and method definition, then C++ within dumped directly to the target .cpp. The code generator will generate the .vapi as it works and compile our wrapper as a static library which libsoy can link to directly.

Kudos again to Waf for making this rather eccentric use of their build system dead simple and clean.

We've been working on the "libsoy" branch of PySoy for some time, a split of the compiled code to a GObject-based C library with Python bindings plus the pure Python code constituting PySoy itself.

Previously we were struggling to build this with Python's distutils system, which didn't support building system-wide C libraries nor using valac to do so. Jon and I started looking into Waf last week. It's pretty impressive; Python based (and works with Python 3), simple, intuitive, does everything we need with only a few dozen lines of script. Last night we setup libsoy in it's own repository and building (just a very basic soy.windows for now) on it's own.

I think we're settled to work with Genie, a comparison between the syntax of soy.windows in Vala vs in Genie makes that choice simple. The only complaint I have with Genie is needing to escape newlines inside struct {}'s (ie, line 32-35), but reportedly this is being fixed in an upcoming release to be more Python-like (ignoring newlines while inside () or {}).

After a bit of searching (through source code, I wish Waf had better API reference in it's docs) I found the gir= option for Vala building, and another hour later both soy-1.0.gir and soy-1.0.typelib were being generated, except both were empty save the "soy" namespace:


<?xml version="1.0"?>
<repository version="1.0" xmlns="http://www.gtk.org/introspection/core/1.0" xmlns:c="http://www.gtk.org/introspection/c/1.0" xmlns:glib="http://www.gtk.org/introspection/glib/1.0">
  <package name="soy"/>
  <namespace name="soy" version="1.0" c:prefix="soy">
  </namespace>
</repository>


I've hit this before with trying to generate GObject Introspection from distutils but chalked it up to not building it correctly. As it turns out, this is a known bug in GIR and a previously unknown bug in valac to fail silently.

I'm still hoping the GIR developers will come to their senses and support nested namespaces, but given that they've stubbornly rejected this for months already I'm afraid we'll have to build a cross-language bindings builder based on Vala/Genie instead of helping to update PyGI for Python 3.

Wasted quite a few hours today trying to get a Vala/Genie IDE working.

VTG is basically broken, crashing at random. It had problems before this; having a fairly rigid concept of a "project", not supporting non-autotools build systems or libraries very easily, not supporting Genie, and the symbol mapping having odd quirks, so this is not a huge loss.

A newer Vala plugin for gEdit, Valencia, looks promising but doesn't support recent Vala versions and attempting to build it raised pages of errors.

Anjuta has always looked interesting, but I've never managed to get the Anjuta Vala plugin to compile. At present, it doesn't appear to work with Gnome 2.28.

I don't use Eclipse or Mono, so the final option is Valide. To my surprise, it's been packaged for Ubuntu and works out of the box. It supports Genie, it seems to have features on-par with VTG and the other IDEs, and works with the latest version of Vala.

My only complaint so far is the new project wizard doesn't list the AGPLv3 as a license option, but then none of the Vala IDE options do yet.

Here’s a short, 5 questions, 2009 Python meme. Copy-paste the questions, and blog your answers !

1. What’s the coolest Python application, framework or library you have discovered in 2009 ?

SQLAlchemy, not only does it fully (properly) abstract the SQL database for you, but provides a full Object Relational Mapping. What's more, it's Python 3 ready.

2. What new programming technique did you learn in 2009 ?

Using GObject Introspection to mix several languages (ie, Python and Javascript) in the same application.

3. What’s the name of the open source project you contributed the most in 2009 ? What did you do ?

Concordance XMPP, an XMPP service framework for Python I started January 1st 2009.

4. What was the Python blog or website you read the most in 2009 ?

Python Planet

5. What are the three top things you want to learn in 2010 ?

1. SHARC DSPs
2. Welding
3. Aikido

We've seen a greater than average share of service outages at Serverbeach; incompetent electricians blowing out the battery backup system causing an evening of downtime, network failures taking the server down for hours at a time, other random outages without explanation.

Of course these outages affect large numbers of customers so you get 500 errors when trying to access the customer portal or put on hold for up to an hour waiting to talk to someone only able to enter a ticket into the system on your behalf.

This latest outage tops them all, however. Yesterday afternoon they attempted to merge the Peer1 and Serverbeach customer portals which knocked out DNS for a "large number" of customers. They sent out an email to everyone and I fully expected it to be a short outage.

Last night at around 8pm all of our domains were still down. I attempted to log into the customer portal to find my old password not working, and after a lengthy process to recover it I found myself locked out of it unless I agreed to a lengthy new service agreement which I certainly don't have time to read through in the middle of an outage.

After sitting on hold with customer service a ticket was filed. Which I replied to, and replied to, and have only received blanket emails asking to reply again if we're still out. More than 24 hours later we are still down.

Serverbeach claims they refund customers for outages, but I have yet to receive a dime credited for any past issues and I fully expect them to make getting credit for this difficult as well. At this point I don't think it's fair to myself or the communities which rely on this server to continue hosting with them.

I've found a really decent deal with Hurricane Electric with the added perks of IPV6, more bandwidth, and newer hardware for less than I'm paying now.

Despite being heavily involved in Ubuntu advocacy and promotion, being a member of the Ubuntu project, and using it on my personal systems, I've had to keep my servers and development machines running Gentoo.

When developing software, we need to work on what will be "mainstream" in 6 to 12 months, not what's already mainstream. When a new version of our physics library is released, it may depreciate certain methods or include new features we'll really want in our release, but that version won't be packaged until 2-8 months after the new library was released, given the typical release cycles and "feature freeze" stopping new packages from being included too close to release time.

Python 3 is a great example of this; Gnome 3 will apparently use Python 3 for many applets, and Gnome 3's release schedule sets it for inclusion in Ubuntu 10.10 (next Fall), but at present Python 3.1 is packaged but none of the 3rd party packages are available for it - even those which explicitly support Python 3 already.

While doing my periodic Gentoo upgrade, I just noticed that Portage, the package system for Gentoo, can now be built to run on Python 3 directly. When you have both Python 2 and 3 installed on your system, packages wich support both will build and install for both, and their packaged versions are often available just days after release. From what I've seen, Gentoo is one of the best distributions for Python developers right now.

I wouldn't recommend that my family run Gentoo, many of whom happily run Ubuntu, but it's lack of support for developers is concerning to me. I do not believe it should be difficult to both support new Linux users and experienced developers with the same distribution.

I just learned that the Dell employee discount, which is offered to employees of numerous Dell partners, does not apply to their N-series laptops, even though many people only have this discount because of their work with Ubuntu.

The irony continues to drip as the models I compared between the employee purchase price and the Ubuntu versions of the same, such as the Inspiron 15 and Inspiron 15n, roughly discounted the added cost of Windows 7 only. It appears that the employee discount equates to free Windows 7.

The current Dell Ubuntu offerings lack in many other areas, such as not offering gigabit ethernet or 802.11n as options. While I would love to support Dell in working with the Ubuntu community more, I really can't justify buying a sub-par laptop at the prices they're offering them at.

My friend Mackenzie showed off her System76 laptop at the 9.10 release party a few weeks ago. All of their laptops include gigabit ethernet, 802.11agn wireless, and many other options Dell doesn't offer. They're a bit more pricey, but according to Mackenzie their service is significantly better.

Bullet has some really great features over ODE; soft body physics, particle physics, and hardware acceleration among the top.

After talking with a Crystal Space developer about this, I put it on my list of things to look into when PySoy is a bit more mature. However, a Debian packager's decision has accelerated these plans.

ODE is not designed to be installed as a single shared library, it has many build-time options which radically change it's behaviour - such as whether the library uses single or double precision, or whether to calculate gyroscopic force. As a general rule, distributions stick to the defaults when packaging as much as possible, but Debian (and thus Ubuntu and other Debian-based distros) ship a single system-wide library with non-default compile flags (Double precision) that would result in sub-par performance, network sync issues, and in some cases bugs in games.

Debian could have shipped four or eight different versions of the ODE library, such that games could be linked against the one it was designed for and work as expected, or ODE could have been designed to build both a single and double precision library with the other options as runtime flags, but instead we game developers are caught in the middle.

We fought this battle before with Soya, for some time actually, and in the end Soya had to ship their own version of ODE embedded in the source and statically linked to get around the problem. Even then it was an issue to get the package including a static ODE included. It's a battle not worth fighting.

While Bullet also supports building as double precision, it's an option so rarely used that no distro builds with it (Gentoo doesn't even have a USE flag to switch it), and given that Bullet is currently not packaged for Debian it'd be easier to add it as a new proper package than fight Debian packages to improve their build of ODE.

I am not looking forward to working with Bullet's incomplete C-API, likely having to fill in some gaps as we work on it, nor writing the Vala bindings for Bullet, but it'd be a lot faster and fun - not to mention resulting in new features.

Man, Go sucks, the creators went and invented their own encoding, which Go source code is required to be in.

Compilation environment

I have been playing with the Go programming language. The gc compilation environment is broadly similar to gcc's, but with enough subtle differences that I am seeing a lot of confusion about how it works.

Traditionally, C is thought of as consisting of .c source files and .h header files. Each .c file compiles to a .o object file. The object files are then linked into some manner of final binary, be it a .a static library, a .so dynamic library, or an executable binary.

In this scenario, the binary depends on the object files which are linked into it, the object files each depend on their respective source file, and each source file depends on the headers which it includes (directly or indirectly).

The gc compiler for Go (the one also known variously as 5g, 6g, and 8g) works slightly differently. Here, we have .go source files. Every .go file belongs to exactly one package, and a package may be comprised of as many .go files as you desire. Packages have a .5, .6, or .8 extension, depending on the architecture you are compiling for. A package is obtained by compiling all of the .go files which comprise it.

Packages are the result of compiling .go files. They fill two roles. They are roughly the equivalent of object files in C (though, as I will explore in a moment, they are not exactly equivalent). They also fill the role that header files fill in C. Packages contain meta-data describing what the public interface of the package is.

Therefore, in order for one package to use another, you require the other package to be compiled, and you do not require the source files for the other package. Further, when a package is compiled, it statically links in all of its dependencies.

Once you have a "main" package, you may then link it into an executable binary. The only file required to do this is the compiled "main" package.

An example

Here is an example Makefile for a simple executable binary. The binary will be named hello. It depends on two custom packages, foo and bar, which each consist of a number of .go files. The "main" package itself contains two .go files: hello.go and goodbye.go.

For the sake of clarity, all of these source files live in the same directory (in a real application, each package's source files might live in their own directory), and I have hardcoded the architecture to amd64.

MAIN_FILES=hello.go goodbye.go
FOO_FILES=foo1.go foo2.go foo3.go
BAR_FILES=bar1.go bar2.go bar3.go

hello: main.6
        6l -o hello main.6
main.6: $(MAIN_FILES) foo.6 bar.6
        6g -I. -o main.6 $(MAIN_FILES)
foo.6: $(FOO_FILES)
        6g -o foo.6 $(FOO_FILES)
bar.6: $(BAR_FILES)
        6g -o bar.6 $(BAR_FILES)
.PHONY: clean
clean:
        rm -f hello main.6 foo.6 bar.6

I will explain each of these rules one at a time.

hello: main.6
        6l -o hello main.6

The hello executable binary is obtained by linking the main package. This package already (statically) contains the other packages which it depends on, thus, the only prerequisite is the main package itself.

main.6: $(MAIN_FILES) foo.6 bar.6
        6g -I. -o main.6 $(MAIN_FILES)

The main package is obtained by compiling the source files which comprise the package, and statically linking in the packages which they depend on. Thus, the foo and bar packages are prerequisites of the main package. However, we do not explicitly pass these packages to the compiler. Instead, we pass the compiler the -I option to inform it of where these packages may be found. It is therefore important for a package's filename to match the name it is imported as.

foo.6: $(FOO_FILES)
        6g -o foo.6 $(FOO_FILES)
bar.6: $(BAR_FILES)
        6g -o bar.6 $(BAR_FILES)

The foo and bar packages are simple, and consist only of their own source files.

.PHONY: clean
clean:
        rm -f hello main.6 foo.6 bar.6

We can make clean by removing all of the binary files. These are the binary, and the packages.

Simplifying matters

It would be relatively straightforward to write a tool which, given a list of source files, could determine which files belong to which package, what the dependencies between these packages are, and automatically rebuild what is required in a make-like fashion. Writing such a tool may be a project for the future.

I updated the Roadmap for 1.0-beta3 today.

In short, we're dropping ambitions for audio support in this release. Once our migration to Vala is complete and the related changes implemented we'll release without further delay. After that we'll begin work on the newer features like audio, networking, and browser embedding for beta4 while the community tests and gives feedback on beta3.

There are a lot of new features already compared to the beta2 release. We're aiming for a beta3 release by January 1st 2010, 2 years from the beta2 release.

The largest problem with integrating Mercurial with our new web framework is being resolved; they are upgrading Mercurial's license to GPLv2 or later.

Thanks to this, our favoured dVCS will be available for integrating into GPLv3 and AGPLv3 licensed projects without any weird hacks or work-arounds.