Skip to content

The Wise Old Geek Posts

What I learned from game developpment

A decade ago, while I was finishing my master’s degree, I also started developing games with a few friends. Nowadays, I can’t overestimate the benefits those years of late-night-coding brought me, and here are a few lessons I learned in the meantime:

Data is important to the user

In online games, players are your customers, and they pay the high price of their free time to use your product. As most of us know, free time is a scarce resource, and your players won’t stay for long if you mess with their time, such as because of a bug, they lose all their possessions (technical issue), or if their character is killed too often (game logic issue), …

Having no backup of your database is the surest way to lose all your players at the first server crash.

Lesson: backup systems should be designed at the same time as your product, so you never lose a customer because you screwed up 10 years of accounting and the backups are unreadable.

Data quality is important to the coder

If you want to know where you are going, what happens in your game, and feel the effects of the changes you make to the world, you need some feedback systems. And there you start to pull your hair when you try to automate checks or data aggregation with non normalized data. For instance, if you want to make statistics over genders of in-game characters, and let’s say gender is a free-text field, then you’re in trouble: most people won’t fill the box, some will fill garbage, and some will try to fill it, but are not able to spell properly (yes, it happens). Therefore, you have useless statistics, stating that 30% people play a girl, 60% a boy, 1% a ‘dude’, and 9% of players actually play a droid.

Lesson: the earlier you freeze possible entries for your data, the less corruption you will have in the future. You never know what data could be useful one day.

“Hell is other people”

Amongst all the different behaviors you can meet, the script kiddie is the most difficult you can have to deal with. This guy will try to input all kind of data into your forms, hoping it will crash and get him access to something interesting. He will hammer your server with requests, because he read somewhere he could become root somehow by doing it.

Needless to say, he will more often crash your game than root it, and also make a lot of people frustrated because they just try to play and the game is down again. He will also create a myriad of fake game accounts, and automate them so he becomes a one man army.

Lesson: you should build your security model first, and then add features on top of it it, not the other way around, or there will always be a hole in which a malicious user will try to sneak in.

Copy-Paste is your best friend as long as you’re not sleep deprived

“I’m in a rush, I’ll just copy this old code and change it so I’m done earlier”. Except you don’t change it. Or worse, you don’t change it enough, and suddenly have all those crazy bugs coming from everywhere. You check and double-check the changes you just made, but can’t find where the issue is. Usually, it’s because you just replaced 3 parameters, but forgot the 4th, and your brain skips it every time you read it.

Lesson: instead of copy-paste, think about refactoring your code, write generalist functions and don’t duplicate code anymore !

Version control or die

There was a time, I did not know about version control at all, and trusted my luck to revert changes made over 3-5 files. It can sometimes work, but sooner or later, you’ll forget one of the changes, and then you’re stuck in a looong bug hunt. Repeat again, and then your entire code base start working against you, showing incorrect data to your players, or granting them with unwanted powers. This actually happened to me, and I decided to restart the whole project, from scratch, but with version control from day one.

Lesson: programming without version control is like walking on a tightrope: one day or another, you’ll fall, and this day, you’ll regret not to have a safety net.

Comments closed

openpyxl 1.5.0 released

It has been almost one full year since I started this project, and it has now reached a state where it is well suited for production, and intensively tested by an increasing number of people around the globe.

The most recent additions I’m the most proud of are the optimized reader and writer, that become stable in this version. It took around 6 months to get them working properly (the reader was the hardest actually), and now they’re here !

You can read and write workbooks of any size, with low and almost constant memory consumption (which is not the case with Excel actually).

I’m also really happy that people keep sending patches and asking for features, so the project continues to live, even when I’m not fully available to work on it.

You can get the latest version of openpyxl either with easy_install:


easy_install openpyxl

or from the official website.

7 Comments

SSL Secured Piston Webservice

On FreeBSD, there are a few gotchas to work with Apache + SSL + Piston.

Here are my findings:

  • Enabling SSL in Apache 2.0

As most SSL-related functions are enclosed in <IfDefine SSL> blocks, adding

apache2_enable=&quot;YES&quot;
apache2_flags=&quot;-D SSL&quot;

to /etc/rc.conf will enable them.

  • Disabling _default_ SSL Virtualhost

There’s a _default_ virtalenv defined in the ssl.conf file, and activated at the same time as the rest of the SSL config.

I didn’t find a “clean” way to disable it, and it was conflicting with my own virtualhost, so I encapsulated if between <IfDefine SSLVH> tags and it did the trick 🙂

  • Generating SSL keys

I followed a guide found on google (in French). Extremely useful.

Copied them to /usr/local/etc/apache2/ssl.key/ and /usr/local/etc/apache2/ssl.crt/

  • Updating my virtualhosts to fetch HTTPS requests

As I disabled the _default_ virtualhost, I needed to make a copy of my existing (port 80) virtualhost, and merge it with what was defined in the _default_ one.

&lt;VirtualHost *:443&gt;

  ServerName servername.com

  SSLEngine On
  SSLOptions +FakeBasicAuth +ExportCertData +StrictRequire

  SSLCertificateKeyFile /usr/local/etc/apache2/ssl.key/server.key
  SSLCertificateFile /usr/local/etc/apache2/ssl.crt/server.crt

SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL
&lt;FilesMatch &quot;.(cgi|shtml|phtml|php3?)$&quot;&gt;
    SSLOptions +StdEnvVars
&lt;/FilesMatch&gt;
usr/local/www/cgi-bin&quot;&gt;
    SSLOptions +StdEnvVars
&lt;/Directory&gt;

SetEnvIf User-Agent &quot;.*MSIE.*&quot; 
         nokeepalive ssl-unclean-shutdown 
         downgrade-1.0 force-response-1.0

CustomLog /var/log/httpd-ssl_request.log 
          &quot;%t %h %{SSL_PROTOCOL}x %{SSL_CIPHER}x &quot;%r&quot; %b&quot;

[...]
&lt;/VirtualHost&gt;
  • Open port 443 on the firewall

Almost forgot this one 🙂

Comments closed

Piston + HTTP Authentication + mod_FastCGI

I’ve spent much more time than necessary on this issue, so if it can help someone, I’m posting a working config for Django Piston + Apache’s mod_fastcgi + HTTP Basic Authentication.

Works with :

  • Apache 2.0.63
  • Django 1.2.3
  • Piston 0.2.2

I’ve updated the default config file found on the official Django website:

FastCGIExternalServer /project/dir/mysite.fcgi -socket /project/dir/mysite.sock -pass-header Authorization

&lt;VirtualHost 123.456.78.9&gt;
ServerName servername.com
DocumentRoot /project/dir
Alias /media /project/dir/media
RewriteEngine On
RewriteRule ^/(media.*)$ /$1 [QSA,L,PT]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteRule ^/(.*)$ /mysite.fcgi/$1 [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},QSA,L]
&lt;/VirtualHost&gt;

tl;dr: -pass-header Authorization and [E=HTTP_AUTHORIZATION:%{HTTP:Authorization},QSA,L]

2 Comments

openpyxl starts being used

When I started working on openpyxl a few months ago, I didn’t know it would catch that much activity around it. I’m very happy to see that it can apparently help  so many people 🙂

I’ll try to follow-up on the bug fixes and new features as far as my time permits, and will usually answer emails within the day. Thank you everyone for using the library, even though it is still far from being perfect 😉 Keep posting bugs on the tracker or ideas and requests on the mailing list !

1 Comment

openpyxl is on PyPi

Yes, I know, yet another post about openpyxl 😉

This time I’m announcing the release of the 1.1.0 version on PyPi.

As mentioned in the Wiki, you can now just type easy_install openpyxl to get the latest released version, if you are not much into getting the snapshop from bitbucket.

Also, thanks to Yaroslav’s great job, openpyxl is also packaged under Debian testing !

Now there is no valid reason for not giving it a try… maybe except that darn memory footprint…

1 Comment

openpyxl turns 1.1

After two weeks of intense activity around openpyxl, I’m releasing version 1.1 today. This new version brings support for dates and number formats.

Several bugs have been fixed, thanks to the careful testing of two new contributors, Jonathan Peirce and Yaroslav Halchenko, both working on the PsychoPy project.

Thanks guys for boosting my morale, providing valuable advises and patches !

Many thanks goes to Marko Loparic for his support and enthusiasm 😉

You can get the sources for the latest version here http://bitbucket.org/ericgazoni/openpyxl. I expect a lot of bug reports with this new version, as it is stable but not extensively tested yet, and that the number of possibilities have seriously increased with the introduction of number formatting.

Keep in mind that the memory footprint is still high, but that it is the target for milestone 1.2. It should perform reasonably well if your needs are moderate (<100.000 cells), but if you want to add more data, then it might start consuming RAM pretty quickly. This holds for writing and reading.

Memory consumption is almost linear, and a 15MB workbook results in 450MB in RAM.

There is also a new mailing list for the project: http://groups.google.com/group/openpyxl-users. It’s pretty empty for now, but feel free to ask questions there, I’ll be reading it regularly.

Bug reports will be better handled if they are filed on the project bug tracker: http://bitbucket.org/ericgazoni/openpyxl/issues/new.

Happy coding !

Comments closed

Why we do what we do

Yesterday, I went to a Master degree graduation ceremony at the local university. The introduction speech addressed to the soon-former-students was extremely interesting.

It recalled me why we work.

If  you listen to the biblical sayings, it’s because mankind is a band of sinners that were  thrown out from Paradise. However, there are still people when given the opportunity not to work at all that still find some occupation instead of just sunbathing on a mexican beach.

The speaker made a clear distinction between “job” and “work”, between what he called “paid work” and “life work”.

For many of us, working is a kind of prostitution of ourselves: we trade our time against money. But maybe there’s more than just money ? Maybe there are greater extends, greater causes, greater expectations ?
For many of us, working is just what we were told to do to “earn” our life, but in the end, is a life spent hunting for our food is a life worth living ?

For many of us, we could completely miss the point of our own life, because the society says we shouldn’t look for our inner desires, dreams and thoughts, and instead get rich so we can afford everything salesmen try to sell us, like a new(er) mobile phone, a bigger car, a bigger house to put our bigger LED 3D TV.

Or … maybe there is some art to do in this world. Maybe there is some mystery awaiting for us to solve it ? Maybe we could replace the word “work” by “craft”, that conveys a totally different meaning ?

Maybe we can use our work as a medium to express ourselves, to add meaning over marketable work.

As Leonardo da Vinci said:

As a well-spent day brings happy sleep, so a life well spent brings happy death.

Had he only done his job (painting, sculpting), even very well, he wouldn’t have achieved such fame as he has now, as museums are filled with less famous artists, but who had of course great skills. What made the difference is that da Vinci was inventing new techniques that made his work astoundingly better that any competitors. He spent his life looking at the World straight in the eyes, and constantly invented new painting techniques, described physical and biological principles, designed music instruments, bridges and weapons, etc…

He was well paid to do so, but even the most well paid engineer wouldn’t spend his life working in such way if it wasn’t coming from within his own self. I don’t think Leonardo felt he traded his time for money. I believe he would not have stopped if the payment stopped, because it was a personal quest for beauty and knowledge he was on.

Masterpieces are always made by passionate people, who are not looking for money, but to achieve a higher level of mastery in their “art”, artistic or technological.

So now, what about transforming our daily job into masterpieces ?

1 Comment

openpyxl reaches 1.0 mark

After a few more efforts, I am pleased to announce the release of the first version of openpyxl.

The reader and the writer are working and tested for strings and numbers.

I have been able to read and write simple Excel 2007 xlsx files from Python and open them with Excel.

You can clone the repository using Mercurial:

hg clone https://ericgazoni@bitbucket.org/ericgazoni/openpyxl

or download the release in zip format.

Edit: 1.0 release is really outdated, you might want to get a more recent version here.

The (sparse for now) documentation can be found on the wiki.

Reader usage (using the “empty_book.xlsx” file from the previous example)

from openpyxl.reader.excel import load_workbook

wb = load_workbook(filename = r'empty_book.xlsx')

sheet_ranges = wb.get_sheet_by_name(name = 'range names')

print sheet_ranges.cell('D18').value # should display D18

Code is published under the MIT licence, so you can use it for whatever use you need, and I’d be very happy if  you drop me an email if  you use it 🙂

If you don’t find it useful, spot a bug, or want to suggest an enhancement, you can do so by filling a ticket on the tracker.

Features that will be added in the next version are listed here, so if you need something in this list, please be patient or send me a message to tell me to hurry 😉

11 Comments

openpyxl: simple writer done

I’ve been very busy on openpyxl the last few days, and I managed to get a working writer for basic data types (strings, numerics).

For the impatient, you can clone my bitbucket repository:

hg clone https://ericgazoni@bitbucket.org/ericgazoni/openpyxl

It’s still a work in progress, so expect some quirks here and there, and if that happens, please file a new issue here.

If you like it, you can also drop a comment below or send me an email (see Contact page).

Usage is pretty simple as you can see:

from openpyxl.workbook import Workbook
from openpyxl.writer.excel import ExcelWriter

from openpyxl.cell import get_column_letter

wb = Workbook()

ew = ExcelWriter(workbook = wb)

dest_filename = r'empty_book.xlsx'

ws = wb.worksheets[0]

ws.title = &quot;range names&quot;

for col_idx in xrange(1, 40):
    col = get_column_letter(col_idx)
    for row in xrange(1, 600):
        ws.cell('%s%s'%(col, row)).value = '%s%s' % (col, row)

ws = wb.create_sheet()

ws.title = 'Pi'

ws.cell('F5').value = 3.14

ew.save(filename = dest_filename)

Next features are:

  1. a working reader (so that I can read back files generated by the writer)
  2. dates support
  3. calculations
  4. formatting
Comments closed