Django and Memcached

This entry was posted on Monday, June 30th, 2008 at 6:21 pm and is filed under django. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.

I just ran into several problems while setting up memcached for a django installation, that is replicated over several servers. One of the problems was the compiling of cmemcache, a python extension for libmemcache, where I received the following error:

‘CmemcacheObject’ has no member named ‘mc_ctxt’


First I didn’t realized the additional file, that resides at the file list of downloadable cmemcache versions. With this file you can patch the libmemcache library so cmemcache can be build. Here the steps to compile and install cmemcache:

cd ~
wget http://people.freebsd.org/~seanc/libmemcache/libmemcache-1.4.0.rc2.tar.bz2
wget http://gijsbert.org/downloads/cmemcache/libmemcache-1.4.0.rc2.patch
wget http://gijsbert.org/downloads/cmemcache/cmemcache-0.95.tar.bz2
# the patch was made using a folder called reference
mkdir reference
cd reference
tar xjvf ../libmemcache-1.4.0.rc2.tar.bz2
cd ..
patch -p0 < libmemcache-1.4.0.rc2.patch
cd reference/libmemcache-1.4.0.rc2
./configure && make
sudo make install
cd ../../
tar xjvf cmemcache-0.95.tar.bz2
cd cmemcache-0.95
./configure && make
sudo make install

After doing these steps you should be able to do this in your python console:

import cmemcache

Instead of the cmemcache library you can also install python-memcache, that is a memcache library completely written in python. But the cmemcache is definitely faster and if you have installed both libraries, django will favor the c implementation in front of the python one.

The other problem with memcached and django relies on a misunderstanding of mine. I thought, that if I define several memcached instances in my CACHE_BACKEND setting of django, it sends every new or changed key/value pair to each server in the list and holds each one redundant on all memcached instances.

So I've defined the following CACHE_BACKEND setting on Django Server A (192.168.1.1):

CACHE_BACKEND = 'memcached://192.168.1.1:11211;192.168.1.2:11211/'

and on Django Server B (192.168.1.2):

CACHE_BACKEND = 'memcached://192.168.1.2:11211;192.168.1.1:11211/'

and assumed, when a request on the cache was made, that Django Server B would first try to access the memcache instance on Server B. This assumption was totally wrong and on top of that I also recognized, that no saved cache-entry from Django Server A was available on Django Server B.

So I've started to read the documentation of memcached and came across the following faq entry. There you can read, how memcached is working and how values are saved across several instances:

When doing a memcached lookup, first the client hashes the key against the whole list of servers

A result of this is, that the setting of CACHE_BACKEND for memcached servers must be the same on all django installations. Otherwise the hash for a key/value pair would be a different one on each server and you are not able to access cached values across the installations. Another result is, that a cascade of memcached servers act as one big cache server with the main goal to scale large. You never know on which of the memached servers your cached value is saved.

Additional note!
If you are running memcached instances on servers that are far distant from each other (like a server in Germany and one in the US), you shouldn't use several memcached instances as one cluster. Replicate your memcached instances with the modified memcached server called repcached instead. Otherwise the cache lookups will be too slow.

5 Responses to “Django and Memcached”

  1. Comment by Anderson — July 20, 2008 @ 5:40 am

    thanks so much, now I got cmemcached installed ;)
    on my freebsd server, I had to install libmemcached only using ports otherwise it didn’t work


  2. Comment by Igal — October 2, 2008 @ 8:39 pm

    Hi,
    Thanks a lot for this article.
    The only problem I had is a “undefined symbol: mcm_buf_len” error. So I have found the “good” patch here:
    http://cia.vc/stats/author/semen/.message/8c85
    then it worked like a charm )))

    Cheers


  3. Comment by Andrew — October 28, 2008 @ 1:55 am

    Same error as Igal, but the patch he mentioned didn’t work for me — said it was malformed. I used this one instead: https://svn.pardus.org.tr/pardus/devel/programming/libs/libmemcache/files/libmemcache.patch


  4. Pingback by Installing cmemcache on Ubuntu 8.10 (almost) « Development Doodles — February 12, 2009 @ 1:42 am

    [...] not the first one experiencing this issue, as can be seen here and here. One of the comments in the second link mentions another patch, which supposedly should work better [...]


  5. Comment by Greg Metsker — March 6, 2010 @ 9:37 pm

    Thanks for the article. I am new at python and this was a big help.


Leave a Reply