WEP

There must be plenty of examples of when encryption was used incorrectly, and how it lead to unforeseen consequences. The most famous example I am aware of is WEP. I won’t go into the details. There are plenty of resources online describing the flaws in great detail. The essence of it however was that WEP used relatively good ciphers incorrectly, or insufficiently. The encryption algorithm it used (RC4), was not ‘broken’. The algorithm itself is quite robust (No algorithm is perfect, but this wasn’t the major flaw in WEP). It’s the design of WEP, not the core algorithm, that were broken.

Some illustration

Lets walk through a simplified, imaginary scenario, to illustrate when NOT to use encryption. It goes something like this:

Developer: “We need to allow access to the X section of the website to more people”
Me: “Ok, then ask them to login before they can have access”
Developer: “No. We want to send them a secure link with expiry date and not all users will have an account”
Me: “In that case, we should use something like HMAC or OAUTH. It’s designed for this purpose”
Developer: “No. I don’t have time to read about Oauth, it’s too confusing. But I’ve written a real cool function that encrypts the part of the URL, and only if we can decrypt it properly, we allow access… It uses Triple-DES, so it’s super-secure. Wikipedia says that The algorithm is believed to be practically secure in the form of Triple DES, although there are theoretical attacks”
Me: “Lets have a look at your ‘secure’ solution then…”

The solution was something along those lines (hugely simplified for the sake of illustration):

http://site.com/secure-access/{encrypted_string}

where encrypted_string contained the expiry date of the page, e.g. 20120315 (15th March 2012).
Can you see where this is flawed??


There are at least a couple of possible attacks here, that don’t involve anything with breaking the encryption or trying to get the key:

1. Randomly change the encrypted string – perhaps not very sophisticated, but since all we need is a date larger than today, there is still a fair bit of chance we might be lucky. Even without calculating the entire search space, it’s quite easy to see it is quite narrow.
2. Replace it with another encrypted string – this is even easier. All you need as another URL string with a known-to-work encrypted string. Copy this string and paste it, and you’re good to go!

Another flaw the developer hasn’t worked out was that he was using triple-DES in ECB mode. This is the most basic mode, which makes the attacks I described much easier. So even if the algorithm itself is robust, the way it is applied can be much more important. In addition to that, under some circumstances, if the encryption process failed (which it could easily, since there was no problem manipulating the encrypted string), the code was very kind as to output the decrypted string inside the error message… Back to the user (or wannabe hacker as the case may be).

Of course, the solution was a little more complicated. It contained not just an encrypted date, but some other data. Nevertheless, the same principles of attack applies. There are many bit-flipping attacks, that can alter encrypted data and generate a predictable output. The bottom line is this: Encrypting data does not prevent modification.

The alternative

In this case, all we want to achieve is authentication. We want to verify that the request was legitimate, and that nobody else can fake such a request. There’s nothing in particular that needs hiding. The expiry date of the access is not such a big secret. All we need is some kind of a signature, to validate the legitimacy of the request. This is where hash functions, and HMAC / Oauth come into play. Those mechanisms, for some strange reason, are less appealing to many developers. I’m not entirely sure why, but maybe it’s just not as fun to see an extra hash at the end of the url as it is to encrypt a string. These mechanisms are much more effective in this case. This is what they’re designed to do.


So how does this work? Again, for the sake of simplicity I’m not going to cover the detailed aspects of those algorithms. But the principle is quite simple: You take a string you want to ‘sign’, plus a secret key, and generate a unique hash value from both of them. This value will be totally different even with the slightest modification to the original string (or the key). The key will never be published or shown, only the string with the generated ‘signature’ (this unique hash value we produced). How is this more secure? As I mentioned earlier, even the slightest modification of the string will produce a completely different signature. This will prevent any undetected modification of the url. Producing the unique hash signature is virtually impossible without the knowledge of the secret key. Voila! Simple. Secure. Elegant. Of course, as always, god is in the details, so even these algorithms can be used wrongly, producing an insecure outcome. However, from my experience, following the guidelines and using the oauth/hmac libraries is far easier and less error-prone than using any encryption algorithm.

Size doesn’t matter, it’s how you use it

The Security of the WEP algorithm page sums it up quite nicely: “The [WEP] protocol’s problems are a result of misunderstanding of some cryptographic primitives and therefore combining them in insecure ways”. Even if you pick the best and most secure encryption algorithm, it might not be enough to make your solution secure. In fact, as I tried to illustrate, encryption might not be necessary at all.

Dynamic Goal Conversion Values

I was trying to get some dynamic goal conversion values into Analytics. I ended up reading about Ecommerce tracking and it seemed like the way to go. Not only would I be able to pick the goal conversion value dynamically, it gives you a breakdown of each and every transaction. Very nice. After implementing it, I was quite impressed to see each transaction, product, sku etc appear neatly on the ecommerce reports. So far so good. But somehow, goals – which were set on the very same page as the ecommerce tracking code – failed to add the transaction value. The goals were tracked just fine, I could see them adding up, but not the goal value. grrrr…

I was particularly frustrated after meticulously reading through the Ecommerce Transaction Page Goals, which is very clear about this being possible, as long as you set the conversion value to zero.

The solution

There’s a small thing that doesn’t seem to get mentioned anywhere, and I’m still not sure why it causes a problem. If it doesn’t work for you, I have a few alternative approaches worth trying too…

I discovered that in my case, the google analytics code was included on the goal conversion page twice. Once, as in every page, the standard tracking code on the header. And again on the body, together with the ecommerce tracking values. Both were running fine, and as I mentioned the results appeared. But somehow it seems to create some conflict. Removing the ‘standard’ tracking section seemed to have solved this mysterious problem.

Here’s what the HTML looked like – Remove the first part

<head>
...
<!-- FIRST SECTION ON THE HEADER : TO REMOVE -->
<script>

  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-xxxxxxxx-x']);
  _gaq.push(['_trackPageview']);

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();

</script>
<!-- DON'T REMOVE ANY FURTHER -->
</head>
<body>
...
<!-- SECOND SECTION - KEEP THIS ! -->
<script>
  var _gaq = _gaq || [];
  _gaq.push(['_setAccount', 'UA-XXXXX-X']);
  _gaq.push(['_trackPageview']); // this tracks the page view
  _gaq.push(['_addTrans',
    '1234',           // order ID - required
    'Acme Clothing',  // affiliation or store name
    '11.99',          // total - required
    '1.29',           // tax
    '5',              // shipping
    'San Jose',       // city
    'California',     // state or province
    'USA'             // country
  ]);

   // add item might be called for every item in the shopping cart
   // where your ecommerce engine loops through each item in the cart and
   // prints out _addItem for each
  _gaq.push(['_addItem',
    '1234',           // order ID - required
    'DD44',           // SKU/code - required
    'T-Shirt',        // product name
    'Green Medium',   // category or variation
    '11.99',          // unit price - required
    '1'               // quantity - required
  ]);
  _gaq.push(['_trackTrans']); //submits transaction to the Analytics servers

  (function() {
    var ga = document.createElement('script'); ga.type = 'text/javascript'; ga.async = true;
    ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';
    var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);
  })();
</script>
...
</body>

Alternative Method

Since analytics seem to encompass lots of black magic, I’m still not 100% sure that’s the only possible solution. However, along the way I discovered some other interesting things…

Event Tracking

Instead of goal tracking by URL, you can actually track an event. If you’ve already coded the ecommerce tracking section anyway, adding event tracking to it is a doddle. Simply add this line somewhere before or after the ecommerce tracking code

_gaq.push(['_trackEvent', 'Cart', 'Checkout', 'Success', 39.99]);

In this case I defined the event category to be ‘Cart’, the action to be ‘Checkout’, the option to ‘Success’, and the value to 39.99. Once you do that, you can set your goal to use an Event instead of a URL. Here’s more on info on setting the event goals and choosing the value as the conversion value.

If your main concern is tracking conversion values, and you’re not too fussed about the whole ecommerce tracking, then this is actually a much easier way. All you need is one line of code. It can however work perfectly well together with your existing ecommerce tracking.

Wet paint – slow updates

Analytics seems to update rather slowly. It also seems to update and un-update some data if you keep refreshing. You see a new transaction. You refresh. It’s gone. Refresh again, it’s back… This is normally not a huge problem, just confusing, but when you’re trying to debug something, can make things much more difficult. For example, if you update your Goal configuration. Say you change the URL, or the tracking event, or added a new goal. It might take a good few minutes for it to update. If you trigger the goal, e.g. by going to the URL page, it might not trigger – because Analytics still uses the old goal settings. You simply have no way to find out whether the goal is active or not. You must wait for the paint to dry before you try anything new.

Other stuff

A couple of more things to watch out for:

• The Ecommerce page says that
  

  For URL: Supply the URL for your shopping cart. For example: http://www.we-sell-for-you.com/mysite/myCart.asp

  In most cases, you only need to supply the relative path, e.g. /mysite/myCart.asp

• I’m not sure whether it makes a difference, but if you’re using _gaq.push(['_setDomainName', '.domain.com']);, this might also cause some issues? (didn’t test this explicitly though)

Final grunt

Hope this is helping someone. I was pulling my hair over this so hope I can save some other people’s hairline from receding. The time it takes for the goal/ecommerce tracking to appear on analytics also makes testing it rather long and much more difficult to resolve. Having tested a few too many different combos, I’m still not 100% sure this is a bullet-proof solution, but I’m definitely seeing some goal conversion values now.

UPDATE: looks like it will only work if you also use event tracking. With URL tracking alone it doesn’t seem to pick the goal values.

This is not normally a problem, unless the URL is already encoded. Since all unicode-based urls use this encoding, they are more prone to these errors. What happens then is that a URL that looks like this:
http://www.frau-vintage.com/2011/%E3%81%95%E3%81%8F%E3%82%89 …

will be encoded again to this:
http://www.frau-vintage.com/2011/%25E3%2581%2595%25E3%25 …

So clicking on such a double-encoded link will unfortunately lead to a 404 page (don’t try it with the links above, because the workaround was already applied there).

A workaround

This workaround is specific to wordpress 404.php, but can be applied quite easily in other frameworks like django, drupal, and maybe even using apache htaccess rule(?).

<?php
/* detecting 'double-encoded' urls
 *  if the request uri contain %25 (the urlncoded form of '%' symbol)
 *  within the first few characeters, we try to decode the url and redirect
 */
$pos = strpos($_SERVER['REQUEST_URI'],'%25');
if ($pos!==false && $pos < 10) :
    header("Status: 301 Moved Permanently");
    header("Location:" . urldecode($_SERVER['REQUEST_URI']));
else:
    get_header(); ?>
    <h2>Error 404 - Page Not Found</h2>
    <?php get_sidebar(); ?>
    <?php get_footer();
endif; ?>

This is placed only in the 404 page. It then grabs the request URI and checks if it contains the string ‘%25′ within the first 10 characters (you can modify the check to suit your needs). If it finds it, it redirects to a urldecoded version of the same page…

Efficient code

Django makes a lot of things very easy. One of its most prominent features is its ORM (Object Relational Mapper). The django ORM really makes it easy to write complex database queries without a single line of SQL. This however might come at a price of sub-optimal queries, which can take a very long time to process, and also consume more memory. If you hit django memory issues, it’s very likely related to some heavy data being loaded from the database, consuming lots and lots of memory. Luckily, there are a few rules that if followed can hugely improve both the queries and memory usage. Most of those are documented clearly on the Database access optimization page. I won’t repeat everything that’s being documented there, but rather focus on a few key points.

Where do you start?

First of all, you might already see crashes on your django, get reports from users about pages loading slowly, and you’ve hopefully checked your server (e.g. using top/htop or free commands) and noticed high memory utilization. The recommendations on part I should give a little stability until you manage to figure out where the problem lies, but it is by no means a sufficient solution.

Monitor log files

The easiest place to start is your log files. If you’re not logging requests already, you seriously should. One helpful method that doesn’t add much overhead to django logging is adding the time that the request took to the logs. This can be achieved with a simple logging middleware. The time each request takes usually gives a fair indication of what’s going on, and from my experience there’s usually a close link between execution time and memory footprint. The same logging middleware can be used with DEBUG=True to view how many SQL queries are executed for each request. This can give another indication of ‘hot-spots’ to look for.

Profiling

The next step is to profile django and see which requests consume high amount of memory. I would suggest following the instructions on Using Guppy to debug Django memory leaks. This should really help pin-pointing areas of code that consume large amounts of memory. Once those are identified, it is easier to try to optimize the code there. Don’t try to do everything at once, but from optimizing one area of code you would learn a lot and could easily apply the same methods across the entire codebase.

Quick-n-dirty improvements

There are a few recommendations that generally help with reducing memory footprint. Be aware not to use them blindly though. They can have a knock-on effect on performance in other areas. So exercise some judgment. The most useful pointers are:

• Make sure DEBUG=False
• .iterator() – In many cases, there’s simply a need to retrieve a list of objects from the database, and return those in a certain format. If you go over the list more-or-less sequentially and use each object only once, it makes a lot of sense to use iterator() – this will eliminate django internal caching, which can save on memory. If, however, you might go back to the same object – this caching could really speed things up, so be aware of what you’re doing
• Using .update()/.delete() – can be a huge saver if you perform a simple update of many objects, or delete a bunch of them. Rather than walking through objects one by one, you can perform this operation with one query
• Using .values() – As above, if all you need is a bunch of values from the database, you can save a lot of memory by fetching them as values instead of as complex queryset objects
• Using .only() – can also save a lot of memory, if you only use a portion of data from each object.

These are just a few pointers to the same page on the django documentation. Please spend time reading through the entire page, as it contains many very useful techniques and other information to help you better understand how the ORM works and how to use it more effectively.

External Monitoring

Even if you manage to optimize your code, tweak the maxrequests parameter to ensure memory is cleared regularly, and brush your teeth morning and night, there are still cases when it simply won’t be enough. One user will make too many requests, the data-set will just be too big, or some query will stay sub-optimal, and you’d still end up with bloated processes eating through your memory. If that happens, you’re almost back to where we started. This is where some external tools can work wonders. They can:

1. Let you know when this happens, perhaps even before memory is completely depleted.
2. Take (automatic) action, and more gracefully restart django, to avoid a full-scale crash.

Monit

My weapon of choice in this case (and many others) is Monit. It’s lightweight, powerful and has a very easy and intuitive configuration syntax, which makes it a snap to use. There are so many uses of this little devil, but in this case I will focus on monitoring process memory. It only takes a few lines to let monit watch over django, and make sure things are running smoothly:

check process your-django-process with pidfile /path/to/your/django.pid
    start program "/etc/init.d/your-django reload"
    stop program "/etc/init.d/your-django stop"
    if totalmem > 70% then exec "/usr/local/bin/highmem" # more about this highmem later...
    if totalmem > 85% for 2 cycles then restart

Lets go over this monit config snippet quickly. What it does is very simple:

1. Monit checks our django process. Make sure you specify the correct pid file (which we set when invoking django using manage.py)
2. start and stop program directives should point to your django daemon script
3. If the total memory of the django process (including children) exceeds 70% of available memory, then it executes a custom ‘highmem’ command. The highmem command will get django to clear some memory by restarting its internal processes. We will cover this command shortly. This will also automatically send an email to alert you (make sure you configure your monit alert settings correctly)
4. The second check is a “safety-belt”, to restart django if memory stays high for too long despite all our efforts

highmem

The highmem command is a very simple 1 line bash script:

#!/bin/bash
kill -SIGUSR1 `cat /path/to/your/django.pid`

All it does is send a SIGUSR1 to our django process. So what does SIGUSR1 do? When django runs in fastcgi, it uses flup for process handling. Since version 1.0.3, flup uses the SIGUSR1 signal to safely respawn those django processes. Popping those balloons. If a request is in-progress, it will wait until it finishes, which is a very nice feature. Just hope that this process waiting to complete won’t take out ALL memory left… You can read more about it on How to Gracefully Restart Django Running FastCGI (look for the comments section in particular). Please note that you’d need flup version 1.0.3. If you’re using an older version, it might just kill your django processes instead of respawning them safely. It’s easy to get flup, simply use sudo pip install flup (or sudo easy_install flup), and you’re done.

The last resort

The last thing on our arsenal of tools, the last resort, is if some process is eating our memory so fast, that even the highmem command didn’t manage to release it, and we simply have no choice but to restart django. However, even then, we’d probably want to do it as gracefully as possible. I’m not going to repeat it, but the page on How to Gracefully Restart Django Running FastCGI covers what you need to do to modify your manage.py in order to give django just a few seconds before it’s restarted. Then just make sure that the scripts starting/restarting django send a KILL -HUP to your django process to restart it nicely.

Does django leak memory?

In actual fact, No. It doesn’t. The title is therefore misleading. I know. However, if you’re not careful, your memory usage or configuration can easily lead to exhausting all memory and crashing django. So whilst django itself doesn’t leak memory, the end result is very similar.

Memory management in Django – with (bad) illustrations

Lets start with the basics. Lets look at a django process. A django process is a basic unit that handles requests from users. We have several of those on the server, to allow handling more than one request at the time. Each process however handles one request at any given time.

But lets look at just one.

cute, isn’t it? it’s a little like a balloon actually (and balloons are generally cute). The balloon has a certain initial size to allow the process to do all the stuff it needs to. Lets say this is balloon size 1.

Now every request that comes to the server gets sent to one of those (cute) django processes. Then to serve the request, the process loads objects into memory. Like this

Those little bubbles are the objects loaded into memory. Once the process finishes processing a request it will clear all the objects from memory and go back to being ‘empty’. It is still size 1 since all the objects fitted within the space.

But some time the request is a bit heavier. It needs to load more objects than its size.

So the process simply inflates itself and grows a little. Easy. Now it’s size 2. More space for bubbles.

and of course, once the request finishes, it clears all those bubbles and there’s space for the next ones.

An import thing to note: The balloon (process) never shrinks. It can only grow. But this is (kind-of) ok, since it will never grow bigger than the biggest request we can get. So even a very big request (lets say one that uses 1Gb memory), we can probably handle. Right??

Not quite. So what’s the problem?

Well, like this little cute process we have other processes. Remember we have to serve more than one user at a time. So we must keep a few of those balloons running. So if more than one BIG request come at roughly the same time, they will inflate not just one balloon, but a few of those. And these balloons compete for space on the server (which is like a big room that contains the balloons, but the room does not grow).
This is our room:

Of course we can clear the room and start empty – this is what we do when we reboot the server or even just restart django. This is what we have to do when the server crashes. When in fact the balloons grew so big that other balloons couldn’t grow any more. So rebooting all the time is not an option. When we do that, everything stops. Including requests that are being processed. Even if they’re half-way through.

So – how about we ‘pop’ those balloons every now and then – when they’re NOT processing a request (other balloons do it), and start from a small balloon? That’s actually possible. However, there are two limitations to be aware of:

• To create a new balloon takes some effort. We have to ‘make’ the balloon. While we make a balloon the others are responding slower.
• We cannot just ‘pop’ a balloon based on its size. Instead we can only create an ‘automatic balloon popper’ that pops the balloon after X requests.

Our degree of control is as follows:

1. minspare – How many empty balloons do we start with. This will potentially save us effort later by having a few ready. The ‘cost’ in term of memory is the {number of balloons} X {balloon initial size}. The benefit, is saving time creating a new process for simultaneous requests. However, This parameter is not very helpful to our problem.
2. maxchildren/maxspare – What is the maximum number of balloons/processes we want to have on the system. This determines the maximum number of simultaneous requests we can deal with. The ‘cost’ is the {number of balloons} X {balloon size}. The balloon size can obviously grow over time!
3. maxrequests – this is the ‘auto-popper’. We can decide after how many requests we ‘pop’ a balloon and start a new one.

So if we set maxrequests too low, say 1 – then the system will work very hard to create a new process/balloon for every request. This is silly if the request is very small and doesn’t need a big balloon. With too high value however, the balloons might grow too much before they’re popped. Even if the maxrequests is 1, if we get a few requests at the same time, each causing our balloons to grow too much, we might still run out of space!

Our worse-case scenario is calculated by : {number of simultaneous requests} X {size of the request}. Lets say our server have 4Gb memory in total, which probably leaves about 3Gb memory for django itself. However, with requests that might take ~1Gb in memory (worst-case-scenario), we can only serve a maximum of 3 such requests. Not even simultaneously. Just in proximity to each other, before the server runs out of memory…

Conclusion

One of the core issues I wasn’t addressing here is obviously how to prevent high-memory usage within the django process. I hope to cover this on the next part. There are certainly some recommendations and best-practices when it comes to memory usage. However, with some types of requests, it might be impossible to avoid high-memory usage. Given enough simultaneous requests, even with optimization that leads to ‘reasonable’ memory utilization, django might still run out of memory. The minspare, maxchildren, maxspare, and most importantly maxrequests parameters are therefore crucial to having a more stable django service. It’s not a bullet-proof solution, but from my experience it helps a lot.

Sweet-Spot settings

So what are my recommended settings? I found that setting maxrequests=100 seems to give a reasonably good performance overall. Simply run django in prefork mode with something like this:

manage.py runfcgi method=prefork host=$DAEMON_HOST port=$BACKUP_DAEMON_PORT pidfile=$PIDFILE maxrequests=100

I didn’t see any need to change the default minspare, maxchildren, or maxspare parameters however.

What’s next?

On Part II I am going to cover some more advanced tweaks. Those are designed to detect and recover from situations where django runs out of memory. Using the balloon popping analogy, those tweaks/methods allow ‘popping balloons’ when memory runs out, rather than only after 100 requests. This gives another layer of protection against memory-related crashes. However, these require monitoring tools outside django. In addition, I hope to give at least some pointers on how to better utilize memory within the code.

Luckily no websites I know or maintain were affected, possibly since the htaccess change I used shouldn’t allow using remote URLs in the first place (and also it renamed timthumb.php from the url string, making it slightly obfuscated). I still very strongly advise anybody using timthumb to upgrade to the latest version to avoid risks.

Ajaxize

The plugin allows you to take any wordpress function and ‘ajaxize’ it into a special div. Typically, all plugins and core wordpress functionality boils down to a number of php functions. If you are able to figure out which function your plugin uses to output content, you can ajaxize it. How do you find the function name? This is not that complicated. Many plugins will actually tell you which function to use. For example, if you want to embed the output in one of your templates. They will instruct you to use something like this:

<?php echo plugin_function_name(); ?>

So all you have to do is take this ‘plugin_function_name’, and ajaxize it using my plugin. The output is a div which looks like this:

<div id="ajaxize_this:plugin_function_name:68e46660f7ce3bc77a51465219df5743879544bc"></div>

Then place this div inside your page, post, widget, header, anywhere really. The ajaxize plugin adds a small javascript that will find this div, and convert it automatically into an ajax call for you!


There are a few limitations though:

• Functions must return valid HTML – this will be called in php and returned via the Ajax call
• Functions cannot accept any parameters (at least at the moment)
• Functions that work within a context (e.g. of a post, page, category), will most likely lose the context information

UPDATE: version 1.1 of the plugin now handles context much better. Ajaxize is now hooking in the right place, so the ajax call is made exactly where the div element is placed. This means plugins that use a post/category/taxonomy context information can now also be ajaxized. Special thanks to One Trick Pony for helping me figure out how to hook this correctly.


This was a perfect solution for mixing caching with dynamic content. I can convert almost any plugin or widget into a div. I can also write very simple PHP functions that will show dynamic content on my pages, with zero extra javascript code. The div itself can be cached, but the content will be pulled automatically by the browser when the page loads. I also found it useful for loading plugin buttons like Facebook like and Twitter tweet. Those can take a while to load and slow the page. When converted via ajaxize, they still take a while to load, but don’t seem to hold the page content from loading first.

What about security?

Some of you may have already started thinking… “but hang on a minute. If you can ajaxize one function, what stops somebody from calling other functions on my wordpress??!!”. Very true. This is why ajaxize was built-in with security in mind. It uses a very powerful algorithm called HMAC, with a secret key, so you can use ajax on any function you like, but only those functions and not others. This also means zero-configuration. The plugin only stores one value in the database – this is your secret key! I might cover the security aspects of the plugin on a future post. I encourage people to look through the code and validate the security I’ve implemented.

Feel free to try it out and let me know what you think!

no image specified
Query String :
TimThumb version : 1.28

The second article gave me the idea for a solution actually. It talks about optimizing timthumb with apache rewrite rules, such that once a thumbnail is generated, it can be served directly by apache, bypassing any php processing on subsequent access to thumbnails. Nice idea, and looks like a good solution too, but it had a couple of disadvantages for me:
1. It required modifying timthumb.php. It’s not a huge issue, but for future compatibility, allowing an easy update of timthumb to the next version etc, I didn’t like patching the code
2. It didn’t support CDN. (Well, it didn’t intend to do that I guess).

My solution in a way is a cocktail of both suggestions, which somehow produced an even simpler solution. The idea in principle was very simple: Change timthumb.php?src=… url to a “cdn-friendly” url. i.e. something without php, and without a real query string, Something like http://my.host.com/cdn-thumb/src=… – which will in turn also work on http://my.cdn.com/cdn-thumb/src=…

All it took was a fairly simple Apache rewrite rule. I don’t really like rewrite rules and however many times I try to figure it out, I always get confused and decide to abandon it. This time it was simple enough even for me to work it out.

<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule ^cdn-thumb/(.*)$ /relative/path/to/thumb.php?$1 [L]
</IfModule>

Place this inside any folder along the wordpress path. Then change from thumb.php?src=… to cdn-thumb/src= in your templates

You can add this rule to any folder within your wordpress installation. It can go in the root, or in your theme sub-path. The path to the ‘real’ php however is relative to the root of the site, not the OS path. For example:

If your root wordpress installation is in /home/joe/wordpress and thumb.php is in /home/joe/wordpress/wp-content/themes/xyz/thumb.php then use

RewriteRule ^cdn-thumb/(.*)$ /wp-content/themes/xyz/thumb.php?$1 [L]

* There was a similar solution on timthumb’s website. However it also required changes to timthumb.php…

I was recently playing around with fail2ban. It’s a really cool little tool that monitors your log files, matches certain patterns, and can act on it. Fail2ban would typically monitor your authentication log file, and if for example it spots 5 or more consecutive failures, it would simply add a filter to your iptables to block this IP address for a certain amount of time. I like fail2ban because it’s simple and effective. It does not try to be too sophisticated, or have too many features. It does one thing, and does it very well.

I was trying to build a custom-rule to watch a specific application log-file. I had a reasonably simple regular expression and I was able to test it successfully using fail2ban-regex. It matched the lines in the log file, and gave me a successful result

Success, the total number of match is 6

However, when running fail2ban, even though it loaded the configuration file correctly, and detected changes in the log files, fail2ban, erm, failed to ban… I couldn’t work out what was the problem.

As it turns-out, the timestamps on my log file was set to a different time-zone, so fail2ban treated those log entries as too old and did not take action. Make sure your timestamps are correct and on the same timezone as your system!! Once the timezone was set, fail2ban was working just fine.

Most people I know tend to simply use the same password on ALL websites. Email, Paypal, Amazon, Ebay, Facebook, Twitter. This is obviously a very bad idea.

Passwords are always a problem. Difficult to remember, hard to think of a good one when you need a new one, tricky to keep safe. For the moderately-paranoid and the sufficiently-techie there are many good solutions out there. Password managers. Online, offline, commercial, free. So I usually suggest to my friends and colleagues to use a password manager.

I personally like to use clipperz (online). I also used keepass (multi-platform). Both free and open source tools and do a good job.

However, I doubt many of my friends actually follow my advice. They’ll have to install something, or log on to somewhere JUST FOR THAT. It’s a little annoying to use and make every login complicated. It might not be available when they’re using the computer at work/friend’s house. So they end up doing the same thing and simply use one password.

So what’s the solution?? Well, lets refine the problem. The main concern for me is that IF I use the same password and it gets compromised. Even if it’s super-strong, ALL my ‘online assets’ almost immediately get compromised too. By the time I log in to change the password on ALL those websites, it’s probably too late. That’s assuming I know it was compromised on one of those websites. So I tend to trust the security of Amazon and Paypal (not that they are 100% immune to attacks and leaks), but what about this website I order coffee at (great coffee and a great website, and I do not imply that their security is not good, it probably is as good as their coffee), or that other website I ordered some computer parts at 2 years ago… The thing is, it only takes ONE. And then if someone grabs this password, the first thing they’re going to try is logging in to paypal, amazon, ebay etc.

It got me thinking. What if I carried on using the same (super-strong) password, but instead of using my usual email, used a different one for each website??! you must be thinking now “How can I use a different email for each website?? Sign up with a gazillion hotmail accounts??” No, there’s a simpler way, but lets leave it aside for now, I’ll show you how to in a moment. So what are the benefits? Even if someone grabs my (super-strong) password from this one website, the email address won’t work on any other website. And they won’t be able to guess my other email accounts Because each email address is different and hard to guess!!

How do I get those gazillion email addresses without signing gazillion times for an email account then? Do you have a gmail account? Hotmail? Yahoo?! No??!! Well, you should probably get one of those (although others may allow the same thing). All these webmail accounts allow you to create aliases. (here’s a quick overview for gmail, hotmail and yahoo). Essentially it’s another email address that is linked to your main email. So instead of john.smith@gmail.com you can use john.smith+f9230382@gmail.com. Not the most friendly address, but virtually impossible to guess. All you need to do now is sign-up for an account using this alias. Don’t forget to create a new alias for every online account you create though! And make sure the alias is hard to guess. Just stick a bunch of random characters and digits at the end. the longer the better (size DOES matter).

The only remaining question is therefore “how do I know which alias I used for <insert name of website>?”. My suggestion is relatively simple. Keep it inside your email account. Keep a draft email, send an email to yourself, add a task/note or whatever you can use inside your online webmail account. The list would look something like this:

facebook – john.smith+kjdi23982ndsa@gmail.com
ebay – john.smith+484jqcqwl2@gmail.com
amazon – john.smith+hgqozcmn21kf@gmail.com
…

It’s not super-secure, but:
1. Only if your email account gets hacked they would see this list (and they need to know to look for it too)
2. The list of aliases would NEVER include your super-secure password. This is something you still have to remember.
3. If you make sure you use a different (super-secure) password for your email account, then even if your regular (super-secure) password gets compromised, they won’t be able to get into your email account and get this list.

So you end up with two super-secure passwords you have to remember, and a (not super secure) list of email aliases inside your email account. That’s the passwordless password-manager.

NOTE:
* Be very careful of losing this list. Without it, you won’t be able to log in to your online accounts. So if you do plan on using this, make sure you have a few copies of this list elsewhere. A simple solution is every time you use an email alias, send yourself an email with a note about it. Try to use the same subject so you can find it later and send this email to more than one email if you have more than one. Then you can simply search through your email account(s). I suggest somthing like

From: john.smith@gmail.com
To: john.smith@gmail.com, john.smith@hotmail.com
Subject: new alias for <website name>

john.smith+fkfjdl93823@gmail.com