Monthly Archives: August 2010

How to tame Jive’s backup happy nature

I've been looking after a Jive box at work, and the other day, the site it provided went down round about the same we were moving offices (do sites ever go down at convenient times?). After checking the logs, it turned out that the culprit was Jive's sometimes handy, but greedy habit of creating a full dump of all the data, files added to a site every day, for the last 30 days.

When you're working with large uploads of Photoshop mockups this quickly burns through drive space, so I had to add a cronjob here to sweep away the older backups. There's a thread on Jive' forum's asking the same issue, so I thought I'd document it here:

First, open up your crontab as a user with the correct privileges:

  crontab -e

then add a line like this:

1
2
3
  # find all db dumps older than 7 days, and delete them
  # at minute 02, on hour 04, every day, every month, every year
  02 04 * * * find /usr/local/jive/var/data/backup/postgres/full -mtime +7 -exec rm -f {} \;

I've annotated the cron syntax cronwtf to help make it more readable, but the second snippet requires some explanation. This line here:

  find /usr/local/jive/var/data/backup/postgres/full -mtime +7 -exec rm -f {} \;

Roughly translates as:

run a 'find' all files older than 7 days, and execute a command to delete (rm -f) each file in the collection

This creates our 'sweeper' script that clears away database dump files that would otherwise bring down the machine, leading you to lose face, and ultimately your place in society.

Note -mtime refers to the last modified time of a file. You can also think of this as the creation time of the file, as Unix does not distinguish between creation and modification. There's more here in the manpages

There’s no need to type your password when you restart Apache, really…

When you're developing with PHP on a mac, if you're not using MAMP, you'll often end up having to do a lot of manual restarts when you make changes to how you've set up Apache (assuming you haven't joined all the cool kids and moved onto Nginx yet...). This usually involves calling up a terminal window and typing in the usual Apache restart command on OS X:

  sudo apachectl restart

This isn't a really destructive command, and having to type in your admin password every time when doing this in development on your own computer gets old quickly. It's also error prone. Surely there's a better way?

Fortunately, when browsing the Aegir OS X install documentation, I came across as handy fix to this problem. The Aegir hackers let Aegir handle server restarts in a fairly elegant fashion, by tweaking the sudoers file on your mac, which is basically a short list of who is allowed to do what on your machine. I've borrowed a few tricks, and adaprted them to use in my sudoers file here, and after showing it in full, I'll explain how it works.

Bear in mind, you can't edit the sudoers file directly - you need to use the visudo command, (this works as a precaution to stop this file getting screwed up by letting more than one person is edit it at a time for example).

Also, to make things more complex, you need to edit this inside the terminal, to you may need to force this by typing EDITOR='vim' first

Okay, now that's out the way, lets look at that file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
  # Run as alias specification
 
  # User privilege specification
  root    ALL=(ALL) ALL
  %admin  ALL=(ALL) ALL
 
  # Uncomment to allow people in group wheel to run all commands
  # %wheel        ALL=(ALL) ALL
 
  # Same thing without a password
  # %wheel        ALL=(ALL) NOPASSWD: ALL
  %staff          ALL=(ALL) NOPASSWD: /usr/sbin/apachectl
  # Samples
  # %users  ALL=/sbin/mount /cdrom,/sbin/umount /cdrom
  # %users  localhost=/sbin/shutdown -h now

Lets look at the first lines, with root and %wheel. If you're even bothering to read this, the chances are you know that root refers to the all powerful user that can do anything on a system, but you may not be familiar with the percent prefix on %admin nor the ALL=(ALL) ALL. The %admin basically means 'anyone in the admin group, but the ALL=(ALL) ALL is somewhat more cryptic. The rough translation goes like this though:

from ALL terminals, let these users run ALL commands and as ALL of the users in the system.

We see the same trick visible again with the %wheel group, but the line starting with %staff deserves more attention:

  %staff          ALL=(ALL) NOPASSWD: /usr/sbin/apachectl

Translated, this means:

for ALL members in the staff group, let them use ALL terminals, to run the command /usr/sbin/apachectl as ALL users (in particular, the root user) without needing a password (that's the NOPASSWD: bit).

This is the line that lets us run the familiar sudo apachectl restart without needing to constantly type our password credentials in.

Which over the course of a year, will easily save you tons of typing over the year, and leave some time to skim the sudoers man page, and suggest a similar trick here for others to try.

Over to you now...

How to get back into Drupal site if you’ve locked yourself out

A few posts back, I shared a one liner to get you back into a WordPress site if you manage to lock yourself out, and forget your database password.

Assuming you've access to the command line and drush, you can pull a similar trick with Drupal, by typing the following query in:

    drush sql-query "update users set pass=md5('NEWPASSWORD') where uid = 1;"

What's happening here?

The first thing we're doing is calling drush sql-query, a sub-command of drush.

If you haven't used Drush yet, you really, really should. It totally transforms how you work with Drupal, by making the kinds of tasks you had to do manually through the website possible from a commandline, which means yes, you can get up to all kinds of handy scripting shenanigans.

As you might expect drush sql-query lets you pass a single arbitrary query to the database described in your site's settings file, without you needing to fish the credentials out yourself. Here's our query too now:

 update users set pass=md5('NEWPASSWORD') where uid = 1;

In short, we're updating the users table in the Drupal database, by setting the pass_word value for the the first user id (_where uid=1), to a md5 hash of the phrase NEWPASSWORD.

If you don't haves access to drush, nor the command line, but you still can change a file over SFTP, you can to the same by adding a snippet like this on to the site:

  $doh_forgot_my_password = db_query("update users set pass=md5('NEWPASSWORD') where uid = 1;");

Of course you really should be paramaterizing this like so:

  $doh_forgot_my_password = db_query("update users set pass=md5('%s') where uid = %d;", array("NEWPASSWORD", "1") );

But this given the fact that this snippet should only existing in a template for 15-20 seconds at the most you'd probably be forgiven for taking the short cut...

Quick heads up on a super handy cheat sheet for Drupal

I'm working with Drupal a lot at work now, and in the process of creating my last post, I stumbled across this cheat sheet for developing with Drupal.

I had no idea could pull ddebug_backtrace() to get an instant stacktrace, or DARGS() to see the arguments being passing into a function at any point, or DD() to log directly to a text file (which you can tail in a terminal window to see stuff as it happens).

Now, if only Simpletest wasn't so depressingly slow...

How to setup Snow Leopard for LAMP development and debugging

Over this weekend, I've been looking at ways to make it easier for me to work with PHP, largely because I've been using it more and more at work, and I've felt spoilt by the tools available when using Ruby, like the [ruby debugger], or when playing with Django or Twisted, python's own interactive pdb debugger, so I decided to give MacGDBp a try.

It's pretty handy - it allows you to step your code when things break, giving you an idea of what exactly is happening, under the hood, or letting you understand the actual a request takes through your code.

I'm not using MAMP, because where possible, I'm trying to minimise duplication of software on the my computer, and previously I've confused myself something horrible with trying to keep track of multiple instances of MySQL and Apache before, so if you're not using MAMP, following these instructions should help get you set up with a decent debugging tool, on Snow Leopard, and along the way get a fairly maintainable lamp stack, if you're not enamoured with MAMP.

Get PHP 5.2

This isn't strictly necessary, but right now, I've found PHP 5.2 to be less hassle when developing than 5.3, and for time being, Drupal and WordPress to be standardising on it first before moving to the newer version, which means I am too.

The best instructions I've found to do this are here PHP 5.2 for compatibility with Drupal, and provide a good introduction to the amazing Homebrew, which you really should be using instead of Macports or Fink, if you're not already.

Lets get the hard commandline work out of the way first.

Setup PEAR to complement Homebrew's PHP 5.2

Just like how we have pip or easy_install with Python, CPAN for Perl, and Rubygems for Ruby, with PHP, we have PECL, and PEAR. PEAR is repository of PHP classes, like PHP Unit for unit testing, whereas PECL is a repository for C extensions like APC or Memcache, that help with performance, and caching, or link to other programs.

By default we do have PEAR and PECL installed, but in keeping with Homebrew's example of storing stuff in /usr/local/ to avoid needless sudo'ing we're going to use our own versions of PEAR, using this handy one liner. What's happening here is that we're using curl to fetch the data at http://pear.php.net/go-pear, and then streaming it into the php command:

    curl http://pear.php.net/go-pear | sudo php

We'll be presented with some text, once we've entered our admin password, along the lines of:

  Welcome to go-pear!
 
  Go-pear will install the 'pear' command and all the files needed by
  it.  This command is your tool for PEAR installation and maintenance.
 
  Go-pear also lets you download and install the following optional PEAR
  packages: PEAR_Frontend_Web-beta, PEAR_Frontend_Gtk2, MDB2.
 
  If you wish to abort, press Control-C now, or press Enter to continue: 
 
  HTTP proxy (http://user:password@proxy.myhost.com:port), or Enter for none::

We probably don't need a proxy, so you can just hit enter, at which point we'll be asked the following:

  Below is a suggested file layout for your new PEAR installation.  To
  change individual locations, type the number in front of the
  directory.  Type 'all' to change all of them or simply press Enter to
  accept these locations.
 
   1. Installation prefix ($prefix) : .
   2. Temporary files directory     : $prefix/temp
   3. Binaries directory            : $prefix/bin
   4. PHP code directory ($php_dir) : $prefix/PEAR
   5. Documentation base directory  : $php_dir/docs
   6. Data base directory           : $php_dir/data
   7. Tests base directory          : $php_dir/tests
 
  1-7, 'all' or Enter to continue:

We want to change the installation prefix, to make sure we're putting this stuff into /usr/local - I found that I had to do this explicitly, because the default value for $prefix, ended up with me getting build errors.

Once we've added this, we'll get a load of text flying past as PEAR is built, and end up with some text saying something like:

    WARNING!  The include_path defined in the currently used php.ini does not
    contain the PEAR PHP directory you just specified:
 
    If the specified directory is also not in the include_path used by
    your scripts, you will have problems getting any PEAR packages working.
 
    Would you like to alter php.ini ? [Y/n] :

Why yes, we would like this added. This will add this snippet to your php.ini file:

    ;***** Added by go-pear
    include_path=".:/usr/local/PEAR"
    ;*****

One important thing here - now that you have this, make sure your PEAR

Now, any thing you add to pear will be available in future, so installing PHP Unit for unit testing is as simple as calling this on the commandline (note):

  pear install phpunit/PHPUnit

In a similar fashion to Rubygems, these libraries of classes are available by calling require_once as if you were in the path /usr/local/PEAR, so to pull in PHP Unit, you just type:

  require_once 'PHPUnit/Framework.php';

Setup PECL with PHP 5.2

Now that we have PEAR setup, we should spend a it of time on making PECL simple to administer without sudo. All I needed to do here was check that everything in /usr/local/Cellar/php52/ belonged to my normal user account, by calling:

    sudo chown -R /usr/local/Cellar/php52/

This does away with the need to compile things as root, and makes installing extensions like Memcache, Mongodb, APC, as simple as:

    pecl install apc

If you find yourself having trouble here, you may want to check the permissions on any extra directories created during the PECL installation process - I had a couple of issues because some extensions had been installed as root earlier when following Hunter Ford's instructions at Cupcake with Sprinkles, which stopped my user account being able to install into the nested directory structure, because my account was trying to put files into directories owned by root.

Fetch Xdebug

If we didn't need xdebug, this would be all we needed, but sadly things aren't as simple as that. If we try calling something like pecl install xdebug, we DO get a version of debug installed, but it's not quite what we need, and doesn't seem to do anything useful. If we want this to work with MacGDBp, we need to install the version compiled by the chaps at Active state from their Remote Debugging page, that's designed to work with their own IDE, but also other tools. Once we've downloaded the tar file, we need to choose the correct version,

And then copy it to where the other extensions are:

    cp xdebug /usr/local/Cellar/php52/5.2.13/lib/php/extensions/xdebug.so

And make the relevant changes in our php.ini file for them, to point to the xdebug shared object (that's the .so suffix on extensions), and provide a few defaults as directed on the MacGDBp help page:

  ; Adding xdebug
  zend_extension=/usr/local/Cellar/php52/5.2.13/lib/php/extensions/xdebug.so
  xdebug.remote_enable=1
  xdebug.remote_autostart=1
  xdebug.remote_host=localhost
  xdebug.remote_port=9000

In short we're telling xdebug to switch on by default, and and listen on localhost:9000 for any clients that want to connect to it when we want to inspect what's happening with code.

Finally turning on MacGDBp

Still with me? Good. At long last, we can finally start looking through code in our debugger. Fire up MacGDBp, and when you run your next PHP script, you should get to see something like this pic pilfered from Particle Tree's own post on this subject:

There's far more to this, but in general the key to using the debugger is knowing how to set breakpoints, and remembering that your choices in the blue buttons are (from left to right)

  • stepping into functions to see what's happening as a request is passed from function to function
  • stepping out when you no longer need to see what's happening in a particular function
  • ...and stepping past a particular function, without needing to look into its working at all

The green button is a fast-forward button, to take you to the next breakpoint you might have in the code, and the green power button is analogous to a power button, to refresh a connection to the debugger. You really, really should look into this post here by Tim Sabat at Particle Tree, and this one here by Matt Butcher, at TechnoSophos.

Debugging without MacGDBp

Of course, you don't need to use MacGDBp all the time, In most cases, just having a stack trace will help find the source of the error. Once you've got xdebug setup, you should get a handy track trace whenever you throw an error - if not, be sure to make sure you have the display errors setting set to 'on' in your php.ini setting (it should be around line 370 in your file, normally):

    display_errors = On

By the time you've followed these steps, you should have an easier to main installation of the LAMP stack, with a simple way to install extra PHP libraries and C extensions, a couple much more effective ways to debug than simply typing echo $variable everywhere, and an ideal environment to get on with hacking on Drupal, WordPress and any other PHP code in future.

Phew!

This guide is largely a synthesis of earlier blog posts about this subject by Justin Hileman, Matt Butcher, Tim Sabat, Hunter Ford, and Boris Gordon, I'd recommend subscribing to their blogs, as they definitely know more about PHP than I do, and they're a great source of handy info.

If there's anything that isn't clear, please let me know, so I can improve this guide - I don't want anyone else to have to blunder through working out all this themselves in future, as working all this out has taken, far, far longer than I'd like, and well, it seems churlish not to share now that I've got a setup that seems fairly stable.

Now, time to actually do something useful with it...

Switching iPhone contracts

I came across this video when trawling across web, researching which provider I wanted to leave O2's woeful reception for with my iphone contract, and while sadly it's not far from the current experience of using my iPhone now that I've upgraded to the new iPhone OS, it is funny enough to raise a smile whenever my phone locks up when I do something really computationally expensive on the phone, like open up iTunes:

It does feel at first like an extremely cynical move from Apple, degrading the user experience of an existing handset this dramatically around the same time as launching a another shiny new expensive handset with four times the memory - but truth be told this is nothing new with Apple - if you own a mac, you'll be aware of their 6 month product cycle already, so seeing this happen with the iPhone is hardly earth shattering. Lame yes, but not shocking. The main saving grace is that there is still a downgrade path back to iOS 3 if Apple can't release a new update in the coming few weeks that sorts out the performance issues.

In the meantime, there's a few steps here you can take to minimise the ridiculous slowness, listed on lifehacker - there's no guarantee they'll work, but there seems to be at least some kind of placebo effect in place, since I switched off spotlight indexing when I tried a few of them just now.

If the slowness of the new OS hasn't put you off of staying with your existing iPhone

As I mentioned before, I took out a contract with 02 a little less than than two years ago now, and if like me you're looking for a sim only tariff, Orange and Vodafone both offer extremely similar deals, though it looks like Vodafone just about edge out Orange. Both offer tethering at a surprisingly reasonable rates.

You can pay a fiver for half a gig of tethered connectivity if need be, and 15 quid for 3Gb of data, which is the same as what they charge for internet access using their 3G dongles normally, but Vodafone's data cap is slightly higher at 1Gb per month instead of 750mb, and their network is generally seen as the most reliable in the UK .

If you want to lock yourself into a twelve month contract, you can get the sim only deal from Vodafone for a fiver less too, at £20 per month, which is about as reasonable as we're gonna see iPhone sim-only pricing get, while still providing some decent coverage.

Goodbye O2 - it's been a fun 4 or so years, but your network is just too flaky to see myself using in the near future. Lets talk again when you have reception again near Shad Thames.

_Update - I want with Vodafone in the end, and bought it through their website. I don't know why I waited so long until I did this to be honest...