How to tame Jive’s backup happy nature

I've been looking after a Jive box at work, and the other day, the site it provided went down round about the same we were moving offices (do sites ever go down at convenient times?). After checking the logs, it turned out that the culprit was Jive's sometimes handy, but greedy habit of creating a full dump of all the data, files added to a site every day, for the last 30 days.

When you're working with large uploads of Photoshop mockups this quickly burns through drive space, so I had to add a cronjob here to sweep away the older backups. There's a thread on Jive' forum's asking the same issue, so I thought I'd document it here:

First, open up your crontab as a user with the correct privileges:

  crontab -e

then add a line like this:

1
2
3
  # find all db dumps older than 7 days, and delete them
  # at minute 02, on hour 04, every day, every month, every year
  02 04 * * * find /usr/local/jive/var/data/backup/postgres/full -mtime +7 -exec rm -f {} \;

I've annotated the cron syntax cronwtf to help make it more readable, but the second snippet requires some explanation. This line here:

  find /usr/local/jive/var/data/backup/postgres/full -mtime +7 -exec rm -f {} \;

Roughly translates as:

run a 'find' all files older than 7 days, and execute a command to delete (rm -f) each file in the collection

This creates our 'sweeper' script that clears away database dump files that would otherwise bring down the machine, leading you to lose face, and ultimately your place in society.

Note -mtime refers to the last modified time of a file. You can also think of this as the creation time of the file, as Unix does not distinguish between creation and modification. There's more here in the manpages

  • Share/Bookmark
Posted in sysadmin | Tagged , , , , | Leave a comment

There’s no need to type your password when you restart Apache, really…

When you're developing with PHP on a mac, if you're not using MAMP, you'll often end up having to do a lot of manual restarts when you make changes to how you've set up Apache (assuming you haven't joined all the cool kids and moved onto Nginx yet...). This usually involves calling up a terminal window and typing in the usual Apache restart command on OS X:

  sudo apachectl restart

This isn't a really destructive command, and having to type in your admin password every time when doing this in development on your own computer gets old quickly. It's also error prone. Surely there's a better way?

Fortunately, when browsing the Aegir OS X install documentation, I came across as handy fix to this problem. The Aegir hackers let Aegir handle server restarts in a fairly elegant fashion, by tweaking the sudoers file on your mac, which is basically a short list of who is allowed to do what on your machine. I've borrowed a few tricks, and adaprted them to use in my sudoers file here, and after showing it in full, I'll explain how it works.

Bear in mind, you can't edit the sudoers file directly - you need to use the visudo command, (this works as a precaution to stop this file getting screwed up by letting more than one person is edit it at a time for example).

Also, to make things more complex, you need to edit this inside the terminal, to you may need to force this by typing EDITOR='vim' first

Okay, now that's out the way, lets look at that file:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
  # Run as alias specification
 
  # User privilege specification
  root    ALL=(ALL) ALL
  %admin  ALL=(ALL) ALL
 
  # Uncomment to allow people in group wheel to run all commands
  # %wheel        ALL=(ALL) ALL
 
  # Same thing without a password
  # %wheel        ALL=(ALL) NOPASSWD: ALL
  %staff          ALL=(ALL) NOPASSWD: /usr/sbin/apachectl
  # Samples
  # %users  ALL=/sbin/mount /cdrom,/sbin/umount /cdrom
  # %users  localhost=/sbin/shutdown -h now

Lets look at the first lines, with root and %wheel. If you're even bothering to read this, the chances are you know that root refers to the all powerful user that can do anything on a system, but you may not be familiar with the percent prefix on %admin nor the ALL=(ALL) ALL. The %admin basically means 'anyone in the admin group, but the ALL=(ALL) ALL is somewhat more cryptic. The rough translation goes like this though:

from ALL terminals, let these users run ALL commands and as ALL of the users in the system.

We see the same trick visible again with the %wheel group, but the line starting with %staff deserves more attention:

  %staff          ALL=(ALL) NOPASSWD: /usr/sbin/apachectl

Translated, this means:

for ALL members in the staff group, let them use ALL terminals, to run the command /usr/sbin/apachectl as ALL users (in particular, the root user) without needing a password (that's the NOPASSWD: bit).

This is the line that lets us run the familiar sudo apachectl restart without needing to constantly type our password credentials in.

Which over the course of a year, will easily save you tons of typing over the year, and leave some time to skim the sudoers man page, and suggest a similar trick here for others to try.

Over to you now...

  • Share/Bookmark
Posted in Coding, sysadmin | Tagged , | Leave a comment

How to get back into Drupal site if you’ve locked yourself out

A few posts back, I shared a one liner to get you back into a WordPress site if you manage to lock yourself out, and forget your database password.

Assuming you've access to the command line and drush, you can pull a similar trick with Drupal, by typing the following query in:

    drush sql-query "update users set pass=md5('NEWPASSWORD') where uid = 1;"

What's happening here?

The first thing we're doing is calling drush sql-query, a sub-command of drush.

If you haven't used Drush yet, you really, really should. It totally transforms how you work with Drupal, by making the kinds of tasks you had to do manually through the website possible from a commandline, which means yes, you can get up to all kinds of handy scripting shenanigans.

As you might expect drush sql-query lets you pass a single arbitrary query to the database described in your site's settings file, without you needing to fish the credentials out yourself. Here's our query too now:

 update users set pass=md5('NEWPASSWORD') where uid = 1;

In short, we're updating the users table in the Drupal database, by setting the pass_word value for the the first user id (_where uid=1), to a md5 hash of the phrase NEWPASSWORD.

If you don't haves access to drush, nor the command line, but you still can change a file over SFTP, you can to the same by adding a snippet like this on to the site:

  $doh_forgot_my_password = db_query("update users set pass=md5('NEWPASSWORD') where uid = 1;");

Of course you really should be paramaterizing this like so:

  $doh_forgot_my_password = db_query("update users set pass=md5('%s') where uid = %d;", array("NEWPASSWORD", "1") );

But this given the fact that this snippet should only existing in a template for 15-20 seconds at the most you'd probably be forgiven for taking the short cut...

  • Share/Bookmark
Posted in Coding | Tagged , | Leave a comment

Quick heads up on a super handy cheat sheet for Drupal

I'm working with Drupal a lot at work now, and in the process of creating my last post, I stumbled across this cheat sheet for developing with Drupal.

I had no idea could pull ddebug_backtrace() to get an instant stacktrace, or DARGS() to see the arguments being passing into a function at any point, or DD() to log directly to a text file (which you can tail in a terminal window to see stuff as it happens).

Now, if only Simpletest wasn't so depressingly slow...

  • Share/Bookmark
Posted in Coding, Journal | Tagged , , | Leave a comment

How to setup Snow Leopard for LAMP development and debugging

Over this weekend, I've been looking at ways to make it easier for me to work with PHP, largely because I've been using it more and more at work, and I've felt spoilt by the tools available when using Ruby, like the [ruby debugger], or when playing with Django or Twisted, python's own interactive pdb debugger, so I decided to give MacGDBp a try.

It's pretty handy - it allows you to step your code when things break, giving you an idea of what exactly is happening, under the hood, or letting you understand the actual a request takes through your code.

I'm not using MAMP, because where possible, I'm trying to minimise duplication of software on the my computer, and previously I've confused myself something horrible with trying to keep track of multiple instances of MySQL and Apache before, so if you're not using MAMP, following these instructions should help get you set up with a decent debugging tool, on Snow Leopard, and along the way get a fairly maintainable lamp stack, if you're not enamoured with MAMP.

Get PHP 5.2

This isn't strictly necessary, but right now, I've found PHP 5.2 to be less hassle when developing than 5.3, and for time being, Drupal and WordPress to be standardising on it first before moving to the newer version, which means I am too.

The best instructions I've found to do this are here PHP 5.2 for compatibility with Drupal, and provide a good introduction to the amazing Homebrew, which you really should be using instead of Macports or Fink, if you're not already.

Lets get the hard commandline work out of the way first.

Setup PEAR to complement Homebrew's PHP 5.2

Just like how we have pip or easy_install with Python, CPAN for Perl, and Rubygems for Ruby, with PHP, we have PECL, and PEAR. PEAR is repository of PHP classes, like PHP Unit for unit testing, whereas PECL is a repository for C extensions like APC or Memcache, that help with performance, and caching, or link to other programs.

By default we do have PEAR and PECL installed, but in keeping with Homebrew's example of storing stuff in /usr/local/ to avoid needless sudo'ing we're going to use our own versions of PEAR, using this handy one liner. What's happening here is that we're using curl to fetch the data at http://pear.php.net/go-pear, and then streaming it into the php command:

    curl http://pear.php.net/go-pear | sudo php

We'll be presented with some text, once we've entered our admin password, along the lines of:

  Welcome to go-pear!
 
  Go-pear will install the 'pear' command and all the files needed by
  it.  This command is your tool for PEAR installation and maintenance.
 
  Go-pear also lets you download and install the following optional PEAR
  packages: PEAR_Frontend_Web-beta, PEAR_Frontend_Gtk2, MDB2.
 
  If you wish to abort, press Control-C now, or press Enter to continue: 
 
  HTTP proxy (http://user:password@proxy.myhost.com:port), or Enter for none::

We probably don't need a proxy, so you can just hit enter, at which point we'll be asked the following:

  Below is a suggested file layout for your new PEAR installation.  To
  change individual locations, type the number in front of the
  directory.  Type 'all' to change all of them or simply press Enter to
  accept these locations.
 
   1. Installation prefix ($prefix) : .
   2. Temporary files directory     : $prefix/temp
   3. Binaries directory            : $prefix/bin
   4. PHP code directory ($php_dir) : $prefix/PEAR
   5. Documentation base directory  : $php_dir/docs
   6. Data base directory           : $php_dir/data
   7. Tests base directory          : $php_dir/tests
 
  1-7, 'all' or Enter to continue:

We want to change the installation prefix, to make sure we're putting this stuff into /usr/local - I found that I had to do this explicitly, because the default value for $prefix, ended up with me getting build errors.

Once we've added this, we'll get a load of text flying past as PEAR is built, and end up with some text saying something like:

    WARNING!  The include_path defined in the currently used php.ini does not
    contain the PEAR PHP directory you just specified:
 
    If the specified directory is also not in the include_path used by
    your scripts, you will have problems getting any PEAR packages working.
 
    Would you like to alter php.ini ? [Y/n] :

Why yes, we would like this added. This will add this snippet to your php.ini file:

    ;***** Added by go-pear
    include_path=".:/usr/local/PEAR"
    ;*****

One important thing here - now that you have this, make sure your PEAR

Now, any thing you add to pear will be available in future, so installing PHP Unit for unit testing is as simple as calling this on the commandline (note):

  pear install phpunit/PHPUnit

In a similar fashion to Rubygems, these libraries of classes are available by calling require_once as if you were in the path /usr/local/PEAR, so to pull in PHP Unit, you just type:

  require_once 'PHPUnit/Framework.php';

Setup PECL with PHP 5.2

Now that we have PEAR setup, we should spend a it of time on making PECL simple to administer without sudo. All I needed to do here was check that everything in /usr/local/Cellar/php52/ belonged to my normal user account, by calling:

    sudo chown -R /usr/local/Cellar/php52/

This does away with the need to compile things as root, and makes installing extensions like Memcache, Mongodb, APC, as simple as:

    pecl install apc

If you find yourself having trouble here, you may want to check the permissions on any extra directories created during the PECL installation process - I had a couple of issues because some extensions had been installed as root earlier when following Hunter Ford's instructions at Cupcake with Sprinkles, which stopped my user account being able to install into the nested directory structure, because my account was trying to put files into directories owned by root.

Fetch Xdebug

If we didn't need xdebug, this would be all we needed, but sadly things aren't as simple as that. If we try calling something like pecl install xdebug, we DO get a version of debug installed, but it's not quite what we need, and doesn't seem to do anything useful. If we want this to work with MacGDBp, we need to install the version compiled by the chaps at Active state from their Remote Debugging page, that's designed to work with their own IDE, but also other tools. Once we've downloaded the tar file, we need to choose the correct version,

And then copy it to where the other extensions are:

    cp xdebug /usr/local/Cellar/php52/5.2.13/lib/php/extensions/xdebug.so

And make the relevant changes in our php.ini file for them, to point to the xdebug shared object (that's the .so suffix on extensions), and provide a few defaults as directed on the MacGDBp help page:

  ; Adding xdebug
  zend_extension=/usr/local/Cellar/php52/5.2.13/lib/php/extensions/xdebug.so
  xdebug.remote_enable=1
  xdebug.remote_autostart=1
  xdebug.remote_host=localhost
  xdebug.remote_port=9000

In short we're telling xdebug to switch on by default, and and listen on localhost:9000 for any clients that want to connect to it when we want to inspect what's happening with code.

Finally turning on MacGDBp

Still with me? Good. At long last, we can finally start looking through code in our debugger. Fire up MacGDBp, and when you run your next PHP script, you should get to see something like this pic pilfered from Particle Tree's own post on this subject:

There's far more to this, but in general the key to using the debugger is knowing how to set breakpoints, and remembering that your choices in the blue buttons are (from left to right)

  • stepping into functions to see what's happening as a request is passed from function to function
  • stepping out when you no longer need to see what's happening in a particular function
  • ...and stepping past a particular function, without needing to look into its working at all

The green button is a fast-forward button, to take you to the next breakpoint you might have in the code, and the green power button is analogous to a power button, to refresh a connection to the debugger. You really, really should look into this post here by Tim Sabat at Particle Tree, and this one here by Matt Butcher, at TechnoSophos.

Debugging without MacGDBp

Of course, you don't need to use MacGDBp all the time, In most cases, just having a stack trace will help find the source of the error. Once you've got xdebug setup, you should get a handy track trace whenever you throw an error - if not, be sure to make sure you have the display errors setting set to 'on' in your php.ini setting (it should be around line 370 in your file, normally):

    display_errors = On

By the time you've followed these steps, you should have an easier to main installation of the LAMP stack, with a simple way to install extra PHP libraries and C extensions, a couple much more effective ways to debug than simply typing echo $variable everywhere, and an ideal environment to get on with hacking on Drupal, WordPress and any other PHP code in future.

Phew!

This guide is largely a synthesis of earlier blog posts about this subject by Justin Hileman, Matt Butcher, Tim Sabat, Hunter Ford, and Boris Gordon, I'd recommend subscribing to their blogs, as they definitely know more about PHP than I do, and they're a great source of handy info.

If there's anything that isn't clear, please let me know, so I can improve this guide - I don't want anyone else to have to blunder through working out all this themselves in future, as working all this out has taken, far, far longer than I'd like, and well, it seems churlish not to share now that I've got a setup that seems fairly stable.

Now, time to actually do something useful with it...

  • Share/Bookmark
Posted in Coding, Journal | Tagged | Leave a comment

Switching iPhone contracts

I came across this video when trawling across web, researching which provider I wanted to leave O2's woeful reception for with my iphone contract, and while sadly it's not far from the current experience of using my iPhone now that I've upgraded to the new iPhone OS, it is funny enough to raise a smile whenever my phone locks up when I do something really computationally expensive on the phone, like open up iTunes:

It does feel at first like an extremely cynical move from Apple, degrading the user experience of an existing handset this dramatically around the same time as launching a another shiny new expensive handset with four times the memory - but truth be told this is nothing new with Apple - if you own a mac, you'll be aware of their 6 month product cycle already, so seeing this happen with the iPhone is hardly earth shattering. Lame yes, but not shocking. The main saving grace is that there is still a downgrade path back to iOS 3 if Apple can't release a new update in the coming few weeks that sorts out the performance issues.

In the meantime, there's a few steps here you can take to minimise the ridiculous slowness, listed on lifehacker - there's no guarantee they'll work, but there seems to be at least some kind of placebo effect in place, since I switched off spotlight indexing when I tried a few of them just now.

If the slowness of the new OS hasn't put you off of staying with your existing iPhone

As I mentioned before, I took out a contract with 02 a little less than than two years ago now, and if like me you're looking for a sim only tariff, Orange and Vodafone both offer extremely similar deals, though it looks like Vodafone just about edge out Orange. Both offer tethering at a surprisingly reasonable rates.

You can pay a fiver for half a gig of tethered connectivity if need be, and 15 quid for 3Gb of data, which is the same as what they charge for internet access using their 3G dongles normally, but Vodafone's data cap is slightly higher at 1Gb per month instead of 750mb, and their network is generally seen as the most reliable in the UK .

If you want to lock yourself into a twelve month contract, you can get the sim only deal from Vodafone for a fiver less too, at £20 per month, which is about as reasonable as we're gonna see iPhone sim-only pricing get, while still providing some decent coverage.

Goodbye O2 - it's been a fun 4 or so years, but your network is just too flaky to see myself using in the near future. Lets talk again when you have reception again near Shad Thames.

_Update - I want with Vodafone in the end, and bought it through their website. I don't know why I waited so long until I did this to be honest...

  • Share/Bookmark
Posted in Journal | 2 Comments

Looking for the best way to keep data on servers safe

We know we should all be doing it, but most of us don't do it enough.

I put out a request today to my followers Twitter asking this question:

I've had the following services recommended to me by a number of fellow developers whose opinions I have a lot of respect for:

A couple of ex-Headshifters pointed me to Duplicity, a free tool that also looks very promising. Using it looks pretty straight forward.

    duplicity /home/my_directory scp://backup@other.host//usr/backup

A variant of it can be used to backup to Amazon S3 too, and there's a helpful blog post to show how to do this here, if you use Ubuntu or OSX.

Kalv recommended Safe, a clever Ruby gem by the Astrails crew that was originally designed to automate the backing up of Rails projects in a simple fashion.

One service that looks really interesting is though is Tarsnap, as recommended by [Jon Gilbrath][]. It's similarly simple to use, but takes care of the some of the more awkwards backup issues, and saves you having to setup your own S3 account.

  # Create an archive named "mybackup" containing /usr/home and /other/stuff:
  tarsnap -c -f mybackup /usr/home /other/stuff

The developer, Colin Percival also has written quite extensively about how it works on his own blog - he's only making a few cents per gigabyte on providing this service for people, yet it still looks to be something viable for him to run - amazing.

It looks like it's almost exactly what I'm after - the only thing missing is the option to back up storage inside the EU. This requirement is mainly one coming from data protection concerns from previous clients, because rules for processing data and storing in in the EU are different to that in the US, but given the strength of encryption, I'm not sure how much of an issue this really is, these days.

I'd really appreciate some light shed on this actually - technology is moving so much faster than law these days, it's frustrating not being able to take advantage of these kinds of services.

Anyway that's what I've found. What do you use, and why?

  • Share/Bookmark
Posted in Coding, sysadmin | Tagged , , | 1 Comment

Why it’s worth volunteering for medical science

Back in 2007, I volunteered to give a blood sample at Guys Hospital, as part of a research project called Indices of Renal Tolerance , which investigated how some kidney transplant patients recovered after operations.

Normally after a transplant operation, our bodies' immune systems would attack any new organ placed inside a body because they see it as a foreign body that must be destroyed before it can do any further damage to us, which is precisely what we don't want to happen when we've replaced a diseased malfunctioning one, with a new healthy one.

To avoid this, patients need to take immunosuppressant drugs, that, as you'd imagine form their name, inhibit these defence systems, allowing the new kidney to work without being rejected by the body.

The problem with immunosuppressants

Sadly these aren't without side effects, and as a rule, you really want to have your immune system at full strength to avoid infections or general illnesses. So transplant patients often end up spending the rest of their natural lives in a somewhat debilitated state as a result of these drugs.

Some people however, have immune systems can develop a tolerance to the newly transplanted organs, meaning they'd be free to live lives free of the side effects from the immunosuppressants, if it weren't for the fact that coming off medication runs carries the risk of killing them if it turns out that their bodies weren't as tolerant as they thought after all.

Coming off the drugs

However, one of the outcomes the research project I was in was the discovery of a chemical signature in patients who may carry this tolerant trait. As we get better at detecting this trait, more people may be able to live lives either free of these drugs, or with much lower dependencies on them.

Why am I writing any this?

To be honest, I had pretty much forgotten about being involved in this project, and the only memories I have are of almost passing out like a total pansy when giving the blood sample ( I needed to drink loads of sweet tea , and sit with my feet up until I felt well enough to stand...), but this month, I received a letter telling me that the study had been published in the Journal of Clinical Investigation, with a few snippets of the press coverage.

And, well... I do feel like a bit of a douche writing about this here, because it I can easily sound like that guy who reminds you about how much they give to charity they are, or how much they like to recycle and so on, and how wonderful a person they are.

But the pleasure and the feeling of pride from the letter far outweighs any potential embarrassment I might feel, especially if it helps persuade anyone else to volunteer for any similar medical research projects.

If you have the time, and you're a healthy person, I'd really recommend volunteering like this - it's incredibly fulfilling, and even including my comical fainting episode, we're talking a total time outlay of maybe two hours of my time, to help someone else live the rest of their life free from the side effects of drugs that they might not even need to to be taking.

So, in short - yay science!

  • Share/Bookmark
Posted in Journal | Tagged , | 1 Comment

The quickest way into WordPress when you’re locked out.

I love WordPress as a platform for building sites.

But no matter how many times I see it, I don't think I'll ever get used to having this snippet available to me if I get locked out of a WordPress install :

  <?php wp_set_password('new_admin_password',1); 
// this changes the password for user with an id of 1 (i.e. the admin)  ?>

Surely it shouldn't be that easy to get root on a WordPress site?

Ladies and gentlemen, this is why you should always sequester access on a database, so each app has it's own db user. If a charmless bad guy DOES gain access, they can only reck one WordPress site database, not all of them.

  • Share/Bookmark
Posted in Coding, meta, security, sysadmin, wordpress | Tagged , | Leave a comment

Nerdy Vim tip ahoy! How to save a file as the root user from inside Vim

At work, I'm always managing files while working on a remote server which invariably means I'm stuck using Vim or Vi, and finding out after I've made the my necessary changes to crucial config file, I don't have the rights to actually save them - so I can see this tip by catonmat saving lots of time in future. Thank you internets!

If you're using Vim, and you find that you need to save a file as root, the normal commands like :wq won't work. Instead you'll need to call this command:

  :w !sudo tee %

Okay that's nice. What is it actually doing?

The above command is basically saying call the write command, passing in sudo tee % as the argument. Now sudo we should be fairly comfortable with by now, leaving us just puzzling over what tee and % actually mean. Thankfully, it's not too complex.

The tee command takes the the standard input (which in this case is the contents of the file we're editing), and writes that into the filename referenced by % which is a vim shorthard for the current filename.

Again, in English please

Sadly for these purposes, my first language for thinking in still English rather than Bash or Vim, so the only way I think I'll ever retain this particular gem is to think of this in the following terms:

From within vim, take the contents of the file I'm editing, and as sudo, write it into the file I'm currently in, doing so as root.

One day this will all be second nature to me when I'm a real programmer, but til then, I'll stick with Textmate - it's massively powerful, but when you come against these strange cases, it defers to OS X's much slicker handling of permissions issues.

  • Share/Bookmark
Posted in Coding, sysadmin | Tagged | 2 Comments