– Michael McIntyre

So much has happened, far more than I could or should ever write about.

Recently we’ve been migrating MySQL servers from one hardware to another – for anyone (me!) that finds this useful in the future, firstly “Hello, and I hope the future is better than the current ‘past’!”, and secondly “I hope this does what you need”:

#!/bin/sh
HOST="$1"
USER="$2"
PASS="$3"
mysql -h "$HOST" -u "$USER" -p"$PASS" -e \
"SELECT CONCAT(\"SHOW GRANTS FOR '\", \
user,\"'@'\",host,\"'\;\") FROM mysql.user;" -B -N | \
mysql -h "$HOST" -u "$USER" -p"$PASS" | egrep -v "^Grants"

That small bash script accepts 3 arguments, HOSTNAME, USERNAME and PASSWORD. Executing it will output “CREATE USER” statements for all users on the MySQL instance.

While migrating the MySQL servers, we took the opportunity to setup a new monitoring solution – Zabbix. I’ve used it a couple of times in the past, and in my most humble opinion it’s failing is it’s flexibility. It’s just so flexible!

We have a fair number of Cisco switches and routers at our disposal here in the office, and setting up Zabbix templates for each is not high on my bucket list. Why do something by hand that can be done automagically by a computer?

Thus this (very hacky, PHP) script which automatically generates Cisco (SNMP) templates for Zabbix.

Simply run “show snmp mib ifmib ifindex” on your Cisco device of choice, and shove the output into the script’s STDIN. Hey-presto, out comes a Zabbix template!

Modifying the default options in the template is very easy; just edit what you like after line 34.

It’s been for several weeks that I’ve been trying (and failing) to aquire Domain Admin status on our Samba domain.

With Windows 2000 and XP, administrative access was easy.  A local admin account called “administrator”, and a local policy stating who’s an administrator – all peachy.

You can tell I’m not a Windows administrator by job!  Obviously domain admins is a far cleaner way to go.

So I dip my toe into Windows 7 on a Samba 3 domain.

It works well.  Too well.  There’s a small issue with the workstations claiming “there’s no logon servers available to handle the request” for 60+ seconds from a cold-start, but I think that’s solvable (I suspect WINS, NetBIOS or DNS is failing to warm up in a timely manner).  The domain admins however eluded me.

“So it’s OK” I think to myself.  “Login as the local administrator – set some local policies up.  …  Oh wait, Windows 7 disabled those because I joined the domain and didn’t first give them passwords.  Sheesh!  Joining the domain admin group it is then.”

Well I searched the Internet and ripped my hair out.  Week after week the issue prevailed.  Windows 7 wouldn’t obey the “Domain Admin”.  “Elevated access required.”

Finally I trip across this: http://fixunix.com/smb/64004-domain-admin-group-samba-3-a.html

The second guy motions that “Domain Admins” must be group ID 512 (in the Windows mapping) – and blam.

One Samba restart later … all works.

Check the documentation out at http://www.samba.org/samba/docs/man/manpages-3/net.8.html … specifically you’re looking for something like this:

net groupmap add rid=512 unixgroup=MYUNIXGROUPHERE type=domain

I would like to say a huge thank-you to the guys at PSN, there’s no question over how hard you’ve been working to get things this far. Despite how many will disagree with me, I’d like to thank the non-tech guys at Sony too, for your honesty and speed with dealing with things.

I work at a PCI-DSS regulated company myself, as a programmer, and don’t have 1% of the customers PSN does. We also don’t provide a service that millions of people rely on “real-time” for entertainment purposes, nor which they expect to find working 24/7.

You guys have done a good job, please keep it up.

This could have happened to anyone, although it’s always easy to point the finger when you’re the ignorant customer.

Don’t they just look great

About a year ago I looked into the possibility of using SMTP to validate email addresses.  As it turns out, it’s not perfect (a lot of false-positives) but it’s very reliable (I’ve failed to get any false-negatives).

The following code outlines how this works (written in PHP, because I was thinking about it in the context of web-apps):

<?php
function validate_email_regex ($email) {
   return preg_match ('/\b[A-Z0-9._%+-]+@[A-Z0-9.-]+\.[A-Z]{2,4}\b/i', $email);
}

function validate_email_smtp ($email) {
   if (!validate_email_regex ($email)) return false;
   list ($user, $domain) = split ('@', $email);

   if (!getmxrr ($domain, $mxhosts)) return false;
   $i = 0;
   $fp = false;
   $thishost = isset ($_SERVER['SERVER_NAME']) ? $_SERVER['SERVER_NAME'] : exec ("hostname");
   foreach ($mxhosts as $mxhost) {
      if (++$i > 3) break;    // don't check more than 3 MX records
      if ($fp) fclose ($fp);  // cleanup from last iteration

      $fp = fsockopen ("tcp://$mxhost", 25, $errno, $errstr, 10);
      if (!$fp) continue;
      if (!$str = fgets ($fp)) continue;
      if (substr ($str, 0, 3) != '220') continue;

      fwrite ($fp, "HELO $thishost\r\n");
      $str = fgets ($fp);
      if (substr ($str, 0, 3) != '250') continue;

      fwrite ($fp, "MAIL FROM:<mail@$thishost>\r\n");
      $str = fgets ($fp);
      if (substr ($str, 0, 3) != '250') continue;

      fwrite ($fp, "RCPT TO:<$email>\r\n");
      $str = fgets ($fp);
      if (substr ($str, 0, 3) != '250') {
         fclose ($fp);
         return false;  /* don't proceed with other MX records because of a bad user */
      }

      fclose ($fp);
      break;
   }
   return true;
}
?>

Hmmmm.  Thank goodness for SVN hooks – a wonderfully understated feature of subversion!

Go to your subversion directory (that created by “svnadmin create”), and enter the “hooks” directory.

Create a file, executable and owned by your webserver, called “post-commit”:

touch post-commit
sudo chown www-data:www-data post_commit     # RH and other distros will want apache:apache here
chmod +x post_commit

Enter the following as it’s content (this is a fairly self explanatory shell script):

#!/bin/sh
set -e
REPOS="$1"
REV="$2"
TO="email@address.com"
FROM="email@address.com"
BY=`svnlook author $REPOS -r$REV`
MSG=`svnlook log $REPOS -r$REV`
CHANGED=`svnlook changed $REPOS -r$REV`
DIFF=`svnlook diff $REPOS -r$REV | pygmentize -l diff -f html`
STYLE="borland"
CSS=`pygmentize -S $STYLE -f html`
echo "To: $TO\nFrom: $FROM\nSubject: SVN commit r$REV - ($BY) \"$MSG\"\nContent-Type: text/html; charset=us-ascii\n\n<html>\n<style type='text/css'>\n$CSS</style>\n<body>\n<p><a href='https://your.trac.url.com/trac/changeset/$REV/'>View changeset r$REV on Trac</a></p>\n<h3>Files changed:</h3><pre>$CHANGED</pre>\n<h3>Changes made:</h3><p>$DIFF</p>\n</body>\n</html>" | /usr/sbin/sendmail -t

You’ll need to apt-get or yum install pygmentizer (package named python-pygments on Debian/Ubuntu) to get the cool diff colouring.

And that’s it!  (The “sendmail -t” reference works with Exim on Debian Squeeze, your mileage as always, may vary.)

The main problem I’ve had with it is the portal pages not looking so good on mobile devices (particularly iPhone and Androids).

A little bit of Googling revealed a solution (at least, a solution for websites in general): insert the following meta tag into the HTML header.

<meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;" />

In the CoovaAP configuration interface, you can apply this for every portal page by adding it to the “HotSpot”/”Portal”/”HTML Title”.

Those who’re into tinkering with things they shouldn’t will notice that Ebuyer’s search pages include a “limit” GET variable in the results page URL. The upper limit appears to be 30, but going negatively causes a different kettle of fish:
http://www.ebuyer.com/search?q=a%25&x=0&y=0&limit=-3000
(That URL is not for the feint hearted!)

It makes one wonder, is there a little bit of sanitization fail?

I’ve made a lot of progress with the Cloud-Bible project (the name is growing on me, slowly).

Switching from the simpler-to-follow (but very assuming) big “switch” clause which was previously handling reverse-engineered tags (oh, I didn’t mention the XML I have is not precise OSIS format did I? ) to the more complex (but elegant?) recursive-function method brought not only more robust operation (a good thing – variances in the XML layout are handled now) but the performance has been greatly decrease (a bad thing – down from 0.008s to 0.014s average).

I’ve tried a lot of ideas to increase the performance again, including passing everything by reference (so the stack isn’t filled with copies of SimpleXML objects) – all to no avail.

I did however have (some) victory by reworking this:

This is slower by 0.000006s per iteration than the code in use below.

function process_xml ($x, $children=false) {
if ($subs) foreach ($z->children () as $x) process_xml ($x);
else {
// ... processing here

I wasn’t sure from the start that the additional recursion (in the foreach()) was a good thing, and my trial-and-error agreed:

function process_xml ($xml, $children=false) {
$to_process = ($children)?$xml->children ():array ($xml);
foreach ($to_process as $x) {
// ... processing here


Surprisingly (to me) the latter performed faster. ~0.000006s per iteration to be precise. Not much, but it adds up (larger XML files can reach 1000 or more iterations, recovering around half the performance lost earlier!)

The jury (in my mind) is therefore still out on whether to handle the XML each on each request (allowing users to turn on and off features, and include user-specific markup like highlighting etc) or whether to build a cache of HTML files alongside the XML counterparts. (Certainly, the latter would be faster. And given the Bible isn’t suspected to be changing any time soon…)

Some of the changes introduced include:

• Red-letter (Jesus’ words in red)
• Handling of quotation marks (IE doesn’t conform to the standards, but beside that the XML I use is UTF-8 encoded and provides sets of quotation marks as needed)
• (Corrected) handling of references, variances etc. Before the recursive code was introduced, I was missing any “notes” which bundled more than one reference up in them.

So it’s progress.

While on the subject of performance, I’ve been trying to work out some targets. How much memory will I allow this script to consume before it’s “too much”? How fast must it execute to remain acceptable?

Currently the figures for Mark chapter 1 come in at:

• 0.013s execution time
• 0.18MB peak memory usage

Interestingly, memory usage jumped up hugely after a tiny change in the recursive function (I can’t even remember what it was now). I thought about this for a moment at the time, and hypothesised that this really makes a lot of sense. Each iteration is shoving data onto the stack, eating memory and generally doing a whole lot of “weight throwing”. I went as far as jotting down my expectations of memory usage per iteration. I expect it to look like a very wide but short kind of bell-curve, initially shooting up as XML depth is reached, then bobbing up and down, finally tailing rapidly off as the end is reached.

Due to my geeky nature, I recorded the memory usage on Mark 1, and went to graph them. But I never got that far. Just looking at the results were quite clear: an immediate shoot upward (9KB across the first 9 iterations) then a very slow increase (about 1.5KB every 20 iterations) thereafter. And it never decreased.

PHP I believe uses a garbage collector, and presumably it considers my usage so low that it would have a negative effect to execute mid-script. Fair play!

My concern really boils down to this: let’s assume that this project really takes off. If we aim to handle 100 requests per second (that’s around 40 times what Wikipedia’s English wiki received on average 2003q1) … I want to service each request within 0.1s each and not exceed 512MB memory on a dual-core Intel (comparable to what I’m running on right now).

I’m working on the grossly over simplified (and wildly inaccurate) calculation:

MaxHits = Min((CpuCores/ExecutionTime),(MemoryLimit/PeakMemoryUsage))

Currently that stands at:

MaxHits = Min((2/0.013),(512/0.26)) = 154

So we’re in. We’re limited hugely by the execution time. I can afford to go up as far as 0.02s per request, but that’s it.

We’ll see what happens!