Archive for the ‘Uncategorized’ Category

Improving CoovaAP portal pages for iPhone and Android - March 19th, 2011

Over the last couple of weeks, I’ve been spending my spare time playing with the Coova AP (ChilliSpot) wireless hotspot service on a WRT54GL.

The main problem I’ve had with it is the portal pages not looking so good on mobile devices (particularly iPhone and Androids).

A little bit of Googling revealed a solution (at least, a solution for websites in general): insert the following meta tag into the HTML header.

<meta name="viewport" content="width=320; initial-scale=1.0; maximum-scale=1.0; user-scalable=0;" />

In the CoovaAP configuration interface, you can apply this for every portal page by adding it to the “HotSpot”/”Portal”/”HTML Title”.

Ebuyer – maxing out the result limit - February 22nd, 2011

This might be useful to someone, somewhere, out there.

Those who’re into tinkering with things they shouldn’t will notice that Ebuyer’s search pages include a “limit” GET variable in the results page URL. The upper limit appears to be 30, but going negatively causes a different kettle of fish:
http://www.ebuyer.com/search?q=a%25&x=0&y=0&limit=-3000
(That URL is not for the feint hearted!)

It makes one wonder, is there a little bit of sanitization fail?

Cloud-Bible progress - February 6th, 2011

Two blogs in one weekend. Impressive for me!

I’ve made a lot of progress with the Cloud-Bible project (the name is growing on me, slowly).

Switching from the simpler-to-follow (but very assuming) big “switch” clause which was previously handling reverse-engineered tags (oh, I didn’t mention the XML I have is not precise OSIS format did I? :() to the more complex (but elegant?) recursive-function method brought not only more robust operation (a good thing – variances in the XML layout are handled now) but the performance has been greatly decrease (a bad thing – down from 0.008s to 0.014s average).

I’ve tried a lot of ideas to increase the performance again, including passing everything by reference (so the stack isn’t filled with copies of SimpleXML objects) – all to no avail.

I did however have (some) victory by reworking this:

This is slower by 0.000006s per iteration than the code in use below.

function process_xml ($x, $children=false) {
if ($subs) foreach ($z->children () as $x) process_xml ($x);
else {
// ... processing here

I wasn’t sure from the start that the additional recursion (in the foreach()) was a good thing, and my trial-and-error agreed:

function process_xml ($xml, $children=false) {
$to_process = ($children)?$xml->children ():array ($xml);
foreach ($to_process as $x) {
// ... processing here

Surprisingly (to me) the latter performed faster. ~0.000006s per iteration to be precise. Not much, but it adds up (larger XML files can reach 1000 or more iterations, recovering around half the performance lost earlier!)

The jury (in my mind) is therefore still out on whether to handle the XML each on each request (allowing users to turn on and off features, and include user-specific markup like highlighting etc) or whether to build a cache of HTML files alongside the XML counterparts. (Certainly, the latter would be faster. And given the Bible isn’t suspected to be changing any time soon…)

Some of the changes introduced include:

  • Red-letter (Jesus’ words in red)
  • Handling of quotation marks (IE doesn’t conform to the standards, but beside that the XML I use is UTF-8 encoded and provides sets of quotation marks as needed)
  • (Corrected) handling of references, variances etc. Before the recursive code was introduced, I was missing any “notes” which bundled more than one reference up in them.

So it’s progress.

While on the subject of performance, I’ve been trying to work out some targets. How much memory will I allow this script to consume before it’s “too much”? How fast must it execute to remain acceptable?

Currently the figures for Mark chapter 1 come in at:

  • 0.013s execution time
  • 0.18MB peak memory usage

Interestingly, memory usage jumped up hugely after a tiny change in the recursive function (I can’t even remember what it was now). I thought about this for a moment at the time, and hypothesised that this really makes a lot of sense. Each iteration is shoving data onto the stack, eating memory and generally doing a whole lot of “weight throwing”. I went as far as jotting down my expectations of memory usage per iteration. I expect it to look like a very wide but short kind of bell-curve, initially shooting up as XML depth is reached, then bobbing up and down, finally tailing rapidly off as the end is reached.

Due to my geeky nature, I recorded the memory usage on Mark 1, and went to graph them. But I never got that far. Just looking at the results were quite clear: an immediate shoot upward (9KB across the first 9 iterations) then a very slow increase (about 1.5KB every 20 iterations) thereafter. And it never decreased.

PHP I believe uses a garbage collector, and presumably it considers my usage so low that it would have a negative effect to execute mid-script. Fair play!

My concern really boils down to this: let’s assume that this project really takes off. If we aim to handle 100 requests per second (that’s around 40 times what Wikipedia’s English wiki received on average 2003q1) … I want to service each request within 0.1s each and not exceed 512MB memory on a dual-core Intel (comparable to what I’m running on right now).

I’m working on the grossly over simplified (and wildly inaccurate) calculation:

MaxHits = Min((CpuCores/ExecutionTime),(MemoryLimit/PeakMemoryUsage))

Currently that stands at:

MaxHits = Min((2/0.013),(512/0.26)) = 154

So we’re in. We’re limited hugely by the execution time. I can afford to go up as far as 0.02s per request, but that’s it.

We’ll see what happens!

Bible study on the cloud (“cloud-sword”?) - February 5th, 2011

I’ve for a long time enjoyed using e-Sword. It’s a superbly well built package which runs on Microsoft Windows (and partially under Wine on Linux) … which is where I see the downfall.

The advantages to sticking with one platform are obvious – speed, usability, feature range, you name it. But that’s just it. You’re sticking with one platform.

I didn’t realise until very recently, “e-Sword live” has been launched. I couldn’t actually login, so I was stuck with the basic version … and that (as I expected) only contained a fraction of the features the full e-Sword does. (That is not a gripe, that’s an observation.)

On my Android phone I use “YouBible”, which does the job. But it requires internet connectivity …

So what am I getting at. Well, for the last 3 months I’ve been toying with the idea of writing an e-Sword like application, on the cloud. Yes. That’s a massive undertaking. No, it’s not because I believe there’s anything wrong with e-Sword, neither do I believe there’s anything wrong with biblegateway.com or e-Sword live!

This is to give myself a better understanding of XML, PHP and webapps, to play around with using direct filesystem-based storage over MySQL, and to possibly provide something really usable for those who cannot run e-Sword (i.e. Mac users).

Long term possibilities include Facebook integration. Twitter output. Etc.

I can’t link to the work I’ve done so far, there’s no copyright messages displayed or anything. And it’s not yet that impressive.

Currently I’ve written a parser for the OSIS document format, which splits and outputs the Bible in smaller chunks (chapter level), adding XML tags where I deem necessary. SimpleXML powers the viewer; which can load, process and output a chapter in currently 0.004s. (That’s on low powered hardware too.)

References, “explanations”, passage chapters, verse numbers, footnotes etc are already mostly handled. (3 evening’s programming so far.) A lot to of gratitude to the OSIS and SWORD guys. Awesome stuff. I wouldn’t be 40% of the way there if it wasn’t for you.

I’ve not yet got a domain, or anywhere to properly host it. Dreamhost may do to begin with, you never know, if things really take off – RapidSwitch’s dedicated servers may have me back once more!

I’ll post here as things progress.

A poem I wrote - December 2nd, 2010

Regarding safe hex, on Linux – a topic at work today:

“A wise man once said to me,
‘The root account, you should not be.’
Alas my pride, for great it was,
‘I shall be root, just because!’
Rename this, move off that,
Getting rid of some old tat.
Then, behold! A stray forward-slash…
My entire filesystem, in the trash.

~

‘Take heed my children, obey these words.
Running as root, is not for nerds!’
So my lesson learnt, my hard drive burnt,
The painful task of rebuilding starts.”

Copyright (c) Matthew Hall (wally), 2010.

Creative Commons Licence

A poem of root. by Matthew Hall is licensed under a Creative Commons Attribution-ShareAlike 2.0 UK: England & Wales License.
Based on a work at matt.matzi.org.uk.

Hamlet - November 29th, 2010

To be, or not to be.

That is the question.

In pseudo code, it could be expressed:
question=(2b)||(!2b)

Therefore of course, the question is “false” – except for when 2b is naught.

I’m so bored.

PS3 and iPlayer - November 18th, 2010

Recently my PS3’s ability to use iPlayer stopped functioning properly: I couldn’t play programs, and the screen didn’t fit.properly.

Today I discovered a fix.  Just press triangle for the menu, and under tools clear all of the browser cookies, cache and finally reload.  Voila!

Hopefully this will prove useful for someone, if not only myself in 6 months time.

My idea for this year - October 30th, 2010

Before some big company steals it and I have a hard time proving “I thought of that first”, here is as captured by Google bots around the globe:

Wii sensor bar built into a TV.

Yeah. That’s one heck of a USP.

Wait no you cannot be PCI-DSS compliant – Security Metrics found an IPS! - October 20th, 2010

The last 3 days of my life have been spent wallowing in the deep grave of mud and syrup which is (drum-roll please) “Security Metrics’ PCI-DSS Vulnerability Scanner“.

I could rant for 3 pages, oh I really could. I admin a half-dosen of Linux boxes (shortly to be replaced with quite a few more) – which I take as much pride over as I do my own property. In fact, I probably take more pride in them than I do my own property.

If someone were to so much as accidently cancel a page load (port 80, HTTP) to our customer facing site, I’d know about it.  Once I caught the telephone system glitching out – I wondered why there were several 404 errors for “ourdomain.com/dotheyanswerthephones”.  Yes, I’m that observant.

These boxes are so secure, I’d happily put everything I own against them not being hacked.  They are the essence of epic.  Every T is crossed and lower-case J dotted.  And  then some.  And then some more.

Alas, I digress.

So what’s led me to blog this?

Well it’s not the score of 45 points which didn’t exist 4 weeks ago.  (Yes, I’m that proud of my work – I won’t settle for even the “warning” points).  Oh no.  It’s not the repetative emails saying “You’re not PCI-DSS compliant” (which is a load of rubbish – we are PCI-DSS compliant – we’re just just flagging some false positives on some third party’s Nessus install), and it’s not even the “Open SMTP relay” which relays mail for our own domains! (Google reveals I’m not the only one with this problem … some sanity remains).  It’s not even even even even the POP3 and SMTP ports which provide outdated SSL certificates (because obviously POP3S is POP3 with the S standing for “super”-SSL.  Oh no wait…).  And I will skip the detecting my Debian Lenny box as a Buffalo Wireless Access Point.  Idiots.

No today, this one started over the “You’re using an outdated OpenSSH version, please upgrade it – or wait for your vendor to provide a new version” (which is in-line with PCI-DSS) – yet continually report me as failing while I wait.

And then it finished with this (and I’m only a third of the way through fixing over 300 points of fail):

Description: scan may have been dynamically blocked by an IPS Severity: Potential Problem Impact: The scan results may be inconclusive. Background: An Intrusion Prevention System (IPS) is a device which protects a system or network from attackers. An IPS works by monitoring a network for malicious activity, and blocking that activity. Resolution Temporarily disable the Intrusion Prevention System or configure an exception for the scanner’s IP address before starting the scan.

Oh my goodness.  You’re actually giving me a score of 3 points because you’ve (incorrectly) detected a possible IPS – which is protecting my servers from attacks?!

</life>

</sanity>

Mario Bros - October 4th, 2010

As some people (… family, co-workers, wife) know, I’m quite a fan of a particular classic Nintendo game …

Today I found myself the coolest ringtone ever!
http://www.supermariobrothers.org/mario-music.html

When you’ve grabbed that to your phone, checkout the utter awesomeness:
http://www.youtube.com/watch?v=kcKurvm_0oE&feature=related