Sean's blog

Pos Malaysia Shipping rates / quotations for osCommerce and Zen Cart

The shipping quote API and shipping modules for osCommerce and Zen Cart is now at Shipping-Quote.net

Pos Malaysia have updated their website in the last few days, with new page markup and new URLs. Either one of those changes would have broken the old osCommerce shipping modules I wrote. I won’t be updating the old shipping modules any more, as random, non-improving changes to postal service websites that break business functionality are completely intolerable.

The extra bad news is that I haven’t quite finished updating the spider.my quotation functionality to cope with Pos Malaysia’s web changes. The good news is that I should finish either late tonight (16 Jan 2011) or sometime tomorrow. I have test versions of shipping plugins available for both osCommerce and Zen Cart, and I’ll put those online probably tomorrow afternoon.

The good news is that the new shipping modules should greatly improve:

• the responsiveness of your website by transferring only the data (no bloated markup) needed to give a shipping quote
• the ease with which you can add shipping quotations to your website – there’s only one shipping module for each e-commerce suite, with config to select your favourite postal services
• the reliability of your shipping quotations – scraping Pos’ website for data just completely fails whenever Pos fiddles about with its website. Spider.my will provide a dedicated data-only API for shipping quotations that will continue to work whether Pos’ website is messed about with or even offline.

Enough of that, I thought I’d better post a head’s up. The first report I had of a problem was from a Malaysian online shop whose last international delivery went out on Thursday 13th, so these changes have happened since then. More later when I have the code ready.

Posted in Broken, Money, Spider.my | No Comments »

Expires header fails, max-age works: RFC1123 compliance

While I was testing the mobile version of the search at spider.my I noticed that my Nokia phone’s browser was briefly presenting an unformatted version of the page, then applying styles, then adding a small image. Doing that once would have been expected, but it shouldn’t have been doing it on subsequent requests of the search page. I serve my CSS files and images with an Expires header field which should prompt user agents to re-use the previously loaded (and cached) CSS and images.

I saw this article about Expires versus the Cache-Control header’s max-age, so I tried max-age and it worked! With max-age, my Nokia phone was no longer requesting the CSS and image it should already have cached. Doubt that Nokia (their phones just work really, really well) would have broken something so basic and a comment in mnot’s article about missing out a leading ‘0’ on the hour part of the Expires date made me check my code again. Here’s what I was sending:

Expires: Mon, 2 May 2011 01:53:37 GMT

Nothing wrong with the hour, but there is an obvious candidate for a leading zero there. After changing the way my webserver formats dates (I had been formatting them to RFC1123/822, which does specify 2-digit hours), so that it sent 2-digit day-of-month, it worked! The new Expires field looks like this:

Expires: Mon, 02 May 2011 02:03:37 GMT

RFC822 says you can specify 1 or 2 digits for the day-of-month part and RFC1123 (specified as the date format for the Expires header in RFC2616) doesn’t alter that. Sending 1-digit day-of-month parts in a RFC1123 date is to specification.

I just lodged a bug report with Nokia – it seems to me that their phones *should* be accepting the RFC1123-compliant Expires date. I wonder how many other user agents will have this issue, as it seems some people believe that the day-of-month part should have two digits – see Dave Bauman’s comment at the foot of this StackOverflow thread.

Posted in Broken, software, Spider.my, Useful | No Comments »

SSH port forwarding from VPS to home server

Spider.my is served by a very small VPS provided by Nocser. The search function of spider.my is currently served by 2 reasonably-sized Xeon servers near my desk. I can’t currently afford to rent the equivalent amount of computing power from a hosting company, so the front-end of spider.my is all served by the VPS, with search results and indexing all done at home. I did have a horrible cross-domain javascript hack which caused a user’s browser to fetch the search results from my servers at home, but it was a horrible kludge and according to the server logs didn’t seem to work on all browsers.

I briefly considered writing some sort of proxy code for the webserver on the VPS, but this also seemed a kludge. The backend of spider.my is implemented as separate RMI services. One obvious solution would be to open up the RMI service ports on my router at home and use Dynamic DNS to keep a domain name resolving to whatever my home IP address is, and make RMI requests from code running on the VPS. This also seemed unnecessarily complicated and an obvious security risk.

What I plumped for in the end is SSH port forwarding – connecting my backend service with the frontend on the VPS over a secure socket. The SSH connection is established by the backend so there’s no need for Dynamic DNS. An SSH connection is setup by an unprivileged account on the backend server which opens sockets on the frontend server. Connections made by applications running on the frontend server are then forwarded to corresponding ports on the backend server.

Here’s the command I use on my backend server to show use of the -R argument:

ssh -R 1099:localhost:1099 -R 1100:localhost:1100 -R 1101:localhost:1101 nopriv@frontend.box -N

Any connections to 1099 on the frontend server are forward to port 1099 (the default rmiregistry port) on the backend server. I have two RMI services running on the backend server, on ports 1100 and 1101. The frontend code can connect to the rmiregistry as though it was running on localhost, but all communication is forwarded over the SSH connection. It works a treat!

I came up against just 2 or 3 ‘wrinkles’. The first was host naming – I ended up making an entry in the frontend’s /etc/hosts to map the backend server’s hostname (for rmiregistry discovery) onto 127.0.0.1 – that allows me to use the application unmodified. To enable the connection to be established without typing a password, ssh-keygen has to be used exactly as stated in ssh’s manpages. Last, I had a problem with the SSH session dying, possibly due to lack of activity as in this article. I used the ‘-o ServerAliveInterval=’ command line option and that seemed to make it rock-solid. I’m also using ‘-c blowfish’ just because the manpages say “it appears very secure and is much faster than [the default]”.

Posted in Uncategorized, Useful | No Comments »

Spider.my simple mobile version

Spider.my mobile search on my Nokia 6500 Classic

Spider.my really wasn’t working very well on any of our mobile phones (handphones as they’re called in Malaysia), as the search had been using a terrible cross-domain javascript hack to fetch results from the back-end. A couple of things I’ve been doing recently makes that more or less redundant, so a few simple pages that work ‘well-enough’ on a mobile phone should be straightforward. A very cut-down search page is now available at http://spider.my/m/search.xhtml as well as redirects at spider.my/m and spider.my/m/.

I’m trying to adhere as much as possible to the W3’s Mobile Web Best Practices guide which seems both informative and well-written. It’s definitely worth a read if you’re starting out in developing webs for mobile phones!

Posted in Search, Spider.my, Uncategorized, Useful | 1 Comment »

Malaysian shipping quote widget revisited

A lot of the shipping quotation stuff that I did using Pos Malaysia’s rates has been broken while I’ve been stumbling through getting spider.my updated.

To add the widget to your own web page, just copy and paste the code from below, and next time you load the page you should see the widget:

<div id="mySpiderShippingWidget"><script type="text/javascript" src="http://spider.my/static/js/shippingwidget.js"></script></div>

I’m currently finalising the detail on the API, and should have a couple of shipping modules up soon for Zen-Cart and osCommerce which will give quotes for Pos-Malaysia’s Pos Laju, Surface Parcel and Air Parcel for documents and parcels. Like the old shipping modules these will also be free, but only for very occasional use.

What I’m intending to do is to offer a paid ‘API key’ to heavier users. The API key will be less than RM50 per year and will help me cover the cost of hosting. Until the API key scheme is finalised, access to the shipping quotations will be free and unlimited.

(Update 11th Jan 2011: Widget has been off and on for some time while I’ve been beating the API into shape. It should continue to work in its present form for the foreseeable future. Check out the spider.my API to see what the widget uses for shipping data. There is a RateLimit in the API at the moment of about 50 responses per hour. That rate will be reduced when I get the APIKEY interface finalised.)

Posted in software, Spider.my, Uncategorized, Useful | No Comments »