Sean’s blog

Pos Malaysia Shipping Widget for your website

(If you’re in Malaysia, the latency to spider.my’s server in the USA is going to make this appear awful. As long as my local Internet bomoh is smiling at me, you might be able to see a Malaysian copy at http://pos.spider.my/ – just don’t depend on it being available.)

I’ve never written a widget before! This one hasn’t turned out bad – try changing the parcel weight and destination. See? It works! And you can have it*1 on your website too! All you need to do is to add this little bit of HTML:

<div id=”pos.spider.box”><script type=”text/javascript” src=”http://spider.my/static/js/posmalaysia4.js”></script></div>

…and next time you look at your webpage, it’ll have Pos Malaysia shipping rates on it, courtesy of spider.my and a very dodgy browser-security workaround to allow cross-domain … actually it isn’t really AJAX, it’s more like Asynchronous JavaScript And More JavaScript, but you’ll have to wait for the HowTo post to see how I do it (or just view my page source).

At the start of the week I said I’d demonstrate a how a Pos Malaysia API (Application Programmer’s Interface) should work. As a recap, it was to have two major parts: a simple, reliable, fast technique for obtaining a single quote in a format that can be readily used by any application, and a technique for providing a complete set of rating tables for applications that need higher performance or reliability, but also want easy access to updated rates.

It has actually been a very solid week’s work, but it’s now all done. On Monday, I demonstrated what the problem is. Pos Malaysia’s online shipping quotation page is OK*2 for humans to look at, but is too ‘fat’ and arbitrarily formatted for efficient and reliable use in e-commerce. I hinted at an AJAX replacement by caching quotes on my ‘old-technique’ page.

On Tuesday I started ‘ripping’ Pos Malaysia’s shipping rates. The whole problem is the difficulty of access to their rates, so having my own set would make the rest of the demo much easier. To avoid upsetting the Pos Malaysia sysadmin, I wrote a set of persistent classes for the rating data on spider.my, so that I would only have to make the hundreds of requests needed to rip the data just once.

On Wednesday I tidied up the server-side access to the ripped rates and provided a reliable AJAX interface which accepts a weight and a destination and returns quotes from all qualifying methods. I currently only have Pos Laju Document and Parcel rates, so if you ask for a weight over 1kg, you get only 1 quote, 2 quotes for anything lighter (now more quotes since Pos Parcel rates added). At this stage, part 1 of the job is finished: there’s now a simple technique for Pos Malaysia’s customers and partners to instantly acquire a single quote.

The method for complete rating data updates was quite straightforward to provide once the server-side persistence was finished. On Thursday I provided a rates download facility which can give full rating data for all countries and all methods, all methods for one country, or all countries for a single method. I have an e-commerce project ‘on a back burner’ at the moment which will use stored shipping rates to simplify and speed up its checkout. With Pos Malaysia’s current online facility that’s practically impossible. If Pos provided this kind of API it would be straightforward.

As a demonstration piece, I started writing a ‘widget’ that can be used on anybody’s blog or website on Thursday night. I’ve never written much in the way of javascript before this week, so I’ve been spending a lot of time at w3schools.com! I thought I’d finished the widget last night, but discovered the “Same Origin Policy” which prevents javascript on a page from one website loading data from another website. There seem to be a lot of very complicated solutions for this problem, but this afternoon I stumbled on a simple one.

The widget is not very pretty, but I’ve got no intention of providing shipping rates from spider.my in the long run. I want Pos Malaysia to allow me to set up this API (or something similar – there are already improvements I have in mind!) on their pos.com.my domain – possibly as api.pos.com.my – so that everyone who uses Pos’ services in their e-businesses can benefit. If Pos likes the widget idea, then I think a little bit of effort to make it more ‘beautiful’ will be good for Pos, as they’ll have their branding on other people’s websites.

If Pos wants my help to get this up and running on their server, I will do it FOR FREE. The benefit to me in having a reliable quotation system in place at Pos will be huge for my own projects. If other businesses in Malaysia also benefit, and if Pos Malaysia win some prizes for competent IT, then I’d like to think I might get some free cups of teh tarik, or some wang ehsan or something. I’ll let you know how I get on with Pos next week.

*1 – Yes, you can put it on your blog or website. It’s not very pretty, but if you’d like to try it out I would be very pleased that you did. Let me know what you think of it functionally. If any Malaysians with design skills fancy demonstrating just how beautiful a Pos Malaysia widget could be, give it your best shot and if it looks nice, I’ll adopt your design, and give you all the credit (but no cash, soz. You could try asking Pos, I guess). One last thing – this is very experimental, so it might stop working from time to time (might, but might also keep going forever), and watch out for the rates: they don’t include Pos’ silly random surcharges.

*2 – OK… but it needs some basic maintenance! There are very many quotes which claim to allow shipments up to 999kg. It looks like Pos has never got round to setting the maximum values for all their countries. Also the country names – some of them are oddly styled like ‘Ukraine (Kiev)’ and ‘Serbia & Montenegro’. The latter hasn’t existed since 2006, according to Wikipedia! The technique Pos uses to request a quote from their server doesn’t actually work with names like these (try getting a quote for shipping to ‘Serbia & Montenegro’ from Pos’ website). The names form the URL for the quote, but contain invalid characters for URLs, so some countries return errors instead of quotes!

Posted in Fixed, software | 2 Comments »

Pos Malaysia shipping rates download

Download Pos Malaysia shipping rates at spider.my

Nearly there! Yesterday’s marathon session to rip Pos Malaysia’s rating data and make it persistent on spider.my made today’s job a little bit easier. The first part of the API is done: there’s a minimal interface for obtaining a single shipping quotation. Today sees the second part complete: a method for obtaining a complete set of rating data for a 3rd-party application.

If you visit part 3 of my Pos Malaysia API demo, you can now select country and shipping method (or all of one, or all of both), and the page will give you a link from which you can download a full set of rates in XML format. There is just one little wrinkle, and that is that I’m using unchanged data from Pos Malaysia. On some of their (less-popular, I’m guessing) destinations, they claim to deliver parcels up to 999kg! I store the shipping data as a table of prices against all the different weights for a shipping method, so with 0.5kg divisions, I end up with 2,000 entries for some methods. My storage method is quite efficient, so it’s not a problem on the server, but if you request All Countries + All Methods, the XML data file is several MB in size.

For an example of one of the smaller data files, see this XML file containing the data for Pos Laju Document and Parcel deliveries to the UK:

http://spider.my/pos-malaysia-rates-3/-ANY–to-United%20Kingdom.xml

See? That’s a very simple format, easily incorporated into your current application. There are just a couple of little things I’d like to do in the way of demonstrations, and then I’ll try to find someone at Pos who might be interested in providing an API to local e-businesses.

Posted in Fixed, Life, Money, software | No Comments »

Pos Malaysia shipping quote – part 2

Pos Laju shipping rates quotation demo 2

With a little bit of ‘reverse engineering‘, it’s possible to write a program to obtain a full set of postage rates from Pos Malaysia’s website. As far as I know, they don’t make their rates available in a form that can be easily used by 3rd parties. That’s simply not good enough in an age of e-business. One positive feature of Pos’ website must be noted: when they show a shipping quotation, they also show how they calculated it. That’s enough information to make my own copy of their rates.

I thought each step of this job would take me 1 day. I have to confess that reverse-engineering Pos’ rates, creating a persistent format for spider.my and finally the AJAX script and XML-generator to go with it took me one very large day. I started at 9am yesterday and had everything working satisfactorily at 2:30am this morning.

Try out part 2 of the shipping quotation demo: there’s no button to submit requests any more. Now that I have all of Pos Malaysia’s Pos Laju rates stored on spider.my, there’s no need for a ‘POST’ method. It would be semantically wrong, as the state of the server is no longer updated by a request. It was right for the part 1 demo, because the request loaded a page from Pos’ website, used pattern matching to extract the price, and added it to the server’s result cache.

I’m pleased with the way the new request method works out. All that a request needs to specify is a destination country and a weight. The persistent data on the server is stored as a per-country list of shipping methods. The shipping methods are retrieved for the specified country, and are tested to see if they are appropriate for the specified weight. For example, a document method will reject a 5kg request. I’ve made this feature part of the generic interface for shipping methods so it should be easy to add any number of different shipping methods (from different shipping agents), all potentially quoting on a single request.

Just in case the ‘accept/reject’ thing isn’t clear, compare the results of these two similar queries. The first is a request for a quote for shipping 0.5kg to China:

http://spider.my/pos-malaysia-shipping-quote-2/0.5-to-China.xml

(Just spotted my URLs are case-sensitive. Bummer. Experimental code mah! )and the second is a similar request, but this time for a heavier shipment – 2.5kg to China:

http://spider.my/pos-malaysia-shipping-quote-2/2.5-to-China.xml

You see two quotes for the lighter parcel because it qualifies (by weight, shape’s another matter) for Pos Laju’s Document service. The heavier parcel is too heavy for the Pos Laju Document service, so no quote is given.

It’s not an earth-shattering difference for the user over the previous instalment, but having a complete set of rating tables under my control is a big step for me. Now the ‘lag’ imposed by the fat request to Pos’ website is gone, so that means I can do things like provide a full set of up-to-date rating tables to 3rd parties for inclusion in their systems and products. That’s the next instalment!

For the interested train-spotter: the rates are ‘ripped’ with some Java code that makes shipping requests to Pos Malaysia’s website. It’s very similar to the code used in part 1, except instead of looking for the “Total Amount” value, I break down the info in the table to extract Method Name, Min Price, Weight Divisions, Increment per Division and Max Weight. It’s actually quite quick, though possible not terribly welcome by Pos Malaysia’s sysadmin. I only download the page text, not the associated images and resources, so making one request per country leads to around 200 requests being made, for something like 2-4MB of transfer. Because I’m extracting data from some fairly messy HTML (never use w3’s HTML validator? Adoi…), even the tiniest change to the page would completely break my update code.

I’ve implemented persistent storage for the shipping rates on spider.my, so there’s no need for me to run the ‘ripping’ code again, unless Pos update their rates. I might write ripping code for their local deliveries and their non-expedited shipping methods, but I don’t think it’s necessary for the sake of this demo. Their sysadmin doesn’t need to get worked up about the rip process: it’s only going to happen once!

For Pos Malaysia: Using arbitrarily-styled country names in your queries doesn’t work. Try requesting a quotation for a delivery to ‘Serbia & Montenegro’ (which hasn’t existed as a country since 2006). See? Here’s a request for 1kg to ‘Serbia & Montenegro’ to make it easy for you. What? It seems WordPress can’t handle that URL either! OK, here’s an image of the result of that request:

URLs with spaces and ampersands in them don't work

You need to encode the data you use in your URLs. Here’s a link for a quote for a parcel to ‘Serbia & Montenegro’ which does actually work on your website:

http://www.pos.com.my/V1/main2.asp?c=/v1/tariff/poslaju/EmsRatedb1.asp&negara=Serbia%20%26%20Montenegro&berat=1

I have to ask you though: Would you really accept a 999kg parcel to Eastern Europe? Hint: Look at the quote result. I’d like to see the envelope for that!

Posted in Fixed, software | 2 Comments »

Pos Malaysia e-commerce shipping quote with AJAX

Pos Malaysia shipping quote at spider.my - part 1

Here’s a first shot at an AJAX-powered Pos Laju shipping quote. As I said in my previous article on a Pos Malaysia shipping quote API, there should ideally be two: one simple one for blogs and low-volume e-commerce sites, and one for large-volume businesses. The shipping quote demo at spider.my is very simple: just enter a weight and a destination, and click on a ‘Quote’ button to get an up-to-date quote from Pos Malaysia’s website. This is a terrible way of doing it, as spider.my retrieves a whole web-page from Pos’ website, just for one price!

You can see how slow it is if you submit the form. Part of that is nothing to do with Pos Malaysia – spider.my is on a server in the USA, so there’s massive latency between your browser (probably, if you’re in Malaysia), the spider.my server in the USA, then back to Malaysia for the quote request, then back to USA with the quote page, where it’s processed using pattern matching to extract the price, then back to your browser with the finished page.

To demonstrate the simple Shipping rate quote API, I’ve added an AJAX updater to the page. That’s a little bit of asynchronous javascript that executes in your browser. When you change the weight of the parcel, or its destination, that script will request a quote matching the new details from spider.my. To give you a feel for how an API at Pos Malaysia’s website would respond, the AJAX updater only provides responses for previously-cached quotes at spider.my. That way, there’s no need to make another long request to www.pos.com.my!

Request timings for AJAX XML shipping quote request

Have a look at the XML sent by the shipping rates demo – it’s extremely small. Comparing the timing (again from Chromium) for the transfer of the XML response from spider.my to the timing of the transfer of the quotation webpage from www.pos.com.my from yesterday. Remember that the spider.my server is in the USA, so network latency is going to be higher. Pos’s local latency isn’t great – but their server must build that page before it can transfer it. Spider.my is only building a very small (less than 100bytes!) XML message. The real benefit comes when the data is transferred. The XML message is so small, Chromium is probably defaulting to 1ms for a timing. The whopping great page from pos.com.my takes nearly a quarter of a second to transfer! For user interaction, that leaves very little time for anything else.

Notice that getting the price update via AJAX from the USA takes a total of just under 300ms, while the local method, from Pos’ website, took just under 350ms. Even with the handicap of being served on the other side of the planet, the AJAX technique is much faster. It should be clear that if the AJAX response was coming from Pos’ server, not only could the response be hundreds of times faster, it would be hundreds of times more efficient in bandwidth.

I’m not ready to try to convince Pos again to provide an API. To make it more convincing, I need to handle any quote that Pos can handle. For that, I need to acquire their rates. That sounds like hard work, since Pos don’t make their entire rates available in any other format than photocopies (I have a set – I won’t be using them). I have a cunning plan for that. Once I can provide super-efficient one-shot quotations from my server, I’m going to tackle a client-side solution for businesses that might need to (a) not depend entirely on Pos’ server being available (b) get the fastest possible quotes. That depends on my cunning plan working so that there’s an efficient way of getting an update from Pos whenever they update their rates across the board.

Posted in Broken, Spider.my, software | 1 Comment »

Pos Malaysia shipping quotes for e-commerce

Pos Malaysia

I maintain some Malaysian shipping modules for the popular e-commerce software osCommerce which allow local e-shops to include up-to-date automatic shipping quotes from Pos Malaysia (Pos Laju and Pos Air Parcel) in the order total. If you run an e-shop, being able to pass on accurate shipping costs to your customers – compared with ‘free’ or fixed-price shipping – means you can give your customers better prices than your competitors. If you can offer your customer a choice of shipping options, this also allows them to save money or expedite their delivery. Doing all of this automatically means you don’t have to attend to your online business – it completes orders and collects money for you, and tells you when to pack deliveries!

Pos Malaysia have a shipping quotation service at their website. It’s obviously intended for human visitors, as it also has advertising, branding and links to other services on it. I use this web page in my osCommerce module. In order to give an automatic, up-to-date quote, an osCommerce shop using my modules must send a request to Pos Malaysia’s website, read the entire page, and use pattern matching to extract the shipping price.

The current technique is very wasteful of resources, the page that must be searched for the shipping price is 13 to 15KB – for just a few characters! The technique is also horrendously sensitive. If Pos Malaysia change the appearance of the page, the pattern-matching technique can’t work and the shipping module fails to return a quote, which will usually mean that the osCommerce e-shop won’t be able to complete the order. The Internet being the ‘instant’ experience that it is, the sales opportunity is probably lost.

One further annoyance is that the shipping module requesting one or two (for different shipping methods) quotes from Pos Malaysia’s website introduces a considerable delay in the user’s experience of the e-shop. When a customer goes to the checkout page on which the shipping cost appears, the server starts to build the page, sends a request to Pos Malaysia’s website, waits for Pos Malaysia’s webserver to build its response, reads the (large) response, performs pattern matching on it (repeating the last 4 steps for each additional shipping method) and finally sends the page to your customer’s browser. Several-second delays are not uncommon.

Pos Malaysia quotation page timings

Take a look at these two resource timing graphs from the Chromium browser. If you haven’t used Chromium yet, and you’re a developer, there are some great tools in there – and it’s shaping up to be a very competent browser too. The first image is from Pos Malaysia’s website, a shipping quote for a parcel (surface and air quotes on the same page). The blue bar is for the page itself – the other bars are for resources the page needs to be rendered. We only need the page, as we’re not interested in images, stylesheets and scripts, only in the text of the shipping amounts. There’s a redirect (a 302 Found) which further slows things down, and then a 104ms wait (perhaps the page is being built by Pos’s server?) and then nearly a quarter of a second while the ~14KB of the page text is transferred.

AJAX load time from spider.my's currency conversion page

Compare that with this image, which shows the resources being loaded from spider.my’s currency conversion page. The line to note is that for the AJAX update of the exchange rate. The latency is horrible, as spider.my is hosted in the USA, but the time to load the AJAX update is only 1ms – hundreds of times faster than Pos’ shipping quote page. That’s understandable because the reply from spider.my is only about 100bytes long!

What I think Pos Malaysia should do is to provide an API to Malaysian e-business so that their postal services can be more efficiently and reliably built into local business systems. I think they should offer 2 APIs – one a simple AJAX responder for use on blog pages and low-volume shops, and the other a shipping rates updater for larger-volume businesses. The simple responder would take the place of the current osCommerce modules and should be able to easily serve thousands of shipping quote requests per second on any old PC that Pos has lying around. I envisage some heavy client-side caching for the high-volume API, so apart from days on which Pos update their rates, the server should see very little traffic at all for a more complex solution suitable for large businesses.

It was obvious to me when I first wrote the shipping modules a few years ago that they were inefficient and fragile, so I emailed and wrote to Pos Malaysia, even met their regional management to try to get someone to improve their end of the shipping quote system. A few years have passed and nothing has happened, so I’m going to write it myself and try to get someone from Pos to have a look at it and hopefully host it themselves. It’s not difficult and it’ll make a nice test of Spinneret. After all, when I finally get round to making Spinneret available, it will be FOSS buatan Malaysia!

If Pos wants the system, I’d be quite happy to do all the necessary work to integrate it into Pos Malaysia’s current IT systems FOR FREE. I don’t want to maintain the osCommerce modules in their current condition any more, and I think Pos exposing a decent API to local e-businesses would make a big difference to the ability of Malaysian businesses to compete online. That would make my life easier, and hopefully earn me a few free cups of tea at the kopitiam.

Posted in Broken, Life, Money, Philosophy, software | 3 Comments »