Sean's blog

Pos Malaysia shipping quote – part 2

The shipping quotes described here moved long ago to shipping-quote.net

Pos Laju shipping rates quotation demo 2

With a little bit of ‘reverse engineering‘, it’s possible to write a program to obtain a full set of postage rates from Pos Malaysia’s website. As far as I know, they don’t make their rates available in a form that can be easily used by 3rd parties. That’s simply not good enough in an age of e-business. One positive feature of Pos’ website must be noted: when they show a shipping quotation, they also show how they calculated it. That’s enough information to make my own copy of their rates.

I thought each step of this job would take me 1 day. I have to confess that reverse-engineering Pos’ rates, creating a persistent format for spider.my and finally the AJAX script and XML-generator to go with it took me one very large day. I started at 9am yesterday and had everything working satisfactorily at 2:30am this morning.

Try out part 2 of the shipping quotation demo: there’s no button to submit requests any more. Now that I have all of Pos Malaysia’s Pos Laju rates stored on spider.my, there’s no need for a ‘POST’ method. It would be semantically wrong, as the state of the server is no longer updated by a request. It was right for the part 1 demo, because the request loaded a page from Pos’ website, used pattern matching to extract the price, and added it to the server’s result cache.

I’m pleased with the way the new request method works out. All that a request needs to specify is a destination country and a weight. The persistent data on the server is stored as a per-country list of shipping methods. The shipping methods are retrieved for the specified country, and are tested to see if they are appropriate for the specified weight. For example, a document method will reject a 5kg request. I’ve made this feature part of the generic interface for shipping methods so it should be easy to add any number of different shipping methods (from different shipping agents), all potentially quoting on a single request.

Just in case the ‘accept/reject’ thing isn’t clear, compare the results of these two similar queries. The first is a request for a quote for shipping 0.5kg to China:

http://spider.my/pos-malaysia-shipping-quote-2/0.5-to-China.xml

(Just spotted my URLs are case-sensitive. Bummer. Experimental code mah! )and the second is a similar request, but this time for a heavier shipment – 2.5kg to China:

http://spider.my/pos-malaysia-shipping-quote-2/2.5-to-China.xml

You see two quotes for the lighter parcel because it qualifies (by weight, shape’s another matter) for Pos Laju’s Document service. The heavier parcel is too heavy for the Pos Laju Document service, so no quote is given.

It’s not an earth-shattering difference for the user over the previous instalment, but having a complete set of rating tables under my control is a big step for me. Now the ‘lag’ imposed by the fat request to Pos’ website is gone, so that means I can do things like provide a full set of up-to-date rating tables to 3rd parties for inclusion in their systems and products. That’s the next instalment!

For the interested train-spotter: the rates are ‘ripped’ with some Java code that makes shipping requests to Pos Malaysia’s website. It’s very similar to the code used in part 1, except instead of looking for the “Total Amount” value, I break down the info in the table to extract Method Name, Min Price, Weight Divisions, Increment per Division and Max Weight. It’s actually quite quick, though possible not terribly welcome by Pos Malaysia’s sysadmin. I only download the page text, not the associated images and resources, so making one request per country leads to around 200 requests being made, for something like 2-4MB of transfer. Because I’m extracting data from some fairly messy HTML (never use w3’s HTML validator? Adoi…), even the tiniest change to the page would completely break my update code.

I’ve implemented persistent storage for the shipping rates on spider.my, so there’s no need for me to run the ‘ripping’ code again, unless Pos update their rates. I might write ripping code for their local deliveries and their non-expedited shipping methods, but I don’t think it’s necessary for the sake of this demo. Their sysadmin doesn’t need to get worked up about the rip process: it’s only going to happen once!

For Pos Malaysia: Using arbitrarily-styled country names in your queries doesn’t work. Try requesting a quotation for a delivery to ‘Serbia & Montenegro’ (which hasn’t existed as a country since 2006). See? Here’s a request for 1kg to ‘Serbia & Montenegro’ to make it easy for you. What? It seems WordPress can’t handle that URL either! OK, here’s an image of the result of that request:

URLs with spaces and ampersands in them don't work

You need to encode the data you use in your URLs. Here’s a link for a quote for a parcel to ‘Serbia & Montenegro’ which does actually work on your website:

http://www.pos.com.my/V1/main2.asp?c=/v1/tariff/poslaju/EmsRatedb1.asp&negara=Serbia%20%26%20Montenegro&berat=1

I have to ask you though: Would you really accept a 999kg parcel to Eastern Europe? Hint: Look at the quote result. I’d like to see the envelope for that!