Sean's blog

Pos Malaysia shipping rates download

Download Pos Malaysia shipping rates at spider.my

Update January 2011 – there won’t be any rates download for some time. If you’re looking for a shipping module for osCommerce / Zen Cart try this article. If you’re looking for simplified access to Pos Malaysia’s shipping rates, you could use the spider.my API here.

Nearly there! Yesterday’s marathon session to rip Pos Malaysia‘s rating data and make it persistent on spider.my made today’s job a little bit easier. The first part of the API is done: there’s a minimal interface for obtaining a single shipping quotation. Today sees the second part complete: a method for obtaining a complete set of rating data for a 3rd-party application.

If you visit part 3 of my Pos Malaysia API demo, you can now select country and shipping method (or all of one, or all of both), and the page will give you a link from which you can download a full set of rates in XML format. There is just one little wrinkle, and that is that I’m using unchanged data from Pos Malaysia. On some of their (less-popular, I’m guessing) destinations, they claim to deliver parcels up to 999kg! I store the shipping data as a table of prices against all the different weights for a shipping method, so with 0.5kg divisions, I end up with 2,000 entries for some methods. My storage method is quite efficient, so it’s not a problem on the server, but if you request All Countries + All Methods, the XML data file is several MB in size.

For an example of one of the smaller data files, see this XML file containing the data for Pos Laju Document and Parcel deliveries to the UK:

http://spider.my/pos-malaysia-rates-3/-ANY–to-United%20Kingdom.xml

See? That’s a very simple format, easily incorporated into your current application. There are just a couple of little things I’d like to do in the way of demonstrations, and then I’ll try to find someone at Pos who might be interested in providing an API to local e-businesses.

Posted in Fixed, Life, Money, software | 1 Comment »

Pos Malaysia shipping quote – part 2

The shipping quotes described here moved long ago to shipping-quote.net

Pos Laju shipping rates quotation demo 2

With a little bit of ‘reverse engineering‘, it’s possible to write a program to obtain a full set of postage rates from Pos Malaysia’s website. As far as I know, they don’t make their rates available in a form that can be easily used by 3rd parties. That’s simply not good enough in an age of e-business. One positive feature of Pos’ website must be noted: when they show a shipping quotation, they also show how they calculated it. That’s enough information to make my own copy of their rates.

I thought each step of this job would take me 1 day. I have to confess that reverse-engineering Pos’ rates, creating a persistent format for spider.my and finally the AJAX script and XML-generator to go with it took me one very large day. I started at 9am yesterday and had everything working satisfactorily at 2:30am this morning.

Try out part 2 of the shipping quotation demo: there’s no button to submit requests any more. Now that I have all of Pos Malaysia’s Pos Laju rates stored on spider.my, there’s no need for a ‘POST’ method. It would be semantically wrong, as the state of the server is no longer updated by a request. It was right for the part 1 demo, because the request loaded a page from Pos’ website, used pattern matching to extract the price, and added it to the server’s result cache.

I’m pleased with the way the new request method works out. All that a request needs to specify is a destination country and a weight. The persistent data on the server is stored as a per-country list of shipping methods. The shipping methods are retrieved for the specified country, and are tested to see if they are appropriate for the specified weight. For example, a document method will reject a 5kg request. I’ve made this feature part of the generic interface for shipping methods so it should be easy to add any number of different shipping methods (from different shipping agents), all potentially quoting on a single request.

Just in case the ‘accept/reject’ thing isn’t clear, compare the results of these two similar queries. The first is a request for a quote for shipping 0.5kg to China:

http://spider.my/pos-malaysia-shipping-quote-2/0.5-to-China.xml

(Just spotted my URLs are case-sensitive. Bummer. Experimental code mah! )and the second is a similar request, but this time for a heavier shipment – 2.5kg to China:

http://spider.my/pos-malaysia-shipping-quote-2/2.5-to-China.xml

You see two quotes for the lighter parcel because it qualifies (by weight, shape’s another matter) for Pos Laju’s Document service. The heavier parcel is too heavy for the Pos Laju Document service, so no quote is given.

It’s not an earth-shattering difference for the user over the previous instalment, but having a complete set of rating tables under my control is a big step for me. Now the ‘lag’ imposed by the fat request to Pos’ website is gone, so that means I can do things like provide a full set of up-to-date rating tables to 3rd parties for inclusion in their systems and products. That’s the next instalment!

For the interested train-spotter: the rates are ‘ripped’ with some Java code that makes shipping requests to Pos Malaysia’s website. It’s very similar to the code used in part 1, except instead of looking for the “Total Amount” value, I break down the info in the table to extract Method Name, Min Price, Weight Divisions, Increment per Division and Max Weight. It’s actually quite quick, though possible not terribly welcome by Pos Malaysia’s sysadmin. I only download the page text, not the associated images and resources, so making one request per country leads to around 200 requests being made, for something like 2-4MB of transfer. Because I’m extracting data from some fairly messy HTML (never use w3’s HTML validator? Adoi…), even the tiniest change to the page would completely break my update code.

I’ve implemented persistent storage for the shipping rates on spider.my, so there’s no need for me to run the ‘ripping’ code again, unless Pos update their rates. I might write ripping code for their local deliveries and their non-expedited shipping methods, but I don’t think it’s necessary for the sake of this demo. Their sysadmin doesn’t need to get worked up about the rip process: it’s only going to happen once!

For Pos Malaysia: Using arbitrarily-styled country names in your queries doesn’t work. Try requesting a quotation for a delivery to ‘Serbia & Montenegro’ (which hasn’t existed as a country since 2006). See? Here’s a request for 1kg to ‘Serbia & Montenegro’ to make it easy for you. What? It seems WordPress can’t handle that URL either! OK, here’s an image of the result of that request:

URLs with spaces and ampersands in them don't work

You need to encode the data you use in your URLs. Here’s a link for a quote for a parcel to ‘Serbia & Montenegro’ which does actually work on your website:

http://www.pos.com.my/V1/main2.asp?c=/v1/tariff/poslaju/EmsRatedb1.asp&negara=Serbia%20%26%20Montenegro&berat=1

I have to ask you though: Would you really accept a 999kg parcel to Eastern Europe? Hint: Look at the quote result. I’d like to see the envelope for that!

Posted in Fixed, software | 6 Comments »

Pos Malaysia e-commerce shipping quote with AJAX

spider.my is long gone: the shipping API has moved to Shipping-Quote.net

Pos Malaysia shipping quote at spider.my – part 1

Here’s a first shot at an AJAX-powered Pos Laju shipping quote. As I said in my previous article on a Pos Malaysia shipping quote API, there should ideally be two: one simple one for blogs and low-volume e-commerce sites, and one for large-volume businesses. The shipping quote demo at spider.my is very simple: just enter a weight and a destination, and click on a ‘Quote’ button to get an up-to-date quote from Pos Malaysia’s website. This is a terrible way of doing it, as spider.my retrieves a whole web-page from Pos’ website, just for one price!

You can see how slow it is if you submit the form. Part of that is nothing to do with Pos Malaysia – spider.my is on a server in the USA, so there’s massive latency between your browser (probably, if you’re in Malaysia), the spider.my server in the USA, then back to Malaysia for the quote request, then back to USA with the quote page, where it’s processed using pattern matching to extract the price, then back to your browser with the finished page.

To demonstrate the simple Shipping rate quote API, I’ve added an AJAX updater to the page. That’s a little bit of asynchronous javascript that executes in your browser. When you change the weight of the parcel, or its destination, that script will request a quote matching the new details from spider.my. To give you a feel for how an API at Pos Malaysia’s website would respond, the AJAX updater only provides responses for previously-cached quotes at spider.my. That way, there’s no need to make another long request to www.pos.com.my!

Request timings for AJAX XML shipping quote request

Have a look at the XML sent by the shipping rates demo – it’s extremely small. Comparing the timing (again from Chromium) for the transfer of the XML response from spider.my to the timing of the transfer of the quotation webpage from www.pos.com.my from yesterday. Remember that the spider.my server is in the USA, so network latency is going to be higher. Pos’s local latency isn’t great – but their server must build that page before it can transfer it. Spider.my is only building a very small (less than 100bytes!) XML message. The real benefit comes when the data is transferred. The XML message is so small, Chromium is probably defaulting to 1ms for a timing. The whopping great page from pos.com.my takes nearly a quarter of a second to transfer! For user interaction, that leaves very little time for anything else.

Notice that getting the price update via AJAX from the USA takes a total of just under 300ms, while the local method, from Pos’ website, took just under 350ms. Even with the handicap of being served on the other side of the planet, the AJAX technique is much faster. It should be clear that if the AJAX response was coming from Pos’ server, not only could the response be hundreds of times faster, it would be hundreds of times more efficient in bandwidth.

I’m not ready to try to convince Pos again to provide an API. To make it more convincing, I need to handle any quote that Pos can handle. For that, I need to acquire their rates. That sounds like hard work, since Pos don’t make their entire rates available in any other format than photocopies (I have a set – I won’t be using them). I have a cunning plan for that. Once I can provide super-efficient one-shot quotations from my server, I’m going to tackle a client-side solution for businesses that might need to (a) not depend entirely on Pos’ server being available (b) get the fastest possible quotes. That depends on my cunning plan working so that there’s an efficient way of getting an update from Pos whenever they update their rates across the board.

Posted in Broken, software, Spider.my | 4 Comments »

Pos Malaysia shipping quotes for e-commerce

Pos Malaysia

I maintain some Malaysian shipping modules for the popular e-commerce software osCommerce which allow local e-shops to include up-to-date automatic shipping quotes from Pos Malaysia (Pos Laju and Pos Air Parcel) in the order total. If you run an e-shop, being able to pass on accurate shipping costs to your customers – compared with ‘free’ or fixed-price shipping – means you can give your customers better prices than your competitors. If you can offer your customer a choice of shipping options, this also allows them to save money or expedite their delivery. Doing all of this automatically means you don’t have to attend to your online business – it completes orders and collects money for you, and tells you when to pack deliveries!

Pos Malaysia have a shipping quotation service at their website. It’s obviously intended for human visitors, as it also has advertising, branding and links to other services on it. I use this web page in my osCommerce module. In order to give an automatic, up-to-date quote, an osCommerce shop using my modules must send a request to Pos Malaysia’s website, read the entire page, and use pattern matching to extract the shipping price.

The current technique is very wasteful of resources, the page that must be searched for the shipping price is 13 to 15KB – for just a few characters! The technique is also horrendously sensitive. If Pos Malaysia change the appearance of the page, the pattern-matching technique can’t work and the shipping module fails to return a quote, which will usually mean that the osCommerce e-shop won’t be able to complete the order. The Internet being the ‘instant’ experience that it is, the sales opportunity is probably lost.

One further annoyance is that the shipping module requesting one or two (for different shipping methods) quotes from Pos Malaysia’s website introduces a considerable delay in the user’s experience of the e-shop. When a customer goes to the checkout page on which the shipping cost appears, the server starts to build the page, sends a request to Pos Malaysia’s website, waits for Pos Malaysia’s webserver to build its response, reads the (large) response, performs pattern matching on it (repeating the last 4 steps for each additional shipping method) and finally sends the page to your customer’s browser. Several-second delays are not uncommon.

Pos Malaysia quotation page timings

Take a look at these two resource timing graphs from the Chromium browser. If you haven’t used Chromium yet, and you’re a developer, there are some great tools in there – and it’s shaping up to be a very competent browser too. The first image is from Pos Malaysia’s website, a shipping quote for a parcel (surface and air quotes on the same page). The blue bar is for the page itself – the other bars are for resources the page needs to be rendered. We only need the page, as we’re not interested in images, stylesheets and scripts, only in the text of the shipping amounts. There’s a redirect (a 302 Found) which further slows things down, and then a 104ms wait (perhaps the page is being built by Pos’s server?) and then nearly a quarter of a second while the ~14KB of the page text is transferred.

AJAX load time from spider.my's currency conversion page

Compare that with this image, which shows the resources being loaded from spider.my’s currency conversion page. The line to note is that for the AJAX update of the exchange rate. The latency is horrible, as spider.my is hosted in the USA, but the time to load the AJAX update is only 1ms – hundreds of times faster than Pos’ shipping quote page. That’s understandable because the reply from spider.my is only about 100bytes long!

What I think Pos Malaysia should do is to provide an API to Malaysian e-business so that their postal services can be more efficiently and reliably built into local business systems. I think they should offer 2 APIs – one a simple AJAX responder for use on blog pages and low-volume shops, and the other a shipping rates updater for larger-volume businesses. The simple responder would take the place of the current osCommerce modules and should be able to easily serve thousands of shipping quote requests per second on any old PC that Pos has lying around. I envisage some heavy client-side caching for the high-volume API, so apart from days on which Pos update their rates, the server should see very little traffic at all for a more complex solution suitable for large businesses.

It was obvious to me when I first wrote the shipping modules a few years ago that they were inefficient and fragile, so I emailed and wrote to Pos Malaysia, even met their regional management to try to get someone to improve their end of the shipping quote system. A few years have passed and nothing has happened, so I’m going to write it myself and try to get someone from Pos to have a look at it and hopefully host it themselves. It’s not difficult and it’ll make a nice test of Spinneret. After all, when I finally get round to making Spinneret available, it will be FOSS buatan Malaysia!

If Pos wants the system, I’d be quite happy to do all the necessary work to integrate it into Pos Malaysia’s current IT systems FOR FREE. I don’t want to maintain the osCommerce modules in their current condition any more, and I think Pos exposing a decent API to local e-businesses would make a big difference to the ability of Malaysian businesses to compete online. That would make my life easier, and hopefully earn me a few free cups of tea at the kopitiam.

Posted in Broken, Life, Money, Philosophy, software | 9 Comments »

AJAX-enabled exchange rates

I finally got round to adding client-side scripting to the Currency Conversion page at spider.my! AJAX is ‘Asynchronous Java And XML’. Basically, it allows you to embed some functionality in your web page so that your page can communicate with your server after it has been rendered in your visitor’s we browser, but without reloading the entire page.

I want to use AJAX for some other projects, so I did some research this morning and used the Currency Conversion page to test it out on. I previously wrote a plain text exchange rate query facility at spider.my which returns only the rate and nothing else. It’s not entirely straightforward to retrieve plain text data using javascript, as far as I can tell. The XMLHttpRequest object that javascript uses to make web requests will only work if it receives a response with a valid XML type, so my text/plain initial effort just isn’t good enough!

A little bit of AJAX on my Currency Conversion page!

If you visit spider.my’s currency conversion page now, you’ll see that the current exchange rate for the two selected currencies is displayed next to the ‘Convert’ button (see image above). Using ‘onchange’ events in the select elements, the page now invokes a javascript function that requests the latest exchange rate from spider.my and updates the displayed rate. That’s a very simple example, but similar code could easily give you an exchange rate widget on your own web pages.

What happens when the selected currencies are changed is that the javascript in the page sends a request to spider.my for the appropriate exchange rate. For example, http://localhost:22791/exchange-rate/GBP-USD.xml is the URL requested for the UK Sterling / US Dollar rate. It’s a very simple response – here it is:

<exchangerate rate="1.5392" source="http://www.ecb.int/stats/eurofxref/eurofxref.zip (Date=19 February 2010)">

All that the javascript has to do is to extract the rate part of the XML response and update the div that holds the visible exchange rate:

var rate = xmlResponse.getElementsByTagName(
"exchangerate")[0].getAttribute("rate");
divRate.innerHTML = '(@'+rate+')';

Easy huh?

I’ve got a better example coming up in the next day or two – stay tuned!

Posted in Fixed, Money, software, Spider.my | No Comments »