{"id":391,"date":"2009-06-26T13:34:28","date_gmt":"2009-06-26T05:34:28","guid":{"rendered":"http:\/\/blog.lolyco.com\/sean\/?p=391"},"modified":"2009-06-26T17:37:24","modified_gmt":"2009-06-26T09:37:24","slug":"la-times-reports-michael-jackson-dead-head-missing","status":"publish","type":"post","link":"https:\/\/blog.lolyco.com\/sean\/2009\/06\/26\/la-times-reports-michael-jackson-dead-head-missing\/","title":{"rendered":"LA Times reports Michael Jackson dead, HEAD missing"},"content":{"rendered":"
\"Michael<\/a>

Michael Jackson - from Wikipedia<\/p><\/div>\n

I was manually adding some reports of Michael Jackson’s death<\/a> to the crawl queue at spider.my<\/a> this morning, when I noticed that one of the machines doing indexing had choked on a page. It wasn’t long ago that I added some code to detect <meta> elements<\/a> in pages being used to specify the character set for the page. The regular expression that I’d come up with to extract the\u00a0 content-type from the http-equiv header field just seemed to be looping, using 100% CPU.<\/p>\n

I wrote some code that was a little more pedestrian, that I hoped would be more robust. It failed to find the content-type that I could see in the page source, but at least it didn’t hang! It took me a little while to work out that it was caused by the page having no <head> element<\/a>. I’m relatively new to developing for the Web, so it’s sometimes surprising it all works so well, when you see the quality of the data!<\/p>\n

\"LA<\/a>

LA Times missing HEAD element<\/p><\/div>\n

Just for completeness, here’s the cached copy of the LA Times page from spider.my’s page cache<\/a>. Have a look at the page source. The LA Times page begins about 6 lines down with an HTML element. See? No HEAD.<\/p>\n","protected":false},"excerpt":{"rendered":"

I was manually adding some reports of Michael Jackson’s death to the crawl queue at spider.my this morning, when I noticed that one of the machines doing indexing had choked on a page. It wasn’t long ago that I added some code to detect <meta> elements in pages being used to specify the character set […]<\/p>\n","protected":false},"author":3,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[30,14,40,35],"tags":[23,107,34,112,105],"class_list":["post-391","post","type-post","status-publish","format-standard","hentry","category-breaktime","category-broken","category-life","category-spidermy","tag-blogging","tag-broken","tag-network","tag-search","tag-software"],"_links":{"self":[{"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/posts\/391","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/users\/3"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/comments?post=391"}],"version-history":[{"count":10,"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/posts\/391\/revisions"}],"predecessor-version":[{"id":408,"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/posts\/391\/revisions\/408"}],"wp:attachment":[{"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/media?parent=391"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/categories?post=391"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.lolyco.com\/sean\/wp-json\/wp\/v2\/tags?post=391"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}