The files:

• Socket.php (highlighted; plaintext)
• webpage.php (highlighted; plaintext)

What good is a full-on Socket class and function, you may ask, when you can simply run file_get_contents with allow_url_fopen turned on? Well, consider that you want to send specific headers to the server, to POST data for example. Also, there are other uses for a Socket class than webpage downloading; I use this Socket implementation in a PHP-based IRC bot that I run (by the name of ecode on the Freenode network). You can use the socket to interact with many different servers.

Socket

The Socket class essentially encapsulates PHP’s socket_* functions, making it easy and straightforward to make a TCP connection with a server. The socket is given a timeout period in microseconds; while I’m not 100% sure what this means, I do know that it emulates a sort of “busy wait” while waiting for content, allowing applications to do processing while waiting for data to come in through the pipe.

To use the socket, include the Socket.php file and create a variable to hold a new Socket.

require_once("Socket.php");

$s = new Socket(1000);

The Socket will automatically set up the outgoing TCP socket for usage. The parameter passed to the new socket (1000 in this case) is the microsecond timeout — this can be left blank if you wish.

Once created, the socket can connect to any service through the connect() function. The function takes two arguments: host and port. Both parameters are optional to allow reuse of the connection; if called without parameters, the Socket will establish a new connection using a host and port from the last connection. The host is the address you would ping to see if the servers are up — for example, either google.com or 74.125.67.100, the Socket doesn’t care which as long as it’s a server it can connect to. The port is optional: if you specify a port number, that will be used; if no port number is specified, it attempts to extract a port number from the host (for example, if you used google.com:80); failing that, it will use the last port number specified (default of 80).

Once connected, you can use the send() and get() functions to — you guessed it — send() and get() data. The send function takes a string parameter, the message to be sent, sends the message through socket_write, and returns the value returned by socket_write (the number of bytes successfully sent through the socket). The get function takes an array parameter to specify fetching options: line (boolean) and length (integer). If line is set to true, then get will fetch one character at a time until a newline (\n) is encountered and return the line along with any trailing carriage returns (\r) and newlines (\n). If length is specified as greater than 0, get will fetch one character at a time until the specified number of characters have been read. If neither is specified, the function will read and return the next 512 characters. Of course, all cases will terminate if the end of data has been reached.

As a quick example, this portion uses the previously created socket to download the HTTP response headers from google.com’s homepage:

if ($s->connect("www.google.com", 80)) {
    $get;
    $s->setVerbose(false);
    $s->send("GET / HTTP/1.1");
    $s->send("Host: www.google.com");
    $s->send("");

    $response = "";
    while (strpos($response, "\r\n\r\n")===false) {
        $response .= $s->get(array("line"=>true));
    }
}

The Socket supports logging and verbosity functions. The setLog function takes a boolean (true/false for logging) and a string (filename to log to). The setVerbose function takes only a boolean (true/false — default false — for printing out extra information to the console).

Finally, when you’re finished with the connection, you can call the disconnect() function to cleanly sever the connection.

webpage()

The webpage function takes three parameters: a url, a headers array, and a “headers only” flag. The url is simple enough; this is the full url that you want to fetch, including both the host name and the file path (and, optionally, a port number after the hostname). The headers array goes two ways; you can pass in headers to send and you’ll get back any headers received. The headers only flag specifies whether or not to fetch the content of the page; if not, it saves the time of downloading that content, and leaves you with only the response headers.

The headers parameter is two-way, as I mentioned. You can pass in an associative array of header-name => header-value pairs, which will be formatted properly and sent to the server along with the HTTP request.

$h = array(
    "User-Agent" => "PHP-webscraper/1.0 php/5",
    "Authorization" => "Basic ".base64_encode("username:password")
);
$page = webpage("http://twitter.com/statuses/mentions.json", $h);

The above will send the User-Agent, for identification, and Authorization, to login, headers to the Twitter API, fetching any tweets that mention the user name “username” (using the password “password”). The response will be in JSON, and will require extra processing, but this will get you the data. For more information on HTTP headers, check out the HTTP 1.1 RFC, section 14.

Once the page is received from the server, the headers array you passed in will be overwritten by the server’s response headers, which you can then examine with external code. With this method, you can also pass in a simple empty array, which will then be populated with the response headers.

The third parameter to the webpage function is simple enough. Pass true if you want to receive only the server’s response headers, cutting off the response once those have been received; a blank string will be returned from the function. Pass false (or no third parameter) if you wish to receive the entire page, the contents of which will be returned as a string.

With these two files (and, of course, the PHP constructs they define), you can download any webpage you would have access to with your browser, provided you are able to use sockets on your server. Need to scrape some data from a page? Download that page and apply some regexes to the content. HTTP Auth required? No problem, send an extra header with your creds. The function could also be modified to POST data to a page, or even download and save images and other media; however, those functions will be left as an exercise for the reader.

Image: Warped by ~eternicode (me).

My PHP script can take any jpg, png, or static gif image (supplied through the “src” GET variable) and apply rounded corners to them. The space left by the corners is transparent, and so blends in with any background you may have applied to your content section.

Using GET variables, you can control various aspects of the output image. Using “rad” or “radius”, you can increase and decrease the size of the corners. The default value for the radius is 20, which I’ve found to be reasonable for most images.

Image: Let It Shine by ~eternicode.

Using “width” or “height”, you can scale down the image to meet your needs — the specified width or height will be taken as the maximum width or height, in pixels, and the aspect ratio will be maintained. If both height and width are given, the smallest result of the two possible outputs will be used.

Image: Minty by ~eternicode.

There is also a “squarification” option, which will reduce either the width or height (whichever is largest) to produce a squared image. If “squared” is specified as true, radius and scaling will take effect as normal, using the new squared aspect ratio in place of the old ratio.

Image: Convergence by ~eternicode.

The final option, which was kind of an afterthought, is the “format” option — you can choose either “png” or “gif” as the output format. I only added this because with png format, in IE6, the method I used would produce magenta-colored corners, rather than transparent colors. Supplying a gif output fixes this problem for IE6, though the png option is generally preferred.

The source of the script is very well documented, but feel free to ask questions if I missed something.
The example directory.
Active script using a local image, for your experimentation (any other image can be specified through “src”).
Highlighted script source.
Plain text script source.

I’ve been developing an IRC bot as a side project, and three of its functions require the use of web services. It can shorten urls using http://is.gd/, search google, and validate web pages using the W3C’s HTML and CSS validators. Fortunately, and to my great elation, all four services have APIs available.

Is.gd’s API is straightforward and easy to use: call the page http://is.gd/api.php?longurl=http://really-long-url-goes-here.com , and is.gd will return the shortened url (or an error message) as the only text. You’ll have to parse out the HTTP headers, of course, but that’s the easy part.

The W3C’s APIs are just as easy to use. For HTML validation, fetch the page http://validator.w3.org/check?uri=http://webpage-to-check.com . This is the same page you would navigate to in a browser, and returns a full-on HTML page with the results. However, it also puts custom headers in the HTTP response. To get the results, you have to parse out the values of the X-W3C-Validator-Status, X-W3C-Validator-Errors, and X-W3C-Validator-Warnings headers. For CSS, the same info comes in the headers of a call to http://jigsaw.w3.org/css-validator/validator?uri=http://webpage-to-check.com . In the case of the CSS validator, though, the X-W3C-Validator-Warnings header is not returned.

Google’s search API is a little more involved. Not only because it uses JSON, but because it’s a secret. Google’s AJAX Search package is no secret of course, but the API calls that it makes are hidden away. Fortunately, Google set up an experimental search site called SearchMash.com, along with an API of its own. Simply fetch the page http://www.searchmash.com/results/search+terms+go+here , and searchmash will return a JSON string which can then be parsed out to obtain the google search results, alternate spelling suggestions, estimated number of results, the works. Even better, you can append “?n=X” to the end to limit the search to X results. Using ?n=1 returns the top result only.

These APIs all save processing power that would otherwise be spent parsing out possibly badly-formed HTML from HTTP responses.

Update: The SearchMash API has “gone the way of the dinosaur.” If you’re looking for something similar to what SM offered, check out Google’s RESTful search API: AJAX API for non-JS environments. It gives the same results in basically the same format — I was able to switch over with very few adjustments to my own code.

Aside: If you would like to check the bot out, it currently resides on #ecode (and, probably, other channels) on the freenode network. It goes by the name of “ecode”.

The features touted on Google’s promotional website include searchable history / smart address bar, a “portal” page as the default “new tab” view, individual tab processes, dynamic tabs (ala Safari 3.1), and “incognito mode” (is that the Red Hat mascot I see in the corner?). Underneath the hood, I’ve heard it uses a combination of opensource code from Safari’s Webkit and Mozilla’s Gecko engines, so it should be interesting from a web development perspective.

All in all, from what I’ve read others saying, it looks like Google took some opensource browser code, implemented a few nifty FireFox addons as default features, and slapped a shiny Google interface on it. My test run for Chrome won’t happen until it’s available for linux, but if you’re itching to see it, check out LifeHacker’s first look at Chrome (using the Windows release).

I found this trick over at labnol.org. Simply change the about:config key keyword.url to http://www.google.com/search?q= and make sure keyword.enabled is set to true.

The default value of keyword.url is http://www.google.com/search?ie=UTF-8&oe=UTF-8&sourceid=navclient&gfns=1&q=. Most likely, OpenDNS has this particular string programmed into their system, to catch navbar searches from FireFox (and IE) with default settings. Change the setting, and you can skirt the redirect .

In addition to gmail-style email hosting for your domain (and ThunderBird and Outlook hookup instructions to boot), GAppsFYD, as LifeHacker calls it, sports Docs, Chat, and Calendar, all private to specified users of your domain. It also includes Web Pages, giving you space to host static web pages (but you have your own server for that — right?); Sites, a collaborative intranet-style hosting space; and a Start Page, a central page from which all domain users can access the activated services. Best of all, you can activate/deactivate whichever services you choose, allowing only what you need.

Each domain you set up with Google Apps can accomodate up to 100 users under the free version. If the email service is activated, each user gets their own inbox with up to 6GB of storage. You can also setup multiple addresses per user (with one shared inbox amongst them), and a “catchall” email address, where mail sent to a nonexistent user will end up. And, of course, you get all the standard advantages of gmail.

To host your own website, you need a computer with an HTTP server, and a decent (not dialup) always-on internet connection. A spare computer (used as a “dedicated server”) with, say, a 333MHz processor and 256MB memory is plenty sufficient to serve a low-traffic website, especially if you put an operating system with low resource requirements on it. If you have a “modern” home computer (usually 1GHz+ and 512MB+), you could even install the web server on that machine, provided you always leave the machine running. Hard drive requirements are very small as well; if you are using a dedicated server, you could use a 10GB hard drive and have more than enough space for your website.

Once you have a machine designated as your server, you need to set it up. If you have a dedicated server, and you plan on using the existing operating system, make sure that no unnecessary programs start running at startup. Once the operating system is prepped (or installed, as the case may be; your home computer can most likely be run normally), you need to install the web server.

In terms of web servers, Apache is the most widely used. And it’s free. And it’s bundled with Ubuntu server edition, should you choose to add it during installation. These reasons and more make Apache a good web server to start with (and, ultimately, stick with). Once you have Apache installed, it needs to be setup to know what files to serve as what website. Check out the Apache Virtual Host documentation for details on setting up individual websites.

Once you have your domain bought, you need an IP to point it at. For this series, I’m going to assume you’re using your home IP, assigned by your ISP. Usually, an ISP will assign its users dynamic IPs, meaning that whenever you connect to your ISP, you get a different IP. Some ISPs offer packages with static IPs, giving you an IP that will always point at your computer. While a static IP would make hosting easier, hosting with a dynamic IP is completely doable.

After getting a domain name and an IP, you need to link them together. This is done through DNS, the Domain Name System. I gave a basic explanation of DNS in my last article, “Is your DNS cache poisoned?“. To tell the system to point your domain at your IP, you need to specify your domain’s nameservers through your registrar, and then set up those nameservers with the appropriate information. I use FreeDNS’s nameservers for my domains. FreeDNS requires registration to use their service, but there is no mandatory fee. They do have a premium membership option, but the free membership provides all of the basics for managing a domain.

If you have a dynamic IP address, you’ll need some way to alert your Dynamic DNS provider (FreeDNS being an example) when your IP has changed. FreeDNS has a list of programs that are capable of updating that service, and other Dynamic DNS services often provide their own clients. I run a linux server, and I’ve settled on using inadyn (manual page, download). Inadyn is a command-line program available for Windows and Linux (and, presumably, is capable of running on Mac); it takes a number of command line options and, once called, can run in the background as a service, updating the appropriate Dynamic DNS service when a different IP is detected.

Next article, I’ll talk about setting up a server. In the meantime, what domain registrar and DNS solutions do you use, if any?

Don’t get me wrong, Blogger was great to use as I began my blogging journey. A freely-hosted blog is probably the best way to test the waters, so you don’t spend $10 on a domain and who knows how much on hosting for a blog that’ll be dead in a few months. It’s just that hacking functionality into Blogger, when that functionality can easily be gained through a WordPress plugin, gets old after a while. And besides, I had to do something with my new server .

The main problem with moving is that free services like Blogger give you very, very little control over the server. The best way to maintain whatever incoming links and “Google juice” you’ve built up, assuming you’re moving to a new domain, is to set up a 301 Permanent Redirect to the new blog. And, well, Blogger doesn’t like that idea .

The next best thing, though nowhere near as desirable, is to setup various client-side redirects, and hope that at least one of them works. This, of course, pretty much leaves behind whatever link juice you may have, but it’s better than users coming to a seemingly dead blog without some way to find the regularly-updated version.

Here’s what I did: throughout the course of using blogspot, I’ve been using “post pages”, which is essentially a permalink. You can setup self-hosted WordPress to use the .htaccess file to redirect users from the old URL version (with year, month, and trailing .html) to the new version used by WordPress. For details, see BlogBloke.com’s article (currently under construction, see the google cached version). To test if your implementation works (after importing all your Blogger posts), pull up an article on the blogspot blog, replace the blog.blogspot.com in the url with your WordPress blog’s base address, and hit enter. You should be redirected to that same article on your WP blog.

As for performing the actual redirect, there are several ways to go, including javascript, meta redirects, and plain links within your pages. I set up a simple javascript redirect that goes in the head of the blog template. This takes the current URL, replaces the blogspot part with the new blog’s URL, and sends the user there. Instant client-side redirect, provided the user has javascript enabled.

<script type='text/javascript'>
// < ![CDATA[

document.location.href=document.location.href.replace(/oldblog.blogspot.com/, "newblog.com");

// ]]>
</script>

I put this directly under the title tags, so it would be executed as soon as possible. Then, it occurred to me to move the Google Analytics tracker code above that, so that I would still receive stats from that site.
Note: The CDATA tags are from the fact that New Blogger templates use XML. Occasionally, I would get issues with using my javascript without CDATA tags surrounding the code. These may or may not be necessary.

In addition to the javascript redirect, I put a Text widget above the main blog content, alerting users to the new location, in case they’ve disabled javascript.

I wouldn’t recommend using a meta tag redirect, unless you can somehow incorporate the current article’s URL into it. You don’t want to auto-redirect the user from the article they want to read to the homepage of your new blog, even if you give them a 6-second warning. It’s just downright inconsiderate. If you can’t “seamlessly” redirect them, tell them how they can get to the new place, and leave it at that.

Once you’ve got the web-based redirects working, you need to take care of your RSS readers. If you used Blogger (which, since you’re reading this, I assume you have), check out their RSS redirect service. If you’re using FeedBurner, it’s almost as simple. FeedBurner has acquired a FeedBurner WordPress plugin called FeedSmith. Simply install the plugin, re-burn your FeedBurner feed (”Edit feed details” from your feed’s dashboard, and enter your WordPress feed, which should be at http://blog.com/feed/), and enter your FeedBurner URL in the appropriate field in WordPress’s Settings → FeedBurner page. Your existing readers shouldn’t notice any change (they may receive duplicate entries of your recent articles), and any future subscribers will be redirected to your FeedBurner feed.

First, some background on DNS; feel free to skip this paragraph if you know what it is . “Domain Name System” is a distributed system for translating web addresses (Google.com) to IP addresses (64.233.187.99). You type “Google.com” into your browser, the browser (through the computer) sends that address to the DNS, the DNS looks up the address and retrieves the corresponding IP, the DNS sends that IP to your computer, and, finally, the browser can use that IP to contact the appropriate website’s server to fetch the content. Each domain has at least one “authoritative” DNS server, which holds the master information for that domain. Other servers in the DNS system contact the authoritative server, and cache the IP address in their own tables. This way, if an authoritative server goes down, less-reliable-yet-still-useful results can be obtained from secondary servers. (This is the way I understand it, please correct me if I’m wrong). OK, moving on.

I was doing some domain name management on FreeDNS a few days ago, when, seemingly out of nowhere, I was getting “error loading page” for that site. I tried a few other sites to make sure my internet wasn’t down; I could access Google, but not eBay, with the same situation for a number of other sites. Very strange. Having recently been educating myself about DNS (that happens when you start looking into the details of dynamic DNS ), I immediately assumed that my ISP’s (AT&T’s) DNS servers were on the fritz. Which, to my knowledge, has never happened before.

So, naturally, I went straight to google with a query: “att dns servers broke”. Not the best grammar in the world, but it worked . Nothing especially recent popped up under Web results, so I checked out the Blog results, and found this: Your DNS Server is Broken, and Can’t Be Fixed. Naturally, that site was one of the sites that my working DNS server(s) couldn’t find, so I had to call up a google cached version. ‘Twas a scary article.

Basically, there is an inherent flaw in the very design of the DNS system. This flaw allows malicious entities with knowledge of this flaw to poison the DNS cache. This means that they can update the system’s cache to point a domain name at their IP address, which, in theory, could be a perfect copy of the original website. This would allow them to redirect major bank websites, for example, to their perfect copies designed to steal your account information. And you, of course, would never know the difference, because the URL bar still says “mybank.com”.

Fortunately, the discoverer of the flaw, Dan Kaminsky of Doxpara.com, was a security researcher. And, fortunately, he kept this monumental news quiet and got together with some other security guys and programmers, as well as some of the big names in technology, to work up a workaround. The flaw isn’t fixable, but we can make it harder to exploit.

After reading this article, which didn’t exactly answer my first question (”Are AT&T’s DNS servers down, broken, or worse?”), I headed to the news search. The most recent article there was on Forbes.com: Hackable Broadband Left Unpatched. This article, besides describing the flaw, detailed some major ISPs who hadn’t updated their servers as of the day the article was published. To my disdain, AT&T was among them. This may explain the DNS hiccups, though, if implementing this workaround to the main servers is at all a major undertaking.

While sooner is better, the major ISPs (or, really, anyone who runs a DNS server) have until August 6th to update their systems. That’s when Kaminsky will be discussing all the gritty details of the exploit in his talk at the Black Hat Briefings. In other words? DNS D-Day for anyone who hasn’t patched up yet.