One worrying thing about DNS is that the entire domain like wikipedia.org or facebook.com seems to map to a single IP address. Fortunately, there are ways of mitigating the bottleneck: * **Round-robin DNS** is a solution where the DNS lookup returns multiple IP addresses, rather than just one. For example, facebook.com actually maps to four IP addresses. * **Load-balancer** is the piece of hardware that listens on a particular IP address and forwards the requests to other servers. Major sites will typically use expensive high-performance load balancers. * **Geographic DNS** improves scalability by mapping a domain name to different IP addresses, depending on the client’s geographic location. This is great for hosting static content so that different servers don’t have to update shared state. * **Anycast** is a routing technique where a single IP address maps to multiple physical servers. Unfortunately, anycast does not fit well with TCP and is rarely used in that scenario. Most of the DNS servers themselves use anycast to achieve high availability and low latency of the DNS lookups. Users of an anycast service (DNS is an excellent example) will always connect to the 'closest' (from a routing protocol perspective) DNS server. This reduces latency, as well as providing a level of load-balancing (assuming that your consumers are evenly distributed around your network). ## Opening of a socket + TLS handshake * Once the browser receives the IP address of the destination server, it takes that and the given port number from the URL (the HTTP protocol defaults to port 80, and HTTPS to port 443), and makes a call to the system library function named socket and requests a [TCP](https://github.com/vasanthk/how-web-works/blob/master/http://www.webopedia.com/TERM/T/TCP.html) [socket](https://github.com/vasanthk/how-web-works/blob/master/http://www.webopedia.com/TERM/S/socket.html) stream. * The client computer sends a ClientHello message to the server with its TLS version, list of cipher algorithms and compression methods available. * The server replies with a ServerHello message to the client with the TLS version, selected cipher, selected compression methods and the server's public certificate signed by a CA (Certificate Authority). The certificate contains a public key that will be used by the client to encrypt the rest of the handshake until a symmetric key can be agreed upon. * The client verifies the server digital certificate against its list of trusted CAs. If trust can be established based on the CA, the client generates a string of pseudo-random bytes and encrypts this with the server's public key. These random bytes can be used to determine the symmetric key. * The server decrypts the random bytes using its private key and uses these bytes to generate its own copy of the symmetric master key. * The client sends a Finished message to the server, encrypting a hash of the transmission up to this point with the symmetric key. * The server generates its own hash, and then decrypts the client-sent hash to verify that it matches. If it does, it sends its own Finished message to the client, also encrypted with the symmetric key. * From now on the TLS session transmits the application (HTTP) data encrypted with the agreed symmetric key. ## HTTP protocol You can be pretty sure that dynamic sites such as Facebook/Gmail will not be served from the browser cache because dynamic pages expire either very quickly or immediately (expiry date set to past). If the web browser used was written by Google, instead of sending an HTTP request to retrieve the page, it will send a request to try and negotiate with the server an "upgrade" from HTTP to the SPDY protocol. Note that SPDY is being deprecated in favor of HTTP/2 in latest versions of Chrome. ```txt GET http://www.google.com/ HTTP/1.1 Accept: application/x-ms-application, image/jpeg, application/xaml+xml, [...] User-Agent: Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; [...] Accept-Encoding: gzip, deflate Connection: Keep-Alive Host: google.com Cookie: datr=1265876274-[...]; locale=en_US; lsd=WW[...]; c_user=2101[...] ``` The GET request names the URL to fetch: “[http://www.google.com/](https://github.com/vasanthk/how-web-works/blob/master/http://www.google.com/)”. The browser identifies itself (User-Agent header), and states what types of responses it will accept (Accept and Accept-Encoding headers). The Connection header asks the server to keep the TCP connection open for further requests. The request also contains the cookies that the browser has for this domain. As you probably already know, cookies are key-value pairs that track the state of a web site in between different page requests. And so the cookies store the name of the logged-in user, a secret number that was assigned to the user by the server, some of user’s settings, etc. The cookies will be stored in a text file on the client, and sent to the server with every request. HTTP/1.1 defines the "close" connection option for the sender to signal that the connection will be closed after completion of the response. For example, Connection: close. After sending the request and headers, the web browser sends a single blank newline to the server indicating that the content of the request is done. The server responds with a response code denoting the status of the request and responds with a response of the form: **200 OK [response headers]** Followed by a single newline, and then sends a payload of the HTML content of www.google.com. The server may then either close the connection, or if headers sent by the client requested it, keep the connection open to be reused for further requests. If the HTTP headers sent by the web browser included sufficient information for the web server to determine if the version of the file cached by the web browser has been unmodified since the last retrieval (ie. if the web browser included an ETag header), it may instead respond with a request of the form: **304 Not Modified [response headers]** and no payload, and the web browser instead retrieves the HTML from its cache. After parsing the HTML, the web browser (and server) repeats this process for every resource (image, CSS, favicon.ico, etc) referenced by the HTML page, except instead of GET / HTTP/1.1 the request will be **GET /$(URL relative to www.google.com) HTTP/1.1.** If the HTML referenced a resource on a different domain than www.google.com, the web browser goes back to the steps involved in resolving the other domain, and follows all steps up to this point for that domain. The Host header in the request will be set to the appropriate server name instead of google.com. **Gotcha:** * The trailing slash in the URL “[http://facebook.com/](https://github.com/vasanthk/how-web-works/blob/master/http://facebook.com/)” is important. In this case, the browser can safely add the slash. For URLs of the form http://example.com/folderOrFile, the browser cannot automatically add a slash, because it is not clear whether folderOrFile is a folder or a file. In such cases, the browser will visit the URL without the slash, and the server will respond with a redirect, resulting in an unnecessary roundtrip. * The server might respond with a 301 Moved Permanently response to tell the browser to go to “[http://www.google.com/](https://github.com/vasanthk/how-web-works/blob/master/http://www.google.com/)” instead of “[http://google.com/](https://github.com/vasanthk/how-web-works/blob/master/http://google.com/)”. There are interesting reasons why the server insists on the redirect instead of immediately responding with the web page that the user wants to see. One reason has to do with search engine rankings. See, if there are two URLs for the same page, say http://www.vasanth.com/ and http://vasanth.com/, search engines may consider them to be two different sites, each with fewer incoming links and thus a lower ranking. Search engines understand permanent redirects (301), and will combine the incoming links from both sources into a single ranking. Also, multiple URLs for the same content are not cache-friendly. When a piece of content has multiple names, it will potentially appear multiple times in caches. **Note:** HTTP response starts with the returned status code from the server. Following is a very brief summary of what a status code denotes: * 1xx indicates an informational message only * 2xx indicates success of some kind * 3xx redirects the client to another URL * 4xx indicates an error on the client's part * 5xx indicates an error on the server's part ## HTTP Server Request Handle The HTTPD (HTTP Daemon) server is the one handling the requests/responses on the server side. The most common HTTPD servers are Apache or nginx for Linux and IIS for Windows. * The HTTPD (HTTP Daemon) receives the request. * The server breaks down the request to the following parameters: * HTTP Request Method (either GET, POST, HEAD, PUT and DELETE). In the case of a URL entered directly into the address bar, this will be GET. * Domain, in this case - google.com. * Requested path/page, in this case - / (as no specific path/page was requested, / is the default path). * The server verifies that there is a Virtual Host configured on the server that corresponds with google.com. * The server verifies that google.com can accept GET requests. * The server verifies that the client is allowed to use this method (by IP, authentication, etc.). * If the server has a rewrite module installed (like mod_rewrite for Apache or URL Rewrite for IIS), it tries to match the request against one of the configured rules. If a matching rule is found, the server uses that rule to rewrite the request. * The server goes to pull the content that corresponds with the request, in our case it will fall back to the index file, as "/" is the main file (some cases can override this, but this is the most common method). * The server parses the file according to the request handler. A request handler is a program (in ASP.NET, PHP, Ruby, …) that reads the request and generates the HTML for the response. If Google is running on PHP, the server uses PHP to interpret the index file, and streams the output to the client. Note: One interesting difficulty that every dynamic website faces is how to store data. Smaller sites will often have a single SQL database to store their data, but sites that store a large amount of data and/or have many visitors have to find a way to split the database across multiple machines. Solutions include sharding (splitting up a table across multiple databases based on the primary key), replication, and usage of simplified databases with weakened consistency semantics. ## Server Response Here is the response that the server generated and sent back: ```txt HTTP/1.1 200 OK Cache-Control: private, no-store, no-cache, must-revalidate, post-check=0, pre-check=0 Expires: Sat, 01 Jan 2000 00:00:00 GMT P3P: CP="DSP LAW" Pragma: no-cache Content-Encoding: gzip Content-Type: text/html; charset=utf-8 X-Cnection: close Transfer-Encoding: chunked Date: Fri, 12 Feb 2010 09:05:55 GMT 2b3 Tn@[...] ``` The entire response is 36 kB, the bulk of them in the byte blob at the end that I trimmed. The **Content-Encoding** header tells the browser that the response body is compressed using the gzip algorithm. After decompressing the blob, you’ll see the HTML you’d expect: ```html ... ``` Notice the header that sets Content-Type to text/html. The header instructs the browser to render the response content as HTML, instead of say downloading it as a file. The browser will use the header to decide how to interpret the response, but will consider other factors as well, such as the extension of the URL. ## Behind the scenes of the Browser Once the server supplies the resources (HTML, CSS, JS, images, etc.) to the browser it undergoes the below process: * Parsing - HTML, CSS, JS * Rendering - Construct DOM Tree → Render Tree → Layout of Render Tree → Painting the render tree ## The browser's high level structure 1. **User Interface:** Includes the address bar, back/forward button, bookmarking menu, etc. Every part of the browser display except the window where you see the requested page. 2. **Browser Engine:** [Marshals](https://github.com/vasanthk/how-web-works/blob/master/http://stackoverflow.com/a/5600887/1672655) actions between the UI and the rendering engine. 3. **Rendering Engine:** Responsible for displaying requested content. For eg. the rendering engine parses HTML and CSS, and displays the parsed content on the screen. 4. **Networking:** For network calls such as HTTP requests, using different implementations for different platforms (behind a platform-independent interface). 5. **UI Backend:** Used for drawing basic widgets like combo boxes and windows. This backend exposes a generic interface that is not platform specific. Underneath it uses operating system user interface methods. 6. **JavaScript Engine:** Interpreter used to parse and execute JavaScript code. 7. **Data Storage:** This is a persistence layer. The browser may need to save data locally, such as cookies. Browsers also support storage mechanisms such as [localStorage](https://github.com/vasanthk/how-web-works/blob/master/https://developer.mozilla.org/en-US/docs/Web/API/Window/localStorage), [IndexedDB](https://github.com/vasanthk/how-web-works/blob/master/https://developer.mozilla.org/en-US/docs/Web/API/IndexedDB_API/Using_IndexedDB) and [FileSystem](https://github.com/vasanthk/how-web-works/blob/master/https://developer.chrome.com/apps/fileSystem).
Let’s start, with the simplest possi ... ...