Bug (fix) in Firebug's Net Panel

Firebug is awesome, but early on I noticed a flaw in the resources displayed in Net Panel. It didn't jibe with what I saw in my packet sniffer running side-by-side. The problem is that resources read directly from the browser's cache, without any HTTP traffic, may still show up in Net Panel. This is confusing since many people assume Net Panel, as the name implies, shows actual network traffic. This has caused some wild goose chases at Yahoo, so I decided to document the bug. Then it occurred to me that I could fix the bug with a new release of YSlow. I first checked with Joe Hewitt and got his okay to charge ahead. The fix described below is now available in YSlow 0.8.

The Bug(s)

The Net Panel bugs are easy to see with any page that has cacheable resources. I'll use Yahoo Search to demonstrate. Before starting, fire up your favorite packet sniffer, such as LiveHTTPHeaders. Open Firebug and select the Net tab. Clear your cache and go to http://search.yahoo.com. This generates six HTTP requests as shown in Figure 1: the HTML document, a stylesheet, and four images.

Figure 1. Empty cache - six requests

The stylesheet and four images are cacheable, with an Expires date set years in the future. If we click the Reload button, we still see six bars in the Net Panel, but the five cacheable components have a bar that is lighter in color, as shown in Figure 2. The lighter bar indicates a conditional GET request was made for the resource, and the web server returned a 304 Not Modified response with no data. The browser read the data from its cache. Even though these resources have a future Expires date, clicking the Reload button tells the browser to double-check that the cached components are up-to-date, hence the conditional GET request.

Figure 2. Clicking the Reload button - light gray bars are 304 responses

So far everything is working perfectly. Now, instead of clicking Reload, just type the URL, http://search.yahoo.com, and hit return. Here's where things start to go wrong. The page still needs the stylesheet and four images. We know those all have a future Expires date, and since we didn't click Reload the browser should just read them from disk without any network traffic. The only HTTP request that should be necessary is the one for the HTML document, search.yahoo.com. If you look in your packet sniffer, you'll see that in fact that is the only HTTP request. But, as shown in Figure 3, Net Panel incorrectly shows two HTTP requests; one for the HTML document (search.yahoo.com) and one for the stylesheet (ysch_hp_041018.css).

Figure 3. Second page view - the stylesheet is incorrectly listed

Why does the stylesheet show up in Net Panel even though it didn't generate any network traffic? Joe Hewitt explained that in Firefox, scripts and stylesheets are stored in a different cache than images. When a script or stylesheet is read directly from cache (with no network traffic) it unfortunately appears to Firebug as if it were an HTTP request, and so shows up in Net Panel. This example shows this behavior with stylesheets, but it also happens with scripts.

In addition to this problem with scripts and stylesheets, there's a similar, yet subtly different, problem with images. To see this problem, close your browser, restart it, and enter the URL, http://search.yahoo.com, again. There should only be one HTTP request, for search.yahoo.com, and your packet sniffer will confirm this. Looking at Net Panel, we're not surprised to see the stylesheet show up, based on what we saw above. But notice that the four images are also incorrectly listed in Net Panel, as shown in Figure 4, despite the fact that no HTTP requests were made for these images, not even conditional GET requests.

Figure 4. Restart - the stylesheet and the four images are incorrectly listed

Why do the images show up in Net Panel even though they didn't generate any network traffic? By restarting the browser, we cleared its memory cache. The first time the images are read from cache, it again appears to Firebug as if they were HTTP requests, and so they show up in Net Panel. Once the images are in memory cache, this problem doesn't reoccur. That's why the images didn't show up in Figure 3.

All of this poking around points out two flaws with Net Panel:

  • Scripts and stylesheets will always show up in Net Panel, even if they are read from cache without any HTTP traffic.
  • Images that don't have any HTTP requests will show up in Net Panel the first time they are loaded from cache.

The Fix

YSlow 0.8 fixes these Net Panel bugs by using the Date header from the HTTP response and some other heuristics. If the resource is read from cache, the Date will be in the past indicating that no HTTP request was made. For example, if it's currently 27 Sep 17:53 GMT and the Date header says 26 Sep 11:17 GMT, it means this response was read from cache. I added a two line patch to the relevant Firebug function that checks this condition. If the resource has a Date in the past it's removed from the list of requests in Net Panel.

Figure 5 was produced under the same conditions as Figure 4, but this time YSlow 0.8 was installed. As seen here, Firebug's Net Panel now correctly lists the true number of HTTP requests - just one for the HTML document search.yahoo.com.

Figure 5. Restart with the Net Panel Patch - things look good!


During testing I was disappointed to bump into more than a few servers that were sending Date values that didn't match the time the response was sent. The HTTP specification explains that the Date value "SHOULD represent the best available approximation of the date and time of message generation." Incorrect Date values from web servers can cause bigger problems than being an edge case in this bug fix. Browser caching heuristics take the Date value into consideration, as well. I added code to check other information, such as whether any HTTP headers are present and the total load time of the resources. These enhancements catch most of the cases where an unexpected Date value is received.

In Figure 5, notice that in the summary at the bottom it still says "6 requests". Since Firebug thought it saw six HTTP requests, that's the total shown. I didn't alter this so that users would be able to see how many requests had been removed by the bug fix, especially since web servers with incorrect Date values may cause inaccurate results. I also added Patch Net Panel from YSlow to the Options menu, so it's easy for users to turn this patch on and off, just to double-check.

In the table below I show some results of adding the patch. The test was to load the page, thus filling the disk cache with any cacheable resources. Then I restarted the browser and entered the URL for a second time. The results show that enabling the Net Panel Patch matches the actual number of HTTP requests sent.


http://www.google.com/ 1 3 1
http://www.wikipedia.com/ 10 18 10
http://search.yahoo.com/ 1 6 1
http://www.yahoo.com/ 9 33 9