One way of dramatically speeding up applications that are built using the Yahoo! Web Services APIs is to make heavy use of caching. With caching, calls that have been previously made in a specified timeframe can be answered using cached data rather than making an API call over the network. This HOWTO describes caching options provided by ASP.NET.
There are many ways to cache data received from Yahoo! web services and choosing the correct type is heavily dependant on the type of application you are creating, expected loads and even server hardware. For smaller scale applications with low user amounts and small data sizes a memory based caching system may suffice, but under heavier loads, or in a server farm environment where memory isn't shared, a disk or database based approach may be better. Another issue to consider is how frequently a call to a web service will be made and how soon the data returned becomes stale. Keeping these in mind will help you design a more responsive and less resource intensive application.
The .NET Framework provides built-in support for various caching technologies. When using ASP.NET to create web applications
you will most likely use a combination of
Cache objects. In some
cases you may also want to use
ViewState for small bits of information that must survive roundtrips. ASP.NET also
provides page output caching, but that is not what we are concerned with here.
Cache object is probably your number one choice for caching when you only need the cache to last for the
duration your application is running. Usually this is until the site, pool, IIS or server is restarted. It provides many options to automatically
expire cached data. You can invalidate the contents when a file changes, when another item in the cache changes, or set
an expiration timeout. You can even set up callbacks to run when items are removed. While it is possible to use the
Cache object in any type of application, you will most likely use it with ASP.NET applications.
To access the
Cache object, you can use the
HttpContext class or by using the
Page object. It is important to remember that the data placed in the cache is not tied to a user's
session and storing large amounts of data for multiple users will tax your server's memory.
ASP.NET session state provides a simple means to cache per-user data. Session state can be configured to store data in memory, on a state server or in a SQL Server database. Using a state server or SQL server also enabled sharing the data across multiple servers.
Session object is easy but isn't suitable for large data items and does not provide automatic
If you need to cache a large amount of data or have it stored persistently, creating a disk caching class may be a good option. Saving data onto a disk however raises questions about security, thread safety, management and can be complicated if more than one server is running your application. There are also issues with clearing out the temporary files after they have become stale.
The Microsoft Patterns & Practices Developer Center has an Application Cache Block as part of their Enterprise Library that provides a full-blown solution for caching with disk support.
You can use SQL Server 2000 and 2005 and their free counterparts MSDE and SQL Server Express Edition to store data returned by web services that need to be saved for longer periods of time or needs to be accessed by multiple front end servers.
When thinking about using SQL Server as the persistent store you should be aware that it is a relatively slow solution for caching purposes. However, the reliability, robust security model and support for server farms may outweigh the downsides. It is always wise to run a comparison on retrieving the data directly from the web service versus storing it in the database.
SQL Server caching with ADO.NET is relatively easy to implement, but will not be discussed here in detail since there are lots of resources on .NET database access available online.
Related information on the web.