Cache Yahoo! Web Service REST Calls

Caching is essential to decrease network loads and improve overall reliability of your web applications. In this HOWTO, we demonstrate a simple caching technique that will reduce transaction times by saving a search result to disk. The best Java caching solution depends on your application.

Caching Requests With a Flat-File Cache

Caching is essential to decrease network loads and improve overall reliability of your web applications. The following example demonstrates a simple flat-file caching technique.

We start with a familiar search query:

String request = "http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=YahooDemo&query=umbrella&results=10";

We'll arbitrarily choose 10 minutes for our cache timeout. You should decide how long to cache based on how often the data changes.

//Replace cached result if it is older than 10 minutes.
int maxAgeMins = 10;

We arbitrarily choose the cache file name to match this particular web services request. Typically, however, you might have a unique filename corresponding to each variation of web services request. So, you would need to come up with a mapping function to map requests into filenames if you want to use the caching with different variations of web services requests.

String fileName = "searchresults.xml";

TransformerFactory is used to format XML documents.

//Initialize XML TransformerFactory
TransformerFactory transformerFactory = TransformerFactory.newInstance();
Transformer transformer = transformerFactory.newTransformer();
transformer.setOutputProperty("indent", "yes"); File xmlResultFile = new File(fileName);
boolean fileExists = xmlResultFile.exists();
if (!fileExists) {

The Search result does not exist, so perform the query and save it to disk.

HttpClient client = new HttpClient();
GetMethod method = new GetMethod(request);
int statusCode = client.executeMethod(method);
InputStream rstream = null;
rstream = method.getResponseBodyAsStream();
Document response = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(rstream);
FileOutputStream fos;
fos = new FileOutputStream(xmlResultFile);
transformer.transform(new DOMSource(response), new StreamResult(fos));
String fileLocation = xmlResultFile.getAbsolutePath();
System.out.println("Saved search results to: " + fileLocation);
} else {

The Search result was found, so check to see how old it is.

String fileLocation = xmlResultFile.getAbsolutePath();
long lastModified = xmlResultFile.lastModified();
long now = new Date().getTime();
long diff = now - lastModified;
float timeSecs = diff / 1000;
float timeMins = timeSecs / 60;

The age of the file is the difference between the current time and the last modified date of the file.

System.out.println("Search results found, saved " + timeMins + " minutes ago: " + fileLocation);
if (timeMins > maxAgeMins) {

The search result exists, but it is too old. Perform query and save it to disk.

 rstream = method.getResponseBodyAsStream();
Document response = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(rstream);
FileOutputStream fos;
fos = new FileOutputStream(xmlResultFile);
transformer.transform(new DOMSource(response), new StreamResult(fos));
System.out.println("Replaced search results older than 10 minutes: " + fileLocation);
}

The source code for this example can be found in the file YahooCacheSearchResults.java

Caching Requests With an In-Memory Cache

Whirlycache is a fast, configurable in-memory object cache for Java. It can be used, for example, to speed up a website or an application by caching objects that would otherwise have to be created by querying a database or by another expensive procedure.

We start with a familiar search query:

String request = "http://api.search.yahoo.com/WebSearchService/V1/webSearch?appid=YahooDemo&query=umbrella&results=10";

We'll choose 5 seconds for our cache timeout in the example source to show two cache misses in a 10 second span. You should choose a timeout based on how ofter the data being cached changes.

//Use the cache manager to create the default cache
Cache c = CacheManager.getInstance().getCache();

//Check the cache for saved response
Document cachedResponse = (Document) c.retrieve(request);
if (cachedResponse == null) {...

//Cache miss, so do HTTP request as above
HttpClient client = new HttpClient();
GetMethod method = new GetMethod(request);
int statusCode = client.executeMethod(method);
InputStream rstream = null;
rstream = method.getResponseBodyAsStream();

//Save response as a Document object
cachedResponse = DocumentBuilderFactory.newInstance().newDocumentBuilder().parse(rstream);

//Store response in cache, expire after 5 seconds
c.store(request, cachedResponse, 5000);

The source code for this example can be found in the file YahooWhirlyCacheSearchResults.java

For More Information

For more information on using Java with Yahoo! Web Services APIs, see The Yahoo! Developer Network Java Developer Center.

Yahoo Forum Discussions