Back to Yahoo! Mail Application Platform Forum
0

per app data questions

Hi,
I'm taking my first look at Yahoo Mail App development, and have a few questions. First is regarding the per application data persistence feature. I've read the api -- very simple! For one, I'm interested in data size limitations. Is there a strict upper limit on the size of object that can be stored? The number of objects that can be stored per application?

Second, the api is key/value based, but it looks like there is no unindexed value associated with the key. Without understanding the indexing implementation - I can imagine performance problems using this api to implement something like a diary, where an key might be composed of diary name, entry date, and the entry content. A typical key/value store would have a key composed of [diary name, date] with a value of [entry content], however it looks like the Yahoo mail app api would implement this as a key of [diary name, date, entry content].

Partial key searches? Of course, I'd like to be able to fetch a list of diary entries based on just the diary name, or even a range based on a beginning and ending key.

Any additional technical information or roadmap for this service? I mean, if this is meant to be the Yahoo answer to Google's BigTable, etc. it needs a bit more meat on the bones.

Am I barking up the wrong tree? For more sophisticated data storage, are apps supposed to use their own web service?

Thanks for considering these questions,
Erik.

by
Report Abuse
5 Replies
  • Hi Erik,

    I am not actually sure what the strict upper limits are, but will research and get back to you.

    In practice, I believe you should be able to store at least a MB per key, but if you're doing that from a client application, it's likely unusable if you're doing sets that big from the browser.  I believe the key limit shouldn't be an issue for a diary, but will try to get you an exact number.  

    Only keys are indexed.  The API's name choice is poor for the args, "keys" is actually an object specifying both the keys and vals.   I'll try to update the API docs to be clearer about this and request we update the API as well.  You just specify key vals as a javascript object, e.g. for key "a" with val "foo" and key "b" with val "bar", you'd use:  openmail.Application.setData( { keys : { a : foo, b : bar }}, . . .);  

    Have a look at the Memories sample app and see if that makes sense:

    http://developer.yahoo.com/mailapplications/samples/memories.html

    -j

    QUOTE(Erik Pearson @ 21 Aug 2011 12:08 PM)
    Hi,
    I'm taking my first look at Yahoo Mail App development, and have a few questions. First is regarding the per application data persistence feature. I've read the api -- very simple! For one, I'm interested in data size limitations. Is there a strict upper limit on the size of object that can be stored? The number of objects that can be stored per application?

    Second, the api is key/value based, but it looks like there is no unindexed value associated with the key. Without understanding the indexing implementation - I can imagine performance problems using this api to implement something like a diary, where an key might be composed of diary name, entry date, and the entry content. A typical key/value store would have a key composed of [diary name, date] with a value of [entry content], however it looks like the Yahoo mail app api would implement this as a key of [diary name, date, entry content].

    Partial key searches? Of course, I'd like to be able to fetch a list of diary entries based on just the diary name, or even a range based on a beginning and ending key.

    Any additional technical information or roadmap for this service? I mean, if this is meant to be the Yahoo answer to Google's BigTable, etc. it needs a bit more meat on the bones.

    Am I barking up the wrong tree? For more sophisticated data storage, are apps supposed to use their own web service?

    Thanks for considering these questions,
    Erik.
    Report Abuse
    0
  • You can do partial matches by using "?" and "*" in the key strings you specify to getData, i.e. to return what you stored for both "january" and "joy", this should work:

    getData( { keys : ['j*y'], . . .);

    This is not meant to compete with BigTable persay, it's just exposing part of our mail metadata storage system to devs.

    You are welcome to use an external backend, lots of devs do.

    -j

    QUOTE(Erik Pearson @ 21 Aug 2011 12:08 PM)
    Hi,
    I'm taking my first look at Yahoo Mail App development, and have a few questions. First is regarding the per application data persistence feature. I've read the api -- very simple! For one, I'm interested in data size limitations. Is there a strict upper limit on the size of object that can be stored? The number of objects that can be stored per application?

    Second, the api is key/value based, but it looks like there is no unindexed value associated with the key. Without understanding the indexing implementation - I can imagine performance problems using this api to implement something like a diary, where an key might be composed of diary name, entry date, and the entry content. A typical key/value store would have a key composed of [diary name, date] with a value of [entry content], however it looks like the Yahoo mail app api would implement this as a key of [diary name, date, entry content].

    Partial key searches? Of course, I'd like to be able to fetch a list of diary entries based on just the diary name, or even a range based on a beginning and ending key.

    Any additional technical information or roadmap for this service? I mean, if this is meant to be the Yahoo answer to Google's BigTable, etc. it needs a bit more meat on the bones.

    Am I barking up the wrong tree? For more sophisticated data storage, are apps supposed to use their own web service?

    Thanks for considering these questions,
    Erik.
    Report Abuse
    0
  • Thanks, I'll tinker around. I do appreciate having some sort of persistence in there.
    Erik.

    QUOTE(joecomotion @ 22 Aug 2011 4:12 PM)
    You can do partial matches by using "?" and "*" in the key strings you specify to getData, i.e. to return what you stored for both "january" and "joy", this should work:

    getData( { keys : ['j*y'], . . .);

    This is not meant to compete with BigTable persay, it's just exposing part of our mail metadata storage system to devs.

    You are welcome to use an external backend, lots of devs do.

    -j

    QUOTE(Erik Pearson @ 21 Aug 2011 12:08 PM)
    Hi,
    I'm taking my first look at Yahoo Mail App development, and have a few questions. First is regarding the per application data persistence feature. I've read the api -- very simple! For one, I'm interested in data size limitations. Is there a strict upper limit on the size of object that can be stored? The number of objects that can be stored per application?

    Second, the api is key/value based, but it looks like there is no unindexed value associated with the key. Without understanding the indexing implementation - I can imagine performance problems using this api to implement something like a diary, where an key might be composed of diary name, entry date, and the entry content. A typical key/value store would have a key composed of [diary name, date] with a value of [entry content], however it looks like the Yahoo mail app api would implement this as a key of [diary name, date, entry content].

    Partial key searches? Of course, I'd like to be able to fetch a list of diary entries based on just the diary name, or even a range based on a beginning and ending key.

    Any additional technical information or roadmap for this service? I mean, if this is meant to be the Yahoo answer to Google's BigTable, etc. it needs a bit more meat on the bones.

    Am I barking up the wrong tree? For more sophisticated data storage, are apps supposed to use their own web service?

    Thanks for considering these questions,
    Erik.
    Report Abuse
    0
  • Seem like part of my previous reply evaporated. 

    They key is the only part indexed.  Part of your confusion is that setData's data.keys arg should actually be called data.keyvals, since it's an object whose member names specify key names and whose member values specify the keys' corresponding values.  I've opened a bug to clean up the doc and/or API.

    I will research the limits and try to document -- embarrassed to say I'm not sure what they are.  As a working guideline, I know you'll run into issues with app performance transferring large vals from your browser before you'll hit our limit for vals -- i.e. if you're setting vals of 1MB+, you're going to have a bad UX.  As far as the max key count, I'll go out on a limb and say you can probably do a million at least.  I'll follow up with real numbers once I hunt them down.
    Report Abuse
    0
  • Thanks for the tips. It looks like the solution to doing what I want, sans any sort of complex querying, is to use a compound key to store anything interesting. In the example I was using, a compound key of "diary_mydiary_2011-08-24" would allow storing diary entries into "books" (mydiary) with one entry per date. It can be queried to yield all diary entries, all per book, etc. Combine this with storing an arbitrary object, and voila, you can do a lot of simple things.



    QUOTE(joecomotion @ 22 Aug 2011 4:12 PM)
    You can do partial matches by using "?" and "*" in the key strings you specify to getData, i.e. to return what you stored for both "january" and "joy", this should work:

    getData( { keys : ['j*y'], . . .);

    This is not meant to compete with BigTable persay, it's just exposing part of our mail metadata storage system to devs.

    You are welcome to use an external backend, lots of devs do.

    -j

    QUOTE(Erik Pearson @ 21 Aug 2011 12:08 PM)
    Hi,
    I'm taking my first look at Yahoo Mail App development, and have a few questions. First is regarding the per application data persistence feature. I've read the api -- very simple! For one, I'm interested in data size limitations. Is there a strict upper limit on the size of object that can be stored? The number of objects that can be stored per application?

    Second, the api is key/value based, but it looks like there is no unindexed value associated with the key. Without understanding the indexing implementation - I can imagine performance problems using this api to implement something like a diary, where an key might be composed of diary name, entry date, and the entry content. A typical key/value store would have a key composed of [diary name, date] with a value of [entry content], however it looks like the Yahoo mail app api would implement this as a key of [diary name, date, entry content].

    Partial key searches? Of course, I'd like to be able to fetch a list of diary entries based on just the diary name, or even a range based on a beginning and ending key.

    Any additional technical information or roadmap for this service? I mean, if this is meant to be the Yahoo answer to Google's BigTable, etc. it needs a bit more meat on the bones.

    Am I barking up the wrong tree? For more sophisticated data storage, are apps supposed to use their own web service?

    Thanks for considering these questions,
    Erik.
    Report Abuse
    0
✕ This forum is locked.
Ask a Question

Recent Posts

in Yahoo! Mail Application Platform