7

Issues when using Yahoo Finance tables

Yesterday various YQL queries started returning the following error: "The content of elements must consist of well-formed character data or markup".

Everything worked fine before. What's wrong?

Example query: select * from yahoo.finance.stocks where symbol="yhoo"

by
33 Replies
  • I experienced the same problem -- working fine around 2am GMT - hopefully it can be addressed in the near future (does this service often fall over ? - I'm pretty new to the yahoo.finance.* services via yql )

    1
  • Something is definitely broken. Hopefully the developers are reading this forum.

    1
  • This requires urgent attention from Yahoo. I am not certain what is going on. Tables that use download.finance.yahoo.com seem unaffected. Those that use finance.yahoo.com are all bombing. I am seeing this in diagnostics too: "execution-start-time": "239", "execution-stop-time": "343", "execution-time": "104", "id": "753e83e2-946c-48c9-b689-23f01dc580b2", "content": "http://finance.yahoo.com/robots.txt" },

    I think this may not be a problem with the tables (query xml) itself but something Yahoo has done on the front end

    2
  • @lawrence Are you saying there is a workaround to use download.* tables? I tried but with no luck. Could you please give some advice? Thanks

    0
  • Hey guys.. Has anyone found a solution for this issue? All of the perfectly running queries are now returning this error..

    0
  • Has anyone heard anything from Yahoo's side?

    0
  • Still no response from Yahoo on this yet on any of the peripheral message boards. No response on twitter either.... Has anyone heard anything?

    0
  • If this problem is serious, it might be a bit too early to know if the problem has been resolved or not.

    0
  • Any word on what's going on? Is this working for anyone?

    0
  • I am guessing that this xml need to be ://github.com/yql/yql-tables/blob/master/yahoo/finance/yahoo.finance.stocks.xml

    Any one knows how to run this file in yql?

    0
  • typo, i meant this ://github.com/yql/yql-tables/blob/master/yahoo/finance/yahoo.finance.stocks.xml

    0
  • one mor time, h t t p s://github.com/yql/yql-tables/blob/master/yahoo/finance/yahoo.finance.stocks.xml

    0
  • There is a yql dev console where you can paste that XML and run it. Yesterday I experimented with it, but it seems that their API for parsing HTML pages is throwing exceptions when trying to parse the yahoo finance pages. I am not sure there is something that can be done here. They should fix their page or their API :(

    0
  • it's a community table, yhoo probably wont fix it. i made some progress last night, will see if i can fix it tonight.

    0
  • Min,

    Thanks very much for your efforts. I apologize if these are noob questions, but what exactly are you trying to fix? The XML table generated by the YQL API? Are you writing html code to parse the source into an XML? Will you then submit it to YQL, so they can run it whenever someone wants the XML? If you fix the Stock table, can we take your code and apply it to the other YQL Finance tables?

    1
  • The yahoo.finance.stock table is probably outdated. I just fixed it and added more stuff. Yes, I'll submit a pull request back to opentable github. The new results output will be like:

    "http://www.yahooapis.com/v1/base.rng" <yahoo:count=>"2" <yahoo:created=>"2014-08-20T08:39:25Z" <yahoo:lang=>"en-US"> <![CDATA[TABLE.query.yahooapis.com.mystocks.a35ac5a7f97fb9ed66cb8403175e5447]]> true <![CDATA[http://finance.yahoo.com/q/pr?s=ntap]]> <![CDATA[http://finance.yahoo.com/q/pr?s=ntap]]> <![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//table[@class="yfnc_datamodoutline1"]/tbody/tr/td/table/tbody/tr | //td[@class="yfnc_modtitlew1"]/b' limit 5]]> <![CDATA[http://finance.yahoo.com/q?s=yhoo]]> <![CDATA[http://finance.yahoo.com/q?s=yhoo]]> <![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//div[@id="yfi_investing_head"]/h1 | //div[@class="yfi_quote_summary"]/div[1]']]> <![CDATA[http://finance.yahoo.com/q/hp?s=ntap]]> <![CDATA[http://finance.yahoo.com/q/hp?s=ntap]]> <![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//option[@selected="selected"] | //input[@maxlength="2"] | //input[@maxlength="4"]']]> <![CDATA[http://finance.yahoo.com/q/pr?s=yhoo]]> <![CDATA[http://finance.yahoo.com/q/pr?s=yhoo]]> <![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//table[@class="yfnc_datamodoutline1"]/tbody/tr/td/table/tbody/tr | //td[@class="yfnc_modtitlew1"]/b' limit 5]]> <![CDATA[http://finance.yahoo.com/q/hp?s=yhoo]]> <![CDATA[http://finance.yahoo.com/q?s=ntap]]> <![CDATA[http://finance.yahoo.com/q?s=ntap]]> <![CDATA[http://finance.yahoo.com/q/hp?s=yhoo]]> <![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//div[@id="yfi_investing_head"]/h1 | //div[@class="yfi_quote_summary"]/div[1]']]> <![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//option[@selected="selected"] | //input[@maxlength="2"] | //input[@maxlength="4"]']]> 249 2120 0.2.2666 1995-11-21 2014-08-20 NetApp, Inc. Technology Data Storage Devices 12490 41.11 41.08 42.02 1.86 12-Nov-14 40.78 41.25 33.34 45.96 3,268,329 3,933,330 13.21B 22.31 1.83 0.66 (1.80%) 1996-04-12 2014-08-20 Yahoo! Inc. Technology Internet Information Providers 12200 37.38 37.58 39.50 1.07 Oct 13 - Oct 17 (Est.) 37.50 37.94 26.75 41.72 17,093,691 18,154,700 37.63B 31.55 1.20 N/A (N/A)

    0
  • repost with the xml inside a code block:



    <?xml version="1.0" encoding="UTF-8"?> <query <xmlns:yahoo=>"<http://www.yahooapis.com/v1/base.rng>" <yahoo:count=>"2" <yahoo:created=>"2014-08-20T08:39:25Z" <yahoo:lang=>"en-US"> <diagnostics> <cache execution-start-time="0" execution-stop-time="2" execution-time="2" method="GET" type="MEMCACHED"><![CDATA[TABLE.query.yahooapis.com.mystocks.a35ac5a7f97fb9ed66cb8403175e5447]]></cache> <publiclyCallable>true</publiclyCallable> <url execution-start-time="4" execution-stop-time="94" execution-time="90"><![CDATA[<http://finance.yahoo.com/q/pr?s=ntap>]]></url> <url execution-start-time="4" execution-stop-time="94" execution-time="90"><![CDATA[<http://finance.yahoo.com/q/pr?s=ntap>]]></url> <query execution-start-time="4" execution-stop-time="97" execution-time="93" params="{url=[<http://finance.yahoo.com/q/pr?s=ntap>]}"><![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//table[@class="yfnc_datamodoutline1"]/tbody/tr/td/table/tbody/tr | //td[@class="yfnc_modtitlew1"]/b' limit 5]]></query> <url execution-start-time="4" execution-stop-time="144" execution-time="140"><![CDATA[<http://finance.yahoo.com/q?s=yhoo>]]></url> <url execution-start-time="4" execution-stop-time="144" execution-time="140"><![CDATA[<http://finance.yahoo.com/q?s=yhoo>]]></url> <query execution-start-time="4" execution-stop-time="149" execution-time="145" params="{url=[<http://finance.yahoo.com/q?s=yhoo>]}"><![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//div[@id="yfi_investing_head"]/h1 | //div[@class="yfi_quote_summary"]/div[1]']]></query> <url execution-start-time="4" execution-stop-time="180" execution-time="176"><![CDATA[<http://finance.yahoo.com/q/hp?s=ntap>]]></url> <url execution-start-time="4" execution-stop-time="180" execution-time="176"><![CDATA[<http://finance.yahoo.com/q/hp?s=ntap>]]></url> <query execution-start-time="4" execution-stop-time="183" execution-time="179" params="{url=[<http://finance.yahoo.com/q/hp?s=ntap>]}"><![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//option[@selected="selected"] | //input[@maxlength="2"] | //input[@maxlength="4"]']]></query> <url execution-start-time="4" execution-stop-time="214" execution-time="210"><![CDATA[<http://finance.yahoo.com/q/pr?s=yhoo>]]></url> <url execution-start-time="4" execution-stop-time="214" execution-time="210"><![CDATA[<http://finance.yahoo.com/q/pr?s=yhoo>]]></url> <query execution-start-time="4" execution-stop-time="217" execution-time="213" params="{url=[<http://finance.yahoo.com/q/pr?s=yhoo>]}"><![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//table[@class="yfnc_datamodoutline1"]/tbody/tr/td/table/tbody/tr | //td[@class="yfnc_modtitlew1"]/b' limit 5]]></query> <url execution-start-time="4" execution-stop-time="241" execution-time="237"><![CDATA[<http://finance.yahoo.com/q/hp?s=yhoo>]]></url> <url execution-start-time="5" execution-stop-time="240" execution-time="235"><![CDATA[<http://finance.yahoo.com/q?s=ntap>]]></url> <url execution-start-time="5" execution-stop-time="240" execution-time="235"><![CDATA[<http://finance.yahoo.com/q?s=ntap>]]></url> <url execution-start-time="4" execution-stop-time="241" execution-time="237"><![CDATA[<http://finance.yahoo.com/q/hp?s=yhoo>]]></url> <query execution-start-time="4" execution-stop-time="242" execution-time="238" params="{url=[<http://finance.yahoo.com/q?s=ntap>]}"><![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//div[@id="yfi_investing_head"]/h1 | //div[@class="yfi_quote_summary"]/div[1]']]></query> <query execution-start-time="4" execution-stop-time="243" execution-time="239" params="{url=[<http://finance.yahoo.com/q/hp?s=yhoo>]}"><![CDATA[select * from html where <url=@url> and compat="html5" and xpath='//option[@selected="selected"] | //input[@maxlength="2"] | //input[@maxlength="4"]']]></query> <javascript execution-start-time="3" execution-stop-time="247" execution-time="244" instructions-used="140111" table-name="mystocks"/> <javascript execution-start-time="3" execution-stop-time="248" execution-time="245" instructions-used="156563" table-name="mystocks"/> <user-time>249</user-time> <service-time>2120</service-time> <build-version>0.2.2666</build-version> </diagnostics> <results> <stock symbol="ntap"> <start>1995-11-21</start> <end>2014-08-20</end> <CompanyName>NetApp, Inc.</CompanyName> <Sector>Technology</Sector> <Industry>Data Storage Devices</Industry> <FullTimeEmployees>12490</FullTimeEmployees> <PrevClose>41.11</PrevClose> <Open>41.08</Open> <OneYearTargetEst>42.02</OneYearTargetEst> <Beta>1.86</Beta> <NextEarningsDate>12-Nov-14</NextEarningsDate> <DaysRange> <From>40.78</From> <To>41.25</To> </DaysRange> <FiftyTwoWkRange> <From>33.34</From> <To>45.96</To> </FiftyTwoWkRange> <Volume>3,268,329</Volume> <AvgVol3m>3,933,330</AvgVol3m> <MarketCap>13.21B</MarketCap> <PEttm>22.31</PEttm> <EPSttm>1.83</EPSttm> <DivYield>0.66 (1.80%)</DivYield> </stock> <stock symbol="yhoo"> <start>1996-04-12</start> <end>2014-08-20</end> <CompanyName>Yahoo! Inc.</CompanyName> <Sector>Technology</Sector> <Industry>Internet Information Providers</Industry> <FullTimeEmployees>12200</FullTimeEmployees> <PrevClose>37.38</PrevClose> <Open>37.58</Open> <OneYearTargetEst>39.50</OneYearTargetEst> <Beta>1.07</Beta> <EarningsDate>Oct 13 - Oct 17 (Est.)</EarningsDate> <DaysRange> <From>37.50</From> <To>37.94</To> </DaysRange> <FiftyTwoWkRange> <From>26.75</From> <To>41.72</To> </FiftyTwoWkRange> <Volume>17,093,691</Volume> <AvgVol3m>18,154,700</AvgVol3m> <MarketCap>37.63B</MarketCap> <PEttm>31.55</PEttm> <EPSttm>1.20</EPSttm> <DivYield>N/A (N/A)</DivYield> </stock> </results> </query>
    0
  • from @YahooCare

    Our engineers are looking into this. Sorry for any confusion.

    0
  • @BeastOrchestra thanks for that !

    0
  • Thanks for looking into this. I've also submitted a pull request and hope it can help:

    https://github.com/yql/yql-tables/pull/432

    0
  • I saw something on github that maybe maybe maybe can be useful in relation to this issue. It was edited only ten days ago I just thought it may help? (without looking into the code):

    https://github.com/trunghung/yql-tables/commit/99094edfee763169075314e35cfcda62fc9c7ecf

    By the way, the tables finance.incomestatement | balancesheet | cashflow also report the error.

    Hope this gets solved!

    0
  • @Min

    Can you tell us what part of your github diff corrects this issue:

    error="The content of elements must consist of well-formed character data or markup."

    knowing that may make it easier to clean up other tables

    0
  • The yahoo.finance tables are not official Yahoo YQL tables. They were created by 3rd parties or were intended as examples. They are also rate limited by the finance site or are occasionally blocked. Also, Yahoo finance data is not approved for commercial usage via YQL.

    From the YQL home page: https://developer.yahoo.com/yql/

    The YQL Service can be used for commercial purposes without prior approval from Yahoo. Developers using the YQL Service fetching data from a content provider need to conform to the Terms of Service of the content provider. Please note that data available on the Yahoo! network (e.g., Finance, Weather) is not approved for commercial usage via YQL, unless mentioned otherwise in the Terms of Service.

    0
  • Interesting, how about accessing other finance websites like google finance via YQL? Any YQL terms on these types of usage?

    0
  • So if they are created by 3rd parties, any reason why they don't work? I mean they work with YQL, so is it a core YQL problem or something related to finance specifically?

    0
  • The table yahoo.finance.keystats is also still down, and the query below generates the error below. So is it just all tables from yahoo finance are not working?

    SELECT * FROM yahoo.finance.keystats WHERE symbol='T'

    error="Exception while converting tidy sanitized html to Dom"

    1
  • Min,

    It is great that you submitted a pull request and this is going to be very helpful when fixing the other tables, but can you please reformat the indentation to match the one in the master branch, because right now it is not visible what you actually changed. It seems that all the lines in the file changed, which is not very helpful.

    BTW, at this point we just moved away from YQL... However I think this is a nice technology and I may reconsider using it if I have more knowledge how to handle situations like this. In this particular case I am curious how you managed to fix the HTML parsing, as it looked to me as an internal API issue.

    Thanks, Valentin

    0
  • The key fix I put in the table is adding "compat=html5" condition while querying the html content, it got rid of the error about "The content of elements must consist of well-formed character data or markup." But because of this, the result HTML content has different xpath than before, so I had to adjust the xpath and the html parsing function as well. In addition to these, I also added more stock summary data to the result so it does has quite a few changes.

    0
  • I tweeted @Yahoocare about the YQL finance tables and got back the following:

    Yahoo Finance doesn't own or provide any support for community-created tables, sorry. Yahoo Finance doesn't provide any official API's to the developers except for the ticker based RSS feeds: https://developer.yahoo.com/finance/

    It appears that we're on our own.

    0
  • Min,

    Thanks again for your time in figuring out the issues with the stocks table. I looked at your code and used the syntax below in the YQL console. It returned a disjointed XML type file. Is there some way to modify the query, so I get the well formed XML that you show above?

    select * from html where url='http://finance.yahoo.com/q?s=goog' and xpath='//div[@id="yfi_investing_head"]/h1 | //div[@class="yfi_quote_summary"]/div[1]' and compat="html5"

    0
  • It should work if you use single quote for the compat property, e.g. compat='html5'

    0
  • Seems to me that whoever maintains these "community tables" made a change. So, the obvious question is who maintains these tables...and, what was changed. Maybe it's time to change to Microsoft or Google...

    0
  • So...further research indicates that perhaps Yahoo has modified the data from which these queries obtain data? I've tried several of the examples and they don't work either. Some YQL table info is on Github and it appears that nothing has changed recently. For example the "options_contracts" table info has the name: Alan Verga associated with it. It appears as he is the author of the code that builds the table itself from the Yahoo data? I can't find an email address for him, so can't go further with this. It may be that since Yahoo isn't doing so well, perhaps it's trying to shut down these types of queries and force people to their website where they can be subjected to advertising? :-)

    0

Recent Posts

in YQL