10:30 a.m.
Hi all
maybe you know, last tuesday Yahoo finance denied automated download of
historical data, putting a mandatory "ID cookie" in the query of
download.
The good news
It seems to me the "ID cookie" is unique for every query caming from a
single browser.
E.G. I opened another browser in my machine and the cookie changed, BUT
when I asked another series of data, did not.
I find this right now, so I dont know if the "ID cookie" changes day by
day.
I think a rough way to bypass the restriction is quite simple.
a) Manually find the our own "ID cookie" (EG going with the mouse over
the download link, then reading the last part of the syntiax of the
query, at the bottom of windows )
b) put it in the query
The bad news
I cant write the changes by myself and so giving a service to the
community.
So please, could somebody verify my hypothesis and - if the trick is
good - writing the additional code, inserting a new voice in the yahoo
download menu (EG "Enter your cookie")? Could be sufficent BUT
moreover I suggest to delay in some way two successive query, to avoid
more stringent measures from Yahoo.
Regards
Guido
2:29 p.m.
New subject: "yahoo download" - bypassing download restriction
There are ways to get the SessionId from Cookie automatically.
*My Favorite Open Source Projects awsome-lists
<https://github.com/sindresorhus/awesome> gretl
<http://gretl.sourceforge.net/> robotframework <http://robotframework.org/>
*(*sponsored/patrocinado*) Recomendo servidores e alojamento Web em:
http://www.proalojamento.pt/
<http://www.proalojamento.com/clientes/aff.php?aff=258>
On Sat, May 20, 2017 at 3:30 PM, guido giaume <guido(a)giaume.it> wrote:
Hi all
maybe you know, last tuesday Yahoo finance denied automated download of
historical data, putting a mandatory "ID cookie" in the query of download.
The good news
It seems to me the "ID cookie" is unique for every query caming from a
single browser.
E.G. I opened another browser in my machine and the cookie changed, BUT
when I asked another series of data, did not.
I find this right now, so I dont know if the "ID cookie" changes day by
day.
I think a rough way to bypass the restriction is quite simple.
a) Manually find the our own "ID cookie" (EG going with the mouse over the
download link, then reading the last part of the syntiax of the query, at
the bottom of windows )
b) put it in the query
The bad news
I cant write the changes by myself and so giving a service to the
community.
So please, could somebody verify my hypothesis and - if the trick is good
- writing the additional code, inserting a new voice in the yahoo download
menu (EG "Enter your cookie")? Could be sufficent BUT
moreover I suggest to delay in some way two successive query, to avoid
more stringent measures from Yahoo.
Regards
Guido
_______________________________________________
Gretl-users mailing list
Gretl-users(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-users
11:05 a.m.
New subject: "yahoo download" - bypassing download restriction
On Sat, 20 May 2017, guido giaume wrote:
I think a rough way to bypass the restriction is quite simple.
a) Manually find the our own "ID cookie" (EG going with the mouse over
the download link, then reading the last part of the syntiax of the
query, at the bottom of windows )
b) put it in the query
I don't think I like this idea very much. The cookie hack is obviously
designed to prevent automatic machine-to-machine mass downloads. Probably
there is a way to circumvent this: for example, a hansl script could
request the yahoo page for the ticker, save it as a file, extract the
"crumble" value from the JSON object in which it's contained and add it to
the query. That wouldn't be particularly efficient, but it's doable. But
then, there is a thousand of ways that Yahoo may use to reformulate
slightly the cookie format, so that everything would have to be
re-engineered every time they feel like playing nasty. (OTOH, I see their
point in forcing you to download things by hand: if you skip their ads,
who pays for their data centres that you're downloading your data from?)
This is more or less what the YouTube-dl guys do
(https://youtube-dl.org/); I'm not particularly interested myself.
frankly, I'd rather use my coding time for econometrics proper.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
2699
days inactive
2700
days old
2 comments
3 participants
participants (3)
-
guido giaume -
Hélio Guilherme -
Riccardo (Jack) Lucchetti