1:10 p.m.
Dear all,
a student of mine just let me know that he has problems with the yahoo_get
package on windows, when it comes to downloading data before 1970. This is
a consequence of the fact that our strptime() function behaves differently
on linux and windows. For example, the code
<hansl>
x = strptime("1964-05-11", "%Y-%m-%d")
</hansl>
returns generate a value for x that is -178074000 on linux but -1 on
windows. more generally, windows truncates any date prior to 1970 to -1.
Of course, writing a workaround for the affected package is relatively
simple, but shouldn't we address this inconsistency? Or this is not for
us, and we should just mention this fact in the docs? What do you think?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
3:29 p.m.
Sorry, I could not resist.
Tell your students to not use Windows :D.
Linux or BSD based OS are OK ;)
On Fri, Jul 22, 2022 at 2:11 PM Riccardo (Jack) Lucchetti <
p002264(a)staff.univpm.it> wrote:
Dear all,
a student of mine just let me know that he has problems with the yahoo_get
package on windows, when it comes to downloading data before 1970. This is
a consequence of the fact that our strptime() function behaves differently
on linux and windows. For example, the code
<hansl>
x = strptime("1964-05-11", "%Y-%m-%d")
</hansl>
returns generate a value for x that is -178074000 on linux but -1 on
windows. more generally, windows truncates any date prior to 1970 to -1.
Of course, writing a workaround for the affected package is relatively
simple, but shouldn't we address this inconsistency? Or this is not for
us, and we should just mention this fact in the docs? What do you think?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------_______________________________________________
Gretl-users mailing list -- gretl-users(a)gretlml.univpm.it
To unsubscribe send an email to gretl-users-leave(a)gretlml.univpm.it
Website:
https://gretlml.univpm.it/postorius/lists/gretl-users.gretlml.univpm.it/
7:54 p.m.
On Fri, Jul 22, 2022 at 9:10 AM Riccardo (Jack) Lucchetti
<p002264(a)staff.univpm.it> wrote:
a student of mine just let me know that he has problems with the yahoo_get
package on windows, when it comes to downloading data before 1970. This is
a consequence of the fact that our strptime() function behaves differently
on linux and windows. For example, the code
<hansl>
x = strptime("1964-05-11", "%Y-%m-%d")
</hansl>
returns generate a value for x that is -178074000 on linux but -1 on
windows. more generally, windows truncates any date prior to 1970 to -1.
Of course, writing a workaround for the affected package is relatively
simple, but shouldn't we address this inconsistency? Or this is not for
us, and we should just mention this fact in the docs? What do you think?
A little background here. Windows doesn't have strptime(), so libgretl
includes an implementation of it, mostly nabbed from Kungliga Tekniska
Högskolan (Sweden). And It's not strptime that's falling over for
dates prior to 1970. Rather it's mktime(), which is required to
convert the "struct tm" pointer obtained from strptime to "epoch
time"
in seconds. Unfortunately the mktime() included in the Windows CRT (C
runtime) doesn't handle pre-1970 dates. So to get this right on
Windows we'd need an alternative implementation of mktime(). A little
googling suggests this isn't trivial, but maybe we can find or devise
something. If not, then I agree, we should add a warning regarding the
limitations of Windows to the strptime() help.
Allin
9:11 p.m.
On Fri, Jul 22, 2022 at 3:54 PM Cottrell, Allin <cottrell(a)wfu.edu> wrote:
On Fri, Jul 22, 2022 at 9:10 AM Riccardo (Jack) Lucchetti
<p002264(a)staff.univpm.it> wrote:
>
> a student of mine just let me know that he has problems with the yahoo_get
> package on windows, when it comes to downloading data before 1970. This is
> a consequence of the fact that our strptime() function behaves differently
> on linux and windows. [...]
A little background here. Windows doesn't have strptime(), so libgretl
includes an implementation of it, mostly nabbed from Kungliga Tekniska
Högskolan (Sweden). And It's not strptime that's falling over for
dates prior to 1970. Rather it's mktime(), which is required to
convert the "struct tm" pointer obtained from strptime to "epoch
time"
in seconds.
Sorry, I realize that my reply may be confusing without a little more
info. The gretl function strptime() uses, but is not equivalent to,
the underlying C library function strptime(). The C function (and also
our remedial C implementation for Windows) does not return an "epoch
time" as such. But it makes available a "struct tm" (something that
has no representation in hansl) from which epoch time can be derived
via mktime(). The gretl function then returns the epoch time as a
"scalar".
Unix-type systems treat this epoch time as a signed integer, so
negative offsets relative to midnight of 1969-12-31/1970-01-01 can be
represented. MS Windows treats it as an unsigned integer and so cannot
handle prior dates. If you think about it for a moment, it's clear
that figuring the offset in seconds of a given date from a given
benchmark (the aforementioned start of 1970) is not at all trivial --
think of leap seconds, never mind leap years. So I doubt there's
anything we can do about this other than post a warning.
Allin
9:11 a.m.
Am 22.07.2022 um 23:11 schrieb Cottrell, Allin:
Unix-type systems treat this epoch time as a signed integer, so
negative offsets relative to midnight of 1969-12-31/1970-01-01 can be
represented. MS Windows treats it as an unsigned integer and so cannot
handle prior dates. If you think about it for a moment, it's clear
that figuring the offset in seconds of a given date from a given
benchmark (the aforementioned start of 1970) is not at all trivial --
think of leap seconds, never mind leap years. So I doubt there's
anything we can do about this other than post a warning.
Would it be an option to return NA instead of -1 in those cases (if I've
understood correctly what's happening) ?
cheers
sven
1 p.m.
On Sat, Jul 23, 2022 at 5:11 AM Sven Schreiber <svetosch(a)gmx.net> wrote:
Am 22.07.2022 um 23:11 schrieb Cottrell, Allin:
>
> Unix-type systems treat this epoch time as a signed integer, so
> negative offsets relative to midnight of 1969-12-31/1970-01-01 can be
> represented. MS Windows treats it as an unsigned integer and so cannot
> handle prior dates. If you think about it for a moment, it's clear
> that figuring the offset in seconds of a given date from a given
> benchmark (the aforementioned start of 1970) is not at all trivial --
> think of leap seconds, never mind leap years. So I doubt there's
> anything we can do about this other than post a warning.
Would it be an option to return NA instead of -1 in those cases (if I've
understood correctly what's happening) ?
Good idea. That's now in git. I've also updated the new English help
text for strptime() to reflect this.
Allin
1:43 p.m.
On Sat, 23 Jul 2022, Cottrell, Allin wrote:
Good idea. That's now in git. I've also updated the new
English help
text for strptime() to reflect this.
Italian too.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
10:21 a.m.
On Fri, 22 Jul 2022, Cottrell, Allin wrote:
Unix-type systems treat this epoch time as a signed integer, so
negative offsets relative to midnight of 1969-12-31/1970-01-01 can be
represented. MS Windows treats it as an unsigned integer and so cannot
handle prior dates. If you think about it for a moment, it's clear
that figuring the offset in seconds of a given date from a given
benchmark (the aforementioned start of 1970) is not at all trivial --
think of leap seconds, never mind leap years. So I doubt there's
anything we can do about this other than post a warning.
A concise note is now in the xml helpfiles in git.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
4:04 p.m.
-----Oorspronkelijk bericht-----
Van: Cottrell, Allin
...
Unix-type systems treat this epoch time as a signed integer, so negative offsets
relative to midnight of 1969-12-31/1970-01-01 can be represented. MS
Windows treats it as an unsigned integer and so cannot handle prior dates.
I think on MS Windows time_t is also a signed integer (int64); see
https://docs.microsoft.com/en-us/cpp/c-runtime-library/standard-types?vie...
Also, mktime could not have -1 as a return value if time_t was unsigned.
Rather it's the implementation that refuses to deal with dates before 1-1-1970; see
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/mktime-m...
BTW, this behavior is not prohibited by the Posix and ISO-C specifications, so there might
also be Unix systems with
similar behavior. See
https://pubs.opengroup.org/onlinepubs/9699919799/basedefs/V1_chap04.html#...
The MS Windows API can deal with dates from 1-1-1601; it measures time in 100-nanoseconds,
and contrary to time_t, this ìs an unsigned integer. See
https://docs.microsoft.com/en-us/windows/win32/api/minwinbase/ns-minwinba...
If you think about it for a moment, it's clear that figuring the
offset in seconds of a
given date from a given benchmark (the aforementioned start of 1970) is not at
all trivial -- think of leap seconds, never mind leap years. So I doubt there's
anything we can do about this other than post a warning.
The Posix specification does not require to take leap seconds into account, only leap
days: "The relationship between the actual time of day and the current value for
seconds since the Epoch is unspecified. How any changes to the value of seconds since the
Epoch are made to align to a desired relationship with the current actual time is
implementation-defined. As represented in seconds since the Epoch, each and every day
shall be accounted for by exactly 86400 seconds." The difficulty seems to lie more in
the difference between local time and UTC, and whether DST should be taken into account,
but these are probably best specified at the user level in the call to strptime.
There are free versions of mktime in glibc, gnulib, and newlib. The newlib version looks
simpler to implement than the glibc and gnulib versions; the strptime version from
Kungliga Tekniska Högskolan is also in newlib.
Kees
4:26 p.m.
On Wed, Jul 27, 2022 at 12:04 PM <kzstatis(a)gmail.com> wrote:
> -----Oorspronkelijk bericht-----
> Van: Cottrell, Allin
> ...
> Unix-type systems treat this epoch time as a signed integer, so negative offsets
> relative to midnight of 1969-12-31/1970-01-01 can be represented. MS
> Windows treats it as an unsigned integer and so cannot handle prior dates.
I think on MS Windows time_t is also a signed integer (int64); see
https://docs.microsoft.com/en-us/cpp/c-runtime-library/standard-types?vie...
Also, mktime could not have -1 as a return value if time_t was unsigned.
Rather it's the implementation that refuses to deal with dates before 1-1-1970; see
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/mktime-m...
Thanks for the correction. So it looks as if we can fix the issue on
Windows by calling _mktime64() instead of plain mktime().
Allin
6:02 p.m.
On Wed, Jul 27, 2022 at 12:26 PM Cottrell, Allin <cottrell(a)wfu.edu> wrote:
On Wed, Jul 27, 2022 at 12:04 PM <kzstatis(a)gmail.com> wrote:
>
> > -----Oorspronkelijk bericht-----
> > Van: Cottrell, Allin
> > ...
> > Unix-type systems treat this epoch time as a signed integer, so negative
offsets
> > relative to midnight of 1969-12-31/1970-01-01 can be represented. MS
> > Windows treats it as an unsigned integer and so cannot handle prior dates.
>
> I think on MS Windows time_t is also a signed integer (int64); see
> https://docs.microsoft.com/en-us/cpp/c-runtime-library/standard-types?vie...
> Also, mktime could not have -1 as a return value if time_t was unsigned.
> Rather it's the implementation that refuses to deal with dates before 1-1-1970;
see
>
https://docs.microsoft.com/en-us/cpp/c-runtime-library/reference/mktime-m...
Thanks for the correction. So it looks as if we can fix the issue on
Windows by calling _mktime64() instead of plain mktime().
No, I misunderstood the Microsoft doc. In fact mktime() calls
_mktime64() by default. Regardless of the signedness or width of
time_t on Windows, their mktime function(s) refuse to return anything
other than -1 for dates prior to 1970. Tant pis!
Although some of what I said here previously was wrong, I believe the
new text in the documentation for gretl's strptime() and strftime(),
regarding the limitations of MS Windows, is correct.
In principle we might think of substituting our own implementation of
mktime() on Windows, but this is a decidedly non-trivial function.
Allin
638
days inactive
643
days old
10 comments
5 participants
participants (5)
-
Cottrell, Allin -
Hélio Guilherme -
kzstatis＠gmail.com -
Riccardo (Jack) Lucchetti -
Sven Schreiber