12:21 p.m.
Hi,
it seems that msortby() only sorts within "blocks" surrounded by
occurrences of NA (showing up as nan in matrices). The rest of the
sorting-like functions seem to work ok. Example:
<hansl>
matrix in = {2; 1; NA; 0; -5}
print in
matrix check = msortby(in, 1) # not ok
print check
matrix check = sort(in) # ok
print check
matrix check = dsort(in) # ok
print check
matrix check = values(in) # ok
print check
matrix check = uniq(in) # ok
print check
</hansl>
Thanks,
sven
4:29 p.m.
On Tue, 2 Dec 2014, Sven Schreiber wrote:
it seems that msortby() only sorts within "blocks"
surrounded by
occurrences of NA (showing up as nan in matrices). The rest of the
sorting-like functions seem to work ok. Example:
<hansl>
matrix in = {2; 1; NA; 0; -5}
print in
matrix check = msortby(in, 1) # not ok
print check
matrix check = sort(in) # ok
print check
matrix check = dsort(in) # ok
print check
matrix check = values(in) # ok
print check
matrix check = uniq(in) # ok
print check
</hansl>
I believe the results you get from these calls will depend on the C
library. We should probably generate an error if you try to sort a
matrix containing NaNs since the sort order is indeterminate.
Allin
9:09 p.m.
Am 02.12.2014 um 17:29 schrieb Allin Cottrell:
On Tue, 2 Dec 2014, Sven Schreiber wrote:
I believe the results you get from these calls will depend on the C
library. We should probably generate an error if you try to sort a
matrix containing NaNs since the sort order is indeterminate.
It's clear that the position of nans is undefined after sorting, but I
still think that being able to sort is quite useful/important. Whether
the nans then show up in the beginning or end or in the middle is
irrelevant and the user's responsibility. So almost all of the functions
are ok IMO. Only msortby() does something very odd.
thanks,
sven
3:59 p.m.
On Tue, 2 Dec 2014, Sven Schreiber wrote:
Am 02.12.2014 um 17:29 schrieb Allin Cottrell:
> On Tue, 2 Dec 2014, Sven Schreiber wrote:
>
> I believe the results you get from these calls will depend on the C
> library. We should probably generate an error if you try to sort a
> matrix containing NaNs since the sort order is indeterminate.
It's clear that the position of nans is undefined after sorting, but I
still think that being able to sort is quite useful/important. Whether
the nans then show up in the beginning or end or in the middle is
irrelevant and the user's responsibility. So almost all of the functions
are ok IMO. Only msortby() does something very odd.
It's more complicated than that, and msortby is actually not doing
anything odd. What's odd, I gather, is the behavior of the Microsoft C
library in the cases you said came out "OK".
Our qsort callback for comparing doubles is a pretty standard trope:
given doubles a and b we return
(a > b) - (a < b)
Now if one or the other (or both) of a or b is NaN, both of these
comparisons should (one would think) return 0 -- a NaN is neither
greater than nor less than any given value. So the callback will
return 0, which happens to signal equality to qsort. It's true that a
NaN is not equal to any value (even itself), but the way our callback
is set up it should always appear to be equal to its comparator.
Therefore a NaN will never move in the sort, and it will create a
barrier such that sorting occurs only before and after. So the msortby
behavior is expected.
If you're seeing "OK" results from sorting functions other than
msortby. I guess your C library must be returning non-zero from a
greater/less comparison of a NaN and a non-NaN. But on Linux (glibc
2.20) this is what I'm getting:
<gretl>
? matrix in = {2; 1; NA; 0; -5}
Generated matrix in
Warning: generated non-finite values
? matrix check = sort(in) ~ dsort(in) ~ values(in) ~ uniq(in)
Generated matrix check
Warning: generated non-finite values
? print check
check (5 x 4)
1 2 1 2
2 1 2 1
nan nan nan nan
-5 0 -5 0
0 -5 0 -5
</gretl>
That's with our original qsort callback. In yesterday's CVS I tried
experimenting with a more complicated qsort callback which arbitrarily
stipulates that a NaN is bigger than anything else. Using that variant
I get the following:
<gretl>
? print check
check (5 x 4)
-5 nan -5 2
0 2 0 1
1 1 1 nan
2 0 2 0
nan -5 nan -5
</gretl>
I guess this is in a sense convenient but I don't think it's really
justified. It will also slow down sorting.
Allin
4:48 p.m.
Am 03.12.2014 um 16:59 schrieb Allin Cottrell:
It's more complicated than that, and msortby is actually not doing
anything odd. What's odd, I gather, is the behavior of the Microsoft C
library in the cases you said came out "OK".
How can I test the C library behavior? Maybe a simple C code snippet --
but compiling on a non-dev Windows machine is tricky I guess?
Therefore a NaN will never move in the sort, and it will create a
barrier such that sorting occurs only before and after. So the msortby
behavior is expected.
Ok, I see. I searched a bit for other software's behavior, FWIW:
- Matlab: "If A includes any NaN elements, sort places these at the high
end." (http://matlab.izmiran.ru/help/techdoc/ref/sort.html)
- Python/Numpy: "In numpy versions >= 1.4.0 nan values are sorted to the
end. " (http://docs.scipy.org/doc/numpy/reference/generated/numpy.sort.html)
- R: "sort ... na.last for controlling the treatment of NAs. If TRUE,
missing values in the data are put last; if FALSE, they are put first;
if NA, they are removed."
(https://stat.ethz.ch/R-manual/R-devel/library/base/html/sort.html)
- gretl (with series, where admittedly it's about NA not NaNs, but for
example 10/0 also produces a NA there not a NaN...): just ignores the
NA, leaves them in their original position and sorts the entire rest.
...
That's with our original qsort callback. In yesterday's CVS I tried
experimenting with a more complicated qsort callback which arbitrarily
stipulates that a NaN is bigger than anything else. Using that variant
I get the following:
<gretl>
? print check
check (5 x 4)
-5 nan -5 2
0 2 0 1
1 1 1 nan
2 0 2 0
nan -5 nan -5
</gretl>
I guess this is in a sense convenient but I don't think it's really
justified. It will also slow down sorting.
I think there are various options: refuse to sort with NaNs and let the
user deal with the NaNs. Or do as the other software does, put it at the
end or at the start. Or mimick the situation with series and leave them
in their original position. Or treat it as always bigger (if that is
what Matlab does). Any or all of this could be done as an optional user
choice. But the current situation is bad IMHO.
Thanks,
sven
10 p.m.
On Wed, 3 Dec 2014, Sven Schreiber wrote:
Am 03.12.2014 um 16:59 schrieb Allin Cottrell:
>
> It's more complicated than that, and msortby is actually not doing
> anything odd. What's odd, I gather, is the behavior of the Microsoft C
> library in the cases you said came out "OK".
How can I test the C library behavior? Maybe a simple C code snippet --
but compiling on a non-dev Windows machine is tricky I guess?
I can send a little test.exe. I'll do that tomorrow; I'd be quite
interested to find out what's actually happening on Windows.
> Therefore a NaN will never move in the sort, and it will create
a
> barrier such that sorting occurs only before and after. So the msortby
> behavior is expected.
Ok, I see. I searched a bit for other software's behavior, FWIW:
- Matlab: "If A includes any NaN elements, sort places these at the high
end." (http://matlab.izmiran.ru/help/techdoc/ref/sort.html)
[...]
Thanks, this is useful info.
I think there are various options: refuse to sort with NaNs and let
the
user deal with the NaNs. Or do as the other software does, put it at the
end or at the start. Or mimick the situation with series and leave them
in their original position. Or treat it as always bigger (if that is
what Matlab does). Any or all of this could be done as an optional user
choice. But the current situation is bad IMHO.
Granted. Our current behavior is no more defensible than sorting
NaNs high, and less convenient for the user. There's "safety in
numbers", proverbially, so maybe we'll just go with my modification
in yesterday's CVS and sort NaNs high like Matlab. This should be
documented, of course.
Allin
5:59 p.m.
On Wed, 3 Dec 2014, Sven Schreiber wrote:
Am 03.12.2014 um 16:59 schrieb Allin Cottrell:
>
> It's more complicated than that, and msortby is actually not doing
> anything odd. What's odd, I gather, is the behavior of the Microsoft C
> library in the cases you said came out "OK".
How can I test the C library behavior? Maybe a simple C code snippet --
but compiling on a non-dev Windows machine is tricky I guess?
I tried a little test program on Windows 8. It turns out that the C
library does return 0 for all comparisons involving a NaN (>, < and
==), as it should. Nonetheless, qsort() with our original simple
callback sorts NaN (invoked via 0.0/0.0) to the start of an array of
doubles. I don't know how that happens!
Allin
8:05 p.m.
Am 04.12.2014 um 18:59 schrieb Allin Cottrell:
On Wed, 3 Dec 2014, Sven Schreiber wrote:
> Am 03.12.2014 um 16:59 schrieb Allin Cottrell:
>
>>
>> It's more complicated than that, and msortby is actually not doing
>> anything odd. What's odd, I gather, is the behavior of the Microsoft C
>> library in the cases you said came out "OK".
>
> How can I test the C library behavior? Maybe a simple C code snippet --
> but compiling on a non-dev Windows machine is tricky I guess?
I tried a little test program on Windows 8. It turns out that the C
library does return 0 for all comparisons involving a NaN (>, < and
==), as it should. Nonetheless, qsort() with our original simple
callback sorts NaN (invoked via 0.0/0.0) to the start of an array of
doubles. I don't know how that happens!
sorry for being clueless, but what is qsort() -- gretl's internal call
to a quicksort algorithm perhaps?
-s
8:27 p.m.
On Thu, 4 Dec 2014, Sven Schreiber wrote:
Am 04.12.2014 um 18:59 schrieb Allin Cottrell:
> On Wed, 3 Dec 2014, Sven Schreiber wrote:
>
>> Am 03.12.2014 um 16:59 schrieb Allin Cottrell:
>>
>>>
>>> It's more complicated than that, and msortby is actually not doing
>>> anything odd. What's odd, I gather, is the behavior of the Microsoft C
>>> library in the cases you said came out "OK".
>>
>> How can I test the C library behavior? Maybe a simple C code snippet --
>> but compiling on a non-dev Windows machine is tricky I guess?
>
> I tried a little test program on Windows 8. It turns out that the C
> library does return 0 for all comparisons involving a NaN (>, < and
> ==), as it should. Nonetheless, qsort() with our original simple
> callback sorts NaN (invoked via 0.0/0.0) to the start of an array of
> doubles. I don't know how that happens!
sorry for being clueless, but what is qsort() -- gretl's internal call
to a quicksort algorithm perhaps?
qsort is the standard C library quicksort function (defined in
stdlib.h). Gretl calls it for sorting -- we haven't felt a need for
our own quicksort implementation. But to use qsort you need to supply
your own comparison function for the objects you're sorting (what I'm
calling the "callback").
Allin
12:54 p.m.
New subject: Bug in "Dataset structure..."
I compacted an hourly dataset to daily (7 days) and tried to assign it
the correct dates. They start at 2012-04-01 but I cannot assign this date.
When I go to /Data/Datset structure..., I write the starting
observation at this date, and click on the forward button, the date is
changed to 2009-07-07.
--
Firma Arista
*Ignacio Díaz-Emparanza*
Zuzendaria/Director
ignacio.diaz-emparanza(a)ehu.es <mailto:ignacio.diaz-emparanza@ehu.es>
94 6013732
*EKONOMIA APLIKATUA III SAILA (EKONOMETRIA ETA ESTATISTIKA)/
DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA)
UPV/EHU*
Avda. Lehendakari Aguirre, 83 | 48015 BILBAO
*T.: +34 946013740* | *F.: +34 946013754*
*www.ea3.ehu.es* <http://www.ea3.ehu.es>
ERNE! Baliteke mezu honen zatiren bat edo mezu osoa legez babestuta
egotea. Mezuak badu bere hartzailea. Okerreko helbidera heldu bada
(helbidea gaizki idatzi, transmisioak huts egin) eman abisu igorleari,
korreo honi erantzunda. Kontuz! Mezua ez bada zuretzat, ez erabili, ez
zabaldu beste inori, ez kopiatu eta ez baliatu.
¡ATENCIÓN! Este mensaje contiene información privilegiada o confidencial
a la que sólo tiene derecho a acceder el destinatario. Si usted lo
recibe por error le agradeceríamos que no hiciera uso de la información
y que se pusiese en contacto con el remitente.
E-mail hau inprimatu baino lehen egiaztatu inprimatzeko beharra.
Antes de imprimir este e-mail piense bien si es necesario hacerlo.
4:06 p.m.
New subject: Bug in "Dataset structure..."
On Fri, 5 Dec 2014, Ignacio Diaz-Emparanza wrote:
I compacted an hourly dataset to daily (7 days) and tried to assign
it the
correct dates. They start at 2012-04-01 but I cannot assign this date.
When I go to /Data/Datset structure..., I write the starting observation at
this date, and click on the forward button, the date is changed to
2009-07-07.
Thanks for the report; that's now fixed in CVS.
Allin
4:02 p.m.
New subject: small bug at periodogram zoom
When you use a zoom in a periodogram, the x-axis scale at the bottom
(frequency) is reduced apropriately but not the scale at the top of the
graph (periods).
(Seen on Ubuntu linux 14.04 with gretl from current CVS)
--
Firma Arista
*Ignacio Díaz-Emparanza*
Zuzendaria/Director
ignacio.diaz-emparanza(a)ehu.es <mailto:ignacio.diaz-emparanza@ehu.es>
94 6013732
*EKONOMIA APLIKATUA III SAILA (EKONOMETRIA ETA ESTATISTIKA)/
DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA)
UPV/EHU*
Avda. Lehendakari Aguirre, 83 | 48015 BILBAO
*T.: +34 946013740* | *F.: +34 946013754*
*www.ea3.ehu.es* <http://www.ea3.ehu.es>
ERNE! Baliteke mezu honen zatiren bat edo mezu osoa legez babestuta
egotea. Mezuak badu bere hartzailea. Okerreko helbidera heldu bada
(helbidea gaizki idatzi, transmisioak huts egin) eman abisu igorleari,
korreo honi erantzunda. Kontuz! Mezua ez bada zuretzat, ez erabili, ez
zabaldu beste inori, ez kopiatu eta ez baliatu.
¡ATENCIÓN! Este mensaje contiene información privilegiada o confidencial
a la que sólo tiene derecho a acceder el destinatario. Si usted lo
recibe por error le agradeceríamos que no hiciera uso de la información
y que se pusiese en contacto con el remitente.
E-mail hau inprimatu baino lehen egiaztatu inprimatzeko beharra.
Antes de imprimir este e-mail piense bien si es necesario hacerlo.
4:31 p.m.
New subject: small bug at periodogram zoom
On 09/12/14 17:02, Ignacio Diaz-Emparanza wrote:
When you use a zoom in a periodogram, the x-axis scale at the bottom
(frequency) is reduced apropriately but not the scale at the top of
the graph (periods).
(Seen on Ubuntu linux 14.04 with gretl from current CVS)
Sorry, there is also another question related to this. When I try to do
a peiodogram with "radians" at the x-axis, and try a zoom I obtain this
error:
"/home/ignacio/.gretl/zoomplot.gp", line 1387: warning: difficulty
making room for xtic labels
--
Firma Arista
*Ignacio Díaz-Emparanza*
Zuzendaria/Director
ignacio.diaz-emparanza(a)ehu.es <mailto:ignacio.diaz-emparanza@ehu.es>
94 6013732
*EKONOMIA APLIKATUA III SAILA (EKONOMETRIA ETA ESTATISTIKA)/
DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA)
UPV/EHU*
Avda. Lehendakari Aguirre, 83 | 48015 BILBAO
*T.: +34 946013740* | *F.: +34 946013754*
*www.ea3.ehu.es* <http://www.ea3.ehu.es>
ERNE! Baliteke mezu honen zatiren bat edo mezu osoa legez babestuta
egotea. Mezuak badu bere hartzailea. Okerreko helbidera heldu bada
(helbidea gaizki idatzi, transmisioak huts egin) eman abisu igorleari,
korreo honi erantzunda. Kontuz! Mezua ez bada zuretzat, ez erabili, ez
zabaldu beste inori, ez kopiatu eta ez baliatu.
¡ATENCIÓN! Este mensaje contiene información privilegiada o confidencial
a la que sólo tiene derecho a acceder el destinatario. Si usted lo
recibe por error le agradeceríamos que no hiciera uso de la información
y que se pusiese en contacto con el remitente.
E-mail hau inprimatu baino lehen egiaztatu inprimatzeko beharra.
Antes de imprimir este e-mail piense bien si es necesario hacerlo.
4:34 p.m.
New subject: small bug at periodogram zoom
On Tue, 9 Dec 2014, Ignacio Diaz-Emparanza wrote:
On 09/12/14 17:02, Ignacio Diaz-Emparanza wrote:
> When you use a zoom in a periodogram, the x-axis scale at the bottom
> (frequency) is reduced apropriately but not the scale at the top of the
> graph (periods).
>
> (Seen on Ubuntu linux 14.04 with gretl from current CVS)
Sorry, there is also another question related to this. When I try to do a
peiodogram with "radians" at the x-axis, and try a zoom I obtain this error:
"/home/ignacio/.gretl/zoomplot.gp", line 1387: warning: difficulty making
room for xtic labels
How important is it to be able to zoom a periodogram plot? This would
be extremely difficult to get right; much the simplest solution would
be to remove the zoom option.
Allin
4:55 p.m.
New subject: small bug at periodogram zoom
On 09/12/14 17:34, Allin Cottrell wrote:
On Tue, 9 Dec 2014, Ignacio Diaz-Emparanza wrote:
> On 09/12/14 17:02, Ignacio Diaz-Emparanza wrote:
>> When you use a zoom in a periodogram, the x-axis scale at the bottom
>> (frequency) is reduced apropriately but not the scale at the top of the
>> graph (periods).
>>
>> (Seen on Ubuntu linux 14.04 with gretl from current CVS)
> Sorry, there is also another question related to this. When I try to do a
> peiodogram with "radians" at the x-axis, and try a zoom I obtain this
error:
>
> "/home/ignacio/.gretl/zoomplot.gp", line 1387: warning: difficulty making
> room for xtic labels
How important is it to be able to zoom a periodogram plot? This would
be extremely difficult to get right; much the simplest solution would
be to remove the zoom option.
Not very important. I was trying to detect an annual peak (period 365)
in daily data (7 days) with a series of 2700 obs. But you always may
have a look at the numbers.
--
Firma Arista
*Ignacio Díaz-Emparanza*
Zuzendaria/Director
ignacio.diaz-emparanza(a)ehu.es <mailto:ignacio.diaz-emparanza@ehu.es>
94 6013732
*EKONOMIA APLIKATUA III SAILA (EKONOMETRIA ETA ESTATISTIKA)/
DEPARTAMENTO DE ECONOMÍA APLICADA III (ECONOMETRÍA Y ESTADÍSTICA)
UPV/EHU*
Avda. Lehendakari Aguirre, 83 | 48015 BILBAO
*T.: +34 946013740* | *F.: +34 946013754*
*www.ea3.ehu.es* <http://www.ea3.ehu.es>
ERNE! Baliteke mezu honen zatiren bat edo mezu osoa legez babestuta
egotea. Mezuak badu bere hartzailea. Okerreko helbidera heldu bada
(helbidea gaizki idatzi, transmisioak huts egin) eman abisu igorleari,
korreo honi erantzunda. Kontuz! Mezua ez bada zuretzat, ez erabili, ez
zabaldu beste inori, ez kopiatu eta ez baliatu.
¡ATENCIÓN! Este mensaje contiene información privilegiada o confidencial
a la que sólo tiene derecho a acceder el destinatario. Si usted lo
recibe por error le agradeceríamos que no hiciera uso de la información
y que se pusiese en contacto con el remitente.
E-mail hau inprimatu baino lehen egiaztatu inprimatzeko beharra.
Antes de imprimir este e-mail piense bien si es necesario hacerlo.
11:28 a.m.
Am 04.12.2014 um 21:27 schrieb Allin Cottrell:
On Thu, 4 Dec 2014, Sven Schreiber wrote:
>>
>> I tried a little test program on Windows 8. It turns out that the C
>> library does return 0 for all comparisons involving a NaN (>, < and
>> ==), as it should. Nonetheless, qsort() with our original simple
>> callback sorts NaN (invoked via 0.0/0.0) to the start of an array of
>> doubles. I don't know how that happens!
>
> sorry for being clueless, but what is qsort() -- gretl's internal call
> to a quicksort algorithm perhaps?
qsort is the standard C library quicksort function (defined in
stdlib.h). Gretl calls it for sorting -- we haven't felt a need for
our own quicksort implementation. But to use qsort you need to supply
your own comparison function for the objects you're sorting (what I'm
calling the "callback").
Ok, anything I could do to investigate this issue further?
thanks,
sven
11:32 p.m.
On Sun, 14 Dec 2014, Sven Schreiber wrote:
Am 04.12.2014 um 21:27 schrieb Allin Cottrell:
> On Thu, 4 Dec 2014, Sven Schreiber wrote:
>
>>>
>>> I tried a little test program on Windows 8. It turns out that the C
>>> library does return 0 for all comparisons involving a NaN (>, < and
>>> ==), as it should. Nonetheless, qsort() with our original simple
>>> callback sorts NaN (invoked via 0.0/0.0) to the start of an array of
>>> doubles. I don't know how that happens!
>>
>> sorry for being clueless, but what is qsort() -- gretl's internal call
>> to a quicksort algorithm perhaps?
>
> qsort is the standard C library quicksort function (defined in
> stdlib.h). Gretl calls it for sorting -- we haven't felt a need for
> our own quicksort implementation. But to use qsort you need to supply
> your own comparison function for the objects you're sorting (what I'm
> calling the "callback").
Ok, anything I could do to investigate this issue further?
Not really, if you're OK with the current CVS gretl behavior,
whereby NaNs are sorted high on all platforms.
Allin
5:13 p.m.
Am 15.12.2014 um 00:32 schrieb Allin Cottrell:
On Sun, 14 Dec 2014, Sven Schreiber wrote:
>
> Ok, anything I could do to investigate this issue further?
Not really, if you're OK with the current CVS gretl behavior,
whereby NaNs are sorted high on all platforms.
sounds good, thanks.
sven
3657
days inactive
3672
days old
17 comments
3 participants
participants (3)
-
Allin Cottrell -
Ignacio Diaz-Emparanza -
Sven Schreiber