Mann-Kendall test
by Sven Schreiber
Hi,
I've just read about the non-parametric Mann-Kendall test for a trend
(hence my recent postings about Kendall's tau and ties in a series).
It looked pretty easy, so I'm attaching an attempt at a basic
implementation. Full disclosure: I've used the description from
https://cran.r-project.org/web/packages/trend/vignettes/trend.pdf (only
the basic version in section 1.1).
All you have to do is call "MannKendall(x)", where x is your series of
interest.
I'd be happy about feedback regarding cross-checks and correctness (for
example in the above document in equ. (4) the case S>0 wrongly appears
twice, and I had to make an educated guess where to switch the
signs...), whether it should become a function package and so on.
thanks,
sven
7 years, 6 months
Preserve row and col names of a matrix after functions
by Henrique Andrade
Dear Gretl Team,
Is it possible to preserve row and col names of a matrix after
"mreverse" and "trimr" functions? Please take a look at the following
code:
<hansl>
matrix M = {1, 2; 2, 1; 4, 1}
strings Row = defarray("Row1", "Row2", "Row3")
strings Col = defarray("Col1", "Col2")
rownames(M, Row)
colnames(M, Col)
matrix Msortby = msortby(M, 1)
matrix Mreverse = mreverse(M)
matrix Mtrimr = trimr(M, 1, 1)
print M Msortby Mreverse Mtrimr
</hansl>
In my humble opinion, "reverse" and "trimr" could have the same
behavior of "msortby", i.e., keep col and row names as in "msortby".
Best,
Henrique Andrade
8 years, 1 month
Pre-specifying names of list members returned from a function call
by Sven Schreiber
Hi,
see feature request #85 (by myself I must admit):
https://sourceforge.net/p/gretl/feature-requests/85/
Currently a function with a list as a return type must internally set
the names of the list members (series) and the those names are
automatically exported to the function caller. If series with those
names already exist in the outer scope (at the caller level), they would
be overwritten. This is an exception of gretl's principle of function
encapsulation.
I'm suggesting: To provide a cleaner solution of this name clash
problem, such a list-returning function should also accept a string
array argument. The string elements in this array would be used by the
function as the names of the exported list members.
In this context it would be useful to have a built-in easy mechanism to
take the labels from the string array and use it for the returned list
(inside the user-defined function). That is the concrete request right now.
I'm not suggesting to make this thing mandatory any time soon, because
of the backward-compatibility issue.
thanks,
sven
8 years, 1 month
Re: [Gretl-devel] MWU package group (Re: Proposal for function package cleanup)
by Sven Schreiber
Hi Yi-Nung,
this is now one year overdue. (I had forgotten about it in the meantime,
and just now stumbled again over the three MWU/mwu/mwu_dummy packages on
the server.)
To workaround the problem that I would forget again, I am now going to
move the "MWU" (uppercase) and "mwu_dummy" packages to the staging area,
so they won't be accessible from within gretl anymore. They will still
be online for some time at:
http://ricardo.ecn.wfu.edu/gretl/staging_fnfiles/
You can consolidate these two removed packages into the "mwu"
(lowercase) package with a new version number and upload the
consolidated version.
Thanks,
sven
Am 13.07.2015 um 02:48 schrieb yinung at Gmail:
> sven, I will integrate those three into one. Please reserve "mwu" for
> these Mann-Whitney test.
> Thanks.
>
> Yi-Nung
>
> 2015-07-13 4:32 GMT+08:00 Sven Schreiber <svetosch(a)gmx.net
> <mailto:svetosch@gmx.net>>:
>
> Am 18.06.2015 um 09:20 schrieb Sven Schreiber:
>
> >>
> >> 8) MWU / mwu / mwu_dummy (Yang): Yi-Nung, please consolidate
> these very
> >> similar things into one package with different options. Please
> respond
> >> to this message, otherwise I propose to remove two of these three
> packages.
> >
> > still waiting for this one
> >
>
> No news here.
>
> Yi-Nung, are there any plans on how to consolidate these three packages?
>
> My "threat" (removing two of three) still stands... But I'm open for
> arguments or deadline extensions, if there are plans and / or good
> reasons.
>
> thanks,
> sven
>
>
8 years, 2 months
Suggestion for new string-checking built-in function
by Sven Schreiber
Hi,
it often occurs (at least in my own and in Artur's hansl programming)
that you want to check whether a certain string variable contains some
shorter string. For example whether variable 's1' contains the string
"USA", without being necessarily exactly equal to "USA".
Currently to my knowledge this can/must be done using the strstr()
function, either like this (producing non-negative integer output):
strlen(strstr(s1, "USA"))
or like this:
strstr(s1, "USA") != ""
I find both variants relatively clumsy, so I'm asking for syntactic
sugar which would be equivalent to the second expression above. For
example an 'instring()' function that would be loosely analogous to the
inbundle() or inlist() functions.
So the above example would simply be rewritten as:
instring(s1, "USA")
But maybe Jack or somebody else have an alternative suggestions what
could be done instead with existing hansl constructs.
Thanks,
sven
8 years, 4 months
slowness with panel sample restrictions (?) or pxnobs() function (?)
by Sven Schreiber
hi,
in relation to the feature request about cross-section dependence
testing in panels I've encountered some performance bottleneck.
Here's working code given some panel variable 'u':
<hansl>
series unit = $obsmajor
N = max(unit)
#### more elegant attempt (but slow for some reason...)
rhos = zeros(N,N)
Tcomms = zeros(N,N)
loop i=1..N -q
loop j=(i+1)..N -q
smpl (unit == i || unit == j) --restrict --replace
smpl pxnobs(u) == 2 --restrict # both units have values there
matrix uij = { u } # hope it's stacked, so that upper half is unit i
Tcomms[i,j] = rows(uij) / 2
rhos[i,j] = corr( uij[1: Tcomms[i,j]], uij[Tcomms[i,j] + 1 : ] )
endloop
endloop
# Pesaran's CD (unbalanced)
scalar CD = sqrt( 2 / (N*(N-1)) ) * sum( sqrt(Tcomms) .* rhos )
</hansl>
This takes quite long to run (several seconds for datasets like
abdata.gdt). I've had a less elegant implementation before with more
code and more copying to temporary matrices and so on (including the
nested loop) which was much faster.
I'm wondering whether perhaps the sample restriction with the pxnobs()
function is slow.
thanks,
sven
8 years, 4 months
Bug in mcorr() with constant input
by Sven Schreiber
Hi,
I'm glad this bug only seems to affect mcorr(), and not corr():
<hansl>
nulldata 6
series c = 2
eval corr(c, c) # gives NA
matrix mc = { c }
eval corr(mc, mc) # gives NA
eval mcorr(mc ~ mc) # gives zero matrix
</hansl>
I think the first two results are correct (correlation is undefined for
non-varying inputs), but the mcorr output of zero is wrong, should also
be NAs.
This is with a recent snapshot.
Thanks,
sven
8 years, 4 months
Modtests as function packages
by Sven Schreiber
Hi,
there's a feature request about a certain type of diagnostic test (see
https://sourceforge.net/p/gretl/feature-requests/97/).
This message here isn't about this specific request or test, but about a
design issue in relation to function packages. It is fairly easy to
implement such a test as a function package, but I have "meta request"
or suggestion for an interface that would provide a tighter integration
and better user experience.
AFAICS, if a new diagnostic test such as "newtest" is implemented as a
contributed function (and package) 'newtest()', then the use would be as
follows:
<current>
include newtest.gfn
ols y 0 x # estimate the model to be tested
matrix testresult = newtest($uhat, $xlist, <other needed input>)
</current>
There's at least two things that bother me here: First the usage is
completely different from gretl's built-in "modtest" command. Secondly
the user needs to know exactly what arguments the 'newtest()' function
needs. This complicates things.
Re the first issue, my suggestion is to extend the "modtest" command to
also apply to installed function packages. So if gretl finds a "modtest
--newtest" line, it would realize that there's no built-in option
"--newtest", and then instead of throwing an error right away, it would
check whether there's an installed function package "newtest.gfn". And
if that's the case, it would execute newtest(...).
As a follow-up issue, because "modtest" provides results in the
accessors $pvalue and $test (AFAIK), the newtest() function would also
be required to return that, presumably wrapped in a bundle. And gretl
would make sure that these return values are then copied to the
outer-scope $pvalue and $test accessors.
Secondly, because "modtest --newtest" wouldn't pass any arguments to the
newtest() function explicitly, I think the convention should be that it
is invoked as "newtest($modbundle)", where "$modbundle" would be a new
accessor that is just a bundle of all the other $-accessors that are
made available by gretl after estimating a model.
BTW, this automatic $modbundle after estimation would be good
irrespective of the specific case of the "modtest" command.
OK, I hope I've made myself clear. I know you're all on vacation and
this is a lenghty post, but it isn't urgent, just a pretty radical
suggestion for the future.
thanks,
sven
8 years, 4 months
MIDAS in Gretl: layout suggestions
by Henrique Andrade
Dear Gretl Team,
I just started reading "MIDAS in gretl: where are we?" by Allin and
Jack and I really happy to see this development. I didn't finish yet,
but I would like to make some suggestions. Please keep in mind (1)
I'm not an expert in this field and, (2) the suggestions are just...
suggestions ;-)
Allin & Jack (2016):
| Since a gretl dataset formally handles only a single data frequency
| it may seem that we have a problem. However, we have adopted a
| straightforward solution: a higher frequency series x_H is represented
| by a set of m series, each holding the value of x_H in a sub-period of
| the “base” (lower-frequency) period (where m is the ratio of the
| higher frequency to the lower).
I think this is a good "internal" solution, but it doesn't look good
in aesthetic terms. IMHO it looks like a temporary patch and is a
little bit far from the general Gretl user interface. When I looked at
"Figure 1: A slice of MIDAS data" I immediately thought: "This
software can't handle different frequencies in a native way and this
figure represents a workaround". I would like to suggest one of these
represantations (attached I'm sending a pdf with more complete
examples):
# Representation 1:
month indpro quarter gdpc96
1947M01 14,3650
1947M02 14,2811 1947Q1 1934,47
1947M03 14,1973
1947M04 14,3091
1947M05 14,3091 1947Q2 1932,28
1947M06 14,2532
1947M07 14,4209
1947M08 14,3091 1947Q3 1930,31
1947M09 14,2253
1947M10 14,8121
1947M11 14,7562 1947Q4 1960,7
1947M12 14,5606
1948M01 14,7563
1948M02 14,9240 1948Q1 1989,54
1948M03 14,8960
1948M04 15,2313
1948M05 15,0357 1948Q2 2021,85
1948M06 14,7842
# Representation 2:
monthly | quarterly
---------------------------------------------
indpro | gdpc96
1947M01 14,3650 |
1947M02 14,2811 | 1947Q1 1934,47
1947M03 14,1973 |
1947M04 14,3091 |
1947M05 14,3091 | 1947Q2 1932,28
1947M06 14,2532 |
1947M07 14,4209 |
1947M08 14,3091 | 1947Q3 1930,31
1947M09 14,2253 |
1947M10 14,8121 |
1947M11 14,7562 | 1947Q4 1960,70
1947M12 14,5606 |
1948M01 14,7563 |
1948M02 14,9240 | 1948Q1 1989,54
1948M03 14,8960 |
1948M04 15,2313 |
1948M05 15,0357 | 1948Q2 2021,85
1948M06 14,7842 |
Allin & Jack (2016):
| So let us define a MIDAS list: this is a list of m series holding
| per-period values of a single high-frequency series, arranged in the
| order of most recent first, as illustrated above. Given the dataset
| shown in Figure 1, an example of a correctly formulated MIDAS list
| would be
| list INDPRO = indpro_m3 indpro_m2 indpro_m1
| Or, since the monthly observations are already in the required order,
| we could define the list by means of a “wildcard”:
| list INDPRO = indpro_m*
| Having created such a list, one can use the setinfo command to tell
| gretl that it’s a bonafide MIDAS list:
| setinfo INDPRO --midas
I think this strategy runs against on of our main feature (according
to Gretl User’s Guide): "User-friendly: Gretl offers an intuitive user
interface; it is very easy to get up and running with econometric
analysis". We could do this operation internally, outside the view of
our users. With that statement in mind I think it could be better to
do this internally, outside user's view.
Let me explain better: internally the solution would be exactly the
one proposed by Allin and Jack, but the "indpro" will looks like a
series on the main Gretl window. But how will could we represent
different frequencies in Gretl's main window? We can use exactly the
same strategy we use with databases. Take a look at the pdf I had
attached to this mail.
Um abraço,
Henrique Andrade
8 years, 4 months
