5:28 p.m.
Dear all,
Gretl have several possibilities for creating
of large numbers of new variables with autonames:
square(list)/ square list
list^list
dummify(discreteVar)
etc.
Below is the script to illustrate the problem
open denmark.gdt
sq_LRM = IBO/IDE
# Descriptive label-> IBO/IDE
square LRM
# Descriptive label-> = LRM squared
series LRY_LRM = IBO
# Descriptive label-> IBO
list x = LRY LRM
square(x,1)
# Descriptive label of LRY_LRM -> LRY times LRM
With denmark.gdt everything is obvious
But what if we work with a large data set
with hundreds of variables and hundreds of
new dummies and interactions some replacements
can be easily overlooked
I think there should be some tools to prevent
casual rewriting variables
May be a precheck function to find possible name
duplicating or rename() to rename variables having
underscore in names: this will prevent duplicating,
since all autonames contain '_'
I'll be glad for opinions
Oleh
6:20 p.m.
On Fri, 23 Sep 2016, oleg_komashko(a)ukr.net wrote:
Dear all,
Gretl have several possibilities for creating
of large numbers of new variables with autonames:
square(list)/ square list
list^list
dummify(discreteVar)
etc.
Below is the script to illustrate the problem
open denmark.gdt
sq_LRM = IBO/IDE
# Descriptive label-> IBO/IDE
square LRM
# Descriptive label-> = LRM squared
series LRY_LRM = IBO
# Descriptive label-> IBO
list x = LRY LRM
square(x,1)
# Descriptive label of LRY_LRM -> LRY times LRM
With denmark.gdt everything is obvious
But what if we work with a large data set
with hundreds of variables and hundreds of
new dummies and interactions some replacements
can be easily overlooked
I think there should be some tools to prevent
casual rewriting variables
May be a precheck function to find possible name
duplicating or rename() to rename variables having
underscore in names: this will prevent duplicating,
since all autonames contain '_'
I'll be glad for opinions
First point: we have a "rename" command, and it can be used for
mass-renaming of series whose names contain an underscore. Here's a
simple illustration:
<hansl>
open data4-10
list D = dataset
list D print
list L = logs(D)
list L print
loop foreach i L -q
string orig = "$i"
string newname = strsub(orig, "_", "")
rename @orig @newname
endloop
list L print
</hansl>
Second point: we do have some code in place to try to ensure that we
don't overwrite existing series by the operations that create a list
of new, auto-named series (we modify the auto-name to avoid
collisions). I can't guarantee, offhand, that this mechanism is
always employed when it should be, or if it's employed that it
always works as intended. However, if we let the null hypothesis be
that it always works fine, do you have evidence to the contrary?
Allin
11:51 p.m.
Dear Allin,
I thought I found such example in the first message:
open denmark.gdt
series LRY_LRM = IBO
Head of LRY_LRM :
LRY_LRM
1974:1 0.1547356
1974:2 0.1779912
list x = LRY LRM
square(x,1)
new head of LRY_LRM:
LRY_LRM
1974:1 68.67460
1974:2 68.16070
rename does the check but square overwrites
Oleh
23 вересня 2016, 21:20:43, від "Allin Cottrell" <cottrell(a)wfu.edu>:
On Fri, 23 Sep 2016, oleg_komashko(a)ukr.net wrote:
> Dear all,
> Gretl have several possibilities for creating
> of large numbers of new variables with autonames:
> square(list)/ square list
> list^list
> dummify(discreteVar)
> etc.
> Below is the script to illustrate the problem
> open denmark.gdt
>
> sq_LRM = IBO/IDE
>
> # Descriptive label-> IBO/IDE
>
> square LRM
>
> # Descriptive label-> = LRM squared
>
> series LRY_LRM = IBO
>
> # Descriptive label-> IBO
>
> list x = LRY LRM
>
> square(x,1)
>
> # Descriptive label of LRY_LRM -> LRY times LRM
>
> With denmark.gdt everything is obvious
> But what if we work with a large data set
> with hundreds of variables and hundreds of
> new dummies and interactions some replacements
> can be easily overlooked
> I think there should be some tools to prevent
> casual rewriting variables
> May be a precheck function to find possible name
> duplicating or rename() to rename variables having
> underscore in names: this will prevent duplicating,
> since all autonames contain '_'
> I'll be glad for opinions
First point: we have a "rename" command, and it can be used for
mass-renaming of series whose names contain an underscore. Here's a
simple illustration:
open data4-10
list D = dataset
list D print
list L = logs(D)
list L print
loop foreach i L -q
string orig = "$i"
string newname = strsub(orig, "_", "")
rename @orig @newname
endloop
list L print
Second point: we do have some code in place to try to ensure that we
don't overwrite existing series by the operations that create a list
of new, auto-named series (we modify the auto-name to avoid
collisions). I can't guarantee, offhand, that this mechanism is
always employed when it should be, or if it's employed that it
always works as intended. However, if we let the null hypothesis be
that it always works fine, do you have evidence to the contrary?
Allin
_______________________________________________
Gretl-devel mailing list
Gretl-devel(a)lists.wfu.edu
http://lists.wfu.edu/mailman/listinfo/gretl-devel
1:32 p.m.
On Sat, 24 Sep 2016, oleg_komashko(a)ukr.net wrote:
Dear Allin,
I thought I found such example
(that is, an example of a list-generating command overwriting an
existing series)
in the first message:
(that is,
http://lists.wfu.edu/pipermail/gretl-devel/2016-September/006938.html
)
open denmark.gdt
series LRY_LRM = IBO
Head of LRY_LRM :
LRY_LRM
1974:1 0.1547356
1974:2 0.1779912
list x = LRY LRM
square(x,1)
new head of LRY_LRM:
LRY_LRM
1974:1 68.67460
1974:2 68.16070
You're right. In my reply to your original post I was misremembering
the nature of the check that we perform when doing things like
square, lags, log and so on (functions that create a set of
automatically named series).
What we ensure is that the auto-names are unique (the set of
auto-names contains no duplicates), NOT that we don't overwite any
existing series.
Here's how the uniqueness issue arises. When you use ldiff, "ld_"
gets stuck in front of the original name, when you generate
cross-products two series names get stuck together with an
underscore, and so on. If your original series names are long, they
may have to truncated to meet the 31-character limit on gretl
identifiers, and two or more generated names may be the same after
truncation. If that happens, we apply a "mangling" rule to ensure
uniqueness.
As for overwriting existing series, avoiding that (if you want to
avoid it -- in some cases, of course, overwriting may be
intentional), that's up to the user. If you have (as you describe in
your original post) "a large data set with hundreds of variables and
hundreds of new dummies and interactions", you would be well advised
to take charge of the naming yourself. You can generate a list of
transformations from a list of inputs using a "foreach" loop plus
the genseries() function.
I suppose we could add a "set" variable -- maybe something like
set auto_overwite off
such that if any built-in list-generating function would overwrite
an existing series it terminates with an error.
Allin
1:59 p.m.
New subject: how to avoid unintentional rewriting series?
On Sun, 25 Sep 2016, Allin Cottrell wrote:
I suppose we could add a "set" variable -- maybe something
like
set auto_overwite off
such that if any built-in list-generating function would
overwrite an existing series it terminates with an error.
Or maybe, introduce a --safe option to commands such as diff, ldiff, lags
etc.
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
3:56 p.m.
New subject: how to avoid unintentional rewriting series?
On Sun, 25 Sep 2016, Riccardo (Jack) Lucchetti wrote:
On Sun, 25 Sep 2016, Allin Cottrell wrote:
> I suppose we could add a "set" variable -- maybe something like
>
> set auto_overwite off
>
> such that if any built-in list-generating function would overwrite an
> existing series it terminates with an error.
Or maybe, introduce a --safe option to commands such as diff, ldiff, lags
etc.
That occurred to me too, and it would be fine for _commands_ of this
sort, but would leave out functions that have a similar effect. A
single "set" variable could be made to apply to both cases.
(Internally, both invoke the mechanisms in the library source file
transforms.c, so that would be the place to insert a conditional
guard.)
Allin
3:38 p.m.
Dear Allin, Dear Riccardo,
25 вересня 2016, 16:33:06, від "Allin Cottrell" <cottrell(a)wfu.edu>:
...
set auto_overwite off
25 вересня 2016, 16:59:59, від "Riccardo (Jack) Lucchetti"
<r.lucchetti(a)univpm.it>:
....
Or maybe, introduce a --safe option to commands such as diff, ldiff,
lags
etc.
Sounds very promising!
Oleh
P.S. I personally didn't encounter so large datasets, but my own son
during his staging at mobile operator company was asked to
estimate a logistic regression with a ~200 levels discrete independent variable
3009
days inactive
3011
days old
6 comments
3 participants
participants (3)
-
Allin Cottrell -
oleg_komashko＠ukr.net -
Riccardo (Jack) Lucchetti