11:11 a.m.
Dear all,
I've stumbled about an annoying issue.
The following script loads the grunfeld data set which includes the
string-valued series "firmname".
Restricting the data set to a single firm (Chrysler), re-opening the
restricted and stored data set shows that "firmname" still comprises the
old meta data as the last print_array() function call shows. Still _all_
firm names are printed, and thus somehow stored on "firmname".
I guess the issue has (partly) to do with the (at least for me) special
behavior of strvals() which _always_ prints all distinct values of a
string-valued series irrespective of the underlying data set is
restricted or not.
This is with current git under Ubuntu.
<hansl>
clear
set verbose off
open grunfeld.gdt -q
function void print_array (const strings S)
loop foreach i S
print "$i"
endloop
end function
printf "\nPrint all distinct string-values of 'firmname'.\n"
print_array(strvals(firmname))
smpl firmname == "Chrysler" --restrict
store foo.gdt
printf "\n\nLoad restricted data set\n"
open foo.gdt -q -p
printf "\nPrint distinct string-valued firmnames.\n"
print_array(strvals(firmname)) # still incl. _all_ firm names
</hansl>
Artur
6:02 p.m.
On Wed, 15 Jul 2020, atecon(a)posteo.de wrote:
Hi,
as nobody replied, yet: Can anybody confirm this behaviour?
The reference is to
https://www.mail-archive.com/gretl-users@gretlml.univpm.it/msg14808.html
Yes, I can confirm the behavior described there, but I don't think
it's a bug. Restricting the sample cannot change the fact that
Chrysler is firm 4 of 10 in the grunfeld dataset, and so the
string-valued "firmname" series must comprise at least 10 strings.
Otherwise it would be impossible to undo the restriction.
If you want to destroy that information and reconstitute the string
values on a smaller scale you can
1) Impose the sample restriction
2) Save the reduced dataset as CSV
3) Open the CSV and save as gdt
On a related matter, you can inspect the string values of a subset
of the grunfeld data in a fairly straightforward way:
<hansl>
open grunfeld.gdt
# e.g. restrict to even-numbered firms
smpl firm % 2 == 0 --restrict
strings S2 = strvals(firmname)[values(firmname)]
print S2
</hansl>
Allin
7:30 p.m.
Am 15.07.20 um 20:02 schrieb Allin Cottrell:
On Wed, 15 Jul 2020, atecon(a)posteo.de wrote:
> Hi,
> as nobody replied, yet: Can anybody confirm this behaviour?
The reference is to
https://www.mail-archive.com/gretl-users@gretlml.univpm.it/msg14808.html
Yes, I can confirm the behavior described there, but I don't think
it's a bug. Restricting the sample cannot change the fact that
Chrysler is firm 4 of 10 in the grunfeld dataset, and so the
string-valued "firmname" series must comprise at least 10 strings.
Otherwise it would be impossible to undo the restriction.
Thanks for your response, Allin.
But what about the case when adding the " --permanent" flag? In this
case one cannot undo the restriction but my example would still return a
string array of length 11.
If you want to destroy that information and reconstitute the string
values on a smaller scale you can
1) Impose the sample restriction
2) Save the reduced dataset as CSV
3) Open the CSV and save as gdt
But this was exactly the example I came up with. The only difference is
that I tried to store a gdt instead of a csv file. And here the surprise
comes: When storing a restricted data set as a csv file and re-opening
it, things are as, at least me, I would expect:
<hansl>
smpl firmname == "Chrysler" --restrict
store foo.csv
open foo.csv -q -p
eval strvals(firmname) # [1] "Chrysler"
</hansl>
But once one tries to store things in the gdt or gdtb format, the length
of the array is still of length 11. Why does it or should it matter in
which format things are stored here?
On a related matter, you can inspect the string values of a subset of
the grunfeld data in a fairly straightforward way:
<hansl>
open grunfeld.gdt
# e.g. restrict to even-numbered firms
smpl firm % 2 == 0 --restrict
strings S2 = strvals(firmname)[values(firmname)]
print S2
</hansl>
Ah thanks, good to know. For some reason I rarely make use of this
modulus operator.
8:17 p.m.
On Wed, 15 Jul 2020, Artur Tarassow wrote:
Am 15.07.20 um 20:02 schrieb Allin Cottrell:
> On Wed, 15 Jul 2020, atecon(a)posteo.de wrote:
>
>> Hi,
>> as nobody replied, yet: Can anybody confirm this behaviour?
>
> The reference is to
> https://www.mail-archive.com/gretl-users@gretlml.univpm.it/msg14808.html
>
> Yes, I can confirm the behavior described there, but I don't think it's a
> bug. Restricting the sample cannot change the fact that Chrysler is firm 4
> of 10 in the grunfeld dataset, and so the string-valued "firmname" series
> must comprise at least 10 strings. Otherwise it would be impossible to undo
> the restriction.
>
Thanks for your response, Allin.
But what about the case when adding the " --permanent" flag? In this case one
cannot undo the restriction but my example would still return a string array
of length 11.
I can see a case for shrinking the strings array when the
--permanent option is given, though it's not totally clear-cut.
> If you want to destroy that information and reconstitute the
string values
> on a smaller scale you can
>
> 1) Impose the sample restriction
> 2) Save the reduced dataset as CSV
> 3) Open the CSV and save as gdt
But this was exactly the example I came up with. The only difference is that
I tried to store a gdt instead of a csv file.
But that's a big difference, and it should be obvious why. When
saving as CSV, where and how would we save the additional strings?
Or the numeric codes associated with the strings? (Even if we wanted
to.)
Allin
12:53 p.m.
On Wed, 15 Jul 2020, Allin Cottrell wrote:
On Wed, 15 Jul 2020, Artur Tarassow wrote:
> But what about the case when adding the " --permanent" flag?
I can see a case for shrinking the strings array when the
--permanent option is given, though it's not totally clear-cut.
Here's a follow-up. You could think of this as a prototype of what
we might do internally with a string-valued series on permanent
sub-sampling.
First, a little CSV file:
<file name="strfoo.csv">
x,s
1,"aaa"
2,"bbb"
3,"ddd"
4,"ccc"
5,"aaa"
6,"ddd"
7,"eee"
8,"bbb"
9,"aaa"
</file>
Then a script that sub-samples it permanently and revises the range
and coding of the string variable appropriately:
<hansl>
function void string_recode (series *y, series s)
matrix snums = uniq(s)
strings S = strvals(s)[snums]
y = replace(s, snums, seq(1, nelem(snums)))
stringify(y, S)
end function
open strfoo.csv -q
series scodes = s # get the numeric codes
print -o
Ss = strvals(s)
print Ss
smpl x % 2 == 0 --restrict --permanent
series y
string_recode(&y, s)
series ycodes = y
print -o
Sy = strvals(y)
print Sy
</hansl>
In the main script (but not in a function) one could finish the job
with:
delete s
rename y s
Allin
1:57 p.m.
Am 16.07.20 um 14:53 schrieb Allin Cottrell:
On Wed, 15 Jul 2020, Allin Cottrell wrote:
> On Wed, 15 Jul 2020, Artur Tarassow wrote:
>
>> But what about the case when adding the " --permanent" flag?
>
> I can see a case for shrinking the strings array when the --permanent
> option is given, though it's not totally clear-cut.
Here's a follow-up. You could think of this as a prototype of what we
might do internally with a string-valued series on permanent
sub-sampling.
Sorry for the late reply, Allin. Yes that looks good to me.
But would "permanent sub-sampling" mean that this only applies when
executing the smpl command with the --permanent flag? Or would it also
apply when storing a sub-sampled data set?
Artur
2:32 p.m.
On Fri, 17 Jul 2020, Artur Tarassow wrote:
Am 16.07.20 um 14:53 schrieb Allin Cottrell:
> On Wed, 15 Jul 2020, Allin Cottrell wrote:
>
>> On Wed, 15 Jul 2020, Artur Tarassow wrote:
>>
>>> But what about the case when adding the " --permanent" flag?
>>
>> I can see a case for shrinking the strings array when the --permanent
>> option is given, though it's not totally clear-cut.
>
> Here's a follow-up. You could think of this as a prototype of what we might
> do internally with a string-valued series on permanent sub-sampling.
Sorry for the late reply, Allin. Yes that looks good to me.
But would "permanent sub-sampling" mean that this only applies when executing
the smpl command with the --permanent flag? Or would it also apply when
storing a sub-sampled data set?
I favour doing this only when the --permanent option is given. It's
an information-destroying move, and I can imagine cases where one
wants to save a sub-sample and yet not lose the information in
question. But if you _want_ to lose it, without using --permanent,
then just store in a format other than gdt or gdtb.
Allin
6:02 a.m.
Am 17.07.20 um 16:32 schrieb Allin Cottrell:
On Fri, 17 Jul 2020, Artur Tarassow wrote:
> Am 16.07.20 um 14:53 schrieb Allin Cottrell:
>> On Wed, 15 Jul 2020, Allin Cottrell wrote:
>>
>>> On Wed, 15 Jul 2020, Artur Tarassow wrote:
>>>
>>>> But what about the case when adding the " --permanent" flag?
>>>
>>> I can see a case for shrinking the strings array when the
>>> --permanent option is given, though it's not totally clear-cut.
>>
>> Here's a follow-up. You could think of this as a prototype of what we
>> might do internally with a string-valued series on permanent
>> sub-sampling.
>
> Sorry for the late reply, Allin. Yes that looks good to me.
>
> But would "permanent sub-sampling" mean that this only applies when
> executing the smpl command with the --permanent flag? Or would it also
> apply when storing a sub-sampled data set?
I favour doing this only when the --permanent option is given. It's an
information-destroying move, and I can imagine cases where one wants to
save a sub-sample and yet not lose the information in question. But if
you _want_ to lose it, without using --permanent, then just store in a
format other than gdt or gdtb.
I understand your point, Allin. And getting it worked when using the
--permanent option would be very useful.
But, let me loudly think about some of my use cases -- maybe some others
have to deal with similar ones.
From time to time I have to deal with large panel data sets (size is
about 800 MB or even larger) originally stored as csv. Reading such csv
data sets is _very_ costly as is also joining additional series to it.
So its natural to store such data as gdt or even better in a binary
format (gdtb).
Let's say, I would like to store several sub-sampled data sets of this
large one, then I would approach it by:
<pseudo-hansl>
open large_dataset.csv
matrix assortments = values(assortment)
loop i=1..rows(assortments)
if i > 1
open large_dataset.csv -p -q
endif
smpl assortment == assortments[i] --restrict --permanent
<store "data_for_assortment_$i.gdtb>
endloop
</pseudo-hansl>
In this case one would have to re-open the "large_dataset.csv" multiple
times (ok, fair enough, I could have stored the "large_dataset.csv" in
gdtb format before the loop and load this -- but still would have a copy
of a 'large' data set.
I just would like to remember that given the tendency that larger and
larger data sets become available also in academics, it may be worth to
got a step further. Don't get me wrong, getting work the hansl example
would be cool but if it's difficult to implement "on-the-fly" this can
wait. I could create a feature request ticket for this if worth.
Artur
9:32 p.m.
On Sat, 18 Jul 2020, Artur Tarassow wrote:
Am 17.07.20 um 16:32 schrieb Allin Cottrell:
> On Fri, 17 Jul 2020, Artur Tarassow wrote:
>
>> Am 16.07.20 um 14:53 schrieb Allin Cottrell:
>>> On Wed, 15 Jul 2020, Allin Cottrell wrote:
>>>
>>>> On Wed, 15 Jul 2020, Artur Tarassow wrote:
>>>>
>>>>> But what about the case when adding the " --permanent"
flag?
>>>>
>>>> I can see a case for shrinking the strings array when the --permanent
>>>> option is given, though it's not totally clear-cut.
>>>
>>> Here's a follow-up. You could think of this as a prototype of what we
>>> might do internally with a string-valued series on permanent
>>> sub-sampling.
>>
>> Sorry for the late reply, Allin. Yes that looks good to me.
>>
>> But would "permanent sub-sampling" mean that this only applies when
>> executing the smpl command with the --permanent flag? Or would it also
>> apply when storing a sub-sampled data set?
>
> I favour doing this only when the --permanent option is given. It's an
> information-destroying move, and I can imagine cases where one wants to
> save a sub-sample and yet not lose the information in question. But if you
> _want_ to lose it, without using --permanent, then just store in a format
> other than gdt or gdtb.
I understand your point, Allin. And getting it worked when using the
--permanent option would be very useful.
But, let me loudly think about some of my use cases -- maybe some others have
to deal with similar ones...
[cases where using the --permanent option with "smpl" would clearly
not be convenient]
OK, here's what's now in git (not yet in snapshots, I'd prefer to
see some testing first):
(1) Imposing a sample restriction with the --permanent option
results in "trimming" of string-valued series: only string values
that appear within the sub-sample are preserved, and the numeric
coding for such series is adjusted accordingly. Note, this means
that any given observation will have the same string value as it had
in the full dataset, but may not have the same numeric code.
(2) When using the "store" command with a native target (gdt or
gdtb) there's a new option --trim-strvals which has a similar
effect. We achieve this as follows:
* Any string-valued series are first backed-up (copied in RAM).
* Before we actually write the data file we "trim" as described
above.
* Once the write is finished we restore the full form of the
string-valued series.
So you can sub-sample, store the data in trimmed form, then restore
the full dataset without loss of information -- or at least that's
the idea! This has worked OK in my limited testing today, but more
testing is wanted.
One further remark: "store --trim-strvals" will work even when
there's no sub-sample in place, in case the dataset contains any
redundant string values. I hadn't noticed before, but gretl's
grunfeld.gdt contains a redundant 11th firmname, "American Steel"
(there are only 10 firms in our dataset). You can remove that by
using store with the new option.
Allin
1617
days inactive
1628
days old
9 comments
3 participants
participants (3)
-
Allin Cottrell -
Artur Tarassow -
atecon＠posteo.de