Re: [Gretl-devel] Data messed up after 'append' with panel data
On Thu, 31 Jul 2014, Sven Schreiber wrote:
this problem turned out to be easy to demonstrate, after it was
recognized and diagnosed:
First consider the workfile 'importcheck'. It is a 5 units by 5 periods
panel dataset, structured by the index variables 'ix' and 'tx'; the
existing values of 'ix' are 25, 27, 30, 35, 40. The only real variable
is called 'somedata' and has mostly missings.
The second file 'tobeimported' also has panel data identified by ix and
tx, but there is an additional ix=33 in there, so it's 6 units by 5 periods.
When you append the second file to the first, gretl says 'appended ok',
but in fact it does not honor the ix/tx-structure, such that the values
of the existing variable 'somedata' are moved around to different
observations. This is pretty bad, especially if you do not notice it
right away!
You need to use "join" to do this sort of thing; "append" is working
as
expected but it won't/can't do what you want here.
When you define a panel structure via the use of index variables this is
just a matter of giving gretl construction guidelines. The series used as
indices have no special status in the resulting panel; individuals are
numbered from 1 to N and time-series observations from 1 to T, as shown
when you print series with --byobs.
So when you do the "append" that you mention, gretl adds one more
individual (number 6) and updates (overwrites) the common series ix with
the values from the second file. So far as gretl is concerned no data have
been "moved": all the previously existing observations are still lined up
with the same individuals, 1 to 5.
If we were to give the index variables used in constructing a panel a
special status thereafter, we'd have to work out a policy regarding those
variables: do they become immutable? if not, what happens if you redefine
one or both of them?
Allin