2:23 p.m.
Hi,
I ran into a problem when using 'join' with a native gdt file (with time
series data). (Background: support for native gdt files was recently
added to the join command, on my request I am afraid.)
The sample and/or workfile ranges did not coincide, stopping the data
import. With a text file one would have to use the --tkey option or
something like that, as explained in the join chapter of the guide. But
I could not find an explanation of how to do that with a native file.
thanks,
sven
2:33 p.m.
On Thu, 19 Nov 2015, Sven Schreiber wrote:
The sample and/or workfile ranges did not coincide, stopping the
data
import.
What do you mean exactly? Non-overlapping samples? Different
periodicities? Both? Something else altogether?
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
2:59 p.m.
Am 19.11.2015 um 15:33 schrieb Riccardo (Jack) Lucchetti:
On Thu, 19 Nov 2015, Sven Schreiber wrote:
> The sample and/or workfile ranges did not coincide, stopping the data
> import.
What do you mean exactly? Non-overlapping samples? Different
periodicities? Both? Something else altogether?
Non-coinciding or perhaps non-contained (in a subset sense) samples. I
can provide a minimal example later, but have to run now.
-s
6:25 p.m.
Am 19.11.2015 um 15:33 schrieb Riccardo (Jack) Lucchetti:
On Thu, 19 Nov 2015, Sven Schreiber wrote:
> The sample and/or workfile ranges did not coincide, stopping the data
> import.
What do you mean exactly? Non-overlapping samples? Different
periodicities? Both? Something else altogether?
Here are some self-contained examples:
<hansl>
nulldata 100 # same example data to be joined
setobs 4 2000:1
series x = normal()
store file1.gdt
nulldata 40 # shorter sample
setobs 4 2000:1
join file1.gdt x # doesn't work
nulldata 120 # longer sample
setobs 4 2000:1
join file1.gdt x # doesn't work
nulldata 100
setobs 4 1990:1 # shifted but overlapping sample
join file1.gdt x # works but is WRONG
</hansl>
With respect to the last example I know you are going to say it's a
feature not a bug, and the responsibility of the user. But I find this
worthy of at least a warning message. And in any case I was originally
asking about the non-working cases.
thanks,
sven
8:16 p.m.
On Thu, 19 Nov 2015, Sven Schreiber wrote:
Am 19.11.2015 um 15:33 schrieb Riccardo (Jack) Lucchetti:
> On Thu, 19 Nov 2015, Sven Schreiber wrote:
>
>> The sample and/or workfile ranges did not coincide, stopping the data
>> import.
>
> What do you mean exactly? Non-overlapping samples? Different
> periodicities? Both? Something else altogether?
Here are some self-contained examples:
<hansl>
nulldata 100 # same example data to be joined
setobs 4 2000:1
series x = normal()
store file1.gdt
nulldata 40 # shorter sample
setobs 4 2000:1
join file1.gdt x # doesn't work
A workaround is to add a suitable tkey column to the join source file:
<hansl>
nulldata 100
setobs 4 2000:1
series yq = $obsmajor + $obsminor/10
series x = normal()
store file1.gdt
nulldata 40 # shorter sample
setobs 4 2000:1
join file1.gdt x --tkey=yq,"%Y.%q"
</hansl>
However, pretty clearly this should not be required. We have an
implicit time-key on the left, and when joining from a gdt file with a
time-series structure there should also be an implicit time-key on the
right. This is just something that's not implemented, and whose
absence hasn't been noticed till now.
nulldata 100
setobs 4 1990:1 # shifted but overlapping sample
join file1.gdt x # works but is WRONG
</hansl>
With respect to the last example I know you are going to say it's a
feature not a bug, and the responsibility of the user.
Well, no, I wouldn't say that ;-) Once again, with two native
time-series datasets the matching should be implicitly by time and
should "just work" (despite what the doc says about joining from
non-native plain text data files). Or at least, it should "just work"
(with no options required) if the two datasets are of the same
frequency.
Allin
9:14 p.m.
Am 19.11.2015 um 21:16 schrieb Allin Cottrell:
A workaround is to add a suitable tkey column to the join source file:
I was also thinking whether it's possible to work around the non-feature
without actually having to change the file from which to import.
Perhaps something like this:
<hansl-untested>
open outerfile.gdt
string outerstart = obslabel($t1)
length = $nobs
nulldata 100 --preserve
setobs 4 2000:1
startnum = obsnum(outerstart)
endnum = startnum + length - 1
smpl startnum endnum
join outerfile.gdt x
</hansl-untested>
cheers,
sven
10:47 p.m.
On Thu, 19 Nov 2015, Sven Schreiber wrote:
Am 19.11.2015 um 15:33 schrieb Riccardo (Jack) Lucchetti:
> On Thu, 19 Nov 2015, Sven Schreiber wrote:
>
>> The sample and/or workfile ranges did not coincide, stopping the data
>> import.
>
> What do you mean exactly? Non-overlapping samples? Different
> periodicities? Both? Something else altogether?
Here are some self-contained examples:
<hansl>
nulldata 100 # same example data to be joined
setobs 4 2000:1
series x = normal()
store file1.gdt
nulldata 40 # shorter sample
setobs 4 2000:1
join file1.gdt x # doesn't work
nulldata 120 # longer sample
setobs 4 2000:1
join file1.gdt x # doesn't work
nulldata 100
setobs 4 1990:1 # shifted but overlapping sample
join file1.gdt x # works but is WRONG
</hansl>
These examples now work correctly with current git. And that should
go for all the common time-series frequencies. I haven't yet
attempted to deal with hourly data.
Allin
3332
days inactive
3332
days old
6 comments
3 participants
participants (3)
-
Allin Cottrell -
Riccardo (Jack) Lucchetti -
Sven Schreiber