2:06 p.m.
Hi,
ok I'm really sorry that adding Python support now leads to additional
bugs and problems, but I hope these are just toothing pains:
First, a prelim question: In the gretl_io.py file that gretl writes out,
why use this function
<python>
def gretl_dotdir():
dotdir = "C:/Users/svenne/AppData/Roaming/gretl/"
return dotdir
</python>
instead of just a plain string definition 'gretl_dotdir =
"<whatever>"'?
Secondly, in the 'gretl_export()' function (also in gretl_io.py) the
seemingly innocuous line
'r, c = M.shape'
might be problematic for vectors (given the next issue this is currently
hard to test, but I will). The point is that in numpy vectors may be
represented as 1-dim-arrays; their 'shape' tuple will be '(n,)' and thus
have only length one. I guess there are two solutions; either do a check
like 'len(M.shape)==1', or force M to be a numpy-matrix which is
guaranteed to be 2-dim like gretl matrices. Let me test and then think
about it.
Third, a real problem: The gretltmp.py file written by gretl does not
preserve the indentation from within gretl's foreign block! Since Python
uses indentation/whitespace as a syntax element, this is completely
broken. Allin, your example in ch. 39 doesn't have any indented blocks
it seems, and so you didn't notice. BTW, this is on Windows 8, haven't
tested on Linux yet.
And finally, another indentation-related less severe problem: gretl's
script editor's on-the-fly indentation is a little annoying when writing
in Python because it does it wrong (understandably). Could this be
somehow turned off inside foreign blocks?
Thanks,
Sven
6:03 p.m.
On Wed, 27 Feb 2013, Sven Schreiber wrote:
ok I'm really sorry that adding Python support now leads to
additional
bugs and problems, but I hope these are just toothing pains:
Yeah, I think so.
First, a prelim question: In the gretl_io.py file that gretl writes
out,
why use this function
<python>
def gretl_dotdir():
dotdir = "C:/Users/svenne/AppData/Roaming/gretl/"
return dotdir
</python>
instead of just a plain string definition 'gretl_dotdir =
"<whatever>"'?
Because we don't like global variables? Actually, I'm not sure
about python and globals, but it seems safer/cleaner, though
admittedly more complex, to have a function.
Secondly, in the 'gretl_export()' function (also in
gretl_io.py) the
seemingly innocuous line
'r, c = M.shape'
might be problematic for vectors (given the next issue this is currently
hard to test, but I will). The point is that in numpy vectors may be
represented as 1-dim-arrays; their 'shape' tuple will be '(n,)' and thus
have only length one. I guess there are two solutions; either do a check
like 'len(M.shape)==1', or force M to be a numpy-matrix which is
guaranteed to be 2-dim like gretl matrices. Let me test and then think
about it.
Could we just wrap M as numpy.matrix(M) ? (The gretl_export
function depends on numpy anyway, via savetxt.)
Third, a real problem: The gretltmp.py file written by gretl does
not
preserve the indentation from within gretl's foreign block! Since Python
uses indentation/whitespace as a syntax element, this is completely
broken. Allin, your example in ch. 39 doesn't have any indented blocks
it seems, and so you didn't notice.
True, that rather important point (and silly feature of
python!) had totally escaped me, doh. Should now be fixed in
CVS.
And finally, another indentation-related less severe problem:
gretl's
script editor's on-the-fly indentation is a little annoying when writing
in Python because it does it wrong (understandably). Could this be
somehow turned off inside foreign blocks?
Yes, I'll work on that.
Allin
7:18 p.m.
Am 27.02.2013 19:03, schrieb Allin Cottrell:
On Wed, 27 Feb 2013, Sven Schreiber wrote:
> First, a prelim question: In the gretl_io.py file that gretl writes out,
> why use this function
> <python>
> def gretl_dotdir():
> dotdir = "C:/Users/svenne/AppData/Roaming/gretl/"
> return dotdir
> </python>
> instead of just a plain string definition 'gretl_dotdir =
"<whatever>"'?
Because we don't like global variables? Actually, I'm not sure
about python and globals, but it seems safer/cleaner, though
admittedly more complex, to have a function.
IIRC the scope of the corresponding string variable should be the module
(here it's the same as the file), this shouldn't really become a global
variable; from another file you would have to access it as
'gretl_io.gretl_dotdir' unless you 'import' it in some other way. I
don't think there is really any advantage of wrapping this one-liner in
a function.
> Secondly, in the 'gretl_export()' function (also in gretl_io.py) the
> seemingly innocuous line
> 'r, c = M.shape'
> might be problematic for vectors (given the next issue this is currently
> hard to test, but I will). The point is that in numpy vectors may be
> represented as 1-dim-arrays; their 'shape' tuple will be '(n,)' and
thus
> have only length one. I guess there are two solutions; either do a check
> like 'len(M.shape)==1', or force M to be a numpy-matrix which is
> guaranteed to be 2-dim like gretl matrices. Let me test and then think
> about it.
Could we just wrap M as numpy.matrix(M) ? (The gretl_export
function depends on numpy anyway, via savetxt.)
Probably we should use 'asmatrix(M)'
(http://docs.scipy.org/doc/numpy/reference/generated/numpy.asmatrix.html),
but yes, this should do it I guess -- I hope there are no side effects
on 'savetxt'.
> Third, a real problem: The gretltmp.py file written by gretl does not
> preserve the indentation from within gretl's foreign block! Since Python
> uses indentation/whitespace as a syntax element, this is completely
> broken. Allin, your example in ch. 39 doesn't have any indented blocks
> it seems, and so you didn't notice.
True, that rather important point (and silly feature of
python!) had totally escaped me, doh. Should now be fixed in
CVS.
It's been a long time since I saw a quarrel about Python's syntax, and I
won't start one here :-) Thanks for fixing.
> And finally, another indentation-related less severe problem: gretl's
> script editor's on-the-fly indentation is a little annoying when writing
> in Python because it does it wrong (understandably). Could this be
> somehow turned off inside foreign blocks?
Yes, I'll work on that.
Thanks again.
-sven
9:20 p.m.
On Wed, 27 Feb 2013, Sven Schreiber wrote:
Am 27.02.2013 19:03, schrieb Allin Cottrell:
> On Wed, 27 Feb 2013, Sven Schreiber wrote:
>
>> First, a prelim question: In the gretl_io.py file that gretl writes out,
>> why use this function
>> <python>
>> def gretl_dotdir():
>> dotdir = "C:/Users/svenne/AppData/Roaming/gretl/"
>> return dotdir
>> </python>
>> instead of just a plain string definition 'gretl_dotdir =
"<whatever>"'?
>
> Because we don't like global variables? Actually, I'm not sure
> about python and globals, but it seems safer/cleaner, though
> admittedly more complex, to have a function.
IIRC the scope of the corresponding string variable should be the module
(here it's the same as the file), this shouldn't really become a global
variable; from another file you would have to access it as
'gretl_io.gretl_dotdir' unless you 'import' it in some other way. I
don't think there is really any advantage of wrapping this one-liner in
a function.
OK, we can change that at some point.
>> Secondly, in the 'gretl_export()' function (also in
gretl_io.py) the
>> seemingly innocuous line
>> 'r, c = M.shape'
>> might be problematic for vectors (given the next issue this is currently
>> hard to test, but I will). The point is that in numpy vectors may be
>> represented as 1-dim-arrays; their 'shape' tuple will be '(n,)'
and thus
>> have only length one. I guess there are two solutions; either do a check
>> like 'len(M.shape)==1', or force M to be a numpy-matrix which is
>> guaranteed to be 2-dim like gretl matrices. Let me test and then think
>> about it.
>
> Could we just wrap M as numpy.matrix(M) ? (The gretl_export
> function depends on numpy anyway, via savetxt.)
Probably we should use 'asmatrix(M)'
OK, that's now done in CVS.
>> And finally, another indentation-related less severe problem:
gretl's
>> script editor's on-the-fly indentation is a little annoying when writing
>> in Python because it does it wrong (understandably). Could this be
>> somehow turned off inside foreign blocks?
>
> Yes, I'll work on that.
Thanks again.
This should be somewhat better now, though it's very fiddly to
get right.
Allin
10:22 p.m.
Hi,
some more issues (with solutions...) on the foreign=python front here.
1) In gretltmp.py, "execfile": I just found out that this was scrapped
in Python 3. There is an official replacement (in Python's 2to3
conversion tool), but here I think it's simpler to just replace it with:
<python>
from gretl_io import gretl_dotdir, gretl_loadmat, gretl_export
</python>
This should work on all Python versions, but it assumes that gretl_io.py
will always be in the same directory as gretltmp.py, but AFAIU that's
the case by construction, no?
2) In gretl_io.py, first I think the gretl_export() function is a little
broken: The target file is opened in text mode but apparently savetxt
writes bytes, which it cannot do on text-mode opened files. So the
export doesn't take place. (Or did it for you?) A solution would be to
tell savetxt to also write the header, and all the other lines would be
unnecessary:
<python>
# f = open(fname, 'wb')
# f.write(repr(r) + '\t' + repr(c) + '\n')
savetxt(fname, M, fmt='%.18e', delimiter=' ',
header=repr(r)+'\t'+repr(c), comments='')
# f.close()
# return
</python>
The drawback is that this is a new feature in Numpy 1.7 which was just
released last month. OTOH it's a new gretl feature as well, so maybe
it's ok to have demanding version requirements.
Secondly, a follow-up on asmatrix(X): It seems that a 1-dim vector is
interpreted as a row vector and as such is written out to the file in a
single row/line (but with the header correct). Is that a problem (for
gretl)?
3) Could it be that gretl_io.py doesn't get re-created if gretl thinks
it created it already? (But it may have been deleted in the meantime.)
Thanks,
sven
10:58 p.m.
On Sat, 2 Mar 2013, Sven Schreiber wrote:
some more issues (with solutions...) on the foreign=python front
here.
1) In gretltmp.py, "execfile": I just found out that this was scrapped
in Python 3. There is an official replacement (in Python's 2to3
conversion tool), but here I think it's simpler to just replace it with:
<python>
from gretl_io import gretl_dotdir, gretl_loadmat, gretl_export
</python>
This should work on all Python versions, but it assumes that gretl_io.py
will always be in the same directory as gretltmp.py, but AFAIU that's
the case by construction, no?
Yes, should be.
2) In gretl_io.py, first I think the gretl_export() function is a
little
broken: The target file is opened in text mode but apparently savetxt
writes bytes, which it cannot do on text-mode opened files. So the
export doesn't take place. (Or did it for you?)
It did work for me, but actually I now see that it may well invoke
undefined behavior: we've got a write handle to fname open and then
we call savetxt which presumably opens the file for writing itself.
Not very nice.
A solution would be to
tell savetxt to also write the header, and all the other lines would
be
unnecessary:
<python>
# f = open(fname, 'wb')
# f.write(repr(r) + '\t' + repr(c) + '\n')
savetxt(fname, M, fmt='%.18e', delimiter=' ',
header=repr(r)+'\t'+repr(c), comments='')
# f.close()
# return
</python>
The drawback is that this is a new feature in Numpy 1.7 which was just
released last month. OTOH it's a new gretl feature as well, so maybe
it's ok to have demanding version requirements.
Presumably another solution would be to bypass savetxt and just do a
low-level write-out of the values in the matrix. I'll experiment.
Secondly, a follow-up on asmatrix(X): It seems that a 1-dim vector
is
interpreted as a row vector and as such is written out to the file in a
single row/line (but with the header correct). Is that a problem (for
gretl)?
That shouldn't be a problem for gretl.
3) Could it be that gretl_io.py doesn't get re-created if gretl
thinks
it created it already? (But it may have been deleted in the meantime.)
gretl_io.py is re-written every session, but in a given session it's
not recreated once it has been written.
Allin
2:42 a.m.
On Sat, 2 Mar 2013, Allin Cottrell wrote:
On Sat, 2 Mar 2013, Sven Schreiber wrote:
> some more issues (with solutions...) on the foreign=python front here.
>
> 1) In gretltmp.py, "execfile": I just found out that this was scrapped
> in Python 3. There is an official replacement (in Python's 2to3
> conversion tool), but here I think it's simpler to just replace it with:
> <python>
> from gretl_io import gretl_dotdir, gretl_loadmat, gretl_export
> </python>
> This should work on all Python versions, but it assumes that gretl_io.py
> will always be in the same directory as gretltmp.py, but AFAIU that's
> the case by construction, no?
Yes, should be.
Now tested, and it works fine. It's in CVS.
> 2) In gretl_io.py, first I think the gretl_export() function is a
little
> broken: The target file is opened in text mode but apparently savetxt
> writes bytes, which it cannot do on text-mode opened files. So the
> export doesn't take place. (Or did it for you?)
It did work for me, but actually I now see that it may well invoke undefined
behavior: we've got a write handle to fname open and then we call savetxt
which presumably opens the file for writing itself. Not very nice.
A solution would be to
> tell savetxt to also write the header, and all the other lines would be
> unnecessary:
> <python>
> # f = open(fname, 'wb')
> # f.write(repr(r) + '\t' + repr(c) + '\n')
> savetxt(fname, M, fmt='%.18e', delimiter=' ',
> header=repr(r)+'\t'+repr(c), comments='')
> # f.close()
> # return
> </python>
> The drawback is that this is a new feature in Numpy 1.7 which was just
> released last month. OTOH it's a new gretl feature as well, so maybe
> it's ok to have demanding version requirements.
Presumably another solution would be to bypass savetxt and just do a
low-level write-out of the values in the matrix. I'll experiment.
This seems to work fine too (with Python 2.7 at any rate), and is in
CVS. Let me know if there are backward/forward compatibility issues
with the solution I chose!
Allin
2:39 p.m.
Am 03.03.2013 03:42, schrieb Allin Cottrell:
On Sat, 2 Mar 2013, Allin Cottrell wrote:
> Presumably another solution would be to bypass savetxt and just
do a
> low-level write-out of the values in the matrix. I'll experiment.
This seems to work fine too (with Python 2.7 at any rate), and is in
CVS. Let me know if there are backward/forward compatibility issues
with the solution I chose!
On my Linux systems I also have 2.7 per default, so for testing on Py3 I
have to wait for the next Win snapshot. BTW, it would be nice to be able
to specify the Python interpreter in gretl's settings, similar to the
way R and Octave paths can be specified. (Why not the same for Ox,
actually?) That way, I could point gretl to the parallel Py3
installation that I have on Linux.
Apart from that I don't see any compatibility issues anymore offhand,
but two more small comments:
First, numpy.ndarray.tofile() may do the trick easier, see
http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.tofile.....
However, the only advantage I see over your low-level solution is three
LOCs less, so maybe just leave it as is.
Finally, the empty 'return' statement at the end is really redundant.
Thanks,
sven
8:09 p.m.
On Sun, 3 Mar 2013, Sven Schreiber wrote:
Am 03.03.2013 03:42, schrieb Allin Cottrell:
> On Sat, 2 Mar 2013, Allin Cottrell wrote:
>> Presumably another solution would be to bypass savetxt and just do a
>> low-level write-out of the values in the matrix. I'll experiment.
>
> This seems to work fine too (with Python 2.7 at any rate), and is in
> CVS. Let me know if there are backward/forward compatibility issues
> with the solution I chose!
On my Linux systems I also have 2.7 per default, so for testing on Py3 I
have to wait for the next Win snapshot. BTW, it would be nice to be able
to specify the Python interpreter in gretl's settings, similar to the
way R and Octave paths can be specified. (Why not the same for Ox,
actually?) That way, I could point gretl to the parallel Py3
installation that I have on Linux.
In today's CVS/snapshots you can set the Python path. The Ox
path is settable if you check "Enable Ox support" under
preferences. (For good measure, the Stata path is now settable
too.)
Allin
4070
days inactive
4074
days old
8 comments
2 participants
participants (2)
-
Allin Cottrell -
Sven Schreiber