7:10 a.m.
I have a question for more experienced users and the guardians of gretl.
I am about to go into Markov Chain Monte Carlo simulations and an
acceptance/rejection algorithm of the generated values. This would
require something like 200,000 loop runs, from which say something like
100,000 will be accepted and should be stored. The number of parameters
is 32.
Up to now with smaller simulations, I used to store such values in
matrix objects, and then turn them into series objects. But now I am
looking at a 100,000 X 32 matrix, or at best at 32 vectors of 100,000 X
1 dimension, and I was wondering whether such a large row dimension will
create problems of speed or any other kind of problems.
Any suggestion on that?
--
Alecos Papadopoulos PhD
Athens University of Economics and Business
web: alecospapadopoulos.wordpress.com/
skype:alecos.papadopoulos
7:30 a.m.
Am 27.04.20 um 13:10 schrieb Alecos Papadopoulos:
I have a question for more experienced users and the guardians of
gretl.
I am about to go into Markov Chain Monte Carlo simulations and an
acceptance/rejection algorithm of the generated values. This would
require something like 200,000 loop runs, from which say something like
100,000 will be accepted and should be stored. The number of parameters
is 32.
Up to now with smaller simulations, I used to store such values in
matrix objects, and then turn them into series objects. But now I am
looking at a 100,000 X 32 matrix, or at best at 32 vectors of 100,000 X
1 dimension, and I was wondering whether such a large row dimension will
create problems of speed or any other kind of problems.
Any suggestion on that?
Hi Alecos,
Regarding the speed problem: I would say, it depends on what you want to
do. Computing column-wise statistics such as mean through the meanc()
function won't be a problem I guess. Sorting will be costly of course.
If you put your code into functions, make use of pointers to avoid
actual copying of big matrices.
Also memory won't be an issue: Gretl stores matrix in double-precision
format. Each float is of size 8 byte. Thus, the matrix will consume
<(100000 * 32 * 8) / 1024 / 1024 = 24.4 MB>.
Artur
9:03 a.m.
On Mon, 27 Apr 2020, Artur Tarassow wrote:
Am 27.04.20 um 13:10 schrieb Alecos Papadopoulos:
> I have a question for more experienced users and the guardians of gretl.
>
> I am about to go into Markov Chain Monte Carlo simulations and an
> acceptance/rejection algorithm of the generated values. This would require
> something like 200,000 loop runs, from which say something like 100,000
> will be accepted and should be stored. The number of parameters is 32.
>
> Up to now with smaller simulations, I used to store such values in matrix
> objects, and then turn them into series objects. But now I am looking at a
> 100,000 X 32 matrix, or at best at 32 vectors of 100,000 X 1 dimension, and
> I was wondering whether such a large row dimension will create problems of
> speed or any other kind of problems.
>
> Any suggestion on that?
Hi Alecos,
Regarding the speed problem: I would say, it depends on what you want to do.
Computing column-wise statistics such as mean through the meanc() function
won't be a problem I guess. Sorting will be costly of course.
Some more detail: internally, storage is column-wise, so if you have to
compute descriptive statistics, it'll be marginally more efficient to
store stuff that way: for example, the code
<hansl>
set verbose off
matrix X = mnormal(2000, 2000)
set stopwatch
loop 100
meanc(X)
endloop
timec = $stopwatch
loop 100
meanr(X)
endloop
timer = $stopwatch
print timec timer
</hansl>
gives, on my home laptop,
<output>
timec = 0.54684248
timer = 1.0227176
</output>
If you put your code into functions, make use of pointers to avoid
actual
copying of big matrices.
Or, better still, use the const modifier. In terms of performance, you get
the same effect, but you will avoid overwriting you data by accident.
Also memory won't be an issue: Gretl stores matrix in
double-precision
format. Each float is of size 8 byte. Thus, the matrix will consume <(100000
* 32 * 8) / 1024 / 1024 = 24.4 MB>.
Like Artur said, memory shouldn't be an issue unless your matrices are
truly massive. That said, you should still strive to keep your RAM
requirements down whenever possible. Some things look good on paper but
are vastly inefficient computationally. For example, if X is a generic
matrix and D is a diagonal matrix with diagonal d, you can translate the
expression Y = DX in 2 ways:
<hansl>
# CPU- and memory-hungry
D = zeros(n,n)
D[diag] = d
Y = D*X
# efficient
Y = d .* X
</hansl>
Try timing the above and prepare for a surprise ;)
-------------------------------------------------------
Riccardo (Jack) Lucchetti
Dipartimento di Scienze Economiche e Sociali (DiSES)
Università Politecnica delle Marche
(formerly known as Università di Ancona)
r.lucchetti(a)univpm.it
http://www2.econ.univpm.it/servizi/hpp/lucchetti
-------------------------------------------------------
1603
days inactive
1603
days old
2 comments
3 participants
participants (3)
-
Alecos Papadopoulos -
Artur Tarassow -
Riccardo (Jack) Lucchetti