On Wed, 8 Sep 2010, Henrique Andrade wrote: Because the current sample range contains missing values. OK, we'll pre-adjust the sample. Allin

X12-ARIMA has its own way of dealing with missing values either at the end of the series or in the middle. Even if the missing values are in the middle of the series X12-Arima in Gretl will fail with a similar message. Perhaps it would be better if Gretl just checked for missing values and if it found them issued a message that X!2-ARIMA does not work if there are missing values in the span of the data. While the X12-ARIMA manual states that missing values are not allowed there are ways around this. If the missing value is replaced by -99999 then -99999 may be replaced by a value that is regarded as an outlier and then this may be replace by the appropriate Reg-ARIMA fitted value for the purpose of estimation seasonal factors etc. This may require specific X12ARIMA options to be in effect and running X12-ARIMA outside of Gretl. I would strongly recommend that users of X12-ARIMA become familiar with X12-ARIMA output and check initial runs of the program to ensure that what they are doing is correct. While X12-ARIMA does produce sensible results in the majority of cases there are times when it may not. In such cases you may need to refine the options used. Don't blame the program if you use the wrong options On 9 September 2010 05:01, Henrique Andrade <henrique.coelho(a)gmail.com&gt; wrote: -- John C Frain Economics Department Trinity College Dublin Dublin 2 Ireland www.tcd.ie/Economics/staff/frainj/home.html mailto:frainj@tcd.ie mailto:frainj@gmail.com

Henrique, My initial encounter with seasonal adjustment was as a statistician in the Central Statistics Office in Dublin. In the mid 1970's we started to use the x11 (previous version of the X12-Arima) program. For each series we asked a variety of questions (e.g. Additive/multiplicative adjustment, Easter adjustment? Trading day adjustment, Treatment of outliers). Initial answers to these problems were suggested by an examination of the x11 output with default settings. This output is still produced by X12-ARIMA. If the initial automatic run was not satisfactory one generally revised the initial parameters until the output was satisfactory. The process was continued until a satisfactory result (specification) was obtained. This specification would then be used in a form of revised automatic seasonal adjustment for that series for a long period afterwards. One did, or should, monitor the x11 quality statistics in later periods. These are real life problems that anyone working in detail on seasonal adjustment might find. Ignoring them will likely lead to later problems. If you feel that you would like to learn more about the X11 output can I recommend Ladiray, D. and Benoit Quenneville (2001), Seasonal Adjustment with the X-11 Method, Springer. My experience is that seasonal adjustment is more difficult than simply pressing a few buttons in a computer program. Seasonal adjustment is much more complex than drawing a graph and often needs manual intervention. Thus the difference. I do use the Gretl X12-ARIMA facilities. It is a very convenient way to access the X12-ARIMA or TRAMO/SEATS program. In particular, I like the facility to import a large amount of data into Gretl and pass the results to X12-ARIMA. I do look at the X12-ARIMA or TRAMO/SEATS output. If the automatic adjustment is not satisfactory I will use the one of the seasonal adjustment programs either directly or through another interface. I have already stated that the opinion of various persons, working on this topic in statistical offices or central banks, who were accustomed to evaluating seasonal adjustments is that the automatic process works well in about 70% of cases. I can not quote any specific study - just an approximation. (It is worth noting that the x12-arima output produced by gretl includes a copy of the material that can be used as a direct input to the x12-arima program) In a real life interior missing values and the treatment of outliers is a problem. The treatment of these in X12-ARIMA is unique. The Gretl interface has problems with interior missing values. If Gretl crashes in such circumstances it might be helpful if the user was referred to the x12-arima manual where he would find details of the process necessary to do seasonal adjustment. If it is considered more convenient for the gretl interface to delete missing values at the start and end of the data then I have no objection. My problem was that I considered the interior missing values to be more important and was hoping to cover both cases in one. There is a considerable debate about using x12-arima adjusted data in econometric models. Some times it is all that you can do. Often it is better to include a specific seasonal model in your econometrics. Best regards John -- John C Frain Economics Department Trinity College Dublin Dublin 2 Ireland www.tcd.ie/Economics/staff/frainj/home.html mailto:frainj@tcd.ie mailto:frainj@gmail.com

Allin Page 2 of the X12 manual states "Observations must be equally spaced in time, and missing values are not allowed. As you say there are circumstances in which -99999 can be used as an indicator of a missing value. However this is not true in all cases. If I insert and interior -99999 in Henrique's dataset I think that It is being transferred from Gretl to X12 as a number and causing X12-ARIMA to crash. For Henrique's data the basic X12-ARIMA specification is series{ period=12 title="teste1" start=2001.1 data=( 2.44242 2.06569 2.7564 2.32389 2.58068 2.35682 2.4838 2.56693 2.0246 2.23309 1.95265 1.5527 1.91115 1.75875 1.7979 1.99489 1.90233 1.80959 2.41727 2.06787 1.99527 2.14675 1.94359 1.70031 1.94852 1.93438 1.9655 2.02454 2.15763 1.91898 2.25941 2.05011 2.43538 2.55376 2.31279 2.26303 2.3231 2.04155 3.01209 2.41058 2.41362 2.95703 2.91066 3.04607 3.1909 3.04472 3.21475 2.9296 2.88241 2.5542 3.104 2.876 3.147 3.41 3.229 3.785 3.162 3.079 3.342 3.18 3.331 2.964 3.708 3.242 3.549 3.55 4.078 4.558 4.125 4.354 4.053 3.61378 4.221 3.702 4.743 4.115 4.946 4.698 5.276 5.888 5.091 6.045 5.632 4.974 6.429 6.014 4.601 5.859 7.254 7.477 8.122 8.026 8.297 8.43 6.772 5.153 4.883 3.752 4.406 3.819 4.286 4.68 5.452 5.42 5.843 6.113 5.605 5.441 -99999 5.629 7.231 6.318 6.55 6.725 7.534 ) } automdl{} x11{ save=( d11 d12 d13 ) } The following code changes the default missing value code to 999999 and inserts a missing value code in the dqata for January 2010. I have also added a transform{function=auto} specification. This selects between a levels and a log transformation of the data using AIC. In the specification below I have also added outlier detection. It might be useful to incorporate these specifications into Gretl's interface. Trading day and Easter adjustments may be important for flow data. series{ period=12 title="teste2" start=2001.1 missingcode=999999 data=( 2.44242 2.06569 2.7564 2.32389 2.58068 2.35682 2.4838 2.56693 2.0246 2.23309 1.95265 1.5527 1.91115 1.75875 1.7979 1.99489 1.90233 1.80959 2.41727 2.06787 1.99527 2.14675 1.94359 1.70031 1.94852 1.93438 1.9655 2.02454 2.15763 1.91898 2.25941 2.05011 2.43538 2.55376 2.31279 2.26303 2.3231 2.04155 3.01209 2.41058 2.41362 2.95703 2.91066 3.04607 3.1909 3.04472 3.21475 2.9296 2.88241 2.5542 3.104 2.876 3.147 3.41 3.229 3.785 3.162 3.079 3.342 3.18 3.331 2.964 3.708 3.242 3.549 3.55 4.078 4.558 4.125 4.354 4.053 3.61378 4.221 3.702 4.743 4.115 4.946 4.698 5.276 5.888 5.091 6.045 5.632 4.974 6.429 6.014 4.601 5.859 7.254 7.477 8.122 8.026 8.297 8.43 6.772 5.153 4.883 3.752 4.406 3.819 4.286 4.68 5.452 5.42 5.843 6.113 5.605 5.441 999999 5.629 7.231 6.318 6.55 6.725 7.534 ) } transform{function=auto} outliers{} automdl{} x11{ save=( d11 d12 d13 ) } The transform specification is probably important if you wish to do an automatic adjustment. As far as I know this is the way TRAMO works. Most economic series need a log transformation before adjustment. On occasion one may wish to transform even though the statistics say otherwise. However in such cases the differences may be very small. An outlier specification is usually included. The default is usually sufficient. In addition for flow data trading day and easter (and other holiday) adjustments are usually included. On 10 September 2010 00:12, Allin Cottrell <cottrell(a)wfu.edu&gt; wrote: -- John C Frain Economics Department Trinity College Dublin Dublin 2 Ireland www.tcd.ie/Economics/staff/frainj/home.html mailto:frainj@tcd.ie mailto:frainj@gmail.com

On Fri, 10 Sep 2010, John C Frain wrote: ... Hmm, if I feed this to x12a (Version 0.3 Build 188) it comes back with Line 18: outliers{} ^ ERROR: outliers is not a valid spec name. However, the use of "missingcode" does lead to what looks like correct handling of interior NAs, very nice. Allin

Allin My apologies. I had intended saving yesterday's draft reply and revising and finishing the email today. It appears that, instead, I sent it in error. Perhaps the best thing to do is ignore yesterday's and read what is below? Page 2 of the X12 manual states "Observations must be equally spaced in time, and missing values are not allowed. As you say there are circumstances in which -99999 can be used as an indicator of a missing value. However this is not true in all cases. If I insert and interior -99999 in Henrique's dataset (Jan 2010). }. The error message is ERROR: Multiplicative or log additive seasonal adjustment cannot be performed when preadjustment factors are derived from a regARIMA model for data which have not been log transformed. Check the values for the power or function arguments of the transform spec and mode of the x11 spec. There would appear to be some problem in the use of this code and the defaults of automdl{} and x11{}. I presume that the message implies that the x11{} spec is trying the default multiplicative adjustment but the regARIMA step was completed using levels of the variables. The spec passed to X12-ARIMA was series{ period=12 title="teste1" start=2001.1 data=( 2.44242 2.06569 2.7564 2.32389 2.58068 2.35682 2.4838 2.56693 2.0246 2.23309 1.95265 1.5527 1.91115 1.75875 1.7979 1.99489 1.90233 1.80959 2.41727 2.06787 1.99527 2.14675 1.94359 1.70031 1.94852 1.93438 1.9655 2.02454 2.15763 1.91898 2.25941 2.05011 2.43538 2.55376 2.31279 2.26303 2.3231 2.04155 3.01209 2.41058 2.41362 2.95703 2.91066 3.04607 3.1909 3.04472 3.21475 2.9296 2.88241 2.5542 3.104 2.876 3.147 3.41 3.229 3.785 3.162 3.079 3.342 3.18 3.331 2.964 3.708 3.242 3.549 3.55 4.078 4.558 4.125 4.354 4.053 3.61378 4.221 3.702 4.743 4.115 4.946 4.698 5.276 5.888 5.091 6.045 5.632 4.974 6.429 6.014 4.601 5.859 7.254 7.477 8.122 8.026 8.297 8.43 6.772 5.153 4.883 3.752 4.406 3.819 4.286 4.68 5.452 5.42 5.843 6.113 5.605 5.441 -99999 5.629 7.231 6.318 6.55 6.725 7.534 ) } automdl{} x11{ save=( d11 d12 d13 ) } I can complete the analysis with the following amended spec. series{ period=12 title="teste3" start=2001.1 data=( 2.44242 2.06569 2.7564 2.32389 2.58068 2.35682 2.4838 2.56693 2.0246 2.23309 1.95265 1.5527 1.91115 1.75875 1.7979 1.99489 1.90233 1.80959 2.41727 2.06787 1.99527 2.14675 1.94359 1.70031 1.94852 1.93438 1.9655 2.02454 2.15763 1.91898 2.25941 2.05011 2.43538 2.55376 2.31279 2.26303 2.3231 2.04155 3.01209 2.41058 2.41362 2.95703 2.91066 3.04607 3.1909 3.04472 3.21475 2.9296 2.88241 2.5542 3.104 2.876 3.147 3.41 3.229 3.785 3.162 3.079 3.342 3.18 3.331 2.964 3.708 3.242 3.549 3.55 4.078 4.558 4.125 4.354 4.053 3.61378 4.221 3.702 4.743 4.115 4.946 4.698 5.276 5.888 5.091 6.045 5.632 4.974 6.429 6.014 4.601 5.859 7.254 7.477 8.122 8.026 8.297 8.43 6.772 5.153 4.883 3.752 4.406 3.819 4.286 4.68 5.452 5.42 5.843 6.113 5.605 5.441 -99999 5.629 7.231 6.318 6.55 6.725 7.534 ) } transform{function=auto} outlier{} regression{variables=(const, td)} automdl{} x11{ save=( d11 ) } Here the spec transform{function=auto} selects levels or log transformation on the basis of AICC. In this case it selects a log transformation ( transform{function=log} would give the the same result). Now the automdl{} is estimating a model based on the regARIMA log of the data which can be used in a multiplicative (default) x11{} estimation. I have also chosen to include trading day adjustment and adjustment for outliers in the model. If the trading day adjustment is not done there are warnings about significant trading day effects not being included. I also think that the outlier adjustment is often important. Is it possible to have an interface to X12-Arima similar to that for a TRAMO/SEATS adjustment? I think that this should only cover the minimum options necessary to produce a reasonable "automatic" adjustment. I am sure that there are others who may have different opinions on what is important. It would be interesting to hear their opinions. The following is a provisional list containing three possible tabs in such a facility. 1) Functional form - no transformation (additive adjustment) - add transform{function=none} (If this is done it is necessary to change the adjustment process to additive in the x11{} spec) - log transformation (multiplicative adjustment) - add transform{function = log} to spec - decided by x12-arima on basis of AIC - add transform{function = auto} to spec (proposed Gretl default) 2) Adjust for outliers - Use X12-ARIMA defaults - add outlier{} to spec (proposed Gretl default) - Do not adjust for outliers 3) Trading day - Include trading day adjustment - add regression{variables = td} - Do not include trading day effect (Gretl default as trading day adjustment is appropriate for flow data) 4) Missing values. (This was my initial way around the -99999 error but it may not be necessary if transform{function = auto} is included in the spec - If missing values are present code them as 999999 (instead of -99999) and add missingcode=999999 to the series spec. - If the series contains interior non-numeric data or a Gretl missing value code There are many other options that might br considered. The inclusion of many of these would over complicate the interface. It is probably better to deal with the difficult cases by direct use of the X12-ARIMA program. Gretl already provides the first step to such analysis as the the X12-AEIMA spec file, .err file and other X12-ARIMA files have been saved in C:\Users\username\AppData\Roaming\gretl\x12arima in windows 7 and probably in the corresponding subdirectory in Linux. The .err file contains a quick summary of errors and warnings. The spec (.spc) file can be edited used for the examination of the effect of other options in the X12-ARIMA program. I hope that I have not misunderstood some aspects of the recent versions of the X12-ARIMA program Best Regards John On 10 September 2010 00:12, Allin Cottrell <cottrell(a)wfu.edu&gt; wrote: -- John C Frain Economics Department Trinity College Dublin Dublin 2 Ireland www.tcd.ie/Economics/staff/frainj/home.html mailto:frainj@tcd.ie mailto:frainj@gmail.com

Allin I have had a quick look at the new version and it looks good. Two small suggestions - 1) The inclusion of the -99999 value makes a bit of a mess of the two graphs which include the original series. Would it be possible to use Table B1 (Original series pre-adjusted for outliers and missing values) 2) Trading day adjustment with the default turned off. This would enable one to get rid of the trading day warning in the .err file which occurs when adjusting the teste data. Would it be worthwhile to present the .err file in a third window? Best Regards John On 12 September 2010 17:33, Allin Cottrell <cottrell(a)wfu.edu&gt; wrote: -- John C Frain Economics Department Trinity College Dublin Dublin 2 Ireland www.tcd.ie/Economics/staff/frainj/home.html mailto:frainj@tcd.ie mailto:frainj@gmail.com

Allin I have attached a pdf file which, I hope, clarifies my proposals for the gretl x12arima interface. Basically there are three proposals 1. An amendment to the graphics output of the x12arima interface in gretl to improve these graphs when there are missing values. 2. A greater emphasis on the .err le to ensure that the user is aquainted with possible problems in the seasonal adjustment process. 3. The inclusion of a trading day option in the gretl interface to x12arima. test.gdt contains the data used to illustrate the points that I have made. The interface will then cover the main facilities required in an automatic x12arima analysis. If particular series require further analysis one can make direct use of the x12a program or use one of the spwcialised interfaces such as Demetra (http://circa.europa.eu/irc/dsis/eurosam/info/data/demetra.htm) Best regards John On 13 September 2010 19:39, Allin Cottrell <cottrell(a)wfu.edu&gt; wrote: -- John C Frain Economics Department Trinity College Dublin Dublin 2 Ireland www.tcd.ie/Economics/staff/frainj/home.html mailto:frainj@tcd.ie mailto:frainj@gmail.com

Allin Thanks for your attention to this matter. I think that you are very close to an ideal solution which does not over-complicate the adjustment process like other implementations. It would be nice if users read the standard output and error files but I am sure that they will not. The x12arima console output does summarise errors and warnings up front. This brings the problems directly to the attention of the user. The trading day warning is mentioned on page 26 of the standard output. Sample console output is below. The default trading day adjustment is a bit more complicated. The td option that I mentioned in an earlier email includes 6 trading day variables for each month. t1 = No of Mondays - No of Sundays ..... t6 = No of Saturdays - No of Sundays. This covers the working days example that you mentioned but is more general and it also includes cases where a particular day or days might have different volume than others. ########### Sample x12a console output ####################### PS C:\Users\frainj\Downloads\x12arimaTest\x12a> .\x12a teste X-12-ARIMA Seasonal Adjustment Program Version Number 0.3 Build 188 PSP = 24 Execution began Oct 1, 2010 20.39.35 Reading input spec file from teste.spc Storing any program output into teste.out Storing any program error messages into teste.err WARNING: At least one visually significant trading day peak has been found in one or more of the estimated spectra. Execution complete for teste.spc ################################################## On 29 September 2010 02:01, Allin Cottrell <cottrell(a)wfu.edu&gt; wrote: -- John C Frain Economics Department Trinity College Dublin Dublin 2 Ireland www.tcd.ie/Economics/staff/frainj/home.html mailto:frainj@tcd.ie mailto:frainj@gmail.com

On Thu, 9 Sep 2010, John C Frain wrote: John, thanks for your contribution to the discussion. It's really nice to have debates of this caliber here on the user list. I just want to add that, in my very very humble opinion, the mindless usage of preprocessed data is one of the reasons of the present comatose state of macroeconometrics. In practice, you take a time series like gdp (which is itself a rough estimate of something that is defined rather vaguely), you take seasonality away via x12 or tramo/seats, then you take the trend away via HP or whatever (oh, and if you use HP you adopt the usual, ridiculously conformist choice as to the value of lambda) and you end up with something you pompously call the "output gap", which is in fact little more than an educated drawing from a random number generator. Then you ramble on for weeks and months across working papers and journals about a 0.1% variation, trying to prove whatever the latest theoretical fad is; while of course, all you really care about is the impact factor of the journal you're submitting to. Riccardo (Jack) Lucchetti Dipartimento di Economia Università Politecnica delle Marche r.lucchetti(a)univpm.it http://www.econ.univpm.it/lucchetti