Outliers Detection

Checking of the last observations (Terror-like)

Terror is a popular tool for detecting anomalies in the last observation(s). The tool simply compares the out of sample forecasts of the series (shortened by the corresponding number of observations) with the actual figures. The differences are expressed in function of the standard error of the forecasts. The class ec.tstoolkit.modelling.arima.CheckLast is a straightforward implementation of Terror.

Observations will be considered as abnormal when the relative forecast errors (in absolute term) will exceed a given threshold. A typical threshold will lie between 4 and 5 (higher values mean lower sensitivity). A CheckLast object will return the relative forecast error (getScores) as well as the forecasts (getActualValues)

We provide below some examples on the use of the tool.

Basic use

        CheckLast tramoTerror = new CheckLast(TramoSpecification.TR5.build());
        tramoTerror.check(Data.X);
        System.out.println("Tramo-Terror");
        System.out.println(tramoTerror.getScore(0));

        CheckLast xTerror = new CheckLast(RegArimaSpecification.RG5.build());
        xTerror.check(Data.X);
        System.out.println("X13-Terror");
        System.out.println(xTerror.getScore(0));

Complete example

        // time series creation
        // Method 1: add couples (date, value) in a "TsDataCollector" object
        TsDataCollector collector = new TsDataCollector();
        int n = 240;
        Day day = Day.toDay();
        Random rnd = new Random();
        double val = 100;
        for (int i = 0; i < n; ++i) {
            val += rnd.nextDouble();
            // Add a new observation (date, value)
            collector.addObservation(day.getTime(), val);
            day = day.plus(31 + rnd.nextInt(10));
        }
        // Creates a new time series. The most suitable frequency is automatically choosen
        // If the frequency is known, it should be used in the first parameter
        // The creation will fail if the collector contains less than 2 observations
        TsData s0 = collector.make(TsFrequency.Undefined, TsAggregationType.None);

        // Method 2: 
        val = 100;
        double[] values = new double[n];
        for (int i = 0; i < n; ++i) {
            val += (.1*i)*rnd.nextDouble();
            values[i] = val;
        }
        // Missing values are identified by Double.NaN
        values[n / 3] = Double.NaN;

        // The monthly series will start in January 2012
        TsPeriod start = new TsPeriod(TsFrequency.Monthly, 2012, 0);
        // Creates directly the series. The last parameter indicates that the data are cloned.
        TsData s1 = new TsData(start, values, true);

        // Creates the module that will detect outliers for the last period(s)
        // By default, Tramo (TR4) should be used for automatically identifying 
        // the most suitable RegArima model.
        // The module is able to check more than 1 last observations
        CheckLast tramoTerror = new CheckLast(TramoSpecification.TR4.build());
        tramoTerror.setBackCount(3);
        System.out.println("Tramo-Terror");
        tramoTerror.check(s0);
        // The scores (corresponding respectively to the periods
        // s0[s0.length - 3], s0[s0.length - 2], s0[s0.length - 1])  
        System.out.println(tramoTerror.getScore(0));
        System.out.println(tramoTerror.getScore(1));
        System.out.println(tramoTerror.getScore(2));
        tramoTerror.check(s1);
        double[] scores = tramoTerror.getScores(), fcasts = tramoTerror.getForecastsValues(),
                vals = tramoTerror.getValues(), actuals = tramoTerror.getActualValues();
        for (int i = 0; i < tramoTerror.getBackCount(); ++i) {
            System.out.print(actuals[i]);
            System.out.print('\t');
            System.out.print(fcasts[i]);
            System.out.print('\t');
            System.out.print(vals[i]);
            System.out.print('\t');
            System.out.println(scores[i]);
        }

        System.out.println(tramoTerror.getScore(0));

        System.out.println("X13-Terror");
        CheckLast xTerror = new CheckLast(RegArimaSpecification.RG4.build());
        xTerror.check(s0);
        System.out.println(xTerror.getScore(0));
        xTerror.check(s1);
        System.out.println(xTerror.getScore(0));

