Extracting Seasonality and Trend from Data: Decomposition Using R

Clique na imagem para seguir o link.

Uma excelente descrição da decomposição clássica com Python e R.

Time series decomposition works by splitting a time series into three components: seasonality, trends and random fluctiation. To show how this works, we will study the decompose( ) and STL( ) functions in the R language.

Understanding Decomposition

Decompose One Time Series into Multiple Series

Time series decomposition is a mathematical procedure which transforms a time series into multiple different time series. The original time series is often split into 3 component series:

  • Seasonal: Patterns that repeat with a fixed period of time. For example, a website might receive more visits during weekends; this would produce data with a seasonality of 7 days.
  • Trend: The underlying trend of the metrics. A website increasing in popularity should show a general trend that goes up.
  • Random: Also call “noise”, “irregular” or “remainder,” this is the residuals of the original time series after the seasonal and trend series are removed.

Tags: , , ,

IFORS Developing Countries OR Resources Website

Clique na imagem para seguir o link

Clique na imagem para seguir o link

Artigos e software relacionado com Investigação Operacional

Click below on required topic headings to access papers or click here to access International Abstracts in OR


The 5 Computer Vision Techniques

clique na imagem para seguir o link

clique na imagem para seguir o link

Boa introdução ao tema da visão por computador

The 5 Computer Vision Techniques That Will Change How You See The World

Computer Vision is one of the hottest research fields within Deep Learning at the moment. It sits at the intersection of many academic subjects, such as Computer Science (Graphics, Algorithms, Theory, Systems, Architecture), Mathematics (Information Retrieval, Machine Learning), Engineering (Robotics, Speech, NLP, Image Processing), Physics (Optics), Biology (Neuroscience), and Psychology (Cognitive Science). As Computer Vision represents a relative understanding of visual environments and their contexts, many scientists believe the field paves the way towards Artificial General Intelligence due to its cross-domain mastery.

So what is Computer Vision?

Tags: ,

How To Use Multivariate Time Series Techniques For Capacity Planning on VMs

clique na imagem para seguir o link

clique na imagem para seguir o link

Métodos multivariados para séries cronológicas com VMs

Capacity planning is an arduous, ongoing task for many operations teams, especially for those who rely on Virtual Machines (VMs) to power their business. At Pivotal, we have developed a data science model capable of forecasting hundreds of thousands of models to automate this task using a multivariate time series approach. Open to reuse for other areas such as industrial equipment or vehicles engines, this technique can be applied broadly to anything where regular monitoring data can be collected.

Tags: ,

Recurrent neural networks, Time series data and IoT – Part One

clique na imagem para seguir o link

clique na imagem para seguir o link

Utilização de redes neuronais para previsão de séries univariadas

RNNs are already used for Time series analysis. Because IoT problems can often be modelled as a Time series, RNNs could apply to IoT data. In this multi-part blog, we first discuss Time series applications and then discuss how RNNs could apply to Time series applications. Finally, we discuss applicability to IoT.

In this article (Part One), we present the overall thought process behind the use of Recurrent neural networks and Time series applications – especially a type of RNN called Long Short Term Memory networks (LSTMs).

Tags: ,

imagens criadas por campos vetoriais

clicar na imagem para seguir o link para a app

This website allows you to explore vector fields in real time.

“Vector field” is just a fancy way of saying that each point on a screen has some vector associated with it. This vector could mean anything, but for our purposes we consider it to be a velocity vector.

Now that we have velocity vectors at every single point, let’s drop thousands of small particles and see how they move. Resulting visualization could be used by scientist to study vector fields, or by artist to get inspiration!

Learn more about this project on GitHub

Stay tuned for updates on Twitter.

With passion,



Free Hadoop Tutorial: Master BigData

clique na imagem para seguir o link

clique na imagem para seguir o link

BigData is the latest buzzword in the IT Industry. Apache’s Hadoop is a leading Big Data platform used by IT giants Yahoo, Facebook & Google. This course is geared to make a Hadoop Expert.

What should I know?

This is an absolute beginner guide to Hadoop. But knowledge of 1) Java 2) Linux will help


Tutorial Introduction to BIG DATA: Types, Characteristics & Benefits
Tutorial Hadoop Tutorial: Features, Components, Cluster & Topology
Tutorial Hadoop Setup Tutorial – Installation & Configuration
Tutorial HDFS Tutorial: Read & Write Commands using Java API
Tutorial What is MapReduce? How it Works – Hadoop MapReduce Tutorial
Tutorial Hadoop & Mapreduce Examples: Create your First Program
Tutorial Hadoop MapReduce Tutorial: Counters & Joins with Example
Tutorial What is Sqoop? What is FLUME – Hadoop Tutorial
Tutorial Sqoop vs Flume vs HDFS in Hadoop
Tutorial Create Your First FLUME Program – Beginner’s Tutorial
Tutorial Hadoop PIG Tutorial: Introduction, Installation & Example
Tutorial Learn OOZIE in 5 Minutes – Hadoop Tutorial
Tutorial Big Data Testing: Functional & Performance
Tutorial Hadoop & MapReduce Interview Questions & Answers


working with alien SPSS files

clique no ícon para seguir o link

clique no ícon para seguir o link

Informação sobre diversos inquéritos e acesso aos dados respetivos

Close Encounters of the Fourth Kind: working with alien SPSS files

[New page 23 Oct 2014: last updated 5 June 2017]

[NB: Notes and commentaries below may arrive as pdf files in your download folder]

Close Encounters of the Fourth Kind: working with alien SPSS files (pdf)

An alternative working title would have been: Sows’ Ears and Silk Purses: working with other peoples’ SPSS files as a follow-up to Old Dog, Old Tricks, my 2006 presentation to ASSESS.  Thought about using Old Dog, New Tricks, but it doesn’t carry the same sense of horror and fun.

Slide-shows covered recent work on other people’s files, including a live demo of Jon Peck’s Python code to move question numbers from the end to the beginning of variable labels and to change labels from UPPER to Mixed case text.  Also included were some new tricks and demos of things I didn’t know SPSS would do until I tried.  I haven’t used PowerPoint since York 2006, but I  found [Alt][PrintScreen] and MS Snip incredibly useful for getting screenshots into Word, and they also copied easily into Ppt.  The presentation ran SPSS live, drawing on my explorations of:

British Social Attitudes
Commentary on SPSS file for British Social Attitudes 2011 (pdf)
Notes on British Social Attitudes 2004 teaching data set (pdf) as used by Marsh and Elliott, 2008

​(See also page British Social Attitudes which has links to later commentaries on the ease of use and understanding of SPSS saved files distributed by UKDS on page British Social Attitudes: ​Exploring the SPSS files and detailed accounts of my creation in 2016 of a cumulative mother fille for all waves 1983 to 2014 on page British Social Attitudes 1983 to 2014: Cumulative SPSS file

Understanding Society
Commentary on Understanding Society 2010 (pdf)

NORC General Social Survey (GSS)
As of March 2016, the NORC GSS website has been completely revamped and is easier to navigate.   Some of the content in the following commentaries may now be otiose.
Commentary on full NORC General Social Survey 2008 (pdf)
Commentary on subset of General Social Survey 2008 (pdf) (as used by Sweet & Grace-Martin)
Commentary on GSS 2008 SPSS files for Babbie et al (pdf) (as used by Babbie, Halley, Wagner & Zaino)

(UK) ONS National Well-being
[New page 2 May 2015]
ONS National Well-being

Commentary on Unrestricted Access Teaching Dataset (ONS Opinions Survey, Well‐Being Module (pdf)
Data set and user guide from the Cathie Marsh Centre for Census and Survey Research, Manchester now renamed the Cathie Marsh Institute for Social Research, . This dataset (SN7146) contains a selection of variables from the April 2011 wave of the ONS Opinions Survey, Well-Being Module, April –  August 2011 (SN 6893) which in turn is part of the regular government survey  Opinions and Lifestyle Survey, run in various guises since 1990

Tags: ,

British Social Attitudes

clique no ícon para seguir o link

clique no ícon para seguir o link

Inquerito e dados respetivos sobre atitudes sociais no UK

British Social Attitudes 1983 onwards
​Cumulative SPSS file

[New page 22 June 2016: last updated 14 Feb 2017]

Cumulative files 1983 onwards
Attempting analyses across waves became increasingly frustrating as I encountered a range of anomalies, incompatibilities and inconsistencies, not to mention universally incomplete and/or incorrect specifications of measurement levels, missing values and value labels.   Accordingly I set myself the  task of generating a complete cumulative SPSS file containing the data from all waves from 1983 to 2014 (one colleague described this undertaking as Herculean) to provide what will hopefully be a valuable resource for teachers, students and researchers.  The 2015 wave was added in January 2017.

Index to UKDS downloads for British Social Attitudes 1983 – 2014 is an Excel file detailing, for each wave 1983 – 2014, year of survey, link to UKDS, download filename, size of file, number of cases, number of variables, number of variables with non-numeric formats and the new working filename assigned to amended files. The amended *.sav files were sent to Natcen for approval and possible deposit with UKDS, but are now superseded.

Non-numeric variables in British Social Attitudes is a step-by-step account of identifying, in each wave, variables with the same name, but different formats.  Several of these variables are specified as Strings with widths varying from A4 to A60, but some are in fact numbers.  Others are dates or times in DATE or TIME format and one is in COMMA1.  These and other factors prevent merging data from different waves using the SPSS command ADD FILES.  It’s been quite complex and tedious tracking them all down, but I eventually managed to create cumulative files for 1983 – 1994 and 2011 – 2014.  Merging 1995 – 2005 and 2006 – 2009 was more daunting, as several pro​blems remained to be resolved, but I eventually managed to generate a draft cumulative file for the whole series.  Much more meticulous and painstaking detective work and editing was required before a beta version was ready for public release.

Cumulative SPSS file 1983 to 2014
This task was completed on 20 June 2016 and the pass-word protected “mother” file (0.99 gb) has now been lodged (via Dropbox) with Natcen and UKDS for approval and distribution.  Custom-written Python code, freely and generously supplied by Jon Peck (retired Senior Software Engineer, IBM-SPSS) has saved me weeks if not months of painstaking needle-in-haystack searches. I also wish to thank Dr Chris Stride (Sheffield) who suggested using the sort facility in Excel to separate variable names with single (positive) missing values from those with paired (positive and equivalent negative) missing values.

For sure, some mini-glitches may remain, but to find and resolve these would at this stage be completely uneconomic of my time.  However users are warned that, because metadata for repeated variables are taken from the most recent wave, the value labels for categories of some variables differ from those of earlier waves.  This is particularly true of ordinal variables for income groups.

Tags: ,

SSRC Survey Unit Quality of Life

clique no ícon para seguir o link

clique no ícon para seguir o link

Dados e descrição de um inquérito sobre qualidade de vida no UK

The abstracts contain details of content, sampling, fieldwork and available data files.  The questionnaires are facsimiles of the actual questionnaires used in the field.  The user manuals contain questionnaires, unweighted frequency counts on the raw data as well as technical information on fieldwork, sampling, coding, show-cards and interviewer instructions.  The SPSS saved files are restorations from original files generated in the 1970s with some editing of SPSS setup files from 1970s versions to SPSS  for Windows (11, 15, 18 and 19): a few (self-explanatory) derived variables have been left in.

Quality of Life in Britain: 1st Pilot Survey,  March 1971

1:  Abstract
2:  Questionnaire
3:  User Manual
4: SPSS saved file for 1st pilot

Quality of Life in Britain: 2nd Pilot Survey, Oct-Nov 1971

1:  Abstract
2:  Questionnaire
3:  User Manual
4: SPSS saved file for 2nd pilot

Quality of Life in Britain: 1st National Survey 1973

(replicated simultaneously in Stoke-on-Trent and Sunderland)

1:  Abstract
2: Questionnaire
3:  User Manual for main GB survey

4a: SPSS saved file for main GB survey 1973
4b: SPSS saved file for Stoke-on-Trent survey 1973
4c: SPSS saved file for Sunderland survey 1973

Quality of Life in Britain: 2nd National Survey 1975

1:  Abstract
2:  Questionnaire
3:  User Manual
4: SPSS saved file for main GB survey 1975

Tags: ,