Useful Softwares for Research

I took a session on ‘Useful software for research’ for the new batch of research scholars. I hope sharing the slides useful to other research students. The software suggestions were sorted based on my experience and colleagues’ experiences. The presentation is prepared in svg (jessyink template) format, and it is best viewed in Firefox/chrome browser.

Let me know if you have any comments or suggestions. I will include those for future sessions.

Useful softwares for research

Data visualisation talk: Presentation using reports package

Why did I use html5 for my today’s talk?   My last presentation was prepared using html5. This time I wanted  some innovation while making the slides.  I prepared  first few slides in Jessyink. Then I got to know that my friend (Trinker) developed an R package, reports,  for creating an amazing presentation including  beamer and reveal. js template.  This is cool!!! Create a presentation in R for R talk. I could write everything in R markdown and convert into beautiful html5 slides.

You could find my presentation just below this blog.  If any of you wanted to recreate the same, here is the R Markdown file Rmd.  I did bit modification in the style file and some tags. Otherwise, you could generate the same. Ensure you have all required packages (check Rmd file) before running the file. The options required for slide creation very well described in the help (reports package).

 Short version of slide creation is as follows:

  1. Install reports package (install.packages(“reports”) or take it from github).
  2. Call the presentation(“your_folder”) function. This creates Rnw and Rmd files inside your_folder/PRESENTATION.
  3. Add contents to Rmd file. Then click knit HTML if you are using RStudio or call knit2html. This would create the html file.
  4. Finally, run reveal. j’s () function. This would create the html5 presentation slides inside your_folder/PRESENTATION/reveal.js(). You could customize the path or theme or transition… I have used the theme as beige and transition as a cube.

This process wouldn’t be smoother for an R beginner who uses Windows. One should make sure all dependencies are set.  Github page provides more help resources, and it is helpful to complete successful creation of your presentation.

Apart from reports one can try slidify  package for creating html5 slides. This got multiple  html5 presentation templates including shower, io2012, etc.

One thing I am a bit unhappy is  that Emacs+ess doesn’t have a Rmd mode. I am sure someone is working on that.  Now it is time for my research work.

Note:Use firefox or chrome browser to view my presentation. left, right, up*, and down* arrows for transition.


PS:  ggplot2 is liked by almost all. As per the feedback, I should spend more time on ggplot2.

Fun with R graphics: A raptor and a cake

R graphics are something I always  explore or experiment with.  Base graphics, ggplot2 are my favorites. On  other graphics packages, I haven’t used plotrix much. I am currently exploring grid package.


First one is using ggplot2. (You could also do this using base graphics).

It would be more fun if we can do animation on Raptors. You can find an animated version of Raptor here. Rcode.


Here is the easy one. This one is a cake using plotrix package.


I have already discussed this graph in my favorite forum talks tats.  I dedicated both the graphs to my friends in the forum. You can find more fun graphs in this forum.


A heuristic enhancement on an optimisation algorithm

Directly or indirectly we heavily use optimization in our life.  Resource or utility optimization is something we hear every day.  Here I will discuss a standard optimization problem in statistics, maximum likelihood estimate. This blog is a small episode of my recent work. I used R language to implement the optimization exercise.

There are packages available for optimization in R. The mostly used are: optim() and  optimize() are in stats package. A dedicated function mle() available in stats4 package. This function is very useful most of the mean part modelling (eg: glm). What we need for this optimization is to prepare a function for the (log) likelihood and gradient (or hessian). We can also specify the algorithm (methods) as any of the following: “Nelder-Mead”, “BFGS”, “CG”, “L-BFGS-B”, “SANN”, and   “Brent”. This optimization output gives the necessary information for the model estimation.

I was working on an extension of bivariate garch model. It includes more than 20 parameters. Let me focus pure garch model, garch(1,1),  as a prototype to explain the algorithm. Here the model equation is in the variance part. Also, the parameters have constraint to ensure the positive variance. The   model equation is given below.

[math]h_t = \omega + \alpha x_{t-1}^2 + \beta h_{t-1}; \omega >0 , \alpha,\beta \geq 0 [/math]

The  optimization function may work for the garch (1,1) estimation. But as the number of parameters increases, there is not much use with optimal functions in R. The main reason for failing the algorithm is the nonconvex surface with sensitive boundaries. So the solution is to write the codes for newton raphson algorithm or BHHH or other modified algorithms.

I wrote BHHH algorithm for my model. I was not able solve it fully, it was not giving the optimum. The reason is the direction for each iteration when it reaches sensitive area is not properly scaled. ie. either it overshoot in some dimension or there is no significant improvement in some direction. So to get appropriate multiplier on optimal direction fails here.

Now I will discuss about the heuristic part of the modification in the algorithm. In this heuristic, certain number of dimensions are randomly selected and perturbed. Remaining dimensions are left unchanged. This way we can overcome the scaling issue. This given me optimal solution but it is not optimised in the execution perspective. It take slightly more time ( I compared few simple models with my heuristic and existing garch estimation).

This approach looks very silly for regression type problems. But in garch like process tweaking like this is very much required. I am still looking for the better way. Anyway the blogging helped me to revise  my approach again. Please let me know if you have any suggestions.

Why Emacs is important to me? : ESS and org-mode

I cannot believe that I lived without emacs!  Now I use emacs more than any application. The usage  of emacs is going to increase as the days go. A non-emacs user  may think that why an editor should get a credit like “emacs is an integral part of my life”. Emacs users may find nothing unusual about my statement. Well! I will brief my emacs story. I will start with my programming life.

My programming life

My programming life is summarized as below. Even though I learned multiple languages I have included the languages where I spend more than 30 days(or more than 200 hours) roughly.

  • 2000-2001: Basic and FORTRAN                ->Windows days
  • 2002: C and C++                                             ->Windows days
  • 2003: VC++ and OpenGL                             ->Windows days
  • 2004: C and C++                              ->Here onwards Linux is my favorite desktop OS
  • 2005-2006: JAVA, VB and VFoxpro* (I don’t think anybody uses this one)
  • 2007-2009: SAS and Shell Script
  • 2010-2011: R
  • 2012 : R, Python and elisp

Out of this I consider 2002, 2003 and 2012 are my best years of programming. I should consider my first few years were my golden days of programming. Those days I didn’t have to do anything (No responsibility) other than coding and some mathematics. At that time I never understood the importance of skill of programming. I treated programming as one subject of mathematics. I did programming mainly for fun. I thank my best friend Jais for my consistent interest in coding. We had lots of intellectual discussion and debates. We spent lots of hours to create graphical applications, screen savers, hacking, etc. Now, after a long break, I am back to programming lifestyle.

When I worked for a software company (2005-2006) I did more cut-copy-paste and trial & error methods. This way I never learnt anything deep. Money and movies where other distractions. Weekends were mainly window shopping and parties. The main reason why I didn’t do any real programming was I didn’t get a good friend who is crazy enough on geek stuff.

What motivated to learn Emacs

I always like simple applications. When I say simple it means quick execution, transparent functionality and customisable. I really like VI/vim editor and it was my favorite editor till 2011 (I don’t consider the Windows era). I tried emacs in 2004 and I didn’t like it. It was a nightmare at that time because to remember hell number of commands. Again, I tried emacs in 2010-2011. This time the motivation was  that fact that ’emacs  topped as the favorite editor for most of the stat tech guys’. But I didn’t learn anything other than C-x C-f and C-x C-c. Now what happened? I was searching a good R editor. I got to know about ESS and ESS is the first reason to learn emacs. So I started using emacs only for R coding. Then I started exploring shortcuts. I found it is very easy to customize. I have copied few short cuts and started customizing my own. Fortunately stackoverflow is very helpful on this. Some of the customisations are

  • auto complete
  • execution short cuts
  • folding mode

First two features are very common ins standard editor but, not the third one. I have seen folding mode in Rstudio. After experimenting on customisation I started learning emacs lisp. This made me think one editor for all programming.

The main reason to learn emacs is Org-mode. I guess this should be one of the key addons to emacs. Like ggplot2 in R. The advantage of org-mode is integrating emacs with:

  • task management
  • connect with other applications (HTML, latex beamer, google calendar)
  • table creation
  • Good for draft/notes preparation, etc.

I am mainly a statistician and coding is not a primary work. But still for all my data modelling and technical report I depend on emacs.  Now I use emacs for latex, python, R, daily planning and blogs (including this one).

One of the main reasons I switched to linux is constant learning. Emacs will ensure the constant learning at best.

Emacs is good enough for preparing daily planning, latex, R, Python, and other programming. With just a console I can use emacs and do most of my work. I am exploring on other applications emacs like reading mail, news feeds, twitting, and other automation. Already the.All files are available on above, but, looking for perfect one. I started exploring on elisp. I will post about my elisp experiment in future.