Aug 5

Dear users,

It is with sadness that I have to inform you that Watexy will be discontinued. This is a direct consequence of Google pulling the plug on Google Wave (see e.g. http://digihub.theage.com.au/node/1792).

Watexy (and associated web-service) gets deleted on 31st of August 2010.

I’m very sorry, but hosting the bot is quite expensive, so I don’t see much point in making the deficit any bigger when Google Wave is going to close anyway. And I have to remind you that Google Wave has always been a beta, so the risk of this happening has been present all the time.

Deleting Watexy and associated web-service means that all your waves with Latex-images/-equations will be useless; the images will not appear any longer. This means that you either have to print the waves before 31st of August 2010 or use the history functionality to get back to the Latex-codes.

I am very sorry for this but I see no other opion.

Jul 14
STA version 0.2
icon1 Mikkel Meyer Andersen | icon4 July 14, 2010 at 19:06 (UTC) | icon3 No Comments »
icon3 , ,

Already an update with the following changes from version 0.1:

General:

  • Icon changed
  • Decimal separator always “.” no matter the chosen locale of the phone (for consistency purposes)
  • Screen rotate issues fixed

Distribution tool:

  • Typing error: Continuous distributions density output changed from “F([input]) = …” to “f([input]) = …”
  • Error description at the parameter tab if the parameters are illegal when trying to plot
  • Descriptives gets calculated automatically when sampling data
  • The link under properties has been made clickable
Jul 13
STA (Statistical Toolbox for Android) version 0.1
icon1 Mikkel Meyer Andersen | icon4 July 13, 2010 at 10:50 (UTC) | icon3 No Comments »
icon3 , ,

Finally, a (“beta”) version 0.1 of STA is available on the market. Just search for STA. Please let me know if you run into trouble or would like certain features!

Jul 6
STA: Statistical Toolbox for Android
icon1 Mikkel Meyer Andersen | icon4 July 6, 2010 at 13:32 (UTC) | icon3 3 Comments »
icon3 , ,

After having done some preliminary application development for Android (and finally finished my master’s), I’ve decided to start a new project. And to blog about the creation of this new project. (As an aside I would really like to point out that I haven’t forgot about Watexy, but for now it is not possible to improve it.)

The aim of the project is to develop an Android-application with basic statistical tools (I really miss R on my phone, but the project won’t be a R-clone nevertheless). So far the codename for the application is Statistical Toolbox for Android (or simply STA).

It is not going to be a programming language such as S, but an easy-to-use graphical statistical toolbox. The features I’ve thought about including in the first version are:

  • Quantiles (and fractiles) for a wide range of univariate probability distributions
  • Descriptive statistics (the first two or three empirical moments, correlation measures)
  • A guide for choosing the right statistical test

The features for the later versions could be:

  • Loading datasets (from mail, files on SD-card, or manual input)
  • A range of statistical tests

If any of you have any comments, please do not hesitate to submit a comment here or by mail (use the contact form accessible from the top menu or by sending an e-mail to the reverse of mikl.dk @ scienco ).

Feb 25

I’ve been working on a robot just like Watexy, but instead of converting Latex to images, it should convert to MathML-fonts. And I’m actually quite far with the robot, so I assume that something will be ready within a couple of days (or maybe a week). Stay tuned!

Oh – and I removed the ads on the site. They was indeed annoying! I hope that you will help the development and server expenses by donating a bit if you can afford it. And thanks to those who already donated – it’s much appreciated!

Nov 14
Watexy version 14 – with both inline and display math mode
icon1 Mikkel Meyer Andersen | icon4 November 14, 2009 at 23:15 (UTC) | icon3 35 Comments »
icon3 , , , ,

Now version 14 of watexy@appspot.com is released.

$ $x$ $ makes an inline equation (with the $-signs immediately following each other and not with a space as here).
\[ 4 + 5 \] makes an equation in “display math mode”, i.e. centred on its own line. Edit by clicking at the equation (the image).

There are still a few misbehaviours/new feature suggestions:

  • Inline equations still doesn’t support < and >
  • It’s still not possible to edit the inline equations
  • Have an align environment so that several equations can be shown underneath each other
  • The height of the equation in display math mode doesn’t always adjust automatically, so it may be necessary to either click it to edit and the press cancel or view another wave and go back to the original one
  • The history of how an equations is changed (editing by clicking on it) is not recorded, so for know it’s not possible to track changes to a single equations. Yet.

I’m of course still working to fix these things, but it might not be solved until January because I’m going to travel the rest of the year.

Thanks a lot for your support and all the feedback. Please continue to comment on how the robot is made, bugs, and suggestions! I’m also on Twitter at http://twitter.com/mikldk.

A last request: If you can afford, please donate money to support my work and expenses. You’ll find the donate-button in the upper right corner of this page.

Nov 13
Watexy-test (with edit and working < and >)
icon1 Mikkel Meyer Andersen | icon4 November 13, 2009 at 01:45 (UTC) | icon3 13 Comments »
icon3 , , , ,

I’m a new version of Watexy with the possibility to edit equations and fixes the bug with < and > not working.

The testing-robot is:
watexy-test@appspot.com

It only works in wave.google.com and not in wavesandbox.com for some reason.

I hope you’ll take the time to give it a try and some feedback to improve it.

watexy-test

Nov 11

First of all, thanks for the feed-back in [1].

I’ve now fixed three things:
1) It’s now possible to put more than one equation in a blip, and they all get nicely/properly displayed, thanks to Michael, post 18 in [1] for a possible solution
2) Instead of using http://www.forkosh.dreamhost.com/mathtex.cgi , I’ve now installed the program on one of my own servers (thanks a lot for both the software and the service)
3) Multi-line is supported, such that the equation can be written over multiple lines and still be matched correctly

Please do not hesitate to comment this version as well!

By the way, I’m sorry for the delay with this new version. My life as an exchange student in Australia has been a bit hectic with all the travelling besides the studies.

[1]: http://www.scienco.org/2009/watexy-latex-robot-for-google-wave/

The new code is as follows:

# Python reference:
# http://wave-robot-python-client.googlecode.com/svn/trunk/pydocs/index.html
 
# Shortcut to the important OpBasedDocument
# http://wave-robot-python-client.googlecode.com/svn/trunk/pydocs/waveapi.ops.OpBasedDocument-class.html
 
__author__ = 'mikl@mikl.dk (Mikkel Meyer Andersen)'
 
import re
 
from waveapi import events
from waveapi import model
from waveapi import robot
from waveapi import document
 
def OnRobotAdded(properties, context):
  """Invoked when the robot has been added."""
  root_wavelet = context.GetRootWavelet()
  root_wavelet.CreateBlip().GetDocument().SetText("Hi. My name is Watexy and I'm here to help you presenting Latex in waves. Just put the latex between $$ and $$, e.g. $$2+2=5$$.")
 
def reversed_iterator(iter):
    return reversed(list(iter))
 
def OnBlipSubmitted(properties, context):
  """Invoked when a blip has been added."""
  blip = context.GetBlipById(properties['blipId']) 
  blip_text_view = blip.GetDocument()
 
  matches = re.finditer('\$\$(.+?)\$\$', blip_text_view.GetText(), re.DOTALL)
 
  """
  Reverse list such that the last items will be changed first, such that
  the positions for the first items doesn't change
  """
  matches = reversed_iterator(matches)
 
  for m in matches:
    """
    The +/- 2 is because of the length of the $$'s. 
    If not removed, the loop will run infintely! 
    """
    blip_text_view.DeleteRange(document.Range(m.start(1)-2, m.end(1)+2))
    image = document.Image('http://meyer.fm/cgi-bin/mathtex.cgi?' + m.group(1), caption=m.group(1))
    blip_text_view.InsertElement(m.start(1)-2, image)
 
if __name__ == '__main__':
  myRobot = robot.Robot('watexy',
      image_url='http://watexy.appspot.com/assets/icon.png',
      version='12',
      profile_url='http://watexy.appspot.com/')
  myRobot.RegisterHandler(events.WAVELET_SELF_ADDED, OnRobotAdded)
  myRobot.RegisterHandler(events.BLIP_SUBMITTED, OnBlipSubmitted)
  myRobot.Run()
Aug 28
qqmultinorm.R version 1.1 – more intelligent plot size
icon1 Mikkel Meyer Andersen | icon4 August 28, 2009 at 07:20 (UTC) | icon3 No Comments »
icon3 ,

I’ve updated the qqmultinorm.R script a bit so that it’s now capable of picking a more optimal plot size (less unused space).

The idea refers to deciding the sides (dimensions) of a rectangle if the areal (number of plots) is known i.e., optimising the dimensions of a rectangle given the areal. We want the perimeter as small as possible (for better viewing), preferably wider than longer if square isn’t possible. A given number n is factorised in to numbers within a given error. For example if the allowed error is 2, then 7 gets factorised in (1,7), (2, 4), (4, 2), and (7, 1) (here redundant factorisation is included). Although 2*4 = 8, but abs(7-8) = 1 <= 2, so it's acceptable. Actually the default error is 10% of the input areal.

The way the proper factorisations is chosen, is by ordering the pairs by ascending the difference of the components i.e, how much the width and height differs. And then ordered ascending by height so that the plot gets wide screen-like instead of poster-like.

The new script is a follows (only find.optimal.mfrow.size(...) and a bit logic in qqmultinorm(...) added):

# File name: qqmultinorm.R
# Version: 1.1
# Last updated: 2009-08-28
#
# This R-code is made by:
# Mikkel Meyer Andersen, Denmark
# mikl [funny-a] math [.] aau [.] dk or 
# mikl [funny-a] mikl [.] dk
#
# Licence: GPLv2
#
# Feel free to use it, but if you do I'll like to hear about it (just for fun).
# If you make corrections, please submit them back so others can enjoy them as well.
 
qqchisq <- function(y, main, df=2, continuity.correction = 0.5)
{
  n <- length(y)
  y <- sort(y)
  c <- numeric(n)
 
  for (i in 1:n)
    c[i] <- qchisq((i-continuity.correction)/n, df=df)
 
  plot(c, y, xlab="Theoretical Quantiles", ylab="Sample Quantiles", main=main)
  lines(c(c[1], c[n]), c(c[1], c[n]), type="l")
}
 
dec2bin <- function(x)
{
  if (!is.vector(x) || length(x) != 1 || x < 0)
    stop("x must be a non-negative integer")
 
  N <- length(x)
  ndigits <- floor(log2(x)) + 1
  bin <- numeric(ndigits)
 
  for (i in (ndigits-1):0)
  {
    tmp <- 2^i
 
    if (x %/% tmp >= 1)
    {
      bin[i+1] <- 1
      x <- x - tmp
    }
  }
 
  return(rev(bin))
}
 
# Returns the power set without the empty set
power.set <- function(v)
{
  n <- length(v)  
  N <- 2^n - 1
  ps <- vector("list", N)
 
  for (i in 1:(N-1))
  {
    Nbin <- dec2bin(i)
    Nbin <- c(numeric(n-length(Nbin)), Nbin)
    Nbin <- rev(Nbin)
    ps[[i]] <- v[which(Nbin == 1)]
  }
 
  ps[[N]] <- v
 
  return(ps)
}
 
# Input: n
#  - here the number of plots
# Returns c(h, w)
#  - the optimal choice of rows and cols to use in in mfrow
find.optimal.mfrow.size <- function(n)
{  
  w0 <- ceiling(sqrt(n))
 
  # The area can at max contain of 10% unused space
  max.error <- round(n*0.1)
 
  n.minus.error <- n - max.error
  n.plus.error <- n + max.error
 
  # If n is a square, fine!
  if (w0^2 == n)
    return(c(w0, w0))
 
  # Col 1 and 2: w and h
  # Col 3: The difference between w and h: this should be as small as possible
  candidates <- matrix(ncol=3)
 
  for (w in w0:1)  
  {    
    h <- ceiling(n / w)
    n0 <- w*h
 
    # Because of ceiling we know that n0 >= n
    if (n0 <= n.plus.error)
      candidates <- rbind(candidates, c(h, w, abs(w-h)))
  }
 
  # First row is NA
  candidates <- candidates[-1,]
 
  # Uups, something went wront - well, don't panic
  if (nrow(candidates) == 0)
    return(c(w0, w0))
 
  # First order by abs(w-h) and then by h to get a widescreen-look 
  # instead of a poster-look
  candidates <- candidates[order(candidates[,3], candidates[,1]), ]
 
  return(candidates[1, c(1,2)])
}
 
# dataset: variables in columns and observations as rows
# subset.min.size, subset.max.size: inclusive limits
# filename: if specified, the plot are saved as a png file with this filename
qqmultinorm <- function(dataset, subset.min.size = 1, subset.max.size = 4, filename = NULL, use.optimale.size = F)
{
  p <- ncol(dataset)
  n <- nrow(dataset)
 
  if (subset.min.size < 1) stop("subset.min.size < 1")
  if (subset.min.size > p) stop("subset.min.size > p")
  if (subset.max.size < 1) stop("subset.max.size < 1")
  if (subset.max.size > p) stop("subset.max.size > p")
  if (subset.min.size > subset.max.size) stop("subset.min.size > subset.max.size")
 
  if (is.null(colnames(dataset)))
    colnames(dataset) <- 1:p
 
  # We have p variables. If all is to be checked against each other,
  # then we have a power-set with 2^p subsets (including the empty set)
 
  # Here we get subset containing indexes of the variables to include
  # Note that power.set doesn't include the empty set.
  subsets <- power.set(1:p)
  subsets.len <- length(subsets)
 
  # To get the plots with the fewest variables first, we do a litte trick:
  # While we find out which plots to include, we build a list
  # where each element of a list is the index of the subset,
  # and the index of the element is the size of the subset.
  # (The +1 is because the limits are includesive!)
  s.included <- vector("list", subset.max.size - subset.min.size + 1)
 
  plots <- 0
 
  for (i in 1:subsets.len)
  {
    s <- subsets[[i]]
    s.len <- length(s)
 
    if (s.len >= subset.min.size && s.len <= subset.max.size)
    {
      plots <- plots + 1
      s.included[[s.len - subset.min.size + 1]] <- c(s.included[[s.len - subset.min.size + 1]], i)
    }
  }
 
  # Now it's possible to build the subset index vector; 
  # we disregard the size of each subset; no more need to know it.
  s.indexes <- c()
 
  for (s in s.included)
  {
    s.indexes <- c(s.indexes, s)
  }
 
  # We want the best view  
  plot.per.row <- ceiling(plots^(1/2))
  plot.per.column <- ceiling(plots^(1/2))
 
  if (use.optimale.size)
  {
    mfrow.parameters <- find.optimal.mfrow.size(plots)
    plot.per.row <- mfrow.parameters[1]
    plot.per.column <- mfrow.parameters[2]
  }
 
  plot.width <- 300
  plot.height <- 200
 
  if (!is.null(filename))
    png(file=paste(filename, ".png", sep=""), bg="white", width = plot.per.row * plot.width, height = plot.per.column * plot.height)
 
  par(mfrow = c(plot.per.row, plot.per.column))  
 
  # There's no need to calculate a whole lot several times:
  ybar <- as.vector(colMeans(dataset))
 
  S <- as.matrix(var(dataset))
 
  current <- 1  
  for (i in s.indexes)
  {
    s <- subsets[[i]]
    s.len <- length(s)
 
    if (s.len > subset.max.size)
      next
 
    cat("Processing subset no.", current, "out of", plots, "\n")
 
    # Container for our values
    squared.dist <- numeric(n)
 
    # qr.solve(A) = A^(-1)  
    Sinv <- qr.solve(S[s,s])
 
    # Then calculate the squared distance for each datapoint
    for (i in 1:n)
    {
      c <- dataset[i,s] - ybar[s]
      squared.dist[i] <- t(c) %*% Sinv %*% c
    }
 
    # Finding the order statistic
    squared.dist <- sort(squared.dist)
 
    qqchisq(squared.dist, paste(colnames(dataset)[s], collapse=", "), s.len)
 
    current <- current + 1
  }
 
  if (!is.null(filename))
    dev.off()
}
 
# Example:
A <- matrix(rnorm(2000, mean=3, sd=2), ncol=8)
qqmultinorm(A, 2, 2, "multinorm-optimal.size", use.optimale.size = T)
qqmultinorm(A, 2, 2, "multinorm", use.optimale.size = F)
Aug 27
qqmultinorm.R – evaluating the normality of a sample
icon1 Mikkel Meyer Andersen | icon4 August 27, 2009 at 11:44 (UTC) | icon3 No Comments »
icon3 ,

I wrote a R-script that can be used to evaluate for multivariate normality of a sample. It’s a kind of generalisation to the qqnorm, but this one just uses sums of the squared statistical distance which is then chi squared distributed with degrees of freedom equalling the number of squared statistical distance summed.

The useful thing about this script, is that it’s able to plot all possible combinations of the variables. It’s possible to specify the minimum number of variables to compare and the maximum number of variables to compare. All possible combinations are found by calculating the power set (using binary representation through the decimal to binary conversion, dec2bin, because if a set has k elements, then its power set has 2^k elements, and the binary representation of the numbers from 1 to 2^k are then used to pick out the subsets in the power set).

Please notice that it’s possible to specify a filename so that the plots are written to a png file. This is by far the easiest thing – and only possibility – if there’s more than a few plots.

I haven’t wrote a lot of documentation besides this, but feel free to ask if in doubt of anything!

# File name: qqmultinorm.R
#
# This R-code is made by:
# Mikkel Meyer Andersen, Denmark
# mikl [funny-a] math [.] aau [.] dk or 
# mikl [funny-a] mikl [.] dk
#
# Licence: GPLv2
#
# Feel free to use it, but if you do I'll like to hear about it (just for fun).
# If you make corrections, please submit them back so others can enjoy them as well.
 
qqchisq <- function(y, main, df=2, continuity.correction = 0.5)
{
  n <- length(y)
  y <- sort(y)
  c <- numeric(n)
 
  for (i in 1:n)
    c[i] <- qchisq((i-continuity.correction)/n, df=df)
 
  plot(c, y, xlab="Theoretical Quantiles", ylab="Sample Quantiles", main=main)
  lines(c(c[1], c[n]), c(c[1], c[n]), type="l")
}
 
dec2bin <- function(x)
{
  if (!is.vector(x) || length(x) != 1 || x < 0)
    stop("x must be a non-negative integer")
 
  N <- length(x)
  ndigits <- floor(log2(x)) + 1
  bin <- numeric(ndigits)
 
  for (i in (ndigits-1):0)
  {
    tmp <- 2^i
 
    if (x %/% tmp >= 1)
    {
      bin[i+1] <- 1
      x <- x - tmp
    }
  }
 
  return(rev(bin))
}
 
# Returns the power set without the empty set
power.set <- function(v)
{
  n <- length(v)  
  N <- 2^n - 1
  ps <- vector("list", N)
 
  for (i in 1:(N-1))
  {
    Nbin <- dec2bin(i)
    Nbin <- c(numeric(n-length(Nbin)), Nbin)
    Nbin <- rev(Nbin)
    ps[[i]] <- v[which(Nbin == 1)]
  }
 
  ps[[N]] <- v
 
  return(ps)
}
 
# dataset: variables in columns and observations as rows
# subset.min.size, subset.max.size: inclusive limits
# filename: if specified, the plot are saved as a png file with this filename
qqmultinorm <- function(dataset, subset.min.size = 1, subset.max.size = 4, filename = NULL)
{
  p <- ncol(dataset)
  n <- nrow(dataset)
 
  if (subset.min.size < 1) stop("subset.min.size < 1")
  if (subset.min.size > p) stop("subset.min.size > p")
  if (subset.max.size < 1) stop("subset.max.size < 1")
  if (subset.max.size > p) stop("subset.max.size > p")
  if (subset.min.size > subset.max.size) stop("subset.min.size > subset.max.size")
 
  if (is.null(colnames(dataset)))
    colnames(dataset) <- 1:p
 
  # We have p variables. If all is to be checked against each other,
  # then we have a power-set with 2^p subsets (including the empty set)
 
  # Here we get subset containing indexes of the variables to include
  # Note that power.set doesn't include the empty set.
  subsets <- power.set(1:p)
  subsets.len <- length(subsets)
 
  # To get the plots with the fewest variables first, we do a litte trick:
  # While we find out which plots to include, we build a list
  # where each element of a list is the index of the subset,
  # and the index of the element is the size of the subset.
  # (The +1 is because the limits are includesive!)
  s.included <- vector("list", subset.max.size - subset.min.size + 1)
 
  plots <- 0
 
  for (i in 1:subsets.len)
  {
    s <- subsets[[i]]
    s.len <- length(s)
 
    if (s.len >= subset.min.size && s.len <= subset.max.size)
    {
      plots <- plots + 1
      s.included[[s.len - subset.min.size + 1]] <- c(s.included[[s.len - subset.min.size + 1]], i)
    }
  }
 
  # Now it's possible to build the subset index vector; 
  # we disregard the size of each subset; no more need to know it.
  s.indexes <- c()
 
  for (s in s.included)
  {
    s.indexes <- c(s.indexes, s)
  }
 
  # We want a squared view
  plot.per.row <- ceiling(plots^(1/2))
  plot.per.column <- ceiling(plots^(1/2))
  plot.width <- 200
  plot.height <- 200
 
  if (!is.null(filename))
    png(file=paste(filename, ".png", sep=""), bg="white", width = plot.per.row * plot.width, height = plot.per.column * plot.height)
 
  par(mfrow = c(plot.per.row, plot.per.column))  
 
  # There's no need to calculate a whole lot several times:
  ybar <- as.vector(colMeans(dataset))
 
  S <- as.matrix(var(dataset))
 
  current <- 1  
  for (i in s.indexes)
  {
    s <- subsets[[i]]
    s.len <- length(s)
 
    if (s.len > subset.max.size)
      next
 
    cat("Processing subset no.", current, "out of", plots, "\n")
 
    # Container for our values
    squared.dist <- numeric(n)
 
    # qr.solve(A) = A^(-1)  
    Sinv <- qr.solve(S[s,s])
 
    # Then calculate the squared distance for each datapoint
    for (i in 1:n)
    {
      c <- dataset[i,s] - ybar[s]
      squared.dist[i] <- t(c) %*% Sinv %*% c
    }
 
    # Finding the order statistic
    squared.dist <- sort(squared.dist)
 
    qqchisq(squared.dist, paste(colnames(dataset)[s], collapse=", "), s.len)
 
    current <- current + 1
  }
 
  if (!is.null(filename))
    dev.off()
}
 
# Example:
A <- matrix(rnorm(2000, mean=3, sd=2), ncol=8)
qqmultinorm(A, 1, 3, "multinorm-1-3")
#qqmultinorm(A, 4, 4, "multinorm-4")
#qqmultinorm(A, 5, 5, "multinorm-5")
#qqmultinorm(A, 6, 8, "multinorm-6-8")
multinorm-1-3

multinorm-1-3

« Previous Entries