Memetic Musings

Vimpy – A vim plugin to make navigation easy on python projects

me — Thu, 22 Sep 2011 18:14:23 +0000

Update: The post below has become obsolete with Vimpy 0.2. Now from inside vim, you just need to:

Use :VimpyCreate path [exclude-path] to create index (One time operation)

:VimpyLoad to load it
And your index will be updated automatically as long as you edit files from Vim.

Old Post:
If you work on lot of python files, it may not be easy to navigate between files with vim. I tried a bunch of tools like ctags/ptags, cscope, or tips with gf etc, but I wasn’t satisfied with any of them. So I wrote this vim script to make life easier. Have a look at the following video to see what it does and try it out!

Note: This is in alpha state and just provides basic ctags like features (hopefully in a better way). I’ll be updating this when I get time. Code is on github. Do let me know your feedback in case you try it.

Timeline:
0.0 – Installation
0.25 – Creating tag/index file
1.50 – Using Vimpy!

Understanding reference cycles in Python

me — Wed, 24 Aug 2011 18:39:16 +0000

Introduction

In Python, garbage collection is done by reference counting, which means objects get garbage collected once their references reach 0. A reference to an object is created via the assignment operation, and it can be removed by a del or automatically when the variable goes out of scope or reassigned. A simple example would be:

1
2
3
4
5

a = "hello" # string object "hello" has refcount = 1
b = a # refcount = 2
a = A() # refcount of "hello" = 1, A = 1
a = None # A is gc'd
del b # "hello" is gc'd

What are reference cycles anyway?

If, for some reason, an object’s reference count does not reach 0, it will never be garbage collected by the above approach. How can this happen? This can happen when the reference graph (a set of objects as nodes and its references as edges) has a cycle. Again a simple example would make it clear:

1
2
3
4
5
6
7
8
9

class A:
def __init__(self, name):
self.name = name
def __repr__(self):
return "A [%s]" % self.name
a = A('a')
b = A('b')
a.ref = b
b.ref = a

Let’s see how its reference graph looks like:

Simple Cycle

So does that mean there is no way to cleanup these objects?
No.
Python’s gc has a mechanism to detect these cycles and get rid of these objects. The gc module provides some useful functions to interact with the garbage collection. The gc.collect() function can be used to invoke a full collection (which also cleans up unused objects in cycles) on demand. It is also called periodically by python based on the thresholds set using set_threshold(). We can also set some debug options to check which objects are stuck in a cycle using set_debug(). So for our example above we get the following output:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

>>> import gc
>>> gc.set_debug(gc.DEBUG_SAVEALL)
>>> gc.collect()
8
>>> import pprint
>>> pprint.pprint(gc.garbage)
[A [b],
{'name': 'b', 'ref': A [a]},
<class __main__.A at 0x00000000029DC9E8>,
A [a],
{'__doc__': None,
'__init__': <function __init__ at 0x0000000002B73D68>,
'__module__': '__main__',
'__repr__': <function __repr__ at 0x0000000002B73DD8>},
{'name': 'a', 'ref': A [b]},
<function __repr__ at 0x0000000002B73DD8>,
<function __init__ at 0x0000000002B73D68>]

This shows the cycle (In the absence of cycles, the gc.garbage list would be empty).

Well, in a typical python script one need nor worry about all this because python would clean this automatically anyway. However, if you have a long running process (say a web server), it is often desirable to disable the full collection or at least to increase the delay. This is to avoid unpredictable gc pauses. In such an application, it is prudent to write code that avoids cycles. It might appear that it would be an easy thing to do. But there are various other ways in which a cycle can be introduced. Let’s look at couple of non obvious cases.

Example 1

Consider the following code. There is a cycle hidden in there, can you spot it?

1
2
3
4
5
6
7

class C:
def cleanup(self):
pass
def __init__(self):
self.mycleanup = self.cleanup

c = C()

Nothing obviously wrong there, so lets check if there is indeed a cycle:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

>>> c = C()
>>> c = None # This should have removed C.
>>> gc.collect()
3
>>> pprint.pprint(gc.garbage)
[{'mycleanup': <bound method C.cleanup of <__main__.C instance at 0x0000000002CAC908>>},
<__main__.C instance at 0x0000000002CAC908>,
<bound method C.cleanup of <__main__.C instance at 0x0000000002CAC908>>,
<__main__.C instance at 0x0000000002B9CDC8>,
<class __main__.C at 0x00000000029DCD58>,
{'cleanup': <bound method C.nocleanup of <__main__.C instance at 0x0000000002B9CDC8>>},
{'__doc__': None,
'__init__': <function __init__ at 0x0000000002B85358>,
'__module__': '__main__',
'cleanup': <function cleanup at 0x0000000002B85278>,
'nocleanup': <function nocleanup at 0x0000000002B852E8>},
<bound method C.nocleanup of <__main__.C instance at 0x0000000002B9CDC8>>,
<function cleanup at 0x0000000002B85278>,
<function nocleanup at 0x0000000002B852E8>,
<function __init__ at 0x0000000002B85358>,
(False,),
{'mycleanup': <bound method C.cleanup of <__main__.C instance at 0x0000000002B69348>>},
<__main__.C instance at 0x0000000002B69348>,
<bound method C.cleanup of <__main__.C instance at 0x0000000002B69348>>,
{'mycleanup': <bound method C.cleanup of <__main__.C instance at 0x0000000002B691C8>>},
<__main__.C instance at 0x0000000002B691C8>,
<bound method C.cleanup of <__main__.C instance at 0x0000000002B691C8>>]

So there’s indeed a cycle. This is how this looks graphically:

Bound Method Cycle

Here’s what happened: We created a bound method object and saved it in a variable. The bound method object stores a reference to its class which has a reference back. When you call an instance method, this bound method object is created and destroyed in the background and hence there is no issue. So if you are saving method references and passing it around, be careful.

Example 2

Consider the following code. Again, there is a cycle hidden in there:

1
2
3
4
5
6
7

def outer(n):
def inner(n):
if n > 1:
inner(n-1)

outer(5)
#At this point there should not be anything left in memory.

Lets do the gc.collect():

1
2
3
4
5

>>> gc.collect()
>>> pprint.pprint(gc.garbage)
[<cell at 0x00000000026D6048: function object at 0x0000000002C9FE48>,
(<cell at 0x00000000026D6048: function object at 0x0000000002C9FE48>,),
<function inner at 0x0000000002C9FE48>]

Or graphically:

Closure Cycle

Here, the cycle is more subtle. Since we have a recursive inner function (rare but useful in some cases), there is a cell object created which holds a reference to the inner function. How to avoid this? Either make Outer a class or explicitly del the reference to inner.

Conclusion

Reference cycles can be tricky to notice in python. If your application has disabled gc, then lookout for such cycles using the gc debug options. The objgraph tool is also quite useful in tracking down memory issues (The images shown above where created using it).

Basics of Concurrent Programming – 1

me — Thu, 24 Dec 2009 00:41:00 +0000

I have some free time and its high time to revive my blog. Since multi-core (r)evolution is expected to make concurrent programming the norm than exception, that is our blog’s focus area (though I’ve not completely bought into it). I’m planning to write a ‘for dummies’ series on this topic.

Let’s start with the basics.

For quite a long period of time, we had concurrency constructs such as Semaphores and Monitors (and other less known/used stuff like conditional critical sections, CSP etc). Among these, monitors are more popular and provides better abstraction model over semaphores. However semaphores and monitors are like C and Java. One is more flexible/powerful; other one is easier to use. We will leave semaphores for now and start with Monitors.

Time for an example.

Let’s take the classical Reader/Writer problem.

We want to have some shared data where multiple writers are allowed, as well as multiple readers, but at no time should readers and writers be allowed in at the same time. However, multiple readers can read the data at the same time.

We will start with a naive java solution (with no concurrency at all). The shared Data can be defined as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13

public class Data {
StringBuilder value = new StringBuilder();

public String read() {
return value.toString();
}

public void write(String s) {
System.out.println("Writing " + s);
this.value.append(s);
System.out.println("Wrote " + s);
}
}

It’s pretty simple. We use StringBuilder as the data type, but this can be generalized easily. Our aim is also straightforward – read and write methods should not execute in parallel. However, multiple reads can happen in parallel.

We also need Readers and Writers to use the data. That is easy too.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46

class Reader extends Thread {
private Data data;
private int readerId;

public Reader(int readerId, Data data) {
this.data = data;
this.readerId = readerId;
}

@Override
public void run() {
//Each reader reads 10 times (say)
for(int i = 0; i < 10; i++) {
System.out.println("Reader "+ readerId + " : "+ data.read());
}
}
}

class Writer extends Thread {
private Data data;
private int writerId;

public Writer(int writerId, Data data) {
this.data = data;
this.writerId = writerId;
}

@Override
public void run() {
//Each writer writes 10 times (say)
for(int i = 0; i < 10; i++) {
data.write("W" + writerId + ":"+ i);
}
}
}

public class Main {
public static void main(String[] args) {
Data d = new Data();
//We have 5 readers and writers as an example.
for (int i=0; i < 5; ++i) {
new Reader(i, d).start();
new Writer(i, d).start();
}
}
}

If we fire up the Main class, we can see the following output (or something similar):
Writing W0:0
Wrote W0:0
Writing W0:1
Wrote W0:1
Reader 3 : W0:0W0:1
Reader 3 : W0:0W0:1
…

As you can guess this solution does not work as read and write can be called simultaneously. So you may see output like:
Writing W0:0
Reader 3 :
Wrote W0:0

That is, some one is reading while you are writing. There are many other race conditions that can happen like two writers writing simultaneously etc.

Ok, let’s make our Data class ready for concurrent access. A naive way of doing that would be to make read and write methods synchronized. But that would block multiple readers accessing the data too, which is restrictive. We will be effectively serializing the access to Data.
Digression: Java initially had Hashtable and Vector classes which suffered from this problem. All the methods were synchronized, so if someone was checking the length of the collection, he will block others who are doing get (or any other read operation). This problem was finally solved from Java 1.5 onwards with concurrent collections.

Back to Reader/Writer. We need to track who is accessing the data and synchronize accordingly. So we maintain no.of readers currently reading. Writers wait if someone is reading. The last reader and writer notifies any waiting writer. Readers can go in simultaneously. Our modified Data class will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

public class Data {
private StringBuilder value = new StringBuilder();
private int readers = 0;

public String read() {
synchronized (this) {
readers++;
}
String val = value.toString();
synchronized (this) {
readers--;
if (readers == 0) notify();
}
return val;
}

public synchronized void write(String s) {
while (readers > 0)
try {
wait();
} catch (InterruptedException e) {
return;
}
System.out.println("Writing " + s);
this.value.append(s);
System.out.println("Wrote " + s);
notify();
}
}

Reader and writer remains the same. However, this is already getting complicated. Java has ReadWriteLock from 1.5 onwards, so you can use it directly instead of reinventing it. With that our solution will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import java.util.concurrent.locks.ReadWriteLock;
import java.util.concurrent.locks.ReentrantReadWriteLock;

public class Data {
private StringBuilder value = new StringBuilder();
private ReadWriteLock lock = new ReentrantReadWriteLock(true);

public String read() {
try {
lock.readLock().lock();
return value.toString();
} finally {
lock.readLock().unlock();
}
}

public void write(String s) {
try {
lock.writeLock().lock();
System.out.println("Writing " + s);
this.value.append(s);
System.out.println("Wrote " + s);
} finally {
lock.writeLock().unlock();
}
}
}

This is much simpler. In the background it will be doing stuff similar to what we did above. The advantage is we don’t have to maintain the queue and logic is much simpler. However, there is limited control on the fairness. In the above example, we had specified the ReadWriteLock to be fair, which means the system will provide a weak FIFO semantics for waiting threads. Other option is no fairness at all. But if we keep track of readers our-self (like in previous implementation) we can provide arbitrary priorities like Reader have more priority over writers and vice verse.

Anyway, that’s all for the first installment. Let me know your comments.

N Queens in Scala

me — Wed, 08 Jul 2009 03:20:00 +0000

Its been a while since I blogged. Even this is not a blog entry, its just a scala code snippet to solve the N queen problem. Its not particularly efficient, but by using lazy lists, we can run it on large numbers and get the first few results. Right now, not so lazy version:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

object q extends Application {

def check(a: Tuple2[Int, Int], b: Tuple2[Int, Int]) =
a._1 == b._2 || (a._1+a._2 == b._1+b._2) || (a._2-a._1 == b._1-b._2)

def safe(p: List[Int], n: Int) = p.zipWithIndex.forall(!check(_, (p.size, n)))

def queens(m: Int)(n: Int): List[List[Int]] =
if (m == 0)
List(Nil)
else
for (p <- queens(m-1)(n); i <- (0 until n); if (safe(p, i)))
yield p ::: List(i)

println(queens(8)(8))
}

Project Euler with Scala

me — Tue, 22 Jul 2008 04:21:00 +0000

I came across this interesting post about solving some Project Euler questions using Scala. Being a fan of Project Euler myself, I though I’ll outline some of the solutions that I came up using Scala. Of course, I have taken trivial examples and the objective is more to learn Scala than to have an efficient solution. So here are some examples:
Warning: Spoilers ahead.

Problem 13.
“Work out the first ten digits of the sum of the following one-hundred 50-digit numbers”. And hundred 50-digit numbers are given. (Not listing the numbers here to avoid clutter).

One possible solution:

1
2
3
4
5

object P13 extends Application {
val s = "..." //The numbers as a string separated by space.
val numbers = for (number val zero:BigInt = 0
println ((zero /: numbers)(_+_).toString.substring(0, 10))
}

Hmm, pretty simple program. We define an object which extends Application, so that we don’t have to write main method. Then we initialize the variable. Next line simply parses the string and creates a List of BigInts. We have a constant ‘zero’ which is BigInt 0. Finally we add the numbers using fold and take first 10 characters. If you don’t like /:, you can also use (numbers.foldLeft(zero)(_+_).

Problem 22.

“Using names.txt, a 46K text file containing over five-thousand first names, begin by sorting it into alphabetical order. Then working out the alphabetical value for each name, multiply this value by its alphabetical position in the list to obtain a name score.”
One possible solution:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

import java.io._
import scala.collection.jcl._
object P22 extends Application {
val res = new BufferedReader(new FileReader("c:\\names.txt")).readLine
val actualNames = res.split(",") map (_.filter(_.isLetter).mkString)
val sorted = new TreeSet[String]
sorted ++= actualNames
val nums = sorted map (_.foldLeft(0)(_+_-64))
var sum: BigInt = 0
var t = 0
for (j t += 1
sum += (t * j)
}
println(sum)
}

First, we read the lines of the file (so easy with scala). Then split based on ‘,’ and strip off the ‘”‘. Create a sorted set (we are using the same TreeSet from Java) and add the names. Calculate the value using fold left (Since they are upcase chars, subtract 64 to get int values). Ok, now we need to multiply each number by its position. Easy thing to do in imperative fashion. So lets do a traditional for each loop and that’s it.

Update: Seth Tisue pointed out that zipWithIndex can be used to do the final calculation. The problem considers index starting with 1, so we cannot directly use zipWithIndex. However zip can be used as below:

1
2
3

val l = nums toList
val indices = 1 to l.size toList
l.zip(indices).foldLeft(sum)((a,b) => a + b._1 * b._2)

Problem 42

1
2
3
4
5
6
7
8
9

import java.io._
import scala.collection.jcl._
object P42 extends Application {
val triangular = new TreeSet[Int]
for (i val lines = new BufferedReader(new FileReader("c:\\words.txt")).readLine
val nums = lines.split(",") map (_.filter(_.isLetter).mkString) map (_.foldLeft(0)(_+_-64))
println((0 /: nums)( (result, num) =>
if (triangular contains num) result+1 else result ))
}

By now this should be obvious. This example is similar to the one above. First we cache first 40 triangular numbers in an Array. Then read the file. Split lines by ‘,’, ignore special characters (by filtering) and find the required sum (using map). I’ve done it in one line as it appears natural. Finally, count the triangular numbers. I used a fold, but it can be done imperatively as well.

Hope you enjoyed it. Do let me know if you think there are better ways of doing this (Not in terms of performance, but in terms of elegance or functional style).

Design Patterns in Scala

me — Sun, 30 Mar 2008 00:40:00 +0000

Design patterns in Scala

The GoF Design Patterns are perhaps the most popular design fads of Software ever. However some people argue that they are more or less solutions to the deficiencies of Language itself (particularly C++/Java) and if the language is powerful enough they are redundant. Consider this excellent presentation from Peter Norvig for example.
I agree that patterns translate to quite natural and concise code in some languages, but the intend of the pattern may still be important. So lets consider some of the popular patterns in Scala – a language which like Ruby/Lisp etc reduces Pattern implementation to triviality (though its not a dynamic language!).

1. Singleton
Singleton is one of the frequently (over)used patterns. In spite of being the simplest of patterns, it is surprisingly difficult to implement a ‘pure’ singleton in Java. The particularly famous Double-checked locking and so on. However, in Scala it is trivial. Lets say you need to create a Registry object which is singleton. You can do that in the following way:

1
2
3
4
5
6
7

object Registry {
def getEntry(): Entry {
}
//Other fields/methods
}
//Create and use the singleton
val entry = Registry.getEntry

Thats it! The only difference from defining a normal class is that instead of the ‘class’ keyword we used ‘object’. If you have to map it back to java, this is equivalent of defining a class and all its fields/methods are static.

2. Strategy
Like any language where functions are first-class objects or where closures are available, Strategy pattern is obvious. For eg. consider the ‘taxing’ example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

trait TaxPayer
case class Employee(sal: Long) extends TaxPayer
case class NonProfitOrg(funds: BigInt) extends TaxPayer

//Consider a generic tax calculation function. (It can be in TaxPayer also).
def calculateTax(t: TaxPayer, strategy: (TaxPayer => double)) = {
strategy(victim)
}

val employee = new Employee(1000)
//A strategy to calculate tax for employees
def empStrategy(e: Employee) = Math.ceil(e.sal * .3) toLong
calculateTax(employee, empStrategy)

val npo = new NonProfitOrg(100000000)
//The tax calculation strategy for npo is trivial, so we can inline it
calculateTax(nonProfit, ((t: TaxPayer) => 0)

3. Factory
Factory pattern addresses the fact that object constructors cannot return arbitrary objects. If you call constructor of object A, you always get an object A. If you need different objects based on parameter types, you typically create a Singleton Factory. In Scala also you need to do the same thing, but in a bit more elegant manner.

1
2
3
4
5
6
7
8
9
10
11
12
13

object Car {
def apply(String type) {
type match {
case "Race" => new RaceCar();
case "Normal" => new NormalCar();
case _ => throw new Exception;
}
}
}
//And you can create cars like:
val myCar = Car("Race")
//instead of more verbose
//Car myCar = CarFactory.getInstance().createCar("Ferrari");

Basically, you can exploit the fact that what apparently looks like a constructor call to a singleton object is syntactic sugar for the “apply” method.
4. Visitor
Visitor pattern is considered harmful by many people, but still it has its value at specific cases. Lets see a typical java implementation of Visitor pattern:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

abstract class Expression {
//...
public void accept(Visitor v);
}

class Identifier extends Expression {
}
class Sum extends Expression {
}
//..

class Visitor {
public void visit(Identifier i) { }
public void visit(Sum s) { }
}

Typical java boilerplate code as you would expect. In Scala you can achieve the same result by using pattern matching:

1
2
3
4
5
6
7
8
9
10
11
12
13
14

trait Expression {
//...
}
case class Identifier(value: Int) extends Expression {
}
case class Sum extends Expression {
}

object EvalVisitor {
def visit(expr: Expression): Int = expr match {
case (Identifier(v)) => v
case (Sum(e1, e2)) => visit(e1) + visit(e2)
}
}

The pattern matching code is also more flexible.
5. Decorator
Finally consider the Decorator pattern. Since Scala supports mixins and implicit conversions, Decorator is also simple and natural. Conceptually, there are two ways you can decorate an object – by adding new functionality and extending existing functionality.
To decorate a class with new functionality, you can use implicit conversions. Consider the RichInt class in scala standard library in action:

1	val range1to10 = 1 to 10

which in scala translates to 1.to(10). However, there is no method “to” in Int class. So how does it work? Scala has an innovative concept called Implicits for pimping your library. In essence it is Ruby’s open classes but lexically scoped. Its interesting stuff and you can read about it elsewhere. For us, it suffices to say that there is an implicit conversion in Predef [implicit def intWrapper(x : Int) : RichInt] that does the trick.
Mixins are also good for decorating your classes and extend its functionality in different dimensions. For example consider the typical Reader interface:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

trait Reader {
type T
def read: T
}
trait SynchronizedReader extends Reader {
abstract override def read: T = synchronized(super.next)
}

trait BufferedReader extends Reader {
abstract override def read: T = {
//buffering code
super.read
}
}

//A concrete implementation
class FileReader extends Reader {
type T = char
def read: char = readSomehow()
}

//Now we can mix in stuff that we need
//Create a FileReader
val f = new FileReader
//create a fileReader which is synchronized
val syncReader = new FileReader with SynchronizedReader
//create a fileReader which is synchronized and buffered
val bsReader = new FileReader with BufferedReader with SynchronizedReader

Conclusion
Design Patterns are good recipes for designing software. However, most of them generally solve a language issue than a design issue. If you have a good Language patterns (at least most of them) will become trivial. For example, in a structured language, the concepts of virtual methods or classes may be a ‘Design Pattern’. Once you move to a powerful language, the design patterns that you deal with will also change. They will be at a higher level of abstraction. More on it later!

To Ruby or To Scala?

me — Mon, 18 Feb 2008 04:47:00 +0000

I have been exploring alternate languages for JVM for some time now.
Why? Certainly not because I think the Java language is dying. Though java is devolving, it has crossed the threshold of immortality – there are way too many applications written in java and java developers can be busy just by maintaining them (and also by inventing ‘frameworks’ to get around the deficiency of the language like DI, AOP etc). However, I have started to realize the power of languages – maybe even to the extend of believing the Sapir–Whorf hypothesis. And finally, I still believe that the JVM is one of the best deployment platforms available today and java has lots of ready to use libraries – so I don’t want to miss them as well. Luckily, there are host of new languages targeted at the JVM – Jython, JRuby, Groovy, Scala, Rhino, Clojure etc.

Finally I’ve narrowed down to two: JRuby and Scala. I’ll try to outline the positives and negatives of both.

Ruby is an extremely dynamic language. It is dynamically (and strongly) typed. In my opinion, Ruby is the best language for internal DSLs (maybe next only to Lisp). Most of the successful Ruby projects have some kind of DSLs – like Rails, Rake, Builders etc. In a way, DSLs are nothing but flexible and easy-to-use interfaces.
Scala is a statically (and strongly) typed language. However, it has a powerful type inference system to avoid the pain of providing the types. It looks suitable for large systems, where precisely defined interfaces are required.

So looks to me that these languages have their own niche in programming. If you need RAD, clean, flexible and easy-to-use interfaces and small to medium sized applications, use Ruby. For large scale applications, where type safety is more important, use Scala without the burden of statically typed language.

Lets see how these fair in the popularity factor:

Popularity: Ruby is already a popular language (thanks mainly to Rails), while Scala is not (yet).
Ease-of-learning: Scala is probably a bit easier for Java folks. But in terms of documentation Ruby clearly excels. Scala documentation is poor, though it is improving.
Ease-of-writing: Ruby maybe an inch ahead as it is dynamically typed.
Ease-of-reading: For large systems, Scala seems to be a winner (thanks to static typing).
Terse: Both languages support functional/OO/imperative style, and not much difference here as well.

I would also outline some cool features of these languages:
Mixins: Both of these languages support mixins (though in Scala it is in a controlled way)
Open classes: Basically you can add methods to existing classes any time you wish. Scala also supports this through implicits.
Functional programming: Both languages has true closures, you can use functions as values etc.
Again, in Ruby you have complete freedom, while in Scala it is more controlled.

Of course, as a last point, the tool support is no way near to that in Java. The IDE support (in terms of auto completion, debugging, profiling etc) is currently mediocre in these languages. Also the performance is not that great. It could only be a matter of time, though. And btw, we needed those IDEs because Java was too bad anyway

To conclude, there is no silver bullet in programming languages. I guess we are moving towards polyglot programming where synergy of multiple languages will greatly help in tackling complexity.

What’s a Software Engineer?

me — Mon, 28 Jan 2008 05:13:00 +0000

Recently there has been some debates on the definition of ‘Engineer’ and what Software Engineers do is not strictly engineering etc. Looks like Reg Brathwaite started it and Ravi Mohan wrote an excellent follow through. Of course not all agree with this. I’ll propound my definition of what we (Software Engineers) do:
Definitely our work (or atleast for the majority) does not resemble ‘Engineering’. To put things into perspective here’s one definition taken from Wikipedia:

“The creative application of scientific principles to design or develop structures, machines, apparatus, or manufacturing processes, or works utilizing them singly or in combination; or to construct or operate the same with full cognizance of their design; or to forecast their behavior under specific operating conditions; all as respects an intended function, economics of operation and safety to life and property.”

[Well, in my opinion that just shows how difficult it is to define a term like 'Engineering'.]

Anyway, applying that to Software, one can readily reflect that ‘Engineering’ is a misnomer. But then what’s a more appropriate term for it? Before coming up with a new term, let’s see what really makes it different from other engineering disciplines:

1. It is impossible to define software.
In most of the Engineering fields, the end product is often cleanly and precisely defined. Consider the building that you are sitting. First, it has definite attributes which can be defined beforehand. Like no.of floors, total area etc.
Now consider any piece of software that you have developed. Unless it was a small piece of code¹, chances are that its definition itself changed over time. Even if it didn’t, you have something which you can’t define completely.

2. Software is inherently complex.
This is so well known that I won’t dwell into it. The take-away is that building software is generally more complex than building a building (no pun intended). I agree with Harold Abelson that Computer Science is all about managing complexity.

3. Software development need not require ‘theoretical backing’
This is the key point which generates lot of heat. In fact this is the answer (or atleast one of the answers) to questions like why is Lisp/FP not popular or why is java popular. Unlike other disciplines of Engineering, software development does not require any Mathematical knowledge. And there has been a plethora of software developers who entered this field just because of this fact and have greatly contributed to the complexity of already complex software development. Needless to say, the minority of software developers who does know theory and develops software that is inexpressibly beautiful or that is indistinguishable from art also are dwarfed to the category of ‘Software Engineers’.

I hope that by now it is clear that ‘Engineering’ is not the right term. I would say a better term would be ‘Software Hacker’.
Yes that is what we do.
And we are lucky to have among us a few ‘Software Artists’ as well.

Footnotes

¹: Even this may not be true. Consider the following simple function that computes sum of two integers:

1
2
3

int sum(int a, int b) {
return a + b;
}

Even the definition of this function is tricky.
This will return the sum of two numbers as long as the sum is less than Integer.MAX_INT in java. In Ruby, this will behave differently.

What make a programming language popular?

me — Mon, 14 Jan 2008 02:37:00 +0000

Recently there were many posts on programming languages (for eg. see excellent posts by Steve Yegge and Mike Vanier). That made me wonder what makes a programming language popular. Popularity seems to have no relation with the quality. Does a language become popular out of luck (or its name)? Or is there something else to it. The following is my take on what make a language popular:

1. Popularity
The main factor that makes something popular is popularity itself. The statement may appear circular, but like recursive functions it is valid. When you decided to learn a new language what did you do? You would’ve read reviews, blogs etc. The more reviews you read the better. And if you are constrained by your employer this is doubly true. I agree that nerds don’t care about popularity. But nerds aren’t so common.

2. Ease of Learning
The next factor is ease of learning. This is the first thing if you leave aside popularity. Btw, this may not mean the same as intuitiveness.
Let’s say you have the option to learn C++ and Python. What would you choose? (Assume that you are not worried about other factors like performance etc)

3. Ease of Reading
Interestingly ease of reading and ease of learning are not the same. For example, Perl is easy to learn (at least to get started), but exceedingly difficult to read (especially large code-bases). Some languages become popular because of the learning factor but over time lose popularity because reading (and hence maintaining) code is difficult. So this is the first major factor that determines the long term popularity of a language.

4. Terse
Smaller code means easier maintenance. The ability of easily creating abstractions (using closures, macros etc) is the main factor here. But then Lisp should be more popular than java right? But remember, this is the 4th factor. And java excels in all the above factors (maybe due to the way computer science is being taught). And many people think that if it tries to add these ‘advanced’ features it’ll become dead. It is unfortunate that the main factor that has to be considered in choosing a language – the level of abstraction it provides – is ignored. But then as Russel said: If fifty million people say a foolish thing, it’s still a foolish thing.

Delegate using Dynamic Proxy

me — Sun, 09 Dec 2007 09:17:00 +0000

Java had the Dynamic proxy since 1.3. Somehow it is not a well known feature, maybe because the documentation is not so good. Of course there are some good articles written by Brian. Anyway I am going to show you how you can use Dynamic proxy to delegate easily.
First lets imagine you want to use the features of an existing class, but keeping good OO practices in mind, you decide to use delegation. Now the problem is you need to add lot of methods just for delegating to the actual class.
For eg, lets say you want to use the functionality of a List. You define your class and delegate to the actual List. It may look like this:

1
2
3
4
5
6
7
8
9
10
11

public class MyList implements List {
private List list;
public MyList() {
this.list = new ArrayList();
}

public void add(Object item) {
list.add(item);
}
//Similarly override all methods
}

This is not particularly interesting. Of course the IDEs help, but if the interface changes you need to add,recompile etc.

Enter Dynamic Proxy.

Simply put, there are two things you need to do to implement dynamic proxy. Create a Proxy (this sits between the user and your object. for eg. MyList) and create an InvocationHandler (This guy will intercept all methods and allows you to override at runtime).
Before we begin, the proxy creation method is not pretty so lets have a utility method that does that for us:

1
2
3

public static <T> T getProxy(Class<T> clazz, Object obj) {
return (T) Proxy.newProxyInstance(clazz.getClassLoader(), new Class[] { clazz }, new Handler(obj));
}

Now we will write the Handler (which is our InvocationHandler):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

class Handler implements InvocationHandler {
private Object obj;

public Handler(Object obj) {
this.obj = obj;
}

public Object invoke(Object proxy, Method method, Object[] args) throws Throwable {
String methodName = method.getName();
try {
//Try the real object first (eg. MyList)
Method m = obj.getClass().getMethod(methodName, method.getParameterTypes());
return m.invoke(obj, args);
} catch (NoSuchMethodException e) {
}
try {
//That failed. So try the underlying object (eg. ArrayList)
Method m = method.getDeclaringClass().getMethod(methodName, method.getParameterTypes());
Object delegator = obj.getClass().getMethod("getUnderlying").invoke(obj);
return delegator.getClass().getMethod(methodName, method.getParameterTypes()).invoke(delegator, args);
} catch (NoSuchMethodException e) {
e.printStackTrace();
}
return null;
}
}

Hmm, so what’s going on here?
We intercept all calls to MyList and check whether it is implemented in MyList. If not, we will get the underlying object (For this we assume a method ‘getUnderlying’ is defined on the main class. Ideally this will be passed as argument) and invoke that. Simple isn’t it?

Finally we don’t even have to implement List. Our class will look like:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

public class MyList {
private List list;

public MyList() {
this.list = new ArrayList();
}

//The methods specific to this class..
public void addItem(Object item) {
list.add(item);
}

//Get the delegated object
public Object getUnderlying() {
return list;
}
}

Ok, all that looks good, but how to use it?

1
2
3
4

List list = ListProxy.getProxy(List.class, new MyList());
list.add("a"); //add method of ArrayList
list.addItem("b"); //addItem method of MyList
println(list.size()); //size method of ArrayList : prints 2

What have we done?
1. We used delegation without the pain of overriding all methods which have default behavior.
2. We used Duck typing (Though MyList does not implement List, we can use it as a List).
3. This code is generic and can be used for any interface/class delegation.

Thats all for now, folks!