Python3 Adventures

At my $dayjob, i work with python, but mostly python 2.7. As you might be aware, python 3 has been out in the wild since a very long time, and unfortunately many projects have not yet embraced it as much as python core devs would like. Our company has also been one of the many holdouts, partly because of the lack of compelling business reasons to move, not many advantages to moving (which is slowly changing with each new python 3.x minor release), and partly because there is much more higher priority work to be done.

However, thanks to some members of the community (burnpanck, who has python3 compatible (though not 100%) forks of parts of our core ETS stack, starting with traits, pyface and traitsui), this is slowly changing.
Based on his work, i have attempted to make chaco work on python3, and have largely been successful. The most significant changes were in the extension modules of enable, as expected, since the setup.py uses 2to3 to migrate python code. Enable's kiva renderer which uses AGG underneath, has significant amount of extension code. Luckily, most of it it written in swig which automatically takes care of python2 and python3 compatibility, though in one place i had to modify swig interface file for bytes<->str<->unicode madness. Chaco also has an extension module for contour plots which uses direct Python<->C api which i could make compatible with both 2 and 3 with some #ifdefs.

In short, migrating from 2 to 3 in not really difficult for most programs. 2to3 makes it even less effort, you just have to figure out the things which 2to3 cannot handle, which is easy if you have tests.

The proof:

Current status: Enable and Chaco seem to work very well under python3. All enable tests pass, though a few chaco tests fail. Also, the chaco toolbar overlay seems to segfault. All other examples in the Chaco demo run very well.
The story is not as so good for traits and traitsui though. The traits `implements` and `adapts` advisors do not work in python3. They are already deprecated in traits, so you must use the corresponding replacements from the new `traits.adaptation` package, even on python2. The same work is also required to be done for traitsui, which uses the old mechanism in quite a lot of place.

Python3 branches of ETS components:
Traits : https://github.com/pankajp/traits/tree/python3 (derived from burnpanck)
Pyface : https://github.com/pankajp/pyface/tree/python3 (derived from burnpanck)
TraitsUI : https://github.com/pankajp/traitsui/tree/python3 (derived from burnpanck)
Enable : https://github.com/pankajp/enable/tree/python3
Chaco : https://github.com/pankajp/chaco/tree/python3

PS: If you are as excited about this as I am, please help the cause by using the python3 branch in your project (they also work with python 2.7), and submit issues and pull requests for both python2 and python2 compatibility. Thanks.

PPS: If you are using a newer gcc version (tested with gcc 4.8.2) and experience segfaults in enable/chaco image plots (enable's `integration_tests/kiva/agg/image_test_case.py` segfaults for you), then you need to disable all optimizations from the compiler (export CFLAGS='-O0'; export CXXFLAGS='-O0') while building enable because apparently gcc's optimizer eliminates some necessary code.

As an alternative, you can use the clang compiler (export CC='clang'; CXX='clang++') which doesn't exhibit this problem at any optimization level. I have yet to figure out whether it is a gcc bug of a bug in enable.

cgroups: Maintain sanity of your development system while you screw it

I am sure many of us developers run into unresponsive machines while doing our usual stuff (developing/debugging) due to a rogue or buggy process consuming all the system's RAM, and swap. It is not usually very common, but happens fairly often for me. For example, if in an ipython shell you start allocating arrays in a loop, and add an extra zero to the size or simply store the arrays without it being garbage collected, it is easy to bog down the system if you are not careful.

For me, the most common instance of this has been using gdb. I usually install all the relevant debuginfo packages available in the Fedora repo to enhance my debugging experience. One downside of this is that gdb consumes a lot of memory. And coupled with python pretty-printing and `thread apply all bt full` command, gdb can go haywire, especially when the stack/heap is corrupted after the segfault. Previously, i would have no recourse but to hard reboot my machine, i once timed that my system to be unresponsive for more than half an hour.

The Cause:

Linux is generally good at time-slicing and not letting rogue processes DOS the system, except for the memory-swap part. The swap system in linux is still not very intelligent in swapping out portions of memory when under pressure. It seems it swaps out data and code of processes equally aggressively.

Here's what happens when you start a process which consumes a lot of memory (more than the total amount of RAM on your system) to force you to swap:

 First data of inactive processes is swapped out, you can see you machine's swap usage increasing but apparently it doesn't seem to affect the system responsiveness;

 next, executable code of processes is swapped out too, and now is when things go out of control.

When the latter happens, your machine is screwed, since your terminal process and bash and window manager and gnome-shell and X server have all likely swapped out their executable code to be able to respond to you soon, and there are only two ways to recover: 1) Hope the rogue process quickly dies, which is not very likely on my system with 8GB RAM and swap (it takes a looong time to allocate so much memory while your executable code is itself swapped out); 2) Hard reboot

Not any more, i present to you a mechanism to maintain sanity of the system and limit the amount of RAM you development processes can consume.

A Workaround:

Welcome to cgroups (Control Groups), a feature in linux kernel for managing resources of processes. You can read more about cgroups from the fedora guide at http://docs.fedoraproject.org/en-US/Fedora/17/html/Resource_Management_Guide/index.html .

Here, i will describe a simple way to restrict the cumulative memory (RAM) consumption of all processes started in a bash terminal so that essential system processes have sufficient RAM available to maintain system responsiveness.

The various tools we will use are provided by the libcgroup-tools package, so install it first using yum or the equivalent package on you distro.

Two important services i will use are cgconfig.service and cgred.service, but before enabling and starting them we will configure them. Their config are respectively located in /etc/cgconfig.conf and /etc/cgrules.conf. 

Here's what we will configure cgoups for:

1) Create a bash_memory group for memory subsystem and limit the total RAM consumption to slightly less than the total available RAM, which on my system is 8GB, so i have set the limit to 6GB. This is done via the /etc/cgconfig.conf file. Add the following content to it:

group bash_memory {

        memory {

                memory.soft_limit_in_bytes="5583457480";

                memory.limit_in_bytes="6442450944";

        }

}

The first line in the memory subsystem states that in case of memory pressure, when the system is actively thrashing, the memory of all processess in the bash_memory cgroup will be reclaimed (by discarding caches and swapping out dirty pages) to reduce it physical memory usage to 5.2 GB.

The second line states that the total physical memory consumption of all processes in bash_memory cgroup will never exceed 6GB, they will be swapped out instead.

You can use similar mechanisms (using different subsystem than memory) to limit various resources, like cpu usage, disk IO, network bandwidth etc.

2) Now all we need to do is add processes to the bash_memory cgroup. We will do this via the cgred service by adding a rule to the /etc/cgrules.conf file. Add the following line to the file:

*:bash          memory          bash_memory/

The first column says that the rule applies to bash process of all users, the second says that the memory controller is being set, and the third column says that the bash_memory group is applied for the memory controller. Now, the cgred.service will take care of automatically applying the bash_memory group to all bash processes whenever they are started. Due to the inheritance of cgroups, all subprocesses started by bash (and their subprocessess too) will belong to the bash_memory cgroup, thus limiting their cumulative RAM consumption.

You can add more lines to limit specific users or processess.

3) Now all we need to do is start the services and reap their benefit.

sudo systemctl enable cgconfig.service

sudo systemctl start cgconfig.service

sudo systemctl enable cgred.service

sudo systemctl start cgred.service

The linux init daemon systemd uses cgroups to control various services, and you can set specific limit on all services in a systemd based system. Also, the lxc project (lightweight linux containers) uses cgroups and the related namespace functionality to sandbox linux containers. This feature of linux is being used for great benefit by various projects.

Please comment if you find this useful or have some better suggestions :)

PS: It seems the gdb memory hog bug which started all this adventure has been fixed: https://bugzilla.redhat.com/show_bug.cgi?id=1013453

Two Years @ Enthought: A Review

Last week, i completed two full years working for Enthought. It has been a great time working with some of the best minds and wonderful people i have known. Two years is a long time for being at same job i hear. None of my flatmates is at the same place where there were when i joined, and the same with two of my best friends. In fact some friends keep asking me where i am, as if its like changing clothes (hey, what color are you wearing today?). To come clean on this, let me clarify: i am very well loving my work here and do not see any change in the status quo foreseeable future. I'm well into my comfort zone.

I still vividly remember when we started off, right after finishing college, when Enthought India was still an idea. For few weeks, we used to work at the my graduate lab in IIT itself. I and Prabhu visited a bunch of places around in search of office space, and Eric has asked us to get a good one. We chose VBC after a few other visits, simply because it would let us start immediately in a pre-furnished office, and it was a good office no doubt. We started work there even before the agreement was officially signed, with Bill's visit. Puneeth also joined us physically on 5th Sept. I remembered him as the hard-core Free Software guy, who bought an expensive "Adorable GNU" in an auction by Richard Stallman when he visited IIT Bombay. For a few months, we worked on a contract with Enthought USA, until Enthought India was officially founded in December. Since then we have seen a significant growth in Enthought India, with more good company in the office.

In the last two years, we have hit several milestones (and probably missed more), and it has been a great journey of learning, doing and sharing. In summary:

The Good:

• Working with some of the best people i have ever known, both intellectually as well as personally.
• The lack of "Corporate Culture". I have only heard horror stories of "corporate culture" from friends, TV series and movies. Thought secretly, i wish i could experience it once (for maybe a week or so), just for the feeling and also that i could empathize with my friends.
• Meaningful work which is used by many scientists and engineers across the world, that which we can be proud (and ashamed) of, and take responsibility for.
• Culture of collaboration and working together, instead of delegating and taking credit.

The Bad:

• Bugs: However you try, bugs are inevitable. They are the nightmare of every developer. The problem is not just the code you wrote, but also the libraries which you use. The former cause embarrassment, and the latter frustration.
• Conflicting demands by users: Though you wish to support all users, it is simply not possible. You have to decide among conflicting ideas based on your insights, the work required for implementation, and sometimes who is paying more.
• Time Management: There is always more to do in life than the time you have to do it, and prioritization is a hard problem :)

The Ugly:

• Mumbai City: One of my friend from Delhi says he can identify Mumbai city by its omnipresent stink. Mumbai is hyper expensive for whatever it is that it offers (i can't think of getting a home in Mumbai without robbing a bank, or joining the government (i.e. robbing the people). Add to that the pollution of the city which reduces average life expectancy by a minimum 10 years, and the lack of any open spaces in the city (we have to commute 13 km to the beach to play ultimate, and that too is unplayable during the monsoons by the stink of the garbage and the injury risk). I still can't say i love Mumbai, despite staying here most of my life. What i'd prefer is staying by the woods, on a hill beside a lake or a river.
• Once i had a funny dream about Prabhu taking a transfer from IIT Bombay to the newly set up IIT Gandhinagar (which is mentored by IIT Bombay) to set up Aerospace Department there, and we (Enthought India) moving the office to SEZ in GIFT, Gujarat for the tax sops and the generally better administration, infrastructure and quality of life :)

QPlainTextEdit with setMaximumBlockCount performance regression

Recently while working with ipython qtconsole, i realized that it is easy to freeze the qtconsole or any app embedding the it by simply doing a "for i in xrange(10000000):print i" loop, then all arguments about separate kernel process safety etc. are voided. Since this is not something i liked, i set about to fix it in true nature of open-source development (scratching my own back). In this post i'll describe some Qt issues, benchmarks and the approaches i took and how you too can deal with similar problem of overwhelming text input to your text buffer.
Note: The ipython pull request https://github.com/ipython/ipython/pull/3409 is part of this experiment.

The Problem:

The qtconsole widget is a QPlainTextEdit, with the maximumBlockCount set to 500 by default, which is a very small buffer size by any standard. However, despite this, in case too much text is written to the QPlainTextEdit, it takes too much time drawing the text and the application appears to be frozen This is because the Qt event loop is blocked by too much time consumed by drawing the QPlainTextEdit and incessant stream of new text to draw at a faster rate than QPlainTextEdit can render.

The Solution Approaches:

My first though at the problem was to use a timer to throttle the stream rendering rate, and append text to the area only every 100ms. That wouldn't cause any perceptible usability loss, but make the program more responsive. Also, another essential idea was to only send the maximumBlockCount number of lines of text ot the QPlainTextEdit. It seems that QPlainTextEdit is very bad at render performance if text is clipped by limiting maximumBlockCount, contrary to its single major use as a way to improve performance and memory usage.

An initial look into the ipython code made it clear that the test code was giving about 8000 lines per stream receive, which i was glad to clip to maximumBlockCount and coalesce multiple text streams into a single appendPlainText call every 100 ms. The qtconsole seemed very responsive, terminating the test print loop without any perceptible delay. All was well, i could go to sleep peacefully now, except for a small glitch which i realized soon enough. Due to a bug, my timer loop wasn't really doing anything. Every stream from ipython kernel was being written to the widget. How then was the widget so responsive, an attentive reader might ask. This post attempts to provide an answer to that very question.

The following are plots of time taken to append plain text to QPlainTextEdit using the code linked here. The x axis is the number of lines appended per call and the different lines are for different maximumBlockCount of the QPlainTextEdit

Appending to an already full buffer

 Clearing and then appending text

 Appending text to empty widget

In all the above cases, the final result is same because the number of lines appended is equal to maximumBlockCount so that all previous content is gone.
As you can see yourself, simply appending text to a full buffer is *very* expensive, so much so that it is almost an order of magnitude larger than clearing the text and then appending for ipython's default case of maxumumBlockCount = 500. All appends are fast until any line overflows the maximumBockCount, when onwards it becomes very expensive to append any more content. I intend to modify the ipython pull request in view of this seemingly bizzare result and attempt to improve the performance further. Hopefully, this would obliterate the need to have a timer based stream output loop and the related complexity. Ideally, someone should fix this right at Qt level, but i do not yet feel confident to do it. Until that happens, this simple workaround should be good enough.


PS: Comments, feedback and further insights welcome

Management lessons from Ultimate CEO: Kaschit Geeta

This post is inspired by a chapter i read from a book titled "DROP", which I got from the Chinmay Mission's Temple near Powai lake. The novel is a nice read, a travelogue of five young people who learn and teach with the experiences they have and share during their journey. This blog post is based on a chapter from the book wherein Kaschit Geeta is replayed in a play in the context of the life of a modern businessman and the principles for managements from the Kaschit Geeta are laid bare for us to benefit.

Background: Kaschit Geeta are the words spoken by Lord Ram to his beloved brother Bharat when he came to bring Ram back to Ayodhya on hearing about his exile to the forest. Note the words, they are not spoken of anger or spite or about general sharing of pleasantries, which is what you would expect most people to do. Instead, Lord Rama enquires Bharat about his handling of the kingdom, and implicit in his enquiries are laid the principles of management. Have a read yourself and see if you can apply the principles in your life too.

Lord Ram's words to Bharata	Crux of the words
"How are you my dear brother? What brings you here? I hope our father, King Dashrath, is in good health? Are you rendering service to our father? Are our mothers Kaushalya, Kaikeyi, and Sumitra in the pink of health. Do you hold in high esteem the Lord, and also your dependents, elders, kinsmen of your father's age, physicians and your teacher?"	Take care of well-being of those who depend on you.
"Getting up regularly everyday do you show yourself well adorned to the people in the assembly hall in the forenoon, O prince?"	Be a role model, appropriately dressed and adhere to all office norms, fresh and ready to start the day and inspire others.
"Has a man of your state, who is learned, clever, ready-witted and capable of delivering messages correctly, nay, who is able to distinguish between right and wrong been appointed by you as an ambassador, O Bharata?"	Choose a right-hand who knows what is right, and can distinguish it from wrong.
"I hope you neither deliberate alone nor take counsel of too many. I hope the decision arrived at by you through deliberation with your counselors does not reach the public before it is carried out."	Choose your advisers carefully, take advise from few trusted people qualified in their domain.
"I hope you distribute daily provisions and disburse the monthly salary due to your employees at the opportune time in a suitable manner and do not defer payment."	Value your employees, make them feel appreciated and recognized.
"Are the forests, which are the home of elephants, preserved by you? Are the milk-cows in abundance with you?"	Be socially responsible.
"Is your income sufficiently large to meet your expenses and your expenditure comparatively less? I hope your wealth does not go to undeserving men."	Balance your books, do not go beyond your means.
"I hope you do not partake by yourself of food nicely cooked. I hope you offer it to friends who seek it."	Share your wealth. Give employees pay hikes to make them feel part of company progress.
"I hope you remain ever mindful of your enemies expelled and since returned, even thought they are weak, O destroyer of foes!"	Know the competition to develop a competitive edge.
"In an unhappy contention between and affluent and a feeble man, icon of Raghu, do your ministers of vast learning, judge the case impartially?"	Have an objective look by not being partial or prejudiced.
"I hope you do not cause interruption in your religious practices by your excessive devotion to wealth."	Do not be too consumed by material desires to forget the goals of life.
"Having obtained his share and ruled in the right way over the entire globe, a wise kshatriya holds sway over the earth, and administering justice to the people -- quite in consonance with righteousness -- surely ascends to heaven when detached from his body."

NOTE: I do not claim copyright over the words of the book, they were spoken by Lord Rama and the produced in the novel DROP. This blog should fall under the "fair use" exemption of the novel.

SPH in movies

Superman returns – but who’s looking after his water?

Is it a plane? No, it’s Smoothed Particle Hydrodynamics.
William West/AFP

Watching films such as Superman Returns or The Day after Tomorrow, you would have seen dramatic sequences of surging water and crumbling buildings.

While doing so, mathematics was probably the last thing you thought about; but without it, scenes of this nature would be virtually impossible.

Take the 2006 film Superman Returns. In one scene, a giant spherical object smashes into a water tank releasing a huge amount of water (see below).

Still image from Superman Returns. Courtesy of Sony Pictures Imageworks

Traditionally, the only possible way to create this kind of sequence would be to use small models – which produce unrealistic results. Or we could create a computer simulation.

Swapping droplets for particles

These days, one of the most popular methods for simulating water is to replace fluid with millions of individual particles within a computer simulation.

And the way these particles move is determined by an algorithm that my colleagues and I invented to simulate the formation of stars in our galaxy’s giant molecular clouds.

The method is known as Smoothed Particle Hydrodynamics (SPH) and the use of SPH in Superman Returns is the work of an American visual effects company called Tweak.

Superman Returns certainly isn’t the only film to feature SPH fluid simulations: think of Gollum falling into the lava of Mount Doom in Lord of the Rings: Return of the King; or the huge alligator splashing through a swamp in Primeval.

These particular scenes are the work of people at a Spanish visual effects company called NextLimit, who received an Oscar for their troubles.

How does SPH work?

Rather than trying to model a body of water as a whole, SPH replaces the fluid with a set of particles. A mathematical technique then uses the position and masses of these particles to determine the density of the fluid being modelled.

Using the density and pressure of the fluid, SPH makes it possible to map the force acting on each particle within the fluid. This technique provides results quite similar to the actual fluid being modelled. And the more particles used in the simulation, the more accurate the model becomes.

This SPH simulation uses 128,000 particles to model a fluid.

Beyond the basics

In Superman Returns, gravity also affects how the body of water behaves (the water spills out of the water tank) and SPH can easily be adapted to accomodate this.

In addition, fluids often need to flow around solid bodies such as rocks and buildings that might be carried, bobbing along, by the flow. The SPH method can be easily extended to handle this combination of solid bodies and fluids by adding sets of particles to the equation, to represent the solid bodies.

These adjustments and extensions to SPH can be made to produce very realistic-looking results.

In industry, SPH is used to describe the motion of offshore rigs in a storm, fluid flow in pumps, and injection moulding of liquid metals. In zoology, it’s being used to investigate the dynamics of fish.

SPH and the stars

As hinted at above, it’s not just water and its inhabitants that can be modelled using this technique.

SPH simulations of star formation by Matthew Bate, from the University of Exeter, and Daniel Price, of Monash, have been able to predict the masses of the stars, and the number of stable two- and three-star systems that form from a typical molecular cloud.

In the case of stable two-star systems (known as binaries) SPH can predict the shape of the orbits in good agreement with astronomical observations.

To get this level of accuracy, millions of particles are used in the SPH calculation, and the motion of these particles is calculated on a number of computer systems that work together in parallel.

SPH is also the method of choice for following the evolution of the universe after the Big Bang. This evolution involves dark matter and gas, and the simulations have one set of SPH particles for the dark matter and one set for the gas.

An advanced SPH code – known as Gadget – used for this purpose was developed by Volker Springel. The code enables astrophysicists to predict the way galaxies form and their distribution in the universe, including the effects of General Relativity.

But for non-astrophysicists, admittedly, the movies may be more of a draw.

So next time you’re watching a film and you see large swathes of water in unusual places or doing incredibly destructive things, think about maths for a moment: without it, such breathtaking scenes would be virtually impossible.

Readibility

Note to myself:
Quoted from http://bethesignal.org/blog/2011/03/18/moving-needle-gnome-leadership/

Contributors are your participants. If there is nothing to contribute, you have no participants. If you have no participants, you barely even have a project, let alone co-development! For FLOSS projects, change has an incentive beyond improving the software.

Readability is the key to creating code that others will use. Because in the end? We can scale silicon, but carbon? People are much harder to scale.

– Brian Aker, Drizzle goes GA, From “What If”, to “What has”

My new mobile

At long last, i've got a new mobile, after years of using the venerable good old fellow Nokia 1100

My old phone: Nokia N8 (circa 2005)

There was not much wrong with the old phone apart from battery problem. However there was one thing which made me buy new phone earlier.
Earlier last month Nokia announced at MWC its deal with MSFT, and the trojan horse that Elop proved himself. I loved Nokia and was waiting for a long time for the N9 with Meego to come to market. However this announcement quashed all my hopes. Hence i bought a Nokia phone while it was still available w/o Windows OS. And the best Nokia phone in market at the time in India was the Nokia N8.
So this is my new phone, a silver Nokia N8.

My new phone: Nokia N8

Many people have asked my why Nokia instead of Android. Well it has really awesome hardware, even if the software Symbian^3 is not as snappy and does not have a million stupid apps in the Ovi store.
Here are some things which impressed me:

• 12 MPx camera with carl-zeiss lenses and xenon flash: I can assure you it can take really awesome pics and i know mega-pixels are not everything in photography. It has a really large sensor for a camera phone. Of course it cannot replace a dedicated DSLR camera (i don't have one), but is handy and good as my primary camera.
• FM Transmitter: Yes you can start your own radio station upto a few meters. Not that i have any idea to use it, but still its a cool technology.
• USB on the go: I can connect usb storage drives (pen drives and external hard disk drives directly to my mobile). USB keyboard and mouse can also be connected to the mobile.A mouse cursor moving in the mobile looks really cool.
• Proper wireless and proxy support: My friend's android cannot connect to ad-hoc wireless networks and also does not support proxy on wlan, so the he needs to be connected to gprs.
• I do not want google to know "ALL" about me.

One con of this mobile is that development on symbian has stopped, and linux support for symbian development is abysmal, and there's no FOSS compiler on linux to compile to symbain, and nokia remote compiler does not support proxy.
If only pyside would work on N8, i'd be really happy. Anyone interested in a GSOC project for this please check here http://developer.qt.nokia.com/wiki/PySide_GSoc_Ideas. Treat assured :)

Stay Healthy By Taking Breaks

Stay Healthy By Taking Breaks: "
Most of us lead sedentary lifestyles these days -- most of our time is spent in front of computers. This slowly is causing a lot of problems people from previous generations haven't experienced: back aches, knee problems, wrist pains, myopia, among others. And just going to a gym or putting in one hour of physical activity a day isn't enough. It doesn't help balance the inactivity over the entire day.

I recently wrote an article in the BenefIT magazine that talks about two tools: Workrave and RSIBreak. Thanks to the publishers, the article is available in pdf format under a CC license.

I've tried both the software but have been using Workrave for quite a while now and am quite happy with it. To briefly introduce them: both software prompt the user to take a break at regular intervals. They have timers that trigger at configured intervals asking the user to take a break. Workrave also has some stretching exercises suggested that can be performed in the longer breaks. The shorter (and more frequent) breaks can be used to take the eyes off the monitor and to relax them. Read the article for more details.

I've reviewed Workrave version 0.9.1 in the article, though the current version as of now is 0.9.3, which has a few differences from those mentioned in the article. The prime difference is the addition of a 'Natural Rest Break' that gets triggered when the screen-saver gets activated, which is nice since if the user walks away from the computer for a prolonged period of time, the rest break in effect has been taken, and the next one is scheduled after the configured duration once the screen-saver is unlocked.

Both software are available in the Fedora repository: Workrave is based on the GTK toolkit (and integrates nicely with the GNOME desktop), whereas RSIBreak is based on the Qt toolkit (and integrates nicely with the KDE desktop). Give these software a try for a cheap but effective way of staying healthy!

"


I've installed Workrave now from the fedora repos. It seems much better than the gnome-typing-break in the keyboard preferences. RSIBreak tries to install a whole lot of KDE dependencies. If anyone of you sits for long hours on a computer, please be nice to yourself and prevent RSI (repetitive strain injury) and other such problems. If you think that working out a few hours in gym will counter it, you are mistaken.

firefox 4b9

Firefox 4 beta 9 is looking good, apart from no hw acceleration on linux.
The minizable menu-bar is a very cool feature. Now all the UI is reduced to a single line on my firefox :)

Here's a screenshot running firefox 4b9 on gnome-shell. See how the menu, back-forward buttons, awesomebar, tabs, tabs-list, panorama all fit into a single line. Now thats some use of the wide laptop screens.

Mathematics, History and worms eating manuscripts…

Mathematics, History and worms eating manuscripts…: "

This is a sad story of forgetten history, indifference towards ancient knowledge and wisdom & callous neglect…Read on.. From A search for India’s mathematical roots, some depressing excerpts (emphasis added):

K. Ramasubramanian is the head of the Indian Institute of Technology, Bombay (IIT-B) research Cell for Indian Science and Technology in Sanskrit (CISTS), the only one of its kind in the country, where doctoral students translate the work of ancient Indian scientists into English, study language technology in Sanskrit that will help computers to analyse a wide range of speech and text, and make the translation and interpretation of Sanskrit texts easy.

…“No country should allow the distortion of its own history,” said Murli Manohar Joshi, former Union minister for human resource development, who had directed all the IIT campuses to set up a CISTS in 2002. Following the directive, IIT-B appointed Kulkarni to spearhead research in Sanskrit language technology in 2003. A year later, the institute brought Ramasubramanian on board.

His students are now at different stages of translating primary Sanskrit texts (dating between the seventh and 15th centuries) of the Kerala School mathematics…All these texts work on the same principles, but work on different timescales. For instance, “Siddhanta texts help predict astronomical positions for a mahayuga (great age), which is about 4,320,000 years. The intermediate Tantra texts work with a yuga, one-tenth of that time—432,000 years. Finally, the Karna texts help quick calculations for as little as one month. My students are working with all three of these texts,” said Ramasubramanian.

…But not every member of the team has scientific training. One of them is a trained astrologer and delighted to read the future. Dinesh Mohan Joshi, (grandson of an astrologer) said: “I saw my grandfather look at kundalis (a graphical representation of planetary positions at birth that charts the life course of the baby) and makes predictions. I saw them come true. I was fascinated. I wanted to be able to do that too. So, I went to (Shri) Lal Bahadur (Shastri) Sanskrit Vidyapeeth, Rashtriya Sanskrit Vidyapeeth and became an acharya (teacher) there. Then a friend told me about this cell and I decided to come.” Unlike Bhatt, his was an uphill struggle to master the mathematics, “because I had no formal training in the subject”.

…And in Joshi’s struggle to learn mathematics, lies the biggest challenge that this venture faces, because “there just aren’t enough people who are skilled in both. If they know Sanskrit, they know little science. And if they are good scientists, they are not interested in Sanskrit or translation of Indian texts”, said Subramanian, explaining why, despite making an enormous effort, IIT has not been able to expand the cell.

Photograph of K. Ramasubramanian, courtesy: IIT Mumbai

Another challenge is of a different nature: Original manuscripts are either rotting or missing. “I had gone to find out some text related to my research at the Kerala University library of manuscripts when I found worms eating four of seven manuscripts. I bought lemongrass oil and gave it to the librarian who said they were too short staffed to look after the documents,” said Ramasubramanian, lamenting that it was the same story across the country. “We simply do not take our historical heritage, intellectual heritage seriously.”

…The professors and students say they have to battle for respect in a country where history, especially the history of science has little value. “Only recently, the cell has started getting more visibility, people have begun asking us to come and talk about our work. Slowly, people are becoming interested…” Kulkarni said.

Reminded me of: Does no one remember the Hindu contribution to Mathematics? and this on the Kerala School

"

whY kernel development is not for the light-hearted

 The linux kernel is a huge behemoth

[pankaj@pankajlaptop pysph-perf]$ su -
Password:
[root@pankajlaptop ~]# yum --enablerepo fedora-debuginfo --enablerepo updates-debuginfo install kernel-debuginfo
Loaded plugins: auto-update-debuginfo, langpacks, presto, refresh-packagekit
Adding en_US to language list
Found 1 installed debuginfo package(s)
Enabling rpmfusion-nonfree-debuginfo: RPM Fusion for Fedora 14 - Nonfree - Debug
Enabling rpmfusion-free-updates-debuginfo: RPM Fusion for Fedora 14 - Free - Updates Debug
Enabling rpmfusion-free-debuginfo: RPM Fusion for Fedora 14 - Free - Debug
Enabling rpmfusion-nonfree-updates-debuginfo: RPM Fusion for Fedora 14 - Nonfree - Updates Debug
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package kernel-debuginfo.x86_64 0:2.6.35.6-48.fc14 set to be installed
--> Processing Dependency: kernel-debuginfo-common-x86_64 = 2.6.35.6-48.fc14 for package: kernel-debuginfo-2.6.35.6-48.fc14.x86_64
--> Running transaction check
---> Package kernel-debuginfo-common-x86_64.x86_64 0:2.6.35.6-48.fc14 set to be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================
 Package                                           Arch                      Version                              Repository                            Size
=============================================================================================================================================================
Installing:
 kernel-debuginfo                                  x86_64                    2.6.35.6-48.fc14                     updates-debuginfo                    239 M
Installing for dependencies:
 kernel-debuginfo-common-x86_64                    x86_64                    2.6.35.6-48.fc14                     updates-debuginfo                     37 M

Transaction Summary
=============================================================================================================================================================
Install       2 Package(s)

Total download size: 276 M
Installed size: 1.6 G
Is this ok [y/N]: y
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 276 M
(1/2): kernel-debuginfo-2.6.35.6-48.fc14.x86_64.rpm                                                                                   | 239 MB     01:11    
(2/2): kernel-debuginfo-common-x86_64-2.6.35.6-48.fc14.x86_64.rpm                                                                     |  37 MB     00:11    
-------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                        3.3 MB/s | 276 MB     01:23    
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing     : kernel-debuginfo-common-x86_64-2.6.35.6-48.fc14.x86_64                                                                                1/2
  Installing     : kernel-debuginfo-2.6.35.6-48.fc14.x86_64                                                                                              2/2

Installed:
  kernel-debuginfo.x86_64 0:2.6.35.6-48.fc14                                                                                                                

Dependency Installed:
  kernel-debuginfo-common-x86_64.x86_64 0:2.6.35.6-48.fc14                                                                                                  

Complete!

[root@pankajlaptop ~]#

python editors in python: spyder and iep

This post is gonna be about python editors (IDEs if you may call, not but quite so)

If you are looking for IDEs, check out pydev and SPE, these are some of the best ones out there with integrated debugging features. There's also a wing editor which many people say is quite good, but i've never used it

Here i'm gonna list opinions about IEP and Spyder. I'm really interested in both of them and run their repository version.
My main issue is i want a good editor for cython, for which i am willing to get my hands dirty (a bit) and add some features myself (see http://powerpan.blogspot.com/2010/10/cython-functions-coverage-revsited.html), so if any of you can help me or have an opinion please do so.

Common features:
Both are python editors written in python (pyqt)
Both provide code completion support and outline
Both have integrated python shells

Differences:
Pros:

Spyder seems a bit more mature project
Spyder has ipython shell which is more useful than a python shell
IEP has outline support for cython

Cons:
Spyder seems to excessively misuse screen real estate - see this issue (editor areas are quite small)
IEP lacks some features such as tree file browser, shell variables, variable occurances marker, pylint support annotation
IEP is officially only for Python3, though you can surely work with python2 files (Also check http://code.google.com/r/pankaj86-iep-python2/source/browse if you really want to run it in python2 only)

Features both lack:
Graphical Debugger: get the code from pydev/winpdb to provide a graphical debugger
Cython support: I could certainly do well with more cython support. Outlining (iep does that), completion support in cython and goto definition (including into cython file from a python file)
Profiling support: Simply run profiling and put the result data into a table (spreadsheet widget)
Documentation: More documentation is needed for both the projects, especially developer documentation to implement new interesting plugins, such as profiling, debugging etc
Also it'd be cool if tooltips could be added to the editor to show documentation and other information on hovering on words in the editor, as in pydev

Few ideas:
Merge features from IEP to Spyder and vice versa. Add new features to both. Having two different projects is good in a way as it keep diversity and help in bringing new ideas. However that shouldn't mean they are independent without any co-operation. Both should surely work together to bring new features.

NOTE: The content may get outdated as you are reading it

linux proxy problem revisited

Some time back i posted how to setup a local forwarding proxy in linux so that u do not need to set your proxy password in each and every program that asks, and that anyone and everyone cannot see your passwords by simply typing "echo $http_proxy"
But again it was not so automatic, you still needed to set the proxy to localhost "http://127.0.0.1:3128/"

Now this post is to make that also redundant. No program connecting to http (port 80) (not https) needs to have the proxy set. This is done by a transparent proxy (intercepting proxy), which 3proxy is by default without requiring any specific configuration (squid needs some option to be set in its config file)

Firstly set up the local forwarding proxy as mentioned in my previous post http://powerpan.blogspot.com/2010/06/linux-proxy-problem.html
Now you need to set an iptables (default linux firewall) policy which will redirect all outgoing traffic to port 80 to the local proxy at port 3128
You also need some way to exclude  the proxy itself from being redirected. So do the following:

Create a specific user for 3proxy:
Create a user on your computer (say 3proxy) with a specific uid (say 480)
# useradd -u 480 3proxy
Now in the 3proxy config file set it to change its user to 3proxy. Just before the "proxy" line in /etc/3proxy.cfg add the following line:
setuid 480
and restart the 3proxy service:
# service 3proxy restart

Redirect outgoing http traffic to local proxy
Here's an iptables rule that forwards outgoing traffic on port 80 (excluding those from user 3proxy) to the local proxy.
# iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner 3proxy --dport 80 -j REDIRECT --to-port 3128

Thats all. Now you are done. To make it persistent across reboots add it to some startup file (/etc/profile.d/).

If you are on Fedora there's a better way:
Create a file (/etc/iptables-transparent-proxy) with the following line:
-A OUTPUT -p tcp -m owner ! --uid-owner 3proxy -m tcp --dport 80 -j REDIRECT --to-ports 3128
Now open system -> config -> firewall and in the Custom Rules (bottommost filter on the left) add a new rule with protocol:ipv4, table:nat and file:/etc/iptables-transparent-proxy and you are done.

To test it, open a terminal, unset http_proxy and wget google.com. The index.html file should be downloaded


NOTE:

• This automatic forwarding does not work for https sites which are specifically designed to prevent such things (man-in-the-middle attacks)

• Outgoing http traffic to any port other than 80 is not redirected

cython functions coverage revsited

A few days back i showed how to get coverage of cython functions using Ned Batchelder's coverage.py
In that post i had posted a patch to coveragepy to enable cython function coverage.

However i realize that many of my friends have never applied a patch before, dont have admin rights on few machines few other problems which may hinder their using this new feature, so here's a good new for you.

I've rewritten the patch into a single file pyx_coverage.py . You can directly use this file instead of 'coverage' command to get cython function coverage, no need to patch anything. You still need to have Ned's coveragepy installed though.

All commands/options/configuration files for coveragepy are applicable here too.

To find coverage of cython files (pyx extension) you need to do following:
1. compile cython code to 'c' with directive profile=True
2. keep source pyx files in same locations as the compiled .so files
    i.e. use 'python setup.py build_ext --inplace' or 'python setup.py develop'
3. run coverage (this file) with the option timid enabled (can also set in in .coveragerc)
    i.e. 'python pyx_coverage.py run --timid my_module.py'

You can use nose test collector as follows:
$ python pyx_coverage.py run /path/to/nosetests /path/to/source

replacing the /paths as appropriate

Download the file from here: https://sites.google.com/site/pankaj86/files/pyx_coverage.py

Reader's Bonus: If you can help me write a python c extension for this treat assured.
Hint: See Ned's coverage.tracer python c extension.

cython functions coverage using coverage.py

For all the coders out there, if you have not been writing unit tests for your code then god bless you, but if you do write tests here's another tool you must use: code coverage.
In python, on of the most popular code coverage tools is Ned Batchelder's coverage.py. It reports statement coverage of all your tests and also can report coverage in beautiful html pages.Its a very nice tool and also integrates well with testing frameworks such as nose to automate your testing and coverage reporting tasks.

But, for all those who use cython, you must surely be aware of the difficulties it brings along while testing, you can never be sure if "all is well". coverage.py doesn't report coverage of cython modules, as those are compiled into native functions.
To mitigate this problem to some extent, i wrote a simple patch to enable coverage.py to report function coverage (not statement coverage) of cython pyx files too, hurray. So now after applying the patch to coverage, all you need to do is:

1. compile cython code with profiling enabled (cython --directive profile=True)
2. run your tests under coverage as you would normally do taking care to add the timid option (coverage --timid test.py)
3. ???
4. profit

So now that you can profit, i'd be greatly thankful if someone comes up and writes this patch into the tracer.c file in the coverage source (it's very simple, get the source using $ hg clone http://bitbucket.org/ned/coveragepy)

Nautilus media tag editor extension by yours truly

Over this weekend, i was searching for something to do (not that i dont have tasks piled up), and i remembered there was no easy way to edit audio files tags. I know there are some excellent programs out there such as easytag and exfalso . However they need you to run a separate program to edit the tags.
Also there is a nautilus extension provided by totem that displays the metadata in the file properties dialog, but cannot edit it. I thought i may be a good idea to make a metadata editor extension for nautilus, so thats what i came up with.


Installation:
The extension requires that exfalso be installed on your computer (since it directly uses the exfalso metadata editor). On fedora, its a simple command:

$ yum install quodlibet

The extension itself is a single python file which you need to put into a directory ~/.nautilus/python-extensions/ (create the directory if it does not exist). The file is available at: https://sites.google.com/site/pankaj86/files/media_tag_editor.py.

Now the mandatory screenshot:

benchmarking pitfalls

In this post i'm going to list some of the pitfalls which can happen when you are trying to optimize your code and test the timings of specific code snippets.

As an example, consider the case of testing the performance of custom array class implemented in pysph at http://code.google.com/p/pysph/source/browse/source/pysph/base/carray.pyx?repo=kunalp-alternate , which is a simple substitute for 1D numpy arrays.

Now in orfer to test the performace i wrote a simple benchmark code:

cpdef dict loopget(ns=Ns):
    cdef double t, t1, t2, num
    cdef int N, i
    cdef dict ret = {}
    empty = numpy.empty
    zeros = numpy.zeros
   
    cdef DoubleArray carr
    cdef numpy.ndarray[ndim=1, dtype=numpy.float64_t] narr
   
    for N in ns:
        carr = DoubleArray(N)
        t = time()
        for i in range(N):
            num = carr[i]
        t = time()-t
        narr = zeros(N)
        t1 = time()
        for i in range(N):
            num = narr[i]
        t1 = time()-t1
        t2 = time()
        for i in range(N):
            num = carr.data[i]
        t2 = time()-t2
        ret['carr loopget %d'%N] = t/N
        ret['carrd loopget %d'%N] = t2/N
        ret['narr loopget %d'%N] = t1/N
    return ret

This snippet simply times the retrieval of a value from a custom DoubleArray class (using its data attribute which is a c array) versus a numpy buffer, both should ideally be at c speed, but the numpy buffer is not unless you disable cython array bounds check.
Now if you run it you would be surprized to see the timings:

carr loopget 100                             9.05990600586e-08
carr loopget 1000                            3.38554382324e-08
carr loopget 10000                           3.42130661011e-08
carr loopget 100000                          3.51309776306e-08
carrd loopget 100                            9.53674316406e-09
carrd loopget 1000                           9.53674316406e-10
carrd loopget 10000                          9.53674316406e-11
carrd loopget 100000                         9.53674316406e-12
narr loopget 100                             9.53674316406e-09
narr loopget 1000                            1.90734863281e-09
narr loopget 10000                           1.09672546387e-09
narr loopget 100000                          1.01089477539e-09

Strangely, getting the value in the c data array is extremely fast, and in fact takes the same time independent of the size of the array.
My first thought was that the access time in C was extremely small as compared to the time it took to call the python's time function. However my readings about gcc and compiler optimizations came to my mind.
Trick: Note that the assignment which is tested in the code snippet does not affect any other part of the code, and the variable num is never even read again. Hence the compiler optimizes it away (this technique is called dead code removal). Thus in the case of C array access the assignments do not occur at all. This does not happen for the other two parts because in calling python functions the compiler can never be sure what is done of the variables, and hence cannot reliably determine whether the assignment has any side-effect or not, so that the assignment is not removed while compilation.

Keeping this fact in mind, let us try to modify the test code so that this specific optimization does not take place. Consider our new test code:

cpdef dict loopget(ns=Ns):
    cdef double t, t1, t2, num
    cdef int N, i
    cdef dict ret = {}
    empty = numpy.empty
    zeros = numpy.zeros
    cdef dict d = {}
   
    cdef DoubleArray carr
    cdef numpy.ndarray[ndim=1, dtype=numpy.float64_t] narr
   
    for N in ns:
        carr = DoubleArray(N)
        t = time()
        for i in range(N):
            num = carr[i]
        t = time()-t
        d[num] = num
        narr = zeros(N)
        t1 = time()
        for i in range(N):
            num = narr[i]
        t1 = time()-t1
        d[num] = num
        t2 = time()
        for i in range(N):
            num = carr.data[i]
        t2 = time()-t2
        d[num] = num
        ret['carr loopget %d'%N] = t/N
        ret['carrd loopget %d'%N] = t2/N
        ret['narr loopget %d'%N] = t1/N
    return ret

The purpose of these added statements is to make sure that the assignment to num is not useless and that the compiler does not optimize it away. Since the new statements occur outside the time() calls it shouldn't affect our tests.
Let us now check the new timings:

carr loopget 100                             1.19209289551e-07
carr loopget 1000                            4.19616699219e-08
carr loopget 10000                           4.52041625977e-08
carr loopget 100000                          4.62603569031e-08
carrd loopget 100                            9.53674316406e-09
carrd loopget 1000                           3.09944152832e-09
carrd loopget 10000                          1.31130218506e-09
carrd loopget 100000                         1.07049942017e-09
narr loopget 100                             2.14576721191e-08
narr loopget 1000                            2.86102294922e-09
narr loopget 10000                           1.69277191162e-09
narr loopget 100000                          2.29835510254e-09

As you can see now the times are more reasonable.
Conclusion: Timing testing code is not a trivial thing to do :)

nearest particle search

Nearest neighbour particle search (NNPS) is  a common requirement of (meshfreee) particle methods, such as SPH. The requirement is to locate all particles within a fixed distance (the kernel support) of a specified particle, and the trick is to avoid doing brute-force distance comparison of every particle with every other particle (O(N^2)). There are many techniques available to implement this. One of the simplest for a fixed kernel support of all particles is to bin the particles and then search for the particles only in the neighbouring bins. Such a technique is implemented here: http://code.google.com/p/pysph/source/browse/source/pysph/base/nnps.pyx?repo=kunalp-alternate.
Here i'm gonna present some timings for the nnps. Note that the timings are old and also include some constant extra times for other operations (calling of rand() numpy function which i've now converted to the c rand() function).

Here are the timings result (Click on image to view the raw data sheet) :

As you can see, it shows that the bin size should be atleast thrice the kernel support size to get good performance.

aero nebula cluster

For those who do not know, i'm currently in my last year of the DD program in Aerospace engineering, and my project is implementation of solid mechanics code using SPH (smoothed particle hydrodynamics) integrated into the pysph project.
It pysph is basically a SPH implementation framework written in python/cython. (Now you know my reason for all those optimization posts :) ).
Now for most CFD codes, you need to run them in parallel on clusters so as to reduce the time required. So i just saw the specs of the nebula cluster (on which i have login) in aero department. Its really wonderful. The specs are:
20 nodes (15 working) each node with 12 six-core AMD opteron 2427 processors with 2.2 GHz xloxk speed and 12 GB RAM, in all 180 6-core processors.
This is sure gonna make parallelizing much more fun and interesting.
PS: I just rad and saw quite a few videos from google about their patented map-reduce technique. It would be interesting to implement SPH in map-reduce and let it run in the "cloud", the buzzword of today.