tag:blogger.com,1999:blog-32229544706823923762024-03-19T10:39:27.272+05:30novice bloggerTo b(log) or not to b(log), that is the questionPankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.comBlogger49125tag:blogger.com,1999:blog-3222954470682392376.post-78542610732705267572016-12-13T12:03:00.000+05:302016-12-13T12:03:13.600+05:30Good Enough Practices for Scientific Computing<div dir="ltr" style="text-align: left;" trbidi="on">
Slides for talk given @ SciPy.in 2016 conference at IIT Bombay<br />
<br />
<a href="https://docs.google.com/presentation/d/1qFaJJmeVrNx_40TfLvyWXcc9tskYll-qmRmfOJ_3yq4">https://docs.google.com/presentation/d/1qFaJJmeVrNx_40TfLvyWXcc9tskYll-qmRmfOJ_3yq4</a><br />
<br />
References:<br />
Collaboratively edited: <a href="https://swcarpentry.github.io/good-enough-practices-in-scientific-computing/">https://swcarpentry.github.io/good-enough-practices-in-scientific-computing/</a><br />
Good enough practices for Scientific Computing: <a href="https://arxiv.org/abs/1609.00037">https://arxiv.org/abs/1609.00037</a><br />
Best practices for Scientific Computing: <a href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745">http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001745</a><br />
<br />
<br /></div>
Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com2tag:blogger.com,1999:blog-3222954470682392376.post-1527375926983279742013-11-09T13:43:00.001+05:302013-11-09T13:50:46.797+05:30Python3 Adventures<div dir="ltr" style="text-align: left;" trbidi="on">
At my <a href="https://www.enthought.com/" target="_blank">$dayjob</a>, i work with python, but mostly python 2.7. As you might be aware, python 3 has been out in the wild since a very long time, and unfortunately many projects have not yet embraced it as much as python core devs would like. Our company has also been one of the many holdouts, partly because of the lack of compelling business reasons to move, not many advantages to moving (which is slowly changing with each new python 3.x minor release), and partly because there is much more higher priority work to be done.<br />
<br />
However, thanks to some members of the community (<a href="https://github.com/burnpanck/" target="_blank">burnpanck</a>, who has python3 compatible (though not 100%) forks of parts of our core <a href="http://code.enthought.com/projects/" target="_blank">ETS</a> stack, starting with traits, pyface and traitsui), this is slowly changing.<br />
Based on his work, i have attempted to make <a href="http://code.enthought.com/projects/chaco/" target="_blank">chaco</a> work on python3, and have largely been successful. The most significant changes were in the extension modules of <a href="http://code.enthought.com/projects/enable/" target="_blank">enable</a>, as expected, since the setup.py uses 2to3 to migrate python code. Enable's kiva renderer which uses AGG underneath, has significant amount of extension code. Luckily, most of it it written in swig which automatically takes care of python2 and python3 compatibility, though in one place i had to modify swig interface file for bytes<->str<->unicode madness. Chaco also has an extension <a href="https://github.com/enthought/chaco/tree/master/chaco/contour" target="_blank">module</a> for contour plots which uses direct Python<->C api which i could make compatible with both 2 and 3 with some #ifdefs.<br />
<br />
In short, migrating from 2 to 3 in not really difficult for most programs. 2to3 makes it even less effort, you just have to figure out the things which 2to3 cannot handle, which is easy if you have tests.<br />
<br />
The proof:<br />
<div class="separator" style="clear: both; text-align: center;">
<object class="BLOGGER-youtube-video" classid="clsid:D27CDB6E-AE6D-11cf-96B8-444553540000" codebase="http://download.macromedia.com/pub/shockwave/cabs/flash/swflash.cab#version=6,0,40,0" data-thumbnail-src="http://i1.ytimg.com/vi/_6EZrAIBaFA/0.jpg" height="532" width="720"><param name="movie" value="http://www.youtube.com/v/_6EZrAIBaFA?version=3&f=user_uploads&c=google-webdrive-0&app=youtube_gdata" /><param name="bgcolor" value="#FFFFFF" /><param name="allowFullScreen" value="true" /><embed width="720" height="532" src="http://www.youtube.com/v/_6EZrAIBaFA?version=3&f=user_uploads&c=google-webdrive-0&app=youtube_gdata" type="application/x-shockwave-flash" allowfullscreen="true"></embed></object></div>
<br />
<br />
Current status: Enable and Chaco seem to work very well under python3. All enable tests pass, though a few chaco tests fail. Also, the chaco toolbar overlay seems to segfault. All other examples in the Chaco demo run very well.<br />
The story is not as so good for <a href="http://code.enthought.com/projects/traits/" target="_blank">traits</a> and <a href="http://code.enthought.com/projects/traitsui/" target="_blank">traitsui</a> though. The traits `implements` and `adapts` advisors do not work in python3. They are already deprecated in traits, so you must use the corresponding replacements from the new `traits.adaptation` package, even on python2. The same work is also required to be done for traitsui, which uses the old mechanism in quite a lot of place.<br />
<br />
Python3 branches of ETS components:<br />
<a href="https://github.com/enthought/traits" target="_blank">Traits</a> : <a href="https://github.com/pankajp/traits/tree/python3">https://github.com/pankajp/traits/tree/python3</a> (derived from burnpanck)<br />
<a href="https://github.com/enthought/pyface" target="_blank">Pyface</a> : <a href="https://github.com/pankajp/pyface/tree/python3">https://github.com/pankajp/pyface/tree/python3</a> (derived from burnpanck)<br />
<a href="https://github.com/enthought/traitsui" target="_blank">TraitsUI</a> : <a href="https://github.com/pankajp/traitsui/tree/python3">https://github.com/pankajp/traitsui/tree/python3</a> (derived from burnpanck)<br />
<a href="https://github.com/enthought/enable" target="_blank">Enable</a> : <a href="https://github.com/pankajp/enable/tree/python3">https://github.com/pankajp/enable/tree/python3</a><br />
<a href="https://github.com/enthought/chaco" target="_blank">Chaco</a> : <a href="https://github.com/pankajp/chaco/tree/python3">https://github.com/pankajp/chaco/tree/python3</a><br />
<br />
PS: If you are as excited about this as I am, please help the cause by using the python3 branch in your project (they also work with python 2.7), and submit issues and pull requests for both python2 and python2 compatibility. Thanks.<br />
<div>
<br /></div>
<div>
PPS: If you are using a newer gcc version (tested with gcc 4.8.2) and experience segfaults in enable/chaco image plots (enable's `integration_tests/kiva/agg/image_test_case.py` segfaults for you), then you need to disable all optimizations from the compiler (export CFLAGS='-O0'; export CXXFLAGS='-O0') while building enable because apparently gcc's optimizer eliminates some necessary code.</div>
<div>
As an alternative, you can use the clang compiler (export CC='clang'; CXX='clang++') which doesn't exhibit this problem at any optimization level. I have yet to figure out whether it is a gcc bug of a bug in enable.</div>
<div>
<br /></div>
</div>
Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-86534665204680140822013-10-20T14:15:00.001+05:302013-11-09T11:26:43.836+05:30cgroups: Maintain sanity of your development system while you screw it<div dir="ltr" style="text-align: left;" trbidi="on">
I am sure many of us developers run into unresponsive machines while doing our usual stuff (developing/debugging) due to a rogue or buggy process consuming all the system's RAM, and swap. It is not usually very common, but happens fairly often for me. For example, if in an ipython shell you start allocating arrays in a loop, and add an extra zero to the size or simply store the arrays without it being garbage collected, it is easy to bog down the system if you are not careful.<br />
<div>
<br /></div>
<div>
For me, the most common instance of this has been using gdb. I usually install all the relevant debuginfo packages available in the Fedora repo to enhance my debugging experience. One downside of this is that gdb consumes a lot of memory. And coupled with python pretty-printing and `<b>thread apply all bt full</b>` command, gdb can go haywire, especially when the stack/heap is corrupted after the segfault. Previously, i would have no recourse but to hard reboot my machine, i once timed that my system to be unresponsive for more than half an hour.</div>
<div>
<br /></div>
<h4 style="text-align: left;">
The Cause:</h4>
<div>
Linux is generally good at time-slicing and not letting rogue processes DOS the system, except for the memory-swap part. The swap system in linux is still not very intelligent in swapping out portions of memory when under pressure. It seems it swaps out data and code of processes equally aggressively.</div>
<div>
Here's what happens when you start a process which consumes a lot of memory (more than the total amount of RAM on your system) to force you to swap:</div>
<div>
First data of inactive processes is swapped out, you can see you machine's swap usage increasing but apparently it doesn't seem to affect the system responsiveness;</div>
<div>
next, executable code of processes is swapped out too, and now is when things go out of control.</div>
<div>
When the latter happens, your machine is screwed, since your terminal process and bash and window manager and gnome-shell and X server have all likely swapped out their executable code to be able to respond to you soon, and there are only two ways to recover: 1) Hope the rogue process quickly dies, which is not very likely on my system with 8GB RAM and swap (it takes a looong time to allocate so much memory while your executable code is itself swapped out); 2) Hard reboot</div>
<div>
<br /></div>
<div>
Not any more, i present to you a mechanism to maintain sanity of the system and limit the amount of RAM you development processes can consume.</div>
<div>
<br /></div>
<h4 style="text-align: left;">
A Workaround:</h4>
<div>
Welcome to cgroups (Control Groups), a feature in linux kernel for managing resources of processes. You can read more about cgroups from the fedora guide at http://docs.fedoraproject.org/en-US/Fedora/17/html/Resource_Management_Guide/index.html .</div>
<div>
<br /></div>
<div>
Here, i will describe a simple way to restrict the cumulative memory (RAM) consumption of all processes started in a bash terminal so that essential system processes have sufficient RAM available to maintain system responsiveness.</div>
<div>
<br /></div>
<div>
The various tools we will use are provided by the <b>libcgroup-tools</b> package, so install it first using yum or the equivalent package on you distro.</div>
<div>
<br /></div>
<div>
Two important services i will use are <b>cgconfig.service</b> and <b>cgred.service</b>, but before enabling and starting them we will configure them. Their config are respectively located in <b>/etc/cgconfig.conf</b> and <b>/etc/cgrules.conf</b>. </div>
<div>
<br /></div>
<div>
Here's what we will configure cgoups for:</div>
<div>
1) Create a <b>bash_memory</b> group for memory subsystem and limit the total RAM consumption to slightly less than the total available RAM, which on my system is 8GB, so i have set the limit to 6GB. This is done via the <b>/etc/cgconfig.conf</b> file. Add the following content to it:</div>
<div>
<br /></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">group bash_memory {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> memory {</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> </span><span style="font-family: 'Courier New', Courier, monospace;">memory.soft_limit_in_bytes="5583457480";</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> memory.limit_in_bytes="6442450944</span><span style="font-family: 'Courier New', Courier, monospace;">";</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;"> }</span></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">}</span></div>
</div>
<div>
<br /></div>
<div>
The first line in the memory subsystem states that in case of memory pressure, when the system is actively thrashing, the memory of all processess in the <b>bash_memory</b> cgroup will be reclaimed (by discarding caches and swapping out dirty pages) to reduce it physical memory usage to 5.2 GB.</div>
<div>
The second line states that the total physical memory consumption of all processes in bash_memory cgroup will never exceed 6GB, they will be swapped out instead.</div>
<div>
<br /></div>
<div>
You can use similar mechanisms (using different subsystem than memory) to limit various resources, like cpu usage, disk IO, network bandwidth etc.</div>
<div>
<br /></div>
<div>
2) Now all we need to do is add processes to the bash_memory cgroup. We will do this via the cgred service by adding a rule to the <b>/etc/cgrules.conf</b> file. Add the following line to the file:</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">*:bash memory bash_memory/</span></div>
<div>
<br /></div>
<div>
The first column says that the rule applies to <b>bash</b> process of all users, the second says that the memory controller is being set, and the third column says that the bash_memory group is applied for the memory controller. Now, the <b>cgred.service</b> will take care of automatically applying the bash_memory group to all bash processes whenever they are started. Due to the inheritance of cgroups, all subprocesses started by bash (and their subprocessess too) will belong to the bash_memory cgroup, thus limiting their cumulative RAM consumption.</div>
<div>
<br /></div>
<div>
You can add more lines to limit specific users or processess.</div>
<div>
<br /></div>
<div>
3) Now all we need to do is start the services and reap their benefit.</div>
<div>
<br /></div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo systemctl enable cgconfig.service</span></div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo systemctl start cgconfig.service</span></div>
</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo systemctl enable cgred.service</span></div>
</div>
<div>
<div>
<span style="font-family: Courier New, Courier, monospace;">sudo systemctl start cgred.service</span></div>
</div>
<div>
<br /></div>
<div>
The linux init daemon systemd uses cgroups to control various services, and you can set specific limit on all services in a systemd based system. Also, the lxc project (lightweight linux containers) uses cgroups and the related namespace functionality to sandbox linux containers. This feature of linux is being used for great benefit by various projects.</div>
<div>
<br /></div>
<div>
Please comment if you find this useful or have some better suggestions :)</div>
<div>
<br />
PS: It seems the gdb memory hog bug which started all this adventure has been fixed: <a href="https://bugzilla.redhat.com/show_bug.cgi?id=1013453">https://bugzilla.redhat.com/show_bug.cgi?id=1013453</a><br />
<br /></div>
</div>
Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-71679791448948883702013-08-25T22:00:00.000+05:302013-08-27T15:27:21.254+05:30Two Years @ Enthought: A Review<div dir="ltr" style="text-align: left;" trbidi="on">
Last week, i completed two full years working for <a href="http://www.enthought.com/" target="_blank">Enthought</a>. It has been a great time working with some of the best minds and wonderful people i have known. Two years is a long time for being at same job i hear. None of my flatmates is at the same place where there were when i joined, and the same with two of my best friends. In fact some friends keep asking me where i am, as if its like changing clothes (hey, what color are you wearing today?). To come clean on this, let me clarify: i am very well loving my work here and do not see any change in the status quo foreseeable future. I'm well into my comfort zone.<br />
<br />
I still vividly remember when we started off, right after finishing college, when Enthought India was still an idea. For few weeks, we used to work at the my graduate lab in IIT itself. I and Prabhu visited a bunch of places around in search of office space, and Eric has asked us to get a good one. We chose <a href="http://www.vatikabusinesscentre.com/supreme-business-park/" target="_blank">VBC</a> after a few other visits, simply because it would let us start immediately in a pre-furnished office, and it was a good office no doubt. We started work there even before the agreement was officially signed, with Bill's visit. Puneeth also joined us physically on 5th Sept. I remembered him as the hard-core Free Software guy, who bought an expensive "Adorable GNU" in an auction by Richard Stallman when he visited IIT Bombay. For a few months, we worked on a contract with Enthought USA, until Enthought India was officially founded in December. Since then we have seen a significant growth in Enthought India, with more good company in the office.<br />
<br />
In the last two years, we have hit several milestones (and probably missed more), and it has been a great journey of learning, doing and sharing. In summary:<br />
<h4 style="text-align: left;">
The Good:</h4>
<ul style="text-align: left;">
<li>Working with some of the best people i have ever known, both intellectually as well as personally.</li>
<li>The lack of "Corporate Culture". I have only heard horror stories of "corporate culture" from friends, TV series and movies. Thought secretly, i wish i could experience it once (for maybe a week or so), just for the feeling and also that i could empathize with my friends.</li>
<li>Meaningful work which is used by many scientists and engineers across the world, that which we can be proud (and ashamed) of, and take responsibility for.</li>
<li>Culture of collaboration and working together, instead of delegating and taking credit.</li>
</ul>
<h4 style="text-align: left;">
The Bad:</h4>
<div>
<ul style="text-align: left;">
<li>Bugs: However you try, bugs are inevitable. They are the nightmare of every developer. The problem is not just the code you wrote, but also the libraries which you use. The former cause embarrassment, and the latter frustration.</li>
<li>Conflicting demands by users: Though you wish to support all users, it is simply not possible. You have to decide among conflicting ideas based on your insights, the work required for implementation, and sometimes who is paying more.</li>
<li>Time Management: There is always more to do in life than the time you have to do it, and prioritization is a hard problem :)</li>
</ul>
<h4 style="text-align: left;">
The Ugly:</h4>
</div>
<div>
<ul style="text-align: left;">
<li>Mumbai City: One of my friend from Delhi says he can identify Mumbai city by its omnipresent stink. Mumbai is hyper expensive for whatever it is that it offers (i can't think of getting a home in Mumbai without robbing a bank, or joining the government (i.e. robbing the people). Add to that the pollution of the city which reduces average life expectancy by a minimum 10 years, and the lack of any open spaces in the city (we have to commute 13 km to the beach to play ultimate, and that too is unplayable during the monsoons by the stink of the garbage and the injury risk). I still can't say i love Mumbai, despite staying here most of my life. What i'd prefer is staying by the woods, on a hill beside a lake or a river.</li>
<li>Once i had a funny dream about Prabhu taking a transfer from IIT Bombay to the newly set up IIT Gandhinagar (which is mentored by IIT Bombay) to set up Aerospace Department there, and we (Enthought India) moving the office to SEZ in <a href="http://giftgujarat.in/home.aspx" target="_blank">GIFT</a>, Gujarat for the tax sops and the generally better administration, infrastructure and quality of life :)</li>
</ul>
<h4 style="text-align: left;">
</h4>
</div>
</div>
Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com2tag:blogger.com,1999:blog-3222954470682392376.post-68091006901181810662013-07-02T01:14:00.001+05:302013-07-02T01:19:41.292+05:30QPlainTextEdit with setMaximumBlockCount performance regression<div dir="ltr" style="text-align: left;" trbidi="on">
Recently while working with ipython <a href="http://ipython.org/ipython-doc/dev/interactive/qtconsole.html" target="_blank">qtconsole</a>, i realized that it is easy to freeze the qtconsole or any app embedding the it by simply doing a "<span style="font-family: Courier New, Courier, monospace;">for i in xrange(10000000):print i</span>" loop, then all arguments about separate kernel process safety etc. are voided. Since this is not something i liked, i set about to fix it in true nature of open-source development (scratching my own back). In this post i'll describe some Qt issues, benchmarks and the approaches i took and how you too can deal with similar problem of overwhelming text input to your text buffer.<br />
Note: The ipython pull request https://github.com/ipython/ipython/pull/3409 is part of this experiment.<br />
<br />
<h4>
The Problem:</h4>
The qtconsole widget is a QPlainTextEdit, with the maximumBlockCount set to 500 by default, which is a very small buffer size by any standard. However, despite this, in case too much text is written to the QPlainTextEdit, it takes too much time drawing the text and the application appears to be frozen This is because the Qt event loop is blocked by too much time consumed by drawing the QPlainTextEdit and incessant stream of new text to draw at a faster rate than QPlainTextEdit can render.<br />
<br />
<h4 style="text-align: left;">
The Solution Approaches:</h4>
My first though at the problem was to use a timer to throttle the stream rendering rate, and append text to the area only every 100ms. That wouldn't cause any perceptible usability loss, but make the program more responsive. Also, another essential idea was to only send the maximumBlockCount number of lines of text ot the QPlainTextEdit. It seems that QPlainTextEdit is <b>very</b> bad at render performance if text is clipped by limiting maximumBlockCount, contrary to its single major use as a way to improve performance and memory usage.<br />
<br />
An initial look into the ipython code made it clear that the test code was giving about 8000 lines per stream receive, which i was glad to clip to maximumBlockCount and coalesce multiple text streams into a single appendPlainText call every 100 ms. The qtconsole seemed very responsive, terminating the test print loop without any perceptible delay. All was well, i could go to sleep peacefully now, except for a small glitch which i realized soon enough. Due to a bug, my timer loop wasn't really doing anything. Every stream from ipython kernel was being written to the widget. How then was the widget so responsive, an attentive reader might ask. This post attempts to provide an answer to that very question.<br />
<br />
The following are plots of time taken to append plain text to QPlainTextEdit using the <a href="https://gist.github.com/pankajp/5903893" target="_blank">code linked here</a>. The x axis is the number of lines appended per call and the different lines are for different maximumBlockCount of the QPlainTextEdit<br />
<div style="text-align: center;">
<b>Appending to an already full buffer</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGxVXr-goJ3e1Ifhbgcou3RlScu3MEq0XB0J0ecEAaYpTjQDv8eLjZ5kAYsfY__Gvi7uJAQQDQaLc5WYYTChKZb8hdpJQBw0l0JtR2gBrTbilK2SRYGdPptNDdrkBB7DlrcVhxvyyIJ3Y/s1600/figure_1_already_full_buffer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjGxVXr-goJ3e1Ifhbgcou3RlScu3MEq0XB0J0ecEAaYpTjQDv8eLjZ5kAYsfY__Gvi7uJAQQDQaLc5WYYTChKZb8hdpJQBw0l0JtR2gBrTbilK2SRYGdPptNDdrkBB7DlrcVhxvyyIJ3Y/s640/figure_1_already_full_buffer.png" width="640" /></a></div>
<div style="text-align: center;">
<b>Clearing and then appending text</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBLvELYV5kUjzGHOFZMY69ljyLBqleoB5Td7aXnlGCj8cJuk2Vj-zf4gb6_W7_A_TkBcWlF2rK28YoNeyJfwacxKxZVR5cQ8QZWhMb-MM5WPJu15tt-qxziVD3DKQeaX1_fOo2hu-GZJs/s1600/figure_1_clear_and_set.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjBLvELYV5kUjzGHOFZMY69ljyLBqleoB5Td7aXnlGCj8cJuk2Vj-zf4gb6_W7_A_TkBcWlF2rK28YoNeyJfwacxKxZVR5cQ8QZWhMb-MM5WPJu15tt-qxziVD3DKQeaX1_fOo2hu-GZJs/s640/figure_1_clear_and_set.png" width="640" /></a></div>
<div style="text-align: center;">
<b>Appending text to empty widget</b></div>
<div class="separator" style="clear: both; text-align: center;">
<a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieapitfcKuWKH8xhQTfd4yD29TG0z5cPfrRq3hP-kbiKzHc9Bjvpj4F0FnQQMyhS_wi_dmLhQtg9lcp4GQbnICiX3N_ZDrjLjvTyuoMECoINpfMMHj55l_-89LuHx6egcxP4vtg4rJcaA/s1600/figure_1_writable_buffer.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="316" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEieapitfcKuWKH8xhQTfd4yD29TG0z5cPfrRq3hP-kbiKzHc9Bjvpj4F0FnQQMyhS_wi_dmLhQtg9lcp4GQbnICiX3N_ZDrjLjvTyuoMECoINpfMMHj55l_-89LuHx6egcxP4vtg4rJcaA/s640/figure_1_writable_buffer.png" width="640" /></a></div>
<br />
In all the above cases, the final result is same because the number of lines appended is equal to maximumBlockCount so that all previous content is gone.<br />
As you can see yourself, simply appending text to a full buffer is *very* expensive, so much so that it is almost an order of magnitude larger than clearing the text and then appending for ipython's default case of maxumumBlockCount = 500. All appends are fast until any line overflows the maximumBockCount, when onwards it becomes very expensive to append any more content. I intend to modify the ipython pull request in view of this seemingly bizzare result and attempt to improve the performance further. Hopefully, this would obliterate the need to have a timer based stream output loop and the related complexity. Ideally, someone should fix this right at Qt level, but i do not yet feel confident to do it. Until that happens, this simple workaround should be good enough.<br />
<br />
<br />
PS: Comments, feedback and further insights welcome<br />
<br /></div>
Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-2016881884033496632012-04-30T00:09:00.000+05:302012-04-30T00:09:40.752+05:30Management lessons from Ultimate CEO: Kaschit Geeta<div dir="ltr" style="text-align: left;" trbidi="on">
This post is inspired by a chapter i read from a book titled "DROP", which I got from the Chinmay Mission's Temple near Powai lake. The novel is a nice read, a travelogue of five young people who learn and teach with the experiences they have and share during their journey. This blog post is based on a chapter from the book wherein Kaschit Geeta is replayed in a play in the context of the life of a modern businessman and the principles for managements from the Kaschit Geeta are laid bare for us to benefit.<br />
<br />
<b>Background:</b> Kaschit Geeta are the words spoken by Lord Ram to his beloved brother Bharat when he came to bring Ram back to Ayodhya on hearing about his exile to the forest. Note the words, they are not spoken of anger or spite or about general sharing of pleasantries, which is what you would expect most people to do. Instead, Lord Rama enquires Bharat about his handling of the kingdom, and implicit in his enquiries are laid the principles of management. Have a read yourself and see if you can apply the principles in your life too.<br />
<b><br /></b><br />
<b><br /></b><br />
<table>
<tbody>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: 1px solid #000000; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0.04in;" width="50%"><b>Lord Ram's words to Bharata</b></td>
<td style="border: 1px solid #000000; padding: 0.04in;" width="50%"><b>Crux of the words</b></td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"How are you my dear brother? What brings you here? I hope our
father, King Dashrath, is in good health? Are you rendering
service to our father? Are our mothers Kaushalya, Kaikeyi, and
Sumitra in the pink of health. Do you hold in high esteem the
Lord, and also your dependents, elders, kinsmen of your father's
age, physicians and your teacher?"</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Take care of well-being of those who depend on you.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"Getting up regularly everyday do you show yourself well adorned
to the people in the assembly hall in the forenoon, O prince?"</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Be a role model, appropriately dressed and adhere to all office
norms, fresh and ready to start the day and inspire others.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"Has a man of your state, who is learned, clever, ready-witted
and capable of delivering messages correctly, nay, who is able to
distinguish between right and wrong been appointed by you as an
ambassador, O Bharata?"</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Choose a right-hand who knows what is right, and can
distinguish it from wrong.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"I hope you neither deliberate alone nor take counsel of too
many. I hope the decision arrived at by you through deliberation
with your counselors does not reach the public before it is
carried out."</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Choose your advisers carefully, take advise from few trusted
people qualified in their domain.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%"><br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%"><br /></td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"I hope you distribute daily provisions and disburse the monthly
salary due to your employees at the opportune time in a suitable
manner and do not defer payment."</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Value your employees, make them feel appreciated and
recognized.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"Are the forests, which are the home of elephants, preserved by
you? Are the milk-cows in abundance with you?"</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Be socially responsible.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%"><br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%"><br /></td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"Is your income sufficiently large to meet your expenses and
your expenditure comparatively less? I hope your wealth does not
go to undeserving men."</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Balance your books, do not go beyond your means.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"I hope you do not partake by yourself of food nicely cooked. I
hope you offer it to friends who seek it."</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Share your wealth. Give employees pay hikes to make them feel
part of company progress.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"I hope you remain ever mindful of your enemies expelled and
since returned, even thought they are weak, O destroyer of foes!"</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Know the competition to develop a competitive edge.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%"><br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%"><br /></td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"In an unhappy contention between and affluent and a feeble man,
icon of Raghu, do your ministers of vast learning, judge the case
impartially?"</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Have an objective look by not being partial or prejudiced.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"I hope you do not cause interruption in your religious
practices by your excessive devotion to wealth."</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%">Do not be too consumed by material desires to forget the goals
of life.</td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%"><br /></td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%"><br /></td>
</tr>
<tr valign="TOP">
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: none; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0in; padding-top: 0in;" width="50%">"Having obtained his share and ruled in the right way over the
entire globe, a wise kshatriya holds sway over the earth, and
administering justice to the people -- quite in consonance with
righteousness -- surely ascends to heaven when detached from his
body."</td>
<td style="border-bottom: 1px solid #000000; border-left: 1px solid #000000; border-right: 1px solid #000000; border-top: none; padding-bottom: 0.04in; padding-left: 0.04in; padding-right: 0.04in; padding-top: 0in;" width="50%"><br />
<br /></td>
</tr>
</tbody></table>
<br />
NOTE: I do not claim copyright over the words of the book, they were spoken by Lord Rama and the produced in the novel DROP. This blog should fall under the "fair use" exemption of the novel.</div>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-58997734784776494522011-06-27T22:32:00.001+05:302011-06-27T22:36:22.871+05:30SPH in movies<h1 class="entry-title five">Superman returns – but who’s looking after his water?</h1><div id="slot1" class="image1"> <img alt="Aapone-20090513000179000144-topshots-australia-film-auction-superman-original" data-id="1146" src="http://cdn.theconversation.edu.au/files/1146/width440/aapone-20090513000179000144-topshots-australia-film-auction-superman-original.jpg"><br />
<div> Is it a plane? No, it’s Smoothed Particle Hydrodynamics.<br />
<span class="source" title="Source">William West/AFP</span><br />
</div> </div> <p>Watching films such as Superman Returns or The Day after Tomorrow, you would have seen dramatic sequences of surging water and crumbling buildings.</p><br />
<p>While doing so, mathematics was probably the last thing you thought about; but without it, scenes of this nature would be virtually impossible.</p><br />
<p>Take the 2006 film Superman Returns. In one scene, a giant spherical object smashes into a water tank releasing a huge amount of water (see below).</p><br />
<p></p><div><img src="http://cdn.theconversation.edu.au/files/1144/width440/Screen_shot_2011-05-18_at_4.02.30_PM.jpg"><div>Still image from Superman Returns. <span style="color:#ccc;">Courtesy of Sony Pictures Imageworks</span></div></div><p></p><br />
<p>Traditionally, the only possible way to create this kind of sequence would be to use small models – which produce unrealistic results. Or we could create a computer simulation.</p><br />
<h2>Swapping droplets for particles</h2><br />
<p>These days, one of the most popular methods for simulating water is to replace fluid with millions of individual particles within a computer simulation.</p><br />
<p>And the way these particles move is determined by an algorithm that my colleagues and I invented to simulate the formation of stars in our galaxy’s giant molecular clouds.</p><br />
<p>The method is known as <a href="http://en.wikipedia.org/wiki/Smoothed-particle_hydrodynamics">Smoothed Particle Hydrodynamics</a> (SPH) and the use of SPH in Superman Returns is the work of an American visual effects company called <a href="http://www.tweaksoftware.com/tweak-films/tweak-films">Tweak</a>.</p><br />
<p>Superman Returns certainly isn’t the only film to feature SPH fluid simulations: think of <a href="http://www.youtube.com/watch?v=UqlaeTphDho#t=0m06s">Gollum falling into the lava of Mount Doom</a> in Lord of the Rings: Return of the King; or the huge alligator splashing through a swamp in Primeval.</p><br />
<p>These particular scenes are the work of people at a Spanish visual effects company called <a href="http://www.nextlimit.com/">NextLimit</a>, who <a href="http://architosh.com/2008/01/next-limit-wins-academy-award/">received an Oscar</a> for their troubles.</p><br />
<h2>How does SPH work?</h2><br />
<p>Rather than trying to model a body of water as a whole, SPH replaces the fluid with a set of particles. A mathematical technique then uses the position and masses of these particles to determine the density of the fluid being modelled.</p><br />
<p>Using the density and pressure of the fluid, SPH makes it possible to map the force acting on each particle within the fluid. This technique provides results quite similar to the actual fluid being modelled. And the more particles used in the simulation, the more accurate the model becomes.</p><br />
<p></p><div><iframe src="http://www.youtube.com/embed/PCS7EgAftRQ?rel=0#t=0m02s" allowfullscreen="" frameborder="0" height="280" width="440"></iframe><div>This SPH simulation uses 128,000 particles to model a fluid.</div></div><p></p><br />
<h2>Beyond the basics</h2><br />
<p>In Superman Returns, gravity also affects how the body of water behaves (the water spills out of the water tank) and SPH can easily be adapted to accomodate this.</p><br />
<p>In addition, fluids often need to flow around solid bodies such as rocks and buildings that might be carried, bobbing along, by the flow. The SPH method can be easily extended to handle this combination of solid bodies and fluids by adding sets of particles to the equation, to represent the solid bodies.</p><br />
<p>These adjustments and extensions to SPH can be made to produce very realistic-looking results.</p><br />
<p>In industry, SPH is used to describe the motion of offshore rigs in a storm, fluid flow in pumps, and injection moulding of liquid metals. In zoology, it’s being used to investigate the dynamics of fish.</p><br />
<h2>SPH and the stars</h2><br />
<p>As hinted at above, it’s not just water and its inhabitants that can be modelled using this technique.</p><br />
<p>SPH simulations of star formation by Matthew Bate, from the University of Exeter, and Daniel Price, of Monash, have been able to <a href="http://users.monash.edu.au/%7Edprice/pubs/magsf/price_stromlo06.pdf">predict the masses of the stars</a>, and the number of stable two- and three-star systems that form from a typical molecular cloud.</p><br />
<p>In the case of stable two-star systems (known as <a href="http://www.universetoday.com/24203/what-is-a-binary-star/">binaries</a>) SPH can predict the shape of the orbits in good agreement with astronomical observations.</p><br />
<p>To get this level of accuracy, millions of particles are used in the SPH calculation, and the motion of these particles is calculated on a number of computer systems that work together in <a href="https://computing.llnl.gov/tutorials/parallel_comp/#Whatis">parallel</a>.</p><br />
<p></p><div><iframe src="http://www.youtube.com/embed/IxKdtjVDh8s?rel=0" allowfullscreen="" frameborder="0" height="360" width="440"></iframe></div><p></p><br />
<p>SPH is also the method of choice for following the evolution of the universe after the Big Bang. This evolution involves <a href="http://theconversation.edu.au/articles/new-chatter-on-dark-matter-1633">dark matter</a> and gas, and the simulations have one set of SPH particles for the dark matter and one set for the gas.</p><br />
<p>An advanced SPH code – known as <a href="http://www.mpa-garching.mpg.de/gadget/">Gadget</a> – used for this purpose was developed by Volker Springel. The code enables astrophysicists to predict the way galaxies form and their distribution in the universe, including the effects of General Relativity.</p><br />
<p>But for non-astrophysicists, admittedly, the movies may be more of a draw.</p><br />
<p>So next time you’re watching a film and you see large swathes of water in unusual places or doing incredibly destructive things, think about maths for a moment: without it, such breathtaking scenes would be virtually impossible.</p>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-50139687701007441792011-03-18T10:15:00.000+05:302011-03-18T10:15:18.155+05:30Readibility<div dir="ltr" style="text-align: left;" trbidi="on"> <br />
Note to myself:<br />
Quoted from <a href="http://bethesignal.org/blog/2011/03/18/moving-needle-gnome-leadership/">http://bethesignal.org/blog/2011/03/18/moving-needle-gnome-leadership/ </a><br />
<br />
<div style="font-family: "Courier New",Courier,monospace;"><br />
</div><div style="font-family: "Courier New",Courier,monospace;">Contributors are your participants. If there is nothing to contribute, you have no participants. If you have no participants, you barely even have a project, let alone co-development! For FLOSS projects, change has an incentive beyond improving the software.</div><blockquote style="font-family: "Courier New",Courier,monospace;">Readability is the key to creating code that others will use. Because in the end? We can scale silicon, but carbon? People are much harder to scale.</blockquote><div style="font-family: "Courier New",Courier,monospace;">– Brian Aker, <a href="http://krow.livejournal.com/700783.html" target="_blank">Drizzle goes GA, From “What If”, to “What has”</a></div></div>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-82320372060343170612011-03-05T15:55:00.000+05:302011-03-05T15:55:33.092+05:30My new mobile<div dir="ltr" style="text-align: left;" trbidi="on">At long last, i've got a new mobile, after years of using the venerable good old fellow Nokia 1100<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJv6acZY2_xCglWs-ysu_FOApsp_AzxaCeD-pKbx8_ARPPkoje-Em22Lfch3gTtTgc7Sd7H8gJyw4TH-yHKc6nRvxkf1ans4J_yB-PTq_Lr1sMC-FaYjQwbOKmFadp1qk9uSfhhVst0IA/s1600/22022011080.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="320" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEiJv6acZY2_xCglWs-ysu_FOApsp_AzxaCeD-pKbx8_ARPPkoje-Em22Lfch3gTtTgc7Sd7H8gJyw4TH-yHKc6nRvxkf1ans4J_yB-PTq_Lr1sMC-FaYjQwbOKmFadp1qk9uSfhhVst0IA/s320/22022011080.jpg" width="178" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My old phone: Nokia N8 (circa 2005)</td></tr>
</tbody></table>There was not much wrong with the old phone apart from battery problem. However there was one thing which made me buy new phone earlier.<br />
Earlier last month Nokia announced at MWC its deal with MSFT, and the trojan horse that Elop proved himself. I loved Nokia and was waiting for a long time for the N9 with Meego to come to market. However this announcement quashed all my hopes. Hence i bought a Nokia phone while it was still available w/o Windows OS. And the best Nokia phone in market at the time in India was the Nokia N8.<br />
So this is my new phone, a silver Nokia N8.<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://www.youtube.com/watch?v=zULDfSS7h20" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="177" src="http://www.nokian8.org/nokia-n8-liggend.jpg" width="320" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">My new phone: Nokia N8</td></tr>
</tbody></table>Many people have asked my why Nokia instead of Android. Well it has really awesome hardware, even if the software Symbian^3 is not as snappy and does not have a million stupid apps in the Ovi store.<br />
Here are some things which impressed me:<br />
<ul style="text-align: left;"><li>12 MPx camera with carl-zeiss lenses and xenon flash: I can assure you it can take really awesome pics and i know mega-pixels are not everything in photography. It has a really large sensor for a camera phone. Of course it cannot replace a dedicated DSLR camera (i don't have one), but is handy and good as my primary camera.</li>
<li>FM Transmitter: Yes you can start your own radio station upto a few meters. Not that i have any idea to use it, but still its a cool technology.</li>
<li>USB on the go: I can connect usb storage drives (pen drives and external hard disk drives directly to my mobile). USB keyboard and mouse can also be connected to the mobile.A mouse cursor moving in the mobile looks really cool.</li>
<li>Proper wireless and proxy support: My friend's android cannot connect to ad-hoc wireless networks and also does not support proxy on wlan, so the he needs to be connected to gprs.</li>
<li>I do not want google to know "ALL" about me. </li>
</ul>One con of this mobile is that development on symbian has stopped, and linux support for symbian development is abysmal, and there's no FOSS compiler on linux to compile to symbain, and nokia remote compiler does not support proxy.<br />
If only pyside would work on N8, i'd be really happy. Anyone interested in a GSOC project for this please check here <a href="http://developer.qt.nokia.com/wiki/PySide_GSoc_Ideas">http://developer.qt.nokia.com/wiki/PySide_GSoc_Ideas</a>. Treat assured :)<br />
<br />
<br />
</div>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com5tag:blogger.com,1999:blog-3222954470682392376.post-82953363535924872642011-01-22T12:41:00.000+05:302011-01-22T12:41:34.842+05:30Stay Healthy By Taking Breaks<a href="http://feedproxy.google.com/%7Er/amit/tech/%7E3/hCGKD-qpunk/stay-healthy-by-taking-breaks.html">Stay Healthy By Taking Breaks</a>: "<img src="http://planet.fedoraproject.org/images/heads/default.png" alt="" style="float: right;" /><br />Most of us lead sedentary lifestyles these days -- most of our time is spent in front of computers. This slowly is causing a lot of problems people from previous generations haven't experienced: back aches, knee problems, wrist pains, myopia, among others. And just going to a gym or putting in one hour of physical activity a day isn't enough. It doesn't help balance the inactivity over the entire day.<br /><br />I recently wrote an <a href="http://amitshah.fedorapeople.org/articles/Healthy-breaks.pdf">article</a> in the <a href="http://benefitmag.com/">BenefIT magazine</a> that talks about two tools: <a href="http://workrave.org/">Workrave</a> and <a href="http://www.rsibreak.org/">RSIBreak</a>. Thanks to the publishers, the article is available in pdf format under a CC license.<br /><br />I've tried both the software but have been using Workrave for quite a while now and am quite happy with it. To briefly introduce them: both software prompt the user to take a break at regular intervals. They have timers that trigger at configured intervals asking the user to take a break. Workrave also has some stretching exercises suggested that can be performed in the longer breaks. The shorter (and more frequent) breaks can be used to take the eyes off the monitor and to relax them. <a href="http://amitshah.fedorapeople.org/articles/Healthy-breaks.pdf">Read the article</a> for more details.<br /><br />I've reviewed Workrave version 0.9.1 in the article, though the current version as of now is 0.9.3, which has a few differences from those mentioned in the article. The prime difference is the addition of a 'Natural Rest Break' that gets triggered when the screen-saver gets activated, which is nice since if the user walks away from the computer for a prolonged period of time, the rest break in effect has been taken, and the next one is scheduled after the configured duration once the screen-saver is unlocked.<br /><br />Both software are available in the Fedora repository: Workrave is based on the GTK toolkit (and integrates nicely with the GNOME desktop), whereas RSIBreak is based on the Qt toolkit (and integrates nicely with the KDE desktop). Give these software a try for a cheap but effective way of staying healthy!<div><img src="https://blogger.googleusercontent.com/tracker/8184990-7748997973838865843?l=log.amitshah.net" alt="" width="1" height="1" /></div><img src="http://feeds.feedburner.com/%7Er/amit/tech/%7E4/hCGKD-qpunk" width="1" height="1" />"<br /><br /><br />I've installed Workrave now from the fedora repos. It seems much better than the gnome-typing-break in the keyboard preferences. RSIBreak tries to install a whole lot of KDE dependencies. If anyone of you sits for long hours on a computer, please be nice to yourself and prevent RSI (repetitive strain injury) and other such problems. If you think that working out a few hours in gym will counter it, you are mistaken.Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-82097581552043084892011-01-18T10:46:00.000+05:302011-01-18T10:46:11.665+05:30firefox 4b9Firefox 4 beta 9 is looking good, apart from no hw acceleration on linux.<br />
The minizable menu-bar is a very cool feature. Now all the UI is reduced to a single line on my firefox :)<br />
<br />
Here's a screenshot running <a href="http://www.mozilla.com/en-US/firefox/beta/">firefox 4b9</a> on <a href="http://live.gnome.org/GnomeShell">gnome-shell</a>. See how the menu, back-forward buttons, awesomebar, tabs, tabs-list, panorama all fit into a single line. Now thats some use of the wide laptop screens.<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaryrTqhACJG5FAph3zjqXACnW0Cl8KIrhwdB5lmZ0eahXlZ7P1aFZqYdTdNveu3SeFvX-9mbGbRUnmJRv4G6XNSoKmtBu8hGiryVttMkLeEnOAgiJEjpJ85avSK-e8paQ21Jj1t6rFK4/s1600/Screenshot-1.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEjaryrTqhACJG5FAph3zjqXACnW0Cl8KIrhwdB5lmZ0eahXlZ7P1aFZqYdTdNveu3SeFvX-9mbGbRUnmJRv4G6XNSoKmtBu8hGiryVttMkLeEnOAgiJEjpJ85avSK-e8paQ21Jj1t6rFK4/s640/Screenshot-1.png" width="640" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1b-7UlhSSLNXamp6e-FCx3FjDOqw3xc0X-C3juzzVwbmP2vn-JnAbFLyilX_uX_y2MCyWNdknzk_aqDM1CSN-KpYQEE2dqugshwRfp9BHmFiYbcxkcMn6hUKs3gm9vfbFL18Q_Mg1s2w/s1600/Screenshot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEg1b-7UlhSSLNXamp6e-FCx3FjDOqw3xc0X-C3juzzVwbmP2vn-JnAbFLyilX_uX_y2MCyWNdknzk_aqDM1CSN-KpYQEE2dqugshwRfp9BHmFiYbcxkcMn6hUKs3gm9vfbFL18Q_Mg1s2w/s640/Screenshot.png" width="640" /></a></div>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-47528787722497292732011-01-18T10:12:00.000+05:302011-01-18T10:12:43.228+05:30Mathematics, History and worms eating manuscripts…<a href="http://feedproxy.google.com/%7Er/SatyamevaJayate/%7E3/wj_BqZcqd8E/">Mathematics, History and worms eating manuscripts…</a>: "<p>This is <strong>a sad story of forgetten history, indifference towards ancient knowledge and wisdom & callous neglect</strong>…Read on.. From <strong><a href="http://www.livemint.com/2011/01/03192858/A-search-for-India8217s-mat.html?atype=tp">A search for India’s mathematical roots</a></strong>, some depressing <strong>excerpts</strong> (emphasis added):</p><br /><blockquote><p>K. Ramasubramanian is the head of the Indian Institute of Technology, Bombay (IIT-B) research Cell for Indian Science and Technology in Sanskrit (CISTS), the only one of its kind in the country, where doctoral students translate the work of ancient Indian scientists into English, study language technology in Sanskrit that will help computers to analyse a wide range of speech and text, and make the translation and interpretation of Sanskrit texts easy.</p><br /><p><strong>…“No country should allow the distortion of its own history,”</strong> said Murli Manohar Joshi, former Union minister for human resource development, who had directed all the IIT campuses to set up a CISTS in 2002. Following the directive, IIT-B appointed Kulkarni to spearhead research in Sanskrit language technology in 2003. A year later, the institute brought Ramasubramanian on board.</p><br /><p>His students are now at different stages of translating primary Sanskrit texts (dating between the seventh and 15th centuries) of the Kerala School mathematics…All these texts work on the same principles, but work on different timescales. For instance, “Siddhanta texts help predict astronomical positions for a mahayuga (great age), which is about 4,320,000 years. The intermediate Tantra texts work with a yuga, one-tenth of that time—432,000 years. Finally, the Karna texts help quick calculations for as little as one month. My students are working with all three of these texts,” said Ramasubramanian.</p><br /><p>…But not every member of the team has scientific training. One of them is a trained astrologer and delighted to read the future. Dinesh Mohan Joshi, (grandson of an astrologer) said: “I saw my grandfather look at kundalis (a graphical representation of planetary positions at birth that charts the life course of the baby) and makes predictions. I saw them come true. I was fascinated. I wanted to be able to do that too. So, I went to (Shri) Lal Bahadur (Shastri) Sanskrit Vidyapeeth, Rashtriya Sanskrit Vidyapeeth and became an acharya (teacher) there. Then a friend told me about this cell and I decided to come.” Unlike Bhatt, his was an uphill struggle to master the mathematics, “because I had no formal training in the subject”.</p><br /><p><strong>…And in Joshi’s struggle to learn mathematics, lies the biggest challenge that this venture faces, because “there just aren’t enough people who are skilled in both. If they know Sanskrit, they know little science. And if they are good scientists, they are not interested in Sanskrit or translation of Indian texts”, said Subramanian, explaining why, despite making an enormous effort, IIT has not been able to expand the cell. </strong></p><br /><p style="text-align:center"><a href="http://satyameva-jayate.org/wp-content/uploads/2011/01/K-Ramasubramanian.jpg"><img title="K Ramasubramanian" src="http://satyameva-jayate.org/wp-content/uploads/2011/01/K-Ramasubramanian-182x300.jpg" alt="" height="300" width="182" /></a></p><br /><p><em>Photograph of <strong>K. Ramasubramanian, </strong></em><em>courtesy: <a href="http://www.hss.iitb.ac.in/faculty.php#ram">IIT Mumbai</a></em></p><br /><p><strong>Another challenge is of a different nature: Original manuscripts are either rotting or missing</strong>. “I had gone to find out some text related to my research at the Kerala University library of manuscripts when I found worms eating four of seven manuscripts. I bought lemongrass oil and gave it to the librarian who said they were too short staffed to look after the documents,” said Ramasubramanian, lamenting that it was the same story across the country. “We simply do not take our historical heritage, intellectual heritage seriously.”</p><br /><p>…<strong>The professors and students say they have to battle for respect in a country where history, especially the history of science has little value</strong>. “Only recently, the cell has started getting more visibility, people have begun asking us to come and talk about our work. Slowly, people are becoming interested…” Kulkarni said.</p><br /></blockquote><br /><p>Reminded me of: <a rel="bookmark" href="http://satyameva-jayate.org/2005/11/25/does-no-one-remember-the-hindu-contribution-to-mathematics/">Does no one remember the Hindu contribution to Mathematics?</a> and t<a href="http://satyameva-jayate.org/2007/08/20/hindu-contribution-to-mathematics-part2/">his on the Kerala School</a></p><br /><div><br /><a href="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?a=wj_BqZcqd8E:I7rmfxwj5sc:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?d=yIl2AUoC8zA" border="0" /></a> <a href="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?a=wj_BqZcqd8E:I7rmfxwj5sc:F7zBnMyn0Lo"><img src="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?i=wj_BqZcqd8E:I7rmfxwj5sc:F7zBnMyn0Lo" border="0" /></a> <a href="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?a=wj_BqZcqd8E:I7rmfxwj5sc:gIN9vFwOqvQ"><img src="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?i=wj_BqZcqd8E:I7rmfxwj5sc:gIN9vFwOqvQ" border="0" /></a> <a href="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?a=wj_BqZcqd8E:I7rmfxwj5sc:qj6IDK7rITs"><img src="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?d=qj6IDK7rITs" border="0" /></a> <a href="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?a=wj_BqZcqd8E:I7rmfxwj5sc:l6gmwiTKsz0"><img src="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?d=l6gmwiTKsz0" border="0" /></a> <a href="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?a=wj_BqZcqd8E:I7rmfxwj5sc:D7DqB2pKExk"><img src="http://feeds.feedburner.com/%7Eff/SatyamevaJayate?i=wj_BqZcqd8E:I7rmfxwj5sc:D7DqB2pKExk" border="0" /></a><br /></div><img src="http://feeds.feedburner.com/%7Er/SatyamevaJayate/%7E4/wj_BqZcqd8E" height="1" width="1" />"Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-68211696928842268882010-11-14T14:41:00.000+05:302010-11-14T14:41:06.891+05:30whY kernel development is not for the light-hearted The linux kernel is a huge behemoth <br />
<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">[pankaj@pankajlaptop pysph-perf]$ su -<br />
Password: <br />
[root@pankajlaptop ~]# yum --enablerepo fedora-debuginfo --enablerepo updates-debuginfo install kernel-debuginfo<br />
Loaded plugins: auto-update-debuginfo, langpacks, presto, refresh-packagekit<br />
Adding en_US to language list<br />
Found 1 installed debuginfo package(s)<br />
Enabling rpmfusion-nonfree-debuginfo: RPM Fusion for Fedora 14 - Nonfree - Debug<br />
Enabling rpmfusion-free-updates-debuginfo: RPM Fusion for Fedora 14 - Free - Updates Debug<br />
Enabling rpmfusion-free-debuginfo: RPM Fusion for Fedora 14 - Free - Debug<br />
Enabling rpmfusion-nonfree-updates-debuginfo: RPM Fusion for Fedora 14 - Nonfree - Updates Debug<br />
Setting up Install Process<br />
Resolving Dependencies<br />
--> Running transaction check<br />
---> Package kernel-debuginfo.x86_64 0:2.6.35.6-48.fc14 set to be installed<br />
--> Processing Dependency: kernel-debuginfo-common-x86_64 = 2.6.35.6-48.fc14 for package: kernel-debuginfo-2.6.35.6-48.fc14.x86_64<br />
--> Running transaction check<br />
---> Package kernel-debuginfo-common-x86_64.x86_64 0:2.6.35.6-48.fc14 set to be installed<br />
--> Finished Dependency Resolution<br />
<br />
Dependencies Resolved<br />
<br />
=============================================================================================================================================================<br />
Package Arch Version Repository Size<br />
=============================================================================================================================================================<br />
Installing:<br />
kernel-debuginfo x86_64 2.6.35.6-48.fc14 updates-debuginfo 239 M<br />
Installing for dependencies:<br />
kernel-debuginfo-common-x86_64 x86_64 2.6.35.6-48.fc14 updates-debuginfo 37 M<br />
<br />
Transaction Summary<br />
=============================================================================================================================================================<br />
Install 2 Package(s)<br />
<br />
<b>Total download size: 276 M<br />
Installed size: 1.6 G</b><br />
Is this ok [y/N]: y<br />
Downloading Packages:<br />
Setting up and reading Presto delta metadata<br />
Processing delta metadata<br />
Package(s) data still to download: 276 M<br />
(1/2): kernel-debuginfo-2.6.35.6-48.fc14.x86_64.rpm | 239 MB 01:11 <br />
(2/2): kernel-debuginfo-common-x86_64-2.6.35.6-48.fc14.x86_64.rpm | 37 MB 00:11 <br />
-------------------------------------------------------------------------------------------------------------------------------------------------------------<br />
Total 3.3 MB/s | 276 MB 01:23 <br />
Running rpm_check_debug<br />
Running Transaction Test<br />
Transaction Test Succeeded<br />
Running Transaction<br />
Installing : kernel-debuginfo-common-x86_64-2.6.35.6-48.fc14.x86_64 1/2 <br />
Installing : kernel-debuginfo-2.6.35.6-48.fc14.x86_64 2/2 <br />
<br />
Installed:<br />
kernel-debuginfo.x86_64 0:2.6.35.6-48.fc14 <br />
<br />
Dependency Installed:<br />
kernel-debuginfo-common-x86_64.x86_64 0:2.6.35.6-48.fc14 <br />
<br />
Complete!</div><div style="font-family: "Courier New",Courier,monospace;">[root@pankajlaptop ~]# </div>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-51808276930425397282010-11-11T18:02:00.002+05:302010-11-12T00:26:51.674+05:30python editors in python: spyder and iepThis post is gonna be about python editors (IDEs if you may call, not but quite so)<br />
<br />
If you are looking for IDEs, check out pydev and SPE, these are some of the best ones out there with integrated debugging features. There's also a wing editor which many people say is quite good, but i've never used it<br />
<br />
Here i'm gonna list opinions about <a href="http://code.google.com/p/iep/">IEP</a> and <a href="http://code.google.com/p/spyderlib/">Spyder</a>. I'm really interested in both of them and run their repository version.<br />
My main issue is i want a good editor for <b>cython</b>, for which i am willing to get my hands dirty (a bit) and add some features myself (see <a href="http://powerpan.blogspot.com/2010/10/cython-functions-coverage-revsited.html">http://powerpan.blogspot.com/2010/10/cython-functions-coverage-revsited.html</a>), so if any of you can help me or have an opinion please do so.<br />
<br />
<b>Common features:</b><br />
Both are python editors written in python (pyqt)<br />
Both provide code completion support and outline<br />
Both have integrated python shells<br />
<br />
<b>Differences:</b><br />
<i>Pros:</i><b><br />
</b><br />
Spyder seems a bit more mature project<br />
Spyder has ipython shell which is more useful than a python shell<br />
IEP has outline support for cython<br />
<br />
<i>Cons:</i><br />
Spyder seems to excessively misuse screen real estate - see this <a href="http://code.google.com/p/spyderlib/issues/detail?id=443">issue</a> (editor areas are quite small)<br />
IEP lacks some features such as tree file browser, shell variables, variable occurances marker, pylint support annotation<br />
IEP is officially only for Python3, though you can surely work with python2 files (Also check <a href="http://code.google.com/r/pankaj86-iep-python2/source/browse">http://code.google.com/r/pankaj86-iep-python2/source/browse</a> if you really want to run it in python2 only)<br />
<br />
<b>Features both lack:</b><br />
<i>Graphical Debugger:</i> get the code from pydev/winpdb to provide a graphical debugger<br />
<i>Cython support:</i> I could certainly do well with more cython support. Outlining (iep does that), completion support in cython and goto definition (including into cython file from a python file)<br />
<i>Profiling support:</i> Simply run profiling and put the result data into a table (spreadsheet widget)<br />
<i>Documentation:</i> More documentation is needed for both the projects, especially developer documentation to implement new interesting plugins, such as profiling, debugging etc<br />
Also it'd be cool if tooltips could be added to the editor to show documentation and other information on hovering on words in the editor, as in pydev<br />
<br />
<b>Few ideas:</b><br />
Merge features from IEP to Spyder and vice versa. Add new features to both. Having two different projects is good in a way as it keep diversity and help in bringing new ideas. However that shouldn't mean they are independent without any co-operation. Both should surely work together to bring new features.<br />
<br />
NOTE: The content may get outdated as you are reading itPankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com1tag:blogger.com,1999:blog-3222954470682392376.post-42902961495582995852010-10-31T10:35:00.003+05:302010-10-31T11:07:26.937+05:30linux proxy problem revisitedSome time <a href="http://powerpan.blogspot.com/2010/06/linux-proxy-problem.html">back</a> i posted how to setup a local forwarding proxy in linux so that u do not need to set your proxy password in each and every program that asks, and that anyone and everyone cannot see your passwords by simply typing "<span style="font-family: "Courier New",Courier,monospace;">echo $http_proxy</span>"<br />
But again it was not so automatic, you still needed to set the proxy to localhost "<span style="font-family: "Courier New",Courier,monospace;">http://127.0.0.1:3128/</span>"<br />
<br />
Now this post is to make that also redundant. No program connecting to http (port 80) (not https) needs to have the proxy set. This is done by a <b>transparent proxy (intercepting proxy)</b>, which 3proxy is by default without requiring any specific configuration (squid needs some option to be set in its config file)<br />
<br />
Firstly set up the local forwarding proxy as mentioned in my previous post <a href="http://powerpan.blogspot.com/2010/06/linux-proxy-problem.html">http://powerpan.blogspot.com/2010/06/linux-proxy-problem.html</a><br />
Now you need to set an iptables (default linux firewall) policy which will redirect all outgoing traffic to port 80 to the local proxy at port 3128<br />
You also need some way to exclude the proxy itself from being redirected. So do the following:<br />
<br />
<b>Create a specific user for 3proxy</b>: <br />
Create a user on your computer (say 3proxy) with a specific uid (say 480)<br />
<span style="font-family: "Courier New",Courier,monospace;"># useradd -u 480 3proxy</span><br />
Now in the 3proxy config file set it to change its user to 3proxy. Just before the "<span style="font-family: "Courier New",Courier,monospace;">proxy</span>" line in /etc/3proxy.cfg add the following line:<br />
<span style="font-family: "Courier New",Courier,monospace;">setuid 480</span><br />
and restart the 3proxy service:<br />
<span style="font-family: "Courier New",Courier,monospace;"># service 3proxy restart</span><br />
<br />
<b>Redirect outgoing http traffic to local proxy</b><br />
Here's an iptables rule that forwards outgoing traffic on port 80 (excluding those from user 3proxy) to the local proxy.<br />
<span style="font-family: "Courier New",Courier,monospace;"># iptables -t nat -A OUTPUT -p tcp -m owner ! --uid-owner 3proxy --dport 80 -j REDIRECT --to-port 3128</span><br />
<br />
Thats all. Now you are done. To make it persistent across reboots add it to some startup file (/etc/profile.d/).<br />
<br />
If you are on <a href="http://fedoraproject.org/">Fedora</a> there's a better way:<br />
Create a file (<span style="font-family: "Courier New",Courier,monospace;">/etc/iptables-transparent-proxy</span>) with the following line:<br />
<span style="font-family: "Courier New",Courier,monospace;">-A OUTPUT -p tcp -m owner ! --uid-owner 3proxy -m tcp --dport 80 -j REDIRECT --to-ports 3128</span><br />
Now open <span style="font-family: "Courier New",Courier,monospace;">system -> config -> firewall</span> and in the <span style="font-family: "Courier New",Courier,monospace;">Custom Rules</span> (bottommost filter on the left) add a new rule with <span style="font-family: "Courier New",Courier,monospace;">protocol:ipv4</span>, <span style="font-family: "Courier New",Courier,monospace;">table:nat</span> and <span style="font-family: "Courier New",Courier,monospace;">file:/etc/iptables-transparent-proxy</span> and you are done.<br />
<br />
To test it, open a terminal, <span style="font-family: "Courier New",Courier,monospace;">unset http_proxy</span> and <span style="font-family: "Courier New",Courier,monospace;">wget google.com</span>. The <span style="font-family: "Courier New",Courier,monospace;">index.html</span> file should be downloaded<br />
<b><br />
</b><br />
<b>NOTE:</b><br />
<ul><li>This automatic forwarding does not work for https sites which are specifically designed to prevent such things (man-in-the-middle attacks)</li>
</ul><ul><li>Outgoing http traffic to any port other than 80 is not redirected</li>
</ul>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-16092111933657362242010-10-30T19:04:00.000+05:302010-10-30T19:04:00.500+05:30cython functions coverage revsitedA <a href="http://powerpan.blogspot.com/2010/10/cython-functions-coverage-using.html">few days back</a> i showed how to get coverage of cython functions using Ned Batchelder's coverage.py<br />
In that post i had posted a patch to coveragepy to enable cython function coverage.<br />
<br />
However i realize that many of my friends have never applied a patch before, dont have admin rights on few machines few other problems which may hinder their using this new feature, so here's a good new for you.<br />
<br />
I've rewritten the patch into a single file <a href="https://sites.google.com/site/pankaj86/files/pyx_coverage.py">pyx_coverage.py</a> . You can directly use this file instead of 'coverage' command to get cython function coverage, no need to patch anything. You still need to have Ned's coveragepy installed though.<br />
<br />
All commands/options/configuration files for coveragepy are applicable here too.<br />
<br />
To find coverage of cython files (pyx extension) you need to do following:<br />
1. compile cython code to 'c' with directive profile=True<br />
2. keep source pyx files in same locations as the compiled .so files<br />
i.e. use 'python setup.py build_ext --inplace' or 'python setup.py develop'<br />
3. run coverage (this file) with the option timid enabled (can also set in in .coveragerc)<br />
i.e. 'python pyx_coverage.py run --timid my_module.py'<br />
<br />
You can use nose test collector as follows:<br />
$ python pyx_coverage.py run /path/to/nosetests /path/to/source<br />
<br />
replacing the /paths as appropriate<br />
<br />
<b>Download the file from here</b>: <a href="https://sites.google.com/site/pankaj86/files/pyx_coverage.py">https://sites.google.com/site/pankaj86/files/pyx_coverage.py</a><br />
<br />
<b>Reader's Bonus</b>: If you can help me write a python c extension for this treat assured.<br />
Hint: See Ned's coverage.tracer python c extension.Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-60243252166164592282010-10-25T15:50:00.000+05:302010-10-25T15:50:42.330+05:30cython functions coverage using coverage.pyFor all the coders out there, if you have not been writing <a href="http://en.wikipedia.org/wiki/Unit_testing">unit tests</a> for your code then god bless you, but if you do write tests here's another tool you must use: <a href="http://en.wikipedia.org/wiki/Code_coverage">code coverage</a>.<br />
In python, on of the most popular code coverage tools is Ned Batchelder's coverage.py. It reports statement coverage of all your tests and also can report coverage in beautiful html pages.Its a very nice tool and also integrates well with testing frameworks such as <a href="http://code.google.com/p/python-nose/">nose</a> to automate your testing and coverage reporting tasks.<br />
<br />
But, for all those who use <a href="http://www.cython.org/">cython</a>, you must surely be aware of the difficulties it brings along while testing, you can never be sure if "all is well". coverage.py doesn't report coverage of cython modules, as those are compiled into native functions.<br />
To mitigate this problem to some extent, i wrote a simple <a href="https://sites.google.com/site/pankaj86/files/pyx_patch.diff">patch </a>to enable coverage.py to report function coverage (not statement coverage) of cython pyx files too, hurray. So now after applying the patch to coverage, all you need to do is:<br />
<ol><li>compile cython code with profiling enabled (cython --directive profile=True)</li>
<li>run your tests under coverage as you would normally do taking care to add the timid option (coverage --timid test.py)</li>
<li>???</li>
<li>profit</li>
</ol>So now that you can profit, i'd be greatly thankful if someone comes up and writes this patch into the <b>tracer.c</b> file in the coverage source (it's very simple, get the source using $ <i>hg clone http://bitbucket.org/ned/coveragepy</i>)Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-50834313611405593852010-07-13T14:38:00.000+05:302010-07-13T14:38:03.792+05:30Nautilus media tag editor extension by yours trulyOver this weekend, i was searching for something to do (not that i dont have tasks piled up), and i remembered there was no easy way to edit audio files tags. I know there are some excellent programs out there such as <a href="http://easytag.sourceforge.net/">easytag</a> and <a href="http://code.google.com/p/quodlibet/">exfalso</a> . However they need you to run a separate program to edit the tags.<br />
Also there is a nautilus extension provided by totem that displays the metadata in the file properties dialog, but cannot edit it. I thought i may be a good idea to make a metadata editor extension for nautilus, so thats what i came up with.<br />
<br />
<br />
<b>Installation:</b><br />
The extension requires that exfalso be installed on your computer (since it directly uses the exfalso metadata editor). On fedora, its a simple command:<br />
<br />
<b><span style="font-family: "Courier New",Courier,monospace;">$ yum install quodlibet</span></b><br />
<br />
The extension itself is a single python file which you need to put into a directory ~/.nautilus/python-extensions/ (create the directory if it does not exist). The file is available at: <a href="https://sites.google.com/site/pankaj86/files/media_tag_editor.py">https://sites.google.com/site/pankaj86/files/media_tag_editor.py</a>.<br />
<br />
Now the mandatory screenshot:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxKr4IyrPpKGuxlQYjfGxDtwDcjrXH5GMzoSCZ3Es9mmYMHATpeBAWPf7cHtIOohClmpSYZxbjZB6tniqVrw7TVp5shXcYXfxPu9TXshbm-Aap8RmajvynREORPUKtQUdRtotUCBvwrHA/s1600/Screenshot.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="400" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhxKr4IyrPpKGuxlQYjfGxDtwDcjrXH5GMzoSCZ3Es9mmYMHATpeBAWPf7cHtIOohClmpSYZxbjZB6tniqVrw7TVp5shXcYXfxPu9TXshbm-Aap8RmajvynREORPUKtQUdRtotUCBvwrHA/s640/Screenshot.png" width="640" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe8Y1r8wDdBf6o0dSnbDp_zXOZco9RQvc3f-ytetmXP7AC1t0WwIquYdd8m_G1geqFz3tRa3xkAgXiMBZQm66TrHQoBNZkuNYc64o1wUT2RjBmZYKUJAV2jCsczLPJhXdw0otUyodHRag/s1600/Screenshot-file.mp3+Properties.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="325" src="https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhe8Y1r8wDdBf6o0dSnbDp_zXOZco9RQvc3f-ytetmXP7AC1t0WwIquYdd8m_G1geqFz3tRa3xkAgXiMBZQm66TrHQoBNZkuNYc64o1wUT2RjBmZYKUJAV2jCsczLPJhXdw0otUyodHRag/s400/Screenshot-file.mp3+Properties.png" width="400" /></a></div>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com5tag:blogger.com,1999:blog-3222954470682392376.post-40172821597455037242010-07-12T12:03:00.001+05:302010-07-12T12:04:25.322+05:30benchmarking pitfallsIn this post i'm going to list some of the pitfalls which can happen when you are trying to optimize your code and test the timings of specific code snippets.<br />
<br />
As an example, consider the case of testing the performance of custom array class implemented in pysph at <a href="http://code.google.com/p/pysph/source/browse/source/pysph/base/carray.pyx?repo=kunalp-alternate">http://code.google.com/p/pysph/source/browse/source/pysph/base/carray.pyx?repo=kunalp-alternate</a> , which is a simple substitute for 1D numpy arrays.<br />
<br />
Now in orfer to test the performace i wrote a simple benchmark code:<br />
<br />
<div style="font-family: "Courier New",Courier,monospace;">cpdef dict loopget(ns=Ns):<br />
cdef double t, t1, t2, num<br />
cdef int N, i<br />
cdef dict ret = {}<br />
empty = numpy.empty<br />
zeros = numpy.zeros<br />
<br />
cdef DoubleArray carr<br />
cdef numpy.ndarray[ndim=1, dtype=numpy.float64_t] narr<br />
<br />
for N in ns:<br />
carr = DoubleArray(N)<br />
t = time()<br />
for i in range(N):<br />
num = carr[i]<br />
t = time()-t<br />
narr = zeros(N)<br />
t1 = time()<br />
for i in range(N):<br />
num = narr[i]<br />
t1 = time()-t1<br />
t2 = time()<br />
for i in range(N):<br />
num = carr.data[i]<br />
t2 = time()-t2<br />
ret['carr loopget %d'%N] = t/N<br />
ret['carrd loopget %d'%N] = t2/N<br />
ret['narr loopget %d'%N] = t1/N<br />
return ret</div><br />
This snippet simply times the retrieval of a value from a custom DoubleArray class (using its data attribute which is a c array) versus a numpy buffer, both should ideally be at c speed, but the numpy buffer is not unless you disable cython array bounds check.<br />
Now if you run it you would be surprized to see the timings:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">carr loopget 100 9.05990600586e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carr loopget 1000 3.38554382324e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carr loopget 10000 3.42130661011e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carr loopget 100000 3.51309776306e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 100 9.53674316406e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 1000 9.53674316406e-10</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 10000 9.53674316406e-11</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 100000 9.53674316406e-12</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 100 9.53674316406e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 1000 1.90734863281e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 10000 1.09672546387e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 100000 1.01089477539e-09</span><br />
<br />
Strangely, getting the value in the c data array is extremely fast, and in fact takes the same time independent of the size of the array.<br />
My first thought was that the access time in C was extremely small as compared to the time it took to call the python's time function. However my readings about gcc and compiler optimizations came to my mind.<br />
Trick: Note that the assignment which is tested in the code snippet does not affect any other part of the code, and the variable num is never even read again. Hence the compiler optimizes it away (this technique is called dead code removal). Thus in the case of C array access the assignments do not occur at all. This does not happen for the other two parts because in calling python functions the compiler can never be sure what is done of the variables, and hence cannot reliably determine whether the assignment has any side-effect or not, so that the assignment is not removed while compilation.<br />
<br />
Keeping this fact in mind, let us try to modify the test code so that this specific optimization does not take place. Consider our new test code:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">cpdef dict loopget(ns=Ns):</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> cdef double t, t1, t2, num</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> cdef int N, i</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> cdef dict ret = {}</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> empty = numpy.empty</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> zeros = numpy.zeros</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <span style="background-color: yellow;">cdef dict d = {}</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"> </span><br />
<span style="font-family: "Courier New",Courier,monospace;"> cdef DoubleArray carr</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> cdef numpy.ndarray[ndim=1, dtype=numpy.float64_t] narr</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> </span><br />
<span style="font-family: "Courier New",Courier,monospace;"> for N in ns:</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> carr = DoubleArray(N)</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> t = time()</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> for i in range(N):</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> num = carr[i]</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> t = time()-t</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <span style="background-color: yellow;">d[num] = num</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"> narr = zeros(N)</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> t1 = time()</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> for i in range(N):</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> num = narr[i]</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> t1 = time()-t1</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <span style="background-color: yellow;">d[num] = num</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"> t2 = time()</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> for i in range(N):</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> num = carr.data[i]</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> t2 = time()-t2</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> <span style="background-color: yellow;">d[num] = num</span></span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ret['carr loopget %d'%N] = t/N</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ret['carrd loopget %d'%N] = t2/N</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> ret['narr loopget %d'%N] = t1/N</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> return ret</span><br />
<br />
The purpose of these added statements is to make sure that the assignment to num is not useless and that the compiler does not optimize it away. Since the new statements occur outside the time() calls it shouldn't affect our tests.<br />
Let us now check the new timings:<br />
<div style="font-family: "Courier New",Courier,monospace;"><br />
</div><span style="font-family: "Courier New",Courier,monospace;">carr loopget 100 1.19209289551e-07</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carr loopget 1000 4.19616699219e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carr loopget 10000 4.52041625977e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carr loopget 100000 4.62603569031e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 100 9.53674316406e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 1000 3.09944152832e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 10000 1.31130218506e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">carrd loopget 100000 1.07049942017e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 100 2.14576721191e-08</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 1000 2.86102294922e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 10000 1.69277191162e-09</span><br />
<span style="font-family: "Courier New",Courier,monospace;">narr loopget 100000 2.29835510254e-09</span><br />
<br />
As you can see now the times are more reasonable.<br />
Conclusion: Timing testing code is not a trivial thing to do :)Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-28133154869798462192010-07-09T22:04:00.001+05:302010-07-09T22:12:40.402+05:30nearest particle searchNearest neighbour particle search (NNPS) is a common requirement of (meshfreee) particle methods, such as SPH. The requirement is to locate all particles within a fixed distance (the kernel support) of a specified particle, and the trick is to avoid doing brute-force distance comparison of every particle with every other particle (O(N^2)). There are many techniques available to implement this. One of the simplest for a fixed kernel support of all particles is to bin the particles and then search for the particles only in the neighbouring bins. Such a technique is implemented here: <a href="http://code.google.com/p/pysph/source/browse/source/pysph/base/nnps.pyx?repo=kunalp-alternate">http://code.google.com/p/pysph/source/browse/source/pysph/base/nnps.pyx?repo=kunalp-alternate</a>.<br />
Here i'm gonna present some timings for the nnps. Note that the timings are old and also include some constant extra times for other operations (calling of rand() numpy function which i've now converted to the c rand() function).<br />
<br />
Here are the timings result (Click on image to view the raw data sheet) :<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://spreadsheets.google.com/pub?key=0AomYDYyBBNkkdGNTME81TWgxN1p2alZndGFXd0NfZlE&authkey=CNWZrsME&hl=en&single=true&gid=0&output=html" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://spreadsheets.google.com/oimg?key=0AomYDYyBBNkkdGNTME81TWgxN1p2alZndGFXd0NfZlE&oid=1&zx=amk8wx-yj1kav" /></a></div><br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://spreadsheets.google.com/pub?key=0AomYDYyBBNkkdGNTME81TWgxN1p2alZndGFXd0NfZlE&authkey=CNWZrsME&hl=en&single=true&gid=0&output=html" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="https://spreadsheets.google.com/oimg?key=0AomYDYyBBNkkdGNTME81TWgxN1p2alZndGFXd0NfZlE&oid=2&zx=ut1jyw-r72nw4" /></a></div><br />
As you can see, it shows that the bin size should be atleast thrice the kernel support size to get good performance.<br />
<br />
<br />
<a href="https://spreadsheets.google.com/pub?key=0AomYDYyBBNkkdGNTME81TWgxN1p2alZndGFXd0NfZlE&authkey=CNWZrsME&hl=en&single=true&gid=0&output=html"></a>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com0tag:blogger.com,1999:blog-3222954470682392376.post-67328136687739768262010-07-07T21:39:00.002+05:302010-07-07T21:40:54.303+05:30aero nebula clusterFor those who do not know, i'm currently in my last year of the DD program in Aerospace engineering, and my project is implementation of solid mechanics code using SPH (smoothed particle hydrodynamics) integrated into the <a href="http://code.google.com/p/pysph/">pysph</a> project.<br />
It pysph is basically a SPH implementation framework written in python/cython. (Now you know my reason for all those optimization posts :) ).<br />
Now for most CFD codes, you need to run them in parallel on clusters so as to reduce the time required. So i just saw the specs of the nebula cluster (on which i have login) in aero department. Its really wonderful. The specs are:<br />
20 nodes (15 working) each node with 12 six-core AMD opteron 2427 processors with 2.2 GHz xloxk speed and 12 GB RAM, in all 180 6-core processors.<br />
This is sure gonna make parallelizing much more fun and interesting.<br />
PS: I just rad and saw quite a few videos from google about their patented map-reduce technique. It would be interesting to implement SPH in map-reduce and let it run in the "cloud", the buzzword of today.Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com2tag:blogger.com,1999:blog-3222954470682392376.post-73015472944313705032010-06-17T21:55:00.005+05:302010-10-31T21:01:44.959+05:30the linux proxy problemFor those of you who use linux for anything more than web browsing (in university/office) must be aware of the problems a proxy can pose. In many places as in my institute, you need to necessarily use a specified proxy server to access outside world, needing authentication for your credentials.<br />
In my college, a common login registered in a central ldap server provides for all authentication services (used for course registration/fees payments/emails/proxy/...). Hence it is very important to protect it. Here i will show one way to avoid anyone easily getting your password.<br />
<br />
<br />
<b>Network proxy loophole in GNOME:</b><br />
If you are using GNOME (default Fedora/Ubuntu) and you set your proxy details in "system->preferences->network proxy" then you open a simple loophole in the settings.<br />
After setting your username/password, open a new terminal and type<br />
<span style="font-family: "Courier New",Courier,monospace;"> echo $http_proxy</span><br />
Now you can clearly see your password as<br />
http://<user>:<pass>@proxy.com:3128/<br />
Now since many people come to your rooms in colleges you can see how simple it is to get your credentials.<br />
<br />
<b>Is there a way out:</b><br />
There may be other ways, but here's the one which i follow. I create a local forwarding proxy server on my own computer and direct all applications to use that proxy. The settings for my proxy server are written in a file only readable by the root.<br />
What follows is a step-by-step guide to set it up. Tested on Fedora<br />
<br />
<b>What do i use:</b><br />
I use a small proxy server <a href="http://www.3proxy.ru/">3proxy</a>, you could also use any other proxy server such as <a href="http://www.squid-cache.org/">squid</a>. In fact i used to use squid before i came to know of 3proxy (when it was packaged in fedora). Squid is a much more feature rich and heavy proxy. When i was using it had a bug whereby it would do at least 100 cpu wakeups per second, using precious power on my laptop. This may have been fixed by now.<br />
<br />
<b>Installation:</b><br />
On Fedora systems you can do<br />
<span style="font-family: "Courier New",Courier,monospace;"> yum install 3proxy</span><br />
A similar command for apt-get may work on Ubuntu (i've never tried)<br />
<br />
<b>Configuration:</b><br />
The configuration you need to do is<br />
<ol><li>Open the file /etc/3proxy.cfg in editor of your choice as root</li>
<li>Locate the line containing 'proxy -n'</li>
<li>Above this line, upto the line 'dnspr', comment out all uncommented lines and instead add the following lines:<br />
<span style="font-family: "Courier New",Courier,monospace;"><br />
auth iponly</span><br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">allow * * 127.0.0.0/24,<local_IPs> * * * *</span> <br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">allow * * * * * * *</span><br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">parent 1000 http <proxy.server.com> <port> <proxy_user> <proxy_pass</span>><br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">proxy -n</span><br />
<br />
The values in angle brackets need to be replaced by you configuration The values for my college are given in parenthesis<br />
<local IPs> = ips not connected through proxy [10.0.0.0/8]<br />
<proxy.server.name> = proxy server [netmon.iitb.ac.in]<br />
<port> = proxy port [80]<br />
<proxy_user> = proxy authentication username<br />
<proxy_pass> = proxy authentication password</li>
<li>Comment out all lines with the content:<br />
<br style="font-family: "Courier New",Courier,monospace;" /><span style="font-family: "Courier New",Courier,monospace;">socks</span><br style="font-family: "Courier New",Courier,monospace;" /> <span style="font-family: "Courier New",Courier,monospace;">pop3p</span><br style="font-family: "Courier New",Courier,monospace;" /> <span style="font-family: "Courier New",Courier,monospace;">ftppr</span><br style="font-family: "Courier New",Courier,monospace;" /> <span style="font-family: "Courier New",Courier,monospace;">admin</span><br style="font-family: "Courier New",Courier,monospace;" /> <span style="font-family: "Courier New",Courier,monospace;">dnspr</span><br style="font-family: "Courier New",Courier,monospace;" /> <span style="font-family: "Courier New",Courier,monospace;">tcppm</span><br style="font-family: "Courier New",Courier,monospace;" /> <span style="font-family: "Courier New",Courier,monospace;">udppm</span></li>
<li>Save the file</li>
<li>as root run (this will make the file only readable by root user)<br />
<span style="font-family: "Courier New",Courier,monospace;"> chmod o-rwx /etc/3proxy.cfg</span><br />
<span style="font-family: "Courier New",Courier,monospace;"> chkconfig 3proxy on</span></li>
<li>??</li>
<li>profit</li>
</ol>The details of the 3proxy.cfg file are documented at <a href="http://www.3proxy.ru/doc/html/man3/3proxy.cfg.3.html">http://www.3proxy.ru/doc/html/man3/3proxy.cfg.3.html</a><br />
<br />
Now in whichever application you need to set the proxy server, set it as<br />
<br />
<b style="font-family: "Courier New",Courier,monospace;">http://127.0.0.1:3128/</b><br />
<br />
without any authentication.<br />
Thats it, now only root knows your ldap password, and no one else can snoop<br />
<br />
<b>EDIT:</b><br />
If you automatically want to set the proxy environment variable of the whole system, then you can create a file <i>/etc/profile.d/proxy.sh</i> with the following content<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">export http_proxy=http://127.0.0.1:3128/</span><br />
<span style="font-family: "Courier New",Courier,monospace;">export https_proxy=$http_proxy</span><br />
<span style="font-family: "Courier New",Courier,monospace;">export ftp_proxy=$http_proxy</span><br />
<br />
Many (not all) programs on linux use these environment variables to get proxy settings.<br />
<br />
<b>EDIT2</b> :<br />
To set multiple proxies (different hosts go through different proxies) you can do something like below (see 3proxy.cfg manual for much more detail and many other options):<br />
<dl class="avatar-comment-indent" id="comments-block"><dd class="comment-body" id="Blog1_cmt-670331920463891991"><span style="font-family: "Courier New",Courier,monospace;"># direct connection </span> <span style="font-family: "Courier New",Courier,monospace;">allow * 127.0.0.1 127.0.0.0/24,<local_IPs> * *</span>
<span style="font-family: "Courier New",Courier,monospace;"># through proxy1 </span> <span style="font-family: "Courier New",Courier,monospace;">allow * * <hosts_thru_proxy1> * *</span>
<span style="font-family: "Courier New",Courier,monospace;">parent 1000 http <proxy1.server.com> <port> <proxy_user> </span> <span style="font-family: "Courier New",Courier,monospace;"># through proxy2 </span> <span style="font-family: "Courier New",Courier,monospace;">allow * * <hosts_thru_proxy2> * *</span>
<span style="font-family: "Courier New",Courier,monospace;">parent 1000 http <proxy2.server.com> <port> <proxy_user> </span> <span style="font-family: "Courier New",Courier,monospace;"># through proxy3 </span> <span style="font-family: "Courier New",Courier,monospace;">allow * * <hosts_thru_proxy3> * *</span>
<span style="font-family: "Courier New",Courier,monospace;">parent 1000 http <proxy3.server.com> <port> <proxy_user></span>
<span style="font-family: "Courier New",Courier,monospace;">allow * * * * *</span>
<span style="font-family: "Courier New",Courier,monospace;">proxy -n</span> </dd><dd class="comment-footer"><span class="comment-timestamp"> </span></dd></dl>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com25tag:blogger.com,1999:blog-3222954470682392376.post-27040811109808588172010-05-30T15:01:00.001+05:302010-05-30T15:02:56.647+05:30research made simple with zoteroIf you are into study/research of any kind (academic/non-academic) which involves reading up things and keeping track of them then you are in for a great productivity boost. This will help if you are reading books/news/articles/wikipedia/journals or any such sort of thing. The too I'm talking about is <a href="http://www.zotero.org/">zotero</a><br />
With zotero you can save proper bibliographic references of lots of material you see on the internet and manage/search/cite them in various forms. Its really difficult to describe all the wonderful things zotero can do for your research, so it'll be really good for you if you watch the screencast <a href="http://www.zotero.org/support/screencast_tutorials/zotero_tour">http://www.zotero.org/support/screencast_tutorials/zotero_tour</a><br />
<br />
Some features you'll find helpful:<br />
Collect: <br />
<ul><li>Single click saving of references. For example single click on any sciencedirect article, if you have subscribed (as in my college), a single click will save all information about the article, including the pdf (with well thought name instead of fulltext.pdf) if its available.</li>
<li>To enable saving pdf select in Zotero Preferences->General tab -> automatically attach associated pdfs and other files when saving. </li>
<li>In search tab in preferences, you may also want to enable indexing of pdfs if you need.</li>
<li>Clicking on sites with references to lots of articles (wikipedia references, cited by in Scopus etc), you can easily select all the references you need to save</li>
</ul>Manage:<br />
<ul><li>You can search all your saved articles, add notes, tags etc</li>
<li>You can group all articles in collections based on topic</li>
<li>You can create saved searches based on various criteria</li>
</ul>Cite:<br />
<ul><li>To cite an article(s) simply select them and right click to 'create bibliography from selectd articles' and choose a format style from the many available (including all popular journals) and you are done</li>
<li>If you are using bibtex to manage bibliographies for your article then select the articles and right click to do 'export selected items' and select bibtex format</li>
<li>Zotero plugins are available for Openoffice and MS Office too, so you can easily insert the references in your articles, without the pain of collecting anf formatting</li>
</ul>Share:<br />
<ul><li>If you work in team then this is a really wonderful feature. Create a simple login on the zotero server (you can also use openid)</li>
<li>In zotero preferences->sync tab enter your zotero login details and enable sync my library and group library.</li>
<li>All synced items (including attached pdfs) are available on the internet anywhere without even installing zotero addon. You just need to login to zotero and see your collection. This is very useful if your college has access to some journals but when you are somewhere else in a conference and you need to check and article. 100 MB space is freely available and you can buy even more. </li>
<li>'My library' is your personal collection. Group libraries are shared collections, which can be shared with other people you are working with.</li>
</ul>So what are you waiting for, install it now. If you did not install it yet, then you need to watch the screencast <a href="http://www.zotero.org/support/screencast_tutorials/zotero_tour">http://www.zotero.org/support/screencast_tutorials/zotero_tour</a> now<br />
<ul></ul>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com1tag:blogger.com,1999:blog-3222954470682392376.post-19965323416692441572010-05-29T15:54:00.003+05:302010-06-16T19:02:33.379+05:30numpy array performance / divide and conquer considered harmfulThis is again a post about python code speed, the data and inference are more than a few months old but still valid.<br />
Here's a spreadsheet showing speed of array math operations (+, -, *, /) between numpy arrays and python lists.<br />
Check this spreadsheet to see the timings of various operations<br />
<a href="https://spreadsheets.google.com/ccc?key=0AomYDYyBBNkkdHAtMkdHMF9TZ29lMmZQV3UwYkxWNFE&hl=en">https://spreadsheets.google.com/ccc?key=0AomYDYyBBNkkdHAtMkdHMF9TZ29lMmZQV3UwYkxWNFE&hl=en</a><br />
<br />
The operations I considered for comparison were:<br />
<ul><li>x+0.1</li>
<li>x-0.1</li>
<li>x*0.1</li>
<li>x/0.1</li>
<li>x*(1/0.1)</li>
<li>x+y</li>
<li>x-y</li>
<li>x*y</li>
<li>x/y</li>
<li>[p+yp[j] for j,p in enumerate(xp)]</li>
<li>[xp[j]+yp[j] for j in xrange(i)]</li>
</ul>where <b>x</b> and <b>y</b> are numpy arrays, <b>xp</b> and <b>yp</b> are python lists, all of size <b>N</b> which is varied for the comparison.<br />
The raw timings data is available here: <br />
<ul></ul><a href="https://spreadsheets.google.com/pub?key=0AomYDYyBBNkkdHAtMkdHMF9TZ29lMmZQV3UwYkxWNFE&hl=en&output=html">https://spreadsheets.google.com/pub?key=0AomYDYyBBNkkdHAtMkdHMF9TZ29lMmZQV3UwYkxWNFE&hl=en&output=html</a><br />
<br />
See the timings plot yourself<br />
<div class="separator" style="clear: both; text-align: center;"><a href="https://doc-0k-b8-docs.googleusercontent.com/docs/secure/ha0ro937gcuc7l7deffksulhg5h7mbp1/tn8kmqubsk7ihmesna4339vi28lu2e4i/1275112800000/09163134879685239802/*/0B4mYDYyBBNkkZDUyMzUxMjEtZWUzMC00OTBjLTkyODctOWFiYzZiOWFiMzlh" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="359" src="https://doc-0k-b8-docs.googleusercontent.com/docs/secure/ha0ro937gcuc7l7deffksulhg5h7mbp1/tn8kmqubsk7ihmesna4339vi28lu2e4i/1275112800000/09163134879685239802/*/0B4mYDYyBBNkkZDUyMzUxMjEtZWUzMC00OTBjLTkyODctOWFiYzZiOWFiMzlh" width="640" /></a></div><b>Conclusion:</b><br />
<ul><li>Use numpy arrays for size > 10</li>
<li>Avoid division as much as you can to improve the speed of your numerical codes</li>
<li>Instead of <b>x/0.1</b> do <b>x*(1/0.1)</b> . This itself causes large speedup as N is increased. </li>
<li><b>x/0.1</b> and <b>x/y</b> take almost the same time</li>
<li><b>+</b>, <b>-</b>, <b>*</b> take almost same time, <b>/</b> takes much more time, and its expense increases as <b>N</b> is increased.</li>
<li>Once again, <b>do not divide</b>.</li>
<li>The same thing is valid in cython code also. Avoid division even in cython code, and even if you are using <b>double</b> instead of numpy arrays (buffer). Rewrite expressions to minimize the usage of division operator.</li>
</ul>Pankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com1tag:blogger.com,1999:blog-3222954470682392376.post-24998648539160765102010-05-26T22:58:00.004+05:302010-05-26T23:24:47.147+05:30cython timings test<h1>The TASK : To optimize cython functions</h1><h1>Detailed: functions which depend on a once initialized attribute value</h1>This often comes handy in many cases, for example to write a Laplacian function of a scalar field in spherical/axisymmetric coordinate system, you would need three independent cases for 1,2,3 dimensions for performance purposes and if u do not write all functions as general 3D functions.<br />
<br />
<br />
<h2>The test CODE : test_kernel.pyx</h2><br />
<table border="1" bordercolor="#000000" cellpadding="3" cellspacing="0" id="at02" style="font-family: Courier New;"><tbody>
<tr><td width="100%">cdef class Kernel:<br />
cdef int dim<br />
cdef double (*func)(Kernel,double)<br />
def __init__(self, dim=1):<br />
self.dim = dim<br />
if dim == 1:<br />
self.func = self.func1<br />
elif dim == 2:<br />
self.func = self.func2<br />
<br />
cdef double func1(self, double x):<br />
return 1+x<br />
<br />
cdef double func2(self, double x):<br />
return 2+x<br />
<br />
cdef double c_func(self, double x):<br />
'''this is only to make function signature compatible with func1 and func2'''<br />
return self.func(self, x)<br />
<br />
def p_func(self, double x):<br />
return self.func(self, x)<br />
<br />
cpdef double py_func(self, double x):<br />
return self.func(self, x)<br />
<br />
cpdef double py_c_func(self, double x):<br />
return self.c_func(x)<br />
<br />
def py_func1(self, x):<br />
return self.func1(x)<br />
<br />
def py_func2(self, x):<br />
return self.func2(x)<br />
<br />
cdef double func_common(self, double x):<br />
cdef int dim = self.dim<br />
if dim == 1:<br />
return 10+x<br />
elif dim == 2:<br />
return 20+x<br />
<br />
def py_func_c_common(self, x):<br />
return self.func_common(x)<br />
<br />
cpdef double py_func_common(self, double x):<br />
cdef int dim = self.dim<br />
if dim == 1:<br />
return 10+x<br />
elif dim == 2:<br />
return 20+x</td></tr>
</tbody></table><br />
Compilation command:<br />
cython -a test_kernel.pyx;<br />
gcc <optimization-flag> -shared -fPIC test_kernel.c -lpython2.6 -I /usr/include/python2.6/ -o test_kernel.so<br />
where optimization flag is either empty or "-O2" or "-O3"<br />
<br />
Cython optimization<br />
<b>Tip 1:</b><br />
Type (cdef) as many variables as you can. You also need to type the locals in each function. Try to try to use C data types wherever possible.<br />
<b>Tip 2:</b><br />
use:<br />
cython -a file.pyx<br />
command to generate a html file which shows lines which cause expensive python functions to be called. Clicking on a line shows the corresponding C code generated, highlighting expensive calls in shades of red. Try to eliminate as many such calls as you can.<br />
<br />
<h2><b>The TEST</b> :</h2>time_kernel.py<br />
<br />
<table border="1" bordercolor="#000000" cellpadding="3" cellspacing="0" id="ex4m" style="font-family: Courier New;"><tbody>
<tr><td width="100%"><div>import timeit</div><br />
<div>def time(s):</div><div> '''returns time in microseconds'''</div><div> t = 1e6*timeit.timeit(s,'import test_kernel;k1=test_kernel.Kernel(1);k2=test_kernel.Kernel(2);',number=1000000)/1000000.</div><div> print s, t</div><div> return t</div><br />
time('k1.p_func(0)')<br />
time('k1.py_func(0)')<br />
time('k1.py_func1(0)')<br />
time('k1.py_c_func(0)')<br />
time('k1.py_func_c_common(0)')<br />
time('k1.py_func_common(0)')<br />
<br />
time('k2.p_func(0)')<br />
time('k2.py_func(0)')<br />
time('k2.py_func2(0)')<br />
time('k2.py_c_func(0)')<br />
time('k2.py_func_c_common(0)')<br />
time('k2.py_func_common(0)')</td></tr>
</tbody></table><br />
<h2>Timings : </h2><br />
<table border="0" cellspacing="0" class="zeroBorder" cols="8" frame="VOID" rules="NONE"><tbody>
<tr><td align="left" height="17" width="39"><br />
</td><td align="left" valign="top" width="167"><b><span style="font-family: Liberation Serif;">function</span></b></td><td align="center" colspan="5" valign="top" width="428"><b><span style="font-family: Liberation Serif;">time (μs)</span></b></td><td align="left" width="86"><b>(ns)</b></td></tr>
<tr><td align="left" height="17"><br />
</td><td align="left"><b><span style="font-family: Liberation Serif;">Optimization flag -></span></b></td><td align="left"><b>None</b></td><td align="left"><b>-O2</b></td><td align="left"><b>-O3</b></td><td align="left"><b>sum</b></td><td align="left"><b>(k1+k2)/2</b></td><td align="left"><b>penalty</b></td></tr>
<tr><td align="right" height="17">1</td><td align="left">k1.p_func(0)</td><td align="right">0.20178</td><td align="right">0.18321</td><td align="right" bgcolor="#00ff00">0.18035</td><td align="right" bgcolor="#00ff00">0.18845</td><td align="right" bgcolor="#00ff00">0.19368</td><td align="right">0.0000</td></tr>
<tr><td align="right" height="17">2</td><td align="left">k1.py_func(0)</td><td align="right">0.23224</td><td align="right">0.18599</td><td align="right">0.18393</td><td align="right">0.20072</td><td align="right">0.19541</td><td align="right">1.7345</td></tr>
<tr><td align="right" height="16">3</td><td align="left">k1.py_func1(0)</td><td align="right">0.21477</td><td align="right">0.18991</td><td align="right" bgcolor="#ff0000">0.19252</td><td align="right">0.19907</td><td align="right" bgcolor="#ff0000">0.19802</td><td align="right">4.3456</td></tr>
<tr><td align="right" height="17">4</td><td align="left">k1.py_c_func(0)</td><td align="right" bgcolor="#ff0000">0.23395</td><td align="right" bgcolor="#ff0000">0.19196</td><td align="right">0.19243</td><td align="right" bgcolor="#ff0000">0.20611</td><td align="right">0.19761</td><td align="right">3.9311</td></tr>
<tr><td align="right" height="17">5</td><td align="left">k1.py_func_c_common(0)</td><td align="right" bgcolor="#00ff00">0.19566</td><td align="right">0.18458</td><td align="right">0.19062</td><td align="right">0.19029</td><td align="right">0.19767</td><td align="right">3.9960</td></tr>
<tr><td align="right" height="17">6</td><td align="left">k1.py_func_common(0)</td><td align="right">0.21981</td><td align="right">0.18707</td><td align="right">0.18984</td><td align="right">0.19891</td><td align="right">0.19510</td><td align="right">1.4237</td></tr>
<tr><td align="right" height="17">7</td><td align="left">k2.p_func(0)</td><td align="right">0.20448</td><td align="right">0.18388</td><td align="right">0.18194</td><td align="right">0.19010</td><td align="left"><br />
</td><td align="left"><br />
</td></tr>
<tr><td align="right" height="17">8</td><td align="left">k2.py_func(0)</td><td align="right">0.21798</td><td align="right">0.18859</td><td align="right">0.18437</td><td align="right" bgcolor="#00ff00">0.19698</td><td align="left"><br />
</td><td align="left"><br />
</td></tr>
<tr><td align="right" height="16">9</td><td align="left">k2.py_func2(0)</td><td align="right">0.20413</td><td align="right" bgcolor="#00ff00">0.18124</td><td align="right">0.18194</td><td align="right">0.18910</td><td align="left"><br />
</td><td align="left"><br />
</td></tr>
<tr><td align="right" height="17">10</td><td align="left">k2.py_c_func(0)</td><td align="right">0.23114</td><td align="right">0.19166</td><td align="right">0.19238</td><td align="right">0.20506</td><td align="left"><br />
</td><td align="left"><br />
</td></tr>
<tr><td align="right" height="17">11</td><td align="left">k2.py_func_c_common(0)</td><td align="right">0.19860</td><td align="right">0.18783</td><td align="right">0.18745</td><td align="right">0.19129</td><td align="left"><br />
</td><td align="left"><br />
</td></tr>
<tr><td align="right" height="17">12</td><td align="left">k2.py_func_common(0)</td><td align="right">0.21609</td><td align="right">0.18747</td><td align="right">0.18640</td><td align="right">0.19666</td><td align="left"><br />
</td><td align="left"><br />
</td></tr>
<tr><td align="left" height="16"><br />
</td><td align="left"><b>Average</b></td><td align="right">0.21560</td><td align="right">0.18681</td><td align="right">0.18703</td><td align="right">0.19648</td><td align="left"><br />
</td><td align="left"><br />
</td></tr>
</tbody></table><br />
<div align="center" style="text-align: left;"><h2>Result :</h2></div>The best is to write separate C function and a python accessor function.<br />
<br />
<table border="1" bordercolor="#000000" cellpadding="3" cellspacing="0" id="uuvu"><tbody>
<tr><td width="50%"><b>task<br />
</b></td><td valign="top"><b>function</b></td><td width="50%"><b>penalty cost (<span style="font-family: Liberation Serif;">ns</span>)<br />
</b></td></tr>
<tr><td width="50%">C function + python accessor : base case</td><td valign="top">p_func</td><td width="50%"><br />
</td></tr>
<tr><td valign="top">cpdef instead of def</td><td valign="top">py_func</td><td valign="top">1.7345</td></tr>
<tr><td valign="top">calling a cdef class method instead of a function pointer attribute</td><td valign="top">py_func1,py_func2</td><td valign="top">4.3456</td></tr>
<tr><td valign="top">one extra c function call</td><td valign="top">py_c_func</td><td valign="top">3.9311</td></tr>
<tr><td valign="top">(def + cdef) instead of (cpdef)</td><td valign="top">py_func_c_common-py_func_common</td><td valign="top">2.5723</td></tr>
<tr><td valign="top">One C comparison vs one C function call</td><td valign="top">py_func_common</td><td valign="top">1.4237</td></tr>
</tbody></table><br />
<h2>Conclusion :</h2>As can be clearly seen that the results are clearly inconclusive :)<br />
This was a small test carried on my laptop with no controlled environment. Also thought the results seemed close to repeatable, nevertheless many trials should be conduction and each value should have a standard deviation also to check the repeatability. However one clear conclusion is do not forget to add optimization flags. Setuptools already does that for you.<br />
Also using a function pointer is not so bad after all. It would become more advantageous in case of more number of comparisons.<br />
Cython provides great speedups (who didn't know that :) ). The pure python version of py_func_common took 0.408<span style="font-family: Liberation Serif;">μs</span> for dim=1 and 0.518<span style="font-family: Liberation Serif;">μs</span> for dim=2<br />
These results are purely from python point of view. The effect of cdef/cpdef should also be considered in c/cython code which calls these functions.<br />
<h2>CAVEAT:</h2>I am no optimization expert. I have done this out of out of sheer boredom :)<br />
If anyone wants to verify, you are welcome<br />
Any information content is purely coincindentalPankajhttp://www.blogger.com/profile/16123915232914933383noreply@blogger.com1