Friday, September 11, 2015 by Sridhar Iyer
I had to take a breather after writing the heading to avoid giving an over the top introduction to Mosh (mobile shell), but it deserves nothing less. It is a relatively new software (2012) from the well known MIT AI labs. Mosh is based on a new protocol SSP (State Sync Protocol), that runs over UDP instead of TCP. It does all the session management at the higher levels, which makes the connection incredibly robust. Few places where Mosh really shines on my setup (Mosh+Tmux):
1. When moving between networks (Wireless to Wired to VPN at home), the connection automatically migrates to the new IP. This wouldn't have been possible if TCP had been used. Consequently, my shell connectivity to servers stay up... No more banging the keyboard when ssh session abruptly freezes, or me trying to clear up a ssh terminal after the computer resumes from sleep. The terminal comes back up at the same place no matter what the network situation is. Its Magic!! I'm running tmux within mosh and the difference between local and remote machines has completely blurred.
2. Mosh has predictive typing! When typing on a ssh client, your keystrokes are relayed to the server, and displayed to you when it appears on the pseudo terminal at the remote end. This works great when you are working on a network with low latency, but try working from a slower connection from home, or god forbid, the caltrain, a coffee shop or from an airplane using the gogo inflight connection, there is a very significant delay between when you type and when the letters appear on your screen, which makes any sort of remote work next to impossible. Mosh gets around this issue by showing you what you type, bunching the updates to your screen in a low latency setup and later syncing the updates to your screen in background (e.g. if you type an 'i' in vi command mode, it should not show up, but it will show up when the network is very very laggy, however it will be erased when the state is synced from the remote server).
3. Mosh uses SSH protocol to communicate, so it automatically uses all the SSH configurations and writes to ssh logs. All one needs to do is call "mosh ", just like ssh. One doesn't need to muck with _any_ configuration files.
One important distinction between ssh and mosh is that ssh is a set of protocols whose utility goes well beyond providing a secure shell. It allows tunnelling, file transfer, X forwarding etc. None of those things are supported in mosh. Mosh was only meant to be a more robust shell.
That being said, Mosh hasn't been battle tested the way ssh has (specifically OpenSSH), but the design looks sound and simple with a relatively small attack surface, which gives me a lot of confidence in using it.
Go try it out.. and let me know what you guys think.
Labels: Infotainment, Linux, Technology
Thursday, February 05, 2015 by Sridhar Iyer
NFV(Network Function Virtualization)
is the new buzzword on everyone’s lips in the
networking world. NFV, when done right, promises a big disruption to the
well-established network appliance businesses by virtualizing most of its core
functionality out, which in layman's term means that you don't need to spend
mucho moolah on custom built Layer3-Layer7 hardware appliances. At a fraction
of the cost, it can all be done in the software.
is all great for the businesses and network vendors, but for us, it forces us
to solve problems, the solutions for which you can't Google. These problems,
although critical for manageability, serviceability and billing, tend to get
ignored during initial stages of product development at other companies. One
such problems which I worked on was incrementing a counter... just i++, as
simple as that. E.g. if one packet comes in, you increment the packet_counter
me start with an overview of how it is handled at BOHVs (big old hardware
vendor). Most of the BOHV boxes consist of line cards, each line card will have
one or more asics, and each asic will deal with a fixed number of interfaces.
These have set memory regions carved for the up-keep of each interface. So if
you have to increment the packet_counter for interface G0/1/1, you just go to
0xblahblahblah and add 1 there. It goes for other counters too. Most of the popular
network OSes out there are based on custom built *nixes/Linux kernels. Having a
directly addressable memory region that stays in memory, is not paged out and
will not cause a cache miss when you are incrementing counters when more than a
million packets are coming in on just one of your interfaces, is a HUUUUGE
help. Since most of the hardware is modularized, if you add extra interfaces,
you are inadvertently adding extra memory to support that feature. However, you
can't add a new counter for an existing hardware (unless of course, a separate
reserved memory space is kept… which adds to the cost).
simple cookie cutter design goes out of the window, when the number of
interfaces or the type of services are not fixed or capped. Heck… we don't even
know whether the services are going to be deployed on bare metal or on a
how do you update around 150K counters in memory, when the packets are coming
in at multiple 10Gbps ports, there are multiple threads accessing the same
counter, and you need to be able to query the value of that counter at any time
with minimal or no disruption to the traffic. To make the matters simple, let’s
take a variable X, which needs to be incremented simultaneously from a variable
number of threads, around a million times a second.
you are right out of college, and you have read anything about multithreaded
programming, you'd suggest locks. However have threads taking and releasing
write locks million times a second is a horrible horrible idea.
what about atomic adds, the cute little primitives that gcc provides for
incrementing counters? Sounds like a great idea until you look at the exact use
case. Most of the counters to be updated are close to each other (so if you are
incrementing packets_received, you probably also want to increment
bytes_received). If you don't know how atomic add works, then you'll
inadvertently end up making the performance worse. In some platforms, it
invalidates the cache line, so if you need to increment the next counter, you'll
end up causing a cache miss. And with around 10 threads contending for the same
counter, your code's performance would still be terrible.
say we had a way to update these counters, how do we read them? Some control
process will have to go through each module, each thread and query them one by
one. Again a somewhat bad idea, if you need to maintain a running state of your
system in general.
require some crazy out of the box thinking to solve this problem... or maybe
not. We'll use the "other box" thinking here, use the solutions to
completely different problems and use it ourselves. I looked at various
database implementations, system implementation at Twitter, different “new age”
technologies that were trying to solve the same problem with a different set of
parameters in a different domain.
stack that did catch my eye was Statsd in conjunction with a no-sql db. Many
people had implemented exactly what we wanted, at least partly, and at much
smaller scale. All the users who need to increment X, need to open a connection
to Statsd, which will just aggregate the value of X over certain interval of
time and then send it to the DB. There are tons of fast, slave backed
performance centric DBs to chose from. This was a great design, however it
would never have scaled to our use case of million counter updates per second
(for 1 counter X).
could definitely have the users cache the entries locally and then send the
update at some interval, and we can club the updates together. But this
essentially meant that we are just providing some Statsd like functionality to
each thread... which we did. Adding our own daemon, which uses our own custom
built IPC stack, further improved the performance a bit.
then benchmarked the DB to find the upper limit of counter updates it could
sustain and then designed the system around it.
far, we just narrowed on where to store the counters, but we didn't know how to
get the values to the daemon talking to the DB. The inspiration for this part
came from the tried and tested "malloc". Counters of any type were
allocated from the same page, doled out in blocks, and maintained in the same
way. Since all the counters are in the same page, reading them and updating the
DB became incredibly fast. Each thread maintains it's own block and the
increments, at worst, take the hit of a cache miss.
could have one thread do the collection, but with NUMA, the performance will
again take a hit.
result of the above exercise was an xml file and two simple apis exposed to the
developers. All the other code was auto-generated using python. The amount of
research done, and the technologies used (python, no-sql, xml ... all the way
to Intel assembly, vtune, hyper-threading, prefetching and architecture) was
baffling, but needed to be done in order to squeeze the most amount of work
from a clock tick.
next time you see a counter, number of emails in your inbox, number of tickets
remaining, data used by your cellphone, think of all the sweat that went into
giving you that number J
Sunday, May 18, 2014 by Sridhar Iyer
Are you from a country that has some sort of currency?
Now that you are intrigued by my idiotic question, please allow me the liberty of asking another one. If you are filthy rich in one country, lets say that you are a millionaire in US, living in a house that was featured on "Cribs", and you move to country that doesn't recognize your wealth. Do you instantly become poor? The answer is a resounding yes. Then all this proves is that you were rich only because you and other people around you, attached some value to the pieces of paper with numbers scribbled on them. The whole economy functions only because everyone sees some value in those pieces of paper. If I'm able to convince people that my torn sock has some value, then other people will think I'm rich. I'll be actually rich, when I believe that my torn sock is valuable, or if I exchange it for some belief system that I actually believe in (e.g. hard cash) before the reality catches up.
Let's look at social circles. There isn't much disparity between people within a circle, I.e. you wouldn't find people who are a lot poorer or a lot richer than you in your active circle of friends. Money act as differentiator here because it forces a change in priority, e.g. you wouldn't take a trip to Hawaii with friends if you are short on cash, but would definitely go on local hikes because it hardly costs anything. Gradually friends turn into acquaintances and are sometimes forgotten. Slowly but steadily, you loose touch with people (Note that there are other reasons why people loose touch, but I'm just trying to point out this particular one). If you have lived a relatively fuller life, you'd accept the fact that people move in and out of our lives, just as we move through their's. Without our knowledge, forces of economics drive people closer to each other and rips us apart. All this, because we believe in a common currency!!
Forget money for a second, and think about culture. Think about it as money. What happens when a white catholic from America meets an upper caste Hindu brahmin from India? How does that interaction go? Let's assume for one second that the American has never heard or know anything about India, and the Indian doesn't know anything about America.. what then? Their interaction would completely depend on the place and environment in which they meet. If they meet at the Indian's house, the Indian would probably find the American less modestly dressed, lacking any knowledge of his religion, his traditions, bordering on blasphemy. The American would see the Indian in the same light if the meeting happened in America, he would find the clothes, food, traditions, deities with multiple hands completely weird. Both of them would find each other less cultured. Assuming both of them to be decent humans, the Indian would encourage the American to go to temples and introduce him to scriptures, on the other hand, the American would do the same if he is religious, if not he'd encourage the Indian to watch some opera or Broadway to get some "culture". If either of them disagree, there would be hostilities thrown around.
Sadly that is how most of the wars start, because of both parties fail to understand the different point of view because it is so radically different. We got around the money problem by having currency exchanges which arbitrarily pits currencies against each other based on supply and demand and not the purchasing power it facilitates in the region where it is prevalent. I.e. if a 1 USD = 60 INR, and a couch costs 1 USD in USA but 30INR in India, who is richer, the guy with 30INR or the guy with 1USD? Still, these currency exchanges allow retention of economic status all over the globe... because everyone believes in some currency and they are more or less inter changeable.
Should we setup a culture exchanges like currency exchanges then ? If you can distinguish between Chardonnay and Sauvignon Blanc, should you be considered a high priest in Navajo? Would you consider Dr BalmuraliKrishna and Puccini similar? Would you see Baptism and Upnayanam in the same light regardless of your religious inclinations?
Of course that would be absurd. Cultural, economical, sociological inequalities are all absurd because they depend on a belief on an arbitrarily defined concept perpetuated by people. These concepts might be bit alien to people who have not interacted a lot with other cultures and have spent their whole life in a cocoon.
Attacks on our rigidly held belief system tend to anger us. If the belief system is big enough, it might even start a war. A lack of a belief system, however, evokes nihilism, depression and general sociopathic tendencies. This seems to be a catch 22 situation. A belief system has given us distinct evolutionary advantage, even if has been something which defies logic. However, evolution's only concern is the survival of the species, not it's well being. (A recent study shows that chicken taste good to humans because chickens wouldn't have survived in the wild without human help, so humans started rearing tastier breeds of chickens while ignoring the other breeds). It may very well be the case that a fundamentalist faction exterminates all other races (as has been tried multiple times in the past) on earth.. We might end up becoming a paragraph in a grade school book. We might all be doomed to the same fate as Neanderthals when homo sapiens roamed the earth. Of course, the process won't end there. There'll be fractures in the victor's factions, a power struggle between the sub-factions, and eventual destruction of culture and advancement.
Humanity is stuck in a feedback loop and is shaped by our own intellect, the limits of which, we cannot comprehend. We need a better approach, and we need to shape our own destiny as a race. What matters to you, doesn't matter to anyone else and vice versa. Why shouldn't we accept the disparity in the belief systems without being combative?
Labels: Life, Rants, Zen
Sunday, October 13, 2013 by Sridhar Iyer
Let me articulate a problem that people don't realize they have, and two words that cause this: Happiness & Sadness. We people like to be happy (whatever be the cost), and do not like to be sad. However, almost all of us, mistakenly assume happiness to be the exact opposite of sadness. This assumption is ingrained in our psyche, thrust onto us by our education system, our family, the society, the media, almost universally. But why is this wrong?
Take something banal from your own life, from say 10 years ago, say sharing a cup of road side chai with one of your friends/roommate, or taking a bus ride home on a bus that had barely any place to stand, or maybe just looking outside the window of your room (10 years ago). How do they make you feel now? Nostalgic, yes.. but also happy. How did you feel when you were actually living those moments? Just meh!
Now, take some actual defining moments in your life, when you were actually happy in that particular instant: these includes graduation, marriage, winning some contest that only 50 people in the world care about. How do they make you feel now? Those were just a blips in your existence when you were almost ecstatic. You were definitely happy during those few hours or few days of festivities, but your journey to those defining events, your college years, your courtship period is what made you more happy than those events itself.
So are you happy now? Most of us too jaded by the monotony would respond to this ambiguously, and even if someone replied that they were happy, they wouldn't be able to articulate why. Let me articulate it now for every one of you reading this post; You were happy in the past.. you will always be happy in the past.. 10 years from now, you will think you were happy at this very moment.
Lets switch to sadness. What makes you sad? Traffic Jam, chores at home, office work on weekend, existential crisis, when things don't work according to you.. in short pretty much everything.
What about some sad defining moments in your life? Loss of a loved one, an accident, getting fired from a job? This generally tend to generate two different kind of reaction, either you will look at them fondly (yes you heard me right) because it made you what you are today (the hardship you faced gave to skills to progress in life), or it will make you shed a tear whenever you think about it. The first kind generally tend to invoke happiness indirectly when you look back on your struggle, the second one might put you in a downward spiral or sadness, but you do eventually recover from it.
On an average, when are/were you sad? Now.. and when are/were you happy? In the past. This justifies the predisposition of people to yearn for the past, sometimes elevating it up to the utopian standards. This is a sad universal truth; in America, people want to live the way their parents lived, when the jobs were plentiful and everyone had a car and a house, but they discount and turn a blind eye to slavery. In India, the movies and soaps puts every one on a trip down memory lane, with all the bahus (daugher-in-laws) taking care of the house and guys bringing the moolah, but discount sati and dowry practices.
Everyone is chasing the proverbial happiness carrot, without realizing that it is something that cannot be achieved, only looked at. If you are not sad now, then you'll be happy in the future thinking about this very instant, so don't be sad because you are not happy now, because that'll rob you of your happiness in the future.
Learn from your experiences, and make new ones, but don't try to yearn for old ones.
PS: Inspired by the Republican rhetoric and Jon Stewart :)
Wednesday, May 29, 2013 by Sridhar Iyer
Let's say that you have 5 five friends and you want to find what they are up to. How do you interact with them?
- You meet/call/mail them each individually and talk to them.
- You have a conference call, facebook/watsapp group message, group email thread, where you just ask what is going on. People respond is random order, but you process the response serially. You lose some of that personal one on one conversation that you might have had, but you still easily touch base with them. All of them talk to each other, crack some common joke, and after a while the group messaging stops. You have effectively reduced the amount of work needed to communicate, and you have checked up on them so they know that you exist, but you still managed to alienate them... but you might not have spent the energy on an individual mail had it not been for the group chat, so this is still a win win situation.
- Now imagine that you ask your friends to continuously tell you what they are doing, but instead of communicating serially, they start shouting simultaneously. This is the craziest and the most inefficient way to communicate and that is being thrust down our throats by all the card interface craziness going on.
Let me explain; Pinterest, google currents, flipboard, and as of today, google plus, use card interface. What that means is they'll have each nugget of information on cards, and depending on the application, you can swipe, close, add, append etc these cards. Each of these cards takes a hell lot of screen space and has all the gooey gradient goodness with shadows and what not. Looks great and works great too.. if all you are reading is few occasional tidbits. If you are one of them, just imagine facebook stream arranged into huge squares and thrown about all over your screen.. or just imagine if each email in your inbox was delivered as these boxes. This interface does makes sense if you are discovering content, but not if you are going through a curated set of content, such as RSS feeds or social networking stream.
There have been tons of UI studies where the eyeball of the users were tracked and found to gravitate towards left hand edge of the screen/web page, by forcing these eyeballs all over the screen, these companies are causing a disservice to their customers.
Now, admittedly, I'm still pretty hung up on Google Reader, but that is how I get to the gobs and gobs of information, I read about 20 articles daily, but for that, I go through 200+ feeds daily. The ability to go through everything and selectively read what I want is what makes stream readers such a good interface. Same thing holds for facebook. You scroll endlessly until something catches your eye, then you click that and explore. Imagine if every object in your news feed, the ones that you don't care about, took as much time; you wouldn't be able to process all that information in time.
With that said, card interface is not all bad. They work on mobile devices, but the powers that be should realize that it doesn't make sense on a desktop, and should show a device specific interface. It might even make sense for something like stumbleUpon.. but please.. not my email, not my feeds and not my social stream.
Friday, May 10, 2013 by Sridhar Iyer
Are you living in the perfect world of your own? Do you want something to change? Chances are that you do. If you don't then you probably have achieved nirvana and are reading this blog from a higher state of existence.
Dial your life back by a few years, dial it as far as you can go, access your oldest memory. What were your problems back then? You were probably not old enough to understand any financial problem that your parents might have, you were probably fed by someone, someone did your laundry, somebody chauffeured you around, you had friends who could play with you, and even a discarded carton was a spaceship to you... but what were you worried about? Grades!! thats right. Even though, your life was perfect, so much so that songs have been written about fantasies to get back to childhood, you worried about that 1/2 a grade that you didn't get in history. You thought that life would be perfect if you could just get that 1/2 a grade. Of course, now you know that, your life was perfect, and it was immature of you to want that 1/2 a grade.. or maybe you think that the want of that 1/2 a grade shaped who you turned out now.
Fast forward a few years now, just before you go to college. You think that you could ace all the classes you took few years back, if you could do it now.. and you'd be right. Now you are hot headed enough to ignore any problems that your parents might have, and think of yourself as a new generation of whatever. Still everything is paid by family so life is good. But you are still worried about grades and the college you get into. You obsess over it for a year or two. And you think that life would be good if you could just get into a good college. Of course, now you know that, your life was perfect, and it was immature of you to base your whole life around the want of the greatest university.. or maybe you think that the want shaped who you turned out now.
Few more years into your life, you are now in your prime, just before you graduate. You have the closest friends you'll ever make in your life close to you, you have a clear goal of what you want to do in life. Your bills are still paid by your family and you are too busy to be worried by any problems back home. And what do you obsess about? Your job.. that's right. You obsess and obsess over finding the right job. Maybe you find it, maybe you don't. You still think the life is not perfect.
A good university, a good job, a new gadget... everything gets old and transparent. We humans are evolved only to notice change, not something that is stagnant.
Few years into the job, your life is still not perfect, you have a good job, but now you obsess over your personal life. You think that your life would be perfect if you just got married. Even with all the accolades you won, all the friends you have, you don't have a perfect life... And of course, we want to make it perfect, don't we.
So fast forward a few years, you'll get married. This is where my experience ends.. of course there are still things in my already perfect life that I need to change to make my life "perfect"... But we as a species should realize that our lives are perfect the way they are. Life changes, and since a non-believer in destiny, you have the reigns of your life in your own hands. Life adds more code to your monolithic operating system, and you should know that more code mean more bugs, but remember that more code also means more functionality. If life didn't have imperfections then you'd not move the train of your life to a better destination.
Don't think about that half a grade, your college search, your job search, your spouse search, or any other imperfections that you handled. Think about your schooling, your college, your friends, your job, your spouse. You'll see that you have a rich life.. enjoy it and be thankful for the imperfections that makes it richer.. cause even if you are billionaire space cowboy with pet unicorns, sh*t will alway fly.
Labels: Rants, Zen
Friday, February 08, 2013 by Sridhar Iyer
I've made it very clear in my previous blog posts/status messages and random conversations, my disdain for closed source systems, primarily Windows and MacOSX. I was a happy camper in my last company because I got to use Ubuntu with wmii as my primary work environment.. and I was churning code like there was no tomorrow.
That all changed when I joined Cisco. Even though you can install whatever you want on your machine, I decided to go with the IT supported Windows 7 so I don't have to muck around with any of the internal proprietary softwares that is generally vogue in any company of Cisco's size. At the end of the day, I had to vnc over to a linux/solaris build server so I didn't care much. Windows was just a thin client, and for most part it didn't get in the way. Operating systems don't really matter if all you need them for is IM, browsing and email (the same software is available on all the oses anyway).
Fast forward 3 years, I needed to refresh my laptop. I decided to go with a mac book pro this time, only because it had much higher hardware specs than it's counterpart. I wondered if I had made a wise choice. In the first one week of badly missing the dock (not the OSX dock, but the hardware dock you put the thinkpad on) and the pin mouse(most people think that it is useless, but it is quite useful if you know how to type and don't want to move your hand from the keyboard), my experience with mac seemed like a self fulfilling prophecy.
I was ready to hate this mac book pro, and write rants for the next 3 years.. but then I really started using the damned machine. With few tweaks, I got my .vimrc, .bashrc and .screenrc to work on mac (some of these files, I've been building for over 10 years!). Adium, which to my surprise is implemented on libpurple, filled the void pidgin had left. And chrome.. what can I say about this fantastic piece of software, once you log in to the browser, you won't know the difference if you are browsing on windows, osx or linux. So OSX just got out of the way as Windows 7 did and let me do my work as I used to do earlier. All the apps like evernote, rescueTime that I depend on pretty heavily just worked as expected.
But then the fun started, when it started seducing me to its vertically integrated ecosystem. To install some apps, I had to get AppleID, so I got one. To get the nifty notifications I configured my facebook, twitter, gmail and flickr accounts, which automatically imported the calendar and contacts, so with barely any effort, I was using OSX's calendar, notification system and contacts. Coming from the windows/linux world, I was used to either open a software and search for relevant entries or just grep for configs/data in the respective directories, but spotlight changed all that. Searching for anything became much easier. To my surprise, some software ran faster on a windows virtual machine on mac than on windows running on my old laptop (with similar memory allocation). The douchey way to scroll became "natural". It just looked like a really really well designed ubuntu system (I know people will find it funny because Ubuntu has been taking GUI cues from OSX.. but I come from that world).
The hardware itself is a joy to use. The 15'' retina display is great and it makes the HD IPS display of my photo processing workstation look like the screen of windows with 1024x768 resolution with 256 colors. Keyboard lights that turn on when lights are dim, extremely thin dimensions,a great touchpad, the completely silent 256G SSD and the monstrous 16G RAM makes this machine a joy to use.
Am I a fanboy? No.. and probably will never be. Why? Well the cost of the machine with all the connectors etc have run to about $3000.. Unless someone is paying for it, there is no way in hell I'll be shelling that amount(in this case my company pays for it). I hate non-standard keyboards. I don't like the command key, I don't like the omission of page-up/down, end/home keys, and I'll never get used to the "MAC" way of doing things because I'll always be working in an environment where I'll be using Linux and working with people who own heterogeneous set of machines and operating systems, so I'd want something that doesn't stump a non-mac user. Even if the keys were eliminated from the laptop itself, they shouldn't be disabled when a usb keyboard is connected. Also, don't know what'll take Apple to put a hardware dock support to the mac books, but connecting a keyboard, a mouse, two displays and a power chord is a chore. The available third party solutions are nowhere close to Lenovo docks. The battery life of my 3 year old Lenovo laptop is still much much better than the new Mac Book Pro. Also MS office softwares work much better on Windows machine.
So what's the final verdict; undoubtedly I'm in love with the machine and is much better than my old Lenovo laptop, but since the hardware configs were different, it is not an apple to apples comparison. Do I still feel the same way about closed source software.. absolutely. Would that stop me from using a decked up mac book pro.. absolutely not :).
PS: Incidentally I wrote this entry, on MBP, sitting in a street corner starbuck, sipping a skinny latte, in San Francisco. I don't know the hipster/douchebaggy smugness is my own or one induced by this vile machine :)