Wednesday, May 18, 2016

Takeaways from RedisConf

I have been using Redis as a swiss army knife database, both at work, and for hobby projects. So I decided to attend RedisConf this year. Low cost of entry, more tech talks than marketing spiels, and a chance to meet Salvatore Sanfilippo, the creator of Redis, were the big draw to the event.
The event was a great introduction to upcoming and lesser used redis features (used by me that is). It was a fun and eye opening window into a lot of different domains and use cases. When all you sell is cheese, the concept of sour cream is interesting..
Some that blew my mind were:

  1. IoT: Prior to this conference, I had a limited understanding of it's use-cases. IoT, to me, was just using a web-controlled device. A light-bulb, or a smart fridge, that'd ping me if the milk ran out etc. I.e. a very Silicon Valley centric first world problem solver.

    The use-cases that Laura Merling provided in her talk, just blew me away:
    • Disney MagicBands: When you navigate your way through disney parks, your hotel automatically knows that you are coming, your door automatically unlocks & most importantly, if you get dangerously close to a moving ride, the ride adjusts itself (can stop) to get you out of harm's way!
    • The oil & gas industry: The oil rigs generate a lot of real time information on the equipment’s performance and health — pressure, temperature, flow rates and multitudes of other variables. IoT based solutions provide real time analytics for predictive and preventative measures. This link explains this further.
    • Factory safety: Remember the incident, when an automotive factory robot arm crushed a factory operator into a metal plate. This happened because the robot arm mistook the person as another part of the machinery. There have been around 26 deaths in the past 30 years in US alone. This number, although low, can be further brought down by judicial use of IoT tech (just like Disney's magicBand).
    • Home Automation: Home automation industry is fairly nascent right now. There is an app to control the lights, another one to control your locks, another one for your TV and so on. There needs to be push from the consumers for tighter integration of products for arbitrary control. E.g. to turn down the volume of the music system if someone is sleeping in the other room, or turn on the lights when the doors are unlocked, and so on.

  2. Distributed systems & the rise of DevOps: There were couple of talks by companies like Netflix and Scopely. The amount of data served and generated by companies these days is monumental.

    Avram Lyon from Scopely gave a behind the scenes look at the data usage. He talked about a simple mobile game where the goal is to hit oncoming zombies with some objects. His game has a modest 100,000 users active at any given time. To generate insights into the game (E.g. People in California like to throw rocks at zombies, whereas people in New York like to hurl trash-cans at them, or that no-one seems to like throwing hammer at zombies), the amount of data generated and recorded is huge. This is used to provide the development and marketing teams with actionable intelligence. Few TBs of data is not huge by today's standards, and conventional databases are ill-suited for this endeavor. Redis, along with it's other supporting actors lends well to this use-case.

    The conference was filled with 2-4 member teams who had set up massively scaled systems within 5-6 months using FOSS(kafka, spark, redis etc) on el-cheapo AWS cluster. Horizontal automatic scaling being the key takeaway. One doesn't need to be a star programmer to build these systems. The architecture has matured enough that systems can be deployed with minimal effort (relatively).

    Another emergent database concept was Eventual consistency. Unlike traditional database systems, that dictate that the view of the databases need to be the same for everyone, eventually consistent databases are like.. meh, it's not the end of the world if are able to access the new season of "House of Cards" few minutes after someone else. Heuristics are further tied to extrapolate behavior of the system to get massive scalability. Amazon, Netflix etc have this kind of systems. Take a look at this to get more detailed view.

  3. Hardware advancement: Traditionally, Redis has been used as a memcached replacement. Something that sits in-front of traditional sql databases. Apps like Twitter et al would store the hot keys in a massively distributed redis cluster (in order of TBs) and would later save these to some backend stores. This adds another layer of complexity. With the advent of flash memory, the backend stores typically now sit on flash memory. However, the newer NVDIMMs & NVMs perform orders of magnitude better than flash memory at just 1/10th the price of RAM. With the closing gap between RAM & Flash, the functionality and scalability of Redis is increasing by leaps and bounds.

  4. Redis Modules: Usual databases are just dumb datastores. Their main utility is to store the data efficiently in the memory and provide an easy way to retrieve it. All the intelligence to consume that data goes into the client code. Salvatore announced that starting 4.0, Redis would support loadable modules that can be used to offload client functionality to Redis. You could use this to keep track of counter rates or a running average or use simple ML algos to automatically classify the images being saved on the DB or apply some sort of filter on them. The possibilities are endless.
Smaller advancements in hardware typically bring in much bigger change in software, that exponentially and radically changes the application development paradigm.

About a decade ago, a simple classifier on an MPI Beowulf cluster took few weeks to implement. These days, I could use a python ML classifier to do the same thing in just few hours.

As complicated technology becomes more pervasive, the entry points to the tech itself gets easier, which further paves way for more complicated technology. Redis has already enabled a lot of industries, which are further revolutionizing everything from healthcare to robotics.
We live in interesting times.

Saturday, March 05, 2016

Programming and the art of shoe polishing

One of the most most undocumented occupational hazards of being a software engineer is that the rest of the world is not privy to our thought process, nor do they have a deeper understanding of what we do. Yes, coding is a part of it, and a very big part of it, but, it is understanding of the system that separates good engineers from the bad ones. Coding is easy, coding a system is hard. There is a whole lot of difference between learning english and writing a novel. Just as an exercise, let's imagine that the task at hand is polishing shoes. Our protagonist is the engineer(SE). The product manager(PM), the quality assurance(QA) engineer & shoe polisher(SP) have the supporting roles in this play.

Bad Engineer's Life
PM: I want our company to polish shoes. We will provide supplies to our shoe polisher team who will polish the shoes.
SE: I'll get on it.

- Distribute shoes to each SP(shoe polisher).
- Distribute polish to each SP.
- Ask each SP to use the given polish on each shoe.
- Take shoes from them.

QA: Hey! Your team used brown polish on black shoes. I'm filing a bug.
SE: Oops. I didn't know that. Let me fix it.

- Distribute shoes to each SP(shoe polisher).
- Each shoe has a COLOR.
- Distribute polish to each SP. Each polish has a COLOR.
- Ask each SP to use the given polish(COLOR) on each shoe(COLOR).
- Take shoes from them.

QA: Hey! Your team put black leather polish on my black canvas shoes. I'm filing a bug.
SE: Oops. Hey, why would the customer even want to polish canvas shoes?! Please test my system  properly.
QA: I don't care. You should not ruin your customer's shoes.
SE: Let me talk to PM.

PM: You should not ruin your customer's shoes.
SE: Ok.

- Distribute shoes to each SP(shoe polisher).
- If shoe is CANVAS then return to the customer.
- Each shoe has a COLOR.
- Distribute polish to each SP.
- Each polish has a COLOR.
- Ask each SP to use the given polish(COLOR) on each shoe(COLOR).
- Take shoes from them.

QA: Hey! SP still polishes my hip hemp shoes from SF. And why should the customer wait for shoe distribution to get his canvas shoe back? I'm filing a bug. Hey PM, the code is unstable!!
PM: Hmmm :(
SE: Let me fix it.

- Get shoes from Customer.
- Return shoe if NOT leather shoes.
- Distribute shoes to each SP(shoe polisher).
- Each shoe has a COLOR.
- Distribute polish to each SP.
- Each polish has a COLOR.
- Ask each SP to use the given polish(COLOR) on each shoe(COLOR).
- Take shoes from them.

QA: After 1 work day, the shoes don't look polished. Something is not working.
SE: I don't know, it works for me. Can you try it again and let me know?
-- After one day --
QA: See.. I told you so. I'm filing a bug.
SE: Oops.. the polish got over after a day. Let me add more polish each day.

- Get shoes from Customer.
- Return shoe if NOT leather shoes.
- Distribute shoes to each SP(shoe polisher).
- Each shoe has a COLOR.
- Distribute polish to each SP.
- Each polish has a COLOR.
- Ask each SP to use the given polish(COLOR) on each shoe(COLOR).
- If SP is working more than a day, then distribute more supplies to each SP.
- Take shoes from them.

QA: The product passes the test cases.
PM: We can sell this code to SUPER-MEGA-STORE then. Let us do some scale testing. They order around 100 shoe polishes a day! This could be a big order for us. I feel like Steve Jobs.
QA: Sure, whatever, I'll test out the cases.

QA: PM, we polish 100 shoes in a week! We need to handle that many in a day!
PM: SE.. make it so!
SE: I'll try. We can handle around 15 shoes in a day, we need to get 7 times as many SPs, maybe 6 times if we get stronger and faster SPs.
PM: Hmmm... We'll have to pass the cost to SUPER-MEGA-STORE

SUPER-MEGA-STORE: We like the fonts on sign board outside your establishment, take our money                                           and give us the product please.
--- After Deployment ---

SUPER-MEGA-STORE: Hey, nobody wears black leather shoes in our Sunnyvale store. We have too many unused black polish cans in our stores right now.
SE: I'll fix it.
SUPER-MEGA-STORE: Hey, one of the SP died while working. The shoes are now piling on!
SE: I'll fix it.

- Get shoes from Customer.
- Return shoe if NOT leather shoes.
- Distribute shoes to each SP(shoe polisher).
- Each shoe has a COLOR.
- Distribute polish to each SP.
- Each polish has a COLOR.
- Ask each SP to use the given polish(COLOR) on each shoe(COLOR).
- If SP is working more than a day.
- If polish(COLOR) is over, replace it.
- If SP is dead, get new SP.
- Take shoes from them.

SUPER-MEGA-STORE: Hey, we want canvas shoes to be washed too.
SE quits.

This happens over the period of a year. SE's own unscalable and spaghetti code have tormented him. The fluidity of requirements from the customer adds fuel to the fire. When one SE quits, other SE takes his place, mutating this code even more into an unrecognizable mess.

Now let's see what happens with a more experienced engineer:

Good Engineer's Life
PM: I want our company to polish shoes. We will provide supplies to our shoe polisher team who will polish the shoes.
SE: I'll get on it. What kind of shoes though?
PM: Leather shoes.
SE: Any specific set of colors? Google lists some common colors as BLACK, BROWN, NEUTRAL, WHITE, CHERRY RED & CORDOVAN.
PM: Hmmm..
SE: Also what about speciality leather shoes like those make with crocodile leather?
PM: Market research shows that people mostly wear BLACK and BROWN shoes, so let's stick with that.
SE: Any particular brand of polish you want me to use.
PM: Use the CODEYMAN brand. We can get bulk discount from them.
-- A week later --
SE: Hey I found out that one box of polish can polish about 250 shoes. I also spoke to my SP friend. He can shine 10 shoes in a day. Do you know about how many shoes we'll be shining a day?
PM: You ask too many questions, I somewhat hate you. SUPER-MEGA-STORE might be interested, but that is just a rumor I heard. But you haven't yet started writing code.
SE: So how many shoe shines a day?
PM: About 100 maybe?
SE: So we need about 10 SPs working each day. We can have 20 SPs who work on two rotating shifts. This would reduce the downtime if something were to happen to them.
PM: Stop doing a PhD in this! I want the code.
SE: Also 100 shoes a day is 700 shoes a week, which means 3 shoe cans would be used in a week. We need to keep more inventory of cans, to keep SPs from waiting on a polish can if someone else is using it. You can use this to set a price point for the product.
PM: I hate you! Give me an ETA.
SE: I'll sketch some informal design plans and revert in 2-3 days.

.. Time goes on ..
SE: PM, I'm going to be code complete in some time. I want the QA to add test cases.
PM: So the product is done?
SE: No, I've just written the code and did some unit testing. System testing etc will take at least a  month.
PM: So the product is done? I'll tell the Sales team to sell the product.
QA: I'll start testing it as soon as I'm done testing some other thing the SE doesn't care about.

SE: Here's the code

- Don't allow customers without BLACK or BROWN leather shoes.
- Let there be X BLACK polish cans and Y BROWN polish cans.
- Every 12 hours:
    a. New set of 10 SPs come in.
       - If less than 10 SPs come in. Pay the exiting employees overtime to pick up the slack.
b. SPs distribute the shoes amongst themselves, trying to maximize the amount of shoes of a particular COLOR.
c. Each SP takes the cans of polish corresponding to the shoe colors he has. He decrements the number of shoe polish cans available by 1.
- If the number of shoe polish hits the threshold T, he puts an order into the system to get more polish. This way they don't have to wait for polish if it runs out.
d. If SP runs out of shoes to polish:
- He'll put the polished shoes on the rack.
- Hel'll repeat steps from b. again.
e. When SPs are ready to leave:
- If less than 10 SPs leave then:
  - Find if SP is dead and put a replacement SP for the next rotated shift.
  - If SP is hiding, then ask him to leave.
- Take shoes from them.

PM: (During review): Please use spaces instead of tabs and use the function names according to  coding standards.

QA: Hey, edge condition issues. If I'm getting 100 shoes/day and one SP dies, the backlog never goes away.
SE: Oh yeah. Let's add a fudge factor of 11 SPs. Will take care in release.

PM: How is the quality?
QA: Seems to mostly work. Not many open issues I think.
PM: Great, let's deploy it.

SUPER-MEGA-STORE: Hey, your system doesn't handle bursts of more than 110 shoe-shines. We need a burst of 200 during Christmas holidays.
PM: Sure. We can include redundancy and burstiness of shoe-shine traffic in your tarrif. This would translate to hiring seasonal workers.

SUPER-MEGA-STORE: I need to get canvas shoes washed too.
PM: That feature will be supported in the next release.

SE: Shouldn't be a big deal. I'll design similar product for canvas-shoes and encapsulate both the products behind an api that is multiplexed by shoe type.
PM: Whatever.. just do it.

SE goes home.
SE: Honey.. I'm home.
Wife: Whatever! you spend all your time at work. I hate you.

.. SE is sad.

Friday, September 11, 2015

Mosh: The ssh client replacement you probably didn't think you needed.

I had to take a breather after writing the heading to avoid giving an over the top introduction to Mosh (mobile shell), but it deserves nothing less. It is a relatively new software (2012) from the well known MIT AI labs. Mosh is based on a new protocol SSP (State Sync Protocol), that runs over UDP instead of TCP. It does all the session management at the higher levels, which makes the connection incredibly robust. Few places where Mosh really shines on my setup (Mosh+Tmux):

1. When moving between networks (Wireless to Wired to VPN at home), the connection automatically migrates to the new IP. This wouldn't have been possible if TCP had been used. Consequently, my shell connectivity to servers stay up... No more banging the keyboard when ssh session abruptly freezes, or me trying to clear up a ssh terminal after the computer resumes from sleep. The terminal comes back up at the same place no matter what the network situation is. Its Magic!! I'm running tmux within mosh and the difference between local and remote machines has completely blurred.

2. Mosh has predictive typing! When typing on a ssh client, your keystrokes are relayed to the server, and displayed to you when it appears on the pseudo terminal at the remote end. This works great when you are working on a network with low latency, but try working from a slower connection from home, or god forbid, the caltrain, a coffee shop or from an airplane using the gogo inflight connection, there is a very significant delay between when you type and when the letters appear on your screen, which makes any sort of remote work next to impossible. Mosh gets around this issue by showing you what you type, bunching the updates to your screen in a low latency setup and later syncing the updates to your screen in background (e.g. if you type an 'i' in vi command mode, it should not show up, but it will show up when the network is very very laggy, however it will be erased when the state is synced from the remote server).

3. Mosh uses SSH protocol to communicate, so it automatically uses all the SSH configurations and writes to ssh logs. All one needs to do is call "mosh ", just like ssh. One doesn't need to muck with _any_ configuration files.

One important distinction between ssh and mosh is that ssh is a set of protocols whose utility goes well beyond providing a secure shell. It allows tunnelling, file transfer, X forwarding etc. None of those things are supported in mosh. Mosh was only meant to be a more robust shell.

That being said, Mosh hasn't been battle tested the way ssh has (specifically OpenSSH), but the design looks sound and simple with a relatively small attack surface, which gives me a lot of confidence in using it.

Go try it out.. and let me know what you guys think.

Sunday, May 18, 2014

You are not special..

Are you from a country that has some sort of currency? 
Now that you are intrigued by my idiotic question, please allow me the liberty of asking another one. If you are filthy rich in one country, lets say that you are a millionaire in US, living in a house that was featured on "Cribs", and you move to country that doesn't recognize your wealth. Do you instantly become poor? The answer is a resounding yes. Then all this proves is that you were rich only because you and other people around you, attached some value to the pieces of paper with numbers scribbled on them. The whole economy functions only because everyone sees some value in those pieces of paper. If I'm able to convince people that my torn sock has some value, then other people will think I'm rich. I'll be actually rich, when I believe that my torn sock is valuable, or if I exchange it for some belief system that I actually believe in (e.g. hard cash) before the reality catches up.

Let's look at social circles. There isn't much disparity between people within a circle, I.e. you wouldn't find people who are a lot poorer or a lot richer than you in your active circle of friends. Money act as differentiator here because it forces a change in priority, e.g. you wouldn't take a trip to Hawaii with friends if you are short on cash, but would definitely go on local hikes because it hardly costs anything.  Gradually friends turn into acquaintances and are sometimes forgotten. Slowly but steadily, you loose touch with people (Note that there are other reasons why people loose touch, but I'm just trying to point out this particular one). If you have lived a relatively fuller life, you'd accept the fact that people move in and out of our lives, just as we move through their's. Without our knowledge, forces of economics drive people closer to each other and rips us apart. All this, because we believe in a common currency!!

Forget money for a second, and think about culture. Think about it as money. What happens when a white catholic from America meets an upper caste Hindu brahmin from India? How does that interaction go? Let's assume for one second that the American has never heard or know anything about India, and the Indian doesn't know anything about America.. what then? Their interaction would completely depend on the place and environment in which they meet. If they meet at the Indian's house, the Indian would probably find the American less modestly dressed, lacking any knowledge of his religion, his traditions, bordering on blasphemy. The American would see the Indian in the same light if the meeting happened in America, he would find the clothes, food, traditions, deities with multiple hands completely weird. Both of them would find each other less cultured. Assuming both of them to be decent humans, the Indian would encourage the American to go to temples and introduce him to scriptures, on the other hand, the American would do the same if he is religious, if not he'd encourage the Indian to watch some opera or Broadway to get some "culture". If either of them disagree, there would be hostilities thrown around. 

Sadly that is how most of the wars start, because of both parties fail to understand the different point of view because it is so radically different. We got around the money problem by having currency exchanges which arbitrarily pits currencies against each other based on supply and demand and not the purchasing power it facilitates in the region where it is prevalent. I.e. if a 1 USD = 60 INR, and a couch costs 1 USD in USA but 30INR in India, who is richer, the guy with 30INR or the guy with 1USD? Still, these currency exchanges allow retention of economic status all over the globe... because everyone believes in some currency and they are more or less inter changeable.

Should we setup a culture exchanges like currency exchanges then ? If you can distinguish between Chardonnay and Sauvignon Blanc, should you be considered a high priest in Navajo? Would you consider Dr BalmuraliKrishna and Puccini similar? Would you see Baptism and Upnayanam in the same light regardless of your religious inclinations?
Of course that would be absurd. Cultural, economical, sociological inequalities are all absurd because they depend on a belief on an arbitrarily defined concept perpetuated by people. These concepts might be bit alien to people who have not interacted a lot with other cultures and have spent their whole life in a cocoon. 

Attacks on our rigidly held belief system tend to anger us. If the belief system is big enough, it might even start a war. A lack of a belief system, however, evokes nihilism, depression and general sociopathic tendencies. This seems to be a catch 22 situation. A belief system has given us distinct evolutionary advantage, even if has been something which defies logic. However, evolution's only concern is the survival of the species, not it's well being. (A recent study shows that chicken taste good to humans because chickens wouldn't have survived in the wild without human help, so humans started rearing tastier breeds of chickens while ignoring the other breeds). It may very well be the case that a fundamentalist faction exterminates all other races (as has been tried multiple times in the past) on earth.. We might end up becoming a paragraph in a grade school book. We might all be doomed to the same fate as Neanderthals when homo sapiens roamed the earth. Of course, the process won't end there. There'll be fractures in the victor's factions, a power struggle between the sub-factions, and eventual destruction of culture and advancement. 

Humanity is stuck in a feedback loop and is shaped by our own intellect, the limits of which, we cannot comprehend. We need a better approach, and we need to shape our own destiny as a race. What matters to you, doesn't matter to anyone else and vice versa. Why shouldn't we accept the disparity in the belief systems without being combative?

Sunday, October 13, 2013

Stop yearning for yester years..

Let me articulate a problem that people don't realize they have, and two words that cause this: Happiness & Sadness. We people like to be happy (whatever be the cost), and do not like to be sad. However, almost all of us, mistakenly assume happiness to be the exact opposite of sadness. This assumption is ingrained in our psyche, thrust onto us by our education system, our family, the society, the media, almost universally. But why is this wrong?

Take something banal from your own life, from say 10 years ago, say sharing a cup of road side chai with one of your friends/roommate, or taking a bus ride home on a bus that had barely any place to stand, or maybe just looking outside the window of your room (10 years ago). How do they make you feel now? Nostalgic, yes.. but also happy. How did you feel when you were actually living those moments? Just meh!

Now, take some actual defining moments in your life, when you were actually happy in that particular instant: these includes graduation, marriage, winning some contest that only 50 people in the world care about. How do they make you feel now? Those were just a blips in your existence when you were almost ecstatic. You were definitely happy during those few hours or few days of festivities, but your journey to those defining events, your college years, your courtship period is what made you more happy than those events itself.

So are you happy now? Most of us too jaded by the monotony would respond to this ambiguously, and even if someone replied that they were happy, they wouldn't be able to articulate why. Let me articulate it now for every one of you reading this post; You were happy in the past.. you will always be happy in the past.. 10 years from now, you will think you were happy at this very moment.

Lets switch to sadness. What makes you sad? Traffic Jam, chores at home, office work on weekend, existential crisis, when things don't work according to you.. in short pretty much everything.

What about some sad defining moments in your life? Loss of a loved one, an accident, getting fired from a job? This generally tend to generate two different kind of reaction, either you will look at them fondly (yes you heard me right) because it made you what you are today (the hardship you faced gave to skills to progress in life), or it will make you shed a tear whenever you think about it. The first kind generally tend to invoke happiness indirectly when you look back on your struggle, the second one might put you in a downward spiral or sadness, but you do eventually recover from it.

On an average, when are/were you sad? Now.. and when are/were you happy? In the past. This justifies the predisposition of people to yearn for the past, sometimes elevating it up to the utopian standards. This is a sad universal truth; in America, people want to live the way their parents lived, when the jobs were plentiful and everyone had a car and a house, but they discount and turn a blind eye to slavery. In India, the movies and soaps puts every one on a trip down memory lane, with all the bahus (daugher-in-laws) taking care of the house and guys bringing the moolah, but discount sati and dowry practices.

Everyone is chasing the proverbial happiness carrot, without realizing that it is something that cannot be achieved, only looked at. If you are not sad now, then you'll be happy in the future thinking about this very instant, so don't be sad because you are not happy now, because that'll rob you of your happiness in the future.

Learn from your experiences, and make new ones, but don't try to yearn for old ones.

PS: Inspired by the Republican rhetoric and Jon Stewart :)